fix(tools): stop testing on the text of other files, and ungate contractaudit

Six tests asserted on the wording and formatting of source and config files
rather than on behaviour: four injected text into fixtures and checked that a
regex noticed, and two asserted on api.jsx's contents. They break on an extra
space, a newline, a rename, or a comment that merely mentions the token -- the
single-fetch check had already false-positived on a comment, and I patched it
with a comment-stripper instead of deleting it. A test that cannot tell "the
code is wrong" from "the file is formatted differently" tests the wrong thing,
and the maintenance cost is permanent.

Deleted: tools/contractaudit/negative_test.go in full, and the source-scanning
"api.jsx holds exactly one fetch call" check with its stripComments helper. The
dashboard suite still asserts the browser mutation contract behaviourally, per
method, by inspecting the request each one makes -- which is where the real
defect was found. 142 UI tests pass; the 45 methods remain fully covered by the
two inventories and the completeness check.

The tool's own decoding is now testable without a file: parseDeclarations takes
an io.Reader, so its rules are exercised with in-memory input, and only the
absent-file branch still touches a path (which is never created).

contractaudit is no longer part of task check. Its surfaces are discovered by
scanning source text, so a rename or reformat elsewhere can change its output
with nothing actually wrong -- it reported 16 live routes as unconsumed once,
when the dashboard's request helper was renamed. A check that can fail for
reasons unrelated to what it measures must not be able to break a build. It now
matches tools/reachaudit: advisory, run by hand, --strict available on request.

Also removes five unused indirect requires from tools/go.mod (go mod tidy).
They were residue from the reflection-based first implementation of the RPC
scan, which was replaced by text scanning without tidying. -mod=readonly cannot
catch a superfluous require, so nothing would ever have found them.

Rewrites docs/prds/contract-conformance-audit.md, which claimed RPC coverage was
derived by reflection over the exported client type -- it never was -- listed
surface sizes of 43/12/52 against real counts of 45/21/49, described three
classification classes where the code has four, invented a five-surface Shape
with files that were never written, and blamed raw fetch calls for a 401 defect
that did not exist. Each false claim is now recorded in an errata section rather
than quietly dropped, and the document states plainly that the audit has found
nothing actionable: the 415 refusals and the duplicated client plumbing were
both found by reading the handler and reproducing the request.

Testing rule added to the document, since this is the second time: assert on the
observable effect, never on the content or formatting of generated or
hand-authored text and config files.
This commit is contained in:
sam committed 2026-09-15 14:17:53 +10:00
1 parent cb69cf1979
commit 6fd76ae550
10 files changed
+217 -592

No files matched your search

+16 -12
View File
@@ -226,7 +226,7 @@ tasks:
- go -C tools test -mod=readonly ./... -count=1
check:
desc: "Full quality gate: fmt, go fix, proto, vet, lint, build, test, tools tests, ui tests, contract"
desc: "Full quality gate: fmt, go fix, proto, vet, lint, build, test, tools tests, ui tests"
cmds:
- task: fmt
- go fix ./...
@@ -243,12 +243,6 @@ tasks:
# The dashboard's suite runs here too. Left out of the gate it rots
# silently: nothing else executes it, so a broken DOM primitive ships.
- task: test:ui
# Contract conformance. Static and dependency-light, so it belongs with
# the per-edit checks rather than next to the end-to-end suite. It runs
# -strict here: every unconsumed surface must be declared in
# docs/contract-declarations.json with a reason and a tracker, and a
# stale declaration fails the same way an undeclared one does.
- task: contract
contract:
desc: "Report contract conformance: declared surfaces, their consumers, and the drift between them"
@@ -257,14 +251,24 @@ tasks:
dashboard HTTP API -- and classifies each subject as consumed,
declared-unconsumed, undeclared, or a stale declaration.
Advisory when run directly; `task check` runs it with -strict so an
undeclared or stale entry fails the gate. Declare an intentionally
unconsumed surface in docs/contract-declarations.json with a reason
that states the consequence and the tracker that owns the work.
ADVISORY, and deliberately NOT part of `task check`, matching
tools/reachaudit. The surfaces are discovered by scanning source text:
the .proto files, the adapter bodies that call the generated client, the
route registrations in internal/webui, and the path literals under
ui/src. That makes it useful for a periodic survey and unsafe as a gate
-- a rename or a reformat elsewhere in the tree can change its output
without anything being wrong, and it has already produced a batch of
false "unconsumed" readings once when the dashboard's request helper was
renamed.
Read the findings as a prompt to go and check, not as a verdict. To make
it fail on an undeclared or stale entry, pass --strict by hand:
go -C tools run ./contractaudit --repo-root .. --strict
See docs/prds/contract-conformance-audit.md.
cmds:
- go -C tools run ./contractaudit --repo-root .. --strict
- go -C tools run ./contractaudit --repo-root ..
release:preview:
desc: "Show the release notes that would be generated (VERSION=1.2.0, or GATEWAY=/RUNTIME= to version the two components separately)"
+5 -4
View File
@@ -6,8 +6,9 @@
"is an undeclared finding and an entry that is no longer true is a stale one,",
"and both fail a -strict run. See docs/prds/contract-conformance-audit.md.",
"",
"Managed by tools/contractaudit. Reasons state the consequence, not 'TODO':",
"this file is the only place the debt is legible."
"Managed by tools/contractaudit, an ADVISORY report that task check does not",
"run. Reasons state the consequence, not 'TODO': this file is the only place",
"the debt is legible."
],
"surfaces": {
"proto/state/v1/state.proto": {
@@ -24,11 +25,11 @@
"blindSpots": [
{
"surface": "internal/webui",
"reason": "A path counts as a consumer only when it is the first argument of a call in code. The function need not be named -- naming it broke on rename, when matching `req|fetch` reported 16 live routes as unconsumed after the dashboard's helper became `request(` -- but a bare constant, a dead array, an object property, or a commented-out call must not count, because crediting any quoted literal makes the surface pass on a file that issues no requests at all. Limits: a path passed as any argument other than the first is not seen (accepting a comma would start crediting array elements); a request inside a template-literal interpolation is not seen; a path built from a bare /api fragment is reported as unparsable; and a route is credited to every HTTP verb, so a route whose specific verb has no caller reads as consumed. Routes whose consumer is outside the dashboard (the asset mount, the health probes, the capture-intake webhook) are excluded from this surface by design: they have different consumers and are not part of the dashboard API contract."
"reason": "Extraction is textual, which is why this report is not gated. A path counts as a consumer only when it is the first argument of a call in code, so a bare constant, a dead array, an object property, or a commented-out call does not count; the calling function need not be named, so a rename is survivable. Limits: a path passed as any argument other than the first is not seen (accepting a comma would start crediting array elements); a request inside a template-literal interpolation is not seen; a path built from a bare /api fragment is reported as unparsable rather than silently contributing nothing; and a route is credited to every HTTP verb, so a route whose specific verb has no caller reads as consumed. Routes whose consumer is outside the dashboard (the asset mount, the health probes, the capture-intake webhook) are excluded from this surface by design: they have different consumers and are not part of the dashboard API contract."
},
{
"surface": "proto/state/v1/state.proto",
"reason": "Consumption is derived by scanning adapter bodies for calls through the generated client. An RPC reached only through runtime-loaded code (Yaegi) or reflection would read as unconsumed. No such path exists today."
"reason": "Consumption is derived by scanning adapter bodies for calls through the generated client, so an RPC reached only through runtime-loaded code (Yaegi) or reflection would read as unconsumed. The scan also keys on the generated client field being named 'client'; renaming that field would report every RPC as unconsumed. No such path exists today."
},
{
"surface": "proto/gateway/v1/chat.proto",
+130 -133
View File
@@ -1,177 +1,174 @@
# Contract conformance audit — decision
**Status:** Decided, implementing
**Status:** Implemented as an **advisory** report. Deliberately **not** part of
`task check`.
**Date:** 2026-09-15
**Authority:** sibling to `archie-core-06ag` (standing reachability audit) and
`docs/architecture/dependencies-and-contracts.md`. Does not replace either.
**Extends:** `docs/architecture/generated-documentation.md` (the `docsgen check`
gap it admits at line 331), `docs/prds/config-example-drift.md` (step 3).
`docs/architecture/dependencies-and-contracts.md`. Replaces neither.
**Open:** withdraw it, or rebuild it structurally. See "Open decision" — not to
be resolved by hardening the regexes.
## Question
Archie has strong contract enforcement at exactly one boundary and none at the
rest. `proto/*.proto` is generated, linted against `main`, and regenerated in
`task check`; `proto:check` fails on any drift. The **application** boundary —
where the dashboard, the generated documentation, and the shipped asset bundle
meet their consumers — has no equivalent, and three separate documents already
admit drift there in writing.
Does every declared contract surface have a consumer, or an explicit declaration
that it has none?
Should contract conformance be a standing, runnable audit like `reachaudit`?
Reachability (`tools/reachaudit`) asks the complementary question — *is this code
in a shipped binary?* — and cannot see a capability that is **reachable and
unconsumed**. `ratelimit` was in the binary closure and called by nothing
(`archie-core-6wxb`); reachability would not have found that.
## Decision
**Yes, and it is a sibling of reachability, not a replacement.** Reachability asks
*"is this code in a shipped binary?"*. Conformance asks the complementary
question: *"does every declared contract surface have a consumer, or an explicit
declaration that it has none?"* A capability can be reachable and unconsumed —
`ratelimit` was in the binary closure and called by nothing
(`archie-core-6wxb`). Reachability would not have found that; this would.
Yes, as an advisory survey. `tools/contractaudit` classifies each subject of each
surface:
The audit reports three classifications, and **the third is the only finding**:
| Classification | Meaning | Action |
| Classification | Meaning | Fails `--strict`? |
| --- | --- | --- |
| `CONSUMED` | a non-test client call site exercises it | none |
| `DECLARED-UNCONSUMED` | named in the declaration file with a reason and a tracker id | none — it is tracked work |
| `UNDECLARED-UNCONSUMED` | nobody reaches it and nobody declared it | **the finding** |
| `CONSUMED` | a call site exercises it | no |
| `DECLARED-UNCONSUMED` | allowlisted in `docs/contract-declarations.json`, with a reason and a tracker | no |
| `UNDECLARED-UNCONSUMED` | nobody reaches it, nobody declared it | **yes** |
| `STALE-DECLARATION` | an allowlist entry that is no longer true | **yes** |
`DECLARED-UNCONSUMED` is a status, not a defect — the same posture `reachaudit`
takes with `unreachable-tracked`. The declaration file is an allowlist that
**must match reality exactly**: an entry naming a surface that is now consumed is
itself a failure (stale declaration), because a stale allowlist is how a gate
quietly stops checking anything.
There are **four** classes and **two** of them fail. The allowlist must match
reality exactly, and it fails in both directions: a missing entry is a finding,
and an entry that outlived its subject is a finding, because a stale allowlist is
how a check quietly stops checking anything.
### Surfaces, and what proves each is consumed
**Not gated.** `task contract` is advisory and `task check` does not run it,
matching `tools/reachaudit`. The surfaces are discovered by scanning source text,
so a rename or a reformat elsewhere in the tree can change the output with nothing
actually wrong. It has already produced 16 false "unconsumed" readings once, when
the dashboard's request helper was renamed. A check that can fail for reasons
unrelated to what it measures must not be able to break a build.
| Surface | Contract of record | Consumed means |
## Surfaces actually built — three, not five
| Surface | Subjects | "Consumed" means |
| --- | --- | --- |
| `proto/state/v1` `StateStoreService` (43 RPCs) | the `.proto` | an exported method on `staterpc.Client` whose body calls that RPC |
| `proto/gateway/v1` `ChatService` (12 RPCs) | the `.proto` | an exported method on `gatewayrpc.Client` calling that RPC |
| `internal/webui` HTTP routes (52) | the `registerX` call sites | a path literal in `ui/src/base/api.jsx` or an `EventSource` subscriber |
| `docs/data/generated/contracts.json` | `tools/docsgen` | a committed-artifact comparison (`docsgen check`) |
| `ui/dist` | `ui/src` | a rebuild that produces no diff |
| `proto/state/v1` `StateStoreService` | **45 RPCs** | an adapter body calls `client.<Rpc>(...)` |
| `proto/gateway/v1` `ChatService` | **21 RPCs** | same |
| `internal/webui` dashboard routes | **49 routes** | a path literal appears as a call argument under `ui/src` |
RPC coverage is derived by **reflection over the exported client type**, not by
parsing call sites. If a method exists, the generated `grpc` interface already
forced it to make a real RPC call, so presence of the method *is* the proof of
consumption. This keeps the check exact without a call-graph.
Consumption for the gRPC surfaces is established by **scanning adapter source
text** for calls through the generated client, not by reflection. See the errata
below: the first revision of this document claimed reflection, and the first
implementation did use it, but the shipped mechanism is a regex and the document
was not updated to match.
## The two gaps this closes immediately
## What this audit has actually found
Both are already admitted in writing; neither is implemented.
**Nothing actionable.** On the current tree it reports 0 undeclared and 0 stale.
Its only allowlisted entries are two RPCs superseded by their streaming variants
(`ListCaptures`, `ListUndispatchedCaptures`), which were already known from the
source comment on `staterpc.Client.ListCaptures`.
**1. `docsgen check` is documented and does not exist.**
`generated-documentation.md:118-121` advertises `docsgen data|asyncapi|all|check`
and line 336 gives the invocation `go -C tools run ./docsgen check --repo-root ..`.
`tools/docsgen/main.go` defines exactly two flags, `--repo-root` and `--out`, and
`run()` unconditionally writes. Line 331 admits "`task check` does not run
`docs:check`, so generated drift is currently ungated" — and it cannot, because
there is no `check` mode and no `docs:*` task to run.
Every real defect found in the session that produced this tool was found by other
means:
**2. The dashboard's client is not centralised, and its own comments say it is.**
`ui/src/base/api.jsx` opens with "Single place that knows how to talk to archied.
Every feature folder goes through this, so auth handling and error shape live in
one file." **22 of its 39 calls use a raw `fetch()` that bypasses its own `req()`
wrapper**, and `ui/src/chat/chat.jsx:406` calls `fetch("/api/chat/stream")` from
outside `api.jsx` entirely. The consequence is concrete, not stylistic: `req()`
is the only place that maps `401` to `ApiError("unauthorised", 401)`, which
`classifyActionError` turns into `session-expired`. Every raw-`fetch` path
therefore reports an expired session as a generic `refused`/`broken` failure —
which is the exact defect `archie-core-1786637490536-29-f1dab2c4` ("Make dashboard
authentication and connectivity failures actionable") exists to fix.
- the three 415 refusals, by reading `authorizeTaskMutation` and reproducing the
browser's exact headers against the real handler;
- the dashboard client's duplicated request plumbing, by reading the module.
This is the check the audit's own harness must survive: it asserts the
centralisation claim, and today it fails.
Recorded plainly because a survey that has never found anything is evidence about
the survey, not about the tree.
## Shape
## Errata — claims in the first revision of this document that were false
```text
tools/contractaudit/
main.go # flag parsing, report rendering (advisory by default)
surface.go # Surface interface: Name() + Findings() []Finding
proto.go # RPC coverage by reflection over the exported client types
webui.go # route registration vs ui/src call sites
generated.go # docsgen check-mode comparison
declarations.go # the allowlist, with exact-match staleness
testdata/ # negative-test fixtures
docs/contract-declarations.json # DECLARED-UNCONSUMED allowlist
```
- **"RPC coverage is derived by reflection over the exported client type."**
It is not. `proto.go` regex-scans adapter bodies for `client.Method(` calls.
The reflection version was replaced and the document was not updated. The unused
`google.golang.org/grpc` family in `tools/go.mod` was residue from that
iteration; `go mod tidy` removed it on 2026-09-15.
- **Surface sizes of "43 / 12 / 52".** The real counts are 45 / 21 / 49.
- **"Three classifications, and the third is the only finding."** Four classes,
two of which fail.
- **"Raw `fetch()` calls caused 401s to render as `refused`/`broken`."** Wrong.
Every raw call site passed `res.status` to `ApiError`, so `classifyActionError`
already mapped 401 to `session-expired`; the chat stream never classified at
all. The real defect in that area was different (below).
- **A five-surface "Shape" naming `surface.go`, `generated.go`,
`declarations.go`, and `testdata/`.** None were written. The real files are
`main.go`, `audit.go`, `proto.go`, `webui.go`.
Finding shape, stable because tests and the report both consume it:
## The two gaps this was aimed at
```go
type Finding struct {
Surface string // "proto/state/v1"
Subject string // "StateStoreService/TokensByDay"
Class string // CONSUMED | DECLARED-UNCONSUMED | UNDECLARED-UNCONSUMED
Evidence string // file:line, or the reason string
Tracker string // bead id for DECLARED entries, empty otherwise
}
```
**1. `docsgen check` is documented and does not exist.** Still open, filed as
`archie-core-5gzx`. `generated-documentation.md:118-121` advertises
`docsgen data|asyncapi|all|check`; `tools/docsgen/main.go` has exactly two flags
and always writes, so the documented check mode cannot run and line 331's
"generated drift is currently ungated" stands.
Advisory is the default: findings print, exit status is 0. `--strict` (and
`task contract` in the gate, once the initial findings are triaged) fails on any
`UNDECLARED-UNCONSUMED` or stale declaration.
**2. The dashboard's client was not centralised while its own docstring claimed
it was.** Now closed by `18e8f8f`. The docstring said "Single place that knows how
to talk to archied… so auth handling and error shape live in one file" while 22
call sites built their own `fetch`, headers, timeout and error handling, and
`chat.jsx` built a 23rd. The defect actually present there was that three
bodyless mutations — `bindingApprove`, `mappingDelete`, `bindingDelete` — omitted
`Content-Type` and were refused **415** on every use, because
`authorizeTaskMutation` requires a JSON media type even when there is no body.
The Go tests missed it because their helper set `Content-Type` unconditionally,
including for a nil body.
## Blind spots, labelled rather than omitted
## Testing rule — applies to this tool and to anything like it
Following `reachaudit`'s rule that Yaegi/reflection packages are "labelled, never
presented as dead":
**Do not assert on the wording, formatting, or content of generated or
hand-authored text and config files.** A test that pins a source snippet, injects
text into a fixture, or counts an identifier in a file is testing that file's
formatting rather than the behaviour under test: an extra space, a newline, a
rename, or a comment that merely mentions the token breaks it, and the
maintenance burden is permanent. A test that cannot tell "the code is wrong" from
"the file is formatted differently" is testing the wrong thing.
- **Runtime-loaded code** (Yaegi, reflection) — a route or RPC reached only
through dynamic dispatch reads as unconsumed. Labelled with the package list.
- **Dashboard call-site extraction is textual, and deliberately narrow.** A path
counts as a consumer only when it is the *first argument of a call* in code:
the function need not be named (so renaming `req(` to `request(` is fine), but
a bare constant, a dead array, an object property, or a commented-out call
must not count. Both halves are load-bearing, and they pull in opposite
directions.
Assert behaviour on the observable effect — the request a client makes, the value
a function returns — and keep parsing out of the assertion. Where a parser must
exist, test it through its own interface with in-memory input, not through a file
on disk.
Naming the call broke on rename: matching `req|fetch` reported 16 live routes
as unconsumed the moment the dashboard's helper was renamed. Dropping the
call requirement was **worse**, and is recorded here because it is the more
tempting mistake: crediting any quoted `/api/...` literal meant a comment, a
retired-route array, or a display label proved a route was reachable, so
deleting every real request still read as healthy. The two directions are not
symmetric — a false "unconsumed" is a loud, fixable failure, while a false
"consumed" is a silent hole, which is worse than having no gate.
The first revision of this tool violated this in six tests (four that injected
text into fixtures, two that asserted on a source file's contents). All six were
deleted on 2026-09-15. The dashboard suite's "api.jsx holds exactly one fetch
call" check was deleted for the same reason: it broke on a reformat and on a
comment that mentioned `fetch`, and it was patched with a comment-stripper instead
of being removed. Every `api.*` method is now covered behaviourally instead, by
asserting the headers and body each one sends.
Remaining limits: a path passed as any argument other than the first is not
seen (accepting a comma would start crediting array elements and restore the
vacuity above); a request written inside a template-literal interpolation is
not seen; and a path built from a bare `/api` fragment is reported as
unparsable rather than silently contributing nothing.
- **Routes are credited per path, not per verb.** A literal does not say which
HTTP verb carries it, so a route is credited to every verb and a route whose
specific verb has no caller reads as consumed.
- **An `EventSource` subscription is a consumer.** `/events` and
`/api/logs/stream` are consumed via `EventSource`, not `fetch`; an extractor
that only looks for `fetch` reports them unconsumed and is simply wrong.
## Open decision
1. **Withdraw.** Delete the tool, the allowlist, this document, and the `contract`
task. It has found nothing, its mechanism is unsound, and the same question is
better asked by a behavioural test per method.
2. **Rebuild structurally.** Take the contract from the compiler instead of from
text: the generated `StateStoreServiceClient` / `ChatServiceClient` interfaces
give the exact RPC set, and `go/ast` over the adapter bodies and
`internal/webui/server.go` is formatting-insensitive where a regex is not.
Cost: the dashboard-consumer half has no Go-native JS parser, so it is either
dropped or replaced by a behavioural test per route — which is what would have
caught the 415s.
Neither is "make the regexes cleverer". Recorded here rather than acted on
unilaterally because withdrawing a capability and rebuilding one are both larger
decisions than adding a report.
## Acceptance
1. `go -C tools run ./contractaudit` runs on demand and prints a report; exit 0.
2. Every unconsumed surface appears with a classification and, where declared, a
reason and tracker id.
3. The declaration file fails on a **new** undeclared entry *and* on a **stale**
entry that is no longer true.
4. Blind spots are labelled in the report, never silently counted as findings.
5. Findings are advisory; nothing is deleted or rewritten by this tool.
6. `docsgen check` exists and `task check` runs it, so committed generated data
cannot drift.
7. **Each check is negative-tested**: a violation is injected, the check is
confirmed to fail for the right reason, and the injection is reverted. A check
that has never failed is not a check.
1. `task contract` prints a report and exits 0.
2. Every unconsumed subject appears with a classification, and where declared with
a reason and a tracker id.
3. `--strict` fails on an undeclared entry **and** on a stale one. Verified by
construction against a copy of the tree, not by asserting on the tree's text.
4. The tool neither deletes nor rewrites anything.
5. No test in this tool asserts on the content or formatting of another file.
## Non-goals
- No runtime schema validation of HTTP responses. The webui handlers already
return non-nil slices on every empty path (`[]SkillView{}`, `[]CuratorView{}`,
`make([]taskView, len(tasks))`), so the nil-slice class does not currently
apply; adding a live-stack validator before it has a defect to find is
scaffolding, not a gate. Recorded as deferred, not rejected.
- No re-parse of the proto as a second source of truth. The generated Go types
are the contract; `buf lint`/`buf breaking` already own the `.proto`.
`make([]taskView, len(tasks))`), so the nil-slice class does not apply; adding a
live-stack validator before it has a defect to find is scaffolding. Deferred,
and filed as `archie-core-nddv`.
- No re-parse of the `.proto` as a second source of truth; `buf lint` and
`buf breaking` already own it.
- No generation of `config.example.toml` or `deployments/*.toml` — settled
against in `config-example-drift.md`.
- No change to `tools/reachaudit`'s scope or report shape.
+16 -9
View File
@@ -127,25 +127,32 @@ func New(root, declPath string) (*Audit, error) {
}, nil
}
// parseDeclarations decodes the allowlist from r. Decoding is separate from
// reading the file so its rules can be exercised without authoring a file.
func parseDeclarations(r io.Reader) (Declarations, error) {
var declarations Declarations
if err := json.NewDecoder(r).Decode(&declarations); err != nil {
return Declarations{}, fmt.Errorf("parse declarations: %w", err)
}
if declarations.Surfaces == nil {
declarations.Surfaces = map[string]map[string]Declaration{}
}
return declarations, nil
}
// loadDeclarations reads the allowlist. A missing file is not an error: an
// empty allowlist is the correct starting state and makes every unconsumed
// subject an undeclared finding, which is the honest first report.
func loadDeclarations(path string) (Declarations, error) {
raw, err := os.ReadFile(path)
f, err := os.Open(path)
if os.IsNotExist(err) {
return Declarations{Surfaces: map[string]map[string]Declaration{}}, nil
}
if err != nil {
return Declarations{}, fmt.Errorf("read declarations: %w", err)
}
var declarations Declarations
if err := json.Unmarshal(raw, &declarations); err != nil {
return Declarations{}, fmt.Errorf("parse %s: %w", path, err)
}
if declarations.Surfaces == nil {
declarations.Surfaces = map[string]map[string]Declaration{}
}
return declarations, nil
defer f.Close()
return parseDeclarations(f)
}
// Run walks every surface and returns findings sorted by surface then subject.
+37 -13
View File
@@ -1,8 +1,8 @@
package main
import (
"os"
"path/filepath"
"strings"
"testing"
)
@@ -84,7 +84,8 @@ func TestUndeclaredCountCountsOnlyFindings(t *testing.T) {
// TestLoadDeclarationsMissingFileIsEmpty asserts an absent allowlist is the
// correct starting state: every unconsumed subject becomes undeclared, which
// is the honest first report rather than a silent pass.
// is the honest first report rather than a silent pass. The path is never
// created, so no file content is involved.
func TestLoadDeclarationsMissingFileIsEmpty(t *testing.T) {
declarations, err := loadDeclarations(filepath.Join(t.TempDir(), "absent.json"))
if err != nil {
@@ -95,15 +96,38 @@ func TestLoadDeclarationsMissingFileIsEmpty(t *testing.T) {
}
}
// TestLoadDeclarationsRejectsMalformed asserts a broken allowlist fails loudly
// rather than degrading to "no declarations", which would silently turn every
// declared entry into a finding nobody could clear.
func TestLoadDeclarationsRejectsMalformed(t *testing.T) {
path := filepath.Join(t.TempDir(), "bad.json")
if err := os.WriteFile(path, []byte("{not json"), 0o600); err != nil {
t.Fatal(err)
}
if _, err := loadDeclarations(path); err == nil {
t.Error("loadDeclarations(malformed) error = nil, want a parse failure")
}
// TestParseDeclarations covers the decoding rules in memory, so no file is
// authored and no formatting of an external file can affect the outcome.
func TestParseDeclarations(t *testing.T) {
t.Run("an empty object is usable", func(t *testing.T) {
declarations, err := parseDeclarations(strings.NewReader("{}"))
if err != nil {
t.Fatalf("error = %v, want nil", err)
}
if declarations.Surfaces == nil {
t.Error("surfaces = nil, want a non-nil empty map so a lookup cannot panic")
}
})
t.Run("entries are decoded", func(t *testing.T) {
declarations, err := parseDeclarations(strings.NewReader(
`{"surfaces":{"s":{"Svc/Method":{"reason":"because","tracker":"abc.1"}}}}`,
))
if err != nil {
t.Fatalf("error = %v, want nil", err)
}
entry := declarations.Surfaces["s"]["Svc/Method"]
if entry.Reason != "because" || entry.Tracker != "abc.1" {
t.Errorf("entry = %+v, want the reason and tracker preserved", entry)
}
})
t.Run("malformed input is rejected", func(t *testing.T) {
// A broken allowlist must fail loudly rather than degrading to "no
// declarations", which would turn every declared entry into a finding
// nobody could clear.
if _, err := parseDeclarations(strings.NewReader("{not json")); err == nil {
t.Error("error = nil, want a parse failure")
}
})
}
-336
View File
@@ -1,336 +0,0 @@
package main
import (
"os"
"path/filepath"
"strings"
"testing"
)
// fixture copies the repository subtrees the audit reads into a temporary
// directory, so a test can inject a violation without touching the working
// tree. The audit reads only these paths -- if a surface starts reading
// something else, a test here fails rather than silently auditing a fixture
// that no longer matches the real layout.
func fixture(t *testing.T) string {
t.Helper()
root := t.TempDir()
for _, dir := range []string{
"proto/state/v1",
"proto/gateway/v1",
"internal/infrastructure/staterpc",
"internal/infrastructure/gatewayrpc",
"internal/webui",
"ui/src",
} {
copyTree(t, filepath.Join("../..", dir), filepath.Join(root, dir))
}
return root
}
// copyTree copies a directory recursively. It must recurse: the dashboard's
// call sites live in nested feature folders (ui/src/base, ui/src/chat), so a
// flat copy yields a fixture with zero consumers and every negative test then
// passes for the wrong reason.
func copyTree(t *testing.T, src, dst string) {
t.Helper()
if err := os.MkdirAll(dst, 0o755); err != nil {
t.Fatal(err)
}
entries, err := os.ReadDir(src)
if err != nil {
t.Fatalf("fixture: read %s: %v", src, err)
}
for _, entry := range entries {
from := filepath.Join(src, entry.Name())
to := filepath.Join(dst, entry.Name())
if entry.IsDir() {
copyTree(t, from, to)
continue
}
body, err := os.ReadFile(from)
if err != nil {
t.Fatal(err)
}
if err := os.WriteFile(to, body, 0o644); err != nil {
t.Fatal(err)
}
}
}
func auditOf(t *testing.T, root string) []Finding {
t.Helper()
audit, err := New(root, "docs/contract-declarations.json")
if err != nil {
t.Fatalf("New: %v", err)
}
findings, err := audit.Run()
if err != nil {
t.Fatalf("Run: %v", err)
}
return findings
}
func find(findings []Finding, subject string) (Finding, bool) {
for _, f := range findings {
if f.Subject == subject {
return f, true
}
}
return Finding{}, false
}
// TestNegativeInjectedRouteIsReported proves the webui check catches a route
// nobody consumes. Without this the check is decoration: a gate that has never
// failed is a gate nobody can trust.
func TestNegativeInjectedRouteIsReported(t *testing.T) {
root := fixture(t)
// A new route with no consumer anywhere in ui/src.
serverPath := filepath.Join(root, "internal/webui/server.go")
body, err := os.ReadFile(serverPath)
if err != nil {
t.Fatal(err)
}
injected := strings.Replace(
string(body),
`mux.HandleFunc("GET /api/summary"`,
`mux.HandleFunc("GET /api/injected-orphan", s.handleSummary)
mux.HandleFunc("GET /api/summary"`,
1,
)
if injected == string(body) {
t.Fatal("could not inject the route; the registration shape changed")
}
if err := os.WriteFile(serverPath, []byte(injected), 0o644); err != nil {
t.Fatal(err)
}
finding, ok := find(auditOf(t, root), "GET /api/injected-orphan")
if !ok {
t.Fatal("injected unconsumed route was not reported")
}
if finding.Class != Undeclared {
t.Errorf("class = %q, want %q", finding.Class, Undeclared)
}
}
// TestNegativeRemovedConsumerIsReported proves the same check fails in the
// other direction: remove the dashboard's only call to a route and the route
// becomes an undeclared finding.
func TestNegativeRemovedConsumerIsReported(t *testing.T) {
root := fixture(t)
apiPath := filepath.Join(root, "ui/src/base/api.jsx")
body, err := os.ReadFile(apiPath)
if err != nil {
t.Fatal(err)
}
if !strings.Contains(string(body), "/api/summary") {
t.Fatal("fixture does not contain the expected consumer")
}
trimmed := strings.ReplaceAll(string(body), "/api/summary", "/api/renamed-by-test")
if err := os.WriteFile(apiPath, []byte(trimmed), 0o644); err != nil {
t.Fatal(err)
}
finding, ok := find(auditOf(t, root), "GET /api/summary")
if !ok {
t.Fatal("route whose consumer was removed was not reported")
}
if finding.Class != Undeclared {
t.Errorf("class = %q, want %q", finding.Class, Undeclared)
}
}
// TestNegativeInjectedRPCIsReported proves the proto check catches an RPC no
// adapter reaches.
func TestNegativeInjectedRPCIsReported(t *testing.T) {
root := fixture(t)
protoPath := filepath.Join(root, "proto/state/v1/state.proto")
body, err := os.ReadFile(protoPath)
if err != nil {
t.Fatal(err)
}
injected := strings.Replace(
string(body),
" rpc RegisterTaskGrant(",
" rpc InjectedOrphanRPC(InjectedOrphanRPCRequest) returns (InjectedOrphanRPCResponse);\n rpc RegisterTaskGrant(",
1,
)
if injected == string(body) {
t.Fatal("could not inject the rpc; the declaration shape changed")
}
if err := os.WriteFile(protoPath, []byte(injected), 0o644); err != nil {
t.Fatal(err)
}
finding, ok := find(auditOf(t, root), "StateStoreService/InjectedOrphanRPC")
if !ok {
t.Fatal("injected unconsumed RPC was not reported")
}
if finding.Class != Undeclared {
t.Errorf("class = %q, want %q", finding.Class, Undeclared)
}
}
// TestNegativeStaleDeclarationIsReported proves a declaration that outlived its
// subject fails. This is the direction people forget, and it is what stops an
// allowlist from quietly permitting something nobody checks.
func TestNegativeStaleDeclarationIsReported(t *testing.T) {
findings := classify("s", []Subject{
{Name: "Svc/NowConsumed", Consumed: true, Consumer: "internal/x/y.go:1"},
}, Declarations{Surfaces: map[string]map[string]Declaration{
"s": {"Svc/NowConsumed": {Reason: "was unconsumed", Tracker: "t"}},
}})
finding, ok := find(findings, "Svc/NowConsumed")
if !ok {
t.Fatal("stale declaration was not reported")
}
if finding.Class != Stale {
t.Errorf("class = %q, want %q", finding.Class, Stale)
}
}
// TestNegativeUnparsableCallShapeFailsLoudly proves the extractor refuses to
// under-report. A path built from a bare /api fragment must be an error, never
// a silently smaller consumer set.
func TestNegativeUnparsableCallShapeFailsLoudly(t *testing.T) {
body := `
const base = "/api";
const url = base + "/things";
`
_, err := scanFile("ui/src/base/exotic.jsx", body)
if err == nil {
t.Fatal("a path built from a bare /api fragment was accepted; the extractor would under-report")
}
if !strings.Contains(err.Error(), "would under-report") {
t.Errorf("error = %v, want the under-report explanation", err)
}
}
// TestNegativeCommentedOutCallIsStillReported is the anti-vacuity test at the
// surface level. Commenting out a real call leaves the path text in the file,
// so an extractor that credits any quoted path reports the route as healthy
// while nothing calls it -- a silent hole rather than a loud failure. The route
// must become an undeclared finding.
func TestNegativeCommentedOutCallIsStillReported(t *testing.T) {
root := fixture(t)
apiPath := filepath.Join(root, "ui/src/base/api.jsx")
body, err := os.ReadFile(apiPath)
if err != nil {
t.Fatal(err)
}
live := `summary: () => request("/api/summary"),`
if !strings.Contains(string(body), live) {
t.Fatal("fixture does not contain the expected live call; the client changed shape")
}
commented := strings.Replace(string(body), live, `// retired: () => request("/api/summary"),`, 1)
if err := os.WriteFile(apiPath, []byte(commented), 0o644); err != nil {
t.Fatal(err)
}
finding, ok := find(auditOf(t, root), "GET /api/summary")
if !ok {
t.Fatal("a route whose only remaining mention is a comment was not reported")
}
if finding.Class != Undeclared {
t.Errorf("class = %q, want %q: a comment cannot prove a route is reachable", finding.Class, Undeclared)
}
}
// TestScanFileCreditsAnyCallSignature pins the lesson from a real regression:
// the first version of this extractor matched the call by name (`req|fetch`),
// and rewriting the dashboard's helper from `req(` to `request(` silently
// reported 16 live routes as unconsumed. Extraction must key on the path
// literal as a call argument, not on the name of the function carrying it.
func TestScanFileCreditsAnyCallSignature(t *testing.T) {
cases := []struct {
name string
body string
want []string
}{
{
name: "the original helper name",
body: `const a = req("/api/summary");`,
want: []string{"/api/summary"},
},
{
name: "a renamed helper",
body: `const a = request("/api/summary");`,
want: []string{"/api/summary"},
},
{
name: "a name the extractor has never heard of",
body: `const a = someOtherTransport("/api/summary");`,
want: []string{"/api/summary"},
},
{
name: "an interpolated path in backticks",
body: "const a = request(`/api/tasks/${id}`);",
want: []string{"/api/tasks/${id}"},
},
{
name: "an EventSource subscription",
body: `const s = new EventSource("/events");`,
want: []string{"/events"},
},
{
name: "a path with a query string",
body: `const a = request("/api/logs" + qs(params));`,
want: []string{"/api/logs"},
},
}
for _, tc := range cases {
t.Run(tc.name, func(t *testing.T) {
got, err := scanFile("ui/src/base/client.jsx", tc.body)
if err != nil {
t.Fatalf("scanFile: %v", err)
}
if len(got) != len(tc.want) {
t.Fatalf("scanFile(%q) = %v, want %v", tc.body, got, tc.want)
}
for i := range tc.want {
if got[i] != tc.want[i] {
t.Errorf("scanFile(%q)[%d] = %q, want %q", tc.body, i, got[i], tc.want[i])
}
}
})
}
}
// TestScanFileIgnoresNonCallLiterals is the anti-vacuity half, and the more
// important one. Crediting any quoted path -- rather than a path passed to a
// call -- makes the whole surface pass on a file that issues no requests at
// all: a comment, a retired-route array, or a display label would each count as
// proof of consumption. That failure is silent, which makes it worse than a
// false finding.
func TestScanFileIgnoresNonCallLiterals(t *testing.T) {
cases := []struct {
name string
body string
}{
{"a constant that is never passed to anything", `const p = "/api/summary";`},
{"a comment quoting a path", `// the route "/api/summary" is dead`},
{"an array of retired paths", `const RETIRED = ["/api/summary", "/api/tasks"];`},
{"an object property", `const cfg = { path: "/api/summary" };`},
{"a template constant", "const p = `/api/summary`;"},
}
for _, tc := range cases {
t.Run(tc.name, func(t *testing.T) {
got, err := scanFile("ui/src/base/client.jsx", tc.body)
if err != nil {
t.Fatalf("scanFile: %v", err)
}
if len(got) != 0 {
t.Errorf(
"scanFile(%q) = %v, want no consumers: a literal that is not a call argument cannot prove a route is reachable",
tc.body, got,
)
}
})
}
}
+7 -8
View File
@@ -10,15 +10,14 @@ import (
)
// webUISurface reports HTTP route coverage for the dashboard API. The contract
// of record is the `registerXRoutes` call sites in internal/webui, and the
// consumers are the path literals the dashboard's client actually calls.
// of record is the route registrations in internal/webui, and the consumers are
// the path literals written under ui/src.
//
// This is the one surface with no schema at all: 52 routes and their response
// shapes are hand-matched against ui/src, so it is also the one that needs the
// check the most. The extractor fails loudly rather than under-reporting -- see
// scanFile -- because a call shape it cannot parse would otherwise make every
// route it feeds look unconsumed, and a false finding in a gate is worse than
// no gate.
// This surface has no schema at all, so the two lists are hand-matched and the
// matching is textual. It is advisory for that reason: a rename or a reformat
// elsewhere can change the result with nothing wrong, and it has already
// reported a batch of false "unconsumed" routes once when the dashboard's
// request helper was renamed. Do not gate a build on this.
type webUISurface struct {
root string
}
-5
View File
@@ -20,12 +20,7 @@ require (
github.com/samcharles93/ai-sdk v0.1.32 // indirect
github.com/traefik/yaegi v0.16.1 // indirect
go.yaml.in/yaml/v4 v4.0.0-rc.2 // indirect
golang.org/x/net v0.58.0 // indirect
golang.org/x/sys v0.47.0 // indirect
golang.org/x/text v0.41.0 // indirect
google.golang.org/genproto/googleapis/rpc v0.0.0-20260904194346-d0f1323225a4 // indirect
google.golang.org/grpc v1.83.2 // indirect
google.golang.org/protobuf v1.36.12 // indirect
gopkg.in/yaml.v3 v3.0.1 // indirect
modernc.org/libc v1.75.7 // indirect
modernc.org/mathutil v1.7.1 // indirect
-10
View File
@@ -38,23 +38,13 @@ go.yaml.in/yaml/v4 v4.0.0-rc.2 h1:/FrI8D64VSr4HtGIlUtlFMGsm7H7pWTbj6vOLVZcA6s=
go.yaml.in/yaml/v4 v4.0.0-rc.2/go.mod h1:aZqd9kCMsGL7AuUv/m/PvWLdg5sjJsZ4oHDEnfPPfY0=
golang.org/x/mod v0.38.0 h1:MECBjubtXD7yj4HrhIUcywNaGeNVUdfVnxmPajOk4yk=
golang.org/x/mod v0.38.0/go.mod h1:V6Xz0pq8TQ3dGqVQ1FVHuelZpAL0uNhSkk9ogYP3c40=
golang.org/x/net v0.58.0 h1:ynWG7rqYi4ccpTEuPZ2QGWHktVEM9DMCj9yzDE0Q7To=
golang.org/x/net v0.58.0/go.mod h1:YwCddHnFlT7eLQqVprV19OnhLGtc5xOKgE0RyqgfWAU=
golang.org/x/sync v0.22.0 h1:SZjpbeLmrCk4xhRSZFNZW5gFUeCeFgjekvI/+gfScek=
golang.org/x/sync v0.22.0/go.mod h1:9xrNwdLfx4jkKbNva9FpL6vEN7evnE43NNNJQ2LF3+0=
golang.org/x/sys v0.1.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.47.0 h1:o7XGOvZQCADBQQ4Y7VNq2dRWQR7JmOUW8Kxx4ZsNgWs=
golang.org/x/sys v0.47.0/go.mod h1:4GL1E5IUh+htKOUEOaiffhrAeqysfVGipDYzABqnCmw=
golang.org/x/text v0.41.0 h1:vz/seA0lnX87Othu2f/0L24RcgrXD9/YFTSuGjj3rH8=
golang.org/x/text v0.41.0/go.mod h1:jvf1O8ajNzZqhSrQBPbutR/EB83Cc0CFrezNQIwbb5M=
golang.org/x/tools v0.48.0 h1:3+hClM1aLL5mjMKm5ovokw9epgRXPuu2tILgismM6RE=
golang.org/x/tools v0.48.0/go.mod h1:08xX0orndb/F7jJxGDicx061tyd5pcMto75YMAXr6lk=
google.golang.org/genproto/googleapis/rpc v0.0.0-20260904194346-d0f1323225a4 h1:5t+ZydAFj5kGVLrgCvLmpmCf9ylGRd64hpEronfRaws=
google.golang.org/genproto/googleapis/rpc v0.0.0-20260904194346-d0f1323225a4/go.mod h1:DjtHYE8FKJLivXcBEjGwndXfIC23G0VpXiXKqG179uA=
google.golang.org/grpc v1.83.2 h1:EManeRomTObA0BU7I8vXgg/78uE5MJ9M8B39EX2WscU=
google.golang.org/grpc v1.83.2/go.mod h1:YPI1hK3kDked6iHvgX3tR0y+nX/qpMFKhPgFsokw1S8=
google.golang.org/protobuf v1.36.12 h1:pJOKDDOyeXErUroCihFAd5LQuwXBSpVnKGrj5o/fwxc=
google.golang.org/protobuf v1.36.12/go.mod h1:HTf+CrKn2C3g5S8VImy6tdcUvCska2kB7j23XfzDpco=
gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405 h1:yhCVgyC4o1eVCa2tZl7eS0r+SDo693bJlVdllGtEeKM=
gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
gopkg.in/yaml.v3 v3.0.1 h1:fxVm/GzAzEWqLHuvctI91KS9hhNmmWOoWu0XTYJS7CA=
+6 -62
View File
@@ -1,7 +1,5 @@
import { test } from "node:test";
import assert from "node:assert/strict";
import { readFileSync } from "node:fs";
import { fileURLToPath } from "node:url";
import "./shim.js";
import { api, classifyActionError } from "../src/base/api.js";
@@ -188,63 +186,9 @@ test("every api method is classified by this suite", () => {
);
});
// The module comment claims this file is the single place that talks to
// archied. That claim is only true while there is exactly one fetch() call
// here; a second one added later would silently bypass the shared CSRF,
// timeout, and error handling -- which is how the three 415s arose.
//
// stripComments keeps this count measuring code rather than prose about code:
// a comment that mentions fetch() must not be able to fail the gate, and must
// not be able to hide a real call either.
function stripComments(source) {
let out = "";
let quote = null;
for (let i = 0; i < source.length; ) {
const c = source[i];
const next = source[i + 1];
if (quote) {
out += c;
if (c === "\\") {
out += next ?? "";
i += 2;
continue;
}
if (c === quote) quote = null;
i++;
continue;
}
if (c === '"' || c === "'" || c === "`") {
quote = c;
out += c;
i++;
continue;
}
if (c === "/" && next === "/") {
while (i < source.length && source[i] !== "\n") i++;
continue;
}
if (c === "/" && next === "*") {
i += 2;
while (i < source.length && !(source[i] === "*" && source[i + 1] === "/")) i++;
i += 2;
continue;
}
out += c;
i++;
}
return out;
}
test("api.jsx holds exactly one fetch call", () => {
const source = readFileSync(fileURLToPath(new URL("../src/base/api.jsx", import.meta.url)), "utf8");
const count = (stripComments(source).match(/\bfetch\s*\(/g) ?? []).length;
// A count of zero means the stripper ate the call, not that the file is
// clean; this check must never pass vacuously.
assert.ok(count >= 1, "no fetch() call found in api.jsx; the extractor is broken, not the file");
assert.equal(
count,
1,
`api.jsx contains ${count} fetch() calls, expected exactly 1: every call must go through the shared send() so CSRF, timeout, and error shape stay in one place`,
);
});
// Note: this suite asserts on the requests each method makes, never on the
// text of api.jsx. A source-scanning check ("api.jsx holds exactly one fetch
// call") was removed on purpose: it broke on a reformat and on a comment that
// merely mentioned fetch, so it tested the file's wording rather than the
// client's behaviour. Every method in the two inventories above is covered
// behaviourally instead.