The before-and-after test loop

The question a documentation hub has to answer is not "is it well written". It is could an agent that has never seen this system do the work from what the hub told it — and then, after a change to the hub, could it do the work with fewer calls and fewer tokens. That is a measurement, so it needs a prompt, a scoring sheet, a label, and a reading taken from the server rather than from the agent's own report.

This guide is how that loop runs. It is the method, not one run's results; the baseline run's figures are at the end as the worked example.

1. The shape of the loop

1. A release artifact names what the build is: its version, its surface, its flags.
2. A test prompt is stored as a test pass in the control plane, verbatim. The prompt

is the artifact; a paraphrase is a different test.

1. A client agent is launched with the prompt and nothing else — no repository

access, no release note. It must learn the surface from the surface.

1. The agent self-identifies with the run's label on every call that takes one, runs

the steps in order, and reports in a fixed shape.

1. The run is matched on the server's own rows by that label, and the call sequence,

the session handle and the documents served are read from there.

1. The report is scored against the pass sheet, which the agent never sees.
2. The findings change the hub. The next run's row in the loop log names the change

it followed.

Two rules protect the measurement. The agent is not told the expected answers — they live in the scoring sheet and are compared afterwards, so the agent cannot be led to them. And a run is identified by exactly one label, in the form cuppy-mcp-<release version>-<run ordinal>; an unlabelled run cannot be matched and is therefore a failed run however well it went.

2. The fifteen steps

The prompt asks for fifteen steps in order, and for each one the request sent, the raw response, and one line of the agent's own judgement. In outline:

steps	what they probe
1 to 3	identity and surface: the protocol version, the server name and version, the list key, every tool with its command count, and the total reachable command rows
4 to 6	the boot tier: the banner, the routes for two different intents, whether a handle was minted, and whether the help answer names the same set the tool list did
7	a session: the handle, the capability object, whether it is persisted
8 and 9	retrieval: a search of the agent's own choosing, then a fetch of an id that search returned
10	the hub's boot document — found through a listed tool, or declared unreachable with the server's own sentence
11 and 12	two refusals: a tool that is not in the list, and a command that is not in a listed tool's set. The error code, the typed code inside it, and the fix string
13	the method rules: a GET and a DELETE on the endpoint
14	the list key again, to prove it did not move mid-run
15	the report

Three rules about the report, and they are the point of the exercise. Every number in it must come from a response the agent received, attested by a field the agent sets itself. The agent's judgement is recorded and is not scored for agreement — including the one question that matters most, whether it could have started work from what the server told it. And a missing capability is a legitimate answer: a step that finds no tool for what it asks is scored as declared-not-served, while inventing a route to a missing capability is the one thing that makes a run worthless.

3. What a pass is

Three verdicts: PASS when every required row matches, no prohibited row fires and the report is well formed; PASS-WITH-NOTES when every required row matches and at least one advisory row does not; FAIL when any required row misses, any prohibited row fires, or the report is malformed.

A fourth outcome sits outside that enum. A run against a build whose list key differs from the sheet's is VOID, not failed: it measured a different surface. Record it, re-derive the sheet against the new build, and run again. A void run reported as a pass is the worst outcome available, because it is a number about nothing.

The sheet for this hub's surface carries four classes of row, counted from the sheet itself: 19 required rows, 6 prohibited rows, 8 advisory rows of which 7 are scored for agreement (the eighth is the agent's judgement, recorded and read), and 4 declared-not-served rows, each of which is met by saying so with evidence.

Two prohibited rows are worth naming here because they are what makes the harness more than a feature test. One fires if any response body carries an address or a credential-shaped string — so every run is also a redaction probe. The other fires if any refusal's text carries a database state code, a stack frame, a file path or a SQL fragment — a caller-facing sentence carrying a database error is a raw error wearing a refusal for a hat.

4. The server-side match

The agent's report is evidence about the agent. The server's rows are evidence about the run, and the loop reads the outcome from them: one audit row per call, carrying the label byte-identically, the session handle, the agent's model and the count of documents served, read by one query the scorer runs with the run's label.

What the match settles that a report cannot. The call sequence is reproduced in the server's own order rather than asserted. The handle is in its own column, never the transport's identifier wearing another name. The documents served figure is counted at read time from the retrieval rows, which is the one number a self-declared report cannot fake. And a run with no matching row is void, which is the rule that makes the whole loop a measurement rather than a conversation.

Two honest gaps in the match, both recorded as findings from the first two runs:

• An unlisted tool name writes no audit row at all, so the one refusal probe every

run makes is invisible to the query and a scorer sees an off-by-one between calls made and rows found (tracking/backlog/hub.md item 45).

• A document read by path writes no retrieval or signal row on the Postgres side,

with both sinks on, while recording the Postgres store in the other log — so a hub whose clients navigate by path rather than by search accrues no signal however often it serves a document (the same backlog, item 44).

5. The measurement contract

A pass is binary; the loop's purpose is a trend. From the second run onward each run records:

• tokens per step, from the client's own usage report;
• the documents each step read, from the audit rows' object column;
• retrievals per step;
• the hub's document count and token total at run time, so a content change can be

read against its effect.

Both arms are then comparable: the hub arm and a full-corpus arm run under the same task set and the same completion criterion, with the headline figure task completion per token. The target is not fewer tokens; it is the correct context sooner.

This contract was added after the baseline run and no run has yet produced it. The baseline has a whole-run token figure and a whole-run document count, and nothing per step. Said plainly so no later reader treats the baseline as a per-step measurement: it is not one.

6. How a run's findings change the hub, and nothing else

A build defect a run finds is a finding, not something the run fixes. The run records it with its reproducer, its severity and a one-line fix, and files it. The next run's row in the loop log names the change it followed in its own column, and that change is normally to the hub's content or structure — never to the build unless something is broken.

Four standing rules for anyone operating a run:

1. Never write to someone else's storage. A probe whose safety depends on the control

under test still holding is exactly the probe a read-only run may not make on another party's data. Score such a row as not-run with the reason instead.

1. A write probe is declared and sits beside the fifteen steps, never inside them: no

step of the prompt writes, so a row about writing cannot be scored from a conforming run of it.

1. Correct the prompt and the sheet in place, dated, with the superseded text quoted.

Two of the first three runs found that a corrected step had quietly disarmed another row; the quoted history is how the next operator sees it.

1. File the agent's own "one thing to fix". An unread judgement is the whole loop

wasted.

7. The baseline, as the worked example

The first public run of this loop, against cuppy's own hub. Every figure is read from docs/commander/RUNS/2026-10-15-public-staging.md and the row in docs/commander/LOOP-LOG.md:

figure	baseline run
label	cuppy-mcp-0.24.0-002 (ordinal 002; 001 was a private rehearsal)
hub under test	cuppy's own hub, 20 documents, counted on the server
build	version 0.24.0, both flags on, the list key stable across the run
required rows	19 of 19 matched, all nineteen actually run
prohibited rows	0 of 6 fired; one scored not-armed, because no step writes
advisory rows	7 of 7 scored rows matched
server rows matched	9 audit rows for the fifteen steps, every one carrying the label; a tenth after a declared probe
documents served	12 — ten search hits and one fetch from another product's corpus, plus one document read from this hub by path; 24 after the declared probe added twelve hits from this hub
tokens	260k, whole run, client-side (input, output and cache writes)
wall clock	25 minutes
findings	three: one medium on the search corpus, one medium on the missing retrieval row, one low on the missing audit row
verdict	proposed PASS, matched on the server's rows

The one reading to carry away from that table: the twelve documents the server counted were mostly not this hub's. Ten came from another product's corpus through the tool named search, one was a fetch of one of those ids, and exactly one was a document of this hub read by path. The agent's own worst moment was the same thing: the one answer a client agent could not trust was the one from the tool called search. A loop that only read the report's search_results: 10 would have called that a success.

Next reads

• guides/sdk-and-transport.md — what a client needs to make the calls the steps make.
• guides/approach-an-unknown-system.md — the protocol for building the hub a run like

this measures.

• reference/standard/hub-standard.md — the contract the hub's documents are written to.
• guides/runbooks/register-in-commander.md — the project row a test pass attaches to.