---
cuppy_hub: cuppy
cuppy_path: guides/test-loop.md
cuppy_section: docs
cuppy_doc: 1A01-KSQP
cuppy_doc_uuid: f99010a0-3e3b-4172-87c5-1bb2feb0ede5
cuppy_permalink: https://cuppy.io/cuppy/d/1A01-KSQP
cuppy_version: 1
cuppy_updated: 2026-10-17T06:32:45.511Z
cuppy_canonical: https://cuppy.io/cuppy/docs/guides/test-loop
cuppy_mcp: https://cuppy.io/mcp
cuppy_body_sha256: 70cf1a48f324de299311ed8afd5ab557bb85bbb69378f710db43d75df7dd509c
# cuppy_body_sha256 is over the document body ALONE, excluding both generated blocks.
---

# The before-and-after test loop

The question a documentation hub has to answer is not "is it well written". It is
**could an agent that has never seen this system do the work from what the hub told
it** — and then, after a change to the hub, could it do the work with fewer calls and
fewer tokens. That is a measurement, so it needs a prompt, a scoring sheet, a label, and
a reading taken from the server rather than from the agent's own report.

This guide is how that loop runs. It is the method, not one run's results; the baseline
run's figures are at the end as the worked example.

## 1. The shape of the loop

1. **A release artifact** names what the build is: its version, its surface, its flags.
2. **A test prompt** is stored as a test pass in the control plane, verbatim. The prompt
   is the artifact; a paraphrase is a different test.
3. **A client agent** is launched with the prompt **and nothing else** — no repository
   access, no release note. It must learn the surface from the surface.
4. **The agent self-identifies** with the run's label on every call that takes one, runs
   the steps in order, and reports in a fixed shape.
5. **The run is matched on the server's own rows** by that label, and the call sequence,
   the session handle and the documents served are read from there.
6. **The report is scored** against the pass sheet, which the agent never sees.
7. **The findings change the hub.** The next run's row in the loop log names the change
   it followed.

Two rules protect the measurement. **The agent is not told the expected answers** — they
live in the scoring sheet and are compared afterwards, so the agent cannot be led to
them. And **a run is identified by exactly one label**, in the form
`cuppy-mcp-<release version>-<run ordinal>`; an unlabelled run cannot be matched and is
therefore a failed run however well it went.

## 2. The fifteen steps

The prompt asks for fifteen steps in order, and for each one the request sent, the raw
response, and one line of the agent's own judgement. In outline:

| steps | what they probe |
|---|---|
| 1 to 3 | identity and surface: the protocol version, the server name and version, the list key, every tool with its command count, and the total reachable command rows |
| 4 to 6 | the boot tier: the banner, the routes for two different intents, whether a handle was minted, and whether the help answer names the same set the tool list did |
| 7 | a session: the handle, the capability object, whether it is persisted |
| 8 and 9 | retrieval: a search of the agent's own choosing, then a fetch of an id that search returned |
| 10 | the hub's boot document — found through a listed tool, or declared unreachable with the server's own sentence |
| 11 and 12 | two refusals: a tool that is not in the list, and a command that is not in a listed tool's set. The error code, the typed code inside it, and the fix string |
| 13 | the method rules: a GET and a DELETE on the endpoint |
| 14 | the list key again, to prove it did not move mid-run |
| 15 | the report |

Three rules about the report, and they are the point of the exercise. **Every number in
it must come from a response the agent received**, attested by a field the agent sets
itself. **The agent's judgement is recorded and is not scored for agreement** — including
the one question that matters most, whether it could have started work from what the
server told it. And **a missing capability is a legitimate answer**: a step that finds no
tool for what it asks is scored as declared-not-served, while inventing a route to a
missing capability is the one thing that makes a run worthless.

## 3. What a pass is

Three verdicts: **PASS** when every required row matches, no prohibited row fires and the
report is well formed; **PASS-WITH-NOTES** when every required row matches and at least
one advisory row does not; **FAIL** when any required row misses, any prohibited row
fires, or the report is malformed.

A fourth outcome sits outside that enum. A run against a build whose list key differs
from the sheet's is **VOID**, not failed: it measured a different surface. Record it,
re-derive the sheet against the new build, and run again. **A void run reported as a pass
is the worst outcome available**, because it is a number about nothing.

The sheet for this hub's surface carries four classes of row, counted from the sheet
itself: **19** required rows, **6** prohibited rows, **8** advisory rows of which 7 are
scored for agreement (the eighth is the agent's judgement, recorded and read), and **4**
declared-not-served rows, each of which is met by saying so with evidence.

Two prohibited rows are worth naming here because they are what makes the harness more
than a feature test. One fires if **any response body carries an address or a
credential-shaped string** — so every run is also a redaction probe. The other fires if
**any refusal's text carries a database state code, a stack frame, a file path or a SQL
fragment** — a caller-facing sentence carrying a database error is a raw error wearing a
refusal for a hat.

## 4. The server-side match

The agent's report is evidence about the agent. The server's rows are evidence about the
run, and the loop reads the outcome from them: one audit row per call, carrying the
label byte-identically, the session handle, the agent's model and the count of documents
served, read by one query the scorer runs with the run's label.

What the match settles that a report cannot. The **call sequence** is reproduced in the
server's own order rather than asserted. The **handle** is in its own column, never the
transport's identifier wearing another name. The **documents served** figure is counted
at read time from the retrieval rows, which is the one number a self-declared report
cannot fake. And a **run with no matching row is void**, which is the rule that makes the
whole loop a measurement rather than a conversation.

Two honest gaps in the match, both recorded as findings from the first two runs:

- **An unlisted tool name writes no audit row at all**, so the one refusal probe every
  run makes is invisible to the query and a scorer sees an off-by-one between calls made
  and rows found (`tracking/backlog/hub.md` item 45).
- **A document read by path writes no retrieval or signal row** on the Postgres side,
  with both sinks on, while recording the Postgres store in the other log — so a hub
  whose clients navigate by path rather than by search accrues no signal however often
  it serves a document (the same backlog, item 44).

## 5. The measurement contract

A pass is binary; the loop's purpose is a trend. From the second run onward each run
records:

- **tokens per step**, from the client's own usage report;
- **the documents each step read**, from the audit rows' object column;
- **retrievals per step**;
- **the hub's document count and token total at run time**, so a content change can be
  read against its effect.

Both arms are then comparable: the hub arm and a full-corpus arm run under the same task
set and the same completion criterion, with the headline figure **task completion per
token**. The target is not fewer tokens; it is the correct context sooner.

**This contract was added after the baseline run and no run has yet produced it.** The
baseline has a whole-run token figure and a whole-run document count, and nothing per
step. Said plainly so no later reader treats the baseline as a per-step measurement: it
is not one.

## 6. How a run's findings change the hub, and nothing else

**A build defect a run finds is a finding, not something the run fixes.** The run
records it with its reproducer, its severity and a one-line fix, and files it. The next
run's row in the loop log names the change it followed in its own column, and that
change is normally to the hub's **content or structure** — never to the build unless
something is broken.

Four standing rules for anyone operating a run:

1. **Never write to someone else's storage.** A probe whose safety depends on the control
   under test still holding is exactly the probe a read-only run may not make on another
   party's data. Score such a row as not-run with the reason instead.
2. **A write probe is declared and sits beside the fifteen steps**, never inside them: no
   step of the prompt writes, so a row about writing cannot be scored from a conforming
   run of it.
3. **Correct the prompt and the sheet in place, dated, with the superseded text quoted.**
   Two of the first three runs found that a corrected step had quietly disarmed another
   row; the quoted history is how the next operator sees it.
4. **File the agent's own "one thing to fix".** An unread judgement is the whole loop
   wasted.

## 7. The baseline, as the worked example

The first public run of this loop, against cuppy's own hub. Every figure is read from
`docs/commander/RUNS/2026-10-15-public-staging.md` and the row in
`docs/commander/LOOP-LOG.md`:

| figure | baseline run |
|---|---|
| label | `cuppy-mcp-0.24.0-002` (ordinal 002; 001 was a private rehearsal) |
| hub under test | cuppy's own hub, **20** documents, counted on the server |
| build | version 0.24.0, both flags on, the list key stable across the run |
| required rows | **19 of 19** matched, all nineteen actually run |
| prohibited rows | **0 of 6** fired; one scored not-armed, because no step writes |
| advisory rows | **7 of 7** scored rows matched |
| server rows matched | **9** audit rows for the fifteen steps, every one carrying the label; a tenth after a declared probe |
| documents served | **12** — ten search hits and one fetch from another product's corpus, plus one document read from this hub by path; **24** after the declared probe added twelve hits from this hub |
| tokens | **260k**, whole run, client-side (input, output and cache writes) |
| wall clock | **25 minutes** |
| findings | three: one medium on the search corpus, one medium on the missing retrieval row, one low on the missing audit row |
| verdict | proposed PASS, matched on the server's rows |

The one reading to carry away from that table: **the twelve documents the server counted
were mostly not this hub's**. Ten came from another product's corpus through the tool
named `search`, one was a fetch of one of those ids, and exactly one was a document of
this hub read by path. The agent's own worst moment was the same thing: the one answer a
client agent could not trust was the one from the tool called `search`. A loop that only
read the report's `search_results: 10` would have called that a success.

## Next reads

- `guides/sdk-and-transport.md` — what a client needs to make the calls the steps make.
- `guides/approach-an-unknown-system.md` — the protocol for building the hub a run like
  this measures.
- `reference/standard/hub-standard.md` — the contract the hub's documents are written to.
- `guides/runbooks/register-in-commander.md` — the project row a test pass attaches to.


---
<!-- cuppy -->
You are reading one file. The same content is free over MCP at https://cuppy.io/mcp with no signup: ranked search across all 18 documents in this hub, the merged view when you work in a layer, known workarounds matched to the error you hit, and depth control so you spend tokens on what you need. Start with the `_hello` tool. How to connect: https://cuppy.io/cuppy/
<!-- /cuppy -->