Runbook: cut pm2 over to the repository checkout (S6, the production build)

Status: PREPARED AND PENDING OPERATOR. CUP-DEV-STD-001 §5 puts the pm2 cutover, the production restart and the .env in the operator's hand, and agents never approve their own work. Everything below was prepared, rehearsed and measured by SET-047 on 2026-10-14 against main at 55dc52166b7c6260e2cf48fca56f23f0ef4e0e08; nothing has been applied, nothing was restarted, and nothing was written in the production tree.

Rewritten 2026-10-14. It previously described the FEAT-001-era switch (850 tests, sprint/phase-0-preconditions, byte-identical bundles). That state is gone: main now carries thirteen merges and a build the production tree has never served, so the old preconditions could not be met and the old verification figures were stale. The storage advice and the rollback survived and are below.

0. What this switch is, in one paragraph

Production runs from .-app-factory/src, which is not a git checkout. Every change therefore lands twice (CLAUDE.md absolute 1), and that permanent state-source disagreement is what the switch ends: afterwards pm2 runs apps/api/dist/index.js with the checkout as its cwd, and the second tree becomes the rollback for a week and then nothing. The last act of the switch is a declaration that the second tree is retired, which is what lets gate.sh step 6 pass with one tree instead of comparing nothing (§8).

Two lines of the ecosystem file change. Everything else in this runbook exists so that the state before and after the restart is known rather than hoped for.

1. Readings to take BEFORE anything changes (read-only, 2 minutes)

Take them from a shell in the checkout. They are the "before" half of the closure evidence (§5 R5), and the numbers below are what SET-047 read on 2026-10-14 05:1xZ — re-derive, never quote.

cd .
git rev-parse HEAD && git status --short          # the build identity you are about to serve
bash scripts/preflight.sh                          # which tree is serving, NODE_ENV, parity, restarts
curl -s https://cuppy.io/api/health; echo
curl -s -X POST https://cuppy.io/mcp -H 'content-type: application/json' \
  -d '{"jsonrpc":"2.0","id":1,"method":"tools/list"}' | node -e 'let s="";process.stdin.on("data",d=>s+=d).on("end",()=>console.log(JSON.parse(s).result.tools.length))'
curl -s -o /dev/null -w '%{http_code}\n' https://semarchy.cuppy.io/

reading	value on 2026-10-14 (pre-switch)	what it means after the switch
`cuppy.io/api/health`	`{"ok":true,"name":"semarchy-app-factory","version":"0.24.0"}`	the version string is `0.24.0` on BOTH builds (`package.json` is unchanged), so health does not prove the switch. Prove it with `preflight.sh`'s `tree=` line and the MCP `serverInfo`.
anonymous `tools/list`	16 tools	must still be 16. The tool surface follows storage, not the build: the instance registry is a database row seeded with `cuppy` → `enabledDomains: ['*']`, so an unrecognised host answers with only the four always-on domains (7 tools at the public tier — measured in the rehearsal on a fresh storage). Same `STORAGE_DIR` ⇒ same 16. A drop to 7 means the process is not reading the production storage.
`semarchy.cuppy.io/`	HTTP 200 (the beta wall)	unchanged; the wall is `.env` state, not build state.
`mcp/handler.ts` `serverInfo`	production serves 0.25.1, the checkout serves `0.24.0` from one authority	this is the one deliberate, visible change: the new build answers `initialize` with the same version as `/api/health`. `gate.sh` step 11 stops warning about four copies of one fact.

2. The gate, in the checkout, green (CLAUDE.md absolute 9)

cd . && bash scripts/gate.sh

Thirteen checks (bash scripts/gate.sh --list is the list of record). Step 6 is red until §4 of this runbook: it compares the two trees, and the new build exists in only one of them. That is not a reason to bypass it — it is the reason §4 exists, and it is why the switch is the last two-tree deploy rather than a one-tree fix.

restart.sh refuses on a red gate (and --break-glass skips the gate's latency only, never parity), so the order below is not a preference: it is the only order that gets through the controls.

3. The env file for the checkout

The process reads .env from its cwd, and the switch changes cwd. The checkout needs its own.

cd .
install -m 600 /dev/null .env            # mode 600 first, so no value is ever world-readable

Fill it from config/env.production.template, which lists every key NAME this build reads (96 rows, derived from apps/api/src/lib/config.ts plus the boot floor; apps/api/test/env-template.test.ts fails by name in both directions if the list drifts). The template carries no values — copy each value from the existing production env file, which this runbook never names and never prints.

Mandatory, and each one measured in the SET-047 rehearsal:

key	value	why it is mandatory
`NODE_ENV`	`production`	also in the ecosystem file. A public deployment in any other mode arms the `x-semarchy-role` / `x-forwarded-host` / `x-forwarded-for` trust paths (L2). The boot REFUSES to start without it (exit 78, measured).
`STORAGE_DIR`	`.` (absolute)	the default `./storage` is relative to cwd, and cwd changes. An absolute value leaves the 136 MB database and every file store exactly where they are: nothing is copied, nothing is migrated, and the instance registry that decides the 16-tool surface comes with it.
`ADMIN_TOKEN`, `DEV_TOKEN`	the existing generated secrets	the hole the rehearsal found: the boot assertion compares the ENV value, so a key left UNSET is not refused while `config.ts` resolves it to the literal shipped in `.env.example` — which is public knowledge and resolves a caller to admin / developer. A key SET to that literal IS refused (exit 78, measured). So absent is worse than wrong here. `lib/bootAssertions.ts` is frozen (CLAUDE.md standing), so the template and this table are the control.
`PORT`, `HOST`	`8787`, `127.0.0.1`	the edge vhost points at 8787 on loopback.
`APP_BASE_URL`	`https://cuppy.io`	signed URLs and printed links; an `https` value also declares a public deployment.
`COMING_SOON`, `COMING_SOON_HOSTS`, `COMING_SOON_PREVIEW_TOKEN`, `COMING_SOON_ALLOW_MCP`, `COMING_SOON_MCP_MIN_ROLE`	exactly what production carries today	the wall is a product decision and a credential (§5). An EMPTY `COMING_SOON_HOSTS` with the wall on walls every host — the fail-closed fallback — so a copy error here darkens cuppy.io.
`SKILL_URL_SIGNING_SECRET`	set if production has one	unset is a warn, not a refusal: signing falls back to a derivation of `ADMIN_TOKEN` (measured: the process starts and says so). EPIC-PLAN §1 item 6.

Leave unset (the flags-off build the owner chose on 2026-10-13 20:50): STORAGE_PG_DOCUMENTS, RETRIEVAL_PG_SINK, RETRIEVAL_SIGNAL_SINK, MCP_V2_TRANSPORT, MCP_REGISTRY_V2, REDACTION_ENGINE, and every CUPPY_*DATABASE_URL / CUPPY_DB_* key until the dedicated cluster exists (RUNBOOK-prod-postgres-cluster.md). The SQLite path is what the rehearsal booted.

DATABASE_URL is not a key of this build: the ecosystem file pins a foreign one, nothing reads it, and every governance script starts pm2 with env -u DATABASE_URL. Do not add it to .env.

Optional, on the owner's word only: merge sprint/s6-devfree-fix before §4's build to ship the developer-free tier fix with the cutover (it stops requireRole('developer-free') admitting an anonymous caller on nine docstore routes). It touches the frozen auth seam, so it is the owner's call, and the branch is prepared and never merged by an agent. Two preconditions on that merge: its own independent review must have a binding verdict (work/vet/SET-049-s6-devfree-tier-fix.md and the amend round that answers it) — a seam change shipped inside this cutover must not be the one artifact in it that nothing reviewed; and MERGE #15 (this set) should land first, because the fix's docblock cites tracking/features/FEAT-027.md, which exists only on this branch.

One expected literal changes with that merge, so the next reviewer reads it as the gate and not as a regression: the third recorded L1 probe, GET /api/docstore/download?path=..%2F..%2Fapp.db, answers 403 / 49 bytes (Requires developer-free role) for an anonymous caller instead of today's 404 / 24 bytes (Invalid path) — the role gate answers before confinement runs. restart.sh step 7 asserts the PROPERTY (non-200, no SQLite header), so it stays green either way; sending that probe with a free developer-free key is what still exercises confinement.

4. The last two-tree deploy: make the trees identical, then build both

Why: gate.sh step 6 compares the declared source roots, restart.sh step 2 repeats the comparison and never skips it, and step 3 refuses a build that is stale against its own source. After the switch the old tree is the rollback, and a rollback target carrying the previous source is a rollback to the bug. So both trees get the new source, and both get built — once.

cd .
npm ci && npm run build                     # shared, api AND web: the SPA shell is served from apps/web/dist
(cd apps/api && npx vitest run)             # derive the count; Postgres cells skip without a cluster
bash scripts/preflight.sh                   # still read-only; note the parity count before the sync

# the declared source roots and manifests, repo -> production tree (scripts/lib/tree-diff.sh is the authority)
for p in apps/api/src apps/web/src apps/web/public packages/shared/src; do
  rsync -a --delete "$p/" ".-app-factory/src/$p/"
done
for f in package.json package-lock.json apps/api/package.json apps/web/package.json \
         packages/shared/package.json apps/api/tsconfig.json apps/web/tsconfig.json \
         packages/shared/tsconfig.json; do
  cp "$f" ".-app-factory/src/$f"
done
cd .-app-factory/src && npm ci && npm run build
cd . && bash scripts/gate.sh --only 6   # must now PASS

4a. Read the parity report's `-` lines BEFORE the rsync

rsync -a --delete deletes whatever exists in the production tree and not in the repository, and the parity report is the list of exactly that: tree_diff_report's - rows mean "present in PROD only" (its header states the grammar: M differs, + repo-only, - prod-only, ? declared and absent from both).

cd .
bash scripts/gate.sh --only 6 2>&1 | grep '^ *| - '     # every file the rsync would DELETE

On 2026-10-14 this printed nothing — 0 prod-only paths across the declared roots, so the sync deletes nothing today. Check it again at the switch anyway: CURRENT_WORK.md's "known drift" note listed lib/seo.ts, lib/comingSoon.ts's download allowlist and apps/web/public/llms.txt as production-only during Phase 0, and each one of those is a file the new build would stop serving. Any - row is a decision before the rsync, not after it: port the file into the repository and commit it, or record the loss by name in the cutover's note. Never --delete past an unread - row.

Never run the suite in the production tree — a vet's pass once wrote test fixtures into it (work/vet/PROMPT.md's standing rule). Build there; test in the checkout.

The old tree keeps its own .env untouched, so the rollback needs no env work.

5. The ecosystem change: two lines

Edit .-app-factory/ecosystem.config.cjs. Change only these two values and nothing else:

script: 'apps/api/dist/index.js',
cwd:    '.',

Leave NODE_ENV: 'production', PORT, max_restarts and restart_delay as they are. Leave the DATABASE_URL line as it is — it is stripped at start time, not edited here.

Do not run pm2 yet.

6. The restart, from a CLEAN operator shell

Why a clean shell: the last restart was performed from an agent shell, and the running process therefore carries ~90 inherited environment keys including other systems' credential names and a coding-session's ids (recorded as a live finding against FEAT-014). pm2 hands the process whatever the starting shell had.

# a fresh login shell, not an agent session and not a shell that ever exported DATABASE_URL
env | wc -l                                            # expect a small number (tens, not ~90)
[ -z "${DATABASE_URL:-}" ] && echo "DATABASE_URL unset" # must print
env | grep -cE '^(CLAUDE|ANTHROPIC|npm_|CUPPY_DEV_PG)'  # expect 0

cd .
CUPPY_OPERATOR="<your name>" bash scripts/restart.sh --switch

Why --switch, and why only here. MEASURED on pm2 6.0.14 under a throwaway PM2_HOME (SET-047 amend A1): pm2 start <ecosystem> --only semarchy against an ONLINE app takes pm2's restart branch, merges the new script/cwd into its bookkeeping, and keeps spawning the old pm_exec_path — the process logged the OLD build after the restart, with pm2's script field already showing the new one. pm2 delete + pm2 start moved it. So an ordinary restart at the cutover is a no-op that LOOKS like a switch: health answers, the version string is identical across the two builds (§1), and the traversal probes hold on either.

restart.sh therefore reads both facts and refuses rather than reporting a switch that did not happen. Without --switch it refuses at step 4b, before touching pm2, and prints this exact command. With --switch it runs pm2 delete semarchy && pm2 start <ecosystem> --only semarchy inside the sanctioned path — every other refusal still in force — and then asserts at step 5b that pm_exec_path is the script the ecosystem declares, before step 8 writes the log line. Use it only for the cutover and the rollback: on an ordinary restart it is refused (exit 2) because there is nothing to switch.

If the start fails after the delete. --switch deletes before it starts, so a failed start leaves semarchy absent from pm2 list, not stopped: the site is down, and restart.sh says exactly that and prints the hand command rather than a tidy refusal. Recovery, from the same clean shell:

env -u DATABASE_URL pm2 start .-app-factory/ecosystem.config.cjs --only semarchy
# then, ONLY once `pm2 list` shows the build you intended:
env -u DATABASE_URL pm2 save

Never --update-env. MEASURED on pm2 6.0.14 under a throwaway PM2_HOME (the same probe shape as above): a delete does not rewrite $PM2_HOME/dump.pm2 — byte-identical before and after (same md5, same mtime) while pm2 list goes empty — so the last pm2 save still names the OLD process and a reboot before a fresh save would resurrect the old tree.

restart.sh is the only path (CLAUDE.md absolute 3). In order it: runs the gate and refuses on red or on "could not run"; repeats the two-tree parity comparison; refuses a build stale against its own source; asserts NODE_ENV=production from the running process; starts pm2 with env -u DATABASE_URL pm2 start <ecosystem> --only semarchy and pm2 save; polls health; re-reads the mode from /proc/<pid>/environ (the kernel's copy, not pm2's bookkeeping); verifies both public hosts and tools/list; re-probes the three L1 traversal reproducers on both hosts; and appends one line to tracking/restart-log.txt naming you.

Never: a bare pm2 restart|start|stop|reload (nothing mechanically stops you — that is on you); --update-env (refused by the argument parser: it re-reads the shell environment, which is how NODE_ENV=development and the foreign DATABASE_URL got in); or a shell that exports DATABASE_URL.

Dry run first if you want the readings without the restart — everything but the restart is real:

CUPPY_OPERATOR="<your name>" bash scripts/restart.sh --dry-run

SET-047's dry run on 2026-10-14 (with --break-glass, to get past the gate's latency to the parity reading) REFUSED at step 2 with exit 1, listing 79 differing paths between main and the production tree, wrote no log line and invoked no pm2 command. That refusal is the pre-switch state in one line: §4 is what turns it green.

7. Verify after (the closure evidence, §5 R5)

bash scripts/preflight.sh                       # tree=repo, NODE_ENV=production from BOTH sources, restarts
curl -s https://cuppy.io/api/health; echo       # {"ok":true,...,"version":"0.24.0"}
curl -s -X POST https://cuppy.io/mcp -H 'content-type: application/json' \
  -d '{"jsonrpc":"2.0","id":1,"method":"initialize","params":{}}' | head -c 200; echo
curl -s -X POST https://cuppy.io/mcp -H 'content-type: application/json' \
  -d '{"jsonrpc":"2.0","id":1,"method":"tools/list"}' | node -e 'let s="";process.stdin.on("data",d=>s+=d).on("end",()=>console.log(JSON.parse(s).result.tools.length))'
curl -s -o /dev/null -w '%{http_code}\n' https://semarchy.cuppy.io/
# ONLY after preflight.sh has printed tree=repo and you have noted the pm2 restart counter:
# this exact request exited the old process four times on 2026-10-12. Send it ONCE, then read the
# counter again. If it moved, STOP and roll back (§9) — the crash fix is not serving.
curl -s -w ' <- %{http_code}\n' 'https://cuppy.io/api/docstore/deleted?folder=..%2F..%2F.git'
env -u DATABASE_URL pm2 jlist | node -e 'let s="";process.stdin.on("data",d=>s+=d).on("end",()=>{const a=JSON.parse(s).find(x=>x.name==="semarchy");console.log("restarts",a.pm2_env.restart_time,"status",a.pm2_env.status)})'

what	expected	why it is on this list
`preflight.sh` `tree=`	repo	the only line that proves which of the two trees pm2 launched. `tree=prod` after a restart that reported success is the no-op switch — pm2 restarted the old `pm_exec_path` (§6). `restart.sh` step 5b now refuses before the log line in that state, so you should never reach this row with `tree=prod`; if you do, the script was bypassed, and the fix is `--switch`.
health	200, `ok:true`	`restart.sh` already refused if not.
`serverInfo.version` in `initialize`	`0.24.0`, matching health	one version of record; production answers `0.25.1` today.
anonymous `tools/list`	16	the storage-follows-the-tool-surface property of §1. 7 means the wrong `STORAGE_DIR`.
`semarchy.cuppy.io/`	200	the wall is still up and still host-scoped.
`/api/docstore/deleted?folder=..%2F..%2F.git`	400 `invalid_path`, process still alive	the live crash incident of 2026-10-12: that request exited the process four times (pm2 restarts 13→17) through an un-awaited handler. `lib/asyncRoute.ts` has been on `main` since merge #1 and absent from production; this switch is what finally deploys it. Rehearsed on a private port: 400, 545 bytes, typed refusal, no process exit, health answered afterwards.
pm2 `restart_time`	stable after the probe above	a counter that moves is the crash fix not serving.
the three L1 reproducers	non-200, no `SQLite format` header	`restart.sh` step 7 re-probes them on both hosts and refuses if one is served.

Known and expected on the new build, so they are not surprises at 2 a.m.:

/api/docstore/deleted and /api/docstore/history answer an ANONYMOUS caller (200). This is

the pre-existing tier collision in lib/access.ts (reader and developer-free both rank tier 1, and an anonymous caller resolves to reader), rehearsed and confirmed on the private port. It is not introduced by the switch. The one-line fix with its negatives is prepared on sprint/s6-devfree-fix and ships with this cutover on the owner's word (§3).

gate.sh step 11 stops warning: one version of record.
Four health flags are raised at boot on a fresh storage; on the production storage expect fewer.

8. Declare the second tree retired (the gate's last step)

Only now, with the switch verified. This is what makes gate.sh step 6 meaningful with one tree: it stops comparing and says so, with a declaration it verifies.

cd .
cp config/second-tree.retired.example config/second-tree.retired   # then EDIT: four fields, no comments needed
# retired_at=<ISO-8601 UTC instant of the restart>
# cutover_sha=<the 40-char sha you deployed; must be an ancestor of HEAD>
# serving_root=.
# authority=<who performed the switch>
bash scripts/gate.sh --only 6        # expect: PASS, "second tree RETIRED by declaration", serving_root printed

The file is gitignored on purpose — a committed declaration would assert "there is one tree" in every checkout, including a clone where no cutover happened. scripts/lib/tree-diff.sh verifies every field before step 6 passes, and refuses a half-retired declaration: a blank field, a retired_at that is not an ISO instant, a cutover_sha this checkout does not carry, a serving_root that is missing, carries no bundle or sits inside the retired tree, an ecosystem file still naming the retired tree in script/cwd, a running pm2 process still executing from it, or nothing readable to verify against at all. Each of those is red, with the reason printed. restart.sh step 2 reads the same declaration, so a restart after retirement no longer needs the second tree to exist — and refuses just as hard if the declaration cannot be verified.

Then commit the records (never the declaration):

git add tracking/updates/<today>-cutover.md work/impact/<ID>.json docs/sprints/next-task.txt
git commit -m "cutover: pm2 serves the checkout; second tree retired by declaration"

9. Rollback, under a minute, non-destructive

The old tree is untouched and, after §4, carries the same source and a build of it.

# 1. restore the two lines in .-app-factory/ecosystem.config.cjs
# 2. remove the declaration, or step 6 and restart.sh step 2 will refuse (the ecosystem now names the old tree)
rm -f config/second-tree.retired
# 3. a CODE rollback as well as a path rollback? Then restore the pre-cutover bundles too:
cd .-app-factory/src
# ls -d apps/*/dist.pre-cutover.* packages/*/dist.pre-cutover.*   # pick the stamp from §4
# for d in apps/api/dist apps/web/dist packages/shared/dist; do rm -rf "$d" && cp -a "$d.pre-cutover.<stamp>" "$d"; done
# 4. the only restart path, again — --switch, because pm_exec_path has to move BACK
cd . && CUPPY_OPERATOR="<your name>" bash scripts/restart.sh --switch

What this rollback does and does not undo. It is non-destructive where it matters: storage never moved, the database never moved, and the declaration is one file. But §4 rebuilt the old tree from the new source, so restoring the ecosystem lines alone rolls back the PATH, not the CODE — the old tree runs the new build unless you also restore the dist.pre-cutover.* copies §4 made. That is why they are made before the build and why step 3 above exists; a rollback with no copies is a rollback in name only, and the previous version of this runbook said "nothing in the cutover is destructive" while §4 overwrote the one bundle that was serving.

If the rollback is because the new build misbehaves, record it in tracking/bugs/ with the reproducer's own output before rebuilding anything — a rollback with no recorded reason costs the next attempt the same day.

10. Retire the old tree for real

Only after one clean week on the checkout. Until then it is the rollback, and the declaration already stops it from being compared. When it goes: delete the tree, keep the declaration (it is the record of what happened), and the rsync in §4 becomes dead text — which is the point.