--- cuppy_hub: cuppy cuppy_path: guides/runbooks/cutover-to-checkout.md cuppy_section: docs cuppy_doc: XDGR-960W cuppy_doc_uuid: 8330853a-47d1-4b46-8132-3f6ddb6462c0 cuppy_permalink: https://cuppy.io/cuppy/d/XDGR-960W cuppy_version: 1 cuppy_updated: 2026-10-17T06:32:45.511Z cuppy_canonical: https://cuppy.io/cuppy/docs/guides/runbooks/cutover-to-checkout cuppy_mcp: https://cuppy.io/mcp cuppy_body_sha256: a3ce8cd4e4ccdddb8e2d0bcdc583982863d4d7d663f44c83740eef1be0e9de24 # cuppy_body_sha256 is over the document body ALONE, excluding both generated blocks. --- # Runbook: cut pm2 over to the repository checkout (S6, the production build) **Status: PREPARED AND PENDING OPERATOR.** CUP-DEV-STD-001 §5 puts the pm2 cutover, the production restart and the `.env` in the operator's hand, and agents never approve their own work. Everything below was prepared, rehearsed and measured by SET-047 on 2026-10-14 against `main` at `55dc52166b7c6260e2cf48fca56f23f0ef4e0e08`; **nothing has been applied, nothing was restarted, and nothing was written in the production tree.** Rewritten 2026-10-14. It previously described the FEAT-001-era switch (850 tests, `sprint/phase-0-preconditions`, byte-identical bundles). That state is gone: `main` now carries thirteen merges and a build the production tree has never served, so the old preconditions could not be met and the old verification figures were stale. The storage advice and the rollback survived and are below. --- ## 0. What this switch is, in one paragraph Production runs from `.-app-factory/src`, which is **not a git checkout**. Every change therefore lands twice (CLAUDE.md absolute 1), and that permanent state-source disagreement is what the switch ends: afterwards pm2 runs `apps/api/dist/index.js` with the checkout as its cwd, and the second tree becomes the rollback for a week and then nothing. The last act of the switch is a **declaration** that the second tree is retired, which is what lets `gate.sh` step 6 pass with one tree instead of comparing nothing (§8). Two lines of the ecosystem file change. Everything else in this runbook exists so that the state before and after the restart is known rather than hoped for. --- ## 1. Readings to take BEFORE anything changes (read-only, 2 minutes) Take them from a shell in the checkout. They are the "before" half of the closure evidence (§5 R5), and the numbers below are what SET-047 read on 2026-10-14 05:1xZ — re-derive, never quote. ```bash cd . git rev-parse HEAD && git status --short # the build identity you are about to serve bash scripts/preflight.sh # which tree is serving, NODE_ENV, parity, restarts curl -s https://cuppy.io/api/health; echo curl -s -X POST https://cuppy.io/mcp -H 'content-type: application/json' \ -d '{"jsonrpc":"2.0","id":1,"method":"tools/list"}' | node -e 'let s="";process.stdin.on("data",d=>s+=d).on("end",()=>console.log(JSON.parse(s).result.tools.length))' curl -s -o /dev/null -w '%{http_code}\n' https://semarchy.cuppy.io/ ``` | reading | value on 2026-10-14 (pre-switch) | what it means after the switch | |---|---|---| | `cuppy.io/api/health` | `{"ok":true,"name":"semarchy-app-factory","version":"0.24.0"}` | the version string is `0.24.0` on BOTH builds (`package.json` is unchanged), so **health does not prove the switch**. Prove it with `preflight.sh`'s `tree=` line and the MCP `serverInfo`. | | anonymous `tools/list` | **16** tools | must still be 16. The tool surface follows **storage**, not the build: the instance registry is a database row seeded with `cuppy` → `enabledDomains: ['*']`, so an unrecognised host answers with only the four always-on domains (7 tools at the public tier — measured in the rehearsal on a fresh storage). Same `STORAGE_DIR` ⇒ same 16. A drop to 7 means the process is not reading the production storage. | | `semarchy.cuppy.io/` | HTTP **200** (the beta wall) | unchanged; the wall is `.env` state, not build state. | | `mcp/handler.ts` `serverInfo` | production serves **0.25.1**, the checkout serves `0.24.0` from one authority | this is the one deliberate, visible change: the new build answers `initialize` with the same version as `/api/health`. `gate.sh` step 11 stops warning about four copies of one fact. | --- ## 2. The gate, in the checkout, green (CLAUDE.md absolute 9) ```bash cd . && bash scripts/gate.sh ``` Thirteen checks (`bash scripts/gate.sh --list` is the list of record). **Step 6 is red until §4 of this runbook**: it compares the two trees, and the new build exists in only one of them. That is not a reason to bypass it — it is the reason §4 exists, and it is why the switch is the **last two-tree deploy** rather than a one-tree fix. `restart.sh` refuses on a red gate (and `--break-glass` skips the gate's *latency only*, never parity), so the order below is not a preference: it is the only order that gets through the controls. --- ## 3. The env file for the checkout The process reads `.env` from its **cwd**, and the switch changes cwd. The checkout needs its own. ```bash cd . install -m 600 /dev/null .env # mode 600 first, so no value is ever world-readable ``` Fill it from **`config/env.production.template`**, which lists every key NAME this build reads (96 rows, derived from `apps/api/src/lib/config.ts` plus the boot floor; `apps/api/test/env-template.test.ts` fails by name in both directions if the list drifts). The template carries **no values** — copy each value from the existing production env file, which this runbook never names and never prints. Mandatory, and each one measured in the SET-047 rehearsal: | key | value | why it is mandatory | |---|---|---| | `NODE_ENV` | `production` | also in the ecosystem file. A public deployment in any other mode arms the `x-semarchy-role` / `x-forwarded-host` / `x-forwarded-for` trust paths (L2). **The boot REFUSES to start without it** (exit 78, measured). | | `STORAGE_DIR` | `.` (absolute) | the default `./storage` is relative to cwd, and cwd changes. An absolute value leaves the 136 MB database and every file store exactly where they are: **nothing is copied, nothing is migrated**, and the instance registry that decides the 16-tool surface comes with it. | | `ADMIN_TOKEN`, `DEV_TOKEN` | the existing generated secrets | **the hole the rehearsal found:** the boot assertion compares the ENV value, so a key left UNSET is not refused while `config.ts` resolves it to the literal shipped in `.env.example` — which is public knowledge and resolves a caller to admin / developer. A key SET to that literal IS refused (exit 78, measured). So absent is worse than wrong here. `lib/bootAssertions.ts` is frozen (CLAUDE.md standing), so the template and this table are the control. | | `PORT`, `HOST` | `8787`, `127.0.0.1` | the edge vhost points at 8787 on loopback. | | `APP_BASE_URL` | `https://cuppy.io` | signed URLs and printed links; an `https` value also declares a public deployment. | | `COMING_SOON`, `COMING_SOON_HOSTS`, `COMING_SOON_PREVIEW_TOKEN`, `COMING_SOON_ALLOW_MCP`, `COMING_SOON_MCP_MIN_ROLE` | exactly what production carries today | the wall is a product decision and a credential (§5). **An EMPTY `COMING_SOON_HOSTS` with the wall on walls every host** — the fail-closed fallback — so a copy error here darkens cuppy.io. | | `SKILL_URL_SIGNING_SECRET` | set if production has one | unset is a **warn**, not a refusal: signing falls back to a derivation of `ADMIN_TOKEN` (measured: the process starts and says so). EPIC-PLAN §1 item 6. | Leave **unset** (the flags-off build the owner chose on 2026-10-13 20:50): `STORAGE_PG_DOCUMENTS`, `RETRIEVAL_PG_SINK`, `RETRIEVAL_SIGNAL_SINK`, `MCP_V2_TRANSPORT`, `MCP_REGISTRY_V2`, `REDACTION_ENGINE`, and every `CUPPY_*DATABASE_URL` / `CUPPY_DB_*` key until the dedicated cluster exists (`RUNBOOK-prod-postgres-cluster.md`). The SQLite path is what the rehearsal booted. `DATABASE_URL` is **not** a key of this build: the ecosystem file pins a foreign one, nothing reads it, and every governance script starts pm2 with `env -u DATABASE_URL`. Do not add it to `.env`. **Optional, on the owner's word only:** merge `sprint/s6-devfree-fix` before §4's build to ship the developer-free tier fix with the cutover (it stops `requireRole('developer-free')` admitting an anonymous caller on nine docstore routes). It touches the frozen auth seam, so it is the owner's call, and the branch is prepared and never merged by an agent. **Two preconditions on that merge:** its own independent review must have a binding verdict (`work/vet/SET-049-s6-devfree-tier-fix.md` and the amend round that answers it) — a seam change shipped inside this cutover must not be the one artifact in it that nothing reviewed; and MERGE #15 (this set) should land first, because the fix's docblock cites `tracking/features/FEAT-027.md`, which exists only on this branch. **One expected literal changes with that merge**, so the next reviewer reads it as the gate and not as a regression: the third recorded L1 probe, `GET /api/docstore/download?path=..%2F..%2Fapp.db`, answers **403 / 49 bytes** (`Requires developer-free role`) for an anonymous caller instead of today's 404 / 24 bytes (`Invalid path`) — the role gate answers before confinement runs. `restart.sh` step 7 asserts the PROPERTY (non-200, no SQLite header), so it stays green either way; sending that probe with a free `developer-free` key is what still exercises confinement. --- ## 4. The last two-tree deploy: make the trees identical, then build both Why: `gate.sh` step 6 compares the declared source roots, `restart.sh` step 2 repeats the comparison and **never** skips it, and step 3 refuses a build that is stale against its own source. After the switch the old tree is the rollback, and a rollback target carrying the previous source is a rollback to the bug. So both trees get the new source, and both get built — once. ```bash cd . npm ci && npm run build # shared, api AND web: the SPA shell is served from apps/web/dist (cd apps/api && npx vitest run) # derive the count; Postgres cells skip without a cluster bash scripts/preflight.sh # still read-only; note the parity count before the sync # the declared source roots and manifests, repo -> production tree (scripts/lib/tree-diff.sh is the authority) for p in apps/api/src apps/web/src apps/web/public packages/shared/src; do rsync -a --delete "$p/" ".-app-factory/src/$p/" done for f in package.json package-lock.json apps/api/package.json apps/web/package.json \ packages/shared/package.json apps/api/tsconfig.json apps/web/tsconfig.json \ packages/shared/tsconfig.json; do cp "$f" ".-app-factory/src/$f" done cd .-app-factory/src && npm ci && npm run build cd . && bash scripts/gate.sh --only 6 # must now PASS ``` ### 4a. Read the parity report's `-` lines BEFORE the rsync `rsync -a --delete` deletes whatever exists in the production tree and not in the repository, and the parity report is the list of exactly that: `tree_diff_report`'s `-` rows mean "present in PROD only" (its header states the grammar: `M` differs, `+` repo-only, `-` prod-only, `?` declared and absent from both). ```bash cd . bash scripts/gate.sh --only 6 2>&1 | grep '^ *| - ' # every file the rsync would DELETE ``` **On 2026-10-14 this printed nothing** — 0 prod-only paths across the declared roots, so the sync deletes nothing today. Check it again at the switch anyway: `CURRENT_WORK.md`'s "known drift" note listed `lib/seo.ts`, `lib/comingSoon.ts`'s download allowlist and `apps/web/public/llms.txt` as production-only during Phase 0, and each one of those is a file the new build would stop serving. Any `-` row is a decision before the rsync, not after it: port the file into the repository and commit it, or record the loss by name in the cutover's note. Never `--delete` past an unread `-` row. **Never run the suite in the production tree** — a vet's pass once wrote test fixtures into it (work/vet/PROMPT.md's standing rule). Build there; test in the checkout. The old tree keeps its own `.env` untouched, so the rollback needs no env work. --- ## 5. The ecosystem change: two lines Edit `.-app-factory/ecosystem.config.cjs`. Change **only** these two values and nothing else: ``` script: 'apps/api/dist/index.js', cwd: '.', ``` Leave `NODE_ENV: 'production'`, `PORT`, `max_restarts` and `restart_delay` as they are. Leave the `DATABASE_URL` line as it is — it is stripped at start time, not edited here. Do not run pm2 yet. --- ## 6. The restart, from a CLEAN operator shell Why a clean shell: the last restart was performed from an agent shell, and the running process therefore carries ~90 inherited environment keys including other systems' credential names and a coding-session's ids (recorded as a live finding against FEAT-014). pm2 hands the process whatever the starting shell had. ```bash # a fresh login shell, not an agent session and not a shell that ever exported DATABASE_URL env | wc -l # expect a small number (tens, not ~90) [ -z "${DATABASE_URL:-}" ] && echo "DATABASE_URL unset" # must print env | grep -cE '^(CLAUDE|ANTHROPIC|npm_|CUPPY_DEV_PG)' # expect 0 cd . CUPPY_OPERATOR="" bash scripts/restart.sh --switch ``` **Why `--switch`, and why only here.** MEASURED on pm2 6.0.14 under a throwaway `PM2_HOME` (SET-047 amend A1): `pm2 start --only semarchy` against an ONLINE app takes pm2's restart branch, merges the new `script`/`cwd` into its bookkeeping, and **keeps spawning the old `pm_exec_path`** — the process logged the OLD build after the restart, with pm2's `script` field already showing the new one. `pm2 delete` + `pm2 start` moved it. So an ordinary restart at the cutover is a no-op that LOOKS like a switch: health answers, the version string is identical across the two builds (§1), and the traversal probes hold on either. `restart.sh` therefore reads both facts and refuses rather than reporting a switch that did not happen. Without `--switch` it refuses at **step 4b**, before touching pm2, and prints this exact command. With `--switch` it runs `pm2 delete semarchy && pm2 start --only semarchy` inside the sanctioned path — every other refusal still in force — and then asserts at **step 5b** that `pm_exec_path` is the script the ecosystem declares, before step 8 writes the log line. Use it **only** for the cutover and the rollback: on an ordinary restart it is refused (exit 2) because there is nothing to switch. **If the start fails after the delete.** `--switch` deletes before it starts, so a failed start leaves `semarchy` **absent from `pm2 list`**, not stopped: the site is down, and `restart.sh` says exactly that and prints the hand command rather than a tidy refusal. Recovery, from the same clean shell: ```bash env -u DATABASE_URL pm2 start .-app-factory/ecosystem.config.cjs --only semarchy # then, ONLY once `pm2 list` shows the build you intended: env -u DATABASE_URL pm2 save ``` Never `--update-env`. MEASURED on pm2 6.0.14 under a throwaway `PM2_HOME` (the same probe shape as above): a `delete` does **not** rewrite `$PM2_HOME/dump.pm2` — byte-identical before and after (same md5, same mtime) while `pm2 list` goes empty — so the last `pm2 save` still names the OLD process and a reboot before a fresh save would resurrect the old tree. `restart.sh` is the only path (CLAUDE.md absolute 3). In order it: runs the gate and refuses on red or on "could not run"; repeats the two-tree parity comparison; refuses a build stale against its own source; asserts `NODE_ENV=production` from the **running process**; starts pm2 with `env -u DATABASE_URL pm2 start --only semarchy` and `pm2 save`; polls health; re-reads the mode from `/proc//environ` (the kernel's copy, not pm2's bookkeeping); verifies both public hosts and `tools/list`; re-probes the three L1 traversal reproducers on both hosts; and appends one line to `tracking/restart-log.txt` naming you. **Never:** a bare `pm2 restart|start|stop|reload` (nothing mechanically stops you — that is on you); `--update-env` (refused by the argument parser: it re-reads the shell environment, which is how `NODE_ENV=development` and the foreign `DATABASE_URL` got in); or a shell that exports `DATABASE_URL`. Dry run first if you want the readings without the restart — everything but the restart is real: ```bash CUPPY_OPERATOR="" bash scripts/restart.sh --dry-run ``` SET-047's dry run on 2026-10-14 (with `--break-glass`, to get past the gate's latency to the parity reading) **REFUSED at step 2 with exit 1**, listing **79** differing paths between `main` and the production tree, wrote no log line and invoked no pm2 command. That refusal is the pre-switch state in one line: §4 is what turns it green. --- ## 7. Verify after (the closure evidence, §5 R5) ```bash bash scripts/preflight.sh # tree=repo, NODE_ENV=production from BOTH sources, restarts curl -s https://cuppy.io/api/health; echo # {"ok":true,...,"version":"0.24.0"} curl -s -X POST https://cuppy.io/mcp -H 'content-type: application/json' \ -d '{"jsonrpc":"2.0","id":1,"method":"initialize","params":{}}' | head -c 200; echo curl -s -X POST https://cuppy.io/mcp -H 'content-type: application/json' \ -d '{"jsonrpc":"2.0","id":1,"method":"tools/list"}' | node -e 'let s="";process.stdin.on("data",d=>s+=d).on("end",()=>console.log(JSON.parse(s).result.tools.length))' curl -s -o /dev/null -w '%{http_code}\n' https://semarchy.cuppy.io/ # ONLY after preflight.sh has printed tree=repo and you have noted the pm2 restart counter: # this exact request exited the old process four times on 2026-10-12. Send it ONCE, then read the # counter again. If it moved, STOP and roll back (§9) — the crash fix is not serving. curl -s -w ' <- %{http_code}\n' 'https://cuppy.io/api/docstore/deleted?folder=..%2F..%2F.git' env -u DATABASE_URL pm2 jlist | node -e 'let s="";process.stdin.on("data",d=>s+=d).on("end",()=>{const a=JSON.parse(s).find(x=>x.name==="semarchy");console.log("restarts",a.pm2_env.restart_time,"status",a.pm2_env.status)})' ``` | what | expected | why it is on this list | |---|---|---| | `preflight.sh` `tree=` | **repo** | the only line that proves which of the two trees pm2 launched. **`tree=prod` after a restart that reported success is the no-op switch** — pm2 restarted the old `pm_exec_path` (§6). `restart.sh` step 5b now refuses before the log line in that state, so you should never reach this row with `tree=prod`; if you do, the script was bypassed, and the fix is `--switch`. | | health | 200, `ok:true` | `restart.sh` already refused if not. | | `serverInfo.version` in `initialize` | `0.24.0`, matching health | one version of record; production answers `0.25.1` today. | | anonymous `tools/list` | **16** | the storage-follows-the-tool-surface property of §1. 7 means the wrong `STORAGE_DIR`. | | `semarchy.cuppy.io/` | 200 | the wall is still up and still host-scoped. | | `/api/docstore/deleted?folder=..%2F..%2F.git` | **400** `invalid_path`, process **still alive** | the live crash incident of 2026-10-12: that request exited the process four times (pm2 restarts 13→17) through an un-awaited handler. `lib/asyncRoute.ts` has been on `main` since merge #1 and absent from production; this switch is what finally deploys it. Rehearsed on a private port: 400, 545 bytes, typed refusal, no process exit, health answered afterwards. | | pm2 `restart_time` | stable after the probe above | a counter that moves is the crash fix not serving. | | the three L1 reproducers | non-200, no `SQLite format` header | `restart.sh` step 7 re-probes them on both hosts and refuses if one is served. | Known and expected on the new build, so they are not surprises at 2 a.m.: - **`/api/docstore/deleted` and `/api/docstore/history` answer an ANONYMOUS caller** (200). This is the pre-existing tier collision in `lib/access.ts` (`reader` and `developer-free` both rank tier 1, and an anonymous caller resolves to `reader`), rehearsed and confirmed on the private port. It is **not** introduced by the switch. The one-line fix with its negatives is prepared on `sprint/s6-devfree-fix` and ships with this cutover on the owner's word (§3). - `gate.sh` step 11 stops warning: one version of record. - Four health flags are raised at boot on a fresh storage; on the production storage expect fewer. --- ## 8. Declare the second tree retired (the gate's last step) Only now, with the switch verified. This is what makes `gate.sh` step 6 meaningful with one tree: it stops comparing and **says so, with a declaration it verifies**. ```bash cd . cp config/second-tree.retired.example config/second-tree.retired # then EDIT: four fields, no comments needed # retired_at= # cutover_sha= # serving_root=. # authority= bash scripts/gate.sh --only 6 # expect: PASS, "second tree RETIRED by declaration", serving_root printed ``` The file is **gitignored on purpose** — a committed declaration would assert "there is one tree" in every checkout, including a clone where no cutover happened. `scripts/lib/tree-diff.sh` verifies every field before step 6 passes, and refuses a **half-retired** declaration: a blank field, a `retired_at` that is not an ISO instant, a `cutover_sha` this checkout does not carry, a `serving_root` that is missing, carries no bundle or sits inside the retired tree, an ecosystem file still naming the retired tree in `script`/`cwd`, a running pm2 process still executing from it, or nothing readable to verify against at all. Each of those is red, with the reason printed. `restart.sh` step 2 reads the same declaration, so a restart after retirement no longer needs the second tree to exist — and refuses just as hard if the declaration cannot be verified. Then commit the records (never the declaration): ```bash git add tracking/updates/-cutover.md work/impact/.json docs/sprints/next-task.txt git commit -m "cutover: pm2 serves the checkout; second tree retired by declaration" ``` --- ## 9. Rollback, under a minute, non-destructive The old tree is untouched and, after §4, carries the same source **and** a build of it. ```bash # 1. restore the two lines in .-app-factory/ecosystem.config.cjs # 2. remove the declaration, or step 6 and restart.sh step 2 will refuse (the ecosystem now names the old tree) rm -f config/second-tree.retired # 3. a CODE rollback as well as a path rollback? Then restore the pre-cutover bundles too: cd .-app-factory/src # ls -d apps/*/dist.pre-cutover.* packages/*/dist.pre-cutover.* # pick the stamp from §4 # for d in apps/api/dist apps/web/dist packages/shared/dist; do rm -rf "$d" && cp -a "$d.pre-cutover." "$d"; done # 4. the only restart path, again — --switch, because pm_exec_path has to move BACK cd . && CUPPY_OPERATOR="" bash scripts/restart.sh --switch ``` **What this rollback does and does not undo.** It is non-destructive where it matters: storage never moved, the database never moved, and the declaration is one file. But §4 rebuilt the old tree from the new source, so **restoring the ecosystem lines alone rolls back the PATH, not the CODE** — the old tree runs the new build unless you also restore the `dist.pre-cutover.*` copies §4 made. That is why they are made before the build and why step 3 above exists; a rollback with no copies is a rollback in name only, and the previous version of this runbook said "nothing in the cutover is destructive" while §4 overwrote the one bundle that was serving. If the rollback is because the new build misbehaves, record it in `tracking/bugs/` with the reproducer's own output before rebuilding anything — a rollback with no recorded reason costs the next attempt the same day. --- ## 10. Retire the old tree for real Only after **one clean week** on the checkout. Until then it is the rollback, and the declaration already stops it from being compared. When it goes: delete the tree, keep the declaration (it is the record of what happened), and the `rsync` in §4 becomes dead text — which is the point. --- You are reading one file. The same content is free over MCP at https://cuppy.io/mcp with no signup: ranked search across all 18 documents in this hub, the merged view when you work in a layer, known workarounds matched to the error you hit, and depth control so you spend tokens on what you need. Start with the `_hello` tool. How to connect: https://cuppy.io/cuppy/