docs: Delmia rig stand-in + notifier verification results (A-G incl. active-central kill/takeover 27 s, rejoin); register rows 107 (prod DelmiaRecv path) + 108 (per-node inbound API key store breaks central failover); CLAUDE.md bullet; plan + task file

Claude-Session: https://claude.ai/code/session_013L3XdmVii9i5sY4FChU8pB

docs/plans/2026-07-08-deferred-work-register.md

@@ -99,6 +99,8 @@ All 7 fix-now items landed via PLAN-04/05/06/07/08 (verified in review 08 round
 | 104 | ~~**Existing site LocalDb files stay `auto_vacuum=NONE`; `PRAGMA incremental_vacuum` after a cap trim is a no-op on them**~~ **RESOLVED 2026-08-20 — recipe shipped and applied to the rig.** New `docker/vacuum-site-localdb.sh` (stop both nodes of a pair together → `PRAGMA auto_vacuum=INCREMENTAL; VACUUM` each volume with integrity checks before/after → start both; `DRY_RUN=1` reports) plus the topology-guide section *Site LocalDb files created before 2026-08-20 need a one-off offline VACUUM*. Run on all three rig pairs: all six files `auto_vacuum 0 → 2`, `integrity_check=ok`, ~11 % smaller (site-a 20.3 → 18.1 MB), site-a replication back `Connected=true`/backlog 0 with no watermark or schema change — the VACUUM was replication-invisible as predicted (ledger keys by declared PK; oplog `seq` is an `INTEGER PRIMARY KEY AUTOINCREMENT`). The same maintenance surfaced the unrelated row 106. Original: | Arch-review 2026-08-19 P4-6 (Low-wave T4, `47f03fe6`) | `auto_vacuum` is only honoured on a table-less database, and `LocalDbSchema.EnsureCreated` had always created LocalDb's bookkeeping tables before `SiteEventLogSchema.Apply` ran — so the pragma had never once taken effect on any real file. The fix (`SiteLocalDbSetup.EnsureAutoVacuumOnFreshFile`, run before `AddZbLocalDb`'s lazy singleton constructs `SqliteLocalDb`) only helps **fresh** files; an unconditional `VACUUM` against a live ~1 GB replicated file was explicitly rejected (long exclusive lock on the active node; the rewrite is CDC-invisible but blocks the sync pump). Every node on the docker rig today predates the fix, so freed `site_events` pages there are reused, not returned to the OS — the storage cap (row 81) still bounds the data volume, only the file size stops shrinking. | Operator-scheduled `VACUUM` in a maintenance window (both nodes of a pair stopped together), or when a site's file is next recreated. |
 | 105 | ~~**Design constraint (not a defect): a LocalDb-replicated table must never be the parent of a row-mutating `ON DELETE` foreign key (`CASCADE`/`SET NULL`/`SET DEFAULT`) from another table**~~ **PINNED 2026-08-20.** Enforcement moved from the first replicated boot to the build: `SiteLocalDbReplicatedTableFkGuardTests` (Host.Tests) builds the real site schema via `OnReady` on an unreplicated config (so 0.3.3's own guard can't mask which FK is at fault), walks `SiteLocalDbSetup.ReplicatedTables` (now `internal`), asserts each exists and has no inbound row-mutating `ON DELETE` FK with the same `sqlite_master JOIN pragma_foreign_key_list` query LocalDb uses, and carries a negative control proving the checker finds a cascading child. The constraint itself stands — the row is kept as its record. Original: | Arch-review 2026-08-19 P4-7 → LocalDb 0.3.3 (scadaproj `122940c` + review follow-up `9663279`, Low-wave T5) | `LwwApplier` upserts with `INSERT OR REPLACE` under `foreign_keys=ON`, so the implicit delete fires the child action while CDC capture is suppressed — the child rows change on the applying node only and the pair diverges silently. 0.3.3 makes `RegisterReplicated` refuse such a parent at registration (clear exception naming the child FK), which is a BOOT failure for the site node, so this is enforced, not advisory. None of the ten `SiteLocalDbSetup.ReplicatedTables` has any FK today (zero `REFERENCES` on the live rig file). Model cross-table cleanup as explicit application deletes on each table (as the 3-table deploy cascade already does), never as SQLite FK actions. | Whenever a schema change adds a foreign key between site tables, or an eleventh table joins `ReplicatedTables`. |
 | 106 | ~~**LocalDb: a converged pair re-streams a FULL snapshot on EVERY reconnect once the sender's oplog has been ack-pruned to empty and the receiver's watermark is 0**~~ **RESOLVED 2026-08-20 — LocalDb 0.3.4 (scadaproj `75c1897`), published and pinned.** `SnapshotStreamer.SendAsync` now stamps `as_of_seq = max(MAX(seq), last_acked_seq)`; pinned by `WipedPeer_SnapshotLiftsItsWatermark_SoTheNextReconnectSendsNoSecondSnapshot` (fails on 0.3.3 exactly as the rig did: watermark 0, expected 3). Live proof on the rig: the first restart on 0.3.4 sent one last full snapshot stamped `as_of_seq 149` (was 0) and node-a's `last_applied_remote_seq` went 0 → 149; a second restart of the pair exchanged **zero** snapshots, `Connected=true`, backlog 0. Original: | Found 2026-08-20 while verifying row 104 on the rig (site-a-a received the same 127 k-row full snapshot at both the 13:09 deploy restart and the 13:25 VACUUM restart) | Sender-side defect in `SnapshotStreamer.SendAsync`: `as_of_seq = MAX(seq)` of the oplog, which is **0** once the pair has converged and ack-pruning emptied it. The receiver's `SetLastAppliedRemoteSeqAsync` is MAX-guarded (no regression — good), but a peer whose watermark is genuinely 0 (site-a-a was rebuilt after the 2026-08-15 corruption; any wiped/re-imaged node is the same) can never be lifted by an `as_of_seq 0` snapshot, so the 0.1.2 empty-oplog horizon rule (`streamableFrom = last_acked_seq + 1`) sees `0 < 149` as a gap on every handshake and re-streams everything. Rig state: node-b `last_acked_seq=149`, oplog empty; node-a `last_applied_remote_seq=0`. Not a divergence (LWW absorbs the re-delivered rows) but a cost/availability bug — ~1 s and 127 k rows here, minutes and a full-file read on a 1 GB production pair, on every reconnect, until the sender happens to write a new row. Fix is sender-side and wire-compatible: stamp `as_of_seq = max(MAX(seq), last_acked_seq)` (a full snapshot taken now covers every delta ever pruned below `last_acked_seq`), so the receiver's MAX-guarded watermark lifts to 149 and the loop ends; add a pinning test (converged+pruned sender, zero-watermark receiver → exactly one snapshot across two handshakes). Ships as LocalDb 0.3.4. | Next LocalDb release (0.3.4) — small, well-understood, and worth doing before any production pair is rebuilt from its peer. |
+| 107 | **Prod `CvdReactor.ProcessRecipeDownload` still addresses `Children["DelmiaRecv"]` after the 2026-06-27 `DelmiaRecv` → `DelmiaReceiver` composition rename** — inferred from the deployed script (`docs/api-specs/Delmia-Recipe-Download-API.md` §7 item 1): `ChildrenAccessor` concatenates the name into a path, an unknown path yields null, so every call answers `Receiver ready flag not set to true` regardless of the real flag. No call has hit prod since the rename, so this is inferred, not observed. The docker rig's stand-in (`docker/seed-delmia-sim.sh`) uses `Children["DelmiaReceiver"]` and the same handshake answers correctly, which is the working reference. | Found 2026-09-02 while capturing the as-deployed spec; QA copy unread (credentials) | Separate change with its own deployment record: `scadabridge template script update` on the `CvdReactor` template + redeploy Z28061/Z28062/Z28061Sim, then a live `WWNotifier` call. | Before the first production Delmia recipe download. |
+| 108 | **Inbound API keys are a PER-NODE SQLite store — a central failover invalidates every inbound client's key** *(High, availability + security-adjacent)*. `ZB.MOM.WW.Auth.ApiKeys` keeps keys in `ScadaBridge:InboundApi:ApiKeyStore:SqlitePath` (`/app/data/inbound-api-keys.sqlite`, on each central node's OWN data volume), and key admin (`ManagementActor`, a plain per-node `ActorOf` reached in-process from that node's `/management` endpoints) writes only the node that serves the call — via Traefik that is always the active node. So a key exists on exactly one node: after an active-central crash the survivor answers **401 `KeyNotFound`** to every inbound call (found live 2026-09-02 by `docker/verify-delmia-notifier.sh`: the standby-relay case passes only because the standby forwards BEFORE authenticating). The same asymmetry means a key revoked on one node stays valid on the other. Rig workaround: `docker/sync-inbound-api-keys.sh` mirrors the file (both nodes share one pepper); the verify script's preflight refuses to run unless both nodes list the key. Prod `wonder-app-vd03` is single-node; the QA pair (`vq03`/`vq04`) has this gap today. | `docs/api-specs/Delmia-Recipe-Download-API.md` §8; `docker/sync-inbound-api-keys.sh` header | Fix is an architecture decision, not a patch: move the store to the shared SQL Server (as Secrets 0.5.0 did for its central store) or replicate the file; either way revocation must become cluster-wide. Needs the `ZB.MOM.WW.Auth.ApiKeys` library (scadaproj), not just this repo. | Before any 2-node central goes live with inbound API clients; before the next arch-review wave. |
 | 68 | ~~**`deploy.sh` run from a git worktree silently rehomes every node's bind-mount data dir**~~ **RESOLVED 2026-08-15 (guard added in the same close-out).** The compose files' bind mounts are relative (`./site-a-node-a/data`), so a deploy from a worktree checkout mounts the worktree's copies — the rig keeps running and looks healthy, but its SQLite stores split from the main checkout's and everything written there is orphaned by the next deploy from main. Bit the rows-46–53 close-out live: the seeding agent's `rig-probe-1` deploy persisted into worktree-homed dirs and "vanished" after the orchestrator's redeploy (no tombstone, no product bug — LWW/tombstone forensics ruled out deletion; the row sat in `.claude/worktrees/…/docker/site-a-node-a/data/site-localdb.db` the whole time). Both `docker/deploy.sh` and `docker-env2/deploy.sh` now refuse to run from a `.claude/worktrees/` path (override: `SCADABRIDGE_ALLOW_WORKTREE_DEPLOY=1`). | Found during the rows-46–53 close-out | Closed 2026-08-15. | — |
 
 ## Resolved (verified against the code 2026-07-10)