docs: relabel wonder-app-vd03 as DEV, not production, across plans, records, reviews and API specs

vd03 is the DEV box (vd = DEV, vq03/vq04 = QA). Historical docs called it prod/production. Only vd03-referential uses are changed; generic 'production' wording (production HOCON, production code paths, ASPNETCORE_ENVIRONMENT=Production, real-world production guidance) is unchanged. One verbatim git commit message quoted in PLAN-07 is left as recorded.
2026-11-10 15:11:45 -05:00
parent ee320cd544
commit 48517b49fb
35 changed files with 110 additions and 110 deletions
@@ -85,7 +85,7 @@ The existing `~/Desktop/ScadaBridge` (non-git, older project containing `akka.md

 ## Section 3a — Git-ignored `deploy/` tree (added during implementation)

-`/deploy/` is git-ignored, so the `git grep`-based scrub script (and its completeness gate) structurally could not reach it — a gap surfaced by code review. The `deploy/wonder-app-vd03/` production artifacts (`install.ps1`, `appsettings.Central.json`, `appsettings.Site.json`, `RUNBOOK.md`, `ldap/glauth.cfg`) still held old references that would break against the renamed binary: `SCADALINK_*` env vars, `ScadaLink__*` config keys, the `"ScadaLink"` config-section root, `ScadaLink.Host.exe` (now `ZB.MOM.WW.ScadaBridge.Host.exe`), `scadalink.exe` (CLI `AssemblyName` is `scadabridge`), and `akka.tcp://scadalink@…` seed URIs (the actor system name in code is `scadabridge`).
+`/deploy/` is git-ignored, so the `git grep`-based scrub script (and its completeness gate) structurally could not reach it — a gap surfaced by code review. The `deploy/wonder-app-vd03/` DEV artifacts (`install.ps1`, `appsettings.Central.json`, `appsettings.Site.json`, `RUNBOOK.md`, `ldap/glauth.cfg`) still held old references that would break against the renamed binary: `SCADALINK_*` env vars, `ScadaLink__*` config keys, the `"ScadaLink"` config-section root, `ScadaLink.Host.exe` (now `ZB.MOM.WW.ScadaBridge.Host.exe`), `scadalink.exe` (CLI `AssemblyName` is `scadabridge`), and `akka.tcp://scadalink@…` seed URIs (the actor system name in code is `scadabridge`).

 Per user decision, this tree was scrubbed by hand with the **same** substitution set, **including** the internal LDAP directory domain (`dc=scadalink,dc=local` → `dc=scadabridge,dc=local` and `@scadalink.local` → `@scadabridge.local`) applied in lockstep across both `appsettings` and `glauth.cfg` so LDAP login stays consistent. Because `deploy/` is git-ignored, these edits are **local-only / not committed** — they fix the on-disk artifacts for the next `wonder-app-vd03` deploy. Verified clean with `grep -rniI scadalink deploy/`.

@@ -5,7 +5,7 @@
 **Scope:** ScadaBridge — inbound API `POST /api/IpsenMESMoveIn`, a small inbound-script DB
 capability, and a new template script on the reactor template that writes a MoveIn to the
 correct MES-receiver child.
-**Target environment:** production `wonder-app-vd03` (Central + Site, ConfigDb `ScadaBridge`
+**Target environment:** DEV `wonder-app-vd03` (Central + Site, ConfigDb `ScadaBridge`
 on `wonder-app-vd03,1433`; reactors live on Site 2). BTDB = db-connection id 1 (`dbo.Machine`
 on `wonder-sql-vd03/BT`).

@@ -96,7 +96,7 @@ receiver is ready to accept a MoveIn.
 > `QuerySingle`/`Query` sketched in the original draft above, and **writes are supported**
 > (InboundAPI-026) rather than read-only. Scripts MUST call `await Database.QuerySingleAsync<T>(...)`.
 > Three deployed methods (`IpsenMESMoveIn`, `MesMoveIn`, `MesMoveOut`) were authored from the old
-> `Database.QuerySingle` spelling and failed Roslyn compilation in production until corrected
+> `Database.QuerySingle` spelling and failed Roslyn compilation on DEV (vd03) until corrected
 > (2026-06-25). The authoritative surface lives in `docs/requirements/Component-InboundAPI.md`.

 ### B. Rewrite inbound `/api/IpsenMESMoveIn` script (data, via management API)
@@ -260,7 +260,7 @@ Add `using Microsoft.Extensions.DependencyInjection;` at the top. No `ScriptOpti

 ## Task 5: Deploy ScadaBridge Central (Component A) to wonder-app-vd03

-**Classification:** high-risk (production deploy)
+**Classification:** high-risk (live DEV-host deploy)
 **Estimated implement time:** ~6 min
 **Parallelizable with:** none (depends on Tasks 3–4)
 **Requires explicit go-ahead before running.**
@@ -278,7 +278,7 @@ Add `using Microsoft.Extensions.DependencyInjection;` at the top. No `ScriptOpti

 ## Task 6: Apply Component B — update the inbound `IpsenMESMoveIn` script

-**Classification:** high-risk (production data)
+**Classification:** high-risk (live DEV-host data)
 **Estimated implement time:** ~3 min
 **Parallelizable with:** Task 7 (different surfaces; both depend on Task 5)
 **Requires explicit go-ahead.**
@@ -335,7 +335,7 @@ Preserve the current `ParameterDefinitions`/`ReturnDefinition` (read them first;

 ## Task 7: Apply Component C — `IpsenMoveIn` template script on T1

-**Classification:** high-risk (production data)
+**Classification:** high-risk (live DEV-host data)
 **Estimated implement time:** ~4 min
 **Parallelizable with:** Task 6 (depends on Task 5)
 **Requires explicit go-ahead.**
@@ -389,7 +389,7 @@ return new { WasSuccessful = true, ErrorText = "", BatchID = batchId };

 ## Task 8: On-box end-to-end verification

-**Classification:** standard (production verify)
+**Classification:** standard (live DEV-host verify)
 **Estimated implement time:** ~5 min
 **Parallelizable with:** none (depends on Tasks 6, 7)

@@ -16,7 +16,7 @@ updated to mirror the new methods. This was hit live: deploying a template scrip

 > `'ScriptCompileSurface.CompileAttributeAccessor' does not contain a definition for 'WaitAsync'`

-…which forced a bounded-poll-loop workaround in the DELMIA/MES receiver scripts on prod. Once this plan
+…which forced a bounded-poll-loop workaround in the DELMIA/MES receiver scripts on DEV (vd03). Once this plan
 ships, those scripts can be switched back to the cheaper event-driven `WaitAsync`.

 **Tech Stack:** C#/.NET 10, Roslyn (`Microsoft.CodeAnalysis.CSharp.Scripting`), xUnit. No new dependencies.
@@ -106,8 +106,8 @@ All 7 fix-now items landed via PLAN-04/05/06/07/08 (verified in review 08 round
 | 104 | ~~**Existing site LocalDb files stay `auto_vacuum=NONE`; `PRAGMA incremental_vacuum` after a cap trim is a no-op on them**~~ **RESOLVED 2026-08-20 — recipe shipped and applied to the rig.** New `docker/vacuum-site-localdb.sh` (stop both nodes of a pair together → `PRAGMA auto_vacuum=INCREMENTAL; VACUUM` each volume with integrity checks before/after → start both; `DRY_RUN=1` reports) plus the topology-guide section *Site LocalDb files created before 2026-08-20 need a one-off offline VACUUM*. Run on all three rig pairs: all six files `auto_vacuum 0 → 2`, `integrity_check=ok`, ~11 % smaller (site-a 20.3 → 18.1 MB), site-a replication back `Connected=true`/backlog 0 with no watermark or schema change — the VACUUM was replication-invisible as predicted (ledger keys by declared PK; oplog `seq` is an `INTEGER PRIMARY KEY AUTOINCREMENT`). The same maintenance surfaced the unrelated row 106. Original: | Arch-review 2026-08-19 P4-6 (Low-wave T4, `47f03fe6`) | `auto_vacuum` is only honoured on a table-less database, and `LocalDbSchema.EnsureCreated` had always created LocalDb's bookkeeping tables before `SiteEventLogSchema.Apply` ran — so the pragma had never once taken effect on any real file. The fix (`SiteLocalDbSetup.EnsureAutoVacuumOnFreshFile`, run before `AddZbLocalDb`'s lazy singleton constructs `SqliteLocalDb`) only helps **fresh** files; an unconditional `VACUUM` against a live ~1 GB replicated file was explicitly rejected (long exclusive lock on the active node; the rewrite is CDC-invisible but blocks the sync pump). Every node on the docker rig today predates the fix, so freed `site_events` pages there are reused, not returned to the OS — the storage cap (row 81) still bounds the data volume, only the file size stops shrinking. | Operator-scheduled `VACUUM` in a maintenance window (both nodes of a pair stopped together), or when a site's file is next recreated. |
 | 105 | ~~**Design constraint (not a defect): a LocalDb-replicated table must never be the parent of a row-mutating `ON DELETE` foreign key (`CASCADE`/`SET NULL`/`SET DEFAULT`) from another table**~~ **PINNED 2026-08-20.** Enforcement moved from the first replicated boot to the build: `SiteLocalDbReplicatedTableFkGuardTests` (Host.Tests) builds the real site schema via `OnReady` on an unreplicated config (so 0.3.3's own guard can't mask which FK is at fault), walks `SiteLocalDbSetup.ReplicatedTables` (now `internal`), asserts each exists and has no inbound row-mutating `ON DELETE` FK with the same `sqlite_master JOIN pragma_foreign_key_list` query LocalDb uses, and carries a negative control proving the checker finds a cascading child. The constraint itself stands — the row is kept as its record. Original: | Arch-review 2026-08-19 P4-7 → LocalDb 0.3.3 (scadaproj `122940c` + review follow-up `9663279`, Low-wave T5) | `LwwApplier` upserts with `INSERT OR REPLACE` under `foreign_keys=ON`, so the implicit delete fires the child action while CDC capture is suppressed — the child rows change on the applying node only and the pair diverges silently. 0.3.3 makes `RegisterReplicated` refuse such a parent at registration (clear exception naming the child FK), which is a BOOT failure for the site node, so this is enforced, not advisory. None of the ten `SiteLocalDbSetup.ReplicatedTables` has any FK today (zero `REFERENCES` on the live rig file). Model cross-table cleanup as explicit application deletes on each table (as the 3-table deploy cascade already does), never as SQLite FK actions. | Whenever a schema change adds a foreign key between site tables, or an eleventh table joins `ReplicatedTables`. |
 | 106 | ~~**LocalDb: a converged pair re-streams a FULL snapshot on EVERY reconnect once the sender's oplog has been ack-pruned to empty and the receiver's watermark is 0**~~ **RESOLVED 2026-08-20 — LocalDb 0.3.4 (scadaproj `75c1897`), published and pinned.** `SnapshotStreamer.SendAsync` now stamps `as_of_seq = max(MAX(seq), last_acked_seq)`; pinned by `WipedPeer_SnapshotLiftsItsWatermark_SoTheNextReconnectSendsNoSecondSnapshot` (fails on 0.3.3 exactly as the rig did: watermark 0, expected 3). Live proof on the rig: the first restart on 0.3.4 sent one last full snapshot stamped `as_of_seq 149` (was 0) and node-a's `last_applied_remote_seq` went 0 → 149; a second restart of the pair exchanged **zero** snapshots, `Connected=true`, backlog 0. Original: | Found 2026-08-20 while verifying row 104 on the rig (site-a-a received the same 127 k-row full snapshot at both the 13:09 deploy restart and the 13:25 VACUUM restart) | Sender-side defect in `SnapshotStreamer.SendAsync`: `as_of_seq = MAX(seq)` of the oplog, which is **0** once the pair has converged and ack-pruning emptied it. The receiver's `SetLastAppliedRemoteSeqAsync` is MAX-guarded (no regression — good), but a peer whose watermark is genuinely 0 (site-a-a was rebuilt after the 2026-08-15 corruption; any wiped/re-imaged node is the same) can never be lifted by an `as_of_seq 0` snapshot, so the 0.1.2 empty-oplog horizon rule (`streamableFrom = last_acked_seq + 1`) sees `0 < 149` as a gap on every handshake and re-streams everything. Rig state: node-b `last_acked_seq=149`, oplog empty; node-a `last_applied_remote_seq=0`. Not a divergence (LWW absorbs the re-delivered rows) but a cost/availability bug — ~1 s and 127 k rows here, minutes and a full-file read on a 1 GB production pair, on every reconnect, until the sender happens to write a new row. Fix is sender-side and wire-compatible: stamp `as_of_seq = max(MAX(seq), last_acked_seq)` (a full snapshot taken now covers every delta ever pruned below `last_acked_seq`), so the receiver's MAX-guarded watermark lifts to 149 and the loop ends; add a pinning test (converged+pruned sender, zero-watermark receiver → exactly one snapshot across two handshakes). Ships as LocalDb 0.3.4. | Next LocalDb release (0.3.4) — small, well-understood, and worth doing before any production pair is rebuilt from its peer. |
-| 107 | **→ #52** **Prod `CvdReactor.ProcessRecipeDownload` still addresses `Children["DelmiaRecv"]` after the 2026-06-27 `DelmiaRecv` → `DelmiaReceiver` composition rename** — inferred from the deployed script (`docs/api-specs/Delmia-Recipe-Download-API.md` §7 item 1): `ChildrenAccessor` concatenates the name into a path, an unknown path yields null, so every call answers `Receiver ready flag not set to true` regardless of the real flag. No call has hit prod since the rename, so this is inferred, not observed. The docker rig's stand-in (`docker/seed-delmia-sim.sh`) uses `Children["DelmiaReceiver"]` and the same handshake answers correctly, which is the working reference. | Found 2026-09-02 while capturing the as-deployed spec; QA copy unread (credentials) | Separate change with its own deployment record: `scadabridge template script update` on the `CvdReactor` template + redeploy Z28061/Z28062/Z28061Sim, then a live `WWNotifier` call. | Before the first production Delmia recipe download. |
-| 108 | ~~**Inbound API keys are a PER-NODE SQLite store — a central failover invalidates every inbound client's key**~~ **RESOLVED 2026-09-02 — `ZB.MOM.WW.Auth` 0.3.0 (new `ZB.MOM.WW.Auth.ApiKeys.SqlServer` package) + ScadaBridge `ScadaBridge:InboundApi:ApiKeyStore:Provider=SqlServer`.** A central pair now shares one row set: both nodes' key stores, admin stores, and audit stores point at the same `zbauth` schema in the configuration database (connection string defaults to `ScadaBridge:Database:ConfigurationDb` — never a `${secret:}` reference, to avoid a circular dependency on the SQL-backed secrets store), so create/revoke/rotate/scope changes are cluster-wide immediately instead of per-node. Each node's existing SQLite file is imported once at boot (insert-where-absent, so both nodes importing is safe and the result is the union) and renamed `*.imported`. `Provider` defaults to `Sqlite` (unchanged behavior for single-node installs, MxAccessGateway, HistorianGateway); a clustered Central left on `Sqlite` now gets a boot-time `StartupValidator` Warning naming this design instead of silently keeping the gap. Both docker rigs (`docker/`, `docker-env2/`) run `Provider=SqlServer` on both centrals; `docker/sync-inbound-api-keys.sh` is deleted, superseded by the shared store. Rig proof 2026-09-03 (`docker/`, both centrals redeployed on `Provider=SqlServer`): both nodes imported their SQLite file at the same second with no PK race — central-a `1 key / 264 audit rows inserted`, central-b `0 / 26 inserted, 1 / 264 already present` (the union), files renamed `*.imported`; a key minted via `:9001` answered `200` via `:9002`, `--enabled false` via `:9001` made both nodes answer `401` and `:9002`'s own list showed `enabled=False`, re-enable via `:9002` restored `200` on `:9001`, delete via `:9002` made `:9001` answer `401`; `verify-delmia-notifier.sh` with a freshly minted key and NO file copy: A/B/E/F `YES`, C/G `NO`, active-central `docker kill` takeover **28 s**, rejoin as standby in 2 s. `docker-env2` (second pair, `ScadaBridgeConfig2`): both centrals booted, `zbauth` v1 created, empty files imported. QA (`vq03`/`vq04`) and prod (`vd03`) still run `Provider=Sqlite` — their move is a planned deployment record (`docs/deployment/deployment-records/2026-09-XX-zb-servers-inbound-key-store.md`), not yet applied. Design: `docs/plans/2026-09-02-inbound-api-key-shared-store-design.md`. Original: | `docs/api-specs/Delmia-Recipe-Download-API.md` §8; `docker/sync-inbound-api-keys.sh` header | Fix is an architecture decision, not a patch: move the store to the shared SQL Server (as Secrets 0.5.0 did for its central store) or replicate the file; either way revocation must become cluster-wide. Needs the `ZB.MOM.WW.Auth.ApiKeys` library (scadaproj), not just this repo. | Before any 2-node central goes live with inbound API clients; before the next arch-review wave. |
+| 107 | **→ #52** **DEV (vd03) `CvdReactor.ProcessRecipeDownload` still addresses `Children["DelmiaRecv"]` after the 2026-06-27 `DelmiaRecv` → `DelmiaReceiver` composition rename** — inferred from the deployed script (`docs/api-specs/Delmia-Recipe-Download-API.md` §7 item 1): `ChildrenAccessor` concatenates the name into a path, an unknown path yields null, so every call answers `Receiver ready flag not set to true` regardless of the real flag. No call has hit DEV (vd03) since the rename, so this is inferred, not observed. The docker rig's stand-in (`docker/seed-delmia-sim.sh`) uses `Children["DelmiaReceiver"]` and the same handshake answers correctly, which is the working reference. | Found 2026-09-02 while capturing the as-deployed spec; QA copy unread (credentials) | Separate change with its own deployment record: `scadabridge template script update` on the `CvdReactor` template + redeploy Z28061/Z28062/Z28061Sim, then a live `WWNotifier` call. | Before the first production Delmia recipe download. |
+| 108 | ~~**Inbound API keys are a PER-NODE SQLite store — a central failover invalidates every inbound client's key**~~ **RESOLVED 2026-09-02 — `ZB.MOM.WW.Auth` 0.3.0 (new `ZB.MOM.WW.Auth.ApiKeys.SqlServer` package) + ScadaBridge `ScadaBridge:InboundApi:ApiKeyStore:Provider=SqlServer`.** A central pair now shares one row set: both nodes' key stores, admin stores, and audit stores point at the same `zbauth` schema in the configuration database (connection string defaults to `ScadaBridge:Database:ConfigurationDb` — never a `${secret:}` reference, to avoid a circular dependency on the SQL-backed secrets store), so create/revoke/rotate/scope changes are cluster-wide immediately instead of per-node. Each node's existing SQLite file is imported once at boot (insert-where-absent, so both nodes importing is safe and the result is the union) and renamed `*.imported`. `Provider` defaults to `Sqlite` (unchanged behavior for single-node installs, MxAccessGateway, HistorianGateway); a clustered Central left on `Sqlite` now gets a boot-time `StartupValidator` Warning naming this design instead of silently keeping the gap. Both docker rigs (`docker/`, `docker-env2/`) run `Provider=SqlServer` on both centrals; `docker/sync-inbound-api-keys.sh` is deleted, superseded by the shared store. Rig proof 2026-09-03 (`docker/`, both centrals redeployed on `Provider=SqlServer`): both nodes imported their SQLite file at the same second with no PK race — central-a `1 key / 264 audit rows inserted`, central-b `0 / 26 inserted, 1 / 264 already present` (the union), files renamed `*.imported`; a key minted via `:9001` answered `200` via `:9002`, `--enabled false` via `:9001` made both nodes answer `401` and `:9002`'s own list showed `enabled=False`, re-enable via `:9002` restored `200` on `:9001`, delete via `:9002` made `:9001` answer `401`; `verify-delmia-notifier.sh` with a freshly minted key and NO file copy: A/B/E/F `YES`, C/G `NO`, active-central `docker kill` takeover **28 s**, rejoin as standby in 2 s. `docker-env2` (second pair, `ScadaBridgeConfig2`): both centrals booted, `zbauth` v1 created, empty files imported. QA (`vq03`/`vq04`) and DEV (`vd03`) still run `Provider=Sqlite` — their move is a planned deployment record (`docs/deployment/deployment-records/2026-09-XX-zb-servers-inbound-key-store.md`), not yet applied. Design: `docs/plans/2026-09-02-inbound-api-key-shared-store-design.md`. Original: | `docs/api-specs/Delmia-Recipe-Download-API.md` §8; `docker/sync-inbound-api-keys.sh` header | Fix is an architecture decision, not a patch: move the store to the shared SQL Server (as Secrets 0.5.0 did for its central store) or replicate the file; either way revocation must become cluster-wide. Needs the `ZB.MOM.WW.Auth.ApiKeys` library (scadaproj), not just this repo. | Before any 2-node central goes live with inbound API clients; before the next arch-review wave. |
 | 109 | **→ scadaproj#18** **`ZB.MOM.WW.Secrets.Replicator.SqlServer`'s store migrator retries its DDL with a flat `100 * attempt` backoff and no jitter — a latent concurrent-boot flake, open in Secrets** *(Low)*. Porting that migrator's retry loop verbatim for the Auth 0.3.0 SQL Server API-key store (register row 108) reproduced an intermittent exhaustion — about once in sixty 8-way concurrent-boot trials the five flat-backoff retries ran out before a DDL/lock conflict cleared; adding jitter to the Auth port's backoff took it to 0 failures in 180 trials. Secrets' own migrator has no concurrent-migrate test, so the same defect is presumed still open there — it just hasn't been exercised at 8-way concurrency, and a ScadaBridge central pair is only ever 2-way, so this is unlikely to bite in practice. | Found 2026-09-02 while implementing `docs/plans/2026-09-02-inbound-api-key-shared-store.md` (Auth 0.3.0 `SqlServerApiKeyStoreMigrator`) | Add jitter to `SqlServerSecretsStoreMigrator`'s backoff in `~/Desktop/scadaproj/ZB.MOM.WW.Secrets` (mirror the Auth fix) and add a `Migrate_Concurrently_FromEightTasks_AllSucceed`-shaped live test to catch a regression. Lives entirely in scadaproj, not this repo. | Next time Secrets' SQL Server migrator is touched, or if a wider-than-2-way concurrent boot of a Secrets-backed cluster is ever planned. |
 | 110 | **→ #53** **Flow Debugger follow-up: the *Vertical trace* list toggle (mockup Direction B)** *(Low)*. v1 ships the Direction A swimlane only; the mockup's Direction B — the same steps as a vertical, filterable list with the lane as a column — was kept as an alternate for long flows and screen readers. Additive: a list view over the existing `FlowGraph`. ~~and console capture on `ScriptRun`~~ **The console-capture half is RESOLVED 2026-09-09 (plan `docs/plans/2026-09-09-flow-debugger-step-logs.md`)** — and deliberately not as this row sketched it. `Extra.console` on the `ScriptRun` row was rejected (the 8 KB payload cap would silently drop lines, and helper-step attribution would need every helper to buffer); shipped instead is a leveled `Log` global on all four script surfaces plus one `ScriptTrace.ScriptLog` audit row per line, folded onto the step that produced it, with the failing step's full exception on `errorDetail`. `Console.Out` capture on a site node is recorded as permanently out of scope in the design's §9: `Console.Out` is process-global and shared by every concurrent run, so per-run capture is unsound. | Design `docs/plans/2026-09-05-flow-debugger-design.md` §11; plan `2026-09-05-flow-debugger.md` Task 23 | The list view was not asked for by an operator; the swimlane + Timing/Writes/Log tabs cover the acceptance flow (Delmia recipe download, rig steps H and J). | First operator request for a list view. |
 | 111 | ~~**Flow Debugger outbound nodes: no Retry/Discard on a parked node, and no `?correlationId=` deep link from the node to the Site Calls / Notification report pages** *(Low)*~~ **RESOLVED 2026-09-06 (plan `docs/plans/2026-09-06-flow-debugger-followups.md`).** `FlowStep` now carries the row's `CorrelationId`; the detail panel deep-links to `/site-calls/report?correlationId=` / `/notifications/report?correlationId=` (both pages accept it through additive per-id query filters — `SiteCallQueryRequest.TrackedOperationIdFilter`, `NotificationOutboxQueryRequest.NotificationIdFilter` — pre-filter to the row and auto-open its detail), and a `Parked` node offers Retry / Discard in place through `IFlowDebugService.RetryOutboundAsync` / `DiscardOutboundAsync`, which reuse the report pages' relay, site-scope re-check and confirm wording, after which the page re-polls Replay until the node leaves Parked. Per-attempt request/response diff is still not shown (the lifecycle rows carry the error summary only). Found on the way: the prediction circuit deadlock (`4769e192`). | Plan 2026-09-06 "Out of scope"; `FlowStepDetail.razor` parked hint | The Site Calls page owns the retry semantics and its audit rows; duplicating the action on the debugger node would need the same Deployer gate + confirm flow. | First operator who reaches a parked node from the debugger and asks for the button. |
@@ -119,8 +119,8 @@ All 7 fix-now items landed via PLAN-04/05/06/07/08 (verified in review 08 round
 | 117 | ~~**Flow Debugger site trace pull: `FlowDebugRunner.WriteAndPollAsync` fetches ONE bounded page with a fixed `since`**~~ **RESOLVED 2026-09-09 (`9ee402bb`, arch-review 2026-09-09 MG-6; closed in this register 2026-10-21 — found stale by the 2026-10-21 system review).** `WriteAndPollAsync` now advances a `(SinceUtc, AfterEventId)` keyset cursor between polls and only treats a page as progress when the cursor moved, so a chatty run can no longer push a spawned run past the page end. Original description kept for traceability: *(Low)*. The write-and-poll loop asks the site for `FlowDebugRunner.PullTraceEventsMax` rows per poll, always `since` the run start, and the site serves `ORDER BY OccurredAtUtc ASC LIMIT` — so the page is the OLDEST rows in the window, never a continuation of the previous one. Step logs made that reachable: one traced run can now emit up to 201 `ScriptLog` rows, so a handful of chatty runs inside the window can fill the page and push a later spawned run's `ScriptRun` row past the end of it, at which point the loop never sees the awaited run and reports a false `NO_TRIGGER_WITHIN_WINDOW` for a run that did happen. Final review I1 raised the cap 500 -> 2 000 (the site's own ceiling, `SiteStreamGrpcServer.MaxTraceEventsPerPull`), which makes it ~10x harder but does not remove it. The real fix is to PAGE — advance `since` to the last-seen `OccurredAtUtc` between polls (keeping the already-known ids so nothing is double-counted) — or to exclude `ScriptLog` from the awaited-run poll entirely and pull the log rows once, after the run has settled. | Final review of `docs/plans/2026-09-09-flow-debugger-step-logs.md` (I1); `src/ZB.MOM.WW.ScadaBridge.ManagementService/FlowDebug/FlowDebugRunner.cs` `WriteAndPollAsync` | 2 000 is the site's hard ceiling, so the headroom cannot be bought again with a bigger number; paging needs the poll loop to track a cursor and the awaited-run predicate to survive a partial view, which is more than the raise was worth at review time. | The first `NO_TRIGGER_WITHIN_WINDOW` reported for a run whose Replay of the same root DOES show the spawned `ScriptRun`. |
 | 118 | **Pre-existing bUnit flake: `CentralUI.Tests.Pages.NotificationReportDetailModalTests.CloseButton_DismissesModal` (~10 %)** *(Low)*. Fails intermittently with `UnknownEventHandlerIdException` — the test resolves the close button once and clicks it after a re-render has already retired that element's handler id. Not caused by the step-logs work: the file is untouched since `40944a0d`. The fix is a one-liner — re-`Find` the button immediately before the click (the pattern the other modal tests already use) rather than reusing an element captured earlier in the test. | Observed during the step-logs full-suite runs, 2026-09-09 | It is a test-only race with a known cause and a known fix; it was out of scope for a remediation pass whose rule was to touch only the files under review. | Next time `CentralUI.Tests` is worked on, or the first time the flake blocks a gate run. |
 | 68 | ~~**`deploy.sh` run from a git worktree silently rehomes every node's bind-mount data dir**~~ **RESOLVED 2026-08-15 (guard added in the same close-out).** The compose files' bind mounts are relative (`./site-a-node-a/data`), so a deploy from a worktree checkout mounts the worktree's copies — the rig keeps running and looks healthy, but its SQLite stores split from the main checkout's and everything written there is orphaned by the next deploy from main. Bit the rows-46–53 close-out live: the seeding agent's `rig-probe-1` deploy persisted into worktree-homed dirs and "vanished" after the orchestrator's redeploy (no tombstone, no product bug — LWW/tombstone forensics ruled out deletion; the row sat in `.claude/worktrees/…/docker/site-a-node-a/data/site-localdb.db` the whole time). Both `docker/deploy.sh` and `docker-env2/deploy.sh` now refuse to run from a `.claude/worktrees/` path (override: `SCADABRIDGE_ALLOW_WORKTREE_DEPLOY=1`). | Found during the rows-46–53 close-out | Closed 2026-08-15. | — |
-| 119 | **→ #36** **Port the QA/prod MES receiver scripts onto the reset-aware `WriteBatchAndWaitAsync` overload (Gitea #36)** *(Medium)*. Template scripts CvdReactor MesMoveIn (13) / MesMoveOut (14) / MesSampleData (36) and base MESReceiver MoveIn/MoveOut (15, 16, 25, 26, 37, 38) still reset `<X>CompleteFlag` with a plain write and rely on `Task.Delay(1000)`; QA carries a config-only `WaitAsync(false, 10 s)` patch on 13/14 (deployment record 110). Rewrite: `var hs = await rx.Attributes.WriteBatchAndWaitAsync(values, "<X>Flag", true, "<X>CompleteFlag", true, 25 s, "<X>CompleteFlag", false, 10 s); if (!hs.Matched) return new { WasSuccessful=false, ErrorText = hs.ResetFailed ? "Timeout waiting for <X>CompleteFlag to reset" : "Timeout waiting for <X> to complete", BatchID=0 };` and delete the delay. Requires the #36 build on the target central + sites first (the overload does not exist on older sites → deploy-gate compile failure, by design). | Gitea #36, design `2026-09-10-handshake-reset-echo-design.md` §8 | Repo-only scope was the user's decision for the #36 session; the script bodies live only in the QA/prod databases. | Next QA deployment of a build ≥ this change (write a deployment record; then prod). |
-| 120 | **→ #58** **Data-sourced SCALAR writes reach the OPC UA server as String Variants — every non-String node rejects them with BadTypeMismatch (0x80740000)** *(Medium)*. `InstanceActor.HandleSetDataAttribute` / `HandleWriteAttributeBatch` decode only List values before the DCL write ("Scalars are unchanged"), so the codec-encoded string (`"True"`, `"42.5"`) reaches `RealOpcUaClient.WriteValuesAsync`, which wraps it in `new Variant(value)` — a String variant sent to a Boolean/Double node. Found live 2026-09-10 running `docker/seed-handshake-demo.sh` against opc-plc `Motor.Running`/`Pump.Running`/`Motor.Speed` (`SetAttribute('Complete') failed: OPC UA write failed with status: 0x80740000`); the seed was retargeted onto the rig's only two writable String nodes. The MxGateway path is unaffected in practice: prod's Delmia/MES handshakes write the same strings through the gateway today. Fix candidates: (a) decode scalars by declared `DataType` in the InstanceActor before the DCL write — also changes what MxGateway receives, so it needs a gateway-side compatibility check; (b) adapter-local: `RealOpcUaClient` coerces to the node's DataType (it already resolves `DataTypeIds` in `EnrichVariableTypeInfoAsync`), leaving the MxGateway wire untouched — preferred. Pin with a `RealOpcUaClientArrayWriteTests`-style test for a Boolean scalar. | Found live 2026-09-10 during the #36 rig gate | Out of scope for #36 (handshake ordering); touches the production write path and needs the MxGateway compatibility question answered first. | Before any OPC UA-bound template relies on script writes to non-String nodes (none does today — until this gate the rig's script writes to OPC UA were static-only). |
+| 119 | **→ #36** **Port the QA/DEV MES receiver scripts onto the reset-aware `WriteBatchAndWaitAsync` overload (Gitea #36)** *(Medium)*. Template scripts CvdReactor MesMoveIn (13) / MesMoveOut (14) / MesSampleData (36) and base MESReceiver MoveIn/MoveOut (15, 16, 25, 26, 37, 38) still reset `<X>CompleteFlag` with a plain write and rely on `Task.Delay(1000)`; QA carries a config-only `WaitAsync(false, 10 s)` patch on 13/14 (deployment record 110). Rewrite: `var hs = await rx.Attributes.WriteBatchAndWaitAsync(values, "<X>Flag", true, "<X>CompleteFlag", true, 25 s, "<X>CompleteFlag", false, 10 s); if (!hs.Matched) return new { WasSuccessful=false, ErrorText = hs.ResetFailed ? "Timeout waiting for <X>CompleteFlag to reset" : "Timeout waiting for <X> to complete", BatchID=0 };` and delete the delay. Requires the #36 build on the target central + sites first (the overload does not exist on older sites → deploy-gate compile failure, by design). | Gitea #36, design `2026-09-10-handshake-reset-echo-design.md` §8 | Repo-only scope was the user's decision for the #36 session; the script bodies live only in the QA/DEV databases. | Next QA deployment of a build ≥ this change (write a deployment record; then DEV). |
+| 120 | **→ #58** **Data-sourced SCALAR writes reach the OPC UA server as String Variants — every non-String node rejects them with BadTypeMismatch (0x80740000)** *(Medium)*. `InstanceActor.HandleSetDataAttribute` / `HandleWriteAttributeBatch` decode only List values before the DCL write ("Scalars are unchanged"), so the codec-encoded string (`"True"`, `"42.5"`) reaches `RealOpcUaClient.WriteValuesAsync`, which wraps it in `new Variant(value)` — a String variant sent to a Boolean/Double node. Found live 2026-09-10 running `docker/seed-handshake-demo.sh` against opc-plc `Motor.Running`/`Pump.Running`/`Motor.Speed` (`SetAttribute('Complete') failed: OPC UA write failed with status: 0x80740000`); the seed was retargeted onto the rig's only two writable String nodes. The MxGateway path is unaffected in practice: DEV's (vd03) Delmia/MES handshakes write the same strings through the gateway today. Fix candidates: (a) decode scalars by declared `DataType` in the InstanceActor before the DCL write — also changes what MxGateway receives, so it needs a gateway-side compatibility check; (b) adapter-local: `RealOpcUaClient` coerces to the node's DataType (it already resolves `DataTypeIds` in `EnrichVariableTypeInfoAsync`), leaving the MxGateway wire untouched — preferred. Pin with a `RealOpcUaClientArrayWriteTests`-style test for a Boolean scalar. | Found live 2026-09-10 during the #36 rig gate | Out of scope for #36 (handshake ordering); touches the production write path and needs the MxGateway compatibility question answered first. | Before any OPC UA-bound template relies on script writes to non-String nodes (none does today — until this gate the rig's script writes to OPC UA were static-only). |
 | 121 | **→ #59** **No source-level `IsMesReportable` on native alarm source bindings** *(Low)*. `TemplateNativeAlarmSource` / `InstanceNativeAlarmSourceOverride` carry no per-alarm MES-relevance flag — "every condition under this binding is reportable" is not expressible, only the 900–999 severity-band convention. | `docs/plans/2026-09-11-mes-reportable-alarm-flag-design.md` | Native alarms have no per-alarm config row, and the design chose not to invent one for this slice. | A native-source machine needs per-alarm MES relevance beyond the severity band. |
 | 122 | **→ #60** **`IsMesReportable` does not ride the wire** *(Low)*. `AlarmStateUpdate` (vendored proto) and `DebugAlarmStateDto` have no `IsMesReportable` field, so central-side surfaces (Debug View live stream, Alarm Summary) always see `false` even though the site-side snapshot stamps the real value. Adding it is additive-only: `AlarmStateUpdate` field 25, `DebugAlarmStateDto` field 20, via a manual `regen-proto.sh` toggle-build-copy-untoggle. | `docs/plans/2026-09-11-mes-reportable-alarm-flag-design.md` | No consumer needed it in this slice — the flag exists to serve `Alarms.MesReportableAsync()` for scripts, not a central UI surface. | Operators ask to see the MES-reportable flag live on Debug View / Alarm Summary. |
 | 123 | **→ #61** **Rail nav scroll pane (`.rail-scroll` + the `#theme-rail > .side-rail { height: 100% }` fill rule) lives in app-local `site.css`/`NavMenu.razor`, not the `ZB.MOM.WW.Theme` kit** *(Low)*. Every other kit consumer (OtOpcUa, MxGateway, HistorianGateway, ignitionoee) has the same unbounded-nav-pushes-the-footer-off-screen shape and would need the identical fix copy-pasted. | `docs/plans/2026-09-14-pending-updates-design.md` §3.5 | Scoped to ScadaBridge for this change; upstreaming into the shared kit is a separate cross-repo release, not a one-app CSS tweak. | A second kit consumer hits the same long-nav clipping and copies the fix instead of consuming it from the kit. |
@@ -135,7 +135,7 @@ All 7 fix-now items landed via PLAN-04/05/06/07/08 (verified in review 08 round
 | 132 | **→ #68** **Site compile-cache eviction counter is not exported as a Meter** *(Low)*. `SiteScriptCompileCache` counts evictions and logs a rate-limited Warning (review 01/SR-M2) but the counter is not a `System.Diagnostics.Metrics` instrument, and `ZbTelemetryOptions.Meters` is an allowlist (`SiteServiceRegistration.ObservedMeters`), so exporting it also needs the Host allowlist entry. | 2026-10-21 system review (`docs/plans/2026-10-21-system-review-remediation.md`) (lane B) | The Warning log already surfaces thrash; a Meter is observability polish. | Wanting to alert on compile-cache thrash from Prometheus. |
 | 133 | **→ #69** **Dedicated health field for site audit rejected rows** *(Low)*. Permanent push rejections (`IngestAck.rejected_event_ids`) are counted only by the `scadabridge.site.audit.rejected_rows` counter and the `Rejected` `ForwardState`; there is no `SiteHealthReport` field/dashboard badge. | 2026-10-21 system review (`docs/plans/2026-10-21-system-review-remediation.md`) (lane H) | Rejections only come from row-intrinsic SQL errors and should be near-zero; the counter is sufficient for now. | The first non-zero `rejected_rows` seen in production, or an operator asking for it on the Health dashboard. |
 | 134 | **An oversize `AuditLog` row (larger than the gRPC message cap) wedges both push and pull** *(Low, accepted)*. A single row bigger than the 4 MB gRPC cap cannot be shipped by either the telemetry push or the reconciliation pull, so the queue head stalls. Unreachable at the default caps (8 KB / 64 KB payload caps, `InboundMaxBytes` 1 MiB) — it needs an operator to raise caps past the transport limit. | 2026-10-21 system review (`docs/plans/2026-10-21-system-review-remediation.md`) (lane H) | Defaults keep every row far below the limit; guarding means a skip/park state for unshippable rows. | Any deployment that raises `InboundMaxBytes` or the payload caps toward 4 MB. |
-| 135 | **→ #70** **Config-audit single-column indexes are possibly redundant** *(Low)*. `ReviewRemediationIndexes` added composite `AuditLogEntries` indexes `(EntityType, Timestamp DESC)`, `(User, Timestamp DESC)` and `(Timestamp DESC, Id DESC)`; the older single-column indexes on timestamp/user/entity type may now be dead weight on a write-heavy table. | 2026-10-21 system review (`docs/plans/2026-10-21-system-review-remediation.md`) (lane P / O) | Dropping needs a measured before/after on a populated table, not an assumption. | Measure index usage (`sys.dm_db_index_usage_stats`) after the migration has run in QA/prod for a few weeks, then drop the unused ones. |
+| 135 | **→ #70** **Config-audit single-column indexes are possibly redundant** *(Low)*. `ReviewRemediationIndexes` added composite `AuditLogEntries` indexes `(EntityType, Timestamp DESC)`, `(User, Timestamp DESC)` and `(Timestamp DESC, Id DESC)`; the older single-column indexes on timestamp/user/entity type may now be dead weight on a write-heavy table. | 2026-10-21 system review (`docs/plans/2026-10-21-system-review-remediation.md`) (lane P / O) | Dropping needs a measured before/after on a populated table, not an assumption. | Measure index usage (`sys.dm_db_index_usage_stats`) after the migration has run in QA/DEV for a few weeks, then drop the unused ones. |

 ## Resolved (verified against the code 2026-07-10)
 Rows removed from the Deferred table above once confirmed shipped. Kept here for traceability.
@@ -87,7 +87,7 @@
    {"id": 16, "subject": "Task 16: Clean up DeploymentManagerActor and AkkaHostedService", "status": "completed", "classification": "standard", "blockedBy": [15], "note": "LANDED AS ONE COMMIT with tasks 15+16 - PLAN DEFECT: they cannot compile separately (SiteReplicationActor takes a ReplicationService + calls ReplaceAllAsync; DeploymentManagerActor Tells ReplicationMessages types; AkkaHostedService constructs the actor). Combining also strengthens Task 14's own invariant: never both mechanisms, never neither. Registered 8 tables; notification_lists + smtp_configurations deliberately NOT registered. Deleted ReplaceAllAsync as UNSAFE (a mass DELETE on a replicated table would be captured and shipped), and its test replaced by a comment explaining that. The positional-arg hazard was REAL: 4 DeploymentManagerActor test call sites bound wrong args; converted to named where possible - Props.Create is an expression tree and rejects OUT-OF-POSITION named args, so the rest are padded positionally. Task 7's 'not yet' test INVERTED (exact in both directions) + new security-named test for the SMTP tables + composite-PK test. Harness's temporary registration deleted, so the convergence suites now prove the cutover. My expected table list was mis-sorted: ordinal puts '_' (0x5F) before 'b', so data_connection_definitions precedes database_connections.", "commit": "037798b3"},
    {"id": 17, "subject": "Task 17: Config-key cleanup (incl. ConfigFetchRetryCount; 10 appsettings files)", "status": "completed", "classification": "standard", "blockedBy": [16], "note": "As planned. DEVIATION: deploy/wonder-app-vd03/appsettings.Site.json sits under a GITIGNORED deploy/ tree, so its edit is local-only and must be repeated on the box at deploy time. Comment style is // (JSONC) rather than \"_comment_\" keys: every one of these files already contains // comments and .NET's json config reader accepts them, and a _comment_ key inside a bound section is a phantom config entry. Both relaxations pinned by the INVERSE of the test they replace (Site_MissingSiteDbPath_IsAccepted..., EmptySqliteDbPath_IsAccepted...), each verified red with the old rule restored. DatabaseOptionsValidator needed no change - it was already null-tolerant/blank-rejecting, which is exactly migration-only semantics.", "commit": "605e5682"},
    {"id": 18, "subject": "Task 18: Two-node convergence suite for the Phase 2 tables", "status": "completed", "classification": "high-risk", "blockedBy": [17], "note": "DEVIATION: landed as a NEW file LocalDbPhase2ConvergenceTests.cs rather than extending Phase 1's LocalDbSitePairConvergenceTests.cs, and drives the REAL SiteStorageService instead of hand-written SQL - possible only post-cutover, and what makes the cascade scenario test the shipped transaction rather than a re-creation of it. Scenario 4 was retargeted onto shared_scripts/external_systems/static_attribute_overrides because the plan's version overlapped LocalDbConfigConvergenceTests' N1 scenario almost exactly; the union-survives property is per-table, so re-proving it on untouched tables is the non-redundant half. Cascade test carries a never-removed control instance (absence assertions otherwise cannot distinguish 'cascade converged' from 'node B lost these tables'). Non-vacuity PROVEN as mandated: with the 8 RegisterReplicated calls commented out, 4 failed / 0 passed; restored, 20/20 across the three LocalDb suites.", "commit": "15013156"},
-    {"id": 19, "subject": "Task 19: Rig configuration (MaxOplogRows/MaxOplogAge + MaxBatchSize per D6)", "status": "completed", "classification": "small", "blockedBy": [17], "note": "MaxBatchSize 500->16 (D6: row-count batching x ~70 KB production config_json vs the 4 MB gRPC cap; 16 => ~1.1 MB worst case). MaxOplogRows 1M->250,000 and MaxOplogAge 7d->2d from the soak's 0.80 rows/sec (~69k/day). Tighter-than-default is SAFE because a cap breach prunes + sets needs_snapshot (graceful snapshot resync), not data loss - the Task 1 finding that the plan's stop condition was weaker than written. site-b/site-c left unreplicated so default-OFF stays proven side by side.", "commit": "921edab4"},
+    {"id": 19, "subject": "Task 19: Rig configuration (MaxOplogRows/MaxOplogAge + MaxBatchSize per D6)", "status": "completed", "classification": "small", "blockedBy": [17], "note": "MaxBatchSize 500->16 (D6: row-count batching x ~70 KB DEV-host (vd03) config_json vs the 4 MB gRPC cap; 16 => ~1.1 MB worst case). MaxOplogRows 1M->250,000 and MaxOplogAge 7d->2d from the soak's 0.80 rows/sec (~69k/day). Tighter-than-default is SAFE because a cap breach prunes + sets needs_snapshot (graceful snapshot resync), not data loss - the Task 1 finding that the plan's stop condition was weaker than written. site-b/site-c left unreplicated so default-OFF stays proven side by side.", "commit": "921edab4"},
    {"id": 20, "subject": "Task 20: Live gate on the docker rig", "status": "completed", "classification": "high-risk", "blockedBy": [18, 19], "note": "ALL 10 CHECKS PASS. Evidence: docs/plans/2026-07-19-localdb-phase2-live-gate.md. Key blocker found and fixed mid-run: external systems reach a site ONLY via ArtifactDeploymentService, which `instance deploy` never invokes - `deploy artifacts` was needed both to deliver the probe harness and to propagate the owed ExternalSystemDefinitions restore. THREE METHOD CORRECTIONS to the plan: (1) its instruction to run DB checks host-side against the bind mounts is UNSAFE - host sqlite3 poisons the container WAL; copy the db/-wal/-shm triplet and query the copy. (2) `docker exec ... curl` cannot scrape metrics (no curl in aspnet:10.0) and with 2>/dev/null the failure is silent - it nearly became a false 'metrics missing' finding; use a network-sharing curl sidecar. (3) checks needing S&F load need CachedCall, not Call. CAVEATS recorded not glossed: check 2's zero-count is vacuous alone (legacy source was also empty) and rests on the ABSENCE of CDC triggers; check 7's native_alarm_state leg was empty live and is covered only offline; check 10's sampling is coarse and the real rise/drain evidence comes from check 6's 0->4->0.", "commit": "158e79bb"},
    {"id": 21, "subject": "Task 21: Documentation truth pass", "status": "completed", "classification": "standard", "blockedBy": [20], "note": "Both CLAUDE.md files + Component-StoreAndForward.md:83 (normative resync paragraph rewritten for CDC, stating the duplicate-delivery bound explicitly: limited to messages the OLD primary delivered whose status change had not yet replicated when the gate flipped - one flush interval plus in-flight ack, and it does NOT grow with backlog depth or absence duration) + components/{StoreAndForward,SiteRuntime,Host}.md + the frame-size known-issue (amended: Phase 2 deleted notify-and-fetch itself, so the 128KB Akka frame constraint is gone from the intra-site hop entirely; successor ceiling is the 4MB gRPC cap via MaxBatchSize, and note the failure mode differs - oversized gRPC is REJECTED, not silently dropped) + deployment topology-guide.md and installation-guide.md (D5 stop-both-together, D2 TombstoneRetention resurrection bound). DoD closed: build 0 warnings, all 10 suites green (3509 tests, 0 failures). DoD grep nuance recorded in the plan: 4 matches remain in src/ and are all deliberate COMMENT prose explaining what was deleted - a literal 'no matches' would delete the explanations that stop someone re-introducing the old design.", "commit": "7b5a5a6f"}
  ],
@@ -102,5 +102,5 @@
    }
  ],
  "lastUpdated": "2026-07-20",
-  "phase2Status": "COMPLETE - all 21 tasks done, live gate PASS (docs/plans/2026-07-19-localdb-phase2-live-gate.md), MERGED to main as 28ca04d7 ('LocalDb adoption Phase 1 + 2 ... (#23)'), which carried Phase 1 in the same merge. History below retained for context. Task 1 gate CLOSED (verdict PROCEED) and Task 2 DONE, 2026-07-20. Task 1's original STOP verdict is SUPERSEDED: the 'Phase 1 disk I/O defect' was OBSERVER-INDUCED (host-side sqlite3 against live bind-mounted WAL files resets the WAL across virtiofs and permanently poisons the container's connections) - NOT a product defect. See docs/known-issues/2026-07-20-localdb-disk-io-error-under-load.md. Both earlier isolation claims were confounded: one sampling pass had already poisoned BOTH nodes, and a poisoned standby looks healthy only because it issues almost no statements; 'LocalDb-specific' was sampling-selection bias (only the LocalDb file had ever been host-read). Clean re-run 2026-07-20 with the copy-based snap() helper on restarted nodes, 6 consecutive 60s intervals: sf_messages 48 rows/min = 0.80 rows/sec dead steady, retry-UPDATE rate 0/sec (SUM(retry_count) flat at 200), oplog 0, native_alarm_state 0, max payload_json 76 B, max config_json 721 B, ZERO SQLite errors across 30 min, both site-a nodes converged at 3564 rows. Honest gap: 0.80/s is ~1.6% of the 50/s ceiling and the retry-UPDATE path was never exercised - acceptable because that ceiling is structural (SweepBatchLimit/RetryTimerInterval), not empirical. Rig config rows (721 B) are NOT representative; D6 sizing rests on the documented ~60-70 KB production config_json. Plan-premise corrections stand: D6 (MaxBatchSize 500 -> 16, the one firmly evidence-backed number; Task 19 sets it), D4 (alarm writes bounded by per-SourceReference coalescing at a 100 ms flush, NOT unbounded), sf_messages hard ceiling 50 rows/sec, oplog cap overrun = graceful snapshot resync (needs_snapshot), not data loss. Tasks 3-21 were subsequently executed and all landed. Rig cleanup that was owed before Task 20 is DONE: ExternalSystemDefinitions id 1 restored to http://scadabridge-restapi:5200 (propagation to both site nodes confirmed), and SoakGenerator template 2021 + instances soakgen-1..4 deleted - see 'Rig state as left' in the live-gate doc."
+  "phase2Status": "COMPLETE - all 21 tasks done, live gate PASS (docs/plans/2026-07-19-localdb-phase2-live-gate.md), MERGED to main as 28ca04d7 ('LocalDb adoption Phase 1 + 2 ... (#23)'), which carried Phase 1 in the same merge. History below retained for context. Task 1 gate CLOSED (verdict PROCEED) and Task 2 DONE, 2026-07-20. Task 1's original STOP verdict is SUPERSEDED: the 'Phase 1 disk I/O defect' was OBSERVER-INDUCED (host-side sqlite3 against live bind-mounted WAL files resets the WAL across virtiofs and permanently poisons the container's connections) - NOT a product defect. See docs/known-issues/2026-07-20-localdb-disk-io-error-under-load.md. Both earlier isolation claims were confounded: one sampling pass had already poisoned BOTH nodes, and a poisoned standby looks healthy only because it issues almost no statements; 'LocalDb-specific' was sampling-selection bias (only the LocalDb file had ever been host-read). Clean re-run 2026-07-20 with the copy-based snap() helper on restarted nodes, 6 consecutive 60s intervals: sf_messages 48 rows/min = 0.80 rows/sec dead steady, retry-UPDATE rate 0/sec (SUM(retry_count) flat at 200), oplog 0, native_alarm_state 0, max payload_json 76 B, max config_json 721 B, ZERO SQLite errors across 30 min, both site-a nodes converged at 3564 rows. Honest gap: 0.80/s is ~1.6% of the 50/s ceiling and the retry-UPDATE path was never exercised - acceptable because that ceiling is structural (SweepBatchLimit/RetryTimerInterval), not empirical. Rig config rows (721 B) are NOT representative; D6 sizing rests on the documented ~60-70 KB DEV-host (vd03) config_json. Plan-premise corrections stand: D6 (MaxBatchSize 500 -> 16, the one firmly evidence-backed number; Task 19 sets it), D4 (alarm writes bounded by per-SourceReference coalescing at a 100 ms flush, NOT unbounded), sf_messages hard ceiling 50 rows/sec, oplog cap overrun = graceful snapshot resync (needs_snapshot), not data loss. Tasks 3-21 were subsequently executed and all landed. Rig cleanup that was owed before Task 20 is DONE: ExternalSystemDefinitions id 1 restored to http://scadabridge-restapi:5200 (propagation to both site nodes confirmed), and SoakGenerator template 2021 + instances soakgen-1..4 deleted - see 'Rig state as left' in the live-gate doc."
 }
@@ -182,10 +182,10 @@ says the *escaped Akka envelope* exceeded the 128 KB frame, and that the default
 double-escapes the payload, so **"the raw flattened JSON only needs to be ~60-70 KB to blow the
 128 KB frame."**

-So the largest known real production `config_json` is on the order of **60–70 KB**.
+So the largest known real-world `config_json` (DEV host vd03) is on the order of **60–70 KB**.

 Measured on the rig (4 deployed configs): **max 715 B, avg 714 B** — trivially small, as the plan
-predicted, hence the production figure above is the one to size against. (A direct measurement
+predicted, hence the DEV-host figure above is the one to size against. (A direct measurement
 against wonder was attempted; `wonder-app-vd03.zmr.zimmer.com` resolves over the VPN but the
 servecli SSH service on :2222 refuses connections, so the box was unreachable.)

@@ -323,7 +323,7 @@ exercised at all. This re-run therefore confirms *steady-state health and the ab
 Finding 1 pathology*; it does **not** probe the ceiling. That is acceptable because the ceiling is
 **structural rather than empirical** (Finding 4: `SweepBatchLimit` ÷ `RetryTimerInterval`), so
 sizing does not depend on observing it. Likewise the rig's 721 B config rows are **not**
-representative — D6's sizing rests on the documented ~60–70 KB production `config_json`, not on
+representative — D6's sizing rests on the documented ~60–70 KB DEV-host (vd03) `config_json`, not on
 this number.

 ## 2. What still owes measurement
@@ -338,7 +338,7 @@ Carry forward (items 1–2 partially discharged by §1b above):
   answer ConditionRefresh with a `SnapshotComplete`; the missing piece is ongoing transitions
   and a seeded `TemplateNativeAlarmSource`. (The test's own doc comment claiming the simulator
   "does not reliably expose A&C" is stale.)
-3. A production-representative `config_json` from wonder, to replace the ~60–70 KB inference.
+3. A real-world (DEV-host) `config_json` from wonder, to replace the ~60–70 KB inference.
 4. The empirical oplog drain-under-churn check — Task 20 evidence item 10.

 ## 3. Rig state left behind
@@ -469,7 +469,7 @@ Add `"SelfFormAfter": "00:00:10"` next to `"SplitBrainResolverStrategy"` in the

 Then `dotnet test tests/ZB.MOM.WW.ScadaBridge.Host.Tests` and commit `config(cluster): SelfFormAfter=10s explicit in all node appsettings`.

-> Ops note (do NOT edit here): the gitignored `deploy/wonder-app-vd03/` overlay gets the same key on the next production deploy.
+> Ops note (do NOT edit here): the gitignored `deploy/wonder-app-vd03/` overlay gets the same key on the next DEV (vd03) deploy.

 ---

@@ -20,7 +20,7 @@

 **Empirical proof** (scratch repro, Roslyn 5.0.0, options mirroring the site's): a 5-assembly explicit set has a 74-assembly transitive closure; compiling the *same code* three times drove `ResolveMissingAssembly` 74 times **per compile** (222 total) and returned 222 **distinct** `PortableExecutableReference` instances — i.e. 74 fresh `AssemblyMetadata`+`PEReader`+native-block sets per compile. With a memoizing decorator: compile #1 = 74 inner resolutions, compiles #2/#3 = **0**, and the script still evaluates correctly.

-This matches the production gcdump signature exactly:
+This matches the DEV-host (vd03) gcdump signature exactly:
 - Per-script scaling, untouched by the `5a781c70` fix: Site (21 scripts) 6,640 triple-objects ≈ 21 × ~105 refs × 3 counted types; Central (6 scripts) 2,699 ≈ 6 × ~150 × 3.
 - Bit-identical counts pre/post fix on an identical workload (`NativeHeapMemoryBlock` 2,681 both times): the minting is deterministic per compiled script, and the explicit sets the fix de-duplicated were a small fraction of the closure.
 - Invisible to source greps: no ScadaBridge code calls anything per compile — the `CreateFromFile` calls happen inside `RuntimeMetadataReferenceResolver`.
@@ -555,7 +555,7 @@ git commit -m "docs(scripts): record the shared caching metadata resolver invari

 ### Task 9 (operator-gated): Live verification, keyed to compile count — not elapsed time

-**Classification:** high-risk (production measurement; no code)
+**Classification:** high-risk (live DEV-host measurement; no code)
 **Estimated implement time:** operator session

 Run on the docker rig first, then wonder-app-vd03 after deploy (tooling at `E:\ApiInstall\_tools\diag\`, baselines at `E:\ApiInstall\_leakcheck-post5a781c70\`):
@@ -590,7 +590,7 @@ Measured as a fixed-compile-count A/B (both nodes compile an identical startup s
 | Site / Central working set | 2,814 / 1,374 MB | 612 / 663 MB (post-fix read at ~5 min uptime — not uptime-matched; the metadata counts are the controlled result) |
 | Site / Central working set — **uptime-matched re-read** (26.3 min, same `20f2ae05` boot) | 2,814 / 1,374 MB (24+ min) | **620 / 695 MB** — flat vs the 5-min read (+8 / +32 MB), so the comparison is now clean: **4.5× / 2.0× working-set reduction** |

-`CachingScriptMetadataResolver` shows exactly 1 instance per process in the gcdump — active in production. Calibration note: the steady-state residual is a **fixed** cost of roughly one shared closure set + statics (~800 for 21 scripts ≈ 38/script amortized), not the per-compile ≤~10 growth figure — the growth figure applies to *additional* compiles (rig-verified at +2/compile), the fixed cost to the startup set. Follow-up lead — **re-tested 2026-08-12, no longer reproduces**: the complex `AddTemplateScript` OOM (historically a deterministic 3–8 min hang → `OutOfMemoryException` on a ~20-line handshake body, all three authored forms, reproducible on a throwaway template) was re-run on `20f2ae05` against a fresh throwaway template (id 27, created and deleted for the test) with a faithful re-authoring of the `WriteBatchAndWaitAsync` form AND the `Attributes.WaitAsync` form, driven from a clean off-box client: both returned **HTTP 200 in <1 s**, rows persisted, Central healthy. Recorded as *no longer reproduces* rather than attributed to this fix — four deploys (`3c9b101d`→`20f2ae05`) intervened since the last confirmed repro and are not distinguishable after the fact.
+`CachingScriptMetadataResolver` shows exactly 1 instance per process in the gcdump — active on the DEV host. Calibration note: the steady-state residual is a **fixed** cost of roughly one shared closure set + statics (~800 for 21 scripts ≈ 38/script amortized), not the per-compile ≤~10 growth figure — the growth figure applies to *additional* compiles (rig-verified at +2/compile), the fixed cost to the startup set. Follow-up lead — **re-tested 2026-08-12, no longer reproduces**: the complex `AddTemplateScript` OOM (historically a deterministic 3–8 min hang → `OutOfMemoryException` on a ~20-line handshake body, all three authored forms, reproducible on a throwaway template) was re-run on `20f2ae05` against a fresh throwaway template (id 27, created and deleted for the test) with a faithful re-authoring of the `WriteBatchAndWaitAsync` form AND the `Attributes.WaitAsync` form, driven from a clean off-box client: both returned **HTTP 200 in <1 s**, rows persisted, Central healthy. Recorded as *no longer reproduces* rather than attributed to this fix — four deploys (`3c9b101d`→`20f2ae05`) intervened since the last confirmed repro and are not distinguishable after the fact.

 ## Out of scope / follow-ups

@@ -4,7 +4,7 @@

 **Goal:** Stand up an in-memory (static-attribute) stand-in for a vd03-style machine with a composed `DelmiaReceiver` on the local docker rig, wire the `DelmiaRecipeDownload` inbound method and a scoped API key to it, let `DelmiaNotifier` read that key from `appsettings.json`, and prove the notifier end-to-end on the 2-node central cluster including an active-node crash and rejoin.

-**Architecture:** Two templates seeded by CLI only (no DB edits): `DelmiaReceiverSim` (the ten receiver attributes, all static — no data connection) composed into `SimMachine` as `DelmiaReceiver`. The Galaxy side of the handshake is replaced by a root-scope `ValueChange` script on `SimMachine` that reacts to `DelmiaReceiver.RecipeDownloadFlag`, so the same call script shape as prod runs against pure in-memory tags. The central inbound method is the prod script verbatim (minus its dead `suf` line). The notifier gains an optional `ScadaBridge:ApiKey` config field with the existing `SCADABRIDGE_API_KEY` env var taking precedence, and a bash verification script drives the whole thing against `docker/` (central-a `:9001`, central-b `:9002`, Traefik `:9000`).
+**Architecture:** Two templates seeded by CLI only (no DB edits): `DelmiaReceiverSim` (the ten receiver attributes, all static — no data connection) composed into `SimMachine` as `DelmiaReceiver`. The Galaxy side of the handshake is replaced by a root-scope `ValueChange` script on `SimMachine` that reacts to `DelmiaReceiver.RecipeDownloadFlag`, so the same call script shape as DEV (vd03) runs against pure in-memory tags. The central inbound method is the DEV (vd03) script verbatim (minus its dead `suf` line). The notifier gains an optional `ScadaBridge:ApiKey` config field with the existing `SCADABRIDGE_API_KEY` env var taking precedence, and a bash verification script drives the whole thing against `docker/` (central-a `:9001`, central-b `:9002`, Traefik `:9000`).

 **Tech Stack:** ScadaBridge CLI (`scadabridge`), bash, .NET 10 (`DelmiaNotifier` + xUnit tests), docker compose rig under `docker/`.

@@ -18,7 +18,7 @@
 - `security api-key create --format=json` returns `{ "keyId": …, "token": "sbk_…" }`; the token is shown once.
 - Rig: site-a is `--site-id 1`; Traefik `http://localhost:9000` → active central; direct nodes `:9001` (central-a) and `:9002` (central-b); both centrals have `StandbyForwardPeerEndpoints` set, so a standby **relays** inbound calls to the active peer. Auto-down failover of a crashed active takes ~25–30 s (`docker/failover-drill.sh`).
 - The notifier's failover is **connect-failure-only**: any HTTP answer (incl. 503) is final. During the takeover window a standby that cannot reach its peer answers 503 `ACTIVE_NODE_UNREACHABLE` → the notifier prints `NO`. That is expected and must be documented, not "fixed".
- Prod's deployed method + site script are recorded in `docs/api-specs/Delmia-Recipe-Download-API.md`; prod's site script still references `Children["DelmiaRecv"]` (a known, separate defect). The rig script uses `Children["DelmiaReceiver"]`.
+- DEV's (vd03) deployed method + site script are recorded in `docs/api-specs/Delmia-Recipe-Download-API.md`; DEV's site script still references `Children["DelmiaRecv"]` (a known, separate defect). The rig script uses `Children["DelmiaReceiver"]`.

 **Model assignment summary** (see each task's `Model:` line): Sonnet for scripted CLI seeding, config plumbing, and docs; Opus for the live rig verification and the failover drill (judgment under a live system) and for reviewing the secret-handling change; Fable (this session) for the final integration review.

@@ -200,10 +200,10 @@ set -euo pipefail
 # site-a, so DelmiaNotifier can be exercised end-to-end on the docker rig without a Galaxy.
 #
 # What it creates (all static attributes — no data connection, no bindings):
-#   - Template DelmiaReceiverSim — the ten DelmiaReceiver attributes from prod
+#   - Template DelmiaReceiverSim — the ten DelmiaReceiver attributes from DEV (vd03)
 #       (docs/api-specs/Delmia-Recipe-Download-API.md §5), ReadyFlag defaulting to true.
 #   - Template SimMachine — composes DelmiaReceiverSim as `DelmiaReceiver` and carries
-#       * ProcessRecipeDownload (Call)  — prod's handshake ported to per-attribute
+#       * ProcessRecipeDownload (Call)  — DEV's (vd03) handshake ported to per-attribute
 #         SetAsync/WaitAsync (WriteBatchAndWaitAsync rejects static attributes).
 #       * DelmiaReceiverSimulator (ValueChange on DelmiaReceiver.RecipeDownloadFlag) —
 #         plays the Galaxy object: accepts unless DownloadPath ends in ".bad" or the work
@@ -245,9 +245,9 @@ MACHINE_ID=$("$CLI" template create "${A[@]}" --name SimMachine \
 "$CLI" template composition add "${A[@]}" --template-id "$MACHINE_ID" --composed-template-id "$RECV_ID" --instance-name DelmiaReceiver > /dev/null
 echo "  id=$MACHINE_ID"

-echo "4/6 Adding ProcessRecipeDownload (Call) — prod handshake on static attributes..."
+echo "4/6 Adding ProcessRecipeDownload (Call) — DEV (vd03) handshake on static attributes..."
 read -r -d '' PROCESS_CODE <<'CS' || true
-// Port of prod CvdReactor.ProcessRecipeDownload for STATIC attributes: WriteBatchAndWaitAsync only
+// Port of DEV (vd03) CvdReactor.ProcessRecipeDownload for STATIC attributes: WriteBatchAndWaitAsync only
 // accepts data-sourced attributes, so the payload is written with SetAsync and the handshake waited
 // with WaitAsync (same trigger/response flags, same 25 s budget, same result mapping).
 var recv = Children["DelmiaReceiver"];
@@ -354,7 +354,7 @@ git commit -m "chore(docker): seed-delmia-sim.sh — in-memory SimMachine + comp
 #!/bin/bash
 set -euo pipefail

-# Seed the DelmiaRecipeDownload inbound method (prod's script verbatim, minus its dead `suf`
+# Seed the DelmiaRecipeDownload inbound method (DEV vd03's script verbatim, minus its dead `suf`
 # line — see docs/api-specs/Delmia-Recipe-Download-API.md §4) and one API key scoped to it.
 # Prints the one-time token as DELMIA_API_KEY=... — capture it; it cannot be retrieved again.
 # Not idempotent (method name must be unique): delete with `api-method delete --id`.
@@ -415,7 +415,7 @@ Expected: `{"Result":true,"ResultText":"Simulated download accepted for WO-1"}`.

 ```bash
 git add docker/seed-delmia-api.sh docker/README.md
-git commit -m "chore(docker): seed-delmia-api.sh — DelmiaRecipeDownload inbound method (prod script) + scoped delmia-notifier API key"
+git commit -m "chore(docker): seed-delmia-api.sh — DelmiaRecipeDownload inbound method (DEV vd03 script) + scoped delmia-notifier API key"
 ```

 ---
@@ -546,8 +546,8 @@ git commit -m "test(docker): verify-delmia-notifier.sh — notifier end-to-end o
 **Files:**
 - Modify: `docs/api-specs/Delmia-Recipe-Download-API.md` (new §8 "Docker rig stand-in", pointing at the two seed scripts, the static-attribute port, and the verified outcomes A–G with the measured takeover time)
 - Modify: `src/ZB.MOM.WW.ScadaBridge.DelmiaNotifier/README.md` (`## Manual smoke test` → add the rig recipe: run the three scripts, `export DELMIA_API_KEY`, expected YES/NO table)
- Modify: `docs/plans/2026-07-08-deferred-work-register.md` (one new row: "prod `ProcessRecipeDownload` still addresses `Children[\"DelmiaRecv\"]` after the 2026-06-27 rename — fix + redeploy Z28061/Z28062/Z28061Sim; QA unverified" — only if a row for it does not already exist)
- Modify: `CLAUDE.md` — in the `### Templates & Deployment` / DelmiaNotifier context, one sentence: "Rig stand-in: `docker/seed-delmia-sim.sh` + `seed-delmia-api.sh` + `verify-delmia-notifier.sh` (static-attribute port of the prod handshake — `WriteBatchAndWaitAsync` rejects static attributes, use `SetAsync`/`WaitAsync`)."
+- Modify: `docs/plans/2026-07-08-deferred-work-register.md` (one new row: "DEV (vd03) `ProcessRecipeDownload` still addresses `Children[\"DelmiaRecv\"]` after the 2026-06-27 rename — fix + redeploy Z28061/Z28062/Z28061Sim; QA unverified" — only if a row for it does not already exist)
+- Modify: `CLAUDE.md` — in the `### Templates & Deployment` / DelmiaNotifier context, one sentence: "Rig stand-in: `docker/seed-delmia-sim.sh` + `seed-delmia-api.sh` + `verify-delmia-notifier.sh` (static-attribute port of the DEV (vd03) handshake — `WriteBatchAndWaitAsync` rejects static attributes, use `SetAsync`/`WaitAsync`)."

 **Step 1: Write the sections** using the numbers from Task 4's log (takeover seconds, D's outcome, audit row count).

@@ -560,7 +560,7 @@ Expected: no hits (Task 1 removed the README/env ones; anything left here is sta

 ```bash
 git add docs/api-specs/Delmia-Recipe-Download-API.md src/ZB.MOM.WW.ScadaBridge.DelmiaNotifier/README.md docs/plans/2026-07-08-deferred-work-register.md CLAUDE.md
-git commit -m "docs: Delmia rig stand-in + notifier verification results (A-G incl. active-central takeover); register row for prod DelmiaRecv path defect"
+git commit -m "docs: Delmia rig stand-in + notifier verification results (A-G incl. active-central takeover); register row for DEV (vd03) DelmiaRecv path defect"
 ```

 ---
@@ -576,7 +576,7 @@ git commit -m "docs: Delmia rig stand-in + notifier verification results (A-G in

 ## Out of scope (deliberately)

- Fixing prod vd03's `Children["DelmiaRecv"]` defect or making the prod method idempotent — recorded in the deferred-work register (Task 5); separate change with its own deployment record.
+- Fixing DEV vd03's `Children["DelmiaRecv"]` defect or making the DEV (vd03) method idempotent — recorded in the deferred-work register (Task 5); separate change with its own deployment record.
 - Extending `WriteBatchAndWaitAsync` to static attributes.
 - Site-pair failover (the user asked for the 2-node **central** cluster); `docker/failover-drill.sh` already covers the general case.
 - Native AOT publish of the notifier (Windows-only); the managed build is what the rig exercises.
@@ -16,7 +16,7 @@ exactly one node**. Consequences, all observed or verified on 2026-09-02:
  to copy the file by hand before a failover drill can pass.
 - A key **revoked** on one node stays valid on the other. Revocation is not cluster-wide.
 - The standby-relay path only works because the standby forwards *before* authenticating.
- Prod `wonder-app-vd03` is single-node and unaffected; the QA pair `vq03`/`vq04` has the gap today.
+- DEV `wonder-app-vd03` is single-node and unaffected; the QA pair `vq03`/`vq04` has the gap today.
 - `Component-InboundAPI.md` §Storage still says keys live in the configuration database — stale since
  the 2026-06-02 re-arch (`RetireInboundApiKeyStore` dropped the SQL table and moved keys to SQLite).

@@ -97,7 +97,7 @@ Additive for every existing consumer; the four existing packages bump in lockste
  `docker/sync-inbound-api-keys.sh` is **deleted** once `verify-delmia-notifier.sh` passes without it.
 - **Windows installs**: appsettings are user-held. QA (`vq03`/`vq04`): stop both centrals **together**,
  add `Provider`, start both — the first to boot creates `zbauth` and imports its file, the second
-  imports its own (union). Prod `vd03`: same change for consistency (single node, import once). If the
+  imports its own (union). DEV `vd03`: same change for consistency (single node, import once). If the
  app login lacks DDL rights, a DBA runs `docs/operations/sql/zbauth-schema.sql` first; the migrator
  finds the version row and skips DDL. Recorded per environment in a deployment record.

@@ -547,7 +547,7 @@ Commit — `chore(docker): both rigs' centrals on the shared SQL Server inbound
 - Create: `docs/operations/sql/zbauth-schema.sql` (copy of the library's DDL for DBAs whose app login lacks DDL rights; the migrator then finds the version row and skips).
 - Modify: `docs/deployment/topology-guide.md` (central-pair section: the key store is shared under SqlServer; SQLite on a pair is a documented misconfiguration the validator warns about).
 - Modify: `env_variables.md` (rows for `ScadaBridge__InboundApi__ApiKeyStore__Provider`, `…__SqlServer__ConnectionString`, `…__ImportFromSqlitePath`).
- Modify: `CLAUDE.md` → *Security & Auth*: one bullet — inbound key store per node by default; central pairs MUST run `Provider=SqlServer` (shared `zbauth` schema in the config DB, import once at boot, revoke cluster-wide); rig does; QA/prod appsettings are user-held (deployment record).
+- Modify: `CLAUDE.md` → *Security & Auth*: one bullet — inbound key store per node by default; central pairs MUST run `Provider=SqlServer` (shared `zbauth` schema in the config DB, import once at boot, revoke cluster-wide); rig does; QA/DEV appsettings are user-held (deployment record).
 - Modify: `docs/plans/2026-07-08-deferred-work-register.md` row 108 → resolved (library 0.3.0 + host provider + rig proof; QA/vd03 rollout pending as a deployment record).
 - Modify: `docs/api-specs/Delmia-Recipe-Download-API.md` §8 (the note about the sync script → superseded).
 - Modify: `../scadaproj/CLAUDE.md` ScadaBridge entry (inbound keys on the shared store; Auth 0.3.0 consumer).
@@ -556,7 +556,7 @@ Check: `grep -rn "sync-inbound-api-keys" docs docker CLAUDE.md` → only histori

 ---

-### Task 13: Deployment record template for QA (vq03/vq04) and prod (vd03)
+### Task 13: Deployment record template for QA (vq03/vq04) and DEV (vd03)

 **Classification:** trivial
 **Estimated implement time:** ~3 min
@@ -566,9 +566,9 @@ Check: `grep -rn "sync-inbound-api-keys" docs docker CLAUDE.md` → only histori
 **Files:**
 - Create: `docs/deployment/deployment-records/2026-09-XX-zb-servers-inbound-key-store.md` (a **planned** record with the exact steps and blanks for results — the user runs it; appsettings on those boxes are user-held)

-Content: build/stage per the 2026-08-21 recipe; **QA:** stop both centrals together → add `"ApiKeyStore": { "Provider": "SqlServer" }` under `ScadaBridge:InboundApi` in both `appsettings.Central.json` → if `scadabridge_app` lacks DDL rights run `docs/operations/sql/zbauth-schema.sql` on the QA config DB first → start both → expect one `KeysInserted=N` line and one `KeysSkipped` line → `security api-key list` from `10.220.158.2:8085` and `.3:8085` agree → `sync`-free failover check with a curl against `DelmiaRecipeDownload` from the non-active node. **Prod vd03:** single node, same change (`KeysInserted=5` expected: `mes`, `mes api key`, `delmia-mes-receivers`, `TestKey`, `JOE_TEST_KEY`). Rollback: revert `Provider`, rename `.imported` back — no SQL rows need removing (the SQLite path ignores them).
+Content: build/stage per the 2026-08-21 recipe; **QA:** stop both centrals together → add `"ApiKeyStore": { "Provider": "SqlServer" }` under `ScadaBridge:InboundApi` in both `appsettings.Central.json` → if `scadabridge_app` lacks DDL rights run `docs/operations/sql/zbauth-schema.sql` on the QA config DB first → start both → expect one `KeysInserted=N` line and one `KeysSkipped` line → `security api-key list` from `10.220.158.2:8085` and `.3:8085` agree → `sync`-free failover check with a curl against `DelmiaRecipeDownload` from the non-active node. **DEV vd03:** single node, same change (`KeysInserted=5` expected: `mes`, `mes api key`, `delmia-mes-receivers`, `TestKey`, `JOE_TEST_KEY`). Rollback: revert `Provider`, rename `.imported` back — no SQL rows need removing (the SQLite path ignores them).

-Commit — `docs(deployment): planned record for moving QA/prod central inbound key stores to SQL Server`
+Commit — `docs(deployment): planned record for moving QA/DEV central inbound key stores to SQL Server`

 ---

@@ -588,4 +588,4 @@ Check: (1) scadaproj `git log` shows Tasks 1-7 committed and `artifacts/` holds
 - Site nodes, MxAccessGateway, HistorianGateway (SQLite stays their store).
 - Hard-failing a clustered central on `Provider=Sqlite` (warning only; revisit after QA is on SqlServer).
 - Encrypting `secret_hash` at rest; the pepper never enters SQL.
- Fixing register row 107 (prod `DelmiaRecv`) — separate change.
+- Fixing register row 107 (DEV vd03 `DelmiaRecv`) — separate change.
@@ -109,7 +109,7 @@
    },
    {
      "id": 13,
-      "subject": "Task 13: Planned deployment record for QA vq03/vq04 and prod vd03",
+      "subject": "Task 13: Planned deployment record for QA vq03/vq04 and DEV vd03",
      "status": "completed",
      "blockedBy": [
        12
@@ -13,7 +13,7 @@ rechecked all High source paths, ran build/test/format/package checks, and execu
 probes. Each area revalidated its existing IDs and searched for additional defects. Source-traced
 mechanisms, runtime probes, latent risks, and unproven operational impact are distinguished below.
 This is a focused whole-system review, not a claim that every line or production topology was
-exercised. The review phase ran no rig rebuild, device writes, failover drill, or remote QA/production inspection.
+exercised. The review phase ran no rig rebuild, device writes, failover drill, or remote QA/DEV inspection.
 Sections 1–8 preserve that reviewed baseline; §§9–10 record subsequent implementation and local rig verification.

 **Second-pass changes:** 11 new findings; UI-3 withdrawn as unsupported; XC-6 merged into CM-5;
@@ -479,7 +479,7 @@ source on each adapter. Medium.

 **DC-6 — `deploy site` exits 0 when its domain result reports failed instances `[Medium][CONFIRMED]`** — `CLI/Commands/CommandHelpers.cs:169-193` returns 0 for any 2xx; `DeployCommands.cs:68-84`. A CI `scadabridge deploy site … || rollback` never fires; other command-specific success payloads need separate contract checks; do not assume every JSON command has the same failure shape. Fix: an `onSuccess` handler inspecting `failureCount` → exit 1; document. Small.

-**DC-7 — DelmiaNotifier treats a standby central's 503 as final and never fails over `[Medium][CONFIRMED]`** — `DelmiaNotifier/Notifier.cs:31-52` rolls over only on `ConnectFailed`. Its HA therefore depends entirely on `StandbyForwardPeerEndpoints` being configured on both centrals — which QA (`vq03`/`vq04`) and prod (`vd03`) still lack (CLAUDE.md). Fix: distinguish a known pre-execution standby rejection from an ambiguous post-send failure before retrying; generic retry-on-any-503 can duplicate effects. Document and test the notifier HA contract. QA/prod peer settings quoted here were not remotely rechecked in this pass. Small.
+**DC-7 — DelmiaNotifier treats a standby central's 503 as final and never fails over `[Medium][CONFIRMED]`** — `DelmiaNotifier/Notifier.cs:31-52` rolls over only on `ConnectFailed`. Its HA therefore depends entirely on `StandbyForwardPeerEndpoints` being configured on both centrals — which QA (`vq03`/`vq04`) and DEV (`vd03`) still lack (CLAUDE.md). Fix: distinguish a known pre-execution standby rejection from an ambiguous post-send failure before retrying; generic retry-on-any-503 can duplicate effects. Document and test the notifier HA contract. QA/DEV peer settings quoted here were not remotely rechecked in this pass. Small.

 **DC-8 — `secured-write` is a shipping CLI group absent from the README and from `CommandTreeTests` `[Low][CONFIRMED]`** — `CLI/Program.cs:42`; `CommandTreeTests.cs:26-45, 63-71` asserts 18 groups vs 19 in `Program.cs` and builds `data-connection` without `ConnectionBrowseCommands.AddTo`, so browse/search/verify-endpoint/certs are uncovered too. Fix: add to the test (19), `AddTo` in the test build, README section. Small.

@@ -773,7 +773,7 @@ Repeatable regression recipes are attached to each finding and the §5 acceptanc
 | XC-1 | Use actual listener/local port, not spoofable Host/authority matching, for port isolation |

 All previous SR/DC/MG/UI/DL/EI/CM/XC IDs and ORCH-1 were source-revisited or, for inventory/doc
-observations, checked against their current files. Historical production/QA statements, performance
+observations, checked against their current files. Historical DEV/QA statements, performance
 estimates, and migration/deferred-register inventories are not newly measured simply because
 source is unchanged. The original twelve design commitments describe implementation structure;
 they are not evidence that the script trust gate is a complete security sandbox or that per-site
@@ -667,6 +667,6 @@ Expected: 0 errors; every project `Passed!` (the known env-only Playwright role-

 **Step 2: `git diff main --stat` review**, then `git log --oneline main..HEAD` — one commit per task above.

-**Step 3: Merge** via `superpowers-extended-cc:finishing-a-development-branch` (fast-forward `main`, delete the worktree). Do NOT push unless the user says so; QA/prod (`vq03`/`vq04`/`vd03`) are NOT deployed (remote exec is on hold — CLAUDE.md 2026-09-11 note).
+**Step 3: Merge** via `superpowers-extended-cc:finishing-a-development-branch` (fast-forward `main`, delete the worktree). Do NOT push unless the user says so; QA/DEV (`vq03`/`vq04`/`vd03`) are NOT deployed (remote exec is on hold — CLAUDE.md 2026-09-11 note).

-**Step 4: Memory** — write `mes-reportable-alarm-flag-state.md` (status, merged SHA, rig seeded, key name, QA/prod not deployed, follow-ups = register rows 121/122) and add its line to `MEMORY.md`.
+**Step 4: Memory** — write `mes-reportable-alarm-flag-state.md` (status, merged SHA, rig seeded, key name, QA/DEV not deployed, follow-ups = register rows 121/122) and add its line to `MEMORY.md`.
@@ -47,8 +47,8 @@ Rows 128-135 of `docs/plans/2026-07-08-deferred-work-register.md`: TLS productio

 ## Rollout notes

- **Not deployed:** the docker rig is rebuilt from main; QA (vq03/vq04) and prod (vd03) are not deployed.
- **Migration:** `ReviewRemediationIndexes` adds `AuditLog` `IX_AuditLog_Kind_Occurred` and `IX_AuditLog_Actor_Occurred` (partition-aligned) plus three composite `AuditLogEntries` indexes. Production applies `docs/plans/sql/ReviewRemediationIndexes.sql` by hand (index builds on a large partitioned `AuditLog`: schedule off-hours). Not yet applied to QA/prod.
+- **Not deployed:** the docker rig is rebuilt from main; QA (vq03/vq04) and DEV (vd03) are not deployed.
+- **Migration:** `ReviewRemediationIndexes` adds `AuditLog` `IX_AuditLog_Kind_Occurred` and `IX_AuditLog_Actor_Occurred` (partition-aligned) plus three composite `AuditLogEntries` indexes. Production applies `docs/plans/sql/ReviewRemediationIndexes.sql` by hand (index builds on a large partitioned `AuditLog`: schedule off-hours). Not yet applied to QA/DEV.
 - **Mixed version:** upgrade central first (new `IngestAck` field 2 and proto additions are additive; an old central ignores `rejected_event_ids` and an old site never rejects). Pair upgrades stay both-nodes-together for LocalDb sites.
 - **Config changes:** CLI hosts need a central that serves `/auth/token` (older centrals fall back to Basic automatically); Central must have a >= 32-byte `JwtSigningKey` or fails startup validation; remove `Database:MachineDataDb` and `Transport:SchemaVersionMajor` from appsettings (ignored if left).
 - **New keys (all defaulted):** `ScadaBridge:ConfigAudit:{RetentionDays 0, PurgeBatchSize 5000, MaxBatchesPerTick 200, PurgeInterval 1d}`; `Security:RoleMappingCacheSeconds 30`; `SiteEventLog:{KeywordSearchMaxWindow 7d, CapTrimCountRefreshBatches 10}`; `AuditLog` site writer/reconciler `{MaxRejectedPushAttempts 10, PurgeBatchSize 5000, PurgeVacuumPagesPerBatch 1000, MaxPagesPerTick 50}`; `SiteRuntime:ScriptCompileCacheMaxEntries 1024`; `NotificationOutbox:DispatchSweepBudget 2m`; `ManagementService:{SecuredWriteTerminalRetention 90d, SecuredWriteTerminalPurgeInterval 1h}`; ESG `DefinitionCacheTtl 10s` (central); DCL `MxAlarmStreamReopenDebounce 200ms`; `DeploymentManager:TerminalDeploymentRecordRetention` governs system artifact deployment record purge.