test(secrets): central-shared-store live gate 5/5 — shared SQL store + hub failover proven on the rig

Rig config: central pair gains Secrets__SqlServer__ConnectionString (dedicated
ZbSecretsHub database on the existing scadabridge-mssql, dev credentials); site-a
pair gains Secrets__GrpcHub__FallbackEndpoints__0 = central-b. Gate doc records
5/5 PASS (parity-by-construction, both-direction failover incl. recovered-primary
wrap, delete-while-follower-offline with no resurrection, Layer-A expander
provably reading the shared store via a stale-SQLite decoy discrimination, and
fail-closed negatives), discharging the Program.cs SQL-expander offline-test
residual, plus two defects documented NOT patched: the SqlServer migrator's
concurrent virgin-DB CREATE SCHEMA race (error 2714 not in the retry filter) and
the Host's pre-Serilog crash path wedging at 100% CPU instead of exiting.

Claude-Session: https://claude.ai/code/session_014WNM4vjoVksyyBraTXSZE1

docker/docker-compose.yml

@@ -33,14 +33,29 @@ x-secrets-hub-env: &secrets-hub-env
   # hub and every follower; an unset token is a startup failure on both halves.
   Secrets__GrpcHub__BearerToken: "secrets-hub-docker-dev-token"
 
-# Site half of the same section. NOTE the asymmetry with
-# ScadaBridge:Communication:CentralGrpcEndpoints, which is a LIST that fails over across
-# the central pair: the hub client dials a SINGLE endpoint, so a sweep against a stopped
-# central-a stalls rather than failing over to central-b. That is survivable — the sweep
-# is best-effort and the node keeps serving its full local last-known-good store — but
-# secrets stop converging until central-a returns.
+# Central-only half: the SHARED SQL-Server secret store (scadaproj#4, Secrets 0.5.0).
+# In Grpc mode BOTH central nodes read and write ONE copy of every row in this database,
+# so the two hub instances serve identical manifests by construction — that is what makes
+# the site-side FallbackEndpoints below safe. ZbSecretsHub is a dedicated database on the
+# rig's existing scadabridge-mssql container (default zbsecrets schema, created by the
+# boot-time SqlServerSecretsStoreMigrator; the database itself + the scadabridge_app grant
+# were provisioned once via sqlcmd — see docs/plans/2026-08-07-secrets-central-shared-store-live-gate.md).
+# Same DEV-ONLY credentials as the ScadaBridge__Database__* strings above. Must be a
+# LITERAL value, never a ${secret:} reference — the pre-host expander needs this string to
+# reach the store that would resolve it (bootstrap circularity; registration rejects it).
+# Site nodes must NEVER carry this key: sites talk to central, not to central's database.
+x-secrets-hub-central-env: &secrets-hub-central-env
+  Secrets__SqlServer__ConnectionString: "Server=scadabridge-mssql,1433;Database=ZbSecretsHub;User Id=scadabridge_app;Password=ScadaBridge_Dev1#;TrustServerCertificate=true"
+
+# Site half of the same section. The hub client dials Endpoint first and fails over, per
+# call, to FallbackEndpoints in order (sticky on whichever answered last, Secrets 0.5.0).
+# Listing central-b is safe ONLY because both centrals serve the one shared SQL store
+# above — never list endpoints backed by independent stores: failing over to an emptier
+# hub is a silent convergence stop, the exact defect (scadaproj#4) the shared store
+# exists to prevent.
 x-secrets-hub-site-env: &secrets-hub-site-env
   Secrets__GrpcHub__Endpoint: "http://scadabridge-central-a:8083"
+  Secrets__GrpcHub__FallbackEndpoints__0: "http://scadabridge-central-b:8083"
 
 services:
   central-a:
@@ -51,8 +66,9 @@ services:
     stop_grace_period: 30s
     container_name: scadabridge-central-a
     environment:
-      # Hub half of the pull-only gRPC secrets hub (anchor at the top of this file).
-      <<: *secrets-hub-env
+      # Hub half of the pull-only gRPC secrets hub + the central-only shared SQL store
+      # (anchors at the top of this file).
+      <<: [*secrets-hub-env, *secrets-hub-central-env]
       SCADABRIDGE_CONFIG: Central
       ASPNETCORE_ENVIRONMENT: Development
       ASPNETCORE_URLS: "http://+:5000"
@@ -88,10 +104,10 @@ services:
       - "9013:8083"   # gRPC control plane (CentralControlService, T1A.2)
     volumes:
       - ./central-node-a/appsettings.Central.json:/app/appsettings.Central.json:ro
-      # Added for the gRPC secrets hub: the node's local secret store lives at
-      # Secrets__SqlitePath=/app/data/scadabridge-secrets.db, so it needs the same
-      # per-node volume the site nodes already have. Without it the store sits in the
-      # image's writable layer and is destroyed by every container recreate.
+      # Originally added for the gRPC secrets hub's local SQLite store; since the
+      # central store moved to the shared SQL-Server database (Secrets 0.5.0,
+      # scadaproj#4) the scadabridge-secrets.db here is a pre-0.5.0 residue, but the
+      # volume is still needed (inbound-api-keys.sqlite lives on it).
       - ./central-node-a/data:/app/data
       - ./central-node-a/logs:/app/logs
     networks:
@@ -106,8 +122,9 @@ services:
     stop_grace_period: 30s
     container_name: scadabridge-central-b
     environment:
-      # Hub half of the pull-only gRPC secrets hub (anchor at the top of this file).
-      <<: *secrets-hub-env
+      # Hub half of the pull-only gRPC secrets hub + the central-only shared SQL store
+      # (anchors at the top of this file).
+      <<: [*secrets-hub-env, *secrets-hub-central-env]
       SCADABRIDGE_CONFIG: Central
       ASPNETCORE_ENVIRONMENT: Development
       ASPNETCORE_URLS: "http://+:5000"
@@ -143,10 +160,10 @@ services:
       - "9014:8083"   # gRPC control plane (CentralControlService, T1A.2)
     volumes:
       - ./central-node-b/appsettings.Central.json:/app/appsettings.Central.json:ro
-      # Added for the gRPC secrets hub: the node's local secret store lives at
-      # Secrets__SqlitePath=/app/data/scadabridge-secrets.db, so it needs the same
-      # per-node volume the site nodes already have. Without it the store sits in the
-      # image's writable layer and is destroyed by every container recreate.
+      # Originally added for the gRPC secrets hub's local SQLite store; since the
+      # central store moved to the shared SQL-Server database (Secrets 0.5.0,
+      # scadaproj#4) the scadabridge-secrets.db here is a pre-0.5.0 residue, but the
+      # volume is still needed (inbound-api-keys.sqlite lives on it).
       - ./central-node-b/data:/app/data
       - ./central-node-b/logs:/app/logs
     networks:

docs/plans/2026-08-07-secrets-central-shared-store-live-gate.md

@@ -0,0 +1,382 @@
+# Secrets central shared SQL store + hub failover — live gate on the docker cluster
+
+**Result: PASS — 5/5 checks.** Run 2026-08-07 on the local 8-node docker cluster.
+Two defects were found and are **documented, not patched** (standing rule): one in the
+`ZB.MOM.WW.Secrets.Replicator.SqlServer` library (concurrent first-boot migration race),
+one in the ScadaBridge Host's pre-Serilog crash path (unhandled boot exception wedges the
+process instead of exiting). Neither is a failure of a gate check — every check's own
+criterion passed — but both need a controller decision. See [Defects](#defects-found).
+
+## Purpose
+
+Terminal gate (Task 5) for the central-shared-SQL-store program
+(`scadaproj/docs/plans/2026-08-07-secrets-central-shared-sql-store.md`, resolving
+[scadaproj#4](https://gitea.dohertylan.com/dohertj2/scadaproj/issues/4)): in
+`Secrets:Replication:Mode=Grpc`, both central nodes' `ISecretStore` is now the **shared
+SQL-Server store** (`AddZbSecretsSqlServerStore`), so the two hub instances serve identical
+manifests **by construction**, and the site-side hub client gains ordered
+`Secrets:GrpcHub:FallbackEndpoints` with sticky preference (Secrets family **0.5.0**).
+
+This gate also **discharges the code-review residual that the `Program.cs` SQL expander
+path has no offline test**: check 4 exercises the `expanderUsesSharedSqlStore` branch
+positively (Layer-A `${secret:}` resolution from the shared SQL store, including the
+`SqlServerSecretsStoreMigrator` boot run), and check 5(a) exercises its fail-closed
+negative (`EnsureCentralSharedStoreConnectionString` throwing the designed message from
+`Program.cs` line 84 — the expander block — before any `SqlConnection` sees the value).
+
+**Scope: rig config only.** No product code was changed by this gate.
+
+## Rig state
+
+| Fact | Value |
+|---|---|
+| Repo / branch | `ScadaBridge` @ `main` |
+| Git SHA gated | `68f812eaa4b29b0657ef2d967b1fa0dfc105c214` (`68f812ea`) |
+| Image | `scadabridge:latest` = `b41249cc7bf8`, built 15:04:42Z |
+| `ZB.MOM.WW.ScadaBridge.Host.dll` | `9dba0869ff4dfa7c…` — byte-identical on all 4 participating nodes |
+| `ZB.MOM.WW.Secrets.Replicator.Grpc.dll` | `4c837f627e901641…`, informational version **`0.5.0+7a2980e`**, `FailoverSecretsHubReader` present — byte-identical on all 4 |
+| Secrets libs | `ZB.MOM.WW.Secrets*` **0.5.0** |
+| Hub enabled on | `central-a`, `central-b` (hub, **shared SQL store**) and `site-a-a`, `site-a-b` (followers, local SQLite) |
+| Hub NOT enabled on | `site-b-*`, `site-c-*` — verified: zero `Secrets__*` env vars, zero hub/sweep log lines |
+| Sweep interval | `00:00:30` — the product default, not shortened; timings below are real |
+| KEK | the same shared committed **dev-only** key as the 2026-08-07 hub gate (`kek_id` sha256 `7451bcbc1f1f` on every row observed) |
+| Working tree at build | only `docker/docker-compose.yml` modified (rig config, not in the image) + untracked `.claude/` |
+
+### Rig DB choice (record)
+
+Central's shared store is a **dedicated database `ZbSecretsHub`** on the rig's existing
+`scadabridge-mssql` container (the same SQL Server ScadaBridge central already uses),
+default `zbsecrets` schema, tables created by the boot-time `SqlServerSecretsStoreMigrator`
+(`schema_version` = 1, `secret`). The database itself and the `scadabridge_app` user grant
+(db_owner on that DB only) were provisioned once via `sqlcmd` in the SQL container. The
+connection string uses the rig's **existing committed dev credentials**
+(`User Id=scadabridge_app`, password as already committed in the compose/infra files —
+redacted here), `TrustServerCertificate=true`.
+
+### Rig-config change made by this gate (committed)
+
+`docker/docker-compose.yml`:
+
+- New anchor `x-secrets-hub-central-env` → `Secrets__SqlServer__ConnectionString`
+  pointing at `scadabridge-mssql,1433 / ZbSecretsHub`. Applied to **central-a and
+  central-b only** (`<<: [*secrets-hub-env, *secrets-hub-central-env]`). Site nodes must
+  never carry this key — sites talk to central, not to central's database.
+- `x-secrets-hub-site-env` gains
+  `Secrets__GrpcHub__FallbackEndpoints__0: "http://scadabridge-central-b:8083"` (site-a
+  pair only), with the same-store-only warning in the comment.
+- The stale central-volume comments (SQLite as central's secret store) updated: the
+  `scadabridge-secrets.db` under `central-node-*/data` is now a pre-0.5.0 residue; the
+  volume survives for `inbound-api-keys.sqlite`.
+
+All values remain DEV-ONLY under the same committed-rig-credential exception as before.
+
+## Method
+
+Central-side seeding/deleting used a throwaway helper console (scratchpad, not committed)
+that composes **exactly** what central's Layer-A expander composes —
+`AddZbSecretsSqlServerStore` + `SqlServerSecretsStoreMigrator` + the CLI's own
+`SecretCommands` — against `ZbSecretsHub` over TCP with the rig dev KEK. This is the
+plan's "CLI → shared SQL" seeding path; with one shared store, a row written by any client
+is on exactly the path a row written via central-b would be (proven bidirectionally by
+check 1's byte-identical hub reads). The `/admin/secrets` page was again not used —
+covered separately by ScadaBridge#22.
+
+Site-store assertions honored the gate-method rule from last time: **every** SQLite
+access ran in a throwaway container on the guest kernel (`mcr.microsoft.com/dotnet/aspnet:10.0`
++ the published `secret` CLI, mounting the node's `data` volume). SQL Server, being
+client-server, has no virtiofs coherence problem, so host-side access to the shared store
+is safe.
+
+gRPC calls were made with `fullstorydev/grpcurl` on `scadabridge-net` against the
+committed `secrets_hub.proto`. No token, KEK, secret plaintext, or connection-string
+password appears in this document; ciphertext identity is a SHA-256 over the encrypted
+columns only.
+
+## Results
+
+| # | Check | Result | Timing |
+|---|---|---|---|
+| 1 | Central-pair parity by construction | **PASS** | byte-identical hub reads; 1 SQL row |
+| 2 | Failover (`FallbackEndpoints`, sticky) | **PASS** | converged 18 s with central-a down; reverse failover converged same tick |
+| 3 | Delete-while-follower-offline | **PASS** | tombstone on returning node's first sweep; no resurrection |
+| 4 | Layer-A expander on SQL | **PASS** | value provably landed from the shared store |
+| 5 | Fail-closed negatives | **PASS** | designed boot refusal; hub throws on SQL outage, never serves empty |
+
+Baseline before seeding: `ZbSecretsHub..secret` empty; site-a stores held the previous
+gate's three `hub-gate-smoke-*` rows (pull-only never deletes — they persist as
+last-known-good residue); central-a's residual SQLite held the same three;
+**central-b's residual SQLite was empty — the literal scadaproj#4 divergence, still on
+disk**, which check 4 turns into a decoy.
+
+### Check 1 — central-pair parity by construction · PASS
+
+Seeded `shared-gate-1` into the shared store at 15:10:53Z. Then, authenticated:
+
+- `GetManifest` on **both** `scadabridge-central-a:8083` and `scadabridge-central-b:8083`
+  returned the identical single entry (`shared-gate-1`,
+  `updatedUtc=2026-08-07T15:10:53.8689010+00:00`).
+- `GetSecrets` full-row responses from the two hubs are **byte-identical**
+  (sha256 `ec1f270c9404471d…` for both), `cipher_sha256=40e5a8df9faa8e189c9f0dc6`,
+  `kek sha256 7451bcbc1f1f`.
+- `SELECT COUNT(*) FROM zbsecrets.secret` → **1**. There is one copy of the row; "same
+  HLC/ciphertext" is not a replication outcome to verify but an identity.
+
+The 2026-08-07 hub gate's empty-manifest-from-central-b defect is **unreproducible by
+construction**: central-b has no store of its own to be empty. Both site nodes converged
+the row in **2 s** (seed 15:10:53Z → `pulled 1 row(s)` 15:10:55Z).
+
+### Check 2 — failover · PASS
+
+The failover machinery was exercised in **both directions**, plus once unplanned:
+
+**Boot window (unplanned, evidence kept).** During bring-up central-a was wedged (defect 1)
+and central-b not yet serving. Both site nodes logged the 0.5.0 failover Warning — naming
+both endpoint URIs and nothing else — then the sweep-level failure:
+
+```
+site-a-a [15:04:55 WRN] Secrets hub endpoint http://scadabridge-central-a:8083 did not answer (Unavailable); failing over to http://scadabridge-central-b:8083.
+site-a-a [15:05:25 WRN] Secrets hub endpoint http://scadabridge-central-a:8083 did not answer (Unavailable); failing over to http://scadabridge-central-b:8083.
+```
+
+At 15:05:25 central-b answered → both clients went **sticky on central-b**.
+
+**Primary down (the check proper).** central-a stopped 15:11:43Z. **No new warning
+appeared — correct, and worth understanding:** the clients were already sticky on
+central-b, so the primary's outage cost zero failed calls, which is the documented sticky
+rationale working as designed. `shared-gate-2` was seeded at 15:15:07Z with central-a
+down; **both** site nodes converged it at 15:15:25Z (**18 s**) and decrypted it to the
+exact seeded plaintext (in-container `secret get`).
+
+**Reverse drill (rotation wrap + recovered-primary reuse).** central-a restarted healthy
+(15:16:15Z), then central-b stopped (15:16:24Z) and `shared-gate-2b` seeded (15:16:35Z):
+
+```
+site-a-a [15:16:55 WRN] Secrets hub endpoint http://scadabridge-central-b:8083 did not answer (Unavailable); failing over to http://scadabridge-central-a:8083.
+site-a-a [15:16:55 INF] Secret hub sync converged: pulled 1 row(s) from the central hub.
+```
+
+The sticky client hit the dead fallback, warned naming both endpoints, wrapped to the
+recovered primary and converged **in the same sweep tick** — proving the rotation and
+that central-a serves the same shared store. `site-a-b`'s 15:16:55Z sweep failed over but
+still failed as a whole (stale gRPC subchannel view of the restarted central-a — the same
+bounded post-restart artifact the previous gate recorded) and converged unaided on the
+next tick (15:17:25Z); decrypt-verified. So worst-case post-outage convergence remains
+**two** sweep intervals.
+
+**No flapping**: after central-b returned (15:17:06Z) no further hub warnings appeared on
+either site node until the deliberate SQL outage in check 5.
+
+### Check 3 — delete-while-follower-offline · PASS
+
+This was the untested residual from scadaproj#4 (the previous gate's residual 5).
+
+- 15:17:58Z — `site-a-b` stopped, holding **live** `shared-gate-1` (rev=0).
+- 15:18:29Z — `shared-gate-1` deleted centrally → SQL row `revision=1, is_deleted=1,
+  updated_utc=deleted_utc=2026-08-07T15:18:29.3213100+00:00`. `shared-gate-2`/`2b` left
+  live as controls.
+- 15:18:55Z — **`site-a-a` (the online follower) converged the tombstone** (≤26 s):
+  `rev=1 del=True`, identical timestamp; `secret get` → `{"error":"not-found"}`.
+- 15:19:13Z — `site-a-b` restarted; its `SyncOnStartup` sweep **in the boot second**
+  pulled the tombstone: `rev=1 del=True`, identical timestamp, `get` → not-found. Its
+  stale live copy did **not** resurrect — and structurally cannot push anywhere (no write
+  RPC on the wire).
+- Re-verified ≥10 sweeps later (15:25Z final-state table below): still tombstoned on
+  both followers, controls still live.
+
+### Check 4 — Layer-A expander on SQL · PASS
+
+Designed as a three-way discrimination so "the expanded value landed from the SHARED
+store" is proven by **value**, not by boot success:
+
+- Shared SQL store: `expander-gate-pepper` = a **5-character** value.
+- central-a's **stale residual SQLite** store: same name, a **41-character decoy** value.
+- central-a env (temporary probe): `ScadaBridge__InboundApi__ApiKeyPepper` =
+  `${secret:expander-gate-pepper}`. The unexpanded literal is 30 characters.
+
+`StartupValidator` (which runs immediately **after** the Layer-A expander) requires the
+pepper to be ≥16 characters. So of the three possible sources, **only the shared SQL
+store's value fails validation**. central-a was recreated at 15:20:36Z and its boot
+failed with exactly:
+
+```
+Unhandled exception. System.InvalidOperationException: Configuration validation failed:
+  - ScadaBridge:InboundApi:ApiKeyPepper is required and must be at least 16 characters for Central (backs the inbound API-key peppered-HMAC verifier)
+```
+
+A boot reading the SQLite decoy or leaving the token unexpanded would have passed. The
+expander therefore resolved the reference **from the shared SQL store** — the store the
+running node serves — not from the stale local one (which is the exact divergence the
+store-swap exists to prevent, sitting right there on disk as a live decoy).
+
+Positive half: the store row was then rotated to a valid ≥16-char value (the same literal
+central-b carries, preserving pair pepper parity) and central-a restarted → clean boot
+15:20:54Z, `Now listening` on 5000/8083, `/health/ready` 200. Probe reverted, both probe
+rows tombstoned/removed. Together with check 5(a), this **discharges the offline-test
+residual for the `Program.cs` SQL expander path** — both its resolve path and its
+fail-closed guard ran live.
+
+Seeding was via CLI → shared SQL; check 1 proved both centrals serve that store
+byte-identically, so this is equivalent to the plan's "seed via central-b".
+
+### Check 5 — fail-closed negatives · PASS
+
+**(a) Blank connection string.** central-a recreated with
+`Secrets__SqlServer__ConnectionString: ""` (still Central + `Mode=Grpc`). Boot refused
+with the designed message, thrown from the **expander block** (`Program.cs` line 84 →
+`SecretsRegistration.EnsureCentralSharedStoreConnectionString`), before any
+`SqlConnection` could bury it:
+
+```
+Unhandled exception. System.InvalidOperationException: Secrets:SqlServer:ConnectionString is empty, but this node is Central with Secrets:Replication:Mode=Grpc. Central in Grpc mode requires the SHARED SQL-Server secret store — an independent local store per central node is the divergence scadaproj#4 recorded (a hub failover would 'succeed' against an empty manifest). Supply the connection string via appsettings or the environment (Secrets__SqlServer__ConnectionString).
+```
+
+It names the key, explains why, and echoes no configured value. Probe reverted; node
+healthy again. (The refusal-to-boot criterion PASSED; *how* the refused process then
+lingers is defect 2 below.)
+
+**(b) SQL Server stopped.** `scadabridge-mssql` stopped 15:22:06Z (accepted transient for
+central's core DBs and the sibling env2 rig; both recovered).
+
+- Authenticated `GetManifest` against central-a → **gRPC error**
+  (`Code: Unknown / Exception was thrown by handler`) — **the hub throws; it never
+  serves an empty manifest.** No credential or connection-string material in the error.
+- Site sweeps failed loudly, trying **both** hubs (both depend on the one store):
+
+  ```
+  site-a-a [15:22:40 WRN] Secrets hub endpoint http://scadabridge-central-b:8083 did not answer (Unknown); failing over to http://scadabridge-central-a:8083.
+  site-a-a [15:22:54 WRN] Secret hub sync failed; the node continues serving its local store and will retry on the next interval.
+  ```
+
+- Last-known-good held: `secret get shared-gate-2` on `site-a-a` returned the correct
+  plaintext mid-outage.
+- SQL restarted 15:23:10Z → the hub answered the full 4-entry manifest immediately, no
+  node restarts. Recovery made positively observable: `shared-gate-final` seeded 15:24:17Z
+  converged on `site-a-a` in **8 s** and `site-a-b` in **26 s**, decrypt-verified on both.
+
+### Final state — three-way parity
+
+At 15:25Z the shared SQL store and both site stores held the identical five rows
+(name / revision / tombstone / `updated_utc` all equal):
+
+```
+expander-gate-pepper rev=2 del=1 updated=2026-08-07T15:21:26.9497970+00:00
+shared-gate-1        rev=1 del=1 updated=2026-08-07T15:18:29.3213100+00:00
+shared-gate-2        rev=0 del=0 updated=2026-08-07T15:15:07.5685210+00:00
+shared-gate-2b       rev=0 del=0 updated=2026-08-07T15:16:35.0265220+00:00
+shared-gate-final    rev=0 del=0 updated=2026-08-07T15:24:18.1206230+00:00
+```
+
+(Site stores additionally keep the previous gate's `hub-gate-smoke-*` residue — pull-only
+never deletes rows absent from the manifest, by design.)
+
+**Log hygiene — zero hits, fleet-wide.** All 8 nodes' `docker logs` and every on-disk
+Serilog file under `docker/*/logs/` grepped for: the dev bearer token, the dev KEK, all
+four smoke plaintexts, the decoy pepper value, **and the SQL dev password** (notably: the
+migrator crash stack in defect 1 did NOT leak the connection string):
+
+```
+central-a/b, site-a-a/b, site-b-a/b, site-c-a/b: token=0 kek=0 p1..p4=0 decoy=0 sqlpw=0
+on-disk log files: 0 files matched, for each of the 8 search strings
+```
+
+Default-OFF pin: `site-b-a`/`site-c-a` show zero `Secrets__*` env vars and zero hub log
+lines.
+
+## Defects found
+
+**None against any check's criterion.** Two real defects surfaced around the checks —
+per the gate's standing rule, **documented and NOT patched**; both need a controller
+decision.
+
+### 1. Library — `SqlServerSecretsStoreMigrator` concurrent first-boot race (error 2714)
+
+`ZB.MOM.WW.Secrets.Replicator.SqlServer` (present since 0.2.0, including 0.5.0). The
+class doc claims *"Idempotent: safe to run on every node at every startup, including
+concurrently"* and that Serializable + `IF NOT EXISTS` guards make simultaneous provision
+of a virgin database converge. Live, the ordinary `docker compose up` recreated both
+central nodes at once, both ran the boot migrator against the empty `ZbSecretsHub`,
+central-b won, and central-a died at 15:04:55Z with:
+
+```
+Unhandled exception. Microsoft.Data.SqlClient.SqlException (0x80131904): There is already an object named 'zbsecrets' in the database.
+CREATE SCHEMA failed due to previous errors.
+   at ZB.MOM.WW.Secrets.Replicator.SqlServer.SqlServerSecretsStoreMigrator.MigrateOnceAsync(...)
+Error Number:2714,State:6,Class:16
+```
+
+The retry loop in `MigrateAsync` catches **only** deadlock victims (error 1205,
+`MaxDeadlockRetries=5`); the object-exists race surfaces as **2714**, which propagates
+and aborts the boot. Window: virgin database + N nodes booting together — which is the
+NORMAL first deploy of a central pair, exactly the situation the class comment says it
+handles. Once the schema exists the migrator is genuinely idempotent (central-a's manual
+restart at 15:09:59Z and every recreate since booted clean), so the blast radius is
+one-time-per-virgin-DB — but it lands on the very first production bring-up, and combined
+with defect 2 it left the node wedged rather than restart-recovered.
+
+**Suggested direction (for the controller, not applied):** include 2714 (and arguably
+2617/1913-class "already exists" errors) in the retry filter, or catch-and-recheck —
+retrying is exactly as correct for this race as for the deadlock it already retries.
+
+### 2. ScadaBridge Host — pre-Serilog unhandled boot exception wedges the process instead of exiting
+
+The plan and the compose comments document the pre-Serilog failure window as: container
+"exits with a bare stderr trace and restarts". **Observed twice, it does not exit.** After
+printing the crash banner the main thread spins at ~100% CPU indefinitely; the container
+stays `running` (`RestartCount=0`), `restart: unless-stopped` never fires, and the node
+serves nothing. Reproduced on **both** pre-host failure shapes this gate produced:
+
+- the defect-1 `SqlException` from the expander's migrator run (spun for ~5 minutes,
+  main thread confirmed as the busy thread, until manually restarted);
+- the check-4 `StartupValidator` `InvalidOperationException` (same signature: banner
+  printed, 100% CPU, still `running`).
+
+So it is not specific to SqlClient or to the new SQL path — any throw in the pre-host
+window (including the pre-existing SQLite path's failure modes) turns a clean fail-fast
+into a silent wedge that needs a manual restart, and monitoring that watches container
+state sees a healthy-looking `running` container. Mechanism not root-caused here (gate
+scope); the observable and two reproductions are recorded. Worth a dedicated issue —
+fail-closed is only honest if the process actually exits.
+
+### Gate-method notes (not defects)
+
+- docker compose interpolates `${…}` in YAML itself: a literal `${secret:…}` env value in
+  the compose file must be written `$${secret:…}` or the service silently keeps its old
+  env (compose warns `invalid interpolation format`). Caught before any result was
+  recorded; the check-4 probe used the escaped form and the applied env was verified with
+  `docker inspect`.
+- `docker logs --since` windows overlap pre-restart lines; every "converged after X"
+  assertion above was re-anchored on timestamps, not on grep hits alone.
+
+## Residuals
+
+1. **Defects 1 and 2 above** — filed for controller disposition; neither blocks the
+   topology (defect 1 is one-time-per-virgin-DB and recoverable by restart; defect 2 is
+   pre-existing behavior of the pre-host window, now with two live reproductions).
+2. **`site-a-b`'s one-tick sweep failure after a central restart** (stale subchannel) —
+   same bounded, self-healing artifact the previous gate recorded; worst-case post-outage
+   convergence is two sweep intervals.
+3. **The `/admin/secrets` write path was again not exercised** — same coverage note as
+   the previous gate; ScadaBridge#22 covers the page itself.
+4. **Replication stays default-OFF in the product.** Enabled on this rig only (4 of 8
+   nodes), dev KEK/token/credentials. Production enablement needs a real KEK out of band,
+   a real hub token, a real SQL credential — and, per the plan's closeout, `rewrap-all`
+   on ScadaBridge now runs **once against central's shared SQL store** plus both nodes of
+   every site pair.
+5. **Central secret resolution now depends on SQL Server availability** — the accepted
+   trade from the decision record, observed live in check 5(b): during the outage the hub
+   throws (sites keep last-known-good), and a central node cannot *boot* through the
+   Layer-A expander while SQL is down (container-restart-retryable — subject to defect 2).
+
+## Conclusion
+
+The shared-store topology does what the design says: there is **one** copy of every
+central secret, both hubs serve it byte-identically, site-side failover between the
+central pair is safe *because* of that (proven in both directions, including the
+recovered-primary wrap in a single sweep tick), a delete issued while a follower is down
+lands as a tombstone on its very first sweep back with no resurrection, central's
+pre-host `${secret:}` expander provably reads the shared store rather than the stale
+local file sitting next to it, and every failure mode observed failed **closed and
+loud** — designed refusal messages naming keys, hub errors instead of empty manifests,
+sweep warnings instead of silent stalls — with zero credential material in any log on
+any of the eight nodes. **5/5.** scadaproj#4's divergence is gone by construction; the
+two defects the gate flushed out are recorded above for disposition.