diff --git a/docker/docker-compose.yml b/docker/docker-compose.yml index a8f63d13..b1c17a85 100644 --- a/docker/docker-compose.yml +++ b/docker/docker-compose.yml @@ -33,14 +33,29 @@ x-secrets-hub-env: &secrets-hub-env # hub and every follower; an unset token is a startup failure on both halves. Secrets__GrpcHub__BearerToken: "secrets-hub-docker-dev-token" -# Site half of the same section. NOTE the asymmetry with -# ScadaBridge:Communication:CentralGrpcEndpoints, which is a LIST that fails over across -# the central pair: the hub client dials a SINGLE endpoint, so a sweep against a stopped -# central-a stalls rather than failing over to central-b. That is survivable — the sweep -# is best-effort and the node keeps serving its full local last-known-good store — but -# secrets stop converging until central-a returns. +# Central-only half: the SHARED SQL-Server secret store (scadaproj#4, Secrets 0.5.0). +# In Grpc mode BOTH central nodes read and write ONE copy of every row in this database, +# so the two hub instances serve identical manifests by construction — that is what makes +# the site-side FallbackEndpoints below safe. ZbSecretsHub is a dedicated database on the +# rig's existing scadabridge-mssql container (default zbsecrets schema, created by the +# boot-time SqlServerSecretsStoreMigrator; the database itself + the scadabridge_app grant +# were provisioned once via sqlcmd — see docs/plans/2026-08-07-secrets-central-shared-store-live-gate.md). +# Same DEV-ONLY credentials as the ScadaBridge__Database__* strings above. Must be a +# LITERAL value, never a ${secret:} reference — the pre-host expander needs this string to +# reach the store that would resolve it (bootstrap circularity; registration rejects it). +# Site nodes must NEVER carry this key: sites talk to central, not to central's database. +x-secrets-hub-central-env: &secrets-hub-central-env + Secrets__SqlServer__ConnectionString: "Server=scadabridge-mssql,1433;Database=ZbSecretsHub;User Id=scadabridge_app;Password=ScadaBridge_Dev1#;TrustServerCertificate=true" + +# Site half of the same section. The hub client dials Endpoint first and fails over, per +# call, to FallbackEndpoints in order (sticky on whichever answered last, Secrets 0.5.0). +# Listing central-b is safe ONLY because both centrals serve the one shared SQL store +# above — never list endpoints backed by independent stores: failing over to an emptier +# hub is a silent convergence stop, the exact defect (scadaproj#4) the shared store +# exists to prevent. x-secrets-hub-site-env: &secrets-hub-site-env Secrets__GrpcHub__Endpoint: "http://scadabridge-central-a:8083" + Secrets__GrpcHub__FallbackEndpoints__0: "http://scadabridge-central-b:8083" services: central-a: @@ -51,8 +66,9 @@ services: stop_grace_period: 30s container_name: scadabridge-central-a environment: - # Hub half of the pull-only gRPC secrets hub (anchor at the top of this file). - <<: *secrets-hub-env + # Hub half of the pull-only gRPC secrets hub + the central-only shared SQL store + # (anchors at the top of this file). + <<: [*secrets-hub-env, *secrets-hub-central-env] SCADABRIDGE_CONFIG: Central ASPNETCORE_ENVIRONMENT: Development ASPNETCORE_URLS: "http://+:5000" @@ -88,10 +104,10 @@ services: - "9013:8083" # gRPC control plane (CentralControlService, T1A.2) volumes: - ./central-node-a/appsettings.Central.json:/app/appsettings.Central.json:ro - # Added for the gRPC secrets hub: the node's local secret store lives at - # Secrets__SqlitePath=/app/data/scadabridge-secrets.db, so it needs the same - # per-node volume the site nodes already have. Without it the store sits in the - # image's writable layer and is destroyed by every container recreate. + # Originally added for the gRPC secrets hub's local SQLite store; since the + # central store moved to the shared SQL-Server database (Secrets 0.5.0, + # scadaproj#4) the scadabridge-secrets.db here is a pre-0.5.0 residue, but the + # volume is still needed (inbound-api-keys.sqlite lives on it). - ./central-node-a/data:/app/data - ./central-node-a/logs:/app/logs networks: @@ -106,8 +122,9 @@ services: stop_grace_period: 30s container_name: scadabridge-central-b environment: - # Hub half of the pull-only gRPC secrets hub (anchor at the top of this file). - <<: *secrets-hub-env + # Hub half of the pull-only gRPC secrets hub + the central-only shared SQL store + # (anchors at the top of this file). + <<: [*secrets-hub-env, *secrets-hub-central-env] SCADABRIDGE_CONFIG: Central ASPNETCORE_ENVIRONMENT: Development ASPNETCORE_URLS: "http://+:5000" @@ -143,10 +160,10 @@ services: - "9014:8083" # gRPC control plane (CentralControlService, T1A.2) volumes: - ./central-node-b/appsettings.Central.json:/app/appsettings.Central.json:ro - # Added for the gRPC secrets hub: the node's local secret store lives at - # Secrets__SqlitePath=/app/data/scadabridge-secrets.db, so it needs the same - # per-node volume the site nodes already have. Without it the store sits in the - # image's writable layer and is destroyed by every container recreate. + # Originally added for the gRPC secrets hub's local SQLite store; since the + # central store moved to the shared SQL-Server database (Secrets 0.5.0, + # scadaproj#4) the scadabridge-secrets.db here is a pre-0.5.0 residue, but the + # volume is still needed (inbound-api-keys.sqlite lives on it). - ./central-node-b/data:/app/data - ./central-node-b/logs:/app/logs networks: diff --git a/docs/plans/2026-08-07-secrets-central-shared-store-live-gate.md b/docs/plans/2026-08-07-secrets-central-shared-store-live-gate.md new file mode 100644 index 00000000..e76b2fbe --- /dev/null +++ b/docs/plans/2026-08-07-secrets-central-shared-store-live-gate.md @@ -0,0 +1,382 @@ +# Secrets central shared SQL store + hub failover — live gate on the docker cluster + +**Result: PASS — 5/5 checks.** Run 2026-08-07 on the local 8-node docker cluster. +Two defects were found and are **documented, not patched** (standing rule): one in the +`ZB.MOM.WW.Secrets.Replicator.SqlServer` library (concurrent first-boot migration race), +one in the ScadaBridge Host's pre-Serilog crash path (unhandled boot exception wedges the +process instead of exiting). Neither is a failure of a gate check — every check's own +criterion passed — but both need a controller decision. See [Defects](#defects-found). + +## Purpose + +Terminal gate (Task 5) for the central-shared-SQL-store program +(`scadaproj/docs/plans/2026-08-07-secrets-central-shared-sql-store.md`, resolving +[scadaproj#4](https://gitea.dohertylan.com/dohertj2/scadaproj/issues/4)): in +`Secrets:Replication:Mode=Grpc`, both central nodes' `ISecretStore` is now the **shared +SQL-Server store** (`AddZbSecretsSqlServerStore`), so the two hub instances serve identical +manifests **by construction**, and the site-side hub client gains ordered +`Secrets:GrpcHub:FallbackEndpoints` with sticky preference (Secrets family **0.5.0**). + +This gate also **discharges the code-review residual that the `Program.cs` SQL expander +path has no offline test**: check 4 exercises the `expanderUsesSharedSqlStore` branch +positively (Layer-A `${secret:}` resolution from the shared SQL store, including the +`SqlServerSecretsStoreMigrator` boot run), and check 5(a) exercises its fail-closed +negative (`EnsureCentralSharedStoreConnectionString` throwing the designed message from +`Program.cs` line 84 — the expander block — before any `SqlConnection` sees the value). + +**Scope: rig config only.** No product code was changed by this gate. + +## Rig state + +| Fact | Value | +|---|---| +| Repo / branch | `ScadaBridge` @ `main` | +| Git SHA gated | `68f812eaa4b29b0657ef2d967b1fa0dfc105c214` (`68f812ea`) | +| Image | `scadabridge:latest` = `b41249cc7bf8`, built 15:04:42Z | +| `ZB.MOM.WW.ScadaBridge.Host.dll` | `9dba0869ff4dfa7c…` — byte-identical on all 4 participating nodes | +| `ZB.MOM.WW.Secrets.Replicator.Grpc.dll` | `4c837f627e901641…`, informational version **`0.5.0+7a2980e`**, `FailoverSecretsHubReader` present — byte-identical on all 4 | +| Secrets libs | `ZB.MOM.WW.Secrets*` **0.5.0** | +| Hub enabled on | `central-a`, `central-b` (hub, **shared SQL store**) and `site-a-a`, `site-a-b` (followers, local SQLite) | +| Hub NOT enabled on | `site-b-*`, `site-c-*` — verified: zero `Secrets__*` env vars, zero hub/sweep log lines | +| Sweep interval | `00:00:30` — the product default, not shortened; timings below are real | +| KEK | the same shared committed **dev-only** key as the 2026-08-07 hub gate (`kek_id` sha256 `7451bcbc1f1f` on every row observed) | +| Working tree at build | only `docker/docker-compose.yml` modified (rig config, not in the image) + untracked `.claude/` | + +### Rig DB choice (record) + +Central's shared store is a **dedicated database `ZbSecretsHub`** on the rig's existing +`scadabridge-mssql` container (the same SQL Server ScadaBridge central already uses), +default `zbsecrets` schema, tables created by the boot-time `SqlServerSecretsStoreMigrator` +(`schema_version` = 1, `secret`). The database itself and the `scadabridge_app` user grant +(db_owner on that DB only) were provisioned once via `sqlcmd` in the SQL container. The +connection string uses the rig's **existing committed dev credentials** +(`User Id=scadabridge_app`, password as already committed in the compose/infra files — +redacted here), `TrustServerCertificate=true`. + +### Rig-config change made by this gate (committed) + +`docker/docker-compose.yml`: + +- New anchor `x-secrets-hub-central-env` → `Secrets__SqlServer__ConnectionString` + pointing at `scadabridge-mssql,1433 / ZbSecretsHub`. Applied to **central-a and + central-b only** (`<<: [*secrets-hub-env, *secrets-hub-central-env]`). Site nodes must + never carry this key — sites talk to central, not to central's database. +- `x-secrets-hub-site-env` gains + `Secrets__GrpcHub__FallbackEndpoints__0: "http://scadabridge-central-b:8083"` (site-a + pair only), with the same-store-only warning in the comment. +- The stale central-volume comments (SQLite as central's secret store) updated: the + `scadabridge-secrets.db` under `central-node-*/data` is now a pre-0.5.0 residue; the + volume survives for `inbound-api-keys.sqlite`. + +All values remain DEV-ONLY under the same committed-rig-credential exception as before. + +## Method + +Central-side seeding/deleting used a throwaway helper console (scratchpad, not committed) +that composes **exactly** what central's Layer-A expander composes — +`AddZbSecretsSqlServerStore` + `SqlServerSecretsStoreMigrator` + the CLI's own +`SecretCommands` — against `ZbSecretsHub` over TCP with the rig dev KEK. This is the +plan's "CLI → shared SQL" seeding path; with one shared store, a row written by any client +is on exactly the path a row written via central-b would be (proven bidirectionally by +check 1's byte-identical hub reads). The `/admin/secrets` page was again not used — +covered separately by ScadaBridge#22. + +Site-store assertions honored the gate-method rule from last time: **every** SQLite +access ran in a throwaway container on the guest kernel (`mcr.microsoft.com/dotnet/aspnet:10.0` ++ the published `secret` CLI, mounting the node's `data` volume). SQL Server, being +client-server, has no virtiofs coherence problem, so host-side access to the shared store +is safe. + +gRPC calls were made with `fullstorydev/grpcurl` on `scadabridge-net` against the +committed `secrets_hub.proto`. No token, KEK, secret plaintext, or connection-string +password appears in this document; ciphertext identity is a SHA-256 over the encrypted +columns only. + +## Results + +| # | Check | Result | Timing | +|---|---|---|---| +| 1 | Central-pair parity by construction | **PASS** | byte-identical hub reads; 1 SQL row | +| 2 | Failover (`FallbackEndpoints`, sticky) | **PASS** | converged 18 s with central-a down; reverse failover converged same tick | +| 3 | Delete-while-follower-offline | **PASS** | tombstone on returning node's first sweep; no resurrection | +| 4 | Layer-A expander on SQL | **PASS** | value provably landed from the shared store | +| 5 | Fail-closed negatives | **PASS** | designed boot refusal; hub throws on SQL outage, never serves empty | + +Baseline before seeding: `ZbSecretsHub..secret` empty; site-a stores held the previous +gate's three `hub-gate-smoke-*` rows (pull-only never deletes — they persist as +last-known-good residue); central-a's residual SQLite held the same three; +**central-b's residual SQLite was empty — the literal scadaproj#4 divergence, still on +disk**, which check 4 turns into a decoy. + +### Check 1 — central-pair parity by construction · PASS + +Seeded `shared-gate-1` into the shared store at 15:10:53Z. Then, authenticated: + +- `GetManifest` on **both** `scadabridge-central-a:8083` and `scadabridge-central-b:8083` + returned the identical single entry (`shared-gate-1`, + `updatedUtc=2026-08-07T15:10:53.8689010+00:00`). +- `GetSecrets` full-row responses from the two hubs are **byte-identical** + (sha256 `ec1f270c9404471d…` for both), `cipher_sha256=40e5a8df9faa8e189c9f0dc6`, + `kek sha256 7451bcbc1f1f`. +- `SELECT COUNT(*) FROM zbsecrets.secret` → **1**. There is one copy of the row; "same + HLC/ciphertext" is not a replication outcome to verify but an identity. + +The 2026-08-07 hub gate's empty-manifest-from-central-b defect is **unreproducible by +construction**: central-b has no store of its own to be empty. Both site nodes converged +the row in **2 s** (seed 15:10:53Z → `pulled 1 row(s)` 15:10:55Z). + +### Check 2 — failover · PASS + +The failover machinery was exercised in **both directions**, plus once unplanned: + +**Boot window (unplanned, evidence kept).** During bring-up central-a was wedged (defect 1) +and central-b not yet serving. Both site nodes logged the 0.5.0 failover Warning — naming +both endpoint URIs and nothing else — then the sweep-level failure: + +``` +site-a-a [15:04:55 WRN] Secrets hub endpoint http://scadabridge-central-a:8083 did not answer (Unavailable); failing over to http://scadabridge-central-b:8083. +site-a-a [15:05:25 WRN] Secrets hub endpoint http://scadabridge-central-a:8083 did not answer (Unavailable); failing over to http://scadabridge-central-b:8083. +``` + +At 15:05:25 central-b answered → both clients went **sticky on central-b**. + +**Primary down (the check proper).** central-a stopped 15:11:43Z. **No new warning +appeared — correct, and worth understanding:** the clients were already sticky on +central-b, so the primary's outage cost zero failed calls, which is the documented sticky +rationale working as designed. `shared-gate-2` was seeded at 15:15:07Z with central-a +down; **both** site nodes converged it at 15:15:25Z (**18 s**) and decrypted it to the +exact seeded plaintext (in-container `secret get`). + +**Reverse drill (rotation wrap + recovered-primary reuse).** central-a restarted healthy +(15:16:15Z), then central-b stopped (15:16:24Z) and `shared-gate-2b` seeded (15:16:35Z): + +``` +site-a-a [15:16:55 WRN] Secrets hub endpoint http://scadabridge-central-b:8083 did not answer (Unavailable); failing over to http://scadabridge-central-a:8083. +site-a-a [15:16:55 INF] Secret hub sync converged: pulled 1 row(s) from the central hub. +``` + +The sticky client hit the dead fallback, warned naming both endpoints, wrapped to the +recovered primary and converged **in the same sweep tick** — proving the rotation and +that central-a serves the same shared store. `site-a-b`'s 15:16:55Z sweep failed over but +still failed as a whole (stale gRPC subchannel view of the restarted central-a — the same +bounded post-restart artifact the previous gate recorded) and converged unaided on the +next tick (15:17:25Z); decrypt-verified. So worst-case post-outage convergence remains +**two** sweep intervals. + +**No flapping**: after central-b returned (15:17:06Z) no further hub warnings appeared on +either site node until the deliberate SQL outage in check 5. + +### Check 3 — delete-while-follower-offline · PASS + +This was the untested residual from scadaproj#4 (the previous gate's residual 5). + +- 15:17:58Z — `site-a-b` stopped, holding **live** `shared-gate-1` (rev=0). +- 15:18:29Z — `shared-gate-1` deleted centrally → SQL row `revision=1, is_deleted=1, + updated_utc=deleted_utc=2026-08-07T15:18:29.3213100+00:00`. `shared-gate-2`/`2b` left + live as controls. +- 15:18:55Z — **`site-a-a` (the online follower) converged the tombstone** (≤26 s): + `rev=1 del=True`, identical timestamp; `secret get` → `{"error":"not-found"}`. +- 15:19:13Z — `site-a-b` restarted; its `SyncOnStartup` sweep **in the boot second** + pulled the tombstone: `rev=1 del=True`, identical timestamp, `get` → not-found. Its + stale live copy did **not** resurrect — and structurally cannot push anywhere (no write + RPC on the wire). +- Re-verified ≥10 sweeps later (15:25Z final-state table below): still tombstoned on + both followers, controls still live. + +### Check 4 — Layer-A expander on SQL · PASS + +Designed as a three-way discrimination so "the expanded value landed from the SHARED +store" is proven by **value**, not by boot success: + +- Shared SQL store: `expander-gate-pepper` = a **5-character** value. +- central-a's **stale residual SQLite** store: same name, a **41-character decoy** value. +- central-a env (temporary probe): `ScadaBridge__InboundApi__ApiKeyPepper` = + `${secret:expander-gate-pepper}`. The unexpanded literal is 30 characters. + +`StartupValidator` (which runs immediately **after** the Layer-A expander) requires the +pepper to be ≥16 characters. So of the three possible sources, **only the shared SQL +store's value fails validation**. central-a was recreated at 15:20:36Z and its boot +failed with exactly: + +``` +Unhandled exception. System.InvalidOperationException: Configuration validation failed: + - ScadaBridge:InboundApi:ApiKeyPepper is required and must be at least 16 characters for Central (backs the inbound API-key peppered-HMAC verifier) +``` + +A boot reading the SQLite decoy or leaving the token unexpanded would have passed. The +expander therefore resolved the reference **from the shared SQL store** — the store the +running node serves — not from the stale local one (which is the exact divergence the +store-swap exists to prevent, sitting right there on disk as a live decoy). + +Positive half: the store row was then rotated to a valid ≥16-char value (the same literal +central-b carries, preserving pair pepper parity) and central-a restarted → clean boot +15:20:54Z, `Now listening` on 5000/8083, `/health/ready` 200. Probe reverted, both probe +rows tombstoned/removed. Together with check 5(a), this **discharges the offline-test +residual for the `Program.cs` SQL expander path** — both its resolve path and its +fail-closed guard ran live. + +Seeding was via CLI → shared SQL; check 1 proved both centrals serve that store +byte-identically, so this is equivalent to the plan's "seed via central-b". + +### Check 5 — fail-closed negatives · PASS + +**(a) Blank connection string.** central-a recreated with +`Secrets__SqlServer__ConnectionString: ""` (still Central + `Mode=Grpc`). Boot refused +with the designed message, thrown from the **expander block** (`Program.cs` line 84 → +`SecretsRegistration.EnsureCentralSharedStoreConnectionString`), before any +`SqlConnection` could bury it: + +``` +Unhandled exception. System.InvalidOperationException: Secrets:SqlServer:ConnectionString is empty, but this node is Central with Secrets:Replication:Mode=Grpc. Central in Grpc mode requires the SHARED SQL-Server secret store — an independent local store per central node is the divergence scadaproj#4 recorded (a hub failover would 'succeed' against an empty manifest). Supply the connection string via appsettings or the environment (Secrets__SqlServer__ConnectionString). +``` + +It names the key, explains why, and echoes no configured value. Probe reverted; node +healthy again. (The refusal-to-boot criterion PASSED; *how* the refused process then +lingers is defect 2 below.) + +**(b) SQL Server stopped.** `scadabridge-mssql` stopped 15:22:06Z (accepted transient for +central's core DBs and the sibling env2 rig; both recovered). + +- Authenticated `GetManifest` against central-a → **gRPC error** + (`Code: Unknown / Exception was thrown by handler`) — **the hub throws; it never + serves an empty manifest.** No credential or connection-string material in the error. +- Site sweeps failed loudly, trying **both** hubs (both depend on the one store): + + ``` + site-a-a [15:22:40 WRN] Secrets hub endpoint http://scadabridge-central-b:8083 did not answer (Unknown); failing over to http://scadabridge-central-a:8083. + site-a-a [15:22:54 WRN] Secret hub sync failed; the node continues serving its local store and will retry on the next interval. + ``` + +- Last-known-good held: `secret get shared-gate-2` on `site-a-a` returned the correct + plaintext mid-outage. +- SQL restarted 15:23:10Z → the hub answered the full 4-entry manifest immediately, no + node restarts. Recovery made positively observable: `shared-gate-final` seeded 15:24:17Z + converged on `site-a-a` in **8 s** and `site-a-b` in **26 s**, decrypt-verified on both. + +### Final state — three-way parity + +At 15:25Z the shared SQL store and both site stores held the identical five rows +(name / revision / tombstone / `updated_utc` all equal): + +``` +expander-gate-pepper rev=2 del=1 updated=2026-08-07T15:21:26.9497970+00:00 +shared-gate-1 rev=1 del=1 updated=2026-08-07T15:18:29.3213100+00:00 +shared-gate-2 rev=0 del=0 updated=2026-08-07T15:15:07.5685210+00:00 +shared-gate-2b rev=0 del=0 updated=2026-08-07T15:16:35.0265220+00:00 +shared-gate-final rev=0 del=0 updated=2026-08-07T15:24:18.1206230+00:00 +``` + +(Site stores additionally keep the previous gate's `hub-gate-smoke-*` residue — pull-only +never deletes rows absent from the manifest, by design.) + +**Log hygiene — zero hits, fleet-wide.** All 8 nodes' `docker logs` and every on-disk +Serilog file under `docker/*/logs/` grepped for: the dev bearer token, the dev KEK, all +four smoke plaintexts, the decoy pepper value, **and the SQL dev password** (notably: the +migrator crash stack in defect 1 did NOT leak the connection string): + +``` +central-a/b, site-a-a/b, site-b-a/b, site-c-a/b: token=0 kek=0 p1..p4=0 decoy=0 sqlpw=0 +on-disk log files: 0 files matched, for each of the 8 search strings +``` + +Default-OFF pin: `site-b-a`/`site-c-a` show zero `Secrets__*` env vars and zero hub log +lines. + +## Defects found + +**None against any check's criterion.** Two real defects surfaced around the checks — +per the gate's standing rule, **documented and NOT patched**; both need a controller +decision. + +### 1. Library — `SqlServerSecretsStoreMigrator` concurrent first-boot race (error 2714) + +`ZB.MOM.WW.Secrets.Replicator.SqlServer` (present since 0.2.0, including 0.5.0). The +class doc claims *"Idempotent: safe to run on every node at every startup, including +concurrently"* and that Serializable + `IF NOT EXISTS` guards make simultaneous provision +of a virgin database converge. Live, the ordinary `docker compose up` recreated both +central nodes at once, both ran the boot migrator against the empty `ZbSecretsHub`, +central-b won, and central-a died at 15:04:55Z with: + +``` +Unhandled exception. Microsoft.Data.SqlClient.SqlException (0x80131904): There is already an object named 'zbsecrets' in the database. +CREATE SCHEMA failed due to previous errors. + at ZB.MOM.WW.Secrets.Replicator.SqlServer.SqlServerSecretsStoreMigrator.MigrateOnceAsync(...) +Error Number:2714,State:6,Class:16 +``` + +The retry loop in `MigrateAsync` catches **only** deadlock victims (error 1205, +`MaxDeadlockRetries=5`); the object-exists race surfaces as **2714**, which propagates +and aborts the boot. Window: virgin database + N nodes booting together — which is the +NORMAL first deploy of a central pair, exactly the situation the class comment says it +handles. Once the schema exists the migrator is genuinely idempotent (central-a's manual +restart at 15:09:59Z and every recreate since booted clean), so the blast radius is +one-time-per-virgin-DB — but it lands on the very first production bring-up, and combined +with defect 2 it left the node wedged rather than restart-recovered. + +**Suggested direction (for the controller, not applied):** include 2714 (and arguably +2617/1913-class "already exists" errors) in the retry filter, or catch-and-recheck — +retrying is exactly as correct for this race as for the deadlock it already retries. + +### 2. ScadaBridge Host — pre-Serilog unhandled boot exception wedges the process instead of exiting + +The plan and the compose comments document the pre-Serilog failure window as: container +"exits with a bare stderr trace and restarts". **Observed twice, it does not exit.** After +printing the crash banner the main thread spins at ~100% CPU indefinitely; the container +stays `running` (`RestartCount=0`), `restart: unless-stopped` never fires, and the node +serves nothing. Reproduced on **both** pre-host failure shapes this gate produced: + +- the defect-1 `SqlException` from the expander's migrator run (spun for ~5 minutes, + main thread confirmed as the busy thread, until manually restarted); +- the check-4 `StartupValidator` `InvalidOperationException` (same signature: banner + printed, 100% CPU, still `running`). + +So it is not specific to SqlClient or to the new SQL path — any throw in the pre-host +window (including the pre-existing SQLite path's failure modes) turns a clean fail-fast +into a silent wedge that needs a manual restart, and monitoring that watches container +state sees a healthy-looking `running` container. Mechanism not root-caused here (gate +scope); the observable and two reproductions are recorded. Worth a dedicated issue — +fail-closed is only honest if the process actually exits. + +### Gate-method notes (not defects) + +- docker compose interpolates `${…}` in YAML itself: a literal `${secret:…}` env value in + the compose file must be written `$${secret:…}` or the service silently keeps its old + env (compose warns `invalid interpolation format`). Caught before any result was + recorded; the check-4 probe used the escaped form and the applied env was verified with + `docker inspect`. +- `docker logs --since` windows overlap pre-restart lines; every "converged after X" + assertion above was re-anchored on timestamps, not on grep hits alone. + +## Residuals + +1. **Defects 1 and 2 above** — filed for controller disposition; neither blocks the + topology (defect 1 is one-time-per-virgin-DB and recoverable by restart; defect 2 is + pre-existing behavior of the pre-host window, now with two live reproductions). +2. **`site-a-b`'s one-tick sweep failure after a central restart** (stale subchannel) — + same bounded, self-healing artifact the previous gate recorded; worst-case post-outage + convergence is two sweep intervals. +3. **The `/admin/secrets` write path was again not exercised** — same coverage note as + the previous gate; ScadaBridge#22 covers the page itself. +4. **Replication stays default-OFF in the product.** Enabled on this rig only (4 of 8 + nodes), dev KEK/token/credentials. Production enablement needs a real KEK out of band, + a real hub token, a real SQL credential — and, per the plan's closeout, `rewrap-all` + on ScadaBridge now runs **once against central's shared SQL store** plus both nodes of + every site pair. +5. **Central secret resolution now depends on SQL Server availability** — the accepted + trade from the decision record, observed live in check 5(b): during the outage the hub + throws (sites keep last-known-good), and a central node cannot *boot* through the + Layer-A expander while SQL is down (container-restart-retryable — subject to defect 2). + +## Conclusion + +The shared-store topology does what the design says: there is **one** copy of every +central secret, both hubs serve it byte-identically, site-side failover between the +central pair is safe *because* of that (proven in both directions, including the +recovered-primary wrap in a single sweep tick), a delete issued while a follower is down +lands as a tombstone on its very first sweep back with no resurrection, central's +pre-host `${secret:}` expander provably reads the shared store rather than the stale +local file sitting next to it, and every failure mode observed failed **closed and +loud** — designed refusal messages naming keys, hub errors instead of empty manifests, +sweep warnings instead of silent stalls — with zero credential material in any log on +any of the eight nodes. **5/5.** scadaproj#4's divergence is gone by construction; the +two defects the gate flushed out are recorded above for disposition.