ScadaBridge/docs/plans/2026-08-07-secrets-central-shared-store-live-gate.md

Secrets central shared SQL store + hub failover — live gate on the docker cluster

Result: PASS — 5/5 checks, plus the 0.5.1 re-drill. Run 2026-08-07 on the local 8-node docker cluster in two passes: the first pass (secrets libs 0.5.0, image b41249cc7bf8) passed all five checks and surfaced two defects, documented, not patched (standing rule) — one in the ZB.MOM.WW.Secrets.Replicator.SqlServer library (concurrent first-boot migration race), one in the ScadaBridge Host's pre-Serilog crash path (unhandled boot exception wedges the process instead of exiting). The library defect was fixed at the library and shipped as 0.5.1 (scadaproj main 31ca940); the second pass (libs 0.5.1, image 4b0c36398cb7) re-ran the exact virgin-DB concurrent bring-up drill that exposed it — both centrals booted clean in the same second — and re-smoked end-to-end convergence. The Host wedge (defect 2) remains open, to be filed as its own ScadaBridge issue at closeout. See Defects.

Purpose

Terminal gate (Task 5) for the central-shared-SQL-store program (scadaproj/docs/plans/2026-08-07-secrets-central-shared-sql-store.md, resolving scadaproj#4): in Secrets:Replication:Mode=Grpc, both central nodes' ISecretStore is now the shared SQL-Server store (AddZbSecretsSqlServerStore), so the two hub instances serve identical manifests by construction, and the site-side hub client gains ordered Secrets:GrpcHub:FallbackEndpoints with sticky preference (Secrets family 0.5.0).

This gate also discharges the code-review residual that the Program.cs SQL expander path has no offline test: check 4 exercises the expanderUsesSharedSqlStore branch positively (Layer-A ${secret:} resolution from the shared SQL store, including the SqlServerSecretsStoreMigrator boot run), and check 5(a) exercises its fail-closed negative (EnsureCentralSharedStoreConnectionString throwing the designed message from Program.cs line 84 — the expander block — before any SqlConnection sees the value).

Scope: rig config only. No product code was changed by this gate.

Rig state

Fact	First pass (checks 1–5)	Second pass (0.5.1 re-drill)
Repo / branch	ScadaBridge @ main	same
Git SHA gated	68f812eaa4b29b0657ef2d967b1fa0dfc105c214 (68f812ea)	68f812ea + the 0.5.1 pin bump (committed with this amendment)
Image	scadabridge:latest = b41249cc7bf8, built 15:04:42Z	4b0c36398cb7, built 15:36:41Z
Secrets libs	ZB.MOM.WW.Secrets* 0.5.0 (Grpc dll informational version 0.5.0+7a2980e, FailoverSecretsHubReader present)	0.5.1 — verified in-container: 0.5.1+e62b4a5… (SourceRevisionId is a pack-before-commit stamp; the fix is provably in the binary — IsRetryableMigrationError present, the old MaxDeadlockRetries gone)

First-pass binary identity:

Fact	Value
ZB.MOM.WW.ScadaBridge.Host.dll	9dba0869ff4dfa7c… — byte-identical on all 4 participating nodes
ZB.MOM.WW.Secrets.Replicator.Grpc.dll	4c837f627e901641… — byte-identical on all 4
Hub enabled on	central-a, central-b (hub, shared SQL store) and site-a-a, site-a-b (followers, local SQLite)
Hub NOT enabled on	site-b-*, site-c-* — verified: zero Secrets__* env vars, zero hub/sweep log lines
Sweep interval	00:00:30 — the product default, not shortened; timings below are real
KEK	the same shared committed dev-only key as the 2026-08-07 hub gate (kek_id sha256 7451bcbc1f1f on every row observed)
Working tree at build	only docker/docker-compose.yml modified (rig config, not in the image) + untracked .claude/

Rig DB choice (record)

Central's shared store is a dedicated database ZbSecretsHub on the rig's existing scadabridge-mssql container (the same SQL Server ScadaBridge central already uses), default zbsecrets schema, tables created by the boot-time SqlServerSecretsStoreMigrator (schema_version = 1, secret). The database itself and the scadabridge_app user grant (db_owner on that DB only) were provisioned once via sqlcmd in the SQL container. The connection string uses the rig's existing committed dev credentials (User Id=scadabridge_app, password as already committed in the compose/infra files — redacted here), TrustServerCertificate=true.

Rig-config change made by this gate (committed)

docker/docker-compose.yml:

• New anchor x-secrets-hub-central-env → Secrets__SqlServer__ConnectionString pointing at scadabridge-mssql,1433 / ZbSecretsHub. Applied to central-a and central-b only (<<: [*secrets-hub-env, *secrets-hub-central-env]). Site nodes must never carry this key — sites talk to central, not to central's database.
• x-secrets-hub-site-env gains Secrets__GrpcHub__FallbackEndpoints__0: "http://scadabridge-central-b:8083" (site-a pair only), with the same-store-only warning in the comment.
• The stale central-volume comments (SQLite as central's secret store) updated: the scadabridge-secrets.db under central-node-*/data is now a pre-0.5.0 residue; the volume survives for inbound-api-keys.sqlite.

All values remain DEV-ONLY under the same committed-rig-credential exception as before.

Method

Central-side seeding/deleting used a throwaway helper console (scratchpad, not committed) that composes exactly what central's Layer-A expander composes — AddZbSecretsSqlServerStore + SqlServerSecretsStoreMigrator + the CLI's own SecretCommands — against ZbSecretsHub over TCP with the rig dev KEK. This is the plan's "CLI → shared SQL" seeding path; with one shared store, a row written by any client is on exactly the path a row written via central-b would be (proven bidirectionally by check 1's byte-identical hub reads). The /admin/secrets page was again not used — covered separately by ScadaBridge#22.

Site-store assertions honored the gate-method rule from last time: every SQLite access ran in a throwaway container on the guest kernel (mcr.microsoft.com/dotnet/aspnet:10.0

• the published secret CLI, mounting the node's data volume). SQL Server, being client-server, has no virtiofs coherence problem, so host-side access to the shared store is safe.

gRPC calls were made with fullstorydev/grpcurl on scadabridge-net against the committed secrets_hub.proto. No token, KEK, secret plaintext, or connection-string password appears in this document; ciphertext identity is a SHA-256 over the encrypted columns only.

Results

#	Check	Result	Timing
1	Central-pair parity by construction	PASS	byte-identical hub reads; 1 SQL row
2	Failover (FallbackEndpoints, sticky)	PASS	converged 18 s with central-a down; reverse failover converged same tick
3	Delete-while-follower-offline	PASS	tombstone on returning node's first sweep; no resurrection
4	Layer-A expander on SQL	PASS	value provably landed from the shared store
5	Fail-closed negatives	PASS	designed boot refusal; hub throws on SQL outage, never serves empty

Baseline before seeding: ZbSecretsHub..secret empty; site-a stores held the previous gate's three hub-gate-smoke-* rows (pull-only never deletes — they persist as last-known-good residue); central-a's residual SQLite held the same three; central-b's residual SQLite was empty — the literal scadaproj#4 divergence, still on disk, which check 4 turns into a decoy.

Check 1 — central-pair parity by construction · PASS

Seeded shared-gate-1 into the shared store at 15:10:53Z. Then, authenticated:

• GetManifest on both scadabridge-central-a:8083 and scadabridge-central-b:8083 returned the identical single entry (shared-gate-1, updatedUtc=2026-08-07T15:10:53.8689010+00:00).
• GetSecrets full-row responses from the two hubs are byte-identical (sha256 ec1f270c9404471d… for both), cipher_sha256=40e5a8df9faa8e189c9f0dc6, kek sha256 7451bcbc1f1f.
• SELECT COUNT(*) FROM zbsecrets.secret → 1. There is one copy of the row; "same HLC/ciphertext" is not a replication outcome to verify but an identity.

The 2026-08-07 hub gate's empty-manifest-from-central-b defect is unreproducible by construction: central-b has no store of its own to be empty. Both site nodes converged the row in 2 s (seed 15:10:53Z → pulled 1 row(s) 15:10:55Z).

Check 2 — failover · PASS

The failover machinery was exercised in both directions, plus once unplanned:

Boot window (unplanned, evidence kept). During bring-up central-a was wedged (defect 1) and central-b not yet serving. Both site nodes logged the 0.5.0 failover Warning — naming both endpoint URIs and nothing else — then the sweep-level failure:

site-a-a [15:04:55 WRN] Secrets hub endpoint http://scadabridge-central-a:8083 did not answer (Unavailable); failing over to http://scadabridge-central-b:8083.
site-a-a [15:05:25 WRN] Secrets hub endpoint http://scadabridge-central-a:8083 did not answer (Unavailable); failing over to http://scadabridge-central-b:8083.

At 15:05:25 central-b answered → both clients went sticky on central-b.

Primary down (the check proper). central-a stopped 15:11:43Z. No new warning appeared — correct, and worth understanding: the clients were already sticky on central-b, so the primary's outage cost zero failed calls, which is the documented sticky rationale working as designed. shared-gate-2 was seeded at 15:15:07Z with central-a down; both site nodes converged it at 15:15:25Z (18 s) and decrypted it to the exact seeded plaintext (in-container secret get).

Reverse drill (rotation wrap + recovered-primary reuse). central-a restarted healthy (15:16:15Z), then central-b stopped (15:16:24Z) and shared-gate-2b seeded (15:16:35Z):

site-a-a [15:16:55 WRN] Secrets hub endpoint http://scadabridge-central-b:8083 did not answer (Unavailable); failing over to http://scadabridge-central-a:8083.
site-a-a [15:16:55 INF] Secret hub sync converged: pulled 1 row(s) from the central hub.

The sticky client hit the dead fallback, warned naming both endpoints, wrapped to the recovered primary and converged in the same sweep tick — proving the rotation and that central-a serves the same shared store. site-a-b's 15:16:55Z sweep failed over but still failed as a whole (stale gRPC subchannel view of the restarted central-a — the same bounded post-restart artifact the previous gate recorded) and converged unaided on the next tick (15:17:25Z); decrypt-verified. So worst-case post-outage convergence remains two sweep intervals.

No flapping: after central-b returned (15:17:06Z) no further hub warnings appeared on either site node until the deliberate SQL outage in check 5.

Check 3 — delete-while-follower-offline · PASS

This was the untested residual from scadaproj#4 (the previous gate's residual 5).

• 15:17:58Z — site-a-b stopped, holding live shared-gate-1 (rev=0).
• 15:18:29Z — shared-gate-1 deleted centrally → SQL row revision=1, is_deleted=1, updated_utc=deleted_utc=2026-08-07T15:18:29.3213100+00:00. shared-gate-2/2b left live as controls.
• 15:18:55Z — site-a-a (the online follower) converged the tombstone (≤26 s): rev=1 del=True, identical timestamp; secret get → {"error":"not-found"}.
• 15:19:13Z — site-a-b restarted; its SyncOnStartup sweep in the boot second pulled the tombstone: rev=1 del=True, identical timestamp, get → not-found. Its stale live copy did not resurrect — and structurally cannot push anywhere (no write RPC on the wire).
• Re-verified ≥10 sweeps later (15:25Z final-state table below): still tombstoned on both followers, controls still live.

Check 4 — Layer-A expander on SQL · PASS

Designed as a three-way discrimination so "the expanded value landed from the SHARED store" is proven by value, not by boot success:

• Shared SQL store: expander-gate-pepper = a 5-character value.
• central-a's stale residual SQLite store: same name, a 41-character decoy value.
• central-a env (temporary probe): ScadaBridge__InboundApi__ApiKeyPepper = ${secret:expander-gate-pepper}. The unexpanded literal is 30 characters.

StartupValidator (which runs immediately after the Layer-A expander) requires the pepper to be ≥16 characters. So of the three possible sources, only the shared SQL store's value fails validation. central-a was recreated at 15:20:36Z and its boot failed with exactly:

Unhandled exception. System.InvalidOperationException: Configuration validation failed:
  - ScadaBridge:InboundApi:ApiKeyPepper is required and must be at least 16 characters for Central (backs the inbound API-key peppered-HMAC verifier)

A boot reading the SQLite decoy or leaving the token unexpanded would have passed. The expander therefore resolved the reference from the shared SQL store — the store the running node serves — not from the stale local one (which is the exact divergence the store-swap exists to prevent, sitting right there on disk as a live decoy).

Positive half: the store row was then rotated to a valid ≥16-char value (the same literal central-b carries, preserving pair pepper parity) and central-a restarted → clean boot 15:20:54Z, Now listening on 5000/8083, /health/ready 200. Probe reverted, both probe rows tombstoned/removed. Together with check 5(a), this discharges the offline-test residual for the Program.cs SQL expander path — both its resolve path and its fail-closed guard ran live.

Seeding was via CLI → shared SQL; check 1 proved both centrals serve that store byte-identically, so this is equivalent to the plan's "seed via central-b".

Check 5 — fail-closed negatives · PASS

(a) Blank connection string. central-a recreated with Secrets__SqlServer__ConnectionString: "" (still Central + Mode=Grpc). Boot refused with the designed message, thrown from the expander block (Program.cs line 84 → SecretsRegistration.EnsureCentralSharedStoreConnectionString), before any SqlConnection could bury it:

Unhandled exception. System.InvalidOperationException: Secrets:SqlServer:ConnectionString is empty, but this node is Central with Secrets:Replication:Mode=Grpc. Central in Grpc mode requires the SHARED SQL-Server secret store — an independent local store per central node is the divergence scadaproj#4 recorded (a hub failover would 'succeed' against an empty manifest). Supply the connection string via appsettings or the environment (Secrets__SqlServer__ConnectionString).

It names the key, explains why, and echoes no configured value. Probe reverted; node healthy again. (The refusal-to-boot criterion PASSED; how the refused process then lingers is defect 2 below.)

(b) SQL Server stopped. scadabridge-mssql stopped 15:22:06Z (accepted transient for central's core DBs and the sibling env2 rig; both recovered).

• Authenticated GetManifest against central-a → gRPC error (Code: Unknown / Exception was thrown by handler) — the hub throws; it never serves an empty manifest. No credential or connection-string material in the error.

• Site sweeps failed loudly, trying both hubs (both depend on the one store):

  site-a-a [15:22:40 WRN] Secrets hub endpoint http://scadabridge-central-b:8083 did not answer (Unknown); failing over to http://scadabridge-central-a:8083.
  site-a-a [15:22:54 WRN] Secret hub sync failed; the node continues serving its local store and will retry on the next interval.
  

• Last-known-good held: secret get shared-gate-2 on site-a-a returned the correct plaintext mid-outage.

• SQL restarted 15:23:10Z → the hub answered the full 4-entry manifest immediately, no node restarts. Recovery made positively observable: shared-gate-final seeded 15:24:17Z converged on site-a-a in 8 s and site-a-b in 26 s, decrypt-verified on both.

Final state — three-way parity

At 15:25Z the shared SQL store and both site stores held the identical five rows (name / revision / tombstone / updated_utc all equal):

expander-gate-pepper rev=2 del=1 updated=2026-08-07T15:21:26.9497970+00:00
shared-gate-1        rev=1 del=1 updated=2026-08-07T15:18:29.3213100+00:00
shared-gate-2        rev=0 del=0 updated=2026-08-07T15:15:07.5685210+00:00
shared-gate-2b       rev=0 del=0 updated=2026-08-07T15:16:35.0265220+00:00
shared-gate-final    rev=0 del=0 updated=2026-08-07T15:24:18.1206230+00:00

(Site stores additionally keep the previous gate's hub-gate-smoke-* residue — pull-only never deletes rows absent from the manifest, by design.)

Log hygiene — zero hits, fleet-wide. All 8 nodes' docker logs and every on-disk Serilog file under docker/*/logs/ grepped for: the dev bearer token, the dev KEK, all four smoke plaintexts, the decoy pepper value, and the SQL dev password (notably: the migrator crash stack in defect 1 did NOT leak the connection string):

central-a/b, site-a-a/b, site-b-a/b, site-c-a/b: token=0 kek=0 p1..p4=0 decoy=0 sqlpw=0
on-disk log files: 0 files matched, for each of the 8 search strings

Default-OFF pin: site-b-a/site-c-a show zero Secrets__* env vars and zero hub log lines.

Defects found

None against any check's criterion. Two real defects surfaced around the checks — per the gate's standing rule, documented and not patched by the gate itself. Defect 1 has since been fixed at the library (0.5.1) and re-drilled on this rig; defect 2 remains open pending its own ScadaBridge issue.

1. Library — SqlServerSecretsStoreMigrator concurrent first-boot race (error 2714)

ZB.MOM.WW.Secrets.Replicator.SqlServer (present since 0.2.0, including 0.5.0). The class doc claims "Idempotent: safe to run on every node at every startup, including concurrently" and that Serializable + IF NOT EXISTS guards make simultaneous provision of a virgin database converge. Live, the ordinary docker compose up recreated both central nodes at once, both ran the boot migrator against the empty ZbSecretsHub, central-b won, and central-a died at 15:04:55Z with:

Unhandled exception. Microsoft.Data.SqlClient.SqlException (0x80131904): There is already an object named 'zbsecrets' in the database.
CREATE SCHEMA failed due to previous errors.
   at ZB.MOM.WW.Secrets.Replicator.SqlServer.SqlServerSecretsStoreMigrator.MigrateOnceAsync(...)
Error Number:2714,State:6,Class:16

The retry loop in MigrateAsync catches only deadlock victims (error 1205, MaxDeadlockRetries=5); the object-exists race surfaces as 2714, which propagates and aborts the boot. Window: virgin database + N nodes booting together — which is the NORMAL first deploy of a central pair, exactly the situation the class comment says it handles. Once the schema exists the migrator is genuinely idempotent (central-a's manual restart at 15:09:59Z and every recreate since booted clean), so the blast radius is one-time-per-virgin-DB — but it lands on the very first production bring-up, and combined with defect 2 it left the node wedged rather than restart-recovered.

Disposition: FIXED in 0.5.1 (scadaproj main 31ca940, released to the Gitea feed the same day). The retry filter now retries 1205/2714/1913/2627, scanning the whole SqlException.Errors collection (2714 arrives bundled with 2759 "CREATE SCHEMA failed due to previous errors"), capped at 5 attempts, and the class's "safe … concurrently" doc overclaim is corrected. ScadaBridge bumped all five ZB.MOM.WW.Secrets* pins to 0.5.1 and the rig was rebuilt (image 4b0c36398cb7; in-container ZB.MOM.WW.Secrets.Replicator.SqlServer.dll verified to carry IsRetryableMigrationError and not the old MaxDeadlockRetries).

Re-drill on 0.5.1 — PASS. The exact trigger was reproduced: both centrals stopped, ZbSecretsHub dropped and recreated empty (0 tables, same scadabridge_app grant), then both centrals started in the same docker start invocation (15:37:41Z):

central-a [15:37:42 INF] Application started. Press Ctrl+C to shut down.
central-b [15:37:42 INF] Application started. Press Ctrl+C to shut down.
zbsecrets.schema_version: version=1 applied_utc=2026-08-07T15:37:42.7099417+00:00
central-a /health/ready=200  central-b /health/ready=200  RestartCount=0 both, CPU ~4–5%

Both nodes booted in the same second against the virgin database, the schema was provisioned exactly once, and there was no 2714, no crash, no wedge — where the 0.5.0 first pass crashed central-a at 15:04:55Z under identical conditions. Honest bound: the migrator runs pre-Serilog with no logger, so a fired retry is not itself visible in logs; the evidence is the trigger conditions reproduced plus the outcome flipped. Convergence re-smoked end-to-end on the 0.5.1 image: shared-gate-051 seeded into the (freshly wiped) shared store at 15:38:11Z, pulled by site-a-a at 15:38:24Z (13 s) and decrypt-verified. (The DB drop wiped the first pass's rows from the hub; site stores keep their local copies — pull-only never deletes — so the final-state table above reflects the first pass.)

2. ScadaBridge Host — pre-Serilog unhandled boot exception wedges the process instead of exiting

The plan and the compose comments document the pre-Serilog failure window as: container "exits with a bare stderr trace and restarts". Observed twice, it does not exit. After printing the crash banner the main thread spins at ~100% CPU indefinitely; the container stays running (RestartCount=0), restart: unless-stopped never fires, and the node serves nothing. Reproduced on both pre-host failure shapes this gate produced:

• the defect-1 SqlException from the expander's migrator run (spun for ~5 minutes, main thread confirmed as the busy thread, until manually restarted);
• the check-4 StartupValidator InvalidOperationException (same signature: banner printed, 100% CPU, still running).

So it is not specific to SqlClient or to the new SQL path — any throw in the pre-host window (including the pre-existing SQLite path's failure modes) turns a clean fail-fast into a silent wedge that needs a manual restart, and monitoring that watches container state sees a healthy-looking running container. Mechanism not root-caused here (gate scope); the observable and two reproductions are recorded.

Disposition: OPEN — to be filed as its own ScadaBridge issue at closeout (controller decision 2026-08-07). Fail-closed is only honest if the process actually exits.

Gate-method notes (not defects)

• docker compose interpolates ${…} in YAML itself: a literal ${secret:…} env value in the compose file must be written $${secret:…} or the service silently keeps its old env (compose warns invalid interpolation format). Caught before any result was recorded; the check-4 probe used the escaped form and the applied env was verified with docker inspect.
• docker logs --since windows overlap pre-restart lines; every "converged after X" assertion above was re-anchored on timestamps, not on grep hits alone.

Residuals

1. Defect 2 above (pre-Serilog wedge) — open, pending its own ScadaBridge issue; pre-existing behavior of the pre-host window, now with two live reproductions. Defect 1 is closed: fixed in 0.5.1 and re-drilled on this rig (see its disposition).
2. site-a-b's one-tick sweep failure after a central restart (stale subchannel) — same bounded, self-healing artifact the previous gate recorded; worst-case post-outage convergence is two sweep intervals.
3. The /admin/secrets write path was again not exercised — same coverage note as the previous gate; ScadaBridge#22 covers the page itself.
4. Replication stays default-OFF in the product. Enabled on this rig only (4 of 8 nodes), dev KEK/token/credentials. Production enablement needs a real KEK out of band, a real hub token, a real SQL credential — and, per the plan's closeout, rewrap-all on ScadaBridge now runs once against central's shared SQL store plus both nodes of every site pair.
5. Central secret resolution now depends on SQL Server availability — the accepted trade from the decision record, observed live in check 5(b): during the outage the hub throws (sites keep last-known-good), and a central node cannot boot through the Layer-A expander while SQL is down (container-restart-retryable — subject to defect 2).

Conclusion

The shared-store topology does what the design says: there is one copy of every central secret, both hubs serve it byte-identically, site-side failover between the central pair is safe because of that (proven in both directions, including the recovered-primary wrap in a single sweep tick), a delete issued while a follower is down lands as a tombstone on its very first sweep back with no resurrection, central's pre-host ${secret:} expander provably reads the shared store rather than the stale local file sitting next to it, and every failure mode observed failed closed and loud — designed refusal messages naming keys, hub errors instead of empty manifests, sweep warnings instead of silent stalls — with zero credential material in any log on any of the eight nodes. 5/5. scadaproj#4's divergence is gone by construction; the two defects the gate flushed out are recorded above for disposition.