test(secrets): central-shared-store live gate 5/5 — shared SQL store + hub failover proven on the rig

Rig config: central pair gains Secrets__SqlServer__ConnectionString (dedicated
ZbSecretsHub database on the existing scadabridge-mssql, dev credentials); site-a
pair gains Secrets__GrpcHub__FallbackEndpoints__0 = central-b. Gate doc records
5/5 PASS (parity-by-construction, both-direction failover incl. recovered-primary
wrap, delete-while-follower-offline with no resurrection, Layer-A expander
provably reading the shared store via a stale-SQLite decoy discrimination, and
fail-closed negatives), discharging the Program.cs SQL-expander offline-test
residual, plus two defects documented NOT patched: the SqlServer migrator's
concurrent virgin-DB CREATE SCHEMA race (error 2714 not in the retry filter) and
the Host's pre-Serilog crash path wedging at 100% CPU instead of exiting.

Claude-Session: https://claude.ai/code/session_014WNM4vjoVksyyBraTXSZE1

docker/docker-compose.yml

@@ -33,14 +33,29 @@ x-secrets-hub-env: &secrets-hub-env
   # hub and every follower; an unset token is a startup failure on both halves.
   Secrets__GrpcHub__BearerToken: "secrets-hub-docker-dev-token"
 
-# Site half of the same section. NOTE the asymmetry with
-# ScadaBridge:Communication:CentralGrpcEndpoints, which is a LIST that fails over across
-# the central pair: the hub client dials a SINGLE endpoint, so a sweep against a stopped
-# central-a stalls rather than failing over to central-b. That is survivable — the sweep
-# is best-effort and the node keeps serving its full local last-known-good store — but
-# secrets stop converging until central-a returns.
+# Central-only half: the SHARED SQL-Server secret store (scadaproj#4, Secrets 0.5.0).
+# In Grpc mode BOTH central nodes read and write ONE copy of every row in this database,
+# so the two hub instances serve identical manifests by construction — that is what makes
+# the site-side FallbackEndpoints below safe. ZbSecretsHub is a dedicated database on the
+# rig's existing scadabridge-mssql container (default zbsecrets schema, created by the
+# boot-time SqlServerSecretsStoreMigrator; the database itself + the scadabridge_app grant
+# were provisioned once via sqlcmd — see docs/plans/2026-08-07-secrets-central-shared-store-live-gate.md).
+# Same DEV-ONLY credentials as the ScadaBridge__Database__* strings above. Must be a
+# LITERAL value, never a ${secret:} reference — the pre-host expander needs this string to
+# reach the store that would resolve it (bootstrap circularity; registration rejects it).
+# Site nodes must NEVER carry this key: sites talk to central, not to central's database.
+x-secrets-hub-central-env: &secrets-hub-central-env
+  Secrets__SqlServer__ConnectionString: "Server=scadabridge-mssql,1433;Database=ZbSecretsHub;User Id=scadabridge_app;Password=ScadaBridge_Dev1#;TrustServerCertificate=true"
+
+# Site half of the same section. The hub client dials Endpoint first and fails over, per
+# call, to FallbackEndpoints in order (sticky on whichever answered last, Secrets 0.5.0).
+# Listing central-b is safe ONLY because both centrals serve the one shared SQL store
+# above — never list endpoints backed by independent stores: failing over to an emptier
+# hub is a silent convergence stop, the exact defect (scadaproj#4) the shared store
+# exists to prevent.
 x-secrets-hub-site-env: &secrets-hub-site-env
   Secrets__GrpcHub__Endpoint: "http://scadabridge-central-a:8083"
+  Secrets__GrpcHub__FallbackEndpoints__0: "http://scadabridge-central-b:8083"
 
 services:
   central-a:
@@ -51,8 +66,9 @@ services:
     stop_grace_period: 30s
     container_name: scadabridge-central-a
     environment:
-      # Hub half of the pull-only gRPC secrets hub (anchor at the top of this file).
-      <<: *secrets-hub-env
+      # Hub half of the pull-only gRPC secrets hub + the central-only shared SQL store
+      # (anchors at the top of this file).
+      <<: [*secrets-hub-env, *secrets-hub-central-env]
       SCADABRIDGE_CONFIG: Central
       ASPNETCORE_ENVIRONMENT: Development
       ASPNETCORE_URLS: "http://+:5000"
@@ -88,10 +104,10 @@ services:
       - "9013:8083"   # gRPC control plane (CentralControlService, T1A.2)
     volumes:
       - ./central-node-a/appsettings.Central.json:/app/appsettings.Central.json:ro
-      # Added for the gRPC secrets hub: the node's local secret store lives at
-      # Secrets__SqlitePath=/app/data/scadabridge-secrets.db, so it needs the same
-      # per-node volume the site nodes already have. Without it the store sits in the
-      # image's writable layer and is destroyed by every container recreate.
+      # Originally added for the gRPC secrets hub's local SQLite store; since the
+      # central store moved to the shared SQL-Server database (Secrets 0.5.0,
+      # scadaproj#4) the scadabridge-secrets.db here is a pre-0.5.0 residue, but the
+      # volume is still needed (inbound-api-keys.sqlite lives on it).
       - ./central-node-a/data:/app/data
       - ./central-node-a/logs:/app/logs
     networks:
@@ -106,8 +122,9 @@ services:
     stop_grace_period: 30s
     container_name: scadabridge-central-b
     environment:
-      # Hub half of the pull-only gRPC secrets hub (anchor at the top of this file).
-      <<: *secrets-hub-env
+      # Hub half of the pull-only gRPC secrets hub + the central-only shared SQL store
+      # (anchors at the top of this file).
+      <<: [*secrets-hub-env, *secrets-hub-central-env]
       SCADABRIDGE_CONFIG: Central
       ASPNETCORE_ENVIRONMENT: Development
       ASPNETCORE_URLS: "http://+:5000"
@@ -143,10 +160,10 @@ services:
       - "9014:8083"   # gRPC control plane (CentralControlService, T1A.2)
     volumes:
       - ./central-node-b/appsettings.Central.json:/app/appsettings.Central.json:ro
-      # Added for the gRPC secrets hub: the node's local secret store lives at
-      # Secrets__SqlitePath=/app/data/scadabridge-secrets.db, so it needs the same
-      # per-node volume the site nodes already have. Without it the store sits in the
-      # image's writable layer and is destroyed by every container recreate.
+      # Originally added for the gRPC secrets hub's local SQLite store; since the
+      # central store moved to the shared SQL-Server database (Secrets 0.5.0,
+      # scadaproj#4) the scadabridge-secrets.db here is a pre-0.5.0 residue, but the
+      # volume is still needed (inbound-api-keys.sqlite lives on it).
       - ./central-node-b/data:/app/data
       - ./central-node-b/logs:/app/logs
     networks: