feat(secrets): central's Grpc-mode store is the SHARED SQL-Server store (scadaproj#4)

In Secrets:Replication:Mode=Grpc a central node's ISecretStore is now the
shared SQL-Server store (AddZbSecretsSqlServerStore) instead of a per-node
local SQLite store. Both central hubs read and write ONE copy of every row,
so they serve identical manifests by construction — the 2026-08-07 live gate
observed central-b answering an authenticated GetManifest with an EMPTY
manifest while central-a held every secret, which would turn site-side hub
failover into a silent convergence stop.

- SecretsRegistration: two fail-closed pre-checks before any registration on
  the central+Grpc path — a blank Secrets:SqlServer:ConnectionString throws
  naming the key (an independent store per central node is the recorded
  defect), and a value containing ${secret: throws naming the bootstrap
  circularity (the expander needs this store to resolve references). Site
  registrations are byte-identical to before; SqlServer mode and
  replication-off are untouched.
- Program.cs Layer-A expander follows the store swap: central+Grpc with a
  non-blank connection string migrates and resolves pre-host ${secret:}
  references through the shared SQL store, so expanded values can never
  diverge from what the running node serves. Every other case keeps the
  SQLite path unchanged; blank-connstr central deliberately falls through so
  the clear AddScadaBridgeSecrets message is the one that fails the boot.
- appsettings.json: Secrets:SqlServer _comment now documents the Grpc-mode
  central requirement (literal/env value only, sites leave it empty).
- SecretsReplicationWiringTests: +5 pins (shared store resolves, blank and
  ${secret:} connstrings fail naming the key, sites-have-no-SqlServer-types
  descriptor sweep), central fixtures carry the now-required connstr.

Full suite green (7,474 passed across 30 projects, 0 warnings); the two
failures are pre-existing and unrelated: the Playwright live-rig suite fails
identically on unmodified main (cluster not running), and
GrpcCentralTransportTests.DeadlineExceeded_IsNotRetriedOnThePeer is a timing
flake that passes 3/3 in isolation and 470/470 on the first run of this code.

Claude-Session: https://claude.ai/code/session_014WNM4vjoVksyyBraTXSZE1

src/ZB.MOM.WW.ScadaBridge.Host/Program.cs

@@ -32,6 +32,8 @@ using ZB.MOM.WW.ScadaBridge.Transport;
 using ZB.MOM.WW.Secrets.Abstractions;
 using ZB.MOM.WW.Secrets.Configuration;
 using ZB.MOM.WW.Secrets.DependencyInjection;
+using ZB.MOM.WW.Secrets.Replicator.SqlServer;
+using ZB.MOM.WW.Secrets.Replicator.SqlServer.DependencyInjection;
 using ZB.MOM.WW.Secrets.Sqlite;
 using ZB.MOM.WW.Secrets.Ui;
 using ZB.MOM.WW.Telemetry;
@@ -52,13 +54,46 @@ var configuration = new ConfigurationBuilder()
 
 // Expand ${secret:...} config references before any validator/binder sees them (Layer A).
 // Throwaway provider — disposed here, shares no singletons with the host container.
+//
+// The expander must read the SAME store the running node will serve. On a central node in Grpc
+// replication mode that store is the SHARED SQL-Server store (see SecretsRegistration,
+// scadaproj#4) — an expander left on SQLite there would resolve pre-host ${secret:} references
+// from a stale/empty local store, silently diverging from what the node's own hub serves. Every
+// other case (sites, SqlServer mode, replication off) keeps the local SQLite path exactly as it
+// always was. Central+Grpc with a BLANK connection string deliberately falls through to the
+// SQLite path too: that boot is about to fail in AddScadaBridgeSecrets with the message naming
+// Secrets:SqlServer:ConnectionString, so the throw is not duplicated here.
+var expanderUsesSharedSqlStore =
+    string.Equals(
+        configuration["ScadaBridge:Node:Role"], "Central", StringComparison.OrdinalIgnoreCase)
+    && SecretsRegistration.UsesGrpcHub(configuration)
+    && !string.IsNullOrWhiteSpace(configuration[SecretsRegistration.HubConnectionStringKey]);
+
+var expanderServices = new ServiceCollection();
+if (expanderUsesSharedSqlStore)
+{
+    expanderServices.AddZbSecretsSqlServerStore(configuration, "Secrets");
+}
+else
+{
+    expanderServices.AddZbSecrets(configuration, "Secrets");
+}
+
 #pragma warning disable ASP0000 // deliberate throwaway container
-await using (var secretsProvider = new ServiceCollection()
-    .AddZbSecrets(configuration, "Secrets")
-    .BuildServiceProvider())
+await using (var secretsProvider = expanderServices.BuildServiceProvider())
 #pragma warning restore ASP0000
 {
-    await secretsProvider.GetRequiredService<SqliteSecretsStoreMigrator>().MigrateAsync(default);
+    if (expanderUsesSharedSqlStore)
+    {
+        await secretsProvider.GetRequiredService<SqlServerSecretsStoreMigrator>()
+            .MigrateAsync(default);
+    }
+    else
+    {
+        await secretsProvider.GetRequiredService<SqliteSecretsStoreMigrator>()
+            .MigrateAsync(default);
+    }
+
     var resolver = secretsProvider.GetRequiredService<ISecretResolver>();
     await new SecretReferenceExpander(resolver)
         .ExpandConfigurationAsync(configuration, default);

src/ZB.MOM.WW.ScadaBridge.Host/SecretsRegistration.cs

@@ -47,9 +47,10 @@ public enum SecretsReplicationMode
 /// <remarks>
 /// <para>
 /// Clustered replication is <b>opt-in and off by default</b>. ScadaBridge is hub-and-spoke — one
-/// central cluster plus N separate site clusters — and every supported topology gives each node a
-/// LOCAL store, so a site keeps resolving secrets straight through a WAN outage. What differs
-/// between the two modes is only how that local store is kept converged.
+/// central cluster plus N separate site clusters — and every supported topology gives each SITE
+/// node a LOCAL store, so a site keeps resolving secrets straight through a WAN outage. What
+/// differs between the modes is how convergence works, and — in gRPC mode — where central's own
+/// store lives.
 /// </para>
 /// <para>
 /// <b><see cref="SecretsReplicationMode.Grpc"/> is the production topology</b> (scadaproj#3):
@@ -58,7 +59,17 @@ public enum SecretsReplicationMode
 /// string to central's database, which violates ScadaBridge's standing rule that sites talk to
 /// central, not to central's DB. Pull-only is a property of the wire contract rather than a
 /// setting: the proto carries no write RPC, so a site cannot push even if misconfigured, and
-/// secrets therefore originate at central only.
+/// secrets therefore originate at central only. <b>Central's own store in this mode is the SHARED
+/// SQL-Server store</b> (<c>AddZbSecretsSqlServerStore</c>), not local SQLite: both central nodes
+/// read and write ONE copy of every row, so the two hub instances serve identical manifests by
+/// construction. The alternative — an independent local store per central node — is the recorded
+/// defect (scadaproj#4): one central answering authenticated followers with an empty manifest,
+/// turning site-side hub failover into a silent convergence stop. The accepted trade is that
+/// central secret resolution now depends on SQL Server availability — acceptable because central
+/// is already SQL-coupled to its core, and a SQL outage makes the hub <em>throw</em> (sites keep
+/// last-known-good), never serve empty. The pre-host <c>${secret:}</c> expander in
+/// <c>Program.cs</c> follows this store swap, so pre-host references resolve from the same store
+/// the running node serves.
 /// </para>
 /// <para>
 /// <b>The SQL-Server gate is not a style preference.</b> <c>AddZbSecretsSqlServerReplication</c>
@@ -73,19 +84,23 @@ public enum SecretsReplicationMode
 /// <b>The gRPC mode deliberately does NOT degrade the same way.</b> A missing connection string in
 /// SQL-Server mode falls back to a local-only store with a loud warning, and that behaviour is
 /// preserved untouched. gRPC mode instead fails at startup when its endpoint or bearer token is
-/// missing, because the fallback is the exact outcome the hub exists to prevent — a site serving
-/// secrets that silently never converge with central — and because selecting the mode takes an
-/// explicit second key, so an operator who typed it meant it.
+/// missing — or, on central, when the shared-store connection string is — because the fallback is
+/// the exact outcome the hub exists to prevent — a site serving secrets that silently never
+/// converge with central — and because selecting the mode takes an explicit second key, so an
+/// operator who typed it meant it.
 /// </para>
 /// <para>
 /// <b>Bootstrap constraint.</b> A replication credential can never itself come from the replicated
-/// set — a node cannot read the hub to learn how to reach the hub. The SQL hub connection string
-/// must arrive from outside it: an environment variable, or a <c>${secret:}</c> reference seeded in
-/// that node's own LOCAL store (the pre-host expander in <c>Program.cs</c> runs against a plain
-/// local SQLite store before the host container exists, so such a reference does resolve). The
+/// set — a node cannot read the hub to learn how to reach the hub. In SQL-Server mode the hub
+/// connection string must arrive from outside it: an environment variable, or a <c>${secret:}</c>
+/// reference seeded in that node's own LOCAL store (the pre-host expander in <c>Program.cs</c>
+/// runs against the local SQLite store before the host container exists, so such a reference does
+/// resolve). In gRPC mode the rule is stricter on central: <c>Secrets:SqlServer:ConnectionString</c>
+/// can never be a <c>${secret:}</c> reference at all, because on central the pre-host expander
+/// itself runs against the shared SQL store (see <c>Program.cs</c>) and would need this very
+/// string to reach the store that resolves it — registration rejects such a value outright. The
 /// same holds for <c>Secrets:GrpcHub:BearerToken</c>, which is supplied from appsettings or the
-/// environment exactly as the mesh pre-shared keys are. That expander is deliberately left
-/// un-replicated for exactly this reason.
+/// environment exactly as the mesh pre-shared keys are.
 /// </para>
 /// </remarks>
 public static class SecretsRegistration
@@ -173,7 +188,9 @@ public static class SecretsRegistration
     /// <summary>
     /// Registers the host container's secret store: a plain local SQLite store by default, or a
     /// replicating topology when <see cref="ReplicationEnabledKey"/> is set — a shared SQL-Server
-    /// hub, or the pull-only gRPC hub whose half is chosen by <paramref name="role"/>.
+    /// hub, or the pull-only gRPC hub whose half is chosen by <paramref name="role"/>. In gRPC
+    /// mode a central node's store is the SHARED SQL-Server store, and registration fails closed
+    /// when <see cref="HubConnectionStringKey"/> is blank or is a <c>${secret:}</c> reference.
     /// </summary>
     /// <param name="services">The service collection to register into.</param>
     /// <param name="config">Application configuration for options binding.</param>
@@ -192,14 +209,57 @@ public static class SecretsRegistration
         // disagree about whether the hub is on.
         if (UsesGrpcHub(config))
         {
-            // Neither gRPC extension registers the local store — pull-only means local writes
-            // never leave the node, so there is nothing to decorate and the store stays exactly as
-            // configured. Register it first; both halves resolve it.
-            services.AddZbSecrets(config, SecretsSectionPath);
-
             switch (role)
             {
                 case SecretsNodeRole.Central:
+                    // Both pre-checks run BEFORE any registration so the boot failure names the
+                    // real problem instead of surfacing later as a package validation message or a
+                    // malformed-connection-string fault on first use.
+                    var hubConnectionString = config[HubConnectionStringKey];
+
+                    if (string.IsNullOrWhiteSpace(hubConnectionString))
+                    {
+                        // Fail closed, matching the mode's philosophy: central in Grpc mode
+                        // REQUIRES the shared SQL store. An independent local store per central
+                        // node is exactly the divergence scadaproj#4 recorded — one central hub
+                        // answered authenticated followers with an EMPTY manifest, so a hub
+                        // failover would "succeed" against nothing and stop convergence silently.
+                        throw new InvalidOperationException(
+                            $"{HubConnectionStringKey} is empty, but this node is Central with "
+                                + $"{ReplicationModeKey}=Grpc. Central in Grpc mode requires the "
+                                + "SHARED SQL-Server secret store — an independent local store per "
+                                + "central node is the divergence scadaproj#4 recorded (a hub "
+                                + "failover would 'succeed' against an empty manifest). Supply the "
+                                + "connection string via appsettings or the environment "
+                                + "(Secrets__SqlServer__ConnectionString).");
+                    }
+
+                    if (hubConnectionString.Contains("${secret:", StringComparison.Ordinal))
+                    {
+                        // The secrets store's own connection string can never be a secret
+                        // reference: the pre-host expander needs the store this string points at
+                        // in order to resolve it — bootstrap circularity, the same rule the class
+                        // remarks document for the hub bearer token. The message deliberately
+                        // does not echo the configured value: a mistyped value here could be a
+                        // pasted real credential.
+                        throw new InvalidOperationException(
+                            $"{HubConnectionStringKey} contains a ${{secret:}} reference. The "
+                                + "secret store's own connection string can never be a secret "
+                                + "reference — the pre-host expander needs the store this string "
+                                + "points at in order to resolve it (bootstrap circularity, the "
+                                + "same rule as Secrets:GrpcHub:BearerToken). Supply a literal "
+                                + "value via appsettings or the environment "
+                                + "(Secrets__SqlServer__ConnectionString).");
+                    }
+
+                    // Central's store is the SHARED SQL-Server store: registers
+                    // SqlServerSecretStore as ISecretStore BEFORE calling AddZbSecrets internally,
+                    // displacing SQLite — do NOT also call AddZbSecrets here. One copy of every
+                    // row means both central hubs serve identical manifests by construction, which
+                    // is what makes site-side hub failover between them safe.
+                    services.AddZbSecretsSqlServerStore(
+                        config, SecretsSectionPath, SqlServerSectionPath);
+
                     // Throws when Secrets:GrpcHub:BearerToken is unset. Deliberately not caught: a
                     // hub that starts and refuses every follower looks like a network fault from
                     // the site end, and would be discovered as a stale secret rather than as a
@@ -207,6 +267,13 @@ public static class SecretsRegistration
                     services.AddZbSecretsGrpcHub(config, GrpcHubSectionPath);
                     break;
                 case SecretsNodeRole.Site:
+                    // A site keeps its LOCAL SQLite store — it resolves secrets straight through a
+                    // WAN outage, and sites talk to central, never to central's database. Neither
+                    // gRPC extension registers the local store — pull-only means local writes
+                    // never leave the node, so there is nothing to decorate and the store stays
+                    // exactly as configured. Register it first; the sweep resolves it.
+                    services.AddZbSecrets(config, SecretsSectionPath);
+
                     // Throws when Secrets:GrpcHub:Endpoint or :BearerToken is unset. Same reasoning.
                     services.AddZbSecretsGrpcHubClient(config, GrpcHubSectionPath);
                     break;

src/ZB.MOM.WW.ScadaBridge.Host/appsettings.json

@@ -24,7 +24,7 @@
       "SchemaName": "zbsecrets",
       "SyncInterval": "00:00:30",
       "SyncOnStartup": true,
-      "_comment": "Hub connstr must be seeded in the LOCAL store (or supplied via env) - it cannot come from the hub itself. Enable via Secrets:Replication:Enabled + Mode=SqlServer."
+      "_comment": "Two modes read this section. Mode=SqlServer: the hub connstr must be seeded in the LOCAL store (or supplied via env) - it cannot come from the hub itself. Mode=Grpc: REQUIRED on CENTRAL, where it points BOTH central nodes at ONE shared SQL-Server secret store so the pair cannot diverge (scadaproj#4 - an independent store per central node let one hub serve an authenticated EMPTY manifest); it must be a LITERAL or environment value (Secrets__SqlServer__ConnectionString), never a ${secret:} reference - the pre-host expander needs this store to resolve references, so the reference could never resolve (bootstrap circularity; registration rejects it). SITES leave it empty in every mode - sites talk to central, never to central's database."
     },
     "GrpcHub": {
       "_comment": "Read ONLY when Secrets:Replication:Enabled is true AND Mode=Grpc; inert otherwise. ONE section, both halves: CENTRAL reads BearerToken + MaxNamesPerRequest and hosts the hub on its CentralGrpcPort h2c listener (default 8083, alongside CentralControlService); a SITE reads Endpoint + BearerToken + the sweep timings and pulls. Replication is pull-only by wire contract - the proto has no write RPC - so secrets originate at central and a site cannot push. FAIL-CLOSED: unlike SqlServer mode there is no local-only fallback; a missing BearerToken (either role) or Endpoint (site) is a startup failure naming the key.",