fix(secrets): review round 1 — pre-host expander reads the same store the runtime serves, raw-string Replication:Enabled parse, leftover-key warning

- SecretsRegistration: UsesCentralSharedStore predicate shared by AddScadaBridgeSecrets and the
  new AddPreHostExpander, so a Central node's Layer-A ${secret:} expander composes against the
  shared SQL store whenever Secrets:Store:Provider=SqlServer does — closing the gap where a
  central migrated off the retired Grpc topology would boot-time-resolve config from per-node
  SQLite while serving the shared store.
- Program.cs: expander block now calls AddPreHostExpander + resolves the store-agnostic
  ISecretsStoreMigrator seam; stale Grpc-hub/replication comment rewritten.
- DeployedSecretsMirror.cs: stale "gRPC-hub follower half" doc comment rewritten to state the
  now-permanent local-SQLite-only invariant.
- StartupValidator: Secrets:Replication:Enabled read as a raw string (any non-empty, non-"false"
  value rejected) instead of GetValue<bool>, which threw a raw binder crash on "yes"/"1"; added a
  CollectWarnings entry for leftover Secrets:Replication/Secrets:GrpcHub child keys once Enabled
  is false/unset.
- Tests: +15 (AddPreHostExpander/UsesCentralSharedStore wiring, Replication:Enabled edge values,
  leftover-key warnings). Full solution build and full Host.Tests green (593/593).

src/ZB.MOM.WW.ScadaBridge.Host/Program.cs

@@ -33,7 +33,6 @@ using ZB.MOM.WW.ScadaBridge.Transport;
 using ZB.MOM.WW.Secrets.Abstractions;
 using ZB.MOM.WW.Secrets.Configuration;
 using ZB.MOM.WW.Secrets.DependencyInjection;
-using ZB.MOM.WW.Secrets.Sqlite;
 using ZB.MOM.WW.Secrets.Ui;
 using ZB.MOM.WW.Telemetry;
 using Serilog;
@@ -75,11 +74,35 @@ var configuration = new ConfigurationBuilder()
 // Expand ${secret:...} config references before any validator/binder sees them (Layer A).
 // Throwaway provider — disposed here, shares no singletons with the host container.
 //
-// This always reads the node's LOCAL SQLite store: Task 11 retired the gRPC secrets hub, the only
-// topology that ever put a central node's store somewhere else pre-host. (Secrets:Store:Provider
-// =SqlServer, the still-supported shared-central-store feature, is a post-host DI registration —
-// see SecretsRegistration — and does not change what the Layer-A expander itself reads.)
+// The expander MUST read the SAME store the running node will serve — SecretsRegistration.
+// UsesCentralSharedStore is the single predicate deciding that, shared with the runtime
+// registration in AddScadaBridgeSecrets, so the two can never disagree. On a Central node with
+// Secrets:Store:Provider=SqlServer that is the SHARED SQL-Server store (Task 5): an expander left
+// on SQLite there would resolve pre-host ${secret:} references (e.g. ScadaBridge:Security:
+// JwtSigningKey) from a store the node does not actually serve, so a value rotated through the
+// shared store would never reach the pre-host config, and two central nodes could disagree on
+// what a token resolves to. Every other case (sites, the default per-node SQLite store) keeps the
+// local SQLite path exactly as it always was. A blank or ${secret:}-valued connection string on
+// the shared-store path fails HERE, through the same EnsureCentralSharedStoreConnectionString
+// check AddScadaBridgeSecrets runs — this is the earliest code that would otherwise hand the bad
+// value to SqlConnection, whose generic "initialization string" format error would bury the
+// designed message.
 //
+// Two consequences worth knowing. This block runs BEFORE Serilog exists and outside the
+// try/catch below, so a SQL Server outage at central boot exits with a bare stderr stack
+// trace, no structured log — honest and container-restart-retryable, just unenriched; the
+// SQLite path had the same window but its only failure mode was a local file. (That exit
+// is real only because of the UnhandledException handler at the top of this file — the
+// runtime's own crash path wedges as container PID 1, ScadaBridge#34.) And the role read
+// stays here at the composition root rather than inside SecretsRegistration: that class
+// refuses to read the node role from configuration (a config-read role could silently turn a
+// site into a shared-store central — see SecretsNodeRole), while Program.cs must read it to
+// branch the whole composition anyway.
+var expanderRole = string.Equals(
+    configuration["ScadaBridge:Node:Role"], "Central", StringComparison.OrdinalIgnoreCase)
+        ? SecretsNodeRole.Central
+        : SecretsNodeRole.Site;
+
 // Secrets 0.6.2 validates Secrets:SqlitePath as absolute AND outside the content root, finding the
 // content root from IHostEnvironment — which this throwaway ServiceCollection does not have, and
 // which does not exist yet anyway (the host is not built until further down). Both the resolution
@@ -87,14 +110,17 @@ var configuration = new ConfigurationBuilder()
 // AddPreHostSqliteExpander for why the content-root argument is load-bearing and silent when
 // dropped, and PreHostSecretsContentRootTests for the regression guard.
 var expanderServices = new ServiceCollection();
-expanderServices.AddPreHostSqliteExpander(
-    configuration, SecretsRegistration.ResolveHostContentRoot());
+expanderServices.AddPreHostExpander(
+    configuration, expanderRole, SecretsRegistration.ResolveHostContentRoot());
 
 #pragma warning disable ASP0000 // deliberate throwaway container
 await using (var secretsProvider = expanderServices.BuildServiceProvider())
 #pragma warning restore ASP0000
 {
-    await secretsProvider.GetRequiredService<SqliteSecretsStoreMigrator>()
+    // ISecretsStoreMigrator is the store-agnostic seam both AddZbSecrets (SQLite) and
+    // AddZbSecretsSqlServerStore register — resolving through it means this block does not need
+    // to know, or re-derive, which concrete store AddPreHostExpander just composed.
+    await secretsProvider.GetRequiredService<ISecretsStoreMigrator>()
         .MigrateAsync(default);
 
     var resolver = secretsProvider.GetRequiredService<ISecretResolver>();
@@ -359,14 +385,14 @@ try
                 .GetValue<bool>(nameof(ZB.MOM.WW.ScadaBridge.Security.Auth.AuthDisableLoginOptions.AllowOutsideDevelopment)));
         builder.Services.AddSecurity(disableLogin);
         builder.Services.AddCentralUI();
-        // Local SQLite store by default. With Secrets:Replication:Enabled true it becomes either a
-        // local store synced against a shared SQL-Server hub (Mode=SqlServer, and only when a hub
-        // connection string is present) or — the production topology, scadaproj#3 — the HOST half
-        // of the pull-only gRPC secrets hub (Mode=Grpc), which central serves and sites sweep.
-        // The Central role never registers the sweep client; passing the role explicitly is what
-        // makes that a compile-time property of this composition root rather than a config guess.
-        // See SecretsRegistration for why the SQL gate exists (eager options validation), why the
-        // hub credential can never itself come from the hub, and why gRPC mode fails closed.
+        // Local SQLite store by default. With Secrets:Store:Provider=SqlServer this central node's
+        // store becomes the SHARED SQL-Server database instead — no hub, no follower sweep, just
+        // both central nodes reading and writing the same rows, so a clustered pair can never ship
+        // sites a drifted secret set (Task 5). The Layer-A pre-host expander above resolves this
+        // same predicate (SecretsRegistration.UsesCentralSharedStore) so it always reads the store
+        // this node actually serves. See SecretsRegistration for why the SQL gate exists (eager
+        // options validation) and why the connection string can never itself be a ${secret:}
+        // reference.
         builder.Services.AddScadaBridgeSecrets(builder.Configuration, SecretsNodeRole.Central);
         // Secrets UI authorization: adds the named policies secrets:manage + secrets:reveal
         // (role-based) consumed by the /admin/secrets page. AddSecretsAuthorization only ADDS

src/ZB.MOM.WW.ScadaBridge.Host/SecretsRegistration.cs

@@ -62,9 +62,10 @@ public enum SecretsNodeRole
 /// <b>Bootstrap constraint.</b> The shared store's own connection string can never itself come from
 /// that store — a node cannot read the database to learn how to reach the database. It must arrive
 /// from outside it: an environment variable, or a literal value in appsettings. It can never be a
-/// <c>${secret:}</c> reference, because on central the pre-host expander that would resolve such a
-/// reference runs against the very store this string points at (see <c>Program.cs</c>) — the
-/// reference could never resolve, so registration rejects it outright.
+/// <c>${secret:}</c> reference, because on central the pre-host expander (<see cref="AddPreHostExpander"/>,
+/// called from <c>Program.cs</c> — see <see cref="UsesCentralSharedStore"/>) resolves EVERY
+/// <c>${secret:}</c> token in configuration, including this one, from the very store this string
+/// points at — the reference could never resolve, so registration rejects it outright.
 /// </para>
 /// </remarks>
 public static class SecretsRegistration
@@ -87,6 +88,30 @@ public static class SecretsRegistration
     /// </summary>
     public const string StoreProviderKey = "Secrets:Store:Provider";
 
+    /// <summary>
+    /// Whether this node's secret store is the shared SQL-Server database — the single predicate
+    /// behind BOTH <see cref="AddScadaBridgeSecrets"/>'s runtime registration AND
+    /// <see cref="AddPreHostExpander"/>'s Layer-A <c>${secret:}</c> expander, so the two can never
+    /// disagree about which store a Central node serves. That matters more than tidiness: an
+    /// expander that resolved boot-time <c>${secret:}</c> tokens from the per-node SQLite file
+    /// while the running node served the shared SQL store would let two central nodes boot with
+    /// different resolved values (e.g. a different <c>JwtSigningKey</c>) — exactly the per-node
+    /// drift <see cref="StoreProviderKey"/>=SqlServer exists to prevent.
+    /// </summary>
+    /// <param name="config">Application configuration.</param>
+    /// <param name="role">Which half of the composition this node is.</param>
+    /// <returns>
+    /// <see langword="true"/> when <paramref name="role"/> is <see cref="SecretsNodeRole.Central"/>
+    /// and <see cref="StoreProviderKey"/> is (case-insensitively) <c>SqlServer</c>.
+    /// </returns>
+    internal static bool UsesCentralSharedStore(IConfiguration config, SecretsNodeRole role)
+    {
+        ArgumentNullException.ThrowIfNull(config);
+
+        return role == SecretsNodeRole.Central
+            && string.Equals(config[StoreProviderKey], "SqlServer", StringComparison.OrdinalIgnoreCase);
+    }
+
     /// <summary>
     /// Resolves the content root the HOST will use, from outside the host — the Layer-A
     /// <c>${secret:}</c> expander in <c>Program.cs</c> runs before any builder exists, so
@@ -146,6 +171,62 @@ public static class SecretsRegistration
         return services.AddZbSecrets(config, SecretsSectionPath, contentRoot);
     }
 
+    /// <summary>
+    /// Composes the Layer-A pre-host <c>${secret:}</c> expander stack for <paramref name="role"/> —
+    /// the shared SQL-Server store when <see cref="UsesCentralSharedStore"/> is true, otherwise the
+    /// plain local SQLite store (<see cref="AddPreHostSqliteExpander"/>).
+    /// </summary>
+    /// <remarks>
+    /// <para>
+    /// This is the fix for the gap <see cref="UsesCentralSharedStore"/>'s own remarks describe: the
+    /// expander MUST read the same store <see cref="AddScadaBridgeSecrets"/> will register for the
+    /// running node, or a boot-time <c>${secret:}</c> reference (e.g. in
+    /// <c>ScadaBridge:Security:JwtSigningKey</c>) resolves from a store the node does not actually
+    /// serve. Both this method and <see cref="AddScadaBridgeSecrets"/> branch on the exact same
+    /// <see cref="UsesCentralSharedStore"/> call, so they cannot drift apart the way the old
+    /// Grpc-mode expander branch in <c>Program.cs</c> and the Grpc-mode runtime branch here once
+    /// had to be kept in step by hand.
+    /// </para>
+    /// <para>
+    /// On the shared-store path this also runs <see cref="EnsureCentralSharedStoreConnectionString"/>
+    /// BEFORE composing anything — the same fail-closed rules apply here as at runtime: a blank
+    /// connection string, or one holding a <c>${secret:}</c> reference (which could never resolve;
+    /// this very expander is what would have to resolve it), fails the boot with the designed
+    /// message instead of a generic SqlConnection format error.
+    /// </para>
+    /// </remarks>
+    /// <param name="services">The throwaway expander service collection.</param>
+    /// <param name="config">Application configuration.</param>
+    /// <param name="role">Which half of the composition this node is.</param>
+    /// <param name="contentRoot">The host's content root, from <see cref="ResolveHostContentRoot"/> — only used on the SQLite path.</param>
+    /// <returns>The same collection, for chaining.</returns>
+    /// <exception cref="InvalidOperationException">
+    /// <paramref name="role"/> is Central with the shared store selected, and
+    /// <see cref="HubConnectionStringKey"/> is blank or a <c>${secret:}</c> reference.
+    /// </exception>
+    public static IServiceCollection AddPreHostExpander(
+        this IServiceCollection services,
+        IConfiguration config,
+        SecretsNodeRole role,
+        string contentRoot)
+    {
+        ArgumentNullException.ThrowIfNull(services);
+        ArgumentNullException.ThrowIfNull(config);
+        ArgumentException.ThrowIfNullOrWhiteSpace(contentRoot);
+
+        if (UsesCentralSharedStore(config, role))
+        {
+            EnsureCentralSharedStoreConnectionString(config, $"{StoreProviderKey}=SqlServer");
+
+            // Same 3-arg overload as the runtime registration in AddScadaBridgeSecrets — this is
+            // deliberately the SAME store, not a lookalike one composed independently.
+            return services.AddZbSecretsSqlServerStore(
+                config, SecretsSectionPath, SqlServerSectionPath);
+        }
+
+        return services.AddPreHostSqliteExpander(config, contentRoot);
+    }
+
     /// <summary>
     /// The one definition of "a Central node has a usable shared-store connection string", shared
     /// by every caller that puts a Central node on the shared SQL-Server store — the
@@ -230,9 +311,10 @@ public static class SecretsRegistration
 
         // Secrets:Store:Provider=SqlServer on Central only. A Site never reaches this: it always
         // keeps its local SQLite store, straight through a WAN outage, same as every other
-        // topology, and never decorated — see the class remarks.
-        if (role == SecretsNodeRole.Central
-            && string.Equals(config[StoreProviderKey], "SqlServer", StringComparison.OrdinalIgnoreCase))
+        // topology, and never decorated — see the class remarks. UsesCentralSharedStore is the
+        // SAME predicate AddPreHostExpander branches on, so the two can never disagree about
+        // which store a Central node serves.
+        if (UsesCentralSharedStore(config, role))
         {
             // Fail-closed rules: blank, or a ${secret:} reference (bootstrap circularity — the
             // pre-host expander needs this same store to resolve it).

src/ZB.MOM.WW.ScadaBridge.Host/StartupValidator.cs

@@ -31,7 +31,7 @@ public static class StartupValidator
         // Deploy Artifacts, and a replicated site store would publish DeployedSecretsMirror's local
         // deletes fleet-wide. Checked unconditionally (both roles): a Site used to be the gRPC
         // hub's follower half, so the flag was never Central-only.
-        var secretsReplicationEnabled = configuration.GetValue<bool>("Secrets:Replication:Enabled");
+        var secretsReplicationEnabled = IsSecretsReplicationEnabled(configuration);
 
         // GrpcPort: default 8083 when absent; only fails the range rule when the key is
         // present AND invalid. The out-param assignment mirrors the original so the
@@ -251,6 +251,7 @@ public static class StartupValidator
         var seedNodes = configuration.GetSection("ScadaBridge:Cluster:SeedNodes").Get<List<string>>();
         var provider = InboundApiKeyStoreOptions.ReadProvider(configuration);
         var secretsStoreProvider = configuration["Secrets:Store:Provider"];
+        var secretsReplicationEnabled = IsSecretsReplicationEnabled(configuration);
         var isClusteredCentral = role == "Central" && seedNodes is { Count: >= 2 };
 
         var warnings = new List<string>();
@@ -283,9 +284,46 @@ public static class StartupValidator
                 + "deploy its own secret set to sites; set Secrets:Store:Provider=SqlServer");
         }
 
+        // Review round 1, Minor #4: a config that still carries Secrets:Replication or
+        // Secrets:GrpcHub child keys — left over from before Task 11 retired the hub — is not
+        // itself an error (Validate already rejects Enabled=true outright), but every key under
+        // either section is now silently ignored. That is worth a Warning so an operator cleans
+        // QA/prod appsettings rather than assuming a still-present Mode/BearerToken/Endpoint value
+        // does anything. Excludes the bare "Enabled" key itself: "Enabled: false" alone is the
+        // normal, still-valid shape of every existing deployment's config and must not warn.
+        var hasStaleReplicationKeys = configuration.GetSection("Secrets:Replication").GetChildren()
+            .Any(child => !string.Equals(child.Key, "Enabled", StringComparison.OrdinalIgnoreCase));
+        var hasStaleGrpcHubKeys = configuration.GetSection("Secrets:GrpcHub").GetChildren().Any();
+
+        if (!secretsReplicationEnabled && (hasStaleReplicationKeys || hasStaleGrpcHubKeys))
+        {
+            warnings.Add(
+                "Secrets:Replication / Secrets:GrpcHub settings are ignored — the secrets hub is "
+                + "retired; remove them");
+        }
+
         return warnings;
     }
 
+    /// <summary>
+    /// Whether <c>Secrets:Replication:Enabled</c> is "set" — read as a raw string rather than
+    /// <c>IConfiguration.GetValue&lt;bool&gt;</c> (review round 1, Minor #3), which throws a raw
+    /// binder <see cref="InvalidOperationException"/> on a non-boolean value like <c>"yes"</c> or
+    /// <c>"1"</c>, so a malformed leftover key would surface as an unrelated parse crash instead of
+    /// the designed retirement message in <see cref="Validate"/>. Any value other than absent,
+    /// empty, or (case-insensitively) <c>"false"</c> counts as "set" — deliberately stricter than
+    /// bool parsing, so a typo'd leftover (<c>Enabled: 1</c>, <c>Enabled: yes</c>) still gets the
+    /// explanatory message rather than silently passing as falsy. Shared by <see cref="Validate"/>
+    /// (which rejects "set" outright) and <see cref="CollectWarnings"/> (which, for the "not set"
+    /// case, separately warns about any OTHER leftover key under the same retired sections).
+    /// </summary>
+    private static bool IsSecretsReplicationEnabled(IConfiguration configuration)
+    {
+        var raw = configuration["Secrets:Replication:Enabled"];
+        return !string.IsNullOrEmpty(raw)
+            && !string.Equals(raw, "false", StringComparison.OrdinalIgnoreCase);
+    }
+
     /// <summary>
     /// Extracts the TCP port from an Akka seed-node address of the form
     /// <c>akka.tcp://system@host:port</c>. Returns <c>-1</c> when no port can be parsed.

src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Secrets/DeployedSecretsMirror.cs

@@ -116,13 +116,16 @@ public sealed record MirrorResult(
 /// can observe a stale pre-rotation plaintext for up to the cache TTL.
 /// </para>
 /// <para>
-/// <b>Node-local store assumption (review Minor #5).</b> This mirror assumes the site's
-/// <see cref="ISecretStore"/> is node-local (plain local SQLite, or the gRPC-hub follower half,
-/// both of which keep writes local). It is NOT safe to run against a site node whose store is
-/// decorated by SQL-Server secrets REPLICATION (<c>Secrets:Replication:Mode=SqlServer</c>) — its
-/// <c>DeleteAsync</c> publishes the tombstone to the shared hub, so a site merely un-referencing
-/// a secret would delete it fleet-wide. Task 11 rejects that configuration on the Site role;
-/// no code change is made here.
+/// <b>Node-local store assumption (review Minor #5; Task 11 removed every replication mode).</b>
+/// This mirror assumes the site's <see cref="ISecretStore"/> is always plain, node-local, local
+/// SQLite — the ONLY shape a Site node's store can be after Task 11 retired both the pull-only
+/// gRPC secrets hub and the SQL-Server <c>ReplicatingSecretStore</c> mode that preceded it (see
+/// <c>SecretsRegistration</c>). It would NOT have been safe to run this mirror against a store
+/// decorated by either retired replication mode: <c>DeleteAsync</c> would have published the
+/// tombstone to every other node sharing the decorator, so a site merely un-referencing a secret
+/// would have deleted it fleet-wide. <c>StartupValidator</c> now rejects
+/// <c>Secrets:Replication:Enabled=true</c> outright on either role, so that configuration can no
+/// longer reach this mirror at all.
 /// </para>
 /// <para>
 /// <b>Resolve semantics and the resolver cache.</b> The trial-resolve verify step goes through