feat(secrets): opt-in SQL-Server hub replication for the host secret store

Routes both host-container secret registrations (central role in Program.cs,
site role in SiteServiceRegistration) through a new SecretsRegistration
composition seam that optionally enables ZB.MOM.WW.Secrets.Replicator.SqlServer
hub-replication mode: each node keeps a LOCAL store that syncs bidirectionally
with a shared central SQL hub, so a site cluster keeps resolving secrets
straight through a WAN outage to central.

OPT-IN GATE (the load-bearing part). AddZbSecretsSqlServerReplication validates
its options EAGERLY at registration time, so wiring it unconditionally would
make ScadaBridge fail to START anywhere Secrets:SqlServer:ConnectionString is
unset -- every dev box, every docker node, every existing deployment. The
SQL-Server package is therefore only touched when BOTH Secrets:Replication:
Enabled is true AND a non-blank connection string is present; otherwise the
registration is byte-identical to the previous plain AddZbSecrets call.
Enabled-without-a-connection-string falls back to local-only and logs a warning
rather than failing the node or silently looking healthy.

BOOTSTRAP CYCLE. The hub connection string is itself a secret and can never come
from the hub -- a node cannot read the hub to learn how to reach the hub. It must
arrive from outside the replicated set: an environment variable, or a ${secret:}
reference seeded in that node's own LOCAL store. appsettings.json therefore ships
ConnectionString empty with a _comment saying so (leaf keys starting with '_' are
skipped by the reference expander, verified in SecretReferenceExpander). No real
connection string is committed. The pre-host ${secret:} expander in Program.cs is
deliberately left on a plain local SQLite store for the same reason.

Per-node docker appsettings are intentionally NOT modified -- replication stays
off there for now.

Tests written before the wiring and confirmed red first (2 failed / 4 passed),
green after (6/6). They BUILD a container and RESOLVE from it rather than
asserting over ServiceDescriptors: a decorator can look correct as a descriptor
list and still throw on first resolve because the undecorated concrete store it
depends on is missing -- that exact defect shipped once in this library with all
descriptor-level tests green, so the undecorated SqliteSecretStore gets its own
resolution test.

Verified: Host.Tests 285/285 pass; full build (all projects except the
pre-existing AngleSharp NU1902 CentralUI.Tests restore break) 0 warnings,
0 errors.

Claude-Session: https://claude.ai/code/session_01BL2Vu1ESDQ9SCN4gVKkdts

src/ZB.MOM.WW.ScadaBridge.Host/Program.cs

@@ -179,7 +179,11 @@ try
                 .GetValue<bool>(nameof(ZB.MOM.WW.ScadaBridge.Security.Auth.AuthDisableLoginOptions.AllowOutsideDevelopment)));
         builder.Services.AddSecurity(disableLogin);
         builder.Services.AddCentralUI();
-        builder.Services.AddZbSecrets(builder.Configuration, "Secrets");
+        // Local SQLite store by default; a local store replicating against a shared SQL-Server hub
+        // only when Secrets:Replication:Enabled is true AND a hub connection string is present.
+        // See SecretsRegistration for why the gate exists (eager options validation) and why the
+        // hub connection string can never itself come from the hub.
+        builder.Services.AddScadaBridgeSecrets(builder.Configuration);
         // Secrets UI authorization: adds the named policies secrets:manage + secrets:reveal
         // (role-based) consumed by the /admin/secrets page. AddSecretsAuthorization only ADDS
         // these two policies via Configure<AuthorizationOptions> — it composes additively with

src/ZB.MOM.WW.ScadaBridge.Host/SecretsRegistration.cs

@@ -0,0 +1,92 @@
+using ZB.MOM.WW.Secrets.DependencyInjection;
+using ZB.MOM.WW.Secrets.Replicator.SqlServer.DependencyInjection;
+
+namespace ZB.MOM.WW.ScadaBridge.Host;
+
+/// <summary>
+/// Single composition-root entry point for the host container's secret store, shared by the
+/// central-role registrations in <c>Program.cs</c> and the site-role registrations in
+/// <see cref="SiteServiceRegistration"/> so both roles wire secrets identically.
+/// </summary>
+/// <remarks>
+/// <para>
+/// Clustered replication is <b>opt-in and off by default</b>. ScadaBridge is hub-and-spoke — one
+/// central cluster plus N separate site clusters — and hub mode gives each node a LOCAL store that
+/// syncs bidirectionally with a shared central SQL hub, so a site keeps resolving secrets straight
+/// through a WAN outage.
+/// </para>
+/// <para>
+/// The gate is not a style preference. <c>AddZbSecretsSqlServerReplication</c> validates its options
+/// EAGERLY at registration time, so calling it unconditionally would throw at startup on every node
+/// where <c>Secrets:SqlServer:ConnectionString</c> is unset — which is every dev box, every docker
+/// node and every existing deployment. Both the explicit <c>Secrets:Replication:Enabled</c> flag and
+/// a non-blank connection string are therefore required before the SQL-Server package is touched at
+/// all; otherwise this is byte-for-byte the plain local-SQLite registration it has always been.
+/// </para>
+/// <para>
+/// <b>Bootstrap constraint.</b> The hub connection string is itself a secret, and it can never come
+/// from the hub — a node cannot read the hub to learn how to reach the hub. It must arrive from
+/// outside the replicated set: an environment variable, or a <c>${secret:}</c> reference that is
+/// seeded in that node's own LOCAL store (the pre-host expander in <c>Program.cs</c> runs against a
+/// plain local SQLite store before the host container exists, so such a reference does resolve).
+/// That expander is deliberately left un-replicated for exactly this reason.
+/// </para>
+/// </remarks>
+public static class SecretsRegistration
+{
+    /// <summary>Configuration section holding the core secrets options.</summary>
+    public const string SecretsSectionPath = "Secrets";
+
+    /// <summary>Configuration section holding the SQL-Server hub options.</summary>
+    public const string SqlServerSectionPath = "Secrets:SqlServer";
+
+    /// <summary>Configuration key gating clustered replication. Absent or false = off.</summary>
+    public const string ReplicationEnabledKey = "Secrets:Replication:Enabled";
+
+    /// <summary>Configuration key holding the shared SQL-Server hub connection string.</summary>
+    public const string HubConnectionStringKey = "Secrets:SqlServer:ConnectionString";
+
+    /// <summary>
+    /// Registers the host container's secret store: a plain local SQLite store by default, or a
+    /// local store replicating against a shared SQL-Server hub when replication is explicitly
+    /// enabled and a hub connection string is present.
+    /// </summary>
+    /// <param name="services">The service collection to register into.</param>
+    /// <param name="config">Application configuration for options binding.</param>
+    /// <returns>The same <paramref name="services"/> instance, for chaining.</returns>
+    public static IServiceCollection AddScadaBridgeSecrets(
+        this IServiceCollection services,
+        IConfiguration config)
+    {
+        ArgumentNullException.ThrowIfNull(services);
+        ArgumentNullException.ThrowIfNull(config);
+
+        var enabled = config.GetValue<bool>(ReplicationEnabledKey);
+        var connectionString = config[HubConnectionStringKey];
+
+        if (enabled && !string.IsNullOrWhiteSpace(connectionString))
+        {
+            // Registers the validated hub options, decorates ISecretStore with
+            // ReplicatingSecretStore, and adds the hub-schema migration + bidirectional sweep
+            // hosted services. Calls AddZbSecrets internally — do NOT also call it here.
+            return services.AddZbSecretsSqlServerReplication(
+                config, SecretsSectionPath, SqlServerSectionPath);
+        }
+
+        if (enabled)
+        {
+            // Asked for replication but gave nothing to replicate against. Falling back to the
+            // local-only store keeps the node up, but it is silently NOT participating in the
+            // cluster, so say so loudly rather than letting it look healthy.
+            Serilog.Log.Warning(
+                "{Key} is true but {ConnKey} is empty — clustered secret replication is DISABLED "
+                    + "and this node's secrets are local-only. The hub connection string cannot come "
+                    + "from the hub itself; supply it via environment variable or seed it in this "
+                    + "node's local store.",
+                ReplicationEnabledKey,
+                HubConnectionStringKey);
+        }
+
+        return services.AddZbSecrets(config, SecretsSectionPath);
+    }
+}

src/ZB.MOM.WW.ScadaBridge.Host/SiteServiceRegistration.cs

@@ -50,7 +50,11 @@ public static class SiteServiceRegistration
         var siteDbPath = config["ScadaBridge:Database:SiteDbPath"] ?? "site.db";
         services.AddSiteRuntime($"Data Source={siteDbPath}");
         services.AddDataConnectionLayer();
-        services.AddZbSecrets(config, "Secrets");
+        // Local SQLite store by default; a local store replicating against a shared SQL-Server hub
+        // only when Secrets:Replication:Enabled is true AND a hub connection string is present.
+        // Hub mode matters most here: a site node keeps resolving secrets from its local store
+        // straight through a WAN outage to central. See SecretsRegistration.
+        services.AddScadaBridgeSecrets(config);
         // Adapter that surfaces the site id to
         // StoreAndForwardService through DI WITHOUT introducing a
         // StoreAndForward → HealthMonitoring project-reference cycle. Must be

src/ZB.MOM.WW.ScadaBridge.Host/appsettings.json

@@ -13,7 +13,15 @@
     "SqlitePath": "scadabridge-secrets.db",
     "MasterKey": { "Source": "Environment", "EnvVarName": "ZB_SECRETS_MASTER_KEY" },
     "RunMigrationsOnStartup": true,
-    "ResolveCacheTtl": "00:00:30"
+    "ResolveCacheTtl": "00:00:30",
+    "Replication": { "Enabled": false },
+    "SqlServer": {
+      "ConnectionString": "",
+      "SchemaName": "zbsecrets",
+      "SyncInterval": "00:00:30",
+      "SyncOnStartup": true,
+      "_comment": "Hub connstr must be seeded in the LOCAL store (or supplied via env) - it cannot come from the hub itself. Enable via Secrets:Replication:Enabled."
+    }
   },
   "Serilog": {
     "Using": [

tests/ZB.MOM.WW.ScadaBridge.Host.Tests/SecretsReplicationWiringTests.cs

@@ -0,0 +1,150 @@
+using Microsoft.Extensions.Configuration;
+using Microsoft.Extensions.DependencyInjection;
+using Microsoft.Extensions.Hosting;
+using ZB.MOM.WW.Secrets.Abstractions;
+using ZB.MOM.WW.Secrets.Replication;
+using ZB.MOM.WW.Secrets.Sqlite;
+
+namespace ZB.MOM.WW.ScadaBridge.Host.Tests;
+
+/// <summary>
+/// DI-resolution tests for <see cref="SecretsRegistration.AddScadaBridgeSecrets"/>.
+/// <para>
+/// These deliberately BUILD a container and RESOLVE out of it rather than asserting over
+/// <see cref="ServiceDescriptor"/>s. A decorator registration can look perfectly correct as a
+/// descriptor list and still throw on the first resolve because the undecorated concrete store it
+/// depends on is missing — that exact gap shipped once in this library, with every descriptor-level
+/// unit test green. Resolution is the only assertion that catches it.
+/// </para>
+/// <para>
+/// Registration must not need a reachable SQL Server: the SQL-Server package validates its options
+/// eagerly, but validation only checks that the connection string is non-empty and the schema name
+/// is a legal identifier. Nothing connects until a sweep or a write runs, and these tests do
+/// neither — so the dummy connection string below never resolves to a real host.
+/// </para>
+/// </summary>
+public class SecretsReplicationWiringTests
+{
+    // Syntactically valid, deliberately unreachable. Never connected to by these tests.
+    private const string DummyConnectionString =
+        "Server=unused;Database=x;Integrated Security=true;";
+
+    private static IConfiguration BuildConfig(params (string Key, string Value)[] overrides)
+    {
+        var settings = new Dictionary<string, string?>
+        {
+            ["Secrets:SqlitePath"] = Path.Combine(Path.GetTempPath(), "sb-secrets-wiring-test.db"),
+            ["Secrets:MasterKey:Source"] = "Environment",
+            ["Secrets:MasterKey:EnvVarName"] = "ZB_SECRETS_MASTER_KEY",
+            ["Secrets:RunMigrationsOnStartup"] = "false",
+        };
+
+        foreach ((string key, string value) in overrides)
+        {
+            settings[key] = value;
+        }
+
+        return new ConfigurationBuilder().AddInMemoryCollection(settings).Build();
+    }
+
+    private static ServiceProvider BuildProvider(IConfiguration config)
+    {
+        var services = new ServiceCollection();
+        services.AddLogging();
+        services.AddScadaBridgeSecrets(config);
+        return services.BuildServiceProvider(validateScopes: true);
+    }
+
+    /// <summary>
+    /// Any hosted service contributed by the SQL-Server replicator package. Asserted by assembly
+    /// rather than by type because <c>SqlServerHubMigrationHostedService</c> is internal to it.
+    /// </summary>
+    private static bool IsReplicatorHostedService(ServiceDescriptor descriptor) =>
+        descriptor.ServiceType == typeof(IHostedService)
+        && descriptor.ImplementationType?.Assembly.GetName().Name
+            == "ZB.MOM.WW.Secrets.Replicator.SqlServer";
+
+    [Fact]
+    public void ReplicationDisabledByDefault_ResolvesPlainSqliteStore()
+    {
+        using ServiceProvider provider = BuildProvider(BuildConfig());
+
+        var store = provider.GetRequiredService<ISecretStore>();
+
+        Assert.IsType<SqliteSecretStore>(store);
+        Assert.IsNotType<ReplicatingSecretStore>(store);
+    }
+
+    [Fact]
+    public void ReplicationDisabledByDefault_RegistersNoReplicatorHostedServices()
+    {
+        var services = new ServiceCollection();
+        services.AddLogging();
+        services.AddScadaBridgeSecrets(BuildConfig());
+
+        Assert.DoesNotContain(services, IsReplicatorHostedService);
+    }
+
+    [Fact]
+    public void ReplicationEnabledWithoutConnectionString_StaysOnPlainSqliteStore()
+    {
+        // The gate requires BOTH the flag and a connection string. Enabling the flag alone must not
+        // reach the SQL-Server package, whose options validation would throw at registration and
+        // take the whole node down at startup.
+        var services = new ServiceCollection();
+        services.AddLogging();
+
+        IConfiguration config = BuildConfig(("Secrets:Replication:Enabled", "true"));
+
+        services.AddScadaBridgeSecrets(config);
+
+        using ServiceProvider provider = services.BuildServiceProvider(validateScopes: true);
+
+        Assert.IsType<SqliteSecretStore>(provider.GetRequiredService<ISecretStore>());
+        Assert.DoesNotContain(services, IsReplicatorHostedService);
+    }
+
+    [Fact]
+    public void ReplicationEnabledWithConnectionString_ResolvesReplicatingStore()
+    {
+        IConfiguration config = BuildConfig(
+            ("Secrets:Replication:Enabled", "true"),
+            ("Secrets:SqlServer:ConnectionString", DummyConnectionString));
+
+        using ServiceProvider provider = BuildProvider(config);
+
+        Assert.IsType<ReplicatingSecretStore>(provider.GetRequiredService<ISecretStore>());
+    }
+
+    [Fact]
+    public void ReplicationEnabledWithConnectionString_UndecoratedLocalStoreAlsoResolves()
+    {
+        // ReplicatingSecretStore is constructed from the CONCRETE SqliteSecretStore, not from
+        // ISecretStore (which is the decorator itself — that would recurse). If the concrete
+        // registration were ever dropped, the decorator would fail on first resolve. This is the
+        // exact defect that shipped once, so it gets its own test.
+        IConfiguration config = BuildConfig(
+            ("Secrets:Replication:Enabled", "true"),
+            ("Secrets:SqlServer:ConnectionString", DummyConnectionString));
+
+        using ServiceProvider provider = BuildProvider(config);
+
+        Assert.NotNull(provider.GetRequiredService<SqliteSecretStore>());
+    }
+
+    [Fact]
+    public void ReplicationEnabledWithConnectionString_RegistersReplicatorHostedServices()
+    {
+        var services = new ServiceCollection();
+        services.AddLogging();
+
+        IConfiguration config = BuildConfig(
+            ("Secrets:Replication:Enabled", "true"),
+            ("Secrets:SqlServer:ConnectionString", DummyConnectionString));
+
+        services.AddScadaBridgeSecrets(config);
+
+        // Both the hub-schema migration service and the bidirectional sweep.
+        Assert.Equal(2, services.Count(IsReplicatorHostedService));
+    }
+}