feat(secrets): central's Grpc-mode store is the SHARED SQL-Server store (scadaproj#4)

In Secrets:Replication:Mode=Grpc a central node's ISecretStore is now the
shared SQL-Server store (AddZbSecretsSqlServerStore) instead of a per-node
local SQLite store. Both central hubs read and write ONE copy of every row,
so they serve identical manifests by construction — the 2026-08-07 live gate
observed central-b answering an authenticated GetManifest with an EMPTY
manifest while central-a held every secret, which would turn site-side hub
failover into a silent convergence stop.

- SecretsRegistration: two fail-closed pre-checks before any registration on
  the central+Grpc path — a blank Secrets:SqlServer:ConnectionString throws
  naming the key (an independent store per central node is the recorded
  defect), and a value containing ${secret: throws naming the bootstrap
  circularity (the expander needs this store to resolve references). Site
  registrations are byte-identical to before; SqlServer mode and
  replication-off are untouched.
- Program.cs Layer-A expander follows the store swap: central+Grpc with a
  non-blank connection string migrates and resolves pre-host ${secret:}
  references through the shared SQL store, so expanded values can never
  diverge from what the running node serves. Every other case keeps the
  SQLite path unchanged; blank-connstr central deliberately falls through so
  the clear AddScadaBridgeSecrets message is the one that fails the boot.
- appsettings.json: Secrets:SqlServer _comment now documents the Grpc-mode
  central requirement (literal/env value only, sites leave it empty).
- SecretsReplicationWiringTests: +5 pins (shared store resolves, blank and
  ${secret:} connstrings fail naming the key, sites-have-no-SqlServer-types
  descriptor sweep), central fixtures carry the now-required connstr.

Full suite green (7,474 passed across 30 projects, 0 warnings); the two
failures are pre-existing and unrelated: the Playwright live-rig suite fails
identically on unmodified main (cluster not running), and
GrpcCentralTransportTests.DeadlineExceeded_IsNotRetriedOnThePeer is a timing
flake that passes 3/3 in isolation and 470/470 on the first run of this code.

Claude-Session: https://claude.ai/code/session_014WNM4vjoVksyyBraTXSZE1
This commit is contained in:
Joseph Doherty
2026-08-07 10:33:28 -04:00
parent f6c3f7c593
commit 43e87a7492
4 changed files with 245 additions and 29 deletions
+39 -4
View File
@@ -32,6 +32,8 @@ using ZB.MOM.WW.ScadaBridge.Transport;
using ZB.MOM.WW.Secrets.Abstractions;
using ZB.MOM.WW.Secrets.Configuration;
using ZB.MOM.WW.Secrets.DependencyInjection;
using ZB.MOM.WW.Secrets.Replicator.SqlServer;
using ZB.MOM.WW.Secrets.Replicator.SqlServer.DependencyInjection;
using ZB.MOM.WW.Secrets.Sqlite;
using ZB.MOM.WW.Secrets.Ui;
using ZB.MOM.WW.Telemetry;
@@ -52,13 +54,46 @@ var configuration = new ConfigurationBuilder()
// Expand ${secret:...} config references before any validator/binder sees them (Layer A).
// Throwaway provider — disposed here, shares no singletons with the host container.
//
// The expander must read the SAME store the running node will serve. On a central node in Grpc
// replication mode that store is the SHARED SQL-Server store (see SecretsRegistration,
// scadaproj#4) — an expander left on SQLite there would resolve pre-host ${secret:} references
// from a stale/empty local store, silently diverging from what the node's own hub serves. Every
// other case (sites, SqlServer mode, replication off) keeps the local SQLite path exactly as it
// always was. Central+Grpc with a BLANK connection string deliberately falls through to the
// SQLite path too: that boot is about to fail in AddScadaBridgeSecrets with the message naming
// Secrets:SqlServer:ConnectionString, so the throw is not duplicated here.
var expanderUsesSharedSqlStore =
string.Equals(
configuration["ScadaBridge:Node:Role"], "Central", StringComparison.OrdinalIgnoreCase)
&& SecretsRegistration.UsesGrpcHub(configuration)
&& !string.IsNullOrWhiteSpace(configuration[SecretsRegistration.HubConnectionStringKey]);
var expanderServices = new ServiceCollection();
if (expanderUsesSharedSqlStore)
{
expanderServices.AddZbSecretsSqlServerStore(configuration, "Secrets");
}
else
{
expanderServices.AddZbSecrets(configuration, "Secrets");
}
#pragma warning disable ASP0000 // deliberate throwaway container
await using (var secretsProvider = new ServiceCollection()
.AddZbSecrets(configuration, "Secrets")
.BuildServiceProvider())
await using (var secretsProvider = expanderServices.BuildServiceProvider())
#pragma warning restore ASP0000
{
await secretsProvider.GetRequiredService<SqliteSecretsStoreMigrator>().MigrateAsync(default);
if (expanderUsesSharedSqlStore)
{
await secretsProvider.GetRequiredService<SqlServerSecretsStoreMigrator>()
.MigrateAsync(default);
}
else
{
await secretsProvider.GetRequiredService<SqliteSecretsStoreMigrator>()
.MigrateAsync(default);
}
var resolver = secretsProvider.GetRequiredService<ISecretResolver>();
await new SecretReferenceExpander(resolver)
.ExpandConfigurationAsync(configuration, default);
@@ -47,9 +47,10 @@ public enum SecretsReplicationMode
/// <remarks>
/// <para>
/// Clustered replication is <b>opt-in and off by default</b>. ScadaBridge is hub-and-spoke — one
/// central cluster plus N separate site clusters — and every supported topology gives each node a
/// LOCAL store, so a site keeps resolving secrets straight through a WAN outage. What differs
/// between the two modes is only how that local store is kept converged.
/// central cluster plus N separate site clusters — and every supported topology gives each SITE
/// node a LOCAL store, so a site keeps resolving secrets straight through a WAN outage. What
/// differs between the modes is how convergence works, and — in gRPC mode — where central's own
/// store lives.
/// </para>
/// <para>
/// <b><see cref="SecretsReplicationMode.Grpc"/> is the production topology</b> (scadaproj#3):
@@ -58,7 +59,17 @@ public enum SecretsReplicationMode
/// string to central's database, which violates ScadaBridge's standing rule that sites talk to
/// central, not to central's DB. Pull-only is a property of the wire contract rather than a
/// setting: the proto carries no write RPC, so a site cannot push even if misconfigured, and
/// secrets therefore originate at central only.
/// secrets therefore originate at central only. <b>Central's own store in this mode is the SHARED
/// SQL-Server store</b> (<c>AddZbSecretsSqlServerStore</c>), not local SQLite: both central nodes
/// read and write ONE copy of every row, so the two hub instances serve identical manifests by
/// construction. The alternative — an independent local store per central node — is the recorded
/// defect (scadaproj#4): one central answering authenticated followers with an empty manifest,
/// turning site-side hub failover into a silent convergence stop. The accepted trade is that
/// central secret resolution now depends on SQL Server availability — acceptable because central
/// is already SQL-coupled to its core, and a SQL outage makes the hub <em>throw</em> (sites keep
/// last-known-good), never serve empty. The pre-host <c>${secret:}</c> expander in
/// <c>Program.cs</c> follows this store swap, so pre-host references resolve from the same store
/// the running node serves.
/// </para>
/// <para>
/// <b>The SQL-Server gate is not a style preference.</b> <c>AddZbSecretsSqlServerReplication</c>
@@ -73,19 +84,23 @@ public enum SecretsReplicationMode
/// <b>The gRPC mode deliberately does NOT degrade the same way.</b> A missing connection string in
/// SQL-Server mode falls back to a local-only store with a loud warning, and that behaviour is
/// preserved untouched. gRPC mode instead fails at startup when its endpoint or bearer token is
/// missing, because the fallback is the exact outcome the hub exists to prevent — a site serving
/// secrets that silently never converge with central — and because selecting the mode takes an
/// explicit second key, so an operator who typed it meant it.
/// missing — or, on central, when the shared-store connection string is — because the fallback is
/// the exact outcome the hub exists to prevent — a site serving secrets that silently never
/// converge with central — and because selecting the mode takes an explicit second key, so an
/// operator who typed it meant it.
/// </para>
/// <para>
/// <b>Bootstrap constraint.</b> A replication credential can never itself come from the replicated
/// set — a node cannot read the hub to learn how to reach the hub. The SQL hub connection string
/// must arrive from outside it: an environment variable, or a <c>${secret:}</c> reference seeded in
/// that node's own LOCAL store (the pre-host expander in <c>Program.cs</c> runs against a plain
/// local SQLite store before the host container exists, so such a reference does resolve). The
/// set — a node cannot read the hub to learn how to reach the hub. In SQL-Server mode the hub
/// connection string must arrive from outside it: an environment variable, or a <c>${secret:}</c>
/// reference seeded in that node's own LOCAL store (the pre-host expander in <c>Program.cs</c>
/// runs against the local SQLite store before the host container exists, so such a reference does
/// resolve). In gRPC mode the rule is stricter on central: <c>Secrets:SqlServer:ConnectionString</c>
/// can never be a <c>${secret:}</c> reference at all, because on central the pre-host expander
/// itself runs against the shared SQL store (see <c>Program.cs</c>) and would need this very
/// string to reach the store that resolves it — registration rejects such a value outright. The
/// same holds for <c>Secrets:GrpcHub:BearerToken</c>, which is supplied from appsettings or the
/// environment exactly as the mesh pre-shared keys are. That expander is deliberately left
/// un-replicated for exactly this reason.
/// environment exactly as the mesh pre-shared keys are.
/// </para>
/// </remarks>
public static class SecretsRegistration
@@ -173,7 +188,9 @@ public static class SecretsRegistration
/// <summary>
/// Registers the host container's secret store: a plain local SQLite store by default, or a
/// replicating topology when <see cref="ReplicationEnabledKey"/> is set — a shared SQL-Server
/// hub, or the pull-only gRPC hub whose half is chosen by <paramref name="role"/>.
/// hub, or the pull-only gRPC hub whose half is chosen by <paramref name="role"/>. In gRPC
/// mode a central node's store is the SHARED SQL-Server store, and registration fails closed
/// when <see cref="HubConnectionStringKey"/> is blank or is a <c>${secret:}</c> reference.
/// </summary>
/// <param name="services">The service collection to register into.</param>
/// <param name="config">Application configuration for options binding.</param>
@@ -192,14 +209,57 @@ public static class SecretsRegistration
// disagree about whether the hub is on.
if (UsesGrpcHub(config))
{
// Neither gRPC extension registers the local store — pull-only means local writes
// never leave the node, so there is nothing to decorate and the store stays exactly as
// configured. Register it first; both halves resolve it.
services.AddZbSecrets(config, SecretsSectionPath);
switch (role)
{
case SecretsNodeRole.Central:
// Both pre-checks run BEFORE any registration so the boot failure names the
// real problem instead of surfacing later as a package validation message or a
// malformed-connection-string fault on first use.
var hubConnectionString = config[HubConnectionStringKey];
if (string.IsNullOrWhiteSpace(hubConnectionString))
{
// Fail closed, matching the mode's philosophy: central in Grpc mode
// REQUIRES the shared SQL store. An independent local store per central
// node is exactly the divergence scadaproj#4 recorded — one central hub
// answered authenticated followers with an EMPTY manifest, so a hub
// failover would "succeed" against nothing and stop convergence silently.
throw new InvalidOperationException(
$"{HubConnectionStringKey} is empty, but this node is Central with "
+ $"{ReplicationModeKey}=Grpc. Central in Grpc mode requires the "
+ "SHARED SQL-Server secret store — an independent local store per "
+ "central node is the divergence scadaproj#4 recorded (a hub "
+ "failover would 'succeed' against an empty manifest). Supply the "
+ "connection string via appsettings or the environment "
+ "(Secrets__SqlServer__ConnectionString).");
}
if (hubConnectionString.Contains("${secret:", StringComparison.Ordinal))
{
// The secrets store's own connection string can never be a secret
// reference: the pre-host expander needs the store this string points at
// in order to resolve it — bootstrap circularity, the same rule the class
// remarks document for the hub bearer token. The message deliberately
// does not echo the configured value: a mistyped value here could be a
// pasted real credential.
throw new InvalidOperationException(
$"{HubConnectionStringKey} contains a ${{secret:}} reference. The "
+ "secret store's own connection string can never be a secret "
+ "reference — the pre-host expander needs the store this string "
+ "points at in order to resolve it (bootstrap circularity, the "
+ "same rule as Secrets:GrpcHub:BearerToken). Supply a literal "
+ "value via appsettings or the environment "
+ "(Secrets__SqlServer__ConnectionString).");
}
// Central's store is the SHARED SQL-Server store: registers
// SqlServerSecretStore as ISecretStore BEFORE calling AddZbSecrets internally,
// displacing SQLite — do NOT also call AddZbSecrets here. One copy of every
// row means both central hubs serve identical manifests by construction, which
// is what makes site-side hub failover between them safe.
services.AddZbSecretsSqlServerStore(
config, SecretsSectionPath, SqlServerSectionPath);
// Throws when Secrets:GrpcHub:BearerToken is unset. Deliberately not caught: a
// hub that starts and refuses every follower looks like a network fault from
// the site end, and would be discovered as a stale secret rather than as a
@@ -207,6 +267,13 @@ public static class SecretsRegistration
services.AddZbSecretsGrpcHub(config, GrpcHubSectionPath);
break;
case SecretsNodeRole.Site:
// A site keeps its LOCAL SQLite store — it resolves secrets straight through a
// WAN outage, and sites talk to central, never to central's database. Neither
// gRPC extension registers the local store — pull-only means local writes
// never leave the node, so there is nothing to decorate and the store stays
// exactly as configured. Register it first; the sweep resolves it.
services.AddZbSecrets(config, SecretsSectionPath);
// Throws when Secrets:GrpcHub:Endpoint or :BearerToken is unset. Same reasoning.
services.AddZbSecretsGrpcHubClient(config, GrpcHubSectionPath);
break;
@@ -24,7 +24,7 @@
"SchemaName": "zbsecrets",
"SyncInterval": "00:00:30",
"SyncOnStartup": true,
"_comment": "Hub connstr must be seeded in the LOCAL store (or supplied via env) - it cannot come from the hub itself. Enable via Secrets:Replication:Enabled + Mode=SqlServer."
"_comment": "Two modes read this section. Mode=SqlServer: the hub connstr must be seeded in the LOCAL store (or supplied via env) - it cannot come from the hub itself. Mode=Grpc: REQUIRED on CENTRAL, where it points BOTH central nodes at ONE shared SQL-Server secret store so the pair cannot diverge (scadaproj#4 - an independent store per central node let one hub serve an authenticated EMPTY manifest); it must be a LITERAL or environment value (Secrets__SqlServer__ConnectionString), never a ${secret:} reference - the pre-host expander needs this store to resolve references, so the reference could never resolve (bootstrap circularity; registration rejects it). SITES leave it empty in every mode - sites talk to central, never to central's database."
},
"GrpcHub": {
"_comment": "Read ONLY when Secrets:Replication:Enabled is true AND Mode=Grpc; inert otherwise. ONE section, both halves: CENTRAL reads BearerToken + MaxNamesPerRequest and hosts the hub on its CentralGrpcPort h2c listener (default 8083, alongside CentralControlService); a SITE reads Endpoint + BearerToken + the sweep timings and pulls. Replication is pull-only by wire contract - the proto has no write RPC - so secrets originate at central and a site cannot push. FAIL-CLOSED: unlike SqlServer mode there is no local-only fallback; a missing BearerToken (either role) or Endpoint (site) is a startup failure naming the key.",
@@ -6,6 +6,7 @@ using Microsoft.Extensions.Hosting;
using ZB.MOM.WW.Secrets.Abstractions;
using ZB.MOM.WW.Secrets.Replication;
using ZB.MOM.WW.Secrets.Replicator.Grpc;
using ZB.MOM.WW.Secrets.Replicator.SqlServer;
using ZB.MOM.WW.Secrets.Sqlite;
namespace ZB.MOM.WW.ScadaBridge.Host.Tests;
@@ -77,6 +78,16 @@ public class SecretsReplicationWiringTests
&& descriptor.ImplementationType?.Assembly.GetName().Name
== "ZB.MOM.WW.Secrets.Replicator.SqlServer";
/// <summary>
/// Any descriptor contributed by the SQL-Server replicator package, whatever its shape. The
/// package's factory-lambda registrations (its store, migrator, connection factory) carry the
/// concrete type as the SERVICE type, so checking both sides catches every registration form.
/// </summary>
private static bool IsFromSqlServerReplicatorPackage(ServiceDescriptor descriptor) =>
descriptor.ServiceType.Assembly.GetName().Name == "ZB.MOM.WW.Secrets.Replicator.SqlServer"
|| descriptor.ImplementationType?.Assembly.GetName().Name
== "ZB.MOM.WW.Secrets.Replicator.SqlServer";
/// <summary>
/// The gRPC hub package's one hosted service — the follower sweep. Asserted by assembly and
/// type NAME because <c>GrpcSecretSyncService</c> is internal to the package. The hub SERVER
@@ -87,11 +98,17 @@ public class SecretsReplicationWiringTests
service.GetType().Assembly.GetName().Name == "ZB.MOM.WW.Secrets.Replicator.Grpc"
&& service.GetType().Name == "GrpcSecretSyncService";
/// <summary>
/// The complete central shape for gRPC mode, connection string included: central's store in
/// this mode is the SHARED SQL-Server store (scadaproj#4), so a central config without a
/// connection string is not a valid variant — it is a registration failure, pinned separately.
/// </summary>
private static (string Key, string Value)[] GrpcCentralConfig() =>
[
("Secrets:Replication:Enabled", "true"),
("Secrets:Replication:Mode", "Grpc"),
("Secrets:GrpcHub:BearerToken", DummyBearerToken),
("Secrets:SqlServer:ConnectionString", DummyConnectionString),
];
private static (string Key, string Value)[] GrpcSiteConfig() =>
@@ -264,8 +281,10 @@ public class SecretsReplicationWiringTests
Assert.NotNull(provider.GetService<SecretsHubAuthInterceptor>());
Assert.True(SecretsRegistration.UsesGrpcHub(config));
// The local store is still the plain one — pull-only means nothing decorates it.
Assert.IsType<SqliteSecretStore>(provider.GetRequiredService<ISecretStore>());
// Central's store is the SHARED SQL-Server store — one copy of every row, so both
// central hubs serve identical manifests by construction (scadaproj#4). Undecorated:
// pull-only means nothing publishes, so there is still nothing to wrap it in.
Assert.IsType<SqlServerSecretStore>(provider.GetRequiredService<ISecretStore>());
}
[Fact]
@@ -313,9 +332,12 @@ public class SecretsReplicationWiringTests
var services = new ServiceCollection();
services.AddLogging();
// Connection string present so the missing bearer token is the ONE defect in this config —
// the connection-string pre-check runs first and would otherwise mask it.
IConfiguration config = BuildConfig(
("Secrets:Replication:Enabled", "true"),
("Secrets:Replication:Mode", "Grpc"));
("Secrets:Replication:Mode", "Grpc"),
("Secrets:SqlServer:ConnectionString", DummyConnectionString));
var ex = Assert.Throws<InvalidOperationException>(
() => services.AddScadaBridgeSecrets(config, SecretsNodeRole.Central));
@@ -324,6 +346,91 @@ public class SecretsReplicationWiringTests
Assert.Contains("Secrets:GrpcHub:BearerToken", ex.Message, StringComparison.Ordinal);
}
/// <summary>
/// Central + gRPC + no connection string is the scadaproj#4 defect asked for by name: each
/// central node on an independent local store, one of them answering authenticated followers
/// with an EMPTY manifest. It must be a boot failure naming the key, never a quiet fallback.
/// </summary>
[Theory]
[InlineData(null)]
[InlineData("")]
[InlineData(" ")]
public void GrpcMode_Central_WithBlankConnectionString_FailsAtRegistration(string? connectionString)
{
var services = new ServiceCollection();
services.AddLogging();
(string, string)[] overrides = connectionString is null
?
[
("Secrets:Replication:Enabled", "true"),
("Secrets:Replication:Mode", "Grpc"),
("Secrets:GrpcHub:BearerToken", DummyBearerToken),
]
:
[
("Secrets:Replication:Enabled", "true"),
("Secrets:Replication:Mode", "Grpc"),
("Secrets:GrpcHub:BearerToken", DummyBearerToken),
("Secrets:SqlServer:ConnectionString", connectionString),
];
var ex = Assert.Throws<InvalidOperationException>(
() => services.AddScadaBridgeSecrets(BuildConfig(overrides), SecretsNodeRole.Central));
Assert.Contains(
SecretsRegistration.HubConnectionStringKey, ex.Message, StringComparison.Ordinal);
}
/// <summary>
/// The store's own connection string can never be a <c>${secret:}</c> reference — the pre-host
/// expander needs the store this string points at in order to resolve it (bootstrap
/// circularity, the same rule as the hub bearer token). Rejected by name rather than left to
/// fail as a malformed connection string on first use.
/// </summary>
[Fact]
public void GrpcMode_Central_WithSecretReferenceConnectionString_FailsAtRegistration()
{
var services = new ServiceCollection();
services.AddLogging();
IConfiguration config = BuildConfig(
("Secrets:Replication:Enabled", "true"),
("Secrets:Replication:Mode", "Grpc"),
("Secrets:GrpcHub:BearerToken", DummyBearerToken),
("Secrets:SqlServer:ConnectionString", "${secret:hub-connection-string}"));
var ex = Assert.Throws<InvalidOperationException>(
() => services.AddScadaBridgeSecrets(config, SecretsNodeRole.Central));
Assert.Contains(
SecretsRegistration.HubConnectionStringKey, ex.Message, StringComparison.Ordinal);
Assert.Contains("${secret:", ex.Message, StringComparison.Ordinal);
// The reference NAME is config, not secret material — but the message must still not echo
// the configured value, because a mistyped value here could be a pasted real credential.
Assert.DoesNotContain("hub-connection-string", ex.Message, StringComparison.Ordinal);
}
/// <summary>
/// The sites-don't-talk-to-SQL pin: a site node's container must contain NOTHING from the
/// SQL-Server replicator package — not just resolve a SQLite store, but hold no descriptor
/// that could ever open a connection to central's database.
/// </summary>
[Fact]
public void GrpcMode_Site_HasNoSqlServerPackageTypesInTheContainer()
{
var services = new ServiceCollection();
services.AddLogging();
services.AddScadaBridgeSecrets(BuildConfig(GrpcSiteConfig()), SecretsNodeRole.Site);
Assert.DoesNotContain(services, IsFromSqlServerReplicatorPackage);
using ServiceProvider provider = services.BuildServiceProvider(validateScopes: true);
Assert.IsType<SqliteSecretStore>(provider.GetRequiredService<ISecretStore>());
Assert.Null(provider.GetService<SqlServerSecretStore>());
}
[Fact]
public void GrpcMode_Site_WithoutEndpoint_FailsAtRegistration()
{
@@ -468,13 +575,19 @@ public class SecretsHubEndpointMappingTests
private static bool HasHubEndpoint(WebApplication app) => HubRoutes(app).Count > 0;
// Syntactically valid, deliberately unreachable — central + gRPC requires the shared
// SQL-Server store's connection string at registration; nothing here ever connects.
private const string DummyConnectionString =
"Server=unused;Database=x;Integrated Security=true;";
[Fact]
public void GrpcMode_Central_MapsTheHubEndpoint()
{
using WebApplication app = BuildCentralApp(
("Secrets:Replication:Enabled", "true"),
("Secrets:Replication:Mode", "Grpc"),
("Secrets:GrpcHub:BearerToken", DummyBearerToken));
("Secrets:GrpcHub:BearerToken", DummyBearerToken),
("Secrets:SqlServer:ConnectionString", DummyConnectionString));
Assert.True(HasHubEndpoint(app));
}
@@ -491,7 +604,8 @@ public class SecretsHubEndpointMappingTests
using WebApplication app = BuildCentralApp(
("Secrets:Replication:Enabled", "true"),
("Secrets:Replication:Mode", "Grpc"),
("Secrets:GrpcHub:BearerToken", DummyBearerToken));
("Secrets:GrpcHub:BearerToken", DummyBearerToken),
("Secrets:SqlServer:ConnectionString", DummyConnectionString));
Assert.Equal(
[HubRoutePrefix + "GetManifest", HubRoutePrefix + "GetSecrets"],