dohertj2/ScadaBridge
1
0
Fork 0
Code Issues 2 Pull Requests Actions Packages Projects Releases Wiki Activity

feat(secrets)!: retire the gRPC secrets hub in ScadaBridge — sites get secrets via Deploy Artifacts; rig central on the shared SQL store

Browse Source
This commit is contained in:
Joseph Doherty
2026-09-28 19:52:26 -04:00
parent 52cd4fb096
commit 2bac13170c
11 changed files with 333 additions and 1181 deletions
Download Patch File Download Diff File Expand all files Collapse all files
src/ZB.MOM.WW.ScadaBridge.Host/Program.cs
+11 -62
View File
@@ -33,8 +33,6 @@ using ZB.MOM.WW.ScadaBridge.Transport;
using ZB.MOM.WW.Secrets.Abstractions;
using ZB.MOM.WW.Secrets.Configuration;
using ZB.MOM.WW.Secrets.DependencyInjection;
using ZB.MOM.WW.Secrets.Replicator.SqlServer;
using ZB.MOM.WW.Secrets.Replicator.SqlServer.DependencyInjection;
using ZB.MOM.WW.Secrets.Sqlite;
using ZB.MOM.WW.Secrets.Ui;
using ZB.MOM.WW.Telemetry;
@@ -77,31 +75,11 @@ var configuration = new ConfigurationBuilder()
// Expand ${secret:...} config references before any validator/binder sees them (Layer A).
// Throwaway provider — disposed here, shares no singletons with the host container.
//
// The expander must read the SAME store the running node will serve. On a central node in Grpc
// replication mode that store is the SHARED SQL-Server store (see SecretsRegistration,
// scadaproj#4) — an expander left on SQLite there would resolve pre-host ${secret:} references
// from a stale/empty local store, silently diverging from what the node's own hub serves. Every
// other case (sites, SqlServer mode, replication off) keeps the local SQLite path exactly as it
// always was. A blank or ${secret:}-valued connection string fails HERE, through the same
// EnsureCentralSharedStoreConnectionString check AddScadaBridgeSecrets runs — this is the
// earliest code that would otherwise hand the bad value to SqlConnection, whose generic
// "initialization string" format error would bury the designed message.
// This always reads the node's LOCAL SQLite store: Task 11 retired the gRPC secrets hub, the only
// topology that ever put a central node's store somewhere else pre-host. (Secrets:Store:Provider
// =SqlServer, the still-supported shared-central-store feature, is a post-host DI registration —
// see SecretsRegistration — and does not change what the Layer-A expander itself reads.)
//
// Two consequences worth knowing. This block runs BEFORE Serilog exists and outside the
// try/catch below, so a SQL Server outage at central boot exits with a bare stderr stack
// trace, no structured log — honest and container-restart-retryable, just unenriched; the
// SQLite path had the same window but its only failure mode was a local file. (That exit
// is real only because of the UnhandledException handler at the top of this file — the
// runtime's own crash path wedges as container PID 1, ScadaBridge#34.) And the
// role check is deliberately NOT folded into SecretsRegistration: that class refuses to
// read the node role from configuration (a config-read role could silently turn a site
// into a hub — see SecretsNodeRole), while Program.cs must read it to branch the whole
// composition anyway, so the config read stays here at the composition root.
var expanderUsesSharedSqlStore =
string.Equals(
configuration["ScadaBridge:Node:Role"], "Central", StringComparison.OrdinalIgnoreCase)
&& SecretsRegistration.UsesGrpcHub(configuration);
// Secrets 0.6.2 validates Secrets:SqlitePath as absolute AND outside the content root, finding the
// content root from IHostEnvironment — which this throwaway ServiceCollection does not have, and
// which does not exist yet anyway (the host is not built until further down). Both the resolution
@@ -109,32 +87,15 @@ var expanderUsesSharedSqlStore =
// AddPreHostSqliteExpander for why the content-root argument is load-bearing and silent when
// dropped, and PreHostSecretsContentRootTests for the regression guard.
var expanderServices = new ServiceCollection();
if (expanderUsesSharedSqlStore)
{
SecretsRegistration.EnsureCentralSharedStoreConnectionString(
configuration, $"{SecretsRegistration.ReplicationModeKey}=Grpc");
expanderServices.AddZbSecretsSqlServerStore(configuration, "Secrets");
}
else
{
expanderServices.AddPreHostSqliteExpander(
configuration, SecretsRegistration.ResolveHostContentRoot());
}
expanderServices.AddPreHostSqliteExpander(
configuration, SecretsRegistration.ResolveHostContentRoot());
#pragma warning disable ASP0000 // deliberate throwaway container
await using (var secretsProvider = expanderServices.BuildServiceProvider())
#pragma warning restore ASP0000
{
if (expanderUsesSharedSqlStore)
{
await secretsProvider.GetRequiredService<SqlServerSecretsStoreMigrator>()
.MigrateAsync(default);
}
else
{
await secretsProvider.GetRequiredService<SqliteSecretsStoreMigrator>()
.MigrateAsync(default);
}
await secretsProvider.GetRequiredService<SqliteSecretsStoreMigrator>()
.MigrateAsync(default);
var resolver = secretsProvider.GetRequiredService<ISecretResolver>();
await new SecretReferenceExpander(resolver)
@@ -740,21 +701,9 @@ try
// connection to the right pipeline by listener/protocol.
app.MapGrpcService<ZB.MOM.WW.ScadaBridge.Communication.Grpc.CentralControlGrpcService>();
// The pull-only secrets hub (scadaproj#3), on the SAME h2c listener and the same addressing
// convention — a site dials central's CentralGrpcPort, never central's database. It shares
// the listener with CentralControlService the way the site's LocalDb sync endpoint shares
// its one: two disjoint service prefixes, two independent fail-closed gates. The
// CentralControlAuthInterceptor registered on AddGrpc is prefix-scoped to
// CentralControlService and passes hub calls straight through; the hub's own
// SecretsHubAuthInterceptor — attached per-service by AddZbSecretsGrpcHub — is what gates
// it, on the shared Secrets:GrpcHub:BearerToken.
//
// A NO-OP unless this node registered the hub. That is not an optimisation: mapping the
// service without registering it would map a hub whose interceptor was never attached,
// i.e. an unauthenticated endpoint serving every secret central holds. Both this call and
// the registration above route through SecretsRegistration.UsesGrpcHub so they cannot
// drift apart.
app.MapScadaBridgeSecretsHub(app.Configuration);
// The pull-only secrets hub that used to be mapped here (scadaproj#3) is retired — Task 11.
// Sites receive secrets exclusively via Deploy Artifacts; there is no hub endpoint to map
// on central and nothing left for a site to sweep. See SecretsRegistration.
app.MapStaticAssets();
app.MapCentralUI<ZB.MOM.WW.ScadaBridge.Host.Components.App>();
src/ZB.MOM.WW.ScadaBridge.Host/SecretsRegistration.cs
+76 -301
View File
@@ -1,45 +1,26 @@
using Microsoft.AspNetCore.Routing;
using Microsoft.Extensions.Hosting.WindowsServices;
using ZB.MOM.WW.Secrets.DependencyInjection;
using ZB.MOM.WW.Secrets.Replicator.Grpc.DependencyInjection;
using ZB.MOM.WW.Secrets.Replicator.SqlServer.DependencyInjection;
namespace ZB.MOM.WW.ScadaBridge.Host;
/// <summary>
/// Which half of a clustered secrets topology this node composes. Passed in by the caller rather
/// than read from configuration, because both composition roots already know statically which one
/// they are — <c>Program.cs</c> for central and <see cref="SiteServiceRegistration"/> for a site —
/// and a role read from a key could be got wrong in a way that silently turns a site into a hub.
/// Which half of a composition this node is. Passed in by the caller rather than read from
/// configuration, because both composition roots already know statically which one they are —
/// <c>Program.cs</c> for central and <see cref="SiteServiceRegistration"/> for a site — and a role
/// read from a key could be got wrong in a way that silently turns a site's secret handling into
/// central's.
/// </summary>
public enum SecretsNodeRole
{
/// <summary>The central cluster. Hosts the gRPC secrets hub; never sweeps from one.</summary>
/// <summary>The central cluster.</summary>
Central,
/// <summary>A site cluster. Sweeps from central's hub; never hosts one.</summary>
/// <summary>A site cluster.</summary>
Site,
}
/// <summary>
/// The clustered-replication transport, selected by <c>Secrets:Replication:Mode</c>.
/// </summary>
public enum SecretsReplicationMode
{
/// <summary>
/// Local store bidirectionally synced with a shared SQL-Server hub. The historical mode and
/// therefore the default, so an existing configuration that sets only
/// <c>Secrets:Replication:Enabled</c> keeps behaving exactly as it did.
/// </summary>
SqlServer,
/// <summary>
/// Local store on every node, converged by a pull-only gRPC hub that central hosts. The
/// production topology (scadaproj#3).
/// </summary>
Grpc,
}
/// <summary>
/// Single composition-root entry point for the host container's secret store, shared by the
/// central-role registrations in <c>Program.cs</c> and the site-role registrations in
@@ -47,61 +28,43 @@ public enum SecretsReplicationMode
/// </summary>
/// <remarks>
/// <para>
/// Clustered replication is <b>opt-in and off by default</b>. ScadaBridge is hub-and-spoke — one
/// central cluster plus N separate site clusters — and every supported topology gives each SITE
/// node a LOCAL store, so a site keeps resolving secrets straight through a WAN outage. What
/// differs between the modes is how convergence works, and — in gRPC mode — where central's own
/// store lives.
/// <b>Task 11 retired ALL secrets replication in ScadaBridge</b> — both the pull-only gRPC hub
/// (central hosts, sites sweep) and the shared-SQL-Server <c>ReplicatingSecretStore</c> path that
/// preceded it. ScadaBridge is hub-and-spoke — one central cluster plus N separate site clusters —
/// and every topology now gives each SITE node a permanently LOCAL, undecorated SQLite store, so a
/// site keeps resolving secrets straight through a WAN outage AND never republishes anything it
/// deletes. That second property is load-bearing, not incidental:
/// <c>ZB.MOM.WW.ScadaBridge.SiteRuntime.Secrets.DeployedSecretsMirror</c> deletes a name from the
/// local store the moment a deploy artifact stops referencing it, and a replication-decorated
/// store (the retired SQL-Server mode) would have published that delete to every other node
/// sharing the hub — turning a single site's unreferencing of a secret into a fleet-wide deletion.
/// Sites now receive secrets exclusively via Deploy Artifacts (Tasks 1–10); there is no sweep, no
/// hub, and nothing for a site to converge against but its own local file.
/// </para>
/// <para>
/// <b><see cref="SecretsReplicationMode.Grpc"/> is the production topology</b> (scadaproj#3):
/// central hosts a pull-only gRPC hub and each site node sweeps it. It exists because
/// <see cref="SecretsReplicationMode.SqlServer"/> requires every site node to hold a connection
/// string to central's database, which violates ScadaBridge's standing rule that sites talk to
/// central, not to central's DB. Pull-only is a property of the wire contract rather than a
/// setting: the proto carries no write RPC, so a site cannot push even if misconfigured, and
/// secrets therefore originate at central only. <b>Central's own store in this mode is the SHARED
/// SQL-Server store</b> (<c>AddZbSecretsSqlServerStore</c>), not local SQLite: both central nodes
/// read and write ONE copy of every row, so the two hub instances serve identical manifests by
/// construction. The alternative — an independent local store per central node — is the recorded
/// defect (scadaproj#4): one central answering authenticated followers with an empty manifest,
/// turning site-side hub failover into a silent convergence stop. The accepted trade is that
/// central secret resolution now depends on SQL Server availability — acceptable because central
/// is already SQL-coupled to its core, and a SQL outage makes the hub <em>throw</em> (sites keep
/// last-known-good), never serve empty. The pre-host <c>${secret:}</c> expander in
/// <c>Program.cs</c> follows this store swap, so pre-host references resolve from the same store
/// the running node serves.
/// <b><see cref="StoreProviderKey"/> is a SEPARATE, still-supported feature and is unaffected by
/// the retirement above.</b> A fleet that ships secrets to sites on every artifact deploy still
/// needs a clustered CENTRAL pair to see the same secret set, so <c>Secrets:Store:Provider=SqlServer</c>
/// points both central nodes at ONE shared SQL-Server database directly — no hub, no follower
/// sweep, just two nodes reading and writing the same rows. This is CENTRAL-only; a Site node
/// always ignores the key and keeps its local SQLite store.
/// </para>
/// <para>
/// <b>The SQL-Server gate is not a style preference.</b> <c>AddZbSecretsSqlServerReplication</c>
/// <b>The SQL-Server gate is not a style preference.</b> <c>AddZbSecretsSqlServerStore</c>
/// validates its options EAGERLY at registration time, so calling it unconditionally would throw at
/// startup on every node where <c>Secrets:SqlServer:ConnectionString</c> is unset — which is every
/// dev box, every docker node and every existing deployment. Both the explicit
/// <c>Secrets:Replication:Enabled</c> flag and a non-blank connection string are therefore required
/// dev box, every docker node and every existing deployment. Both
/// <c>Secrets:Store:Provider=SqlServer</c> and a non-blank connection string are therefore required
/// before the SQL-Server package is touched at all; otherwise this is byte-for-byte the plain
/// local-SQLite registration it has always been.
/// </para>
/// <para>
/// <b>The gRPC mode deliberately does NOT degrade the same way.</b> A missing connection string in
/// SQL-Server mode falls back to a local-only store with a loud warning, and that behaviour is
/// preserved untouched. gRPC mode instead fails at startup when its endpoint or bearer token is
/// missing — or, on central, when the shared-store connection string is — because the fallback is
/// the exact outcome the hub exists to prevent — a site serving secrets that silently never
/// converge with central — and because selecting the mode takes an explicit second key, so an
/// operator who typed it meant it.
/// </para>
/// <para>
/// <b>Bootstrap constraint.</b> A replication credential can never itself come from the replicated
/// set — a node cannot read the hub to learn how to reach the hub. In SQL-Server mode the hub
/// connection string must arrive from outside it: an environment variable, or a <c>${secret:}</c>
/// reference seeded in that node's own LOCAL store (the pre-host expander in <c>Program.cs</c>
/// runs against the local SQLite store before the host container exists, so such a reference does
/// resolve). In gRPC mode the rule is stricter on central: <c>Secrets:SqlServer:ConnectionString</c>
/// can never be a <c>${secret:}</c> reference at all, because on central the pre-host expander
/// itself runs against the shared SQL store (see <c>Program.cs</c>) and would need this very
/// string to reach the store that resolves it — registration rejects such a value outright. The
/// same holds for <c>Secrets:GrpcHub:BearerToken</c>, which is supplied from appsettings or the
/// environment exactly as the mesh pre-shared keys are.
/// <b>Bootstrap constraint.</b> The shared store's own connection string can never itself come from
/// that store — a node cannot read the database to learn how to reach the database. It must arrive
/// from outside it: an environment variable, or a literal value in appsettings. It can never be a
/// <c>${secret:}</c> reference, because on central the pre-host expander that would resolve such a
/// reference runs against the very store this string points at (see <c>Program.cs</c>) — the
/// reference could never resolve, so registration rejects it outright.
/// </para>
/// </remarks>
public static class SecretsRegistration
@@ -109,94 +72,21 @@ public static class SecretsRegistration
/// <summary>Configuration section holding the core secrets options.</summary>
public const string SecretsSectionPath = "Secrets";
/// <summary>Configuration section holding the SQL-Server hub options.</summary>
/// <summary>Configuration section holding the shared SQL-Server store options.</summary>
public const string SqlServerSectionPath = "Secrets:SqlServer";
/// <summary>
/// Configuration section holding the gRPC hub options. One section serves both halves: central
/// reads <c>BearerToken</c> + <c>MaxNamesPerRequest</c> from it, a site reads <c>Endpoint</c>,
/// <c>BearerToken</c> and the sweep timings.
/// </summary>
public const string GrpcHubSectionPath = "Secrets:GrpcHub";
/// <summary>Configuration key gating clustered replication. Absent or false = off.</summary>
public const string ReplicationEnabledKey = "Secrets:Replication:Enabled";
/// <summary>
/// Configuration key selecting the replication transport. Absent or blank =
/// <see cref="SecretsReplicationMode.SqlServer"/>, the historical behaviour.
/// </summary>
public const string ReplicationModeKey = "Secrets:Replication:Mode";
/// <summary>Configuration key holding the shared SQL-Server hub connection string.</summary>
/// <summary>Configuration key holding the shared SQL-Server store's connection string.</summary>
public const string HubConnectionStringKey = "Secrets:SqlServer:ConnectionString";
/// <summary>
/// Configuration key selecting a CENTRAL node's own local store provider — independent of
/// <see cref="ReplicationEnabledKey"/>/<see cref="ReplicationModeKey"/> and the gRPC hub.
/// Absent or blank = <c>Sqlite</c> (per-node, the historical default); <c>SqlServer</c> points
/// both central nodes at ONE shared SQL-Server database directly, with no hub in between —
/// for a fleet that ships secrets to sites on every artifact deploy, a clustered central needs
/// both nodes to see the same secret set, and this is the lighter-weight way to get that
/// without standing up the gRPC hub. A Site node always ignores this key.
/// Configuration key selecting a CENTRAL node's own local store provider. Absent or blank =
/// <c>Sqlite</c> (per-node, the historical default); <c>SqlServer</c> points both central
/// nodes at ONE shared SQL-Server database directly, with no hub in between — for a fleet
/// that ships secrets to sites on every artifact deploy, a clustered central needs both nodes
/// to see the same secret set. A Site node always ignores this key.
/// </summary>
public const string StoreProviderKey = "Secrets:Store:Provider";
/// <summary>
/// Reads <see cref="ReplicationModeKey"/>. An absent or blank value is
/// <see cref="SecretsReplicationMode.SqlServer"/>; anything unrecognised throws.
/// </summary>
/// <remarks>
/// The value is validated whenever it is present, even with replication switched off. A typo
/// only surfaces at the boot that introduces it that way — deferring the check to whenever
/// somebody later flips <see cref="ReplicationEnabledKey"/> would fail a node in an unrelated
/// change, pointing at a key nobody touched.
/// </remarks>
/// <param name="config">Application configuration.</param>
/// <returns>The selected mode.</returns>
/// <exception cref="InvalidOperationException">The configured value is not a known mode.</exception>
public static SecretsReplicationMode ResolveReplicationMode(IConfiguration config)
{
ArgumentNullException.ThrowIfNull(config);
var raw = config[ReplicationModeKey];
if (string.IsNullOrWhiteSpace(raw))
{
return SecretsReplicationMode.SqlServer;
}
if (!Enum.TryParse(raw.Trim(), ignoreCase: true, out SecretsReplicationMode mode)
|| !Enum.IsDefined(mode))
{
throw new InvalidOperationException(
$"{ReplicationModeKey} is '{raw}', which is not a known secrets replication mode. "
+ $"Valid values are '{nameof(SecretsReplicationMode.SqlServer)}' and "
+ $"'{nameof(SecretsReplicationMode.Grpc)}'.");
}
return mode;
}
/// <summary>
/// Whether this node participates in the pull-only gRPC secrets hub.
/// </summary>
/// <remarks>
/// The single predicate behind BOTH the service registration and
/// <see cref="MapScadaBridgeSecretsHub"/>, so the two can never disagree. That matters more
/// than tidiness: mapping the hub endpoint without registering it would map a service whose
/// fail-closed bearer interceptor was never attached — an unauthenticated hub serving every
/// secret central holds.
/// </remarks>
/// <param name="config">Application configuration.</param>
/// <returns><see langword="true"/> when replication is enabled in gRPC mode.</returns>
public static bool UsesGrpcHub(IConfiguration config)
{
ArgumentNullException.ThrowIfNull(config);
return config.GetValue<bool>(ReplicationEnabledKey)
&& ResolveReplicationMode(config) == SecretsReplicationMode.Grpc;
}
/// <summary>
/// Resolves the content root the HOST will use, from outside the host — the Layer-A
/// <c>${secret:}</c> expander in <c>Program.cs</c> runs before any builder exists, so
@@ -258,20 +148,15 @@ public static class SecretsRegistration
/// <summary>
/// The one definition of "a Central node has a usable shared-store connection string", shared
/// by every caller that puts a Central node on the shared SQL-Server store — the Grpc-hub
/// branch of <see cref="AddScadaBridgeSecrets"/>, the <see cref="StoreProviderKey"/>=SqlServer
/// branch, and the Layer-A expander in <c>Program.cs</c> — so none of them can diverge on what
/// "usable" means. The expander runs FIRST on a real boot — without this sharing, a bad value
/// would die inside <c>SqlConnection</c> with a generic "initialization string" format error
/// before the designed message ever fired.
/// by every caller that puts a Central node on the shared SQL-Server store — the
/// <see cref="StoreProviderKey"/>=SqlServer branch of <see cref="AddScadaBridgeSecrets"/> — so
/// nothing can diverge on what "usable" means.
/// </summary>
/// <param name="config">Application configuration.</param>
/// <param name="triggerContext">
/// Names the setting that put this Central node on the shared store, e.g.
/// <c>"Secrets:Replication:Mode=Grpc"</c> or <c>"Secrets:Store:Provider=SqlServer"</c> — folded
/// into the blank-connection-string message so a boot failure on either path names its own
/// real trigger instead of misdescribing the other one (a Store:Provider=SqlServer failure
/// must not claim Grpc mode is in effect, and vice versa).
/// <c>"Secrets:Store:Provider=SqlServer"</c> — folded into the blank-connection-string
/// message so a boot failure names its own real trigger.
/// </param>
/// <returns>The validated connection string.</returns>
/// <exception cref="InvalidOperationException">
@@ -287,35 +172,31 @@ public static class SecretsRegistration
if (string.IsNullOrWhiteSpace(hubConnectionString))
{
// Fail closed, matching the mode's philosophy: this configuration REQUIRES the
// shared SQL store. An independent local store per central node is exactly the
// divergence scadaproj#4 recorded — one central hub answered authenticated followers
// with an EMPTY manifest, so a hub failover would "succeed" against nothing and stop
// convergence silently.
// Fail closed: this configuration REQUIRES the shared SQL store. An independent
// local store per central node would silently drift the moment anyone edits a
// secret on only one node, then ship an inconsistent secret set to sites on
// artifact deploy.
throw new InvalidOperationException(
$"{HubConnectionStringKey} is empty, but this node is Central with "
+ $"{triggerContext}. That configuration requires the SHARED "
+ "SQL-Server secret store — an independent local store per "
+ "central node is the divergence scadaproj#4 recorded (a hub "
+ "failover would 'succeed' against an empty manifest). Supply the "
+ "connection string via appsettings or the environment "
+ "(Secrets__SqlServer__ConnectionString).");
+ "central node would silently drift between the two nodes. "
+ "Supply the connection string via appsettings or the "
+ "environment (Secrets__SqlServer__ConnectionString).");
}
if (hubConnectionString.Contains("${secret:", StringComparison.Ordinal))
{
// The secrets store's own connection string can never be a secret reference: the
// pre-host expander needs the store this string points at in order to resolve it —
// bootstrap circularity, the same rule the class remarks document for the hub bearer
// token. The message deliberately does not echo the configured value: a mistyped
// value here could be a pasted real credential.
// bootstrap circularity. The message deliberately does not echo the configured
// value: a mistyped value here could be a pasted real credential.
throw new InvalidOperationException(
$"{HubConnectionStringKey} contains a ${{secret:}} reference. The "
+ "secret store's own connection string can never be a secret "
+ "reference — the pre-host expander needs the store this string "
+ "points at in order to resolve it (bootstrap circularity, the "
+ "same rule as Secrets:GrpcHub:BearerToken). Supply a literal "
+ "value via appsettings or the environment "
+ "points at in order to resolve it (bootstrap circularity). "
+ "Supply a literal value via appsettings or the environment "
+ "(Secrets__SqlServer__ConnectionString).");
}
@@ -323,15 +204,21 @@ public static class SecretsRegistration
}
/// <summary>
/// Registers the host container's secret store: a plain local SQLite store by default, or a
/// replicating topology when <see cref="ReplicationEnabledKey"/> is set — a shared SQL-Server
/// hub, or the pull-only gRPC hub whose half is chosen by <paramref name="role"/>. In gRPC
/// mode a central node's store is the SHARED SQL-Server store, and registration fails closed
/// when <see cref="HubConnectionStringKey"/> is blank or is a <c>${secret:}</c> reference.
/// Registers the host container's secret store: a plain local SQLite store, or — on Central
/// only, with <see cref="StoreProviderKey"/> set to <c>SqlServer</c> — the shared SQL-Server
/// store both central nodes point at directly. Registration fails closed when
/// <see cref="HubConnectionStringKey"/> is blank or is a <c>${secret:}</c> reference.
/// </summary>
/// <remarks>
/// A Site node (<paramref name="role"/> = <see cref="SecretsNodeRole.Site"/>) always resolves
/// to the plain local SQLite store — there is no replicating or decorated store left for a
/// site to be given. This is deliberate (Task 11): <c>DeployedSecretsMirror</c> deletes a name
/// from the local store directly, and a replication decorator would have published that
/// delete fleet-wide.
/// </remarks>
/// <param name="services">The service collection to register into.</param>
/// <param name="config">Application configuration for options binding.</param>
/// <param name="role">Which half of a clustered topology this node composes.</param>
/// <param name="role">Which half of the composition this node is.</param>
/// <returns>The same <paramref name="services"/> instance, for chaining.</returns>
public static IServiceCollection AddScadaBridgeSecrets(
this IServiceCollection services,
@@ -341,135 +228,23 @@ public static class SecretsRegistration
ArgumentNullException.ThrowIfNull(services);
ArgumentNullException.ThrowIfNull(config);
// Secrets:Store:Provider=SqlServer on Central, BEFORE the replication branch below and
// gated OFF whenever clustered replication is enabled (that is a different, existing
// feature — the gRPC hub or SQL-Server replication already decide central's store in that
// case). A Site never reaches this: it always keeps its local SQLite store, straight
// through a WAN outage, same as every other topology.
// Secrets:Store:Provider=SqlServer on Central only. A Site never reaches this: it always
// keeps its local SQLite store, straight through a WAN outage, same as every other
// topology, and never decorated — see the class remarks.
if (role == SecretsNodeRole.Central
&& string.Equals(config[StoreProviderKey], "SqlServer", StringComparison.OrdinalIgnoreCase)
&& !config.GetValue<bool>(ReplicationEnabledKey))
&& string.Equals(config[StoreProviderKey], "SqlServer", StringComparison.OrdinalIgnoreCase))
{
// Same fail-closed rules as the Grpc-mode central branch below: blank, or a
// ${secret:} reference (bootstrap circularity — the pre-host expander needs this
// same store to resolve it). Names ITS OWN trigger (Store:Provider=SqlServer, not
// Replication:Mode=Grpc) — this branch runs only when replication is off.
// Fail-closed rules: blank, or a ${secret:} reference (bootstrap circularity — the
// pre-host expander needs this same store to resolve it).
EnsureCentralSharedStoreConnectionString(config, $"{StoreProviderKey}=SqlServer");
// Registers the shared SQL-Server store as ISecretStore directly — no hub, no
// follower sweep. Both central nodes just point at the same database, so they serve
// identical manifests by construction, the same guarantee the Grpc-mode branch gets
// from the hub, without standing one up.
// identical manifests by construction.
return services.AddZbSecretsSqlServerStore(
config, SecretsSectionPath, SqlServerSectionPath);
}
// UsesGrpcHub is the single predicate shared with MapScadaBridgeSecretsHub — the branch
// condition must never be re-derived here, or registration and endpoint mapping could
// disagree about whether the hub is on.
if (UsesGrpcHub(config))
{
switch (role)
{
case SecretsNodeRole.Central:
// Runs BEFORE any registration so the boot failure names the real problem
// instead of surfacing later as a package validation message or a
// malformed-connection-string fault on first use. On a real boot the Layer-A
// expander in Program.cs has already run the same check even earlier.
EnsureCentralSharedStoreConnectionString(config, $"{ReplicationModeKey}=Grpc");
// Central's store is the SHARED SQL-Server store: registers
// SqlServerSecretStore as ISecretStore BEFORE calling AddZbSecrets internally,
// displacing SQLite — do NOT also call AddZbSecrets here. One copy of every
// row means both central hubs serve identical manifests by construction, which
// is what makes site-side hub failover between them safe.
services.AddZbSecretsSqlServerStore(
config, SecretsSectionPath, SqlServerSectionPath);
// Throws when Secrets:GrpcHub:BearerToken is unset. Deliberately not caught: a
// hub that starts and refuses every follower looks like a network fault from
// the site end, and would be discovered as a stale secret rather than as a
// boot failure.
services.AddZbSecretsGrpcHub(config, GrpcHubSectionPath);
break;
case SecretsNodeRole.Site:
// A site keeps its LOCAL SQLite store — it resolves secrets straight through a
// WAN outage, and sites talk to central, never to central's database. Neither
// gRPC extension registers the local store — pull-only means local writes
// never leave the node, so there is nothing to decorate and the store stays
// exactly as configured. Register it first; the sweep resolves it.
services.AddZbSecrets(config, SecretsSectionPath);
// Throws when Secrets:GrpcHub:Endpoint or :BearerToken is unset. Same reasoning.
services.AddZbSecretsGrpcHubClient(config, GrpcHubSectionPath);
break;
default:
throw new ArgumentOutOfRangeException(nameof(role), role,
"Unknown SecretsNodeRole — a new role must decide explicitly which half " +
"of the gRPC hub it composes rather than fall into either branch.");
}
return services;
}
// Reached with the mode already validated (UsesGrpcHub resolves it on every path), so a
// false answer here means SqlServer mode or replication off — never an unvetted mode value.
var enabled = config.GetValue<bool>(ReplicationEnabledKey);
var connectionString = config[HubConnectionStringKey];
if (enabled && !string.IsNullOrWhiteSpace(connectionString))
{
// Registers the validated hub options, decorates ISecretStore with
// ReplicatingSecretStore, and adds the hub-schema migration + bidirectional sweep
// hosted services. Calls AddZbSecrets internally — do NOT also call it here.
return services.AddZbSecretsSqlServerReplication(
config, SecretsSectionPath, SqlServerSectionPath);
}
if (enabled)
{
// Asked for replication but gave nothing to replicate against. Falling back to the
// local-only store keeps the node up, but it is silently NOT participating in the
// cluster, so say so loudly rather than letting it look healthy.
Serilog.Log.Warning(
"{Key} is true but {ConnKey} is empty — clustered secret replication is DISABLED "
+ "and this node's secrets are local-only. The hub connection string cannot come "
+ "from the hub itself; supply it via environment variable or seed it in this "
+ "node's local store.",
ReplicationEnabledKey,
HubConnectionStringKey);
}
return services.AddZbSecrets(config, SecretsSectionPath);
}
/// <summary>
/// Maps the central-hosted secrets hub endpoint, but only on a node whose configuration
/// actually registered it. A no-op otherwise.
/// </summary>
/// <remarks>
/// Central-only by construction: <see cref="AddScadaBridgeSecrets"/> registers the hub service
/// and its bearer interceptor solely for <see cref="SecretsNodeRole.Central"/>, and a site node
/// never reaches this call. The endpoint lands on the shared endpoint routing, so it is served
/// on the dedicated h2c control-plane listener (<c>ScadaBridge:Node:CentralGrpcPort</c>,
/// default 8083) — the same listener and the same addressing convention sites already use to
/// reach <c>CentralControlService</c>.
/// </remarks>
/// <param name="endpoints">The endpoint route builder.</param>
/// <param name="config">Application configuration.</param>
/// <returns>The same <paramref name="endpoints"/> instance, for chaining.</returns>
public static IEndpointRouteBuilder MapScadaBridgeSecretsHub(
this IEndpointRouteBuilder endpoints,
IConfiguration config)
{
ArgumentNullException.ThrowIfNull(endpoints);
ArgumentNullException.ThrowIfNull(config);
if (!UsesGrpcHub(config))
{
return endpoints;
}
return endpoints.MapZbSecretsHub();
}
}
src/ZB.MOM.WW.ScadaBridge.Host/SiteServiceRegistration.cs
+10 -9
View File
@@ -115,15 +115,16 @@ public static class SiteServiceRegistration
services.AddZbLocalDbReplication(config);
services.AddDataConnectionLayer();
// Local SQLite store by default. With Secrets:Replication:Enabled true it becomes either a
// local store synced against a shared SQL-Server hub (Mode=SqlServer) or — the production
// topology, scadaproj#3 — the FOLLOWER half of the pull-only gRPC secrets hub (Mode=Grpc),
// sweeping central over the same gRPC path the control plane already uses instead of
// holding a connection string to central's database. A site never hosts the hub; passing
// the role explicitly is what makes that a property of this composition root.
//
// Either way the node keeps a FULL local store, so it resolves secrets straight through a
// WAN outage to central and simply converges late. See SecretsRegistration.
// A Site node's secret store is ALWAYS the local SQLite store — never a replication
// decorator. Task 11 retired both the pull-only gRPC secrets hub (central hosts, sites
// sweep) and the shared-SQL-Server replication mode that preceded it. This is deliberate,
// not merely simplification: ZB.MOM.WW.ScadaBridge.SiteRuntime.Secrets.DeployedSecretsMirror
// deletes a name from the local store directly the moment a deploy artifact stops
// referencing it, and a replication-decorated store would have published that delete to
// every other node sharing the hub — turning one site's unreferencing of a secret into a
// fleet-wide deletion. Sites now receive secrets exclusively via Deploy Artifacts, so the
// node keeps a FULL local store and simply picks up whatever the next deploy ships. See
// SecretsRegistration.
services.AddScadaBridgeSecrets(config, SecretsNodeRole.Site);
// Adapter that surfaces the site id to
// StoreAndForwardService through DI WITHOUT introducing a
src/ZB.MOM.WW.ScadaBridge.Host/StartupValidator.cs
+19 -9
View File
@@ -26,6 +26,13 @@ public static class StartupValidator
var seedNodes = configuration.GetSection("ScadaBridge:Cluster:SeedNodes").Get<List<string>>();
// Secrets replication (the gRPC hub, and the SQL-Server ReplicatingSecretStore mode that
// preceded it) is retired in ScadaBridge (Task 11) — sites receive secrets exclusively via
// Deploy Artifacts, and a replicated site store would publish DeployedSecretsMirror's local
// deletes fleet-wide. Checked unconditionally (both roles): a Site used to be the gRPC
// hub's follower half, so the flag was never Central-only.
var secretsReplicationEnabled = configuration.GetValue<bool>("Secrets:Replication:Enabled");
// GrpcPort: default 8083 when absent; only fails the range rule when the key is
// present AND invalid. The out-param assignment mirrors the original so the
// resolved grpcPort feeds the cross-field rules even on a parse failure.
@@ -49,6 +56,12 @@ public static class StartupValidator
.Require("ScadaBridge:Node:RemotingPort",
_ => portValid,
"must be 1-65535")
// Retired for BOTH roles (Task 11) — a Site could set this too, as the gRPC hub's
// follower half. Deliberately NOT folded into a role-scoped block below.
.Require("Secrets:Replication:Enabled",
_ => !secretsReplicationEnabled,
"Secrets replication (the gRPC hub) is retired in ScadaBridge — sites receive "
+ "secrets with Deploy Artifacts; central uses Secrets:Store:Provider=SqlServer")
// SiteId (Site only) — note: OUTSIDE the big Site block in the original,
// so it must run before the unconditional SeedNodes-count rule.
.When(role == "Site", p => p
@@ -238,7 +251,6 @@ public static class StartupValidator
var seedNodes = configuration.GetSection("ScadaBridge:Cluster:SeedNodes").Get<List<string>>();
var provider = InboundApiKeyStoreOptions.ReadProvider(configuration);
var secretsStoreProvider = configuration["Secrets:Store:Provider"];
var secretsReplicationEnabled = configuration.GetValue<bool>("Secrets:Replication:Enabled");
var isClusteredCentral = role == "Central" && seedNodes is { Count: >= 2 };
var warnings = new List<string>();
@@ -257,15 +269,13 @@ public static class StartupValidator
// both central nodes to deploy the SAME secret set; a per-node store means each node has
// silently drifted the moment anyone edits a secret on only one of them.
//
// Gated off when Secrets:Replication:Enabled is true (review round 1, finding #1): that
// flag already puts this central node on a topology that shares state across the pair —
// the pre-existing gRPC hub (whose OWN central-side store is the shared SQL-Server store,
// not per-node SQLite — see SecretsRegistration's Grpc branch) or SQL-Server replication
// (ReplicatingSecretStore). Firing this warning there would be a false positive against
// the design's own named production topology (scadaproj#3), telling an operator to set a
// key that is already functionally moot.
// No Secrets:Replication:Enabled gate here any more (Task 11): that flag is now a hard
// Validate() failure on either role (the gRPC hub and SQL-Server secrets replication are
// both retired), so CollectWarnings never runs with it true on a config that also passed
// Validate — the old gate that suppressed this warning when replication was on is dead code
// and would otherwise mask the exact per-node-drift risk it exists to catch on any caller
// that invokes CollectWarnings directly.
if (isClusteredCentral
&& !secretsReplicationEnabled
&& !string.Equals(secretsStoreProvider, "SqlServer", StringComparison.OrdinalIgnoreCase))
{
warnings.Add(
src/ZB.MOM.WW.ScadaBridge.Host/ZB.MOM.WW.ScadaBridge.Host.csproj
+4 -4
View File
@@ -41,11 +41,11 @@
<PackageReference Include="ZB.MOM.WW.Secrets" />
<PackageReference Include="ZB.MOM.WW.Secrets.Abstractions" />
<PackageReference Include="ZB.MOM.WW.Secrets.Ui" />
<!-- Secrets:Store:Provider=SqlServer only (Task 5): points a clustered central pair at one
shared SQL-Server secret store directly, no hub. The gRPC secrets hub package this project
used to also reference (ZB.MOM.WW.Secrets.Replicator.Grpc) was retired in Task 11 — see
SecretsRegistration. -->
<PackageReference Include="ZB.MOM.WW.Secrets.Replicator.SqlServer" />
<!-- Both halves of the pull-only central secrets hub ship in one package: central
hosts the gRPC service, a site runs the sweep client. Which half a node composes
is decided at registration time by its role — see SecretsRegistration. -->
<PackageReference Include="ZB.MOM.WW.Secrets.Replicator.Grpc" />
<PackageReference Include="ZB.MOM.WW.LocalDb" />
<PackageReference Include="ZB.MOM.WW.LocalDb.Replication" />
</ItemGroup>
src/ZB.MOM.WW.ScadaBridge.Host/appsettings.json
+3 -22
View File
@@ -15,33 +15,14 @@
"RunMigrationsOnStartup": true,
"ResolveCacheTtl": "00:00:30",
"Store": {
"_comment": "CENTRAL ONLY (a Site always ignores this and keeps its local SQLite store). Independent of Replication above and the gRPC hub - no hub is stood up either way. 'Sqlite' (default) is per-node: on a clustered central, each node's secrets can silently drift the moment anyone edits one on only a single node, which then ships an inconsistent secret set to sites on artifact deploy - StartupValidator warns on a >=2-seed central left on Sqlite. 'SqlServer' points BOTH central nodes at ONE shared SQL-Server database directly (Secrets:SqlServer:ConnectionString, same fail-closed blank/${secret:} rules as Grpc mode's shared store), so the pair can never diverge without standing up the gRPC hub.",
"_comment": "CENTRAL ONLY (a Site always ignores this and keeps its local SQLite store, permanently - Task 11 retired secrets replication, so there is no decorator left for a site to be given). 'Sqlite' (default) is per-node: on a clustered central, each node's secrets can silently drift the moment anyone edits one on only a single node, which then ships an inconsistent secret set to sites on artifact deploy - StartupValidator warns on a >=2-seed central left on Sqlite. 'SqlServer' points BOTH central nodes at ONE shared SQL-Server database directly (Secrets:SqlServer:ConnectionString, fail-closed on a blank or ${secret:} value), so the pair can never diverge.",
"Provider": "Sqlite"
},
"Replication": {
"_comment": "Default-OFF pin: no deployment enables clustered secret replication by shipping this file. Mode selects the transport when Enabled is true - 'SqlServer' (the default, kept so an existing config that sets only Enabled behaves exactly as before) or 'Grpc' (the PRODUCTION topology, scadaproj#3: central hosts a pull-only hub under Secrets:GrpcHub and sites sweep it, so a site never needs a connection string to central's database). Every node in either topology must carry the SAME ZB_SECRETS_MASTER_KEY - only ciphertext crosses the wire, so a node with a different KEK fails closed on resolve with a kek_id mismatch that reads like corruption but is a deployment error.",
"Enabled": false,
"Mode": "SqlServer"
},
"_replication": "Secrets replication (the gRPC hub, and the SQL-Server ReplicatingSecretStore mode that preceded it) is retired - Task 11. Secrets:Replication:Enabled=true is now a hard startup failure on either role; sites receive secrets exclusively via Deploy Artifacts, and Secrets:Store:Provider=SqlServer (above) is how a clustered central shares one secret set.",
"SqlServer": {
"ConnectionString": "",
"SchemaName": "zbsecrets",
"SyncInterval": "00:00:30",
"SyncOnStartup": true,
"_comment": "Two modes read this section. Mode=SqlServer: the hub connstr must be seeded in the LOCAL store (or supplied via env) - it cannot come from the hub itself. Mode=Grpc: REQUIRED on CENTRAL, where it points BOTH central nodes at ONE shared SQL-Server secret store so the pair cannot diverge (scadaproj#4 - an independent store per central node let one hub serve an authenticated EMPTY manifest); it must be a LITERAL or environment value (Secrets__SqlServer__ConnectionString), never a ${secret:} reference - the pre-host expander needs this store to resolve references, so the reference could never resolve (bootstrap circularity; registration rejects it). SITES leave it empty in every mode - sites talk to central, never to central's database."
},
"GrpcHub": {
"_comment": "Read ONLY when Secrets:Replication:Enabled is true AND Mode=Grpc; inert otherwise. ONE section, both halves: CENTRAL reads BearerToken + MaxNamesPerRequest and hosts the hub on its CentralGrpcPort h2c listener (default 8083, alongside CentralControlService); a SITE reads Endpoint + BearerToken + the sweep timings and pulls. Replication is pull-only by wire contract - the proto has no write RPC - so secrets originate at central and a site cannot push. FAIL-CLOSED: unlike SqlServer mode there is no local-only fallback; a missing BearerToken (either role) or Endpoint (site) is a startup failure naming the key.",
"_bearerToken": "Shared credential every follower presents. Supply it from appsettings or the environment (Secrets__GrpcHub__BearerToken), NOT as a ${secret:...} reference - resolving that reference is what the hub exists to make possible, so it cannot come from the hub. Same rationale and same handling as the mesh pre-shared keys. Never commit a real value here; the empty default below is fail-closed, not open.",
"BearerToken": "",
"_endpoint": "SITE ONLY. Absolute http/https URI of a CENTRAL node's gRPC (h2c) port - the same address family as ScadaBridge:Communication:CentralGrpcEndpoints, e.g. 'http://central-a-host:8083'. NOT via Traefik (HTTP/1 only). With FallbackEndpoints left empty the hub client dials this SINGLE endpoint only: a sweep against a downed central-a then stalls rather than failing over - survivable because the sweep is best-effort (one warning per interval, then retry) and the site keeps serving its full local last-known-good store, but secrets stop converging until that central node returns. Listing the second central under FallbackEndpoints removes that stall.",
"Endpoint": "",
"_fallbackEndpoints": "SITE ONLY. List the SECOND central node's hub endpoint here (same URI form as Endpoint, e.g. 'http://central-b-host:8083') so a sweep fails over across the central pair instead of stalling on a downed primary. Safe ONLY because both central nodes serve ONE shared SQL secret store, so either hub answers with the same manifest - NEVER list an endpoint backed by an independent store: failing over to an emptier hub is a silent convergence stop, the exact defect scadaproj#4 recorded. Empty by default; leaving it empty keeps the pre-0.5.0 single-endpoint behavior byte-identical.",
"FallbackEndpoints": [],
"SyncInterval": "00:00:30",
"SyncOnStartup": true,
"CallDeadline": "00:00:30",
"MaxNamesPerRequest": 1000
"_comment": "Read ONLY when Secrets:Store:Provider=SqlServer: points BOTH central nodes at ONE shared SQL-Server secret store so the pair cannot diverge. Must be a LITERAL or environment value (Secrets__SqlServer__ConnectionString), never a ${secret:} reference - the pre-host expander needs this store to resolve references, so the reference could never resolve (bootstrap circularity; registration rejects it). SITES leave it empty - sites talk to central, never to central's database. Every node that handles deployed secrets must still carry the SAME ZB_SECRETS_MASTER_KEY - only ciphertext crosses the wire, so a node with a different KEK fails closed on resolve with a kek_id mismatch that reads like corruption but is a deployment error."
}
},
"Serilog": {