Files
ScadaBridge/src/ZB.MOM.WW.ScadaBridge.Host/SecretsRegistration.cs
T
Joseph Doherty fdc6b0c2bb chore(secrets): adopt ZB.MOM.WW.Secrets 0.6.2 and close the pre-host guard gap
0.6.x refuses a secret store whose path is relative or inside the content root,
because a store in the deployment directory is destroyed by an ordinary upgrade —
the failure that wiped the MxGateway API-key store on 2026-08-09 and read as an
auth outage rather than a deployment error.

The pin alone would not have protected this repo. Program.cs expands ${secret:}
before the host exists, composing secrets into a throwaway ServiceCollection with
no IHostEnvironment, so the guard would not run at the moment the migrator creates
the store. That composition now lives in SecretsRegistration with an explicit
content root — resolved to match what the host resolves later, including the
Windows-Service case where the pre-host CWD is still system32 — and is covered by
PreHostSecretsContentRootTests, verified by simulating the regression and
confirming it fails on the leftover file rather than on the exception.

The docker rig needed a fix too: /app/data is absolute but inside the container's
content root, so all 8 nodes would have failed to boot. Each node's data directory
is now mounted a second time at /data; same host directory, so existing stores
carry over untouched.

Verified: build clean, 29 test assemblies green (Playwright's 159 failures are the
pre-existing SEC-36 login baseline). Not yet deployed — the rig runs the old
config until someone redeploys.
2026-08-11 09:16:31 -04:00

431 lines
24 KiB
C#

using Microsoft.AspNetCore.Routing;
using Microsoft.Extensions.Hosting.WindowsServices;
using ZB.MOM.WW.Secrets.DependencyInjection;
using ZB.MOM.WW.Secrets.Replicator.Grpc.DependencyInjection;
using ZB.MOM.WW.Secrets.Replicator.SqlServer.DependencyInjection;
namespace ZB.MOM.WW.ScadaBridge.Host;
/// <summary>
/// Which half of a clustered secrets topology this node composes. Passed in by the caller rather
/// than read from configuration, because both composition roots already know statically which one
/// they are — <c>Program.cs</c> for central and <see cref="SiteServiceRegistration"/> for a site —
/// and a role read from a key could be got wrong in a way that silently turns a site into a hub.
/// </summary>
public enum SecretsNodeRole
{
/// <summary>The central cluster. Hosts the gRPC secrets hub; never sweeps from one.</summary>
Central,
/// <summary>A site cluster. Sweeps from central's hub; never hosts one.</summary>
Site,
}
/// <summary>
/// The clustered-replication transport, selected by <c>Secrets:Replication:Mode</c>.
/// </summary>
public enum SecretsReplicationMode
{
/// <summary>
/// Local store bidirectionally synced with a shared SQL-Server hub. The historical mode and
/// therefore the default, so an existing configuration that sets only
/// <c>Secrets:Replication:Enabled</c> keeps behaving exactly as it did.
/// </summary>
SqlServer,
/// <summary>
/// Local store on every node, converged by a pull-only gRPC hub that central hosts. The
/// production topology (scadaproj#3).
/// </summary>
Grpc,
}
/// <summary>
/// Single composition-root entry point for the host container's secret store, shared by the
/// central-role registrations in <c>Program.cs</c> and the site-role registrations in
/// <see cref="SiteServiceRegistration"/> so both roles wire secrets identically.
/// </summary>
/// <remarks>
/// <para>
/// Clustered replication is <b>opt-in and off by default</b>. ScadaBridge is hub-and-spoke — one
/// central cluster plus N separate site clusters — and every supported topology gives each SITE
/// node a LOCAL store, so a site keeps resolving secrets straight through a WAN outage. What
/// differs between the modes is how convergence works, and — in gRPC mode — where central's own
/// store lives.
/// </para>
/// <para>
/// <b><see cref="SecretsReplicationMode.Grpc"/> is the production topology</b> (scadaproj#3):
/// central hosts a pull-only gRPC hub and each site node sweeps it. It exists because
/// <see cref="SecretsReplicationMode.SqlServer"/> requires every site node to hold a connection
/// string to central's database, which violates ScadaBridge's standing rule that sites talk to
/// central, not to central's DB. Pull-only is a property of the wire contract rather than a
/// setting: the proto carries no write RPC, so a site cannot push even if misconfigured, and
/// secrets therefore originate at central only. <b>Central's own store in this mode is the SHARED
/// SQL-Server store</b> (<c>AddZbSecretsSqlServerStore</c>), not local SQLite: both central nodes
/// read and write ONE copy of every row, so the two hub instances serve identical manifests by
/// construction. The alternative — an independent local store per central node — is the recorded
/// defect (scadaproj#4): one central answering authenticated followers with an empty manifest,
/// turning site-side hub failover into a silent convergence stop. The accepted trade is that
/// central secret resolution now depends on SQL Server availability — acceptable because central
/// is already SQL-coupled to its core, and a SQL outage makes the hub <em>throw</em> (sites keep
/// last-known-good), never serve empty. The pre-host <c>${secret:}</c> expander in
/// <c>Program.cs</c> follows this store swap, so pre-host references resolve from the same store
/// the running node serves.
/// </para>
/// <para>
/// <b>The SQL-Server gate is not a style preference.</b> <c>AddZbSecretsSqlServerReplication</c>
/// validates its options EAGERLY at registration time, so calling it unconditionally would throw at
/// startup on every node where <c>Secrets:SqlServer:ConnectionString</c> is unset — which is every
/// dev box, every docker node and every existing deployment. Both the explicit
/// <c>Secrets:Replication:Enabled</c> flag and a non-blank connection string are therefore required
/// before the SQL-Server package is touched at all; otherwise this is byte-for-byte the plain
/// local-SQLite registration it has always been.
/// </para>
/// <para>
/// <b>The gRPC mode deliberately does NOT degrade the same way.</b> A missing connection string in
/// SQL-Server mode falls back to a local-only store with a loud warning, and that behaviour is
/// preserved untouched. gRPC mode instead fails at startup when its endpoint or bearer token is
/// missing — or, on central, when the shared-store connection string is — because the fallback is
/// the exact outcome the hub exists to prevent — a site serving secrets that silently never
/// converge with central — and because selecting the mode takes an explicit second key, so an
/// operator who typed it meant it.
/// </para>
/// <para>
/// <b>Bootstrap constraint.</b> A replication credential can never itself come from the replicated
/// set — a node cannot read the hub to learn how to reach the hub. In SQL-Server mode the hub
/// connection string must arrive from outside it: an environment variable, or a <c>${secret:}</c>
/// reference seeded in that node's own LOCAL store (the pre-host expander in <c>Program.cs</c>
/// runs against the local SQLite store before the host container exists, so such a reference does
/// resolve). In gRPC mode the rule is stricter on central: <c>Secrets:SqlServer:ConnectionString</c>
/// can never be a <c>${secret:}</c> reference at all, because on central the pre-host expander
/// itself runs against the shared SQL store (see <c>Program.cs</c>) and would need this very
/// string to reach the store that resolves it — registration rejects such a value outright. The
/// same holds for <c>Secrets:GrpcHub:BearerToken</c>, which is supplied from appsettings or the
/// environment exactly as the mesh pre-shared keys are.
/// </para>
/// </remarks>
public static class SecretsRegistration
{
/// <summary>Configuration section holding the core secrets options.</summary>
public const string SecretsSectionPath = "Secrets";
/// <summary>Configuration section holding the SQL-Server hub options.</summary>
public const string SqlServerSectionPath = "Secrets:SqlServer";
/// <summary>
/// Configuration section holding the gRPC hub options. One section serves both halves: central
/// reads <c>BearerToken</c> + <c>MaxNamesPerRequest</c> from it, a site reads <c>Endpoint</c>,
/// <c>BearerToken</c> and the sweep timings.
/// </summary>
public const string GrpcHubSectionPath = "Secrets:GrpcHub";
/// <summary>Configuration key gating clustered replication. Absent or false = off.</summary>
public const string ReplicationEnabledKey = "Secrets:Replication:Enabled";
/// <summary>
/// Configuration key selecting the replication transport. Absent or blank =
/// <see cref="SecretsReplicationMode.SqlServer"/>, the historical behaviour.
/// </summary>
public const string ReplicationModeKey = "Secrets:Replication:Mode";
/// <summary>Configuration key holding the shared SQL-Server hub connection string.</summary>
public const string HubConnectionStringKey = "Secrets:SqlServer:ConnectionString";
/// <summary>
/// Reads <see cref="ReplicationModeKey"/>. An absent or blank value is
/// <see cref="SecretsReplicationMode.SqlServer"/>; anything unrecognised throws.
/// </summary>
/// <remarks>
/// The value is validated whenever it is present, even with replication switched off. A typo
/// only surfaces at the boot that introduces it that way — deferring the check to whenever
/// somebody later flips <see cref="ReplicationEnabledKey"/> would fail a node in an unrelated
/// change, pointing at a key nobody touched.
/// </remarks>
/// <param name="config">Application configuration.</param>
/// <returns>The selected mode.</returns>
/// <exception cref="InvalidOperationException">The configured value is not a known mode.</exception>
public static SecretsReplicationMode ResolveReplicationMode(IConfiguration config)
{
ArgumentNullException.ThrowIfNull(config);
var raw = config[ReplicationModeKey];
if (string.IsNullOrWhiteSpace(raw))
{
return SecretsReplicationMode.SqlServer;
}
if (!Enum.TryParse(raw.Trim(), ignoreCase: true, out SecretsReplicationMode mode)
|| !Enum.IsDefined(mode))
{
throw new InvalidOperationException(
$"{ReplicationModeKey} is '{raw}', which is not a known secrets replication mode. "
+ $"Valid values are '{nameof(SecretsReplicationMode.SqlServer)}' and "
+ $"'{nameof(SecretsReplicationMode.Grpc)}'.");
}
return mode;
}
/// <summary>
/// Whether this node participates in the pull-only gRPC secrets hub.
/// </summary>
/// <remarks>
/// The single predicate behind BOTH the service registration and
/// <see cref="MapScadaBridgeSecretsHub"/>, so the two can never disagree. That matters more
/// than tidiness: mapping the hub endpoint without registering it would map a service whose
/// fail-closed bearer interceptor was never attached — an unauthenticated hub serving every
/// secret central holds.
/// </remarks>
/// <param name="config">Application configuration.</param>
/// <returns><see langword="true"/> when replication is enabled in gRPC mode.</returns>
public static bool UsesGrpcHub(IConfiguration config)
{
ArgumentNullException.ThrowIfNull(config);
return config.GetValue<bool>(ReplicationEnabledKey)
&& ResolveReplicationMode(config) == SecretsReplicationMode.Grpc;
}
/// <summary>
/// Resolves the content root the HOST will use, from outside the host — the Layer-A
/// <c>${secret:}</c> expander in <c>Program.cs</c> runs before any builder exists, so
/// <c>builder.Environment.ContentRootPath</c> is not available to it.
/// </summary>
/// <remarks>
/// Mirrors the framework's own resolution order. The Windows-Service branch is the one that
/// is easy to get wrong: <c>UseWindowsService</c> sets both the current directory and the
/// content root to <see cref="AppContext.BaseDirectory"/>, but only once the host is built —
/// at expander time the process working directory is still <c>C:\Windows\system32</c>, so
/// taking the current directory there would compare the store path against the wrong root.
/// </remarks>
/// <returns>The absolute content-root path the host will resolve.</returns>
public static string ResolveHostContentRoot() =>
Environment.GetEnvironmentVariable("ASPNETCORE_CONTENTROOT")
?? Environment.GetEnvironmentVariable("DOTNET_CONTENTROOT")
?? (WindowsServiceHelpers.IsWindowsService()
? AppContext.BaseDirectory
: Directory.GetCurrentDirectory());
/// <summary>
/// Composes the SQLite secrets stack for the Layer-A pre-host <c>${secret:}</c> expander.
/// </summary>
/// <remarks>
/// <para>
/// This exists as a named method rather than an inline <c>AddZbSecrets</c> call so that the
/// content-root argument is covered by a test (<c>PreHostSecretsContentRootTests</c>). The
/// argument is load-bearing and its absence is SILENT: the expander composes into a throwaway
/// <see cref="IServiceCollection"/> that has no <c>IHostEnvironment</c>, so without it the
/// under-content-root rule does not run, the migrator creates the store at the rejected path,
/// and the boot then fails once the real host registers <c>IHostEnvironment</c> — leaving an
/// empty database behind. That "the file is there, it's just empty" artifact is the confusing
/// half of the 2026-08-09 MxGateway outage the guard exists to prevent.
/// </para>
/// <para>
/// A correctly configured path shows NO symptom either way, so this cannot be verified by
/// observing a clean boot — the exposure is that the guard is not running when the store is
/// created, not that any particular path is wrong.
/// </para>
/// </remarks>
/// <param name="services">The throwaway expander service collection.</param>
/// <param name="config">Application configuration.</param>
/// <param name="contentRoot">The host's content root, from <see cref="ResolveHostContentRoot"/>.</param>
/// <returns>The same collection, for chaining.</returns>
public static IServiceCollection AddPreHostSqliteExpander(
this IServiceCollection services,
IConfiguration config,
string contentRoot)
{
ArgumentNullException.ThrowIfNull(services);
ArgumentNullException.ThrowIfNull(config);
ArgumentException.ThrowIfNullOrWhiteSpace(contentRoot);
// Do NOT drop the third argument. The 3-argument overload still compiles and silently
// reopens the gap described above; the regression would arrive disguised as removing a
// redundant parameter.
return services.AddZbSecrets(config, SecretsSectionPath, contentRoot);
}
/// <summary>
/// The one definition of "a Central node in Grpc mode has a usable shared-store connection
/// string", shared by <see cref="AddScadaBridgeSecrets"/> and the Layer-A expander in
/// <c>Program.cs</c> so the two can never diverge. The expander runs FIRST on a real boot —
/// without this sharing, a bad value would die inside <c>SqlConnection</c> with a generic
/// "initialization string" format error before the designed message ever fired.
/// </summary>
/// <param name="config">Application configuration.</param>
/// <returns>The validated connection string.</returns>
/// <exception cref="InvalidOperationException">
/// <see cref="HubConnectionStringKey"/> is blank or contains a <c>${secret:}</c> reference.
/// </exception>
internal static string EnsureCentralSharedStoreConnectionString(IConfiguration config)
{
ArgumentNullException.ThrowIfNull(config);
var hubConnectionString = config[HubConnectionStringKey];
if (string.IsNullOrWhiteSpace(hubConnectionString))
{
// Fail closed, matching the mode's philosophy: central in Grpc mode REQUIRES the
// shared SQL store. An independent local store per central node is exactly the
// divergence scadaproj#4 recorded — one central hub answered authenticated followers
// with an EMPTY manifest, so a hub failover would "succeed" against nothing and stop
// convergence silently.
throw new InvalidOperationException(
$"{HubConnectionStringKey} is empty, but this node is Central with "
+ $"{ReplicationModeKey}=Grpc. Central in Grpc mode requires the "
+ "SHARED SQL-Server secret store — an independent local store per "
+ "central node is the divergence scadaproj#4 recorded (a hub "
+ "failover would 'succeed' against an empty manifest). Supply the "
+ "connection string via appsettings or the environment "
+ "(Secrets__SqlServer__ConnectionString).");
}
if (hubConnectionString.Contains("${secret:", StringComparison.Ordinal))
{
// The secrets store's own connection string can never be a secret reference: the
// pre-host expander needs the store this string points at in order to resolve it —
// bootstrap circularity, the same rule the class remarks document for the hub bearer
// token. The message deliberately does not echo the configured value: a mistyped
// value here could be a pasted real credential.
throw new InvalidOperationException(
$"{HubConnectionStringKey} contains a ${{secret:}} reference. The "
+ "secret store's own connection string can never be a secret "
+ "reference — the pre-host expander needs the store this string "
+ "points at in order to resolve it (bootstrap circularity, the "
+ "same rule as Secrets:GrpcHub:BearerToken). Supply a literal "
+ "value via appsettings or the environment "
+ "(Secrets__SqlServer__ConnectionString).");
}
return hubConnectionString;
}
/// <summary>
/// Registers the host container's secret store: a plain local SQLite store by default, or a
/// replicating topology when <see cref="ReplicationEnabledKey"/> is set — a shared SQL-Server
/// hub, or the pull-only gRPC hub whose half is chosen by <paramref name="role"/>. In gRPC
/// mode a central node's store is the SHARED SQL-Server store, and registration fails closed
/// when <see cref="HubConnectionStringKey"/> is blank or is a <c>${secret:}</c> reference.
/// </summary>
/// <param name="services">The service collection to register into.</param>
/// <param name="config">Application configuration for options binding.</param>
/// <param name="role">Which half of a clustered topology this node composes.</param>
/// <returns>The same <paramref name="services"/> instance, for chaining.</returns>
public static IServiceCollection AddScadaBridgeSecrets(
this IServiceCollection services,
IConfiguration config,
SecretsNodeRole role)
{
ArgumentNullException.ThrowIfNull(services);
ArgumentNullException.ThrowIfNull(config);
// UsesGrpcHub is the single predicate shared with MapScadaBridgeSecretsHub — the branch
// condition must never be re-derived here, or registration and endpoint mapping could
// disagree about whether the hub is on.
if (UsesGrpcHub(config))
{
switch (role)
{
case SecretsNodeRole.Central:
// Runs BEFORE any registration so the boot failure names the real problem
// instead of surfacing later as a package validation message or a
// malformed-connection-string fault on first use. On a real boot the Layer-A
// expander in Program.cs has already run the same check even earlier.
EnsureCentralSharedStoreConnectionString(config);
// Central's store is the SHARED SQL-Server store: registers
// SqlServerSecretStore as ISecretStore BEFORE calling AddZbSecrets internally,
// displacing SQLite — do NOT also call AddZbSecrets here. One copy of every
// row means both central hubs serve identical manifests by construction, which
// is what makes site-side hub failover between them safe.
services.AddZbSecretsSqlServerStore(
config, SecretsSectionPath, SqlServerSectionPath);
// Throws when Secrets:GrpcHub:BearerToken is unset. Deliberately not caught: a
// hub that starts and refuses every follower looks like a network fault from
// the site end, and would be discovered as a stale secret rather than as a
// boot failure.
services.AddZbSecretsGrpcHub(config, GrpcHubSectionPath);
break;
case SecretsNodeRole.Site:
// A site keeps its LOCAL SQLite store — it resolves secrets straight through a
// WAN outage, and sites talk to central, never to central's database. Neither
// gRPC extension registers the local store — pull-only means local writes
// never leave the node, so there is nothing to decorate and the store stays
// exactly as configured. Register it first; the sweep resolves it.
services.AddZbSecrets(config, SecretsSectionPath);
// Throws when Secrets:GrpcHub:Endpoint or :BearerToken is unset. Same reasoning.
services.AddZbSecretsGrpcHubClient(config, GrpcHubSectionPath);
break;
default:
throw new ArgumentOutOfRangeException(nameof(role), role,
"Unknown SecretsNodeRole — a new role must decide explicitly which half " +
"of the gRPC hub it composes rather than fall into either branch.");
}
return services;
}
// Reached with the mode already validated (UsesGrpcHub resolves it on every path), so a
// false answer here means SqlServer mode or replication off — never an unvetted mode value.
var enabled = config.GetValue<bool>(ReplicationEnabledKey);
var connectionString = config[HubConnectionStringKey];
if (enabled && !string.IsNullOrWhiteSpace(connectionString))
{
// Registers the validated hub options, decorates ISecretStore with
// ReplicatingSecretStore, and adds the hub-schema migration + bidirectional sweep
// hosted services. Calls AddZbSecrets internally — do NOT also call it here.
return services.AddZbSecretsSqlServerReplication(
config, SecretsSectionPath, SqlServerSectionPath);
}
if (enabled)
{
// Asked for replication but gave nothing to replicate against. Falling back to the
// local-only store keeps the node up, but it is silently NOT participating in the
// cluster, so say so loudly rather than letting it look healthy.
Serilog.Log.Warning(
"{Key} is true but {ConnKey} is empty — clustered secret replication is DISABLED "
+ "and this node's secrets are local-only. The hub connection string cannot come "
+ "from the hub itself; supply it via environment variable or seed it in this "
+ "node's local store.",
ReplicationEnabledKey,
HubConnectionStringKey);
}
return services.AddZbSecrets(config, SecretsSectionPath);
}
/// <summary>
/// Maps the central-hosted secrets hub endpoint, but only on a node whose configuration
/// actually registered it. A no-op otherwise.
/// </summary>
/// <remarks>
/// Central-only by construction: <see cref="AddScadaBridgeSecrets"/> registers the hub service
/// and its bearer interceptor solely for <see cref="SecretsNodeRole.Central"/>, and a site node
/// never reaches this call. The endpoint lands on the shared endpoint routing, so it is served
/// on the dedicated h2c control-plane listener (<c>ScadaBridge:Node:CentralGrpcPort</c>,
/// default 8083) — the same listener and the same addressing convention sites already use to
/// reach <c>CentralControlService</c>.
/// </remarks>
/// <param name="endpoints">The endpoint route builder.</param>
/// <param name="config">Application configuration.</param>
/// <returns>The same <paramref name="endpoints"/> instance, for chaining.</returns>
public static IEndpointRouteBuilder MapScadaBridgeSecretsHub(
this IEndpointRouteBuilder endpoints,
IConfiguration config)
{
ArgumentNullException.ThrowIfNull(endpoints);
ArgumentNullException.ThrowIfNull(config);
if (!UsesGrpcHub(config))
{
return endpoints;
}
return endpoints.MapZbSecretsHub();
}
}