2ee84af1c0
Phase 0 of the ClusterClient→gRPC migration
(docs/plans/2026-07-22-clusterclient-to-grpc-plan.md). Standalone hardening: it
closes a gap that exists today and is a precondition for moving command/control
onto gRPC in later phases.
T0.1 — delete the ManagementActor ClusterClientReceptionist registration.
It was built for an out-of-cluster CLI that was never written: the shipped CLI
speaks HTTP Basic to /management, which asks the actor in-process through
ManagementActorHolder. Nothing in the repo ever sent to /user/management. The
actor still runs there; only the cross-boundary advertisement is gone. Six
documents claimed the CLI used ClusterClient — including the CLI's own README
"Architecture Notes" — and are corrected here rather than left to rot.
T0.2 — record, do not port, the dead integration-routing path.
IntegrationCallRequest is unwired at BOTH ends: RouteIntegrationCallAsync has
zero callers anywhere, and RegisterLocalHandler(Integration, …) appears only in
a test, so production always answers "Integration handler not available". It is
excluded from the gRPC contract (28 of 29 commands migrate) rather than
enshrined on an additive-only wire format, and deleting it during a
transport migration would mix a behavioural change into a change whose whole
value is that behaviour is identical. See
docs/known-issues/2026-07-22-integration-call-routing-is-dead-code.md.
T0.3 — preshared-key authentication on SiteStreamService.
The service shipped with no auth at all: plaintext h2c, no interceptor, so
anything that could reach a site node's :8083 could open a live data stream or
read audit rows back via PullAuditEvents/PullSiteCalls. ControlPlaneAuthInterceptor
now gates /sitestream.SiteStreamService/ — modeled on LocalDbSyncAuthInterceptor
(constant-time compare, fail-closed, PermissionDenied) but gating a SET of
service prefixes so phases 1A/1B add services rather than interceptors. LocalDb
sync keeps its own separate key: it authenticates the pair partner, not central,
and collapsing the two would make a site's central-facing key also admit writes
into its database.
Keys are per site (SB-GRPC-PSK-<siteId>), never fleet-wide, so a compromised
site yields only its own. Central attaches them through ControlPlaneCredentials,
which binds CallCredentials to the channel — covering unary and streaming
uniformly, and letting the key resolve asynchronously, which a client
interceptor could not do without blocking. All three central→site channel
creation sites go through it (SiteStreamGrpcClient and both audit pull invokers);
the pull invokers' channel caches are re-keyed by (site, endpoint) because
credentials are per-site and bound to the channel.
Two decisions beyond the plan:
* StartupValidator now requires GrpcPsk on Site nodes. The plan specified only
the runtime gate, but fail-closed with no boot check produces a node that
joins, answers heartbeats and reports healthy while refusing every stream,
audit pull and telemetry ingest — silent and total. Same reasoning as the
existing inbound API-key pepper rule.
* Added Communication:SitePsks as a central-side key map. The plan assumed
central would read the store, seeded via a dev KEK; the docker rig
deliberately boots with no master key, so store-only resolution would leave
it unable to dial its own sites. The store stays primary — it is the only
source that can serve a site added at runtime — with the map covering
key-less hosts and one-off pins. Neither source falling back to
"unauthenticated" is the invariant.
T0.4 — dev keys on both rigs and tests.
34 tests. The seven that matter most exercise a real in-process gRPC stack over
TestServer: the unit tests on either side of the wire would both stay green if
the halves disagreed, and gRPC refuses call credentials on a plaintext channel
by default — the UnsafeUseInsecureChannelCallCredentials opt-in is only provable
by making a real call. They confirm correct key passes on unary AND streaming,
wrong key and no-credentials both get PermissionDenied, and an unresolvable key
fails the call with nothing reaching the service.
OPERATIONAL: a site node upgraded to this build without a key will not boot.
That includes the gitignored deploy/wonder-app-vd03/ overlay.
150 lines
6.5 KiB
C#
150 lines
6.5 KiB
C#
using System.Collections.Concurrent;
|
|
using Microsoft.Extensions.Logging;
|
|
using Microsoft.Extensions.Options;
|
|
using ZB.MOM.WW.ScadaBridge.Communication;
|
|
using ZB.MOM.WW.ScadaBridge.Communication.Grpc;
|
|
using ZB.MOM.WW.Secrets.Abstractions;
|
|
|
|
namespace ZB.MOM.WW.ScadaBridge.Host;
|
|
|
|
/// <summary>
|
|
/// Central's <see cref="ISitePskProvider"/>: resolves each site's gRPC preshared key at
|
|
/// channel-build time — from <c>ScadaBridge:Communication:SitePsks</c> if the site is listed
|
|
/// there, otherwise from the secrets store under the name <c>SB-GRPC-PSK-{siteId}</c>.
|
|
/// </summary>
|
|
/// <remarks>
|
|
/// <para>
|
|
/// <b>Why resolved at runtime rather than expanded into config at boot.</b> Site nodes get their
|
|
/// single key through the pre-host <c>SecretReferenceExpander</c> — a <c>${secret:…}</c> in
|
|
/// appsettings, resolved once before the host is built. Central cannot do that: its set of sites
|
|
/// comes from the configuration database and changes while the process runs, so there is no
|
|
/// fixed list of references to expand at boot. This mirrors the pattern the MxGateway data
|
|
/// connection already uses for its per-connection API keys.
|
|
/// </para>
|
|
/// <para>
|
|
/// <b>Caching.</b> Successful resolves are cached indefinitely; a site's key changes only on
|
|
/// rotation, and rotation restarts the pair. Failures are deliberately NOT cached, so a key
|
|
/// seeded after the first (failed) dial is picked up on the next attempt instead of requiring a
|
|
/// restart. <see cref="Invalidate"/> drops an entry when a site is removed or a key is rotated
|
|
/// in place.
|
|
/// </para>
|
|
/// <para>
|
|
/// <b>Fail-closed.</b> A key absent from BOTH sources throws, and the dial that needed it fails.
|
|
/// It never degrades to an unauthenticated call. The message names the exact config key and the
|
|
/// exact secret name, so the fix is one setting or one <c>secret</c> CLI seed.
|
|
/// </para>
|
|
/// </remarks>
|
|
public sealed class SitePskProvider : ISitePskProvider
|
|
{
|
|
/// <summary>Prefix of the per-site secret name; the site identifier is appended verbatim.</summary>
|
|
public const string SecretNamePrefix = "SB-GRPC-PSK-";
|
|
|
|
private readonly ConcurrentDictionary<string, string> _cache = new(StringComparer.Ordinal);
|
|
private readonly ISecretResolver _resolver;
|
|
private readonly IOptionsMonitor<CommunicationOptions> _options;
|
|
private readonly ILogger<SitePskProvider> _logger;
|
|
|
|
/// <summary>Creates the provider over the host's secret resolver.</summary>
|
|
/// <param name="resolver">Runtime secret resolver from the host container.</param>
|
|
/// <param name="options">
|
|
/// Communication options supplying the optional <c>SitePsks</c> map. Read through an
|
|
/// <see cref="IOptionsMonitor{TOptions}"/> so a reloaded configuration is honoured on the
|
|
/// next uncached resolve rather than pinned at construction.
|
|
/// </param>
|
|
/// <param name="logger">Logger for resolution diagnostics.</param>
|
|
public SitePskProvider(
|
|
ISecretResolver resolver,
|
|
IOptionsMonitor<CommunicationOptions> options,
|
|
ILogger<SitePskProvider> logger)
|
|
{
|
|
ArgumentNullException.ThrowIfNull(resolver);
|
|
ArgumentNullException.ThrowIfNull(options);
|
|
ArgumentNullException.ThrowIfNull(logger);
|
|
|
|
_resolver = resolver;
|
|
_options = options;
|
|
_logger = logger;
|
|
}
|
|
|
|
/// <summary>Builds the secret name for a site.</summary>
|
|
/// <param name="siteId">Site identifier.</param>
|
|
/// <returns>The secret name, e.g. <c>SB-GRPC-PSK-site-a</c>.</returns>
|
|
public static string SecretNameFor(string siteId) => SecretNamePrefix + siteId;
|
|
|
|
/// <inheritdoc />
|
|
public async ValueTask<string> GetAsync(string siteId, CancellationToken ct)
|
|
{
|
|
ArgumentException.ThrowIfNullOrWhiteSpace(siteId);
|
|
|
|
if (_cache.TryGetValue(siteId, out var cached))
|
|
{
|
|
return cached;
|
|
}
|
|
|
|
// Configured map first: it is the explicit, operator-stated answer for this site, and
|
|
// it is the only source available to a host running without a secrets master key.
|
|
var configured = _options.CurrentValue.SitePsks;
|
|
if (configured.TryGetValue(siteId, out var fromConfig) && !string.IsNullOrEmpty(fromConfig))
|
|
{
|
|
_cache[siteId] = fromConfig;
|
|
return fromConfig;
|
|
}
|
|
|
|
// Otherwise the store, which is the only source that can serve a site added at runtime.
|
|
var secretName = SecretNameFor(siteId);
|
|
var value = await ResolveFromStoreAsync(secretName, siteId, ct).ConfigureAwait(false);
|
|
|
|
if (string.IsNullOrEmpty(value))
|
|
{
|
|
_logger.LogError(
|
|
"gRPC control-plane key for site {SiteId} could not be resolved: it is absent from "
|
|
+ "ScadaBridge:Communication:SitePsks and secret '{SecretName}' is missing, empty or "
|
|
+ "tombstoned. Calls to this site are refused until one of the two is set (the site "
|
|
+ "node must carry the same value in ScadaBridge:Communication:GrpcPsk).",
|
|
siteId, secretName);
|
|
throw new InvalidOperationException(
|
|
$"gRPC preshared key for site '{siteId}' could not be resolved from "
|
|
+ $"ScadaBridge:Communication:SitePsks or secret '{secretName}'.");
|
|
}
|
|
|
|
// Two concurrent first dials may both resolve; the read is idempotent and the values
|
|
// identical, so the loser simply overwrites with the same string.
|
|
_cache[siteId] = value;
|
|
return value;
|
|
}
|
|
|
|
/// <summary>
|
|
/// Reads the key from the secrets store, treating a store fault as "not found" rather than
|
|
/// letting it propagate. A host with no master key configured (the development rig) throws
|
|
/// from the resolver; that must produce the same clear "key not configured" error as a
|
|
/// missing secret, not an opaque cryptographic one.
|
|
/// </summary>
|
|
private async Task<string?> ResolveFromStoreAsync(string secretName, string siteId, CancellationToken ct)
|
|
{
|
|
try
|
|
{
|
|
return await _resolver.GetAsync(new SecretName(secretName), ct).ConfigureAwait(false);
|
|
}
|
|
catch (OperationCanceledException)
|
|
{
|
|
throw;
|
|
}
|
|
catch (Exception ex)
|
|
{
|
|
_logger.LogDebug(ex,
|
|
"Secret store lookup for '{SecretName}' (site {SiteId}) faulted; treating as not found.",
|
|
secretName, siteId);
|
|
return null;
|
|
}
|
|
}
|
|
|
|
/// <inheritdoc />
|
|
public void Invalidate(string siteId)
|
|
{
|
|
if (!string.IsNullOrWhiteSpace(siteId))
|
|
{
|
|
_cache.TryRemove(siteId, out _);
|
|
}
|
|
}
|
|
}
|