feat(grpc): PSK-authenticate the site gRPC control plane; drop the vestigial management receptionist registration

Phase 0 of the ClusterClient→gRPC migration
(docs/plans/2026-07-22-clusterclient-to-grpc-plan.md). Standalone hardening: it
closes a gap that exists today and is a precondition for moving command/control
onto gRPC in later phases.

T0.1 — delete the ManagementActor ClusterClientReceptionist registration.
It was built for an out-of-cluster CLI that was never written: the shipped CLI
speaks HTTP Basic to /management, which asks the actor in-process through
ManagementActorHolder. Nothing in the repo ever sent to /user/management. The
actor still runs there; only the cross-boundary advertisement is gone. Six
documents claimed the CLI used ClusterClient — including the CLI's own README
"Architecture Notes" — and are corrected here rather than left to rot.

T0.2 — record, do not port, the dead integration-routing path.
IntegrationCallRequest is unwired at BOTH ends: RouteIntegrationCallAsync has
zero callers anywhere, and RegisterLocalHandler(Integration, …) appears only in
a test, so production always answers "Integration handler not available". It is
excluded from the gRPC contract (28 of 29 commands migrate) rather than
enshrined on an additive-only wire format, and deleting it during a
transport migration would mix a behavioural change into a change whose whole
value is that behaviour is identical. See
docs/known-issues/2026-07-22-integration-call-routing-is-dead-code.md.

T0.3 — preshared-key authentication on SiteStreamService.
The service shipped with no auth at all: plaintext h2c, no interceptor, so
anything that could reach a site node's :8083 could open a live data stream or
read audit rows back via PullAuditEvents/PullSiteCalls. ControlPlaneAuthInterceptor
now gates /sitestream.SiteStreamService/ — modeled on LocalDbSyncAuthInterceptor
(constant-time compare, fail-closed, PermissionDenied) but gating a SET of
service prefixes so phases 1A/1B add services rather than interceptors. LocalDb
sync keeps its own separate key: it authenticates the pair partner, not central,
and collapsing the two would make a site's central-facing key also admit writes
into its database.

Keys are per site (SB-GRPC-PSK-<siteId>), never fleet-wide, so a compromised
site yields only its own. Central attaches them through ControlPlaneCredentials,
which binds CallCredentials to the channel — covering unary and streaming
uniformly, and letting the key resolve asynchronously, which a client
interceptor could not do without blocking. All three central→site channel
creation sites go through it (SiteStreamGrpcClient and both audit pull invokers);
the pull invokers' channel caches are re-keyed by (site, endpoint) because
credentials are per-site and bound to the channel.

Two decisions beyond the plan:

  * StartupValidator now requires GrpcPsk on Site nodes. The plan specified only
    the runtime gate, but fail-closed with no boot check produces a node that
    joins, answers heartbeats and reports healthy while refusing every stream,
    audit pull and telemetry ingest — silent and total. Same reasoning as the
    existing inbound API-key pepper rule.

  * Added Communication:SitePsks as a central-side key map. The plan assumed
    central would read the store, seeded via a dev KEK; the docker rig
    deliberately boots with no master key, so store-only resolution would leave
    it unable to dial its own sites. The store stays primary — it is the only
    source that can serve a site added at runtime — with the map covering
    key-less hosts and one-off pins. Neither source falling back to
    "unauthenticated" is the invariant.

T0.4 — dev keys on both rigs and tests.
34 tests. The seven that matter most exercise a real in-process gRPC stack over
TestServer: the unit tests on either side of the wire would both stay green if
the halves disagreed, and gRPC refuses call credentials on a plaintext channel
by default — the UnsafeUseInsecureChannelCallCredentials opt-in is only provable
by making a real call. They confirm correct key passes on unary AND streaming,
wrong key and no-credentials both get PermissionDenied, and an unresolvable key
fails the call with nothing reaching the service.

OPERATIONAL: a site node upgraded to this build without a key will not boot.
That includes the gitignored deploy/wonder-app-vd03/ overlay.

src/ZB.MOM.WW.ScadaBridge.Communication/CommunicationOptions.cs

@@ -41,6 +41,54 @@ public class CommunicationOptions
     /// </summary>
     public List<string> CentralContactPoints { get; set; } = new();
 
+    /// <summary>
+    /// Preshared key authenticating this node's gRPC control plane — the site↔central
+    /// boundary. On a site node this is the key its inbound gate
+    /// (<c>ControlPlaneAuthInterceptor</c>) expects on every <c>SiteStreamService</c> call, and
+    /// which central must present; central resolves the matching value per site from its own
+    /// secret store under the name <c>SB-GRPC-PSK-{siteId}</c>.
+    /// </summary>
+    /// <remarks>
+    /// <para>
+    /// In production this is supplied as <c>${secret:SB-GRPC-PSK-&lt;siteId&gt;}</c> and expanded
+    /// out of the secrets store before the host is built, so the plaintext never sits in
+    /// appsettings. Development rigs set a literal, mirroring the LocalDb replication key.
+    /// </para>
+    /// <para>
+    /// <b>Empty means closed, not open.</b> With no key set the interceptor rejects every gated
+    /// call. This is not optional configuration: a node that ships without a key serves no
+    /// streams and no audit pulls.
+    /// </para>
+    /// <para>
+    /// Distinct from <c>LocalDb:Replication:ApiKey</c>, which authenticates the pair partner for
+    /// database replication over the same listener. The two are never shared.
+    /// </para>
+    /// </remarks>
+    public string GrpcPsk { get; set; } = "";
+
+    /// <summary>
+    /// Central-side per-site gRPC preshared keys, keyed by site identifier — the mirror image
+    /// of <see cref="GrpcPsk"/>, which is the single key a site node expects on its own inbound
+    /// gate. An entry here takes precedence over the secret store for that site.
+    /// </summary>
+    /// <remarks>
+    /// <para>
+    /// <b>Why both a config map and a secret store.</b> The store is the primary source and the
+    /// only one that works for the real case: sites are added at runtime from the Central UI, so
+    /// their keys cannot be enumerated in configuration at boot, and <c>SitePskProvider</c>
+    /// resolves <c>SB-GRPC-PSK-{siteId}</c> on demand. This map covers the cases the store
+    /// cannot or should not: a development rig that runs with no master key and injects every
+    /// credential as an environment override, and an operator pinning one site's key without
+    /// touching the store. Values may themselves be <c>${secret:…}</c> references, since a map
+    /// declared in configuration IS enumerable at boot.
+    /// </para>
+    /// <para>
+    /// Absence is not a fallback to "unauthenticated" in either source — a site with no key in
+    /// the map and none in the store cannot be dialed at all.
+    /// </para>
+    /// </remarks>
+    public Dictionary<string, string> SitePsks { get; set; } = new();
+
     /// <summary>gRPC keepalive ping interval for streaming connections.</summary>
     public TimeSpan GrpcKeepAlivePingDelay { get; set; } = TimeSpan.FromSeconds(15);
 

src/ZB.MOM.WW.ScadaBridge.Communication/Grpc/ISitePskProvider.cs

@@ -0,0 +1,132 @@
+using Grpc.Core;
+using Grpc.Net.Client;
+
+namespace ZB.MOM.WW.ScadaBridge.Communication.Grpc;
+
+/// <summary>
+/// Resolves the preshared key that authenticates gRPC control-plane traffic for one site
+/// relationship. Central holds one key per site; each site holds only its own.
+/// </summary>
+/// <remarks>
+/// <para>
+/// <b>Why per-site rather than one fleet-wide key.</b> A compromised site yields only its own
+/// key, never another site's. A single shared key would be simpler to seed and strictly worse
+/// on blast radius, which is why it was rejected in the design.
+/// </para>
+/// <para>
+/// <b>Fail-closed.</b> Implementations throw when the key cannot be resolved. They must never
+/// fall back to "no key means no authentication" — that is the failure mode this whole
+/// mechanism exists to remove, and it would silently disable auth on exactly the default
+/// configuration. A dial that cannot be authenticated does not happen.
+/// </para>
+/// <para>
+/// The interface lives in Communication (not Host) because both sides need it: central's
+/// site-dialing clients live here and in AuditLog, while the implementation over
+/// <c>ISecretResolver</c> lives in Host, which owns the secrets container.
+/// </para>
+/// </remarks>
+public interface ISitePskProvider
+{
+    /// <summary>
+    /// Resolves the preshared key for <paramref name="siteId"/>, caching the result.
+    /// </summary>
+    /// <param name="siteId">Site identifier, as used in the <c>Site.SiteIdentifier</c> column.</param>
+    /// <param name="ct">Cancellation token.</param>
+    /// <returns>The preshared key. Never null or empty.</returns>
+    /// <exception cref="InvalidOperationException">
+    /// The key is not configured, not resolvable, or empty — the fail-closed path.
+    /// </exception>
+    ValueTask<string> GetAsync(string siteId, CancellationToken ct);
+
+    /// <summary>
+    /// Drops any cached key for <paramref name="siteId"/>, so the next
+    /// <see cref="GetAsync"/> re-reads the store. Called when a site is removed, and after a
+    /// key rotation.
+    /// </summary>
+    /// <param name="siteId">Site identifier whose cached key should be discarded.</param>
+    void Invalidate(string siteId);
+}
+
+/// <summary>
+/// Builds the call credentials that carry a site's preshared key (and the site's own identity)
+/// on every gRPC call central makes to that site — and, from Phase 1A, on the calls a site makes
+/// to central.
+/// </summary>
+/// <remarks>
+/// <para>
+/// <b>Why <see cref="CallCredentials.FromInterceptor(AsyncAuthInterceptor)"/> rather than a
+/// client <c>Interceptor</c>.</b> The key is resolved asynchronously from the secrets store, and
+/// this is the one extension point in gRPC that is async by design. A client interceptor would
+/// have to block on the resolve inside a synchronous <c>AsyncServerStreamingCall</c> path.
+/// Credentials also apply uniformly to unary and streaming calls, so no call site can forget one.
+/// </para>
+/// <para>
+/// <b>Why <c>UnsafeUseInsecureChannelCallCredentials</c>.</b> gRPC refuses to attach call
+/// credentials to a plaintext channel by default, precisely because a bearer token on h2c is
+/// readable and replayable by anyone on the path. That is a real and accepted limitation here:
+/// these listeners are h2c today and the boundary assumes a trusted network. The PSK raises the
+/// bar from "anyone who can reach the port" to "anyone who can read the traffic"; TLS on these
+/// listeners is the follow-on hardening and requires no change to this code.
+/// </para>
+/// </remarks>
+public static class ControlPlaneCredentials
+{
+    /// <summary>
+    /// Metadata header naming the site a call belongs to. Central needs it to pick which
+    /// per-site key to verify against; a site's own inbound gate ignores it (a site has exactly
+    /// one key). Required by central's interceptor from Phase 1A.
+    /// </summary>
+    public const string SiteHeader = "x-scadabridge-site";
+
+    /// <summary>The bearer metadata header. Lowercase — gRPC lowercases header keys on the wire.</summary>
+    public const string AuthorizationHeader = "authorization";
+
+    /// <summary>
+    /// Creates call credentials that attach <c>authorization: Bearer &lt;psk&gt;</c> and
+    /// <c>x-scadabridge-site: &lt;siteId&gt;</c> to every call.
+    /// </summary>
+    /// <param name="provider">Resolves the site's preshared key.</param>
+    /// <param name="siteId">The site this channel talks to (or, site-side, this site's own id).</param>
+    /// <returns>Call credentials for a channel bound to that site.</returns>
+    public static CallCredentials ForSite(ISitePskProvider provider, string siteId)
+    {
+        ArgumentNullException.ThrowIfNull(provider);
+        ArgumentException.ThrowIfNullOrWhiteSpace(siteId);
+
+        return CallCredentials.FromInterceptor(async (context, metadata) =>
+        {
+            // A throw here fails the call, which is the point: an unauthenticated dial must
+            // not happen. Callers classify the resulting fault the same way they classify any
+            // other — the pull clients degrade to an empty batch and log, the streaming
+            // subscribers retry.
+            var psk = await provider.GetAsync(siteId, context.CancellationToken).ConfigureAwait(false);
+            metadata.Add(AuthorizationHeader, $"Bearer {psk}");
+            metadata.Add(SiteHeader, siteId);
+        });
+    }
+
+    /// <summary>
+    /// Applies per-site call credentials to channel options, if a provider is available.
+    /// A null provider leaves the options untouched — the shape used by test-only and
+    /// default constructors that never dial a gated endpoint.
+    /// </summary>
+    /// <param name="options">Channel options being built.</param>
+    /// <param name="provider">Key provider, or null to leave the channel unauthenticated.</param>
+    /// <param name="siteId">The site this channel talks to.</param>
+    /// <returns>The same options instance, for chaining.</returns>
+    public static GrpcChannelOptions WithSiteCredentials(
+        this GrpcChannelOptions options, ISitePskProvider? provider, string? siteId)
+    {
+        ArgumentNullException.ThrowIfNull(options);
+
+        if (provider is null || string.IsNullOrWhiteSpace(siteId))
+        {
+            return options;
+        }
+
+        options.Credentials = ChannelCredentials.Create(
+            ChannelCredentials.Insecure, ForSite(provider, siteId));
+        options.UnsafeUseInsecureChannelCallCredentials = true;
+        return options;
+    }
+}

src/ZB.MOM.WW.ScadaBridge.Communication/Grpc/SiteStreamGrpcClient.cs

@@ -60,6 +60,27 @@ public class SiteStreamGrpcClient : IAsyncDisposable, IDisposable
     /// <param name="logger">Logger for diagnostics and errors.</param>
     /// <param name="options">Communication options including keepalive settings.</param>
     public SiteStreamGrpcClient(string endpoint, ILogger logger, CommunicationOptions options)
+        : this(endpoint, logger, options, pskProvider: null, siteIdentifier: null)
+    {
+    }
+
+    /// <summary>
+    /// Creates a client that authenticates every call with the site's preshared key.
+    /// This is the production shape: <c>SiteStreamService</c> is gated by
+    /// <c>ControlPlaneAuthInterceptor</c> on the site node, so a client without credentials
+    /// gets <see cref="StatusCode.PermissionDenied"/> on every call.
+    /// </summary>
+    /// <param name="endpoint">The gRPC endpoint address for the site.</param>
+    /// <param name="logger">Logger for diagnostics and errors.</param>
+    /// <param name="options">Communication options including keepalive settings.</param>
+    /// <param name="pskProvider">Resolves the site's preshared key; null leaves the channel unauthenticated.</param>
+    /// <param name="siteIdentifier">Site this channel talks to; null leaves the channel unauthenticated.</param>
+    public SiteStreamGrpcClient(
+        string endpoint,
+        ILogger logger,
+        CommunicationOptions options,
+        ISitePskProvider? pskProvider,
+        string? siteIdentifier)
     {
         Endpoint = endpoint;
         KeepAlivePingDelay = options.GrpcKeepAlivePingDelay;
@@ -72,7 +93,7 @@ public class SiteStreamGrpcClient : IAsyncDisposable, IDisposable
                 KeepAlivePingTimeout = options.GrpcKeepAlivePingTimeout,
                 KeepAlivePingPolicy = HttpKeepAlivePingPolicy.Always
             }
-        });
+        }.WithSiteCredentials(pskProvider, siteIdentifier));
         _client = new SiteStreamService.SiteStreamServiceClient(_channel);
         _logger = logger;
     }

src/ZB.MOM.WW.ScadaBridge.Communication/Grpc/SiteStreamGrpcClientFactory.cs

@@ -26,9 +26,11 @@ public class SiteStreamGrpcClientFactory : IAsyncDisposable, IDisposable
     private readonly ConcurrentDictionary<(string Site, string Endpoint), SiteStreamGrpcClient> _clients = new();
     private readonly ILoggerFactory _loggerFactory;
     private readonly CommunicationOptions _options;
+    private readonly ISitePskProvider? _pskProvider;
 
     /// <summary>
-    /// Test/default constructor — uses default <see cref="CommunicationOptions"/>.
+    /// Test/default constructor — uses default <see cref="CommunicationOptions"/> and creates
+    /// unauthenticated channels.
     /// </summary>
     /// <param name="loggerFactory">Logger factory passed to created clients.</param>
     public SiteStreamGrpcClientFactory(ILoggerFactory loggerFactory)
@@ -37,16 +39,36 @@ public class SiteStreamGrpcClientFactory : IAsyncDisposable, IDisposable
     }
 
     /// <summary>
-    /// DI constructor — flows <see cref="CommunicationOptions"/> into every created
-    /// <see cref="SiteStreamGrpcClient"/> so the configured gRPC keepalive settings
-    /// are applied rather than hard-coded defaults.
+    /// Constructor without a key provider — creates unauthenticated channels, which a gated
+    /// site will refuse. Retained for tests and for hosts that never dial a site.
     /// </summary>
     /// <param name="loggerFactory">Logger factory passed to created clients.</param>
     /// <param name="options">Communication options applied to each created client.</param>
     public SiteStreamGrpcClientFactory(ILoggerFactory loggerFactory, IOptions<CommunicationOptions> options)
+        : this(loggerFactory, options, pskProvider: null)
+    {
+    }
+
+    /// <summary>
+    /// DI constructor — flows <see cref="CommunicationOptions"/> into every created
+    /// <see cref="SiteStreamGrpcClient"/> so the configured gRPC keepalive settings are applied
+    /// rather than hard-coded defaults, and attaches the per-site preshared key that the site's
+    /// <c>ControlPlaneAuthInterceptor</c> requires.
+    /// </summary>
+    /// <param name="loggerFactory">Logger factory passed to created clients.</param>
+    /// <param name="options">Communication options applied to each created client.</param>
+    /// <param name="pskProvider">
+    /// Resolves each site's preshared key. Optional in DI so a host that registers no provider
+    /// (a site node, which never dials another site) still resolves this factory.
+    /// </param>
+    public SiteStreamGrpcClientFactory(
+        ILoggerFactory loggerFactory,
+        IOptions<CommunicationOptions> options,
+        ISitePskProvider? pskProvider)
     {
         _loggerFactory = loggerFactory;
         _options = options.Value;
+        _pskProvider = pskProvider;
     }
 
     /// <summary>
@@ -59,7 +81,7 @@ public class SiteStreamGrpcClientFactory : IAsyncDisposable, IDisposable
     /// <param name="grpcEndpoint">gRPC endpoint (second half of the cache key) the client is bound to.</param>
     /// <returns>The cached or newly-created client bound to <paramref name="grpcEndpoint"/>.</returns>
     public virtual SiteStreamGrpcClient GetOrCreate(string siteIdentifier, string grpcEndpoint) =>
-        _clients.GetOrAdd((siteIdentifier, grpcEndpoint), _ => CreateClient(grpcEndpoint));
+        _clients.GetOrAdd((siteIdentifier, grpcEndpoint), key => CreateClient(key.Site, key.Endpoint));
 
     /// <summary>
     /// Returns the cached client for <c>(site, endpoint)</c>, or <c>null</c> — never creates.
@@ -77,12 +99,13 @@ public class SiteStreamGrpcClientFactory : IAsyncDisposable, IDisposable
     /// can substitute a tracking client while still exercising the factory's real
     /// caching and disposal machinery.
     /// </summary>
+    /// <param name="siteIdentifier">Site the new client talks to; selects which preshared key it presents.</param>
     /// <param name="grpcEndpoint">gRPC endpoint the new client will connect to.</param>
     /// <returns>A new <see cref="SiteStreamGrpcClient"/> connected to <paramref name="grpcEndpoint"/>.</returns>
-    protected virtual SiteStreamGrpcClient CreateClient(string grpcEndpoint)
+    protected virtual SiteStreamGrpcClient CreateClient(string siteIdentifier, string grpcEndpoint)
     {
         var logger = _loggerFactory.CreateLogger<SiteStreamGrpcClient>();
-        return new SiteStreamGrpcClient(grpcEndpoint, logger, _options);
+        return new SiteStreamGrpcClient(grpcEndpoint, logger, _options, _pskProvider, siteIdentifier);
     }
 
     /// <summary>
@@ -99,6 +122,10 @@ public class SiteStreamGrpcClientFactory : IAsyncDisposable, IDisposable
             if (_clients.TryRemove(key, out var client))
                 await client.DisposeAsync();
         }
+
+        // Drop the cached preshared key too, so a site removed and re-added under the same
+        // identifier (with a rotated key) is not dialed with the stale one.
+        _pskProvider?.Invalidate(siteIdentifier);
     }
 
     /// <summary>

src/ZB.MOM.WW.ScadaBridge.Communication/ServiceCollectionExtensions.cs

@@ -1,5 +1,6 @@
 using Microsoft.Extensions.DependencyInjection;
 using Microsoft.Extensions.DependencyInjection.Extensions;
+using Microsoft.Extensions.Logging;
 using Microsoft.Extensions.Options;
 using ZB.MOM.WW.ScadaBridge.Communication.Grpc;
 
@@ -23,7 +24,15 @@ public static class ServiceCollectionExtensions
             ServiceDescriptor.Singleton<IValidateOptions<CommunicationOptions>, CommunicationOptionsValidator>());
 
         services.AddSingleton<CommunicationService>();
-        services.AddSingleton<SiteStreamGrpcClientFactory>();
+
+        // Explicit factory rather than AddSingleton<T>(): the ISitePskProvider dependency is
+        // optional (central registers one, a site node does not), and constructor selection
+        // over a nullable interface parameter is exactly the case the container cannot decide
+        // for itself — GetService returns null cleanly where constructor injection would throw.
+        services.AddSingleton(sp => new SiteStreamGrpcClientFactory(
+            sp.GetRequiredService<ILoggerFactory>(),
+            sp.GetRequiredService<IOptions<CommunicationOptions>>(),
+            sp.GetService<ISitePskProvider>()));
         services.AddSingleton<DebugStreamService>();
 
         // Aggregated live alarm cache (plan #10, Task 4): transient, in-memory, shared