feat(grpc): PSK-authenticate the site gRPC control plane; drop the vestigial management receptionist registration

Phase 0 of the ClusterClient→gRPC migration (docs/plans/2026-07-22-clusterclient-to-grpc-plan.md). Standalone hardening: it closes a gap that exists today and is a precondition for moving command/control onto gRPC in later phases. T0.1 — delete the ManagementActor ClusterClientReceptionist registration. It was built for an out-of-cluster CLI that was never written: the shipped CLI speaks HTTP Basic to /management, which asks the actor in-process through ManagementActorHolder. Nothing in the repo ever sent to /user/management. The actor still runs there; only the cross-boundary advertisement is gone. Six documents claimed the CLI used ClusterClient — including the CLI's own README "Architecture Notes" — and are corrected here rather than left to rot. T0.2 — record, do not port, the dead integration-routing path. IntegrationCallRequest is unwired at BOTH ends: RouteIntegrationCallAsync has zero callers anywhere, and RegisterLocalHandler(Integration, …) appears only in a test, so production always answers "Integration handler not available". It is excluded from the gRPC contract (28 of 29 commands migrate) rather than enshrined on an additive-only wire format, and deleting it during a transport migration would mix a behavioural change into a change whose whole value is that behaviour is identical. See docs/known-issues/2026-07-22-integration-call-routing-is-dead-code.md. T0.3 — preshared-key authentication on SiteStreamService. The service shipped with no auth at all: plaintext h2c, no interceptor, so anything that could reach a site node's :8083 could open a live data stream or read audit rows back via PullAuditEvents/PullSiteCalls. ControlPlaneAuthInterceptor now gates /sitestream.SiteStreamService/ — modeled on LocalDbSyncAuthInterceptor (constant-time compare, fail-closed, PermissionDenied) but gating a SET of service prefixes so phases 1A/1B add services rather than interceptors. LocalDb sync keeps its own separate key: it authenticates the pair partner, not central, and collapsing the two would make a site's central-facing key also admit writes into its database. Keys are per site (SB-GRPC-PSK-<siteId>), never fleet-wide, so a compromised site yields only its own. Central attaches them through ControlPlaneCredentials, which binds CallCredentials to the channel — covering unary and streaming uniformly, and letting the key resolve asynchronously, which a client interceptor could not do without blocking. All three central→site channel creation sites go through it (SiteStreamGrpcClient and both audit pull invokers); the pull invokers' channel caches are re-keyed by (site, endpoint) because credentials are per-site and bound to the channel. Two decisions beyond the plan: * StartupValidator now requires GrpcPsk on Site nodes. The plan specified only the runtime gate, but fail-closed with no boot check produces a node that joins, answers heartbeats and reports healthy while refusing every stream, audit pull and telemetry ingest — silent and total. Same reasoning as the existing inbound API-key pepper rule. * Added Communication:SitePsks as a central-side key map. The plan assumed central would read the store, seeded via a dev KEK; the docker rig deliberately boots with no master key, so store-only resolution would leave it unable to dial its own sites. The store stays primary — it is the only source that can serve a site added at runtime — with the map covering key-less hosts and one-off pins. Neither source falling back to "unauthenticated" is the invariant. T0.4 — dev keys on both rigs and tests. 34 tests. The seven that matter most exercise a real in-process gRPC stack over TestServer: the unit tests on either side of the wire would both stay green if the halves disagreed, and gRPC refuses call credentials on a plaintext channel by default — the UnsafeUseInsecureChannelCallCredentials opt-in is only provable by making a real call. They confirm correct key passes on unary AND streaming, wrong key and no-credentials both get PermissionDenied, and an unresolvable key fails the call with nothing reaching the service. OPERATIONAL: a site node upgraded to this build without a key will not boot. That includes the gitignored deploy/wonder-app-vd03/ overlay.
2026-07-22 17:51:09 -04:00
parent f1ad967083
commit 2ee84af1c0
45 changed files with 1913 additions and 69 deletions
@@ -145,7 +145,7 @@ public sealed class GrpcPullAuditEventsClient : IPullAuditEventsClient
    {
        try
        {
-            var reply = await _invoker.InvokeAsync(endpoint, request, ct).ConfigureAwait(false);
+            var reply = await _invoker.InvokeAsync(siteId, endpoint, request, ct).ConfigureAwait(false);
            return (reply, false);
        }
        catch (RpcException ex) when (IsTolerable(ex.StatusCode))
@@ -226,11 +226,17 @@ public sealed class GrpcPullAuditEventsClient : IPullAuditEventsClient
        /// May throw <see cref="RpcException"/> / <see cref="HttpRequestException"/>
        /// on transport faults — the caller classifies and swallows tolerable ones.
        /// </summary>
+        /// <param name="siteId">
+        /// The site being pulled from. Selects which preshared key the call presents —
+        /// <c>PullAuditEvents</c> is gated by the site's <c>ControlPlaneAuthInterceptor</c>, and
+        /// keys are per-site, so the endpoint alone is not enough to authenticate.
+        /// </param>
        /// <param name="endpoint">The site gRPC authority (e.g. <c>http://site-a:8083</c>).</param>
        /// <param name="request">The wire-format pull request.</param>
        /// <param name="ct">Cancellation token.</param>
        /// <returns>The wire-format pull response.</returns>
-        Task<ProtoPullResponse> InvokeAsync(string endpoint, ProtoPullRequest request, CancellationToken ct);
+        Task<ProtoPullResponse> InvokeAsync(
+            string siteId, string endpoint, ProtoPullRequest request, CancellationToken ct);
    }
 }

@@ -251,8 +257,9 @@ public sealed class GrpcPullAuditEventsClient : IPullAuditEventsClient
 public sealed class GrpcPullAuditEventsInvoker
    : GrpcPullAuditEventsClient.IPullAuditEventsInvoker, IDisposable
 {
-    private readonly ConcurrentDictionary<string, GrpcChannel> _channels = new(StringComparer.Ordinal);
+    private readonly ConcurrentDictionary<(string Site, string Endpoint), GrpcChannel> _channels = new();
    private readonly CommunicationOptions _options;
+    private readonly ISitePskProvider? _pskProvider;

    /// <summary>
    /// Creates the invoker using default <see cref="CommunicationOptions"/>.
@@ -268,15 +275,27 @@ public sealed class GrpcPullAuditEventsInvoker
    /// </summary>
    /// <param name="options">Communication options supplying gRPC keepalive timings.</param>
    public GrpcPullAuditEventsInvoker(CommunicationOptions options)
+        : this(options, pskProvider: null)
+    {
+    }
+
+    /// <summary>
+    /// Creates the invoker with per-site call credentials, the production shape: the site's
+    /// <c>ControlPlaneAuthInterceptor</c> refuses an unauthenticated <c>PullAuditEvents</c>.
+    /// </summary>
+    /// <param name="options">Communication options supplying gRPC keepalive timings.</param>
+    /// <param name="pskProvider">Resolves each site's preshared key; null dials unauthenticated.</param>
+    public GrpcPullAuditEventsInvoker(CommunicationOptions options, ISitePskProvider? pskProvider)
    {
        _options = options ?? throw new ArgumentNullException(nameof(options));
+        _pskProvider = pskProvider;
    }

    /// <inheritdoc />
    public async Task<ProtoPullResponse> InvokeAsync(
-        string endpoint, ProtoPullRequest request, CancellationToken ct)
+        string siteId, string endpoint, ProtoPullRequest request, CancellationToken ct)
    {
-        var channel = GetOrCreateChannel(endpoint);
+        var channel = GetOrCreateChannel(siteId, endpoint);
        var client = new SiteStreamService.SiteStreamServiceClient(channel);
        using var call = client.PullAuditEventsAsync(request, cancellationToken: ct);
        return await call.ResponseAsync.ConfigureAwait(false);
@@ -288,12 +307,13 @@ public sealed class GrpcPullAuditEventsInvoker
    // pool) and the loser would leak. Create-then-GetOrAdd-then-dispose-if-lost
    // mirrors SiteStreamGrpcClientFactory: only the channel actually installed
    // survives; a channel that lost the race is disposed immediately.
-    private GrpcChannel GetOrCreateChannel(string endpoint)
+    private GrpcChannel GetOrCreateChannel(string siteId, string endpoint)
    {
-        if (!_channels.TryGetValue(endpoint, out var channel))
+        var key = (siteId, endpoint);
+        if (!_channels.TryGetValue(key, out var channel))
        {
-            var created = CreateChannel(endpoint);
-            channel = _channels.GetOrAdd(endpoint, created);
+            var created = CreateChannel(siteId, endpoint);
+            channel = _channels.GetOrAdd(key, created);
            if (!ReferenceEquals(channel, created))
            {
                created.Dispose();
@@ -302,7 +322,10 @@ public sealed class GrpcPullAuditEventsInvoker
        return channel;
    }

-    private GrpcChannel CreateChannel(string endpoint) =>
+    // Keyed by (site, endpoint) rather than endpoint alone: the call credentials are bound to
+    // the channel, and they are per-site, so two sites sharing an endpoint string would
+    // otherwise share one channel carrying the first site's key.
+    private GrpcChannel CreateChannel(string siteId, string endpoint) =>
        GrpcChannel.ForAddress(endpoint, new GrpcChannelOptions
        {
            HttpHandler = new SocketsHttpHandler
@@ -311,7 +334,7 @@ public sealed class GrpcPullAuditEventsInvoker
                KeepAlivePingTimeout = _options.GrpcKeepAlivePingTimeout,
                KeepAlivePingPolicy = HttpKeepAlivePingPolicy.Always,
            },
-        });
+        }.WithSiteCredentials(_pskProvider, siteId));

    /// <summary>Disposes all cached channels.</summary>
    public void Dispose()
@@ -174,7 +174,7 @@ public sealed class GrpcPullSiteCallsClient : IPullSiteCallsClient
    {
        try
        {
-            var reply = await _invoker.InvokeAsync(endpoint, request, ct).ConfigureAwait(false);
+            var reply = await _invoker.InvokeAsync(siteId, endpoint, request, ct).ConfigureAwait(false);
            return (reply, false);
        }
        catch (RpcException ex) when (IsTolerable(ex.StatusCode))
@@ -254,11 +254,17 @@ public sealed class GrpcPullSiteCallsClient : IPullSiteCallsClient
        /// May throw <see cref="RpcException"/> / <see cref="HttpRequestException"/>
        /// on transport faults — the caller classifies and swallows tolerable ones.
        /// </summary>
+        /// <param name="siteId">
+        /// The site being pulled from. Selects which preshared key the call presents —
+        /// <c>PullSiteCalls</c> is gated by the site's <c>ControlPlaneAuthInterceptor</c>, and
+        /// keys are per-site, so the endpoint alone is not enough to authenticate.
+        /// </param>
        /// <param name="endpoint">The site gRPC authority (e.g. <c>http://site-a:8083</c>).</param>
        /// <param name="request">The wire-format pull request.</param>
        /// <param name="ct">Cancellation token.</param>
        /// <returns>The wire-format pull response.</returns>
-        Task<ProtoPullResponse> InvokeAsync(string endpoint, ProtoPullRequest request, CancellationToken ct);
+        Task<ProtoPullResponse> InvokeAsync(
+            string siteId, string endpoint, ProtoPullRequest request, CancellationToken ct);
    }
 }

@@ -277,8 +283,9 @@ public sealed class GrpcPullSiteCallsClient : IPullSiteCallsClient
 public sealed class GrpcPullSiteCallsInvoker
    : GrpcPullSiteCallsClient.IPullSiteCallsInvoker, IDisposable
 {
-    private readonly ConcurrentDictionary<string, GrpcChannel> _channels = new(StringComparer.Ordinal);
+    private readonly ConcurrentDictionary<(string Site, string Endpoint), GrpcChannel> _channels = new();
    private readonly CommunicationOptions _options;
+    private readonly ISitePskProvider? _pskProvider;

    /// <summary>Creates the invoker using default <see cref="CommunicationOptions"/>.</summary>
    public GrpcPullSiteCallsInvoker()
@@ -292,15 +299,27 @@ public sealed class GrpcPullSiteCallsInvoker
    /// </summary>
    /// <param name="options">Communication options supplying gRPC keepalive timings.</param>
    public GrpcPullSiteCallsInvoker(CommunicationOptions options)
+        : this(options, pskProvider: null)
+    {
+    }
+
+    /// <summary>
+    /// Creates the invoker with per-site call credentials, the production shape: the site's
+    /// <c>ControlPlaneAuthInterceptor</c> refuses an unauthenticated <c>PullSiteCalls</c>.
+    /// </summary>
+    /// <param name="options">Communication options supplying gRPC keepalive timings.</param>
+    /// <param name="pskProvider">Resolves each site's preshared key; null dials unauthenticated.</param>
+    public GrpcPullSiteCallsInvoker(CommunicationOptions options, ISitePskProvider? pskProvider)
    {
        _options = options ?? throw new ArgumentNullException(nameof(options));
+        _pskProvider = pskProvider;
    }

    /// <inheritdoc />
    public async Task<ProtoPullResponse> InvokeAsync(
-        string endpoint, ProtoPullRequest request, CancellationToken ct)
+        string siteId, string endpoint, ProtoPullRequest request, CancellationToken ct)
    {
-        var channel = GetOrCreateChannel(endpoint);
+        var channel = GetOrCreateChannel(siteId, endpoint);
        var client = new SiteStreamService.SiteStreamServiceClient(channel);
        using var call = client.PullSiteCallsAsync(request, cancellationToken: ct);
        return await call.ResponseAsync.ConfigureAwait(false);
@@ -310,12 +329,13 @@ public sealed class GrpcPullSiteCallsInvoker
    // concurrent first dials of the same endpoint can both build a GrpcChannel;
    // only the channel actually installed survives, the loser is disposed.
    // Mirrors SiteStreamGrpcClientFactory / GrpcPullAuditEventsInvoker.
-    private GrpcChannel GetOrCreateChannel(string endpoint)
+    private GrpcChannel GetOrCreateChannel(string siteId, string endpoint)
    {
-        if (!_channels.TryGetValue(endpoint, out var channel))
+        var key = (siteId, endpoint);
+        if (!_channels.TryGetValue(key, out var channel))
        {
-            var created = CreateChannel(endpoint);
-            channel = _channels.GetOrAdd(endpoint, created);
+            var created = CreateChannel(siteId, endpoint);
+            channel = _channels.GetOrAdd(key, created);
            if (!ReferenceEquals(channel, created))
            {
                created.Dispose();
@@ -324,7 +344,10 @@ public sealed class GrpcPullSiteCallsInvoker
        return channel;
    }

-    private GrpcChannel CreateChannel(string endpoint) =>
+    // Keyed by (site, endpoint) rather than endpoint alone: the call credentials are bound to
+    // the channel, and they are per-site, so two sites sharing an endpoint string would
+    // otherwise share one channel carrying the first site's key.
+    private GrpcChannel CreateChannel(string siteId, string endpoint) =>
        GrpcChannel.ForAddress(endpoint, new GrpcChannelOptions
        {
            HttpHandler = new SocketsHttpHandler
@@ -333,7 +356,7 @@ public sealed class GrpcPullSiteCallsInvoker
                KeepAlivePingTimeout = _options.GrpcKeepAlivePingTimeout,
                KeepAlivePingPolicy = HttpKeepAlivePingPolicy.Always,
            },
-        });
+        }.WithSiteCredentials(_pskProvider, siteId));

    /// <summary>Disposes all cached channels.</summary>
    public void Dispose()
@@ -511,9 +511,14 @@ public static class ServiceCollectionExtensions
            var options = sp
                .GetService<Microsoft.Extensions.Options.IOptions<
                    ZB.MOM.WW.ScadaBridge.Communication.CommunicationOptions>>();
+            // The PSK provider is central-only and optional in DI, so GetService (not
+            // GetRequiredService): a host without one dials unauthenticated and the site
+            // refuses it, which is the fail-closed outcome we want rather than a
+            // resolution crash at composition time.
+            var psk = sp.GetService<ZB.MOM.WW.ScadaBridge.Communication.Grpc.ISitePskProvider>();
            return options is null
-                ? new GrpcPullAuditEventsInvoker()
-                : new GrpcPullAuditEventsInvoker(options.Value);
+                ? new GrpcPullAuditEventsInvoker(new ZB.MOM.WW.ScadaBridge.Communication.CommunicationOptions(), psk)
+                : new GrpcPullAuditEventsInvoker(options.Value, psk);
        });
        services.TryAddSingleton<GrpcPullAuditEventsClient.IPullAuditEventsInvoker>(
            sp => sp.GetRequiredService<GrpcPullAuditEventsInvoker>());
@@ -536,9 +541,14 @@ public static class ServiceCollectionExtensions
            var options = sp
                .GetService<Microsoft.Extensions.Options.IOptions<
                    ZB.MOM.WW.ScadaBridge.Communication.CommunicationOptions>>();
+            // The PSK provider is central-only and optional in DI, so GetService (not
+            // GetRequiredService): a host without one dials unauthenticated and the site
+            // refuses it, which is the fail-closed outcome we want rather than a
+            // resolution crash at composition time.
+            var psk = sp.GetService<ZB.MOM.WW.ScadaBridge.Communication.Grpc.ISitePskProvider>();
            return options is null
-                ? new GrpcPullSiteCallsInvoker()
-                : new GrpcPullSiteCallsInvoker(options.Value);
+                ? new GrpcPullSiteCallsInvoker(new ZB.MOM.WW.ScadaBridge.Communication.CommunicationOptions(), psk)
+                : new GrpcPullSiteCallsInvoker(options.Value, psk);
        });
        services.TryAddSingleton<GrpcPullSiteCallsClient.IPullSiteCallsInvoker>(
            sp => sp.GetRequiredService<GrpcPullSiteCallsInvoker>());
@@ -1987,9 +1987,11 @@ scadabridge --url <url> cached-call discard --site-id <string> --tracked-operati

 ## Architecture Notes

-The CLI connects to the Central cluster using Akka.NET's `ClusterClient`. It does not join the cluster — it contacts the `ClusterClientReceptionist` on one of the configured Central nodes and sends commands to the `ManagementActor` at path `/user/management`.
+The CLI connects to the Central cluster over **HTTP** (`ManagementHttpClient`), posting to the `/management` endpoints at the configured `managementUrl` — normally the Traefik load balancer, which routes to the active central node. It carries HTTP **Basic** credentials from `--username`/`--password`. Central's endpoint handler asks the `ManagementActor` in-process via `ManagementActorHolder`.

-The connection is established per-command invocation and torn down cleanly via `CoordinatedShutdown` when the command completes.
+There is no Akka dependency in the CLI at all: it does not join the cluster, does not use `ClusterClient`, and does not contact a `ClusterClientReceptionist`. (Earlier revisions of this document described a ClusterClient transport that was never built.)
+
+An `HttpClient` is created per command invocation and disposed when the command completes.

 Role enforcement is applied by the ManagementActor on the server side. The CLI authenticates against LDAP using `--username` / `--password`, resolves LDAP group memberships, then maps groups to ScadaBridge roles (Admin, Design, Deployment) via role mappings configured in the security settings. Operations require the appropriate role — for example, creating templates requires `Design`, deploying requires `Deployment`. In the test environment, use the `multi-role` user (password: `password`) which has all three roles.

@@ -41,6 +41,54 @@ public class CommunicationOptions
    /// </summary>
    public List<string> CentralContactPoints { get; set; } = new();

+    /// <summary>
+    /// Preshared key authenticating this node's gRPC control plane — the site↔central
+    /// boundary. On a site node this is the key its inbound gate
+    /// (<c>ControlPlaneAuthInterceptor</c>) expects on every <c>SiteStreamService</c> call, and
+    /// which central must present; central resolves the matching value per site from its own
+    /// secret store under the name <c>SB-GRPC-PSK-{siteId}</c>.
+    /// </summary>
+    /// <remarks>
+    /// <para>
+    /// In production this is supplied as <c>${secret:SB-GRPC-PSK-&lt;siteId&gt;}</c> and expanded
+    /// out of the secrets store before the host is built, so the plaintext never sits in
+    /// appsettings. Development rigs set a literal, mirroring the LocalDb replication key.
+    /// </para>
+    /// <para>
+    /// <b>Empty means closed, not open.</b> With no key set the interceptor rejects every gated
+    /// call. This is not optional configuration: a node that ships without a key serves no
+    /// streams and no audit pulls.
+    /// </para>
+    /// <para>
+    /// Distinct from <c>LocalDb:Replication:ApiKey</c>, which authenticates the pair partner for
+    /// database replication over the same listener. The two are never shared.
+    /// </para>
+    /// </remarks>
+    public string GrpcPsk { get; set; } = "";
+
+    /// <summary>
+    /// Central-side per-site gRPC preshared keys, keyed by site identifier — the mirror image
+    /// of <see cref="GrpcPsk"/>, which is the single key a site node expects on its own inbound
+    /// gate. An entry here takes precedence over the secret store for that site.
+    /// </summary>
+    /// <remarks>
+    /// <para>
+    /// <b>Why both a config map and a secret store.</b> The store is the primary source and the
+    /// only one that works for the real case: sites are added at runtime from the Central UI, so
+    /// their keys cannot be enumerated in configuration at boot, and <c>SitePskProvider</c>
+    /// resolves <c>SB-GRPC-PSK-{siteId}</c> on demand. This map covers the cases the store
+    /// cannot or should not: a development rig that runs with no master key and injects every
+    /// credential as an environment override, and an operator pinning one site's key without
+    /// touching the store. Values may themselves be <c>${secret:…}</c> references, since a map
+    /// declared in configuration IS enumerable at boot.
+    /// </para>
+    /// <para>
+    /// Absence is not a fallback to "unauthenticated" in either source — a site with no key in
+    /// the map and none in the store cannot be dialed at all.
+    /// </para>
+    /// </remarks>
+    public Dictionary<string, string> SitePsks { get; set; } = new();
+
    /// <summary>gRPC keepalive ping interval for streaming connections.</summary>
    public TimeSpan GrpcKeepAlivePingDelay { get; set; } = TimeSpan.FromSeconds(15);

@@ -0,0 +1,132 @@
+using Grpc.Core;
+using Grpc.Net.Client;
+
+namespace ZB.MOM.WW.ScadaBridge.Communication.Grpc;
+
+/// <summary>
+/// Resolves the preshared key that authenticates gRPC control-plane traffic for one site
+/// relationship. Central holds one key per site; each site holds only its own.
+/// </summary>
+/// <remarks>
+/// <para>
+/// <b>Why per-site rather than one fleet-wide key.</b> A compromised site yields only its own
+/// key, never another site's. A single shared key would be simpler to seed and strictly worse
+/// on blast radius, which is why it was rejected in the design.
+/// </para>
+/// <para>
+/// <b>Fail-closed.</b> Implementations throw when the key cannot be resolved. They must never
+/// fall back to "no key means no authentication" — that is the failure mode this whole
+/// mechanism exists to remove, and it would silently disable auth on exactly the default
+/// configuration. A dial that cannot be authenticated does not happen.
+/// </para>
+/// <para>
+/// The interface lives in Communication (not Host) because both sides need it: central's
+/// site-dialing clients live here and in AuditLog, while the implementation over
+/// <c>ISecretResolver</c> lives in Host, which owns the secrets container.
+/// </para>
+/// </remarks>
+public interface ISitePskProvider
+{
+    /// <summary>
+    /// Resolves the preshared key for <paramref name="siteId"/>, caching the result.
+    /// </summary>
+    /// <param name="siteId">Site identifier, as used in the <c>Site.SiteIdentifier</c> column.</param>
+    /// <param name="ct">Cancellation token.</param>
+    /// <returns>The preshared key. Never null or empty.</returns>
+    /// <exception cref="InvalidOperationException">
+    /// The key is not configured, not resolvable, or empty — the fail-closed path.
+    /// </exception>
+    ValueTask<string> GetAsync(string siteId, CancellationToken ct);
+
+    /// <summary>
+    /// Drops any cached key for <paramref name="siteId"/>, so the next
+    /// <see cref="GetAsync"/> re-reads the store. Called when a site is removed, and after a
+    /// key rotation.
+    /// </summary>
+    /// <param name="siteId">Site identifier whose cached key should be discarded.</param>
+    void Invalidate(string siteId);
+}
+
+/// <summary>
+/// Builds the call credentials that carry a site's preshared key (and the site's own identity)
+/// on every gRPC call central makes to that site — and, from Phase 1A, on the calls a site makes
+/// to central.
+/// </summary>
+/// <remarks>
+/// <para>
+/// <b>Why <see cref="CallCredentials.FromInterceptor(AsyncAuthInterceptor)"/> rather than a
+/// client <c>Interceptor</c>.</b> The key is resolved asynchronously from the secrets store, and
+/// this is the one extension point in gRPC that is async by design. A client interceptor would
+/// have to block on the resolve inside a synchronous <c>AsyncServerStreamingCall</c> path.
+/// Credentials also apply uniformly to unary and streaming calls, so no call site can forget one.
+/// </para>
+/// <para>
+/// <b>Why <c>UnsafeUseInsecureChannelCallCredentials</c>.</b> gRPC refuses to attach call
+/// credentials to a plaintext channel by default, precisely because a bearer token on h2c is
+/// readable and replayable by anyone on the path. That is a real and accepted limitation here:
+/// these listeners are h2c today and the boundary assumes a trusted network. The PSK raises the
+/// bar from "anyone who can reach the port" to "anyone who can read the traffic"; TLS on these
+/// listeners is the follow-on hardening and requires no change to this code.
+/// </para>
+/// </remarks>
+public static class ControlPlaneCredentials
+{
+    /// <summary>
+    /// Metadata header naming the site a call belongs to. Central needs it to pick which
+    /// per-site key to verify against; a site's own inbound gate ignores it (a site has exactly
+    /// one key). Required by central's interceptor from Phase 1A.
+    /// </summary>
+    public const string SiteHeader = "x-scadabridge-site";
+
+    /// <summary>The bearer metadata header. Lowercase — gRPC lowercases header keys on the wire.</summary>
+    public const string AuthorizationHeader = "authorization";
+
+    /// <summary>
+    /// Creates call credentials that attach <c>authorization: Bearer &lt;psk&gt;</c> and
+    /// <c>x-scadabridge-site: &lt;siteId&gt;</c> to every call.
+    /// </summary>
+    /// <param name="provider">Resolves the site's preshared key.</param>
+    /// <param name="siteId">The site this channel talks to (or, site-side, this site's own id).</param>
+    /// <returns>Call credentials for a channel bound to that site.</returns>
+    public static CallCredentials ForSite(ISitePskProvider provider, string siteId)
+    {
+        ArgumentNullException.ThrowIfNull(provider);
+        ArgumentException.ThrowIfNullOrWhiteSpace(siteId);
+
+        return CallCredentials.FromInterceptor(async (context, metadata) =>
+        {
+            // A throw here fails the call, which is the point: an unauthenticated dial must
+            // not happen. Callers classify the resulting fault the same way they classify any
+            // other — the pull clients degrade to an empty batch and log, the streaming
+            // subscribers retry.
+            var psk = await provider.GetAsync(siteId, context.CancellationToken).ConfigureAwait(false);
+            metadata.Add(AuthorizationHeader, $"Bearer {psk}");
+            metadata.Add(SiteHeader, siteId);
+        });
+    }
+
+    /// <summary>
+    /// Applies per-site call credentials to channel options, if a provider is available.
+    /// A null provider leaves the options untouched — the shape used by test-only and
+    /// default constructors that never dial a gated endpoint.
+    /// </summary>
+    /// <param name="options">Channel options being built.</param>
+    /// <param name="provider">Key provider, or null to leave the channel unauthenticated.</param>
+    /// <param name="siteId">The site this channel talks to.</param>
+    /// <returns>The same options instance, for chaining.</returns>
+    public static GrpcChannelOptions WithSiteCredentials(
+        this GrpcChannelOptions options, ISitePskProvider? provider, string? siteId)
+    {
+        ArgumentNullException.ThrowIfNull(options);
+
+        if (provider is null || string.IsNullOrWhiteSpace(siteId))
+        {
+            return options;
+        }
+
+        options.Credentials = ChannelCredentials.Create(
+            ChannelCredentials.Insecure, ForSite(provider, siteId));
+        options.UnsafeUseInsecureChannelCallCredentials = true;
+        return options;
+    }
+}
@@ -60,6 +60,27 @@ public class SiteStreamGrpcClient : IAsyncDisposable, IDisposable
    /// <param name="logger">Logger for diagnostics and errors.</param>
    /// <param name="options">Communication options including keepalive settings.</param>
    public SiteStreamGrpcClient(string endpoint, ILogger logger, CommunicationOptions options)
+        : this(endpoint, logger, options, pskProvider: null, siteIdentifier: null)
+    {
+    }
+
+    /// <summary>
+    /// Creates a client that authenticates every call with the site's preshared key.
+    /// This is the production shape: <c>SiteStreamService</c> is gated by
+    /// <c>ControlPlaneAuthInterceptor</c> on the site node, so a client without credentials
+    /// gets <see cref="StatusCode.PermissionDenied"/> on every call.
+    /// </summary>
+    /// <param name="endpoint">The gRPC endpoint address for the site.</param>
+    /// <param name="logger">Logger for diagnostics and errors.</param>
+    /// <param name="options">Communication options including keepalive settings.</param>
+    /// <param name="pskProvider">Resolves the site's preshared key; null leaves the channel unauthenticated.</param>
+    /// <param name="siteIdentifier">Site this channel talks to; null leaves the channel unauthenticated.</param>
+    public SiteStreamGrpcClient(
+        string endpoint,
+        ILogger logger,
+        CommunicationOptions options,
+        ISitePskProvider? pskProvider,
+        string? siteIdentifier)
    {
        Endpoint = endpoint;
        KeepAlivePingDelay = options.GrpcKeepAlivePingDelay;
@@ -72,7 +93,7 @@ public class SiteStreamGrpcClient : IAsyncDisposable, IDisposable
                KeepAlivePingTimeout = options.GrpcKeepAlivePingTimeout,
                KeepAlivePingPolicy = HttpKeepAlivePingPolicy.Always
            }
-        });
+        }.WithSiteCredentials(pskProvider, siteIdentifier));
        _client = new SiteStreamService.SiteStreamServiceClient(_channel);
        _logger = logger;
    }
@@ -26,9 +26,11 @@ public class SiteStreamGrpcClientFactory : IAsyncDisposable, IDisposable
    private readonly ConcurrentDictionary<(string Site, string Endpoint), SiteStreamGrpcClient> _clients = new();
    private readonly ILoggerFactory _loggerFactory;
    private readonly CommunicationOptions _options;
+    private readonly ISitePskProvider? _pskProvider;

    /// <summary>
-    /// Test/default constructor — uses default <see cref="CommunicationOptions"/>.
+    /// Test/default constructor — uses default <see cref="CommunicationOptions"/> and creates
+    /// unauthenticated channels.
    /// </summary>
    /// <param name="loggerFactory">Logger factory passed to created clients.</param>
    public SiteStreamGrpcClientFactory(ILoggerFactory loggerFactory)
@@ -37,16 +39,36 @@ public class SiteStreamGrpcClientFactory : IAsyncDisposable, IDisposable
    }

    /// <summary>
-    /// DI constructor — flows <see cref="CommunicationOptions"/> into every created
-    /// <see cref="SiteStreamGrpcClient"/> so the configured gRPC keepalive settings
-    /// are applied rather than hard-coded defaults.
+    /// Constructor without a key provider — creates unauthenticated channels, which a gated
+    /// site will refuse. Retained for tests and for hosts that never dial a site.
    /// </summary>
    /// <param name="loggerFactory">Logger factory passed to created clients.</param>
    /// <param name="options">Communication options applied to each created client.</param>
    public SiteStreamGrpcClientFactory(ILoggerFactory loggerFactory, IOptions<CommunicationOptions> options)
+        : this(loggerFactory, options, pskProvider: null)
+    {
+    }
+
+    /// <summary>
+    /// DI constructor — flows <see cref="CommunicationOptions"/> into every created
+    /// <see cref="SiteStreamGrpcClient"/> so the configured gRPC keepalive settings are applied
+    /// rather than hard-coded defaults, and attaches the per-site preshared key that the site's
+    /// <c>ControlPlaneAuthInterceptor</c> requires.
+    /// </summary>
+    /// <param name="loggerFactory">Logger factory passed to created clients.</param>
+    /// <param name="options">Communication options applied to each created client.</param>
+    /// <param name="pskProvider">
+    /// Resolves each site's preshared key. Optional in DI so a host that registers no provider
+    /// (a site node, which never dials another site) still resolves this factory.
+    /// </param>
+    public SiteStreamGrpcClientFactory(
+        ILoggerFactory loggerFactory,
+        IOptions<CommunicationOptions> options,
+        ISitePskProvider? pskProvider)
    {
        _loggerFactory = loggerFactory;
        _options = options.Value;
+        _pskProvider = pskProvider;
    }

    /// <summary>
@@ -59,7 +81,7 @@ public class SiteStreamGrpcClientFactory : IAsyncDisposable, IDisposable
    /// <param name="grpcEndpoint">gRPC endpoint (second half of the cache key) the client is bound to.</param>
    /// <returns>The cached or newly-created client bound to <paramref name="grpcEndpoint"/>.</returns>
    public virtual SiteStreamGrpcClient GetOrCreate(string siteIdentifier, string grpcEndpoint) =>
-        _clients.GetOrAdd((siteIdentifier, grpcEndpoint), _ => CreateClient(grpcEndpoint));
+        _clients.GetOrAdd((siteIdentifier, grpcEndpoint), key => CreateClient(key.Site, key.Endpoint));

    /// <summary>
    /// Returns the cached client for <c>(site, endpoint)</c>, or <c>null</c> — never creates.
@@ -77,12 +99,13 @@ public class SiteStreamGrpcClientFactory : IAsyncDisposable, IDisposable
    /// can substitute a tracking client while still exercising the factory's real
    /// caching and disposal machinery.
    /// </summary>
+    /// <param name="siteIdentifier">Site the new client talks to; selects which preshared key it presents.</param>
    /// <param name="grpcEndpoint">gRPC endpoint the new client will connect to.</param>
    /// <returns>A new <see cref="SiteStreamGrpcClient"/> connected to <paramref name="grpcEndpoint"/>.</returns>
-    protected virtual SiteStreamGrpcClient CreateClient(string grpcEndpoint)
+    protected virtual SiteStreamGrpcClient CreateClient(string siteIdentifier, string grpcEndpoint)
    {
        var logger = _loggerFactory.CreateLogger<SiteStreamGrpcClient>();
-        return new SiteStreamGrpcClient(grpcEndpoint, logger, _options);
+        return new SiteStreamGrpcClient(grpcEndpoint, logger, _options, _pskProvider, siteIdentifier);
    }

    /// <summary>
@@ -99,6 +122,10 @@ public class SiteStreamGrpcClientFactory : IAsyncDisposable, IDisposable
            if (_clients.TryRemove(key, out var client))
                await client.DisposeAsync();
        }
+
+        // Drop the cached preshared key too, so a site removed and re-added under the same
+        // identifier (with a rotated key) is not dialed with the stale one.
+        _pskProvider?.Invalidate(siteIdentifier);
    }

    /// <summary>
@@ -1,5 +1,6 @@
 using Microsoft.Extensions.DependencyInjection;
 using Microsoft.Extensions.DependencyInjection.Extensions;
+using Microsoft.Extensions.Logging;
 using Microsoft.Extensions.Options;
 using ZB.MOM.WW.ScadaBridge.Communication.Grpc;

@@ -23,7 +24,15 @@ public static class ServiceCollectionExtensions
            ServiceDescriptor.Singleton<IValidateOptions<CommunicationOptions>, CommunicationOptionsValidator>());

        services.AddSingleton<CommunicationService>();
-        services.AddSingleton<SiteStreamGrpcClientFactory>();
+
+        // Explicit factory rather than AddSingleton<T>(): the ISitePskProvider dependency is
+        // optional (central registers one, a site node does not), and constructor selection
+        // over a nullable interface parameter is exactly the case the container cannot decide
+        // for itself — GetService returns null cleanly where constructor injection would throw.
+        services.AddSingleton(sp => new SiteStreamGrpcClientFactory(
+            sp.GetRequiredService<ILoggerFactory>(),
+            sp.GetRequiredService<IOptions<CommunicationOptions>>(),
+            sp.GetService<ISitePskProvider>()));
        services.AddSingleton<DebugStreamService>();

        // Aggregated live alarm cache (plan #10, Task 4): transient, in-memory, shared
@@ -450,16 +450,22 @@ akka {{
        siteAlarmLiveCache?.SetActorSystem(_actorSystem!);


-        // Management Service — accessible via ClusterClient
+        // Management Service — reached IN-PROCESS only, via ManagementActorHolder.
+        //
+        // This actor used to be registered with the ClusterClientReceptionist as well, for
+        // an out-of-cluster CLI that was never built (REQ-HOST-6a). The shipped CLI speaks
+        // HTTP Basic to /management, which asks this actor through the holder below
+        // (ManagementEndpoints), so the registration had no sender anywhere in the repo —
+        // it only advertised a management surface across the cluster-client boundary for
+        // free. Removed 2026-07-22 (ClusterClient→gRPC migration, T0.1).
        var mgmtLogger = _serviceProvider.GetRequiredService<ILoggerFactory>()
            .CreateLogger<ZB.MOM.WW.ScadaBridge.ManagementService.ManagementActor>();
        var mgmtActor = _actorSystem!.ActorOf(
            Props.Create(() => new ZB.MOM.WW.ScadaBridge.ManagementService.ManagementActor(_serviceProvider, mgmtLogger)),
            "management");
-        ClusterClientReceptionist.Get(_actorSystem).RegisterService(mgmtActor);
        var mgmtHolder = _serviceProvider.GetRequiredService<ZB.MOM.WW.ScadaBridge.ManagementService.ManagementActorHolder>();
        mgmtHolder.ActorRef = mgmtActor;
-        _logger.LogInformation("ManagementActor registered with ClusterClientReceptionist");
+        _logger.LogInformation("ManagementActor started at /user/management (in-process access via ManagementActorHolder)");

        // Notification Outbox — cluster singleton so exactly one node owns ingest,
        // the dispatch sweep and the purge loop. Central actors run on the base
@@ -0,0 +1,211 @@
+using System.Security.Cryptography;
+using System.Text;
+using Grpc.Core;
+using Grpc.Core.Interceptors;
+using Microsoft.Extensions.Logging;
+using Microsoft.Extensions.Options;
+using ZB.MOM.WW.ScadaBridge.Communication;
+using ZB.MOM.WW.ScadaBridge.Communication.Grpc;
+
+namespace ZB.MOM.WW.ScadaBridge.Host;
+
+/// <summary>
+/// Gates the site↔central gRPC control plane with a preshared key.
+/// </summary>
+/// <remarks>
+/// <para>
+/// <b>The gap this closes.</b> <c>SiteStreamService</c> shipped with no authentication at all —
+/// plaintext h2c, no interceptor. Anything that could reach a site node's gRPC port could open a
+/// live data stream or call <c>PullAuditEvents</c>/<c>PullSiteCalls</c> and read audit rows back.
+/// The only gated surface on that listener was LocalDb sync, and only for its own service. That
+/// gap exists independently of the ClusterClient→gRPC migration; it becomes indefensible once
+/// every command crosses this listener.
+/// </para>
+/// <para>
+/// <b>Modeled on <see cref="LocalDbSyncAuthInterceptor"/>,</b> deliberately: same four server
+/// handlers funnelling into one <c>Authorize</c>, same <c>authorization: Bearer</c> extraction,
+/// same <see cref="CryptographicOperations.FixedTimeEquals"/> comparison, same fail-closed
+/// posture, same <see cref="StatusCode.PermissionDenied"/> rejection. Two differences:
+/// </para>
+/// <list type="number">
+/// <item>It gates a <b>set</b> of service prefixes rather than one, so later phases can add the
+/// new command/control services without a second interceptor.</item>
+/// <item>Its expected key comes from <see cref="CommunicationOptions.GrpcPsk"/> — the site's own
+/// key, supplied in production as <c>${secret:SB-GRPC-PSK-&lt;siteId&gt;}</c> and expanded before
+/// the host is built.</item>
+/// </list>
+/// <para>
+/// <b>The two keys are separate on purpose.</b> LocalDb sync keeps its own
+/// <c>LocalDb:Replication:ApiKey</c>, which authenticates a different peer (the pair partner, not
+/// central) over a different trust relationship. Sharing one key would mean a site's central-facing
+/// key also admits writes into its database.
+/// </para>
+/// <para>
+/// <b>Fail-closed, and not optional.</b> With no <c>GrpcPsk</c> configured, every gated call is
+/// rejected — including the ones that work today. That is a deliberate break: LocalDb replication
+/// is an opt-in feature whose "off" state is "no peer", whereas streaming and audit pull are
+/// core paths, so "no key" must not silently mean "no authentication". Every environment must
+/// carry a key before upgrading to this build.
+/// </para>
+/// </remarks>
+public sealed class ControlPlaneAuthInterceptor : Interceptor
+{
+    /// <summary>
+    /// Service prefixes gated by default. Read from the generated <c>sitestream.proto</c>
+    /// package/service names — <c>package sitestream; service SiteStreamService</c>.
+    /// Later phases append their own services here.
+    /// </summary>
+    public static readonly IReadOnlyList<string> DefaultGatedPrefixes =
+        new[] { "/sitestream.SiteStreamService/" };
+
+    private readonly IReadOnlyList<string> _gatedPrefixes;
+    private readonly IOptions<CommunicationOptions> _options;
+    private readonly ILogger<ControlPlaneAuthInterceptor> _logger;
+
+    /// <summary>Creates the interceptor gating <see cref="DefaultGatedPrefixes"/>.</summary>
+    /// <param name="options">Communication options; <c>GrpcPsk</c> is the expected bearer token.</param>
+    /// <param name="logger">Logger for denial diagnostics.</param>
+    public ControlPlaneAuthInterceptor(
+        IOptions<CommunicationOptions> options,
+        ILogger<ControlPlaneAuthInterceptor> logger)
+        : this(options, logger, DefaultGatedPrefixes)
+    {
+    }
+
+    /// <summary>Creates the interceptor gating an explicit set of service prefixes.</summary>
+    /// <param name="options">Communication options; <c>GrpcPsk</c> is the expected bearer token.</param>
+    /// <param name="logger">Logger for denial diagnostics.</param>
+    /// <param name="gatedPrefixes">Method-path prefixes to gate, e.g. <c>/sitestream.SiteStreamService/</c>.</param>
+    public ControlPlaneAuthInterceptor(
+        IOptions<CommunicationOptions> options,
+        ILogger<ControlPlaneAuthInterceptor> logger,
+        IReadOnlyList<string> gatedPrefixes)
+    {
+        ArgumentNullException.ThrowIfNull(options);
+        ArgumentNullException.ThrowIfNull(logger);
+        ArgumentNullException.ThrowIfNull(gatedPrefixes);
+
+        _options = options;
+        _logger = logger;
+        _gatedPrefixes = gatedPrefixes;
+    }
+
+    /// <inheritdoc />
+    public override Task<TResponse> UnaryServerHandler<TRequest, TResponse>(
+        TRequest request,
+        ServerCallContext context,
+        UnaryServerMethod<TRequest, TResponse> continuation)
+    {
+        Authorize(context);
+        return continuation(request, context);
+    }
+
+    /// <inheritdoc />
+    public override Task DuplexStreamingServerHandler<TRequest, TResponse>(
+        IAsyncStreamReader<TRequest> requestStream,
+        IServerStreamWriter<TResponse> responseStream,
+        ServerCallContext context,
+        DuplexStreamingServerMethod<TRequest, TResponse> continuation)
+    {
+        Authorize(context);
+        return continuation(requestStream, responseStream, context);
+    }
+
+    /// <inheritdoc />
+    public override Task<TResponse> ClientStreamingServerHandler<TRequest, TResponse>(
+        IAsyncStreamReader<TRequest> requestStream,
+        ServerCallContext context,
+        ClientStreamingServerMethod<TRequest, TResponse> continuation)
+    {
+        Authorize(context);
+        return continuation(requestStream, context);
+    }
+
+    /// <inheritdoc />
+    public override Task ServerStreamingServerHandler<TRequest, TResponse>(
+        TRequest request,
+        IServerStreamWriter<TResponse> responseStream,
+        ServerCallContext context,
+        ServerStreamingServerMethod<TRequest, TResponse> continuation)
+    {
+        Authorize(context);
+        return continuation(request, responseStream, context);
+    }
+
+    /// <summary>
+    /// Throws <see cref="RpcException"/> with <see cref="StatusCode.PermissionDenied"/> if this
+    /// is a gated call that does not carry the configured bearer token. Calls to services
+    /// outside <c>gatedPrefixes</c> — notably LocalDb sync, which has its own interceptor —
+    /// return immediately.
+    /// </summary>
+    private void Authorize(ServerCallContext context)
+    {
+        if (!IsGated(context.Method))
+        {
+            return;
+        }
+
+        var expected = _options.Value.GrpcPsk;
+        if (string.IsNullOrEmpty(expected))
+        {
+            _logger.LogWarning(
+                "Rejected a control-plane call to {Method}: no ScadaBridge:Communication:GrpcPsk is "
+                + "configured, so the control plane is closed. Set the same key here (in production, "
+                + "as ${{secret:SB-GRPC-PSK-<siteId>}}) and in central's secret store.",
+                context.Method);
+            throw new RpcException(new Status(
+                StatusCode.PermissionDenied,
+                "Control plane is not accepting calls: no preshared key is configured on this node."));
+        }
+
+        var presented = ExtractBearerToken(context.RequestHeaders);
+        if (presented is null || !FixedTimeEquals(presented, expected))
+        {
+            _logger.LogWarning(
+                "Rejected a control-plane call to {Method}: {Reason}.",
+                context.Method,
+                presented is null ? "no bearer token presented" : "bearer token did not match");
+            throw new RpcException(new Status(
+                StatusCode.PermissionDenied,
+                "Control plane authentication failed."));
+        }
+    }
+
+    private bool IsGated(string method)
+    {
+        foreach (var prefix in _gatedPrefixes)
+        {
+            if (method.StartsWith(prefix, StringComparison.Ordinal))
+            {
+                return true;
+            }
+        }
+        return false;
+    }
+
+    private static string? ExtractBearerToken(Metadata headers)
+    {
+        // gRPC lowercases header keys on the wire; compare case-insensitively anyway so a
+        // hand-built Metadata in a test behaves the same as a real request.
+        foreach (var entry in headers)
+        {
+            if (!string.Equals(entry.Key, ControlPlaneCredentials.AuthorizationHeader,
+                    StringComparison.OrdinalIgnoreCase))
+            {
+                continue;
+            }
+
+            var value = entry.Value;
+            if (value is not null && value.StartsWith("Bearer ", StringComparison.OrdinalIgnoreCase))
+            {
+                return value["Bearer ".Length..];
+            }
+        }
+
+        return null;
+    }
+
+    private static bool FixedTimeEquals(string presented, string expected)
+        => CryptographicOperations.FixedTimeEquals(
+            Encoding.UTF8.GetBytes(presented), Encoding.UTF8.GetBytes(expected));
+}
@@ -100,6 +100,14 @@ try
        // Shared components
        builder.Services.AddClusterInfrastructure();
        builder.Services.AddCommunication();
+
+        // Per-site gRPC preshared keys. Central-only: it is the side that dials sites, and
+        // the only side whose key set is dynamic (sites come from the configuration
+        // database, so there is no fixed list of ${secret:} references to expand at boot —
+        // hence a runtime resolver rather than the pre-host SecretReferenceExpander a site
+        // node uses for its single key). Registered before the clients that consume it.
+        builder.Services.AddSingleton<
+            ZB.MOM.WW.ScadaBridge.Communication.Grpc.ISitePskProvider, SitePskProvider>();
        builder.Services.AddHealthMonitoring();
        builder.Services.AddCentralHealthAggregation();
        builder.Services.AddExternalSystemGateway();
@@ -519,12 +527,21 @@ try
            });
        });

-        // gRPC server registration
-        // The interceptor gates ONLY /localdb_sync.v1.LocalDbSync/ — SiteStream calls on
-        // this same pipeline pass through untouched. It is fail-closed: with no
-        // LocalDb:Replication:ApiKey configured, no sync stream is accepted at all.
+        // gRPC server registration. Two interceptors, two disjoint service prefixes, two
+        // separate keys — neither one's absence weakens the other:
+        //
+        //   LocalDbSyncAuthInterceptor  gates /localdb_sync.v1.LocalDbSync/  (LocalDb:Replication:ApiKey)
+        //   ControlPlaneAuthInterceptor gates /sitestream.SiteStreamService/ (ScadaBridge:Communication:GrpcPsk)
+        //
+        // Both are fail-closed: an unset key closes that surface rather than opening it. For
+        // LocalDb that means replication simply does not start; for the control plane it means
+        // this node serves no streams and no audit pulls until a key is configured, which is
+        // why every environment must carry one before running this build.
        builder.Services.AddGrpc(options =>
-            options.Interceptors.Add<LocalDbSyncAuthInterceptor>());
+        {
+            options.Interceptors.Add<LocalDbSyncAuthInterceptor>();
+            options.Interceptors.Add<ControlPlaneAuthInterceptor>();
+        });
        builder.Services.AddSingleton<ZB.MOM.WW.ScadaBridge.Communication.Grpc.SiteStreamGrpcServer>();

        // Existing site service registrations (this is also where LocalDb and its
@@ -0,0 +1,149 @@
+using System.Collections.Concurrent;
+using Microsoft.Extensions.Logging;
+using Microsoft.Extensions.Options;
+using ZB.MOM.WW.ScadaBridge.Communication;
+using ZB.MOM.WW.ScadaBridge.Communication.Grpc;
+using ZB.MOM.WW.Secrets.Abstractions;
+
+namespace ZB.MOM.WW.ScadaBridge.Host;
+
+/// <summary>
+/// Central's <see cref="ISitePskProvider"/>: resolves each site's gRPC preshared key at
+/// channel-build time — from <c>ScadaBridge:Communication:SitePsks</c> if the site is listed
+/// there, otherwise from the secrets store under the name <c>SB-GRPC-PSK-{siteId}</c>.
+/// </summary>
+/// <remarks>
+/// <para>
+/// <b>Why resolved at runtime rather than expanded into config at boot.</b> Site nodes get their
+/// single key through the pre-host <c>SecretReferenceExpander</c> — a <c>${secret:…}</c> in
+/// appsettings, resolved once before the host is built. Central cannot do that: its set of sites
+/// comes from the configuration database and changes while the process runs, so there is no
+/// fixed list of references to expand at boot. This mirrors the pattern the MxGateway data
+/// connection already uses for its per-connection API keys.
+/// </para>
+/// <para>
+/// <b>Caching.</b> Successful resolves are cached indefinitely; a site's key changes only on
+/// rotation, and rotation restarts the pair. Failures are deliberately NOT cached, so a key
+/// seeded after the first (failed) dial is picked up on the next attempt instead of requiring a
+/// restart. <see cref="Invalidate"/> drops an entry when a site is removed or a key is rotated
+/// in place.
+/// </para>
+/// <para>
+/// <b>Fail-closed.</b> A key absent from BOTH sources throws, and the dial that needed it fails.
+/// It never degrades to an unauthenticated call. The message names the exact config key and the
+/// exact secret name, so the fix is one setting or one <c>secret</c> CLI seed.
+/// </para>
+/// </remarks>
+public sealed class SitePskProvider : ISitePskProvider
+{
+    /// <summary>Prefix of the per-site secret name; the site identifier is appended verbatim.</summary>
+    public const string SecretNamePrefix = "SB-GRPC-PSK-";
+
+    private readonly ConcurrentDictionary<string, string> _cache = new(StringComparer.Ordinal);
+    private readonly ISecretResolver _resolver;
+    private readonly IOptionsMonitor<CommunicationOptions> _options;
+    private readonly ILogger<SitePskProvider> _logger;
+
+    /// <summary>Creates the provider over the host's secret resolver.</summary>
+    /// <param name="resolver">Runtime secret resolver from the host container.</param>
+    /// <param name="options">
+    /// Communication options supplying the optional <c>SitePsks</c> map. Read through an
+    /// <see cref="IOptionsMonitor{TOptions}"/> so a reloaded configuration is honoured on the
+    /// next uncached resolve rather than pinned at construction.
+    /// </param>
+    /// <param name="logger">Logger for resolution diagnostics.</param>
+    public SitePskProvider(
+        ISecretResolver resolver,
+        IOptionsMonitor<CommunicationOptions> options,
+        ILogger<SitePskProvider> logger)
+    {
+        ArgumentNullException.ThrowIfNull(resolver);
+        ArgumentNullException.ThrowIfNull(options);
+        ArgumentNullException.ThrowIfNull(logger);
+
+        _resolver = resolver;
+        _options = options;
+        _logger = logger;
+    }
+
+    /// <summary>Builds the secret name for a site.</summary>
+    /// <param name="siteId">Site identifier.</param>
+    /// <returns>The secret name, e.g. <c>SB-GRPC-PSK-site-a</c>.</returns>
+    public static string SecretNameFor(string siteId) => SecretNamePrefix + siteId;
+
+    /// <inheritdoc />
+    public async ValueTask<string> GetAsync(string siteId, CancellationToken ct)
+    {
+        ArgumentException.ThrowIfNullOrWhiteSpace(siteId);
+
+        if (_cache.TryGetValue(siteId, out var cached))
+        {
+            return cached;
+        }
+
+        // Configured map first: it is the explicit, operator-stated answer for this site, and
+        // it is the only source available to a host running without a secrets master key.
+        var configured = _options.CurrentValue.SitePsks;
+        if (configured.TryGetValue(siteId, out var fromConfig) && !string.IsNullOrEmpty(fromConfig))
+        {
+            _cache[siteId] = fromConfig;
+            return fromConfig;
+        }
+
+        // Otherwise the store, which is the only source that can serve a site added at runtime.
+        var secretName = SecretNameFor(siteId);
+        var value = await ResolveFromStoreAsync(secretName, siteId, ct).ConfigureAwait(false);
+
+        if (string.IsNullOrEmpty(value))
+        {
+            _logger.LogError(
+                "gRPC control-plane key for site {SiteId} could not be resolved: it is absent from "
+                + "ScadaBridge:Communication:SitePsks and secret '{SecretName}' is missing, empty or "
+                + "tombstoned. Calls to this site are refused until one of the two is set (the site "
+                + "node must carry the same value in ScadaBridge:Communication:GrpcPsk).",
+                siteId, secretName);
+            throw new InvalidOperationException(
+                $"gRPC preshared key for site '{siteId}' could not be resolved from "
+                + $"ScadaBridge:Communication:SitePsks or secret '{secretName}'.");
+        }
+
+        // Two concurrent first dials may both resolve; the read is idempotent and the values
+        // identical, so the loser simply overwrites with the same string.
+        _cache[siteId] = value;
+        return value;
+    }
+
+    /// <summary>
+    /// Reads the key from the secrets store, treating a store fault as "not found" rather than
+    /// letting it propagate. A host with no master key configured (the development rig) throws
+    /// from the resolver; that must produce the same clear "key not configured" error as a
+    /// missing secret, not an opaque cryptographic one.
+    /// </summary>
+    private async Task<string?> ResolveFromStoreAsync(string secretName, string siteId, CancellationToken ct)
+    {
+        try
+        {
+            return await _resolver.GetAsync(new SecretName(secretName), ct).ConfigureAwait(false);
+        }
+        catch (OperationCanceledException)
+        {
+            throw;
+        }
+        catch (Exception ex)
+        {
+            _logger.LogDebug(ex,
+                "Secret store lookup for '{SecretName}' (site {SiteId}) faulted; treating as not found.",
+                secretName, siteId);
+            return null;
+        }
+    }
+
+    /// <inheritdoc />
+    public void Invalidate(string siteId)
+    {
+        if (!string.IsNullOrWhiteSpace(siteId))
+        {
+            _cache.TryRemove(siteId, out _);
+        }
+    }
+}
@@ -126,6 +126,23 @@ public static class StartupValidator
                p.Require("ScadaBridge:Node:MetricsPort", _ => metricsPort != port, "must differ from RemotingPort");
                p.Require("ScadaBridge:Node:MetricsPort", _ => metricsPort != grpcPort, "must differ from GrpcPort");

+                // The gRPC control-plane preshared key. Same argument as the inbound
+                // API-key pepper above: without it the node boots and looks healthy, but
+                // ControlPlaneAuthInterceptor is fail-closed, so every SiteStream call —
+                // live subscriptions, audit pulls, cached-telemetry ingest — is refused
+                // with PermissionDenied. A silent, total loss of the site's central-facing
+                // surface is far worse than a loud boot failure, so require it here.
+                // Central holds the matching value per site in its secret store as
+                // SB-GRPC-PSK-{SiteId}; production supplies this one as
+                // ${secret:SB-GRPC-PSK-<siteId>}, expanded before the host is built.
+                p.Require("ScadaBridge:Communication:GrpcPsk",
+                    value => !string.IsNullOrWhiteSpace(value),
+                    "is required for Site nodes: it is the preshared key the gRPC control "
+                    + "plane authenticates with, and the interceptor is fail-closed, so an "
+                    + "unset key refuses every SiteStream call. Set the same value here (in "
+                    + "production as ${secret:SB-GRPC-PSK-<siteId>}) and under the secret "
+                    + "name SB-GRPC-PSK-<siteId> in central's secret store");
+
                // ScadaBridge:Database:SiteDbPath was required here until LocalDb
                // Phase 2. The site's tables now live in the consolidated LocalDb
                // database (LocalDb:Path, which SiteServiceRegistration requires),
@@ -42,6 +42,8 @@
      "SqliteDbPath": "./data/store-and-forward.db"
    },
    "Communication": {
+      "_grpcPsk": "REQUIRED on Site nodes (StartupValidator fails the boot without it). The preshared key the gRPC control plane authenticates with: ControlPlaneAuthInterceptor is fail-closed, so an unset key refuses every SiteStream call — live subscriptions, audit pulls, cached-telemetry ingest — while the node still reports healthy. Supply it as ${secret:SB-GRPC-PSK-<siteId>} so the plaintext never sits in this file, and seed the SAME value on central: either as the secret SB-GRPC-PSK-<siteId> in its store, or as ScadaBridge:Communication:SitePsks:<siteId>. BOTH nodes of the pair carry the same key. Distinct from LocalDb:Replication:ApiKey, which authenticates the pair partner, not central — never share the two.",
+      "GrpcPsk": "${secret:SB-GRPC-PSK-site-1}",
      "_centralContactPoints": "Host-016: each entry MUST be a central node's remoting endpoint, NOT this site's own remoting port. The single dev-loopback default below points only at central-a (localhost:8081). In a multi-central deployment add the second central node here (e.g. 'akka.tcp://scadabridge@central-b-host:8081') so ClusterClient can fail over when central-a is down. The previous template listed localhost:8082 as the second contact — that is THIS site's own RemotingPort and is a permanent failure in the initial-contact rotation.",
      "CentralContactPoints": [
        "akka.tcp://scadabridge@localhost:8081"