feat(grpc): PSK-authenticate the site gRPC control plane; drop the vestigial management receptionist registration

Phase 0 of the ClusterClient→gRPC migration
(docs/plans/2026-07-22-clusterclient-to-grpc-plan.md). Standalone hardening: it
closes a gap that exists today and is a precondition for moving command/control
onto gRPC in later phases.

T0.1 — delete the ManagementActor ClusterClientReceptionist registration.
It was built for an out-of-cluster CLI that was never written: the shipped CLI
speaks HTTP Basic to /management, which asks the actor in-process through
ManagementActorHolder. Nothing in the repo ever sent to /user/management. The
actor still runs there; only the cross-boundary advertisement is gone. Six
documents claimed the CLI used ClusterClient — including the CLI's own README
"Architecture Notes" — and are corrected here rather than left to rot.

T0.2 — record, do not port, the dead integration-routing path.
IntegrationCallRequest is unwired at BOTH ends: RouteIntegrationCallAsync has
zero callers anywhere, and RegisterLocalHandler(Integration, …) appears only in
a test, so production always answers "Integration handler not available". It is
excluded from the gRPC contract (28 of 29 commands migrate) rather than
enshrined on an additive-only wire format, and deleting it during a
transport migration would mix a behavioural change into a change whose whole
value is that behaviour is identical. See
docs/known-issues/2026-07-22-integration-call-routing-is-dead-code.md.

T0.3 — preshared-key authentication on SiteStreamService.
The service shipped with no auth at all: plaintext h2c, no interceptor, so
anything that could reach a site node's :8083 could open a live data stream or
read audit rows back via PullAuditEvents/PullSiteCalls. ControlPlaneAuthInterceptor
now gates /sitestream.SiteStreamService/ — modeled on LocalDbSyncAuthInterceptor
(constant-time compare, fail-closed, PermissionDenied) but gating a SET of
service prefixes so phases 1A/1B add services rather than interceptors. LocalDb
sync keeps its own separate key: it authenticates the pair partner, not central,
and collapsing the two would make a site's central-facing key also admit writes
into its database.

Keys are per site (SB-GRPC-PSK-<siteId>), never fleet-wide, so a compromised
site yields only its own. Central attaches them through ControlPlaneCredentials,
which binds CallCredentials to the channel — covering unary and streaming
uniformly, and letting the key resolve asynchronously, which a client
interceptor could not do without blocking. All three central→site channel
creation sites go through it (SiteStreamGrpcClient and both audit pull invokers);
the pull invokers' channel caches are re-keyed by (site, endpoint) because
credentials are per-site and bound to the channel.

Two decisions beyond the plan:

  * StartupValidator now requires GrpcPsk on Site nodes. The plan specified only
    the runtime gate, but fail-closed with no boot check produces a node that
    joins, answers heartbeats and reports healthy while refusing every stream,
    audit pull and telemetry ingest — silent and total. Same reasoning as the
    existing inbound API-key pepper rule.

  * Added Communication:SitePsks as a central-side key map. The plan assumed
    central would read the store, seeded via a dev KEK; the docker rig
    deliberately boots with no master key, so store-only resolution would leave
    it unable to dial its own sites. The store stays primary — it is the only
    source that can serve a site added at runtime — with the map covering
    key-less hosts and one-off pins. Neither source falling back to
    "unauthenticated" is the invariant.

T0.4 — dev keys on both rigs and tests.
34 tests. The seven that matter most exercise a real in-process gRPC stack over
TestServer: the unit tests on either side of the wire would both stay green if
the halves disagreed, and gRPC refuses call credentials on a plaintext channel
by default — the UnsafeUseInsecureChannelCallCredentials opt-in is only provable
by making a real call. They confirm correct key passes on unary AND streaming,
wrong key and no-credentials both get PermissionDenied, and an unresolvable key
fails the call with nothing reaching the service.

OPERATIONAL: a site node upgraded to this build without a key will not boot.
That includes the gitignored deploy/wonder-app-vd03/ overlay.
This commit is contained in:
Joseph Doherty
2026-07-22 17:51:09 -04:00
parent f1ad967083
commit 2ee84af1c0
45 changed files with 1913 additions and 69 deletions
+28 -1
View File
@@ -109,6 +109,31 @@ The streaming protocol is defined in `sitestream.proto` (`src/ZB.MOM.WW.ScadaBri
- The `oneof event` pattern is extensible — future event types (health metrics, connection state changes) are added as new fields without breaking existing consumers.
- Proto field numbers are never reused; new RPCs and message fields are appended additively. Old clients ignore unknown `oneof` variants.
##### Authentication (preshared key, 2026-07-22)
Every RPC on this service — streaming and unary alike — requires a preshared key, presented as
`authorization: Bearer <key>` metadata and verified by `ControlPlaneAuthInterceptor` on the site
node. The gate is **fail-closed**: with no key configured, every call is refused with
`PermissionDenied`, and `StartupValidator` refuses to boot a site node in that state (an unset key
would otherwise leave the node joined and reporting healthy while serving nothing).
Keys are scoped **one per site** — secret `SB-GRPC-PSK-<siteId>`, so a compromised site never
yields another site's key. The site node reads its own key from
`ScadaBridge:Communication:GrpcPsk` (in production a `${secret:}` reference expanded before the
host is built); central resolves each site's key at channel-build time via `SitePskProvider`,
because sites are created at runtime and cannot be enumerated in configuration at boot.
`ControlPlaneCredentials` binds the key to the channel as `CallCredentials`, so no individual call
site can omit it.
This is distinct from `LocalDb:Replication:ApiKey`, which gates
`/localdb_sync.v1.LocalDbSync/` on the same listener via its own interceptor: that key
authenticates the *pair partner* for database replication, a different trust relationship, and the
two are never shared.
The transport remains h2c, so the key is readable and replayable by anyone on the path — the
boundary still assumes a trusted network, with the bar raised from "can reach the port" to "can
read the traffic". TLS is follow-on hardening and does not change this design.
#### Enriched AlarmStateUpdate (Native Alarm Mirror)
`AlarmStateUpdate` carries the read-only native alarm mirror (Computed, native OPC UA, and native MxAccess Gateway alarms) to central over the **existing gRPC real-time stream** — no new transport, no command/control round-trip. The message was extended **additively**: existing fields 17 are unchanged, and fields 823 carry the enriched native-alarm state. Old clients that only read fields 17 continue to work; new fields are populated only where the source provides them.
@@ -306,7 +331,9 @@ Akka.NET guarantees message ordering between a specific sender/receiver actor pa
## ManagementActor and ClusterClient
The ManagementActor is registered at the well-known path `/user/management` on central nodes and advertised via **ClusterClientReceptionist**. External tools (primarily the CLI) connect using Akka.NET ClusterClient, which contacts the receptionist to discover the ManagementActor. This is a separate ClusterClient usage from the inter-cluster ClusterClient connections used for central-site messaging — the CLI does not participate in cluster membership or affect the hub-and-spoke topology.
The ManagementActor runs at the well-known path `/user/management` on central nodes. It is **not** advertised via ClusterClientReceptionist, and no ClusterClient reaches it.
That registration existed until 2026-07-22 for an out-of-cluster CLI that was never built. The shipped CLI speaks **HTTP Basic to the central `/management` endpoints**; those endpoints ask the ManagementActor **in-process** through `ManagementActorHolder` (`ManagementEndpoints.cs`). Nothing in the repo ever sent to `/user/management` across the boundary, so the registration was deleted — leaving exactly one receptionist registration per cluster role (`CentralCommunicationActor` on central, `SiteCommunicationActor` on sites), both of which serve inter-cluster central↔site messaging and are themselves scheduled for removal by the gRPC transport migration (`docs/plans/2026-07-22-clusterclient-to-grpc-plan.md`).
## Connection Failure Behavior