feat(grpc): PSK-authenticate the site gRPC control plane; drop the vestigial management receptionist registration

Phase 0 of the ClusterClient→gRPC migration
(docs/plans/2026-07-22-clusterclient-to-grpc-plan.md). Standalone hardening: it
closes a gap that exists today and is a precondition for moving command/control
onto gRPC in later phases.

T0.1 — delete the ManagementActor ClusterClientReceptionist registration.
It was built for an out-of-cluster CLI that was never written: the shipped CLI
speaks HTTP Basic to /management, which asks the actor in-process through
ManagementActorHolder. Nothing in the repo ever sent to /user/management. The
actor still runs there; only the cross-boundary advertisement is gone. Six
documents claimed the CLI used ClusterClient — including the CLI's own README
"Architecture Notes" — and are corrected here rather than left to rot.

T0.2 — record, do not port, the dead integration-routing path.
IntegrationCallRequest is unwired at BOTH ends: RouteIntegrationCallAsync has
zero callers anywhere, and RegisterLocalHandler(Integration, …) appears only in
a test, so production always answers "Integration handler not available". It is
excluded from the gRPC contract (28 of 29 commands migrate) rather than
enshrined on an additive-only wire format, and deleting it during a
transport migration would mix a behavioural change into a change whose whole
value is that behaviour is identical. See
docs/known-issues/2026-07-22-integration-call-routing-is-dead-code.md.

T0.3 — preshared-key authentication on SiteStreamService.
The service shipped with no auth at all: plaintext h2c, no interceptor, so
anything that could reach a site node's :8083 could open a live data stream or
read audit rows back via PullAuditEvents/PullSiteCalls. ControlPlaneAuthInterceptor
now gates /sitestream.SiteStreamService/ — modeled on LocalDbSyncAuthInterceptor
(constant-time compare, fail-closed, PermissionDenied) but gating a SET of
service prefixes so phases 1A/1B add services rather than interceptors. LocalDb
sync keeps its own separate key: it authenticates the pair partner, not central,
and collapsing the two would make a site's central-facing key also admit writes
into its database.

Keys are per site (SB-GRPC-PSK-<siteId>), never fleet-wide, so a compromised
site yields only its own. Central attaches them through ControlPlaneCredentials,
which binds CallCredentials to the channel — covering unary and streaming
uniformly, and letting the key resolve asynchronously, which a client
interceptor could not do without blocking. All three central→site channel
creation sites go through it (SiteStreamGrpcClient and both audit pull invokers);
the pull invokers' channel caches are re-keyed by (site, endpoint) because
credentials are per-site and bound to the channel.

Two decisions beyond the plan:

  * StartupValidator now requires GrpcPsk on Site nodes. The plan specified only
    the runtime gate, but fail-closed with no boot check produces a node that
    joins, answers heartbeats and reports healthy while refusing every stream,
    audit pull and telemetry ingest — silent and total. Same reasoning as the
    existing inbound API-key pepper rule.

  * Added Communication:SitePsks as a central-side key map. The plan assumed
    central would read the store, seeded via a dev KEK; the docker rig
    deliberately boots with no master key, so store-only resolution would leave
    it unable to dial its own sites. The store stays primary — it is the only
    source that can serve a site added at runtime — with the map covering
    key-less hosts and one-off pins. Neither source falling back to
    "unauthenticated" is the invariant.

T0.4 — dev keys on both rigs and tests.
34 tests. The seven that matter most exercise a real in-process gRPC stack over
TestServer: the unit tests on either side of the wire would both stay green if
the halves disagreed, and gRPC refuses call credentials on a plaintext channel
by default — the UnsafeUseInsecureChannelCallCredentials opt-in is only provable
by making a real call. They confirm correct key passes on unary AND streaming,
wrong key and no-credentials both get PermissionDenied, and an unresolvable key
fails the call with nothing reaching the service.

OPERATIONAL: a site node upgraded to this build without a key will not boot.
That includes the gitignored deploy/wonder-app-vd03/ overlay.
This commit is contained in:
Joseph Doherty
2026-07-22 17:51:09 -04:00
parent f1ad967083
commit 2ee84af1c0
45 changed files with 1913 additions and 69 deletions
+8 -2
View File
@@ -30,7 +30,13 @@ The transports are independent. A gRPC stream interruption does not affect in-fl
**The gRPC dial direction is inverted from its data direction.** Values flow site → central, but each **site node hosts the gRPC server** and **central is the client**. `MapGrpcService<SiteStreamGrpcServer>()` appears exactly once in the tree, inside the Site branch of `Program.cs` (`Host/Program.cs:542`); there is **no gRPC server on a central node at all**. That is why the two `Ingest*` unary RPCs — nominally a central-side ingest surface — are dead in the shipped topology (acknowledged in `AkkaHostedService.cs:505-508`: "when the gRPC server is not registered (current central topology)"); sites push audit telemetry to central over `ClusterClient` instead, and central pulls with `PullAuditEvents` / `PullSiteCalls` by dialling the site.
**None of the three transports carries a mutual-auth or transport-encryption story.** `BuildHocon` emits no Akka `enable-ssl`, no secure cookie and no `trusted-selection-paths`, so remoting and `ClusterClient` are plaintext and unauthenticated. The site gRPC listener is **h2c**`ListenAnyIP(grpcPort, o => o.Protocols = HttpProtocols.Http2)` with no `UseHttps` (`Host/Program.cs:501-506`) — and `SiteStreamGrpcClient` opens a bare `GrpcChannel.ForAddress(endpoint, …)` with no credentials. `LocalDbSyncAuthInterceptor` shares that listener but gates **only** methods under `/localdb_sync.v1.LocalDbSync/`, explicitly letting every `SiteStreamService` call through untouched, so the whole surface — including the `PullAuditEvents` / `PullSiteCalls` RPCs that return audit rows — is callable by anything that can reach the site's gRPC port. The one authenticated piece is the HTTP config fetch, and its per-deployment token is the *entire* security boundary (the endpoint is `AllowAnonymous`). The design assumes a trusted network between central and sites.
**No transport carries transport encryption; two of the three now carry authentication.**
- **Akka remoting / `ClusterClient` — unauthenticated.** `BuildHocon` emits no `enable-ssl`, no secure cookie and no `trusted-selection-paths`, so the command/control path is plaintext and open to anything that can reach the remoting port.
- **gRPC — authenticated by preshared key since 2026-07-22.** The listener is still **h2c**`ListenAnyIP(grpcPort, o => o.Protocols = HttpProtocols.Http2)` with no `UseHttps` — but `ControlPlaneAuthInterceptor` gates every method under `/sitestream.SiteStreamService/`, including the `PullAuditEvents` / `PullSiteCalls` RPCs that return audit rows. It is fail-closed (no key ⇒ everything refused, and `StartupValidator` will not boot a site node in that state), compares with `CryptographicOperations.FixedTimeEquals`, and rejects with `PermissionDenied`. Central attaches the key via `ControlPlaneCredentials`, which binds `CallCredentials` to each channel so unary and streaming calls are covered uniformly — `SiteStreamGrpcClient`, `GrpcPullAuditEventsInvoker` and `GrpcPullSiteCallsInvoker` all build their channels through it. Keys are **per site** (`SB-GRPC-PSK-<siteId>`), so a compromised site yields only its own. `LocalDbSyncAuthInterceptor` shares the listener and keeps its own separate key on `/localdb_sync.v1.LocalDbSync/` — the two authenticate different peers (central vs. the pair partner) and are never shared.
- **HTTP config fetch — token-authenticated.** Its per-deployment token is the *entire* security boundary (the endpoint is `AllowAnonymous`).
A bearer PSK over plaintext h2c is readable and replayable by anyone on the path, so the design still assumes a trusted network between central and sites — but the bar is now "read the traffic" rather than "reach the port". TLS on these listeners is the follow-on hardening and would not change the key design. Operational detail: [`docs/deployment/topology-guide.md`](../deployment/topology-guide.md).
### Notify-and-fetch: the deployment-config HTTP path
@@ -262,7 +268,7 @@ Three layers of dead-client detection protect the gRPC stream path:
- [Notification Outbox (#21)](./NotificationOutbox.md) — `CentralCommunicationActor` routes `NotificationSubmit` / `NotificationStatusQuery` messages from sites to the `NotificationOutboxActor` proxy. `CommunicationService` Asks the proxy directly for central-UI outbox management calls.
- [Site Call Audit (#22)](./SiteCallAudit.md) — `CommunicationService` Asks the `SiteCallAuditActor` proxy directly for query and relay operations. `SiteCallAuditActor` issues `RetryParkedOperation` / `DiscardParkedOperation` relay commands to sites via `SiteEnvelope`; `SiteCommunicationActor` dispatches them to `_parkedMessageHandler`.
- [Store-and-Forward Engine (#6)](./StoreAndForward.md) — the site S&F Engine drives `NotificationSubmit` forwarding and cached-call telemetry emission through `SiteCommunicationActor`. Parked-message queries and retry/discard relay commands flow back the other way.
- [Management Service (#18)](./ManagementService.md) — `ManagementActor` is registered with `ClusterClientReceptionist` at `/user/management` on central; the CLI connects via its own separate `ClusterClient`. This is a distinct `ClusterClient` usage from the inter-cluster hub-and-spoke connections managed by this component. Management Service also hosts `DeploymentConfigEndpoints` — the `GET /api/internal/deployments/{id}/config` route that terminates the third (HTTP) transport, mapped in the central-role block alongside `/api/audit/*` and `/management`.
- [Management Service (#18)](./ManagementService.md) — `ManagementActor` runs at `/user/management` on central and is reached **in-process** through `ManagementActorHolder`; the CLI connects over HTTP, not `ClusterClient`. (It was `ClusterClientReceptionist`-registered until 2026-07-22, for a CLI that was never built that way.) So this component's `ClusterClient` usage is exclusively the inter-cluster hub-and-spoke connections. Management Service also hosts `DeploymentConfigEndpoints` — the `GET /api/internal/deployments/{id}/config` route that terminates the third (HTTP) transport, mapped in the central-role block alongside `/api/audit/*` and `/management`.
- Design spec: [Component-Communication.md](../requirements/Component-Communication.md).
## Troubleshooting
+7 -4
View File
@@ -56,17 +56,18 @@ Mutating handlers that call repositories directly invoke `AuditAsync` (backed by
### Actor lifecycle and registration
`AkkaHostedService` (in the Host) creates the `ManagementActor` under the path `/user/management` and registers it with `ClusterClientReceptionist`:
`AkkaHostedService` (in the Host) creates the `ManagementActor` under the path `/user/management` and publishes it to `ManagementActorHolder`, which is the only way anything reaches it:
```csharp
var mgmtActor = _actorSystem!.ActorOf(
Props.Create(() => new ManagementActor(_serviceProvider, mgmtLogger)),
"management");
ClusterClientReceptionist.Get(_actorSystem).RegisterService(mgmtActor);
var mgmtHolder = _serviceProvider.GetRequiredService<ManagementActorHolder>();
mgmtHolder.ActorRef = mgmtActor;
```
A `ClusterClientReceptionist.Get(_actorSystem).RegisterService(mgmtActor)` call sat between those two statements until 2026-07-22. It was deleted because nothing ever sent to it: the CLI it was built for uses HTTP, not ClusterClient.
`ClusterClientReceptionist` advertises the actor to `ClusterClient` senders without requiring them to join the Akka cluster. The `ManagementActorHolder.ActorRef` property is then the bridge from the HTTP endpoint (which runs in ASP.NET Core middleware) into the Akka actor world.
The actor declares an explicit supervisor strategy — one-for-one with Resume and no retry limit — to match the coordinator-actor convention and remain correct if child actors are added later.
@@ -154,9 +155,11 @@ Content-Type: application/json
A successful response is HTTP 200 with the JSON result. An authorization failure is HTTP 403 with `{ "error": "...", "code": "UNAUTHORIZED" }`.
### Sending a command via ClusterClient
### Sending a command in-process
The `ManagementActor` is also reachable from any `ClusterClient` that has a contact point into the central cluster. The actor is registered under `/system/receptionist` with the path `/user/management`. Callers construct and `Tell` a `ManagementEnvelope` and expect one of `ManagementSuccess`, `ManagementError`, or `ManagementUnauthorized` in reply.
`ManagementEnvelope` is also the in-process contract: a caller holding `ManagementActorHolder.ActorRef` asks the actor directly and expects one of `ManagementSuccess`, `ManagementError`, or `ManagementUnauthorized` in reply. `ManagementEndpoints` is that caller.
There is **no** out-of-process actor path. The actor was advertised via `ClusterClientReceptionist` until 2026-07-22, so a `ClusterClient` with a contact point into the central cluster could `Tell` it a `ManagementEnvelope`; no caller ever did, and the registration is gone. The HTTP endpoints above are the only remote management surface.
## Command Groups
+39 -3
View File
@@ -178,9 +178,45 @@ database from its peer rather than letting it rejoin.
### Central-Site Communication
- Sites connect to central via Akka.NET remoting.
- The `Communication:CentralSeedNode` setting in the site config points to one of the central nodes.
- If that central node is down, the site's communication actor will retry until it connects to the active central node.
Three transports cross the boundary, not one:
- **Akka ClusterClient** — command/control. Sites list every central node in
`ScadaBridge:Communication:CentralContactPoints`; contact rotation reaches whichever node
answers, so no "active central" needs to be identified. (There is no `Communication:CentralSeedNode`
setting — earlier revisions of this guide named one that never existed in the code.)
- **gRPC** — real-time data and audit pull. Note the direction is inverted from the data flow:
each **site node hosts the gRPC server** on `GrpcPort` (default 8083, h2c) and central dials in.
- **Plain HTTP** — the deploy config itself, fetched by the site with a per-deployment token.
#### gRPC control-plane preshared key (required)
Every site node must set `ScadaBridge:Communication:GrpcPsk`, and central must hold the same
value for that site. **`StartupValidator` refuses to boot a site node without it**, deliberately:
the gate is fail-closed, so an unset key would leave the node joined, healthy-looking and
answering heartbeats while refusing every gRPC call — no live subscriptions, no audit pull, no
cached-telemetry ingest.
| Side | Where the key lives |
|---|---|
| Site node (both nodes of the pair, identical) | `ScadaBridge:Communication:GrpcPsk`, in production `${secret:SB-GRPC-PSK-<siteId>}` |
| Central | secret `SB-GRPC-PSK-<siteId>` in its store — **or** `ScadaBridge:Communication:SitePsks:<siteId>` |
The store is the source that matters in production, because sites are added at runtime and their
keys cannot be enumerated in configuration at boot; `SitePsks` covers a host running without a
master key (the docker rig) and one-off pins.
One key **per site**, never one for the fleet: a compromised site must not yield another site's
key. And never share it with `LocalDb:Replication:ApiKey` — that authenticates the *pair partner*
for database replication, a different trust relationship on the same listener.
**Rotation:** set the new value on both sides, then restart the pair (pairs restart together
anyway — see above). **Upgrading to a build that has this gate requires seeding the key first**,
including in the on-host `deploy/` overlays.
The key is a bearer token over plaintext h2c, so it is readable and replayable by anyone on the
path. That is the accepted posture today — the same trusted-network assumption the boundary
already made, now with authentication rather than none. TLS on these listeners is follow-on
hardening and needs no change to the key design.
## Scaling Guidelines
@@ -0,0 +1,59 @@
# Integration call routing (`IntegrationCallRequest`) is dead on both ends
**Date:** 2026-07-22 · **Status:** OPEN (decision needed: wire or delete) · **Severity:** Low (no
runtime impact — the path cannot be reached) · **Area:** CentralSite Communication
## What
"Pattern 4: Integration Routing" — `CommunicationService.RouteIntegrationCallAsync`
`SiteEnvelope(IntegrationCallRequest)``SiteCommunicationActor` → an integration handler — is
plumbed end to end but connected at neither end.
- **No producer.** `RouteIntegrationCallAsync` (`CommunicationService.cs`, "Pattern 4") has **zero
callers** in `src/` or `tests/`. It is the only one of `CommunicationService`'s command methods
with none.
- **No handler.** `SiteCommunicationActor` forwards to `_integrationHandler` when one is
registered, but `RegisterLocalHandler(LocalHandlerType.Integration, …)` appears **only** in
`SiteCommunicationActorTests.cs`. `AkkaHostedService` registers the other three handler types
(`Artifacts`, `EventLog`, `ParkedMessages`) and never this one.
So if anything ever did call it, the site would answer
`IntegrationCallResponse(Success: false, Error: "Integration handler not available")`
(`SiteCommunicationActor.cs`, Pattern 4) — and the two tests that exercise the path both register
the handler themselves first, which is why the suite has never noticed.
Do not confuse this with the **Inbound API**'s routed-site-script path, which is live, tested, and
uses different messages entirely. This is a separate, unused routing pattern that predates it.
## Why it is recorded rather than fixed
Found during the recon for the ClusterClient→gRPC transport migration
([`docs/plans/2026-07-22-clusterclient-to-grpc-plan.md`](../plans/2026-07-22-clusterclient-to-grpc-plan.md),
T0.2), which had to enumerate every command crossing the site↔central boundary. Of the **29**
command types, this is the one that is excluded: **28 migrate to the gRPC contract.**
Porting it would mean designing a proto contract, a `oneof` slot and round-trip mapper tests for a
verb no caller can invoke and no site can service — and enshrining it on a wire format whose
evolution rules are additive-only, so an unused RPC slot is permanent. Deleting it during a
transport migration would mix an unrelated behavioural change into a change whose whole value is
that behaviour is identical. Hence: excluded from the contract, behaviour untouched, decision
deferred to its own change.
## Decision needed
Either:
1. **Delete** — remove `RouteIntegrationCallAsync`, the `IntegrationCallRequest`/`Response`
messages, the `SiteCommunicationActor` receive block, `LocalHandlerType.Integration`, and the
three tests that cover them. This is the default if no consumer is planned.
2. **Wire** — register a real integration handler on site nodes and give the method a caller. This
only makes sense if there is a requirement it serves; none is recorded in
`docs/requirements/`.
Whichever is chosen, do it **before Phase 4** of the migration, since Phase 4 deletes the Akka
transport underneath this path. If it is still dead at that point, option 1 is forced.
## Filing
To be filed as a Gitea issue on `dohertj2/scadabridge` by the repo owner — this note is the
in-repo record of the finding and of the migration exclusion it justifies.
@@ -302,10 +302,10 @@ Critical path ≈ 1B: **~46 weeks total**, matching the design estimate.
## Task checklist (tick as you go; IDs reference the sections above)
**Phase 0 — PSK + dead code** (branch `feat/grpc-phase0-psk`)
- [ ] T0.1 Delete `/user/management` receptionist registration (`AkkaHostedService.cs:459`) + Component-Host.md update
- [ ] T0.2 File dead-`IntegrationCallRequest` issue; record exclusion (28 of 29 migrate)
- [ ] T0.3 `ControlPlaneAuthInterceptor` + `CommunicationOptions.GrpcPsk` + `SitePskProvider`; gate SiteStream; PSK attached on central's streaming + pull clients
- [ ] T0.4 Rig dev keys (all 3 sites + central store seeds) + interceptor/provider/wiring tests
- [x] T0.1 Delete `/user/management` receptionist registration (`AkkaHostedService.cs:459`) + Component-Host.md update
- [x] T0.2 File dead-`IntegrationCallRequest` issue; record exclusion (28 of 29 migrate)
- [x] T0.3 `ControlPlaneAuthInterceptor` + `CommunicationOptions.GrpcPsk` + `SitePskProvider`; gate SiteStream; PSK attached on central's streaming + pull clients
- [x] T0.4 Rig dev keys (all 3 sites + central store seeds) + interceptor/provider/wiring tests
- [ ] Phase 0 DoD: suite green; rig unauthenticated ⇒ `PermissionDenied`, authenticated paths work; PR merged
**Phase 1A — central control plane** (worktree, `feat/grpc-central-control`)
@@ -0,0 +1,241 @@
{
"plan": "docs/plans/2026-07-22-clusterclient-to-grpc-plan.md",
"design": "~/Desktop/scadaproj/scadabridge_clusterclient_to_grpc.md",
"worktrees": {
"feat/grpc-phase0-psk": "/Users/dohertj2/Desktop/ScadaBridge-phase0"
},
"tasks": [
{
"id": "T0.1",
"phase": "0",
"subject": "Delete the vestigial /user/management receptionist registration",
"status": "completed",
"activeForm": "Deleting the /user/management receptionist registration",
"files": [
"src/ZB.MOM.WW.ScadaBridge.Host/Actors/AkkaHostedService.cs",
"docs/requirements/Component-Host.md",
"docs/requirements/Component-ManagementService.md",
"docs/requirements/Component-Communication.md",
"docs/components/ManagementService.md",
"docs/components/Communication.md",
"src/ZB.MOM.WW.ScadaBridge.CLI/README.md"
],
"notes": "Actor stays; only the ClusterClientReceptionist.RegisterService call goes. Verified: CLI is Akka-free, so 6 docs claiming the CLI reaches ManagementActor over ClusterClient are stale and must be corrected in the same change."
},
{
"id": "T0.2",
"phase": "0",
"subject": "Record the dead IntegrationCallRequest exclusion (28 of 29 commands migrate)",
"status": "completed",
"activeForm": "Recording the dead IntegrationCallRequest exclusion",
"files": [
"docs/known-issues/"
],
"notes": "Plan says 'file a Gitea issue' \u2014 outward-facing, handed to the user. In-repo half is a known-issues note + the exclusion record."
},
{
"id": "T0.3",
"phase": "0",
"subject": "ControlPlaneAuthInterceptor + CommunicationOptions.GrpcPsk + SitePskProvider; gate SiteStream; attach PSK on central's clients",
"status": "completed",
"activeForm": "Building the control-plane PSK auth",
"files": [
"src/ZB.MOM.WW.ScadaBridge.Host/ControlPlaneAuthInterceptor.cs",
"src/ZB.MOM.WW.ScadaBridge.Communication/CommunicationOptions.cs",
"src/ZB.MOM.WW.ScadaBridge.Communication/Grpc/ISitePskProvider.cs",
"src/ZB.MOM.WW.ScadaBridge.Host/SitePskProvider.cs",
"src/ZB.MOM.WW.ScadaBridge.Communication/Grpc/SiteStreamGrpcClient.cs",
"src/ZB.MOM.WW.ScadaBridge.Communication/Grpc/SiteStreamGrpcClientFactory.cs",
"src/ZB.MOM.WW.ScadaBridge.AuditLog/Central/GrpcPullAuditEventsClient.cs",
"src/ZB.MOM.WW.ScadaBridge.AuditLog/Central/GrpcPullSiteCallsClient.cs",
"src/ZB.MOM.WW.ScadaBridge.Host/Program.cs"
],
"notes": "Service prefix verified from sitestream.proto: /sitestream.SiteStreamService/. Grpc 2.76 -> CallCredentials.FromInterceptor + UnsafeUseInsecureChannelCallCredentials is the async-safe attach path on h2c."
},
{
"id": "T0.4",
"phase": "0",
"subject": "Rig dev keys (3 sites + central secret seeds) + interceptor/provider/wiring tests",
"status": "completed",
"activeForm": "Seeding rig dev keys and writing the auth tests",
"files": [
"docker/site-a-node-a/appsettings.Site.json",
"docker/site-a-node-b/appsettings.Site.json",
"docker/site-b-node-a/appsettings.Site.json",
"docker/site-b-node-b/appsettings.Site.json",
"docker/site-c-node-a/appsettings.Site.json",
"docker/site-c-node-b/appsettings.Site.json",
"tests/ZB.MOM.WW.ScadaBridge.Host.Tests/ControlPlaneAuthInterceptorTests.cs"
],
"notes": "Fail-closed from day one \u2014 every environment (incl. docker-env2 and the gitignored deploy/wonder-app-vd03 overlay) needs its key before upgrade. Ops item for the user."
},
{
"id": "P0.DoD",
"phase": "0",
"subject": "Phase 0 DoD: suite green; rig unauthenticated => PermissionDenied, authenticated paths work; PR merged",
"status": "in_progress",
"activeForm": "Verifying the Phase 0 DoD",
"blockedBy": [
"T0.1",
"T0.2",
"T0.3",
"T0.4"
],
"notes": "Rig half needs user approval (docker/deploy.sh recreates 8 containers)."
},
{
"id": "T1A.1",
"phase": "1A",
"subject": "central_control.proto (7 RPCs, checked-in codegen) + CentralControlDtoMapper + round-trip golden tests",
"status": "pending",
"activeForm": "Authoring central_control.proto and its mappers",
"blockedBy": [
"P0.DoD"
]
},
{
"id": "T1A.2",
"phase": "1A",
"subject": "Central hosting: AddGrpc + per-site-PSK interceptor, CentralGrpcPort h2c listener, CentralControlGrpcService, readiness gate",
"status": "pending",
"activeForm": "Hosting CentralControlService on central",
"blockedBy": [
"T1A.1"
]
},
{
"id": "T1A.3",
"phase": "1A",
"subject": "ICentralTransport (Akka extract + Grpc impl), CentralChannelProvider, CentralTransport flag, CentralGrpcEndpoints option",
"status": "pending",
"activeForm": "Building the site->central transport seam",
"blockedBy": [
"T1A.1"
]
},
{
"id": "T1A.4",
"phase": "1A",
"subject": "Tests: actor-with-fake-transport x7, TestServer transport tests, S&F/audit/health suites pass unmodified",
"status": "pending",
"activeForm": "Testing the central control plane",
"blockedBy": [
"T1A.2",
"T1A.3"
]
},
{
"id": "P1A.DoD",
"phase": "1A",
"subject": "1A DoD: rig site-a on Grpc proves all 5 site->central paths while site-b/c stay Akka; PR merged before 1B",
"status": "pending",
"activeForm": "Verifying the 1A DoD",
"blockedBy": [
"T1A.4"
]
},
{
"id": "T1B.1",
"phase": "1B",
"subject": "site_command.proto (6 oneof RPCs / 28 commands) + SiteCommandDtoMapper + round-trip golden tests",
"status": "pending",
"activeForm": "Authoring site_command.proto and its mappers",
"blockedBy": [
"P0.DoD"
]
},
{
"id": "T1B.2",
"phase": "1B",
"subject": "SiteCommandDispatcher refactor (actor + SiteCommandGrpcService share it; parked stays node-local)",
"status": "pending",
"activeForm": "Extracting the site command dispatcher",
"blockedBy": [
"T1B.1"
]
},
{
"id": "T1B.3",
"phase": "1B",
"subject": "ISiteCommandTransport in CentralCommunicationActor (Akka extract + Grpc impl), SitePairChannelProvider, SiteTransport flag",
"status": "pending",
"activeForm": "Building the central->site transport seam",
"blockedBy": [
"T1B.1"
]
},
{
"id": "T1B.4",
"phase": "1B",
"subject": "Tests: dispatcher routing x28, actor envelope/reply plumbing, TestServer service tests, existing suites green",
"status": "pending",
"activeForm": "Testing the site command plane",
"blockedBy": [
"T1B.2",
"T1B.3"
]
},
{
"id": "P1B.DoD",
"phase": "1B",
"subject": "1B DoD: rig central on Grpc for site-a proves full command matrix incl. standby parked retry; rebased on 1A; PR merged",
"status": "pending",
"activeForm": "Verifying the 1B DoD",
"blockedBy": [
"T1B.4",
"P1A.DoD"
]
},
{
"id": "P2",
"phase": "2",
"subject": "All sites CentralTransport=Grpc; central-kill S&F soak, failback observed, health sequences clean",
"status": "pending",
"activeForm": "Running the site->central cutover soak",
"blockedBy": [
"P1A.DoD"
]
},
{
"id": "P3",
"phase": "3",
"subject": "Central SiteTransport=Grpc all sites; full UI command matrix; site-kill mid-command clean; zero ClusterClient activity",
"status": "pending",
"activeForm": "Running the central->site cutover soak",
"blockedBy": [
"P1B.DoD"
]
},
{
"id": "P4.1",
"phase": "4",
"subject": "Flip both flag defaults to Grpc + soak; delete Akka transports, ClusterClient creation, DefaultSiteClientFactory, receptionist registrations, CentralContactPoints, then the flags",
"status": "pending",
"activeForm": "Deleting the ClusterClient transport",
"blockedBy": [
"P2",
"P3"
]
},
{
"id": "P4.2",
"phase": "4",
"subject": "Grep-gates pass + docs updated (grpc_streams.md, Component-Host.md, Component-StoreAndForward.md, known-issues cross-ref)",
"status": "pending",
"activeForm": "Running the deletion grep-gates and doc sweep",
"blockedBy": [
"P4.1"
]
},
{
"id": "P5",
"phase": "5",
"subject": "Live gate, 8 checks, recorded in docs/plans/2026-07-22-clusterclient-to-grpc-live-gate.md",
"status": "pending",
"activeForm": "Running the live gate",
"blockedBy": [
"P4.2"
]
}
]
}
+28 -1
View File
@@ -109,6 +109,31 @@ The streaming protocol is defined in `sitestream.proto` (`src/ZB.MOM.WW.ScadaBri
- The `oneof event` pattern is extensible — future event types (health metrics, connection state changes) are added as new fields without breaking existing consumers.
- Proto field numbers are never reused; new RPCs and message fields are appended additively. Old clients ignore unknown `oneof` variants.
##### Authentication (preshared key, 2026-07-22)
Every RPC on this service — streaming and unary alike — requires a preshared key, presented as
`authorization: Bearer <key>` metadata and verified by `ControlPlaneAuthInterceptor` on the site
node. The gate is **fail-closed**: with no key configured, every call is refused with
`PermissionDenied`, and `StartupValidator` refuses to boot a site node in that state (an unset key
would otherwise leave the node joined and reporting healthy while serving nothing).
Keys are scoped **one per site** — secret `SB-GRPC-PSK-<siteId>`, so a compromised site never
yields another site's key. The site node reads its own key from
`ScadaBridge:Communication:GrpcPsk` (in production a `${secret:}` reference expanded before the
host is built); central resolves each site's key at channel-build time via `SitePskProvider`,
because sites are created at runtime and cannot be enumerated in configuration at boot.
`ControlPlaneCredentials` binds the key to the channel as `CallCredentials`, so no individual call
site can omit it.
This is distinct from `LocalDb:Replication:ApiKey`, which gates
`/localdb_sync.v1.LocalDbSync/` on the same listener via its own interceptor: that key
authenticates the *pair partner* for database replication, a different trust relationship, and the
two are never shared.
The transport remains h2c, so the key is readable and replayable by anyone on the path — the
boundary still assumes a trusted network, with the bar raised from "can reach the port" to "can
read the traffic". TLS is follow-on hardening and does not change this design.
#### Enriched AlarmStateUpdate (Native Alarm Mirror)
`AlarmStateUpdate` carries the read-only native alarm mirror (Computed, native OPC UA, and native MxAccess Gateway alarms) to central over the **existing gRPC real-time stream** — no new transport, no command/control round-trip. The message was extended **additively**: existing fields 17 are unchanged, and fields 823 carry the enriched native-alarm state. Old clients that only read fields 17 continue to work; new fields are populated only where the source provides them.
@@ -306,7 +331,9 @@ Akka.NET guarantees message ordering between a specific sender/receiver actor pa
## ManagementActor and ClusterClient
The ManagementActor is registered at the well-known path `/user/management` on central nodes and advertised via **ClusterClientReceptionist**. External tools (primarily the CLI) connect using Akka.NET ClusterClient, which contacts the receptionist to discover the ManagementActor. This is a separate ClusterClient usage from the inter-cluster ClusterClient connections used for central-site messaging — the CLI does not participate in cluster membership or affect the hub-and-spoke topology.
The ManagementActor runs at the well-known path `/user/management` on central nodes. It is **not** advertised via ClusterClientReceptionist, and no ClusterClient reaches it.
That registration existed until 2026-07-22 for an out-of-cluster CLI that was never built. The shipped CLI speaks **HTTP Basic to the central `/management` endpoints**; those endpoints ask the ManagementActor **in-process** through `ManagementActorHolder` (`ManagementEndpoints.cs`). Nothing in the repo ever sent to `/user/management` across the boundary, so the registration was deleted — leaving exactly one receptionist registration per cluster role (`CentralCommunicationActor` on central, `SiteCommunicationActor` on sites), both of which serve inter-cluster central↔site messaging and are themselves scheduled for removal by the gRPC transport migration (`docs/plans/2026-07-22-clusterclient-to-grpc-plan.md`).
## Connection Failure Behavior
+5 -1
View File
@@ -125,7 +125,11 @@ The Host bootstraps the Akka.NET actor system from a **hand-assembled, injection
### REQ-HOST-6a: ClusterClientReceptionist (Central Only)
On central nodes, the Host must configure the Akka.NET **ClusterClientReceptionist** and register the ManagementActor with it. This allows external processes (e.g., the CLI) to discover and communicate with the ManagementActor via ClusterClient without joining the cluster as full members. The receptionist is started as part of the Akka.NET bootstrap (REQ-HOST-6) on central nodes only.
On central nodes, the Host must configure the Akka.NET **ClusterClientReceptionist** and register the **CentralCommunicationActor** with it, so that site clusters' ClusterClients can reach the central command/control endpoint without joining the central cluster.
**The ManagementActor is NOT registered with the receptionist** (removed 2026-07-22). That registration was written for an out-of-cluster CLI that was never built: the shipped CLI speaks HTTP Basic to the central `/management` endpoints, which ask the ManagementActor **in-process** via `ManagementActorHolder` (`ManagementEndpoints.cs`). No sender to `/user/management` existed anywhere in the repo, so the registration only widened the cluster-client surface for nothing. The actor itself still runs at `/user/management`; only its cross-boundary advertisement is gone.
> **Migration note.** This receptionist registration — and the `CentralCommunicationActor` one that remains — are scheduled for deletion once the site↔central transport moves to gRPC. See `docs/plans/2026-07-22-clusterclient-to-grpc-plan.md`.
### REQ-HOST-7: ASP.NET Web Endpoints
@@ -24,7 +24,7 @@ Central cluster only. The ManagementActor runs as a plain actor on **every** cen
### ManagementActor
The central actor that receives and processes all management commands. Registered at a well-known actor path (`/user/management`) and with ClusterClientReceptionist.
The central actor that receives and processes all management commands. Created at the well-known actor path (`/user/management`) and handed to `ManagementActorHolder`, which is how every in-process caller (notably `ManagementEndpoints`) reaches it. It is **not** advertised via ClusterClientReceptionist — see the note under "ManagementActor and ClusterClient" in Component-Communication.
### ManagementEndpoints