feat(comm): Phase 4 — delete Akka ClusterClient site↔central transport, gRPC-only

ClusterClient→gRPC migration Phase 4 (docs/plans/2026-07-22-clusterclient-to-grpc-plan.md).
Phases 2/3 proved both directions on gRPC; this removes the Akka transport underneath.

Deleted:
- AkkaCentralTransport, AkkaSiteTransport (+ their dedicated tests)
- ISiteClientFactory + DefaultSiteClientFactory; CentralCommunicationActor legacy
  ctor + SelectTransport (Host now builds GrpcSiteTransport and injects it)
- ClusterClient creation + both ClusterClientReceptionist.RegisterService calls in
  AkkaHostedService; the RegisterCentralClient message + receive block
- CommunicationOptions.CentralContactPoints; the CentralTransport/SiteTransport
  coexistence flags; the CentralTransportMode/SiteTransportKind enums

gRPC is now the only site↔central transport (site→central CentralControlService via
GrpcCentralTransport; central→site SiteCommandService via GrpcSiteTransport), both
built unconditionally by the Host. NoOpCentralTransport is the fail-loud null-default
so TestKit command-dispatch suites still construct the site actor without a wired
transport; production always injects GrpcCentralTransport.

Config: CentralGrpcEndpoints is now unconditional — CommunicationOptionsValidator
rejects blank entries (role-agnostic), and StartupValidator requires a Site node to
list >=1 endpoint (fail-fast, mirrors GrpcPsk). Rig configs moved
CentralContactPoints -> CentralGrpcEndpoints (docker x6, docker-env2 x2, Host default,
deploy/wonder-app-vd03). Kept Akka.Cluster.Tools (ClusterSingleton still used).

Tests: build 0/0; Communication.Tests 640, Host.Tests 421 green. Removed the
ClusterClient.Send per-site-routing tests (covered by the transport suites), swapped
the ISiteClientFactory-based ctors to a substitute ISiteCommandTransport, converted
the audit-push integration relay to an in-process bridge transport.

Docs: Component-Communication/Host/StoreAndForward, components/Communication,
topology-guide, grpc_streams (SUPERSEDED note), the frame-size known-issue (retired
amendment), and CLAUDE.md transport decisions.

Not included: the dead IntegrationCallRequest path (#32) is a separate user-owned
behavioral decision — SiteEnvelope routing is transport-agnostic so it still compiles.
This commit is contained in:
Joseph Doherty
2026-07-23 12:54:32 -04:00
parent 3a8ddb7087
commit 7fd5cb2b56
42 changed files with 479 additions and 1295 deletions
+20 -17
View File
@@ -1,6 +1,6 @@
# CentralSite Communication
The CentralSite Communication component is the transport layer that connects the central cluster to every site cluster. It provides three independent transports — Akka.NET `ClusterClient` for command/control, gRPC server-streaming for real-time data, and plain token-gated HTTP for the deployment-config fetch — anchored by a pair of actors that each cluster registers with the `ClusterClientReceptionist`.
The CentralSite Communication component is the transport layer that connects the central cluster to every site cluster. It provides three independent transports, **all now gRPC-based across the boundary** (Akka `ClusterClient` was removed in Phase 4 of the ClusterClient→gRPC migration, 2026-07-23): gRPC command/control in both directions (`CentralControlService` on central via `GrpcCentralTransport`; `SiteCommandService` on the site via `GrpcSiteTransport`), gRPC server-streaming for real-time data (`SiteStreamService`), and plain token-gated HTTP for the deployment-config fetch. Cross-cluster Akka remoting and `ClusterClientReceptionist` are gone; Akka remoting is now intra-cluster only.
## Overview
@@ -22,25 +22,26 @@ DI registration is called from the Host composition root via `AddCommunication`.
| Transport | Who dials | Data direction | Purpose |
|-----------|-----------|----------------|---------|
| Akka.NET `ClusterClient` | both (central → site per site; site → central) | bidirectional | Deploy notifies, lifecycle, subscribe/unsubscribe handshake, snapshots, heartbeats, health reports, telemetry, notifications |
| gRPC command/control (`CentralControlService`) | **site dials central** | site → central | Heartbeats, health reports, notification submit/status, audit + cached-telemetry ingest, reconcile |
| gRPC command/control (`SiteCommandService`) | **central dials the site** | central → site | Deploy notifies, lifecycle, OPC UA, remote queries, subscribe/unsubscribe handshake, snapshots, parked, route, failover |
| gRPC (`SiteStreamService`) | **central dials the site** | mostly site → central | Real-time attribute value and alarm state changes (server-streaming), plus the audit ingest/pull unary RPCs |
| HTTP `GET` (`/api/internal/deployments/{id}/config`) | **site dials central** | central → site | The flattened deployment config itself (notify-and-fetch), gated by a per-deployment `X-Deployment-Token` |
The transports are independent. A gRPC stream interruption does not affect in-flight `ClusterClient` commands, and vice versa.
The transports are independent. A gRPC stream interruption does not affect in-flight command/control calls, and vice versa.
**The gRPC dial direction is inverted from its data direction.** Values flow site → central, but each **site node hosts the gRPC server** and **central is the client**. `MapGrpcService<SiteStreamGrpcServer>()` appears exactly once in the tree, inside the Site branch of `Program.cs` (`Host/Program.cs:542`); there is **no gRPC server on a central node at all**. That is why the two `Ingest*` unary RPCs — nominally a central-side ingest surface — are dead in the shipped topology (acknowledged in `AkkaHostedService.cs:505-508`: "when the gRPC server is not registered (current central topology)"); sites push audit telemetry to central over `ClusterClient` instead, and central pulls with `PullAuditEvents` / `PullSiteCalls` by dialling the site.
**`SiteStreamService`'s gRPC dial direction is inverted from its data direction.** Values flow site → central, but each **site node hosts the `SiteStreamService` server** and **central is the client**. Command/control is different: it runs on its own two services — `CentralControlService` (central-hosted, site dials in) and `SiteCommandService` (site-hosted, central dials in). So a central node now DOES host a gRPC server (`CentralControlService`), unlike before Phase 4. The two legacy `Ingest*` unary RPCs on the site-hosted `SiteStreamService` remain dead in the shipped topology (no site dials a site for ingest); audit telemetry is pushed site → central over `CentralControlService`, and central pulls with `PullAuditEvents` / `PullSiteCalls` by dialling the site's `SiteStreamService`.
**No transport carries transport encryption; two of the three now carry authentication.**
- **Akka remoting / `ClusterClient` — unauthenticated.** `BuildHocon` emits no `enable-ssl`, no secure cookie and no `trusted-selection-paths`, so the command/control path is plaintext and open to anything that can reach the remoting port.
- **gRPC — authenticated by preshared key since 2026-07-22.** The listener is still **h2c**`ListenAnyIP(grpcPort, o => o.Protocols = HttpProtocols.Http2)` with no `UseHttps` — but `ControlPlaneAuthInterceptor` gates every method under `/sitestream.SiteStreamService/`, including the `PullAuditEvents` / `PullSiteCalls` RPCs that return audit rows. It is fail-closed (no key ⇒ everything refused, and `StartupValidator` will not boot a site node in that state), compares with `CryptographicOperations.FixedTimeEquals`, and rejects with `PermissionDenied`. Central attaches the key via `ControlPlaneCredentials`, which binds `CallCredentials` to each channel so unary and streaming calls are covered uniformly`SiteStreamGrpcClient`, `GrpcPullAuditEventsInvoker` and `GrpcPullSiteCallsInvoker` all build their channels through it. Keys are **per site** (`SB-GRPC-PSK-<siteId>`), so a compromised site yields only its own. `LocalDbSyncAuthInterceptor` shares the listener and keeps its own separate key on `/localdb_sync.v1.LocalDbSync/` — the two authenticate different peers (central vs. the pair partner) and are never shared.
- **Akka remoting — unauthenticated, but intra-cluster only.** `BuildHocon` emits no `enable-ssl`, no secure cookie and no `trusted-selection-paths`, so remoting is plaintext and open to anything that can reach the remoting port — but as of Phase 4 remoting no longer crosses the site↔central boundary, so this exposure is pair-internal.
- **gRPC — authenticated by preshared key since 2026-07-22.** The listeners are still **h2c**`ListenAnyIP(grpcPort, o => o.Protocols = HttpProtocols.Http2)` with no `UseHttps` — but `ControlPlaneAuthInterceptor` gates the command/control and streaming services alike: every method under `/sitestream.SiteStreamService/` (including the `PullAuditEvents` / `PullSiteCalls` RPCs that return audit rows) **and** the Phase-4 `CentralControlService` (site→central) / `SiteCommandService` (central→site). It is fail-closed (no key ⇒ everything refused, and `StartupValidator` will not boot a site node in that state), compares with `CryptographicOperations.FixedTimeEquals`, and rejects with `PermissionDenied`. The caller attaches the key via `ControlPlaneCredentials`, which binds `CallCredentials` to each channel so unary and streaming calls are covered uniformly. Keys are **per site** (`SB-GRPC-PSK-<siteId>`), so a compromised site yields only its own. `LocalDbSyncAuthInterceptor` shares the listener and keeps its own separate key on `/localdb_sync.v1.LocalDbSync/` — the two authenticate different peers (central vs. the pair partner) and are never shared.
- **HTTP config fetch — token-authenticated.** Its per-deployment token is the *entire* security boundary (the endpoint is `AllowAnonymous`).
A bearer PSK over plaintext h2c is readable and replayable by anyone on the path, so the design still assumes a trusted network between central and sites — but the bar is now "read the traffic" rather than "reach the port". TLS on these listeners is the follow-on hardening and would not change the key design. Operational detail: [`docs/deployment/topology-guide.md`](../deployment/topology-guide.md).
### Notify-and-fetch: the deployment-config HTTP path
An instance deployment does not carry its flattened configuration inside the Akka message. Central stages a `PendingDeployment` row (config JSON + a freshly generated `DeploymentFetchToken` + a TTL) and sends only a small `RefreshDeploymentCommand` over `ClusterClient`, carrying the deployment id, revision hash, `CentralFetchBaseUrl` and the fetch token. The site's Deployment Manager singleton then calls back to central over plain HTTP:
An instance deployment does not carry its flattened configuration inside the command message. Central stages a `PendingDeployment` row (config JSON + a freshly generated `DeploymentFetchToken` + a TTL) and sends only a small `RefreshDeploymentCommand` over gRPC command/control (`SiteCommandService`), carrying the deployment id, revision hash, `CentralFetchBaseUrl` and the fetch token. The site's Deployment Manager singleton then calls back to central over plain HTTP:
```csharp
// SiteRuntime/Deployment/HttpDeploymentConfigFetcher.cs
@@ -49,11 +50,11 @@ using var req = new HttpRequestMessage(HttpMethod.Get, url);
req.Headers.Add("X-Deployment-Token", token);
```
`DeploymentConfigEndpoints.Resolve` (`ManagementService/DeploymentConfigEndpoints.cs:101`) checks existence and TTL *before* the token, so unknown, superseded and expired deployments are all indistinguishable `404`s; a live row with a wrong or missing token is `401`. The token comparison is constant-time. This exists because a flattened config can exceed the default 128 KB Akka frame size, which drops the single oversized message without tearing down the association — heartbeats keep flowing, the site still reports healthy, and the deploy just hangs to its Ask timeout. See `docs/known-issues/2026-06-26-deploy-config-exceeds-akka-frame-size.md`. `DeployArtifactsCommand` was **not** moved to this path and still carries its payload inline.
`DeploymentConfigEndpoints.Resolve` (`ManagementService/DeploymentConfigEndpoints.cs:101`) checks existence and TTL *before* the token, so unknown, superseded and expired deployments are all indistinguishable `404`s; a live row with a wrong or missing token is `401`. The token comparison is constant-time. This path was originally introduced because a flattened config could exceed the default 128 KB Akka frame size when command/control rode ClusterClient — the oversized message was silently dropped and the deploy hung to its timeout (see `docs/known-issues/2026-06-26-deploy-config-exceeds-akka-frame-size.md`). That Akka frame limit no longer applies now that command/control rides gRPC (`SiteCommandService`, 4 MB cap), but notify-and-fetch remains the deploy path. `DeployArtifactsCommand` was **not** moved to this path and still carries its payload inline — now over gRPC, so it is no longer at risk of the 128 KB drop.
### Hub-and-spoke topology
Sites do not communicate with each other. All inter-cluster traffic flows through central. Central maintains one `ClusterClient` per site; each site maintains a single `ClusterClient` pointed at both central nodes.
Sites do not communicate with each other. All inter-cluster traffic flows through central. Central maintains one `GrpcSiteTransport` channel pair per site (dialling `SiteCommandService`); each site maintains a single `GrpcCentralTransport` channel pair pointed at both central nodes (dialling `CentralControlService`).
### `SiteEnvelope` routing
@@ -94,12 +95,14 @@ If a site is unreachable when a command arrives, the caller's Ask times out. Cen
## Architecture
> **Phase 4 note (2026-07-23).** The `ClusterClient` mechanics described in this section — `SiteEnvelope` routing via `ClusterClient.Send("/user/site-communication", …)`, one `ClusterClient` per site on `CentralCommunicationActor`, the site's outbound `_centralClient`, and `ClusterClientReceptionist` registration — were **replaced by gRPC command/control** in Phase 4 of the ClusterClient→gRPC migration. Central→site commands now go over `GrpcSiteTransport` → the site-hosted `SiteCommandService` (per-site NodeA→NodeB failover channel pair), and site→central messages over `GrpcCentralTransport` → the central-hosted `CentralControlService` (sticky central-a→central-b pair). Endpoints are dialled directly (no receptionist, no cross-cluster actor discovery). The per-site address load below still runs, but it now feeds a **per-site gRPC-endpoint cache** (from `GrpcNodeAAddress`/`GrpcNodeBAddress`) consumed by a `SitePairChannelProvider`, not a `ClusterClient` contact set. See [Component-Communication.md](../requirements/Component-Communication.md) for the current design.
### Central-side: `CentralCommunicationActor`
`CentralCommunicationActor` is a `ReceiveActor` created at `/user/central-communication` and registered with `ClusterClientReceptionist` so the site's `ClusterClient` can locate it. It owns:
- A `Dictionary<string, (IActorRef Client, ImmutableHashSet<string> ContactAddresses)>` keyed by site identifier — one `ClusterClient` per site.
- A `RefreshSiteAddresses` periodic timer (60-second cadence, starting immediately). Each tick fires `LoadSiteAddressesFromDb`, which reads every `Site` row from the database, parses `NodeAAddress` and `NodeBAddress` into Akka receptionist paths (`{addr}/system/receptionist`), and pipes a `SiteAddressCacheLoaded` message back to Self. `HandleSiteAddressCacheLoaded` creates, updates, or stops `ClusterClient` actors based on the diff.
- A `RefreshSiteAddresses` periodic timer (60-second cadence, starting immediately). Each tick fires `LoadSiteAddressesFromDb`, which reads every `Site` row from the database and (as of Phase 4) builds a **per-site gRPC-endpoint cache** from `GrpcNodeAAddress`/`GrpcNodeBAddress`, piping a `SiteAddressCacheLoaded` message back to Self. `HandleSiteAddressCacheLoaded` reconciles the diff, updating the `SitePairChannelProvider` that `GrpcSiteTransport` uses to dial each site's `SiteCommandService`. (Before Phase 4 this parsed the Akka `NodeAAddress`/`NodeBAddress` into receptionist paths and created/updated/stopped a `ClusterClient` per site.)
- Proxy references to `NotificationOutboxActor` and `AuditLogIngestActor` cluster singletons, injected post-construction via `RegisterNotificationOutbox` / `RegisterAuditIngest` messages from the Host. Messages that arrive before the proxy is registered are answered with a non-accepted ack (notifications) or an empty reply (audit), so the site retries without data loss.
- Fanout of `SiteHealthReport` to the peer central node via `DistributedPubSub`, keyed on the `site-health-replica` topic, so both central nodes' aggregators stay in sync regardless of which central node the site's `ClusterClient` load-balanced the report to.
@@ -236,7 +239,7 @@ All options are bound from the `ScadaBridge:Communication` section via `Communic
| `IntegrationTimeout` | `00:00:30` | Ask timeout for integration routing and Inbound API routing. |
| `DebugViewTimeout` | `00:00:10` | Ask timeout for debug subscribe/unsubscribe handshake. |
| `NotificationForwardTimeout` | `00:00:30` | Ask timeout for notification submission forwarding. |
| `CentralContactPoints` | `[]` | Site-side: Akka addresses of central nodes, e.g. `akka.tcp://scadabridge@central-a:8081`. |
| `CentralGrpcEndpoints` | `[]` | Site-side: gRPC h2c endpoints of the central nodes' `CentralControlService`, e.g. `http://scadabridge-central-a:8083` (central's `CentralGrpcPort`, default 8083 — direct, not via Traefik, which is HTTP/1 only). **Required on Site nodes** (at least one); empty on Central nodes, which host `CentralControlService` and do not dial it. Replaces the former `CentralContactPoints` Akka-address list. |
| `GrpcKeepAlivePingDelay` | `00:00:15` | HTTP/2 keepalive PING interval on `SiteStreamGrpcClient`. |
| `GrpcKeepAlivePingTimeout` | `00:00:10` | HTTP/2 keepalive PING timeout. |
| `GrpcMaxStreamLifetime` | `04:00:00` | Per-stream session timeout; forces reconnect of zombie streams. |
@@ -259,27 +262,27 @@ Three layers of dead-client detection protect the gRPC stream path:
## Dependencies & Interactions
- [Commons (#16)](./Commons.md) — owns all message contracts used by this component: `DeployInstanceCommand`, `SiteEnvelope`, `HeartbeatMessage`, `SiteHealthReport`, `SiteHealthReportReplica`, `RegisterNotificationOutbox`, `RegisterAuditIngest`, `IngestAuditEventsCommand`, `IngestCachedTelemetryCommand`, and all other request/response records. Commons does not hold an Akka package reference, so `RegisterAuditIngest` (which carries an `IActorRef`) lives in this project.
- [Cluster Infrastructure (#13)](./ClusterInfrastructure.md) — provides `ClusterClientReceptionist` registration and the oldest-`Up` active/standby model that `SiteCommunicationActor`'s `IsActive` stamp depends on, plus the single `"scadabridge"` `ActorSystem` name that makes cross-cluster `ClusterClient` addressing possible at all. `CentralCommunicationActor`'s `DistributedPubSub` fanout keeps both central nodes in sync regardless of which one a site's report landed on.
- [Cluster Infrastructure (#13)](./ClusterInfrastructure.md) — provides ClusterSingleton and the oldest-`Up` active/standby model that `SiteCommunicationActor`'s `IsActive` stamp depends on, plus the single `"scadabridge"` `ActorSystem` name for intra-cluster remoting. (`ClusterClientReceptionist` is no longer used — cross-cluster messaging is gRPC as of Phase 4.) `CentralCommunicationActor`'s `DistributedPubSub` fanout keeps both central nodes in sync regardless of which one a site's report landed on.
- [Configuration Database (#17)](./ConfigurationDatabase.md) — provides `ISiteRepository.GetAllSitesAsync` for address loading; site records carry `NodeAAddress`, `NodeBAddress`, `GrpcNodeAAddress`, `GrpcNodeBAddress`.
- [Deployment Manager (#2)](./DeploymentManager.md) — the primary consumer of command/control patterns 13. `CommunicationService` is injected into the Deployment Manager actor to send deploy notifies, lifecycle commands, and artifact deployments to sites. It also owns the staging half of the notify-and-fetch HTTP path (`PendingDeployment` rows + fetch tokens); the endpoint itself is served by the Management Service.
- [Site Runtime (#3)](./SiteRuntime.md) — `SiteCommunicationActor` forwards inbound commands to the `DeploymentManager` singleton proxy. `SiteStreamManager` (in Site Runtime) implements `ISiteStreamSubscriber` so `SiteStreamGrpcServer` can subscribe relay actors to instance event feeds without referencing Site Runtime directly.
- [Health Monitoring (#11)](./HealthMonitoring.md) — `CentralCommunicationActor` calls `ICentralHealthAggregator.MarkHeartbeat` and `ProcessReport` for every inbound heartbeat and health report. `DistributedPubSub` fanout keeps both central nodes' aggregators in sync.
- [Audit Log (#23)](./AuditLog.md) — `SiteStreamGrpcServer` hosts the `IngestAuditEvents`, `IngestCachedTelemetry`, `PullAuditEvents` and `PullSiteCalls` RPCs. Because there is no central gRPC server, the `Ingest*` pair is unused in the shipped topology: sites push audit telemetry over ClusterClient, and `CentralCommunicationActor` routes `IngestAuditEventsCommand` / `IngestCachedTelemetryCommand` to the `AuditLogIngestActor` proxy. The `Pull*` reconciliation RPCs run the other way, with the central `site-audit-reconciliation` singleton dialling each site.
- [Audit Log (#23)](./AuditLog.md) — `SiteStreamGrpcServer` hosts the `IngestAuditEvents`, `IngestCachedTelemetry`, `PullAuditEvents` and `PullSiteCalls` RPCs. The `Ingest*` pair on the site-hosted service stays unused in the shipped topology; sites push audit telemetry site→central over gRPC to `CentralControlService`, which routes `IngestAuditEventsCommand` / `IngestCachedTelemetryCommand` to the `AuditLogIngestActor` proxy. The `Pull*` reconciliation RPCs run the other way, with the central `site-audit-reconciliation` singleton dialling each site's `SiteStreamService`.
- [Notification Outbox (#21)](./NotificationOutbox.md) — `CentralCommunicationActor` routes `NotificationSubmit` / `NotificationStatusQuery` messages from sites to the `NotificationOutboxActor` proxy. `CommunicationService` Asks the proxy directly for central-UI outbox management calls.
- [Site Call Audit (#22)](./SiteCallAudit.md) — `CommunicationService` Asks the `SiteCallAuditActor` proxy directly for query and relay operations. `SiteCallAuditActor` issues `RetryParkedOperation` / `DiscardParkedOperation` relay commands to sites via `SiteEnvelope`; `SiteCommunicationActor` dispatches them to `_parkedMessageHandler`.
- [Store-and-Forward Engine (#6)](./StoreAndForward.md) — the site S&F Engine drives `NotificationSubmit` forwarding and cached-call telemetry emission through `SiteCommunicationActor`. Parked-message queries and retry/discard relay commands flow back the other way.
- [Management Service (#18)](./ManagementService.md) — `ManagementActor` runs at `/user/management` on central and is reached **in-process** through `ManagementActorHolder`; the CLI connects over HTTP, not `ClusterClient`. (It was `ClusterClientReceptionist`-registered until 2026-07-22, for a CLI that was never built that way.) So this component's `ClusterClient` usage is exclusively the inter-cluster hub-and-spoke connections. Management Service also hosts `DeploymentConfigEndpoints` — the `GET /api/internal/deployments/{id}/config` route that terminates the third (HTTP) transport, mapped in the central-role block alongside `/api/audit/*` and `/management`.
- [Management Service (#18)](./ManagementService.md) — `ManagementActor` runs at `/user/management` on central and is reached **in-process** through `ManagementActorHolder`; the CLI connects over HTTP, not any cluster transport. (It was `ClusterClientReceptionist`-registered until 2026-07-22, for a CLI that was never built that way.) After Phase 4 this component uses no `ClusterClient` at all — its cross-cluster connections are the gRPC `CentralControlService` / `SiteCommandService` transports. Management Service also hosts `DeploymentConfigEndpoints` — the `GET /api/internal/deployments/{id}/config` route that terminates the HTTP deploy-config transport, mapped in the central-role block alongside `/api/audit/*` and `/management`.
- Design spec: [Component-Communication.md](../requirements/Component-Communication.md).
## Troubleshooting
### A site's commands fail immediately
Check that `NodeAAddress` and `NodeBAddress` are populated in the site configuration — if both are empty, `CentralCommunicationActor` logs a warning and skips that site on every refresh, so no `ClusterClient` is created and all commands timeout. `CommunicationService.RefreshSiteAddresses()` triggers an on-demand refresh after an address is added.
Check that `GrpcNodeAAddress` and `GrpcNodeBAddress` are populated in the site configuration — if both are empty, `CentralCommunicationActor` logs a warning and skips that site on every refresh, so no `SiteCommandService` channel is built and all commands timeout. `CommunicationService.RefreshSiteAddresses()` triggers an on-demand refresh after an address is added.
### Commands are timing out but the site is reachable
A single malformed address string for one site can silently prevent `ClusterClient` creation for that site while other sites are unaffected. Check the logs for a `Warning` line from `HandleSiteAddressCacheLoaded` naming the offending site. The actor parse-guard catches the `ActorPath.Parse` exception per-site so the rest of the refresh proceeds.
A single malformed gRPC endpoint string for one site can silently prevent channel creation for that site while other sites are unaffected. Check the logs for a `Warning` line from `HandleSiteAddressCacheLoaded` naming the offending site. The per-site parse-guard skips the bad entry so the rest of the refresh proceeds.
A `Warning` at the `Status.Failure` handler in `CentralCommunicationActor` means `LoadSiteAddressesFromDb` itself threw (typically a SQL connection error); the cache is left stale until the next successful refresh.
@@ -291,7 +294,7 @@ After a site node failover, the `DebugStreamBridgeActor` attempts to reconnect t
### Deployments fail immediately with a config-fetch error
The site received the `RefreshDeploymentCommand` over ClusterClient but could not complete the HTTP leg. Check `CentralFetchBaseUrl` first — it must be reachable *from the site*, so a value that only resolves inside the central network fails every deploy. A `404` from the fetch means the staged row was unknown, superseded, or past `PendingDeploymentTtl`; a `401` means the row is live but the token did not match. Because the endpoint hides existence, a `404` cannot distinguish "wrong id" from "expired".
The site received the `RefreshDeploymentCommand` over gRPC command/control (`SiteCommandService`) but could not complete the HTTP leg. Check `CentralFetchBaseUrl` first — it must be reachable *from the site*, so a value that only resolves inside the central network fails every deploy. A `404` from the fetch means the staged row was unknown, superseded, or past `PendingDeploymentTtl`; a `401` means the row is live but the token did not match. Because the endpoint hides existence, a `404` cannot distinguish "wrong id" from "expired".
### Heartbeats arrive but health reports do not
+15 -7
View File
@@ -178,14 +178,22 @@ database from its peer rather than letting it rejoin.
### Central-Site Communication
Three transports cross the boundary, not one:
Three transports cross the boundary, not one — **all now gRPC or HTTP; Akka ClusterClient was removed
in Phase 4 of the ClusterClient→gRPC migration (2026-07-23), and Akka remoting no longer crosses the
boundary at all:**
- **Akka ClusterClient** — command/control. Sites list every central node in
`ScadaBridge:Communication:CentralContactPoints`; contact rotation reaches whichever node
answers, so no "active central" needs to be identified. (There is no `Communication:CentralSeedNode`
setting — earlier revisions of this guide named one that never existed in the code.)
- **gRPC** — real-time data and audit pull. Note the direction is inverted from the data flow:
each **site node hosts the gRPC server** on `GrpcPort` (default 8083, h2c) and central dials in.
- **gRPC command/control** — both directions, on sticky-failover channel pairs, dialled directly (no
receptionist, no "active central" to identify — each side dials both of the peer's node endpoints):
- *Site → central* to the central-hosted **`CentralControlService`** (`GrpcCentralTransport`): the
site lists the central nodes' gRPC endpoints in `ScadaBridge:Communication:CentralGrpcEndpoints`
(e.g. `http://scadabridge-central-a:8083`, the central's `CentralGrpcPort`, default 8083 — direct
h2c, **not** via Traefik, which is HTTP/1 only). A Site node must list at least one; central nodes
leave it empty.
- *Central → site* to the site-hosted **`SiteCommandService`** (`GrpcSiteTransport`): central dials
the site's `GrpcNodeAAddress` / `GrpcNodeBAddress` (from the Site entity), NodeA→NodeB failover.
- **gRPC streaming + audit pull** — real-time data and audit/telemetry pull on the site-hosted
**`SiteStreamService`**. Note the direction is inverted from the data flow: each **site node hosts
the server** on `GrpcPort` (default 8083, h2c) and central dials in.
- **Plain HTTP** — the deploy config itself, fetched by the site with a per-deployment token.
#### gRPC control-plane preshared key (required)
@@ -31,6 +31,25 @@ Any deployment replicating wide rows must size that key deliberately; see the Ph
Note the failure mode differs from the one documented below: an oversized gRPC message is
**rejected**, not silently dropped.
## Amendment (2026-07-23) — Akka frame-size class RETIRED for site↔central command/control
Phase 4 of the ClusterClient→gRPC migration deleted the Akka `ClusterClient` site↔central transport
entirely. **All site↔central command/control now rides gRPC** — central→site over `SiteCommandService`
(`GrpcSiteTransport`) and site→central over `CentralControlService` (`GrpcCentralTransport`), each with
the gRPC default 4 MB message cap and **no per-message Akka frame limit**. Consequently:
- The 128 KB Akka `maximum-frame-size` constraint **no longer applies to any site↔central command
message**, including `DeployArtifactsCommand`, which still carries its payload inline but now travels
over gRPC.
- The **silent frame-drop failure mode** described below — the transport dropping one oversized message
while heartbeats keep flowing and the deploy hangs to its timeout — **cannot occur on that path any
more.** An over-cap gRPC message is *rejected* with an error, not silently discarded.
- The notify-and-fetch deploy path (the 2026-06-26 resolution) still stands and remains the deploy
mechanism; it is simply no longer the *only* thing keeping large payloads off a frame-limited hop.
- Akka remoting is now intra-cluster only, so its frame size governs only pair-internal traffic.
See `docs/plans/2026-07-22-clusterclient-to-grpc-plan.md`.
The diagnosis below is retained as the historical record of how the bug was found and reasoned about.
## Summary
@@ -327,9 +327,9 @@ Critical path ≈ 1B: **~46 weeks total**, matching the design estimate.
- [x] P3 Central `SiteTransport=Grpc` all sites; full UI command matrix per site; site-kill mid-command clean; no PSK noise; zero ClusterClient log activity on flipped paths — **PASS 2026-07-23** (live gate: `ExecuteQuery`/`ExecuteParked` all 3 sites + `ExecuteLifecycle` disable/enable on site-a, all 200 over `SiteCommandService`; active-site-node hard-kill mid-command → clean `TIMEOUT` at the 30s deadline, next call failed over to site-a-b automatically; 0 PermissionDenied / 0 ClusterClient-to-site on both central. `ExecuteOpcUa`/`ExecuteRoute`/`TriggerFailover` deferred — no OPC-bound instance / no CLI verb, unit-proven)
**Phase 4 — deletion**
- [ ] Defaults flip to `Grpc` + soak; then delete Akka transports, ClusterClient creation, `DefaultSiteClientFactory`, receptionist registrations, `CentralContactPoints`, then the flags
- [ ] Grep-gates pass (`clusterclient|receptionist` → historical docs only; `CentralContactPoints` → empty)
- [ ] Docs updated: `grpc_streams.md`, `Component-Host.md`, `Component-StoreAndForward.md:137`, known-issues cross-ref
- [x] gRPC made the only transport (flags DELETED, not flipped — end state is identical) + deleted `AkkaCentralTransport`/`AkkaSiteTransport`, ClusterClient creation (`AkkaHostedService`), `DefaultSiteClientFactory`+`ISiteClientFactory`, both receptionist registrations (`:436`/`:1001`), `CentralContactPoints` + the `CentralTransport`/`SiteTransport` flags + `CentralTransportMode`/`SiteTransportKind` enums. `NoOpCentralTransport` added as the fail-loud null-default; `CentralGrpcEndpoints` now unconditional (StartupValidator requires ≥1 on Site). Kept `Akka.Cluster.Tools` (ClusterSingleton). Rig configs (docker ×6, docker-env2 ×2, Host default, wonder-app-vd03) moved `CentralContactPoints`→`CentralGrpcEndpoints`. **Full solution build 0/0; Communication.Tests 640, Host.Tests + StartupValidator + SiteActorPath green, audit-push integration green** (2026-07-23)
- [x] Grep-gates pass (`CentralContactPoints` → only "replaces the former" doc refs + plan trackers; deleted symbols → 0 live refs; remaining `clusterclient` in src = the misleadingly-named `ClusterClientSiteAuditClient` [transport-agnostic, works unchanged] + stale inline doc-comments, noted as follow-up)
- [x] Docs updated: `Component-Communication.md`, `components/Communication.md`, `Component-Host.md`, `Component-StoreAndForward.md`, `topology-guide.md`, `grpc_streams.md`, known-issues frame-size amendment, **CLAUDE.md** transport decisions
**Phase 5 — live gate** (record in `2026-07-22-clusterclient-to-grpc-live-gate.md`)
- [ ] 1 PSK negatives · [ ] 2 site→central matrix · [ ] 3 central→site matrix · [ ] 4 failover/failback both directions · [ ] 5 mid-drain kill · [ ] 6 >128 KB frame-class proof · [ ] 7 no cross-boundary Akka association · [ ] 8 full-rig restart clean
@@ -218,22 +218,24 @@
"id": "P4.1",
"phase": "4",
"subject": "Flip both flag defaults to Grpc + soak; delete Akka transports, ClusterClient creation, DefaultSiteClientFactory, receptionist registrations, CentralContactPoints, then the flags",
"status": "pending",
"status": "completed",
"activeForm": "Deleting the ClusterClient transport",
"blockedBy": [
"P2",
"P3"
]
],
"notes": "Branch feat/grpc-phase4-deletion (2026-07-23). Flags DELETED rather than flipped — end state is gRPC-only, identical. Deleted: AkkaCentralTransport, AkkaSiteTransport, ISiteClientFactory+DefaultSiteClientFactory, CentralCommunicationActor legacy ctor + SelectTransport, ClusterClient creation + both ClusterClientReceptionist.RegisterService in AkkaHostedService, CommunicationOptions.CentralContactPoints + CentralTransport/SiteTransport flags + CentralTransportMode/SiteTransportKind enums, RegisterCentralClient message + receive block, AkkaCentralTransportTests + DefaultSiteClientFactoryTests. Added NoOpCentralTransport (fail-loud null-default so command-dispatch TestKit suites keep constructing without a wired transport; production always injects GrpcCentralTransport). CommunicationOptionsValidator: CentralGrpcEndpoints now unconditional (no-blank-entries, role-agnostic); StartupValidator: Site-only requires ≥1 CentralGrpcEndpoint (fail-fast, mirrors GrpcPsk). Host builds GrpcSiteTransport (central) + GrpcCentralTransport (site) unconditionally. Kept Akka.Cluster.Tools (ClusterSingleton). Test rework: deleted the ClusterClient.Send per-site-routing tests (covered by CentralCommunicationActorTransportTests + GrpcSiteTransport suites), swapped ISiteClientFactory→substitute ISiteCommandTransport across 5 files, converted SiteAuditPushFlow's ClusterClientRelay→BridgeCentralTransport, converted Heartbeat_StampsIsActive to a substitute ICentralTransport, repurposed HealthReportAck no-client test to the NoOp fail-loud path, removed SiteActors_CentralClusterClient_Exists. Rig: docker ×6 + docker-env2 ×2 + Host appsettings.Site.json + deploy/wonder-app-vd03 moved CentralContactPoints→CentralGrpcEndpoints. Full solution build 0/0; Communication.Tests 640, StartupValidator 59, SiteActorPath 5, audit-push integration 1 all green. DID NOT fold in the dead IntegrationCallRequest deletion (#32) — SiteEnvelope routing is transport-agnostic so it still compiles; user-owned behavioral decision, left OUT of this PR."
},
{
"id": "P4.2",
"phase": "4",
"subject": "Grep-gates pass + docs updated (grpc_streams.md, Component-Host.md, Component-StoreAndForward.md, known-issues cross-ref)",
"status": "pending",
"status": "completed",
"activeForm": "Running the deletion grep-gates and doc sweep",
"blockedBy": [
"P4.1"
]
],
"notes": "Grep-gates: CentralContactPoints → only intentional 'replaces the former' doc refs + plan trackers + stale bin/ artifacts (regenerate on build); deleted symbols (DefaultSiteClientFactory/ISiteClientFactory/RegisterCentralClient/AkkaCentralTransport/AkkaSiteTransport/CentralTransportMode/SiteTransportKind/the flags) → 0 live refs. Residual 'clusterclient' in src = ClusterClientSiteAuditClient (misleadingly named but transport-agnostic — it Asks SiteCommunicationActor, holds no ClusterClient; works unchanged) + ~25 stale inline XML-doc comments across SiteRuntime/Commons/SiteCallAudit — NOT scrubbed (out of plan scope, no behavior impact); FOLLOW-UP noted. Docs updated (subagent + me): Component-Communication.md, components/Communication.md, Component-Host.md, Component-StoreAndForward.md, deployment/topology-guide.md, plans/grpc_streams.md (SUPERSEDED note + LoadSiteAddressesFromDb correction), known-issues/2026-06-26-deploy-config-exceeds-akka-frame-size.md (2026-07-23 frame-size-retired amendment), and CLAUDE.md (2 transport-decision passages). Phase 5 (live gate) is next and requires a rig redeploy."
},
{
"id": "P5",
+15 -4
View File
@@ -1,10 +1,21 @@
# gRPC Streaming Channel: Site → Central Real-Time Data
> **Status update (2026-07-23) — the "ClusterClient keeps command/control" framing is SUPERSEDED.**
> This plan introduced gRPC for *streaming only*, leaving command/control on Akka ClusterClient. That
> split no longer holds: **Phase 4 of the ClusterClient→gRPC migration moved command/control to gRPC
> too and deleted the ClusterClient site↔central transport entirely.** Command/control now rides gRPC
> in both directions — site→central to the central-hosted `CentralControlService` (`GrpcCentralTransport`)
> and central→site to the site-hosted `SiteCommandService` (`GrpcSiteTransport`) — alongside the
> streaming `SiteStreamService` this plan describes. Wherever the text below says "ClusterClient handles
> command/control", read it as "gRPC command/control (`CentralControlService` / `SiteCommandService`)".
> See `docs/requirements/Component-Communication.md` and
> `docs/plans/2026-07-22-clusterclient-to-grpc-plan.md`.
## Context
Debug streaming events currently flow through Akka.NET ClusterClient (`InstanceActor → SiteCommunicationActor → ClusterClient.Send → CentralCommunicationActor → bridge actor`). ClusterClient wasn't built for high-throughput value streaming — it's a cluster coordination tool with gossip-based routing. As we scale beyond debug view to health streaming, alarm feeds, or future live dashboards, pushing all real-time data through ClusterClient will become a bottleneck.
**Goal**: Add a dedicated gRPC server-streaming channel on each site node. Central subscribes to sites over gRPC for real-time data. ClusterClient continues to handle command/control (subscribe, unsubscribe, deploy, lifecycle) but all streaming values flow through the gRPC channel.
**Goal**: Add a dedicated gRPC server-streaming channel on each site node. Central subscribes to sites over gRPC for real-time data. Command/control (subscribe, unsubscribe, deploy, lifecycle) and streaming values flow over gRPC. *(As originally written, command/control stayed on ClusterClient; Phase 4 later moved it to gRPC — see the status note above.)*
**Scope**: General-purpose site→central streaming transport. Debug view is the first consumer, but the proto and server are designed so future features (health streaming, alarm feeds, live dashboards) can subscribe with different event types and filters.
@@ -57,7 +68,7 @@ flowchart TD
class PB dec
```
**Key separation**: ClusterClient handles subscribe/unsubscribe/snapshot (request-response). gRPC handles the ongoing value stream (server-streaming).
**Key separation**: command/control handles subscribe/unsubscribe/snapshot (request-response); the ongoing value stream is server-streaming. Both ride gRPC after Phase 4 (subscribe/unsubscribe/snapshot over `SiteCommandService`, the value stream over `SiteStreamService`); as originally drawn, the request-response leg was ClusterClient.
## Port & Address Configuration
@@ -164,7 +175,7 @@ Add corresponding columns to the sites list table. Wire `_formGrpcNodeAAddress`
### SiteStreamGrpcClientFactory
Reads `GrpcNodeAAddress` / `GrpcNodeBAddress` from the `Site` entity (loaded by `CentralCommunicationActor.LoadSiteAddressesFromDb()`) when creating per-site gRPC channels. Falls back to NodeB if NodeA connection fails (same pattern as ClusterClient dual-contact-point failover).
Reads `GrpcNodeAAddress` / `GrpcNodeBAddress` from the `Site` entity when creating per-site gRPC channels. As of Phase 4, `CentralCommunicationActor.LoadSiteAddressesFromDb()` no longer builds Akka ClusterClient contacts at all — it builds a **per-site gRPC-endpoint cache** from those same `GrpcNodeAAddress` / `GrpcNodeBAddress` columns, feeding a `SitePairChannelProvider` that both this streaming factory and the command/control `GrpcSiteTransport` (`SiteCommandService`) dial. Falls back to NodeB if NodeA connection fails.
### Docker Compose Port Allocation
@@ -364,7 +375,7 @@ public async Task<StreamSubscription> SubscribeAsync(
### Client Factory
`SiteStreamGrpcClientFactory` caches per-site `GrpcChannel` instances (same pattern as `CentralCommunicationActor._siteClients` caching per-site ClusterClient instances).
`SiteStreamGrpcClientFactory` caches per-site `GrpcChannel` instances. (As originally written this mirrored `CentralCommunicationActor._siteClients` caching per-site ClusterClient instances; after Phase 4 that dictionary caches per-site gRPC endpoints/channels, not ClusterClients.)
## Failover & Reconnection
+53 -52
View File
@@ -2,7 +2,7 @@
## Purpose
The Communication component manages all messaging between the central cluster and site clusters. It provides the transport layer for deployments, instance lifecycle commands, integration routing, debug streaming, health reporting, notification submission, and remote queries (parked messages, event logs). Two transports are used: **Akka.NET ClusterClient** for command/control messaging and **gRPC server-streaming** for real-time data (attribute values, alarm states).
The Communication component manages all messaging between the central cluster and site clusters. It provides the transport layer for deployments, instance lifecycle commands, integration routing, debug streaming, health reporting, notification submission, and remote queries (parked messages, event logs). **gRPC is the only inter-cluster transport** (ClusterClient/ClusterClientReceptionist for site↔central messaging was removed in Phase 4 of the ClusterClient→gRPC migration, 2026-07-23). It carries three concerns: **command/control** in both directions — site→central to the central-hosted `CentralControlService` (heartbeats, health reports, notification submit/status, audit/telemetry ingest, reconcile) and central→site to the site-hosted `SiteCommandService` (deployments, lifecycle, OPC UA, remote queries, parked, route, failover); **gRPC server-streaming** for real-time data (attribute values, alarm states) via `SiteStreamService`; and **plain token-gated HTTP** for the deployment-config fetch (notify-and-fetch). Akka remoting remains, but only *intra-cluster* (pair-internal), never across the site↔central boundary.
## Location
@@ -10,8 +10,8 @@ Both central and site clusters. Each side has communication actors that handle m
## Responsibilities
- Resolve site addresses (Akka remoting and gRPC) from the configuration database and maintain a cached address map.
- Establish and maintain cross-cluster connections using Akka.NET ClusterClient/ClusterClientReceptionist for command/control.
- Resolve site addresses (Akka remoting — intra-cluster only — and gRPC) from the configuration database and maintain a cached address map, including the per-site gRPC command endpoints used by central→site command/control.
- Establish and maintain cross-cluster command/control connections over gRPC: site→central to `CentralControlService` (`GrpcCentralTransport`) and central→site to `SiteCommandService` (`GrpcSiteTransport`).
- Establish and maintain per-site gRPC streaming connections for real-time data delivery (site→central).
- Route messages between central and site clusters in a hub-and-spoke topology.
- Broker requests from external systems (via central) to sites and return responses.
@@ -54,15 +54,15 @@ Both central and site clusters. Each side has communication actors that handle m
- Site applies and acknowledges.
### 6. Debug Streaming (Site → Central)
- **Pattern**: Subscribe/push with initial snapshot. Two transports: **ClusterClient** for the subscribe/unsubscribe handshake and initial snapshot, **gRPC server-streaming** for ongoing real-time events.
- A **DebugStreamBridgeActor** (one per active debug session) is created on the central cluster by the **DebugStreamService**. The bridge actor first opens a **gRPC server-streaming subscription** to the site via `SiteStreamGrpcClient`, then sends a `SubscribeDebugViewRequest` to the site via `CentralCommunicationActor` (ClusterClient). The site's `InstanceActor` replies with an initial snapshot via the ClusterClient reply path.
- **Pattern**: Subscribe/push with initial snapshot. Both legs ride gRPC: the subscribe/unsubscribe handshake and initial snapshot go over **gRPC command/control** (`SiteCommandService`), and ongoing real-time events over **gRPC server-streaming** (`SiteStreamService`).
- A **DebugStreamBridgeActor** (one per active debug session) is created on the central cluster by the **DebugStreamService**. The bridge actor first opens a **gRPC server-streaming subscription** to the site via `SiteStreamGrpcClient`, then sends a `SubscribeDebugViewRequest` to the site as a central→site command over `SiteCommandService` (`GrpcSiteTransport`). The site's `InstanceActor` replies with an initial snapshot on that gRPC request/response.
- **gRPC stream (real-time events)**: The site's **SiteStreamGrpcServer** receives the gRPC `SubscribeInstance` call and creates a **StreamRelayActor** that subscribes to **SiteStreamManager** for the requested instance. Events (`AttributeValueChanged`, `AlarmStateChanged`) flow from `SiteStreamManager``StreamRelayActor``Channel<SiteStreamEvent>` (bounded, 1000, DropOldest) → gRPC response stream → `SiteStreamGrpcClient` on central → `DebugStreamBridgeActor`.
- The `DebugStreamEvent` message type no longer exists — events are not routed through ClusterClient. `SiteCommunicationActor` and `CentralCommunicationActor` have no role in streaming event delivery.
- The `DebugStreamEvent` message type no longer exists — events are not routed through a command/control channel. `SiteCommunicationActor` and `CentralCommunicationActor` have no role in streaming event delivery.
- The bridge actor forwards received events to the consumer via callbacks (Blazor component or SignalR hub).
- **Snapshot-to-stream handoff**: The gRPC stream is opened **before** the snapshot request to avoid missing events. The consumer applies the snapshot as baseline, then replays buffered gRPC events with timestamps newer than the snapshot (timestamp-based dedup).
- Attribute value stream messages: `[InstanceUniqueName].[AttributePath].[AttributeName]`, value, quality, timestamp.
- Alarm state stream messages: `[InstanceUniqueName].[AlarmName]`, state (active/normal), priority, timestamp.
- Central sends an unsubscribe request via ClusterClient when the debug session ends. The gRPC stream is cancelled. The site's `StreamRelayActor` is stopped and the SiteStreamManager subscription is removed.
- Central sends an unsubscribe request over `SiteCommandService` (gRPC command/control) when the debug session ends. The gRPC stream is cancelled. The site's `StreamRelayActor` is stopped and the SiteStreamManager subscription is removed.
- The stream is session-based and temporary.
- **Accepted limitation (central-side session locality):** debug sessions are process-local state on the central node hosting the `DebugStreamBridgeActor`. A central **restart or failover drops all active sessions** with no `OnStreamTerminated` signal to the operator — the engineer simply re-establishes the debug session from the UI. Debug streaming is an interactive, transient diagnostic aid, so this is an accepted trade-off rather than a durability gap (arch review 02, U7).
- **Backpressure is lossy-by-design and now observable:** the per-session `Channel<SiteStreamEvent>` (bounded 1000, `DropOldest`) silently evicts the oldest event when a slow consumer falls behind. The eviction is counted via the channel's `itemDropped` callback and logged at Warning (first eviction, then every 500th) so real event loss on a debug stream is visible instead of silent.
@@ -89,7 +89,7 @@ Delivered 2026-07-10 (`docs/plans/2026-07-10-aggregated-live-alarm-stream-plan.m
#### Central-Side Debug Stream Components
- **DebugStreamService**: Singleton service that manages debug stream sessions. Resolves instance ID to unique name and site, creates and tears down `DebugStreamBridgeActor` instances, and provides a clean API for both Blazor components and the SignalR hub. Injects `SiteStreamGrpcClientFactory` for gRPC stream creation.
- **DebugStreamBridgeActor**: One per active debug session. Opens a gRPC streaming subscription via `SiteStreamGrpcClient` and receives real-time events via callback. Also receives the initial `DebugViewSnapshot` via ClusterClient. Forwards all events to the consumer via callbacks. Handles gRPC stream errors with reconnection logic: tries the other site node endpoint, retries with backoff (max 3 retries), terminates the session if all retries fail.
- **DebugStreamBridgeActor**: One per active debug session. Opens a gRPC streaming subscription via `SiteStreamGrpcClient` and receives real-time events via callback. Also receives the initial `DebugViewSnapshot` over gRPC command/control (`SiteCommandService`). Forwards all events to the consumer via callbacks. Handles gRPC stream errors with reconnection logic: tries the other site node endpoint, retries with backoff (max 3 retries), terminates the session if all retries fail.
- **SiteStreamGrpcClient**: Per-site gRPC client that manages `GrpcChannel` instances and streaming subscriptions. Reads from the gRPC response stream in a background task, converts protobuf messages to domain events, and invokes the `onEvent` callback.
- **SiteStreamGrpcClientFactory**: Caches per-site `SiteStreamGrpcClient` instances. Reads `GrpcNodeAAddress` / `GrpcNodeBAddress` from the `Site` entity (loaded by `CentralCommunicationActor`). Falls back to NodeB if NodeA connection fails. Disposes clients on site removal or address change.
- **DebugStreamHub**: SignalR hub at `/hubs/debug-stream` for external consumers (e.g., CLI). Authenticates via Basic Auth + LDAP and requires the **Deployment** role. Server-to-client methods: `OnSnapshot`, `OnAttributeChanged`, `OnAlarmChanged`, `OnStreamTerminated`.
@@ -98,10 +98,10 @@ Delivered 2026-07-10 (`docs/plans/2026-07-10-aggregated-live-alarm-stream-plan.m
The streaming protocol is defined in `sitestream.proto` (`src/ZB.MOM.WW.ScadaBridge.Communication/Protos/sitestream.proto`):
- **Service**: `SiteStreamService` — hosted on each site node by `SiteStreamGrpcServer` — exposes six RPCs. Two are real-time **server-streaming** subscriptions (`SubscribeInstance`, `SubscribeSite`); the other four are **unary request/response** calls added by the Audit Log (#23) and Site Call Audit (#22) components. A unary call is request/response and is distinct from the command/control ClusterClient channel — gRPC on this service is no longer real-time-stream-only:
- **Service**: `SiteStreamService` — hosted on each site node by `SiteStreamGrpcServer` — exposes six RPCs. Two are real-time **server-streaming** subscriptions (`SubscribeInstance`, `SubscribeSite`); the other four are **unary request/response** calls added by the Audit Log (#23) and Site Call Audit (#22) components. These are distinct from the command/control gRPC services (`CentralControlService` on central, `SiteCommandService` on the site) — `SiteStreamService` is the streaming + audit-pull surface, not the command/control channel:
- `SubscribeInstance(InstanceStreamRequest) returns (stream SiteStreamEvent)` — the per-instance real-time debug stream (§6).
- `SubscribeSite(SiteStreamRequest) returns (stream SiteStreamEvent)` — the **site-wide, alarm-only** aggregated stream (§6.1) added for the operator Alarm Summary live cache. Additive (new RPC + new `SiteStreamRequest { correlation_id }` message; no field renumbering). The server handler mirrors `SubscribeInstance` — same bounded `Channel(1000, DropOldest)`, `GrpcMaxConcurrentStreams`/`GrpcMaxStreamLifetime` limits, `SiteConnectionOpened/Closed` telemetry, and `StreamRelayActor` mapping — but subscribes via `SiteStreamManager.SubscribeSiteAlarms` (all instances, `AlarmStateChanged` only; attribute events dropped, no per-instance filter).
- `IngestAuditEvents(AuditEventBatch) returns (IngestAck)` — central-side **ingest** receiving surface for Audit Log (#23) telemetry; routes the batch to the central `AuditLogIngestActor` proxy and returns the accepted `EventId`s. (The production *push* path is still ClusterClient via `ClusterClientSiteAuditClient`; this RPC is the gRPC-receiving counterpart.)
- `IngestAuditEvents(AuditEventBatch) returns (IngestAck)` a legacy central-side ingest surface on the *site*-hosted service; it is **dead in the shipped topology** because no site ever dials a site for ingest. The production audit-telemetry *push* path is site→central gRPC to `CentralControlService`, which routes the batch to the central `AuditLogIngestActor` proxy and returns the accepted `EventId`s.
- `IngestCachedTelemetry(CachedTelemetryBatch) returns (IngestAck)` — ingest receiving surface for the combined cached-call telemetry packet (audit row + `SiteCalls` operational upsert written in one transaction).
- `PullAuditEvents(PullAuditEventsRequest) returns (PullAuditEventsResponse)` — central→site **reconciliation pull** for the Audit Log self-heal feed; the site serves `Pending`/`Forwarded` rows from its `ISiteAuditQueue`.
- `PullSiteCalls(PullSiteCallsRequest) returns (PullSiteCallsResponse)` — central→site reconciliation pull for the Site Call Audit (#22) self-heal feed; the site serves operation-tracking rows changed since a cursor from its `IOperationTrackingStore`. A separate RPC from `PullAuditEvents` because the tracking store is the operational source of truth, distinct from the site audit queue.
@@ -205,15 +205,15 @@ Keepalive settings are configurable via `CommunicationOptions`:
- The site **Store-and-Forward Engine** sends a `NotificationSubmit` message to central carrying the notification — `NotificationId`, target list name, subject, body, and source provenance.
- Central ingests the submission with an insert-if-not-exists on `NotificationId` and acknowledges **after the row is persisted** to the `Notifications` table in the central configuration database. The site S&F engine clears the buffered message only on that ack.
- The `NotificationId` GUID — generated at the site — is the **idempotency key**. The handoff is at-least-once: a re-sent submission after a lost ack is harmless because central's insert-if-not-exists treats the duplicate as a no-op.
- **Transport**: ClusterClient (site→central command/control), consistent with how other site→central messages are sent.
- **Transport**: gRPC to `CentralControlService` (site→central command/control), consistent with how other site→central messages are sent.
### 10. Cached Call Telemetry (Site → Central)
- **Pattern**: Fire-and-forget telemetry with a periodic reconciliation pull.
- The site **Store-and-Forward Engine** emits a `CachedCallTelemetry` message to central on **every** cached-call lifecycle transition (`Pending → Retrying → Delivered / Parked / Failed / Discarded`). The first telemetry event for an operation carries its initial status — `Pending` when a transient failure has buffered the call, or directly `Delivered`/`Failed` for a cached call that never buffers. The message carries the `TrackedOperationId`, source site, `Kind` (the `TrackedOperationKind` enum), target summary, status, retry count, last error, key timestamps, and source provenance.
- Emission is **best-effort and at-least-once**, **idempotent on `TrackedOperationId`** — central's Site Call Audit component ingests with insert-if-not-exists then upsert-on-newer-status, so a re-sent or out-of-order event is harmless.
- **Reconciliation pull**: because telemetry is best-effort, the central **Site Call Audit** component periodically — and on site reconnect — pulls the changed rows back from each site over the **`PullSiteCalls` unary gRPC RPC** on `SiteStreamService` (not a ClusterClient round-trip). Central sends a `PullSiteCallsRequest` (`since_utc` cursor + `batch_size`); the site reads its `IOperationTrackingStore` and replies with a `PullSiteCallsResponse` carrying the matching operation-tracking rows (as `SiteCallOperationalDto`s) plus a `more_available` flag that signals a saturated batch so central advances the cursor and pulls again. Any telemetry missed during a disconnect self-heals through this pull. The Audit Log (#23) reconciliation feed uses the sibling `PullAuditEvents` RPC the same way.
- **Reconciliation pull**: because telemetry is best-effort, the central **Site Call Audit** component periodically — and on site reconnect — pulls the changed rows back from each site over the **`PullSiteCalls` unary gRPC RPC** on `SiteStreamService` (a distinct surface from the `CentralControlService`/`SiteCommandService` command/control path). Central sends a `PullSiteCallsRequest` (`since_utc` cursor + `batch_size`); the site reads its `IOperationTrackingStore` and replies with a `PullSiteCallsResponse` carrying the matching operation-tracking rows (as `SiteCallOperationalDto`s) plus a `more_available` flag that signals a saturated batch so central advances the cursor and pulls again. Any telemetry missed during a disconnect self-heals through this pull. The Audit Log (#23) reconciliation feed uses the sibling `PullAuditEvents` RPC the same way.
- Central audit is an **eventually-consistent mirror** — the site's operation tracking table remains the source of truth for cached-call status (`Tracking.Status(id)` is always answered site-locally).
- **Transport**: the *push* telemetry emission rides **ClusterClient** (site→central command/control), consistent with how other site→central messages are sent; the *reconciliation pull* rides the **gRPC** unary `PullSiteCalls` RPC (central→site request/response). The two paths are complementary — push is the fast, best-effort feed; pull is the slower self-heal backfill.
- **Transport**: the *push* telemetry emission rides **gRPC to `CentralControlService`** (site→central command/control), consistent with how other site→central messages are sent; the *reconciliation pull* rides the **gRPC** unary `PullSiteCalls` RPC on `SiteStreamService` (central→site request/response). The two paths are complementary — push is the fast, best-effort feed; pull is the slower self-heal backfill.
## Topology
@@ -221,37 +221,38 @@ Keepalive settings are configurable via `CommunicationOptions`:
%%{init: {'theme':'base', 'themeVariables': {'textColor':'#111111','lineColor':'#555555','edgeLabelBackground':'#ffffff','fontSize':'15px'}}}%%
flowchart LR
subgraph Central["Central Cluster"]
CCA["ClusterClient<br/>(command/control)"]
CCB["ClusterClient<br/>(command/control)"]
CCN["ClusterClient<br/>(command/control)"]
CCS["CentralControlService<br/>(gRPC server — site→central cmd/ctrl)"]
SCA["GrpcSiteTransport → SiteCommandService<br/>(gRPC client, per site)"]
SCB["GrpcSiteTransport → SiteCommandService"]
SCN["GrpcSiteTransport → SiteCommandService"]
GRPCC["SiteStreamGrpcClient<br/>(real-time data)"]
end
subgraph SiteA["Site A Cluster"]
SACOMM["SiteCommunicationActor<br/>(via Receptionist)"]
SACMD["SiteCommandService<br/>(gRPC server — central→site cmd/ctrl)"]
SAGRPC["SiteStreamGrpcServer<br/>(Kestrel HTTP/2, port 8083)"]
SACC["ClusterClient to Central<br/>(CentralCommunicationActor)"]
SACC["GrpcCentralTransport → CentralControlService<br/>(gRPC client)"]
end
subgraph SiteB["Site B Cluster"]
SBCOMM["SiteCommunicationActor<br/>(via Receptionist)"]
SBCMD["SiteCommandService"]
SBGRPC["SiteStreamGrpcServer"]
end
subgraph SiteN["Site N Cluster"]
SNCOMM["SiteCommunicationActor<br/>(via Receptionist)"]
SNCMD["SiteCommandService"]
SNGRPC["SiteStreamGrpcServer"]
end
CCA -->|command/control| SACOMM
CCB -->|command/control| SBCOMM
CCN -->|command/control| SNCOMM
SCA -->|"gRPC command/control"| SACMD
SCB -->|"gRPC command/control"| SBCMD
SCN -->|"gRPC command/control"| SNCMD
SAGRPC -->|"gRPC stream (real-time data)"| GRPCC
SBGRPC -->|gRPC stream| GRPCC
SNGRPC -->|gRPC stream| GRPCC
SACC -.->|command/control| Central
SACC -.->|"gRPC command/control"| CCS
NOTE["Sites do NOT communicate with each other.<br/>All inter-cluster communication flows through Central."]
@@ -260,7 +261,7 @@ flowchart LR
classDef dec fill:#fff2cc,stroke:#d6b656,color:#111111;
classDef alt fill:#e1d5e7,stroke:#9673a6,color:#111111;
classDef muted fill:#f5f5f5,stroke:#999999,color:#666666;
class CCA,CCB,CCN,SACOMM,SACC,SBCOMM,SNCOMM dec
class CCS,SCA,SCB,SCN,SACMD,SACC,SBCMD,SNCMD dec
class GRPCC,SAGRPC,SBGRPC,SNGRPC start
class NOTE muted
class Central proc
@@ -269,24 +270,24 @@ flowchart LR
- Sites do **not** communicate with each other.
- All inter-cluster communication flows through central.
- Both **CentralCommunicationActor** and **SiteCommunicationActor** are registered with their cluster's **ClusterClientReceptionist** for cross-cluster discovery.
- Command/control uses gRPC in both directions: central dials each site's **SiteCommandService** (per-site NodeA→NodeB failover channel pair), and each site dials the central **CentralControlService** (sticky central-a→central-b failover channel pair). There is no ClusterClientReceptionist and no cross-cluster actor discovery — the endpoints are dialled directly.
## Site Address Resolution
Central discovers site addresses through the **configuration database**, not runtime registration:
- Each site record in the Sites table includes optional **NodeAAddress** and **NodeBAddress** fields containing base Akka addresses of the site's cluster nodes (e.g., `akka.tcp://scadabridge@host:port`), and optional **GrpcNodeAAddress** and **GrpcNodeBAddress** fields containing gRPC endpoints (e.g., `http://host:8083`).
- The **CentralCommunicationActor** loads all site addresses from the database at startup and creates one **ClusterClient per site**, configured with both NodeA and NodeB as contact points. The **SiteStreamGrpcClientFactory** uses `GrpcNodeAAddress` / `GrpcNodeBAddress` to create per-site gRPC channels for streaming.
- The address cache is **refreshed every 60 seconds** and **on-demand** when site records are added, edited, or deleted via the Central UI or CLI. ClusterClient instances are recreated when contact points change.
- When routing a message to a site, central sends via `ClusterClient.Send("/user/site-communication", msg)`. **ClusterClient handles failover between NodeA and NodeB internally** — there is no application-level NodeA preference/NodeB fallback logic.
- Each site record in the Sites table includes optional **GrpcNodeAAddress** and **GrpcNodeBAddress** fields containing the site nodes' gRPC endpoints (e.g., `http://host:8083`), used for **both** central→site command/control (`SiteCommandService`) and site→central streaming (`SiteStreamService`). The legacy **NodeAAddress** / **NodeBAddress** Akka fields are retained but no longer drive cross-cluster dialling — Akka remoting is now intra-cluster only.
- The **CentralCommunicationActor** loads all site addresses from the database at startup and builds a per-site gRPC-endpoint cache. `GrpcSiteTransport` uses it (via a `SitePairChannelProvider`) to dial each site's `SiteCommandService` as a **per-site NodeANodeB failover channel pair**; the **SiteStreamGrpcClientFactory** uses the same `GrpcNodeAAddress` / `GrpcNodeBAddress` for per-site streaming channels.
- The address cache is **refreshed every 60 seconds** and **on-demand** when site records are added, edited, or deleted via the Central UI or CLI. Channels are rebuilt when a site's gRPC endpoints change.
- When routing a command to a site, central calls `SiteCommandService` on the site's active gRPC endpoint; on error it flips to the other node endpoint. The failover is an explicit NodeA→NodeB channel-pair flip, not internal transport routing.
- **Heartbeats** from sites serve **health monitoring only** — they do not serve as a registration or address discovery mechanism.
- If no addresses are configured for a site, messages to that site are **dropped** and the caller's Ask times out.
- If no gRPC endpoints are configured for a site, commands to that site fail and the caller's call times out.
### Site → Central Communication
- Site nodes configure a list of **CentralContactPoints** (both central node addresses) instead of a single `CentralActorPath`.
- The site creates a **ClusterClient** using the central contact points and sends heartbeats, health reports, and other messages via `ClusterClient.Send("/user/central-communication", msg)`.
- ClusterClient handles automatic failover between central nodes — if the active central node goes down, the site's ClusterClient reconnects to the standby node transparently.
- Site nodes configure a list of **CentralGrpcEndpoints** — the gRPC h2c URLs of the central nodes (e.g., `http://scadabridge-central-a:8083`, on the central's `CentralGrpcPort`, default 8083) — replacing the former `CentralContactPoints` list of Akka addresses. A site node must list at least one; central nodes leave it empty (they host `CentralControlService`, they do not dial it). Note these are the central nodes' **direct** gRPC endpoints, not Traefik (which is HTTP/1 only).
- `GrpcCentralTransport` dials `CentralControlService` over these endpoints as a **sticky central-a→central-b failover channel pair**, and the site sends heartbeats, health reports, notification submit/status, audit/telemetry ingest, and reconcile over it.
- If the active central node goes down, the site's transport flips to the standby central endpoint transparently.
## Message Timeouts
@@ -301,17 +302,17 @@ Each request/response pattern has a default timeout that can be overridden in co
| 5. Recipe/Command Delivery | 30 seconds | Fire-and-forget with ack |
| 8. Remote Queries | 30 seconds | Querying parked messages or event logs |
| 9. Notification Submission | 30 seconds | Fire-and-forget with ack; central acks after persisting the row |
| 10. Cached Call Telemetry | 30 seconds | Telemetry emission (ClusterClient) is fire-and-forget; the reconciliation pull is the unary gRPC `PullSiteCalls` request/response (its deadline is the gRPC call timeout, not the Akka ask) |
| 10. Cached Call Telemetry | 30 seconds | Telemetry emission (gRPC to `CentralControlService`) is fire-and-forget; the reconciliation pull is the unary gRPC `PullSiteCalls` request/response (its deadline is the gRPC call timeout) |
Timeouts use the Akka.NET **ask pattern**. If no response is received within the timeout, the caller receives a timeout failure.
Command/control request/response now uses **gRPC calls with a deadline** in place of the Akka ask; the per-pattern values above are applied as the gRPC call timeout. If no response is received within the timeout, the caller receives a timeout failure.
## Transport Configuration
Akka.NET remoting provides the underlying transport for both intra-cluster communication and ClusterClient connections. The following transport-level settings are **explicitly configured** (not left to framework defaults) for predictable behavior:
Akka.NET remoting provides the underlying transport for **intra-cluster** (pair-internal) communication only; it no longer crosses the site↔central boundary. Cross-cluster command/control, streaming, and audit pulls all ride gRPC. The following settings apply to each concern:
- **Transport heartbeat interval**: Configurable interval at which heartbeat messages are sent over remoting connections (e.g., every 5 seconds).
- **Failure detection threshold**: Number of missed heartbeats before the connection is considered lost (e.g., 3 missed heartbeats = 15 seconds with a 5-second interval).
- **Reconnection**: ClusterClient handles reconnection and failover between contact points automatically for cross-cluster communication. No custom reconnection logic is required.
- **Transport heartbeat interval** (Akka remoting, intra-cluster): Configurable interval at which heartbeat messages are sent over remoting connections within a cluster (e.g., every 5 seconds).
- **Failure detection threshold** (Akka remoting, intra-cluster): Number of missed heartbeats before an intra-cluster connection is considered lost.
- **gRPC reconnection/failover**: Both command/control transports handle reconnection and node failover via their channel pairs — `GrpcCentralTransport` (site→central, central-a→central-b) and `GrpcSiteTransport` (central→site, NodeA→NodeB). No ClusterClient reconnection is involved.
These settings should be tuned for the expected network conditions between central and site clusters.
@@ -327,32 +328,32 @@ This provides protocol-level safety beyond Akka.NET's transport guarantees, whic
## Message Ordering
Akka.NET guarantees message ordering between a specific sender/receiver actor pair. The Communication Layer relies on this guarantee — messages to a given site are processed in the order they are sent. Callers do not need to handle out-of-order delivery.
Within a cluster, Akka.NET guarantees message ordering between a specific sender/receiver actor pair. Across the site↔central boundary, ordering is preserved by the single gRPC channel each transport holds to a given endpoint — commands to a given site over `SiteCommandService` are issued in order over that channel. Callers do not need to handle out-of-order delivery.
## ManagementActor and ClusterClient
## ManagementActor and cross-cluster access
The ManagementActor runs at the well-known path `/user/management` on central nodes. It is **not** advertised via ClusterClientReceptionist, and no ClusterClient reaches it.
The ManagementActor runs at the well-known path `/user/management` on central nodes. It is **not** advertised for cross-cluster access, and nothing outside the central cluster reaches it directly.
That registration existed until 2026-07-22 for an out-of-cluster CLI that was never built. The shipped CLI speaks **HTTP Basic to the central `/management` endpoints**; those endpoints ask the ManagementActor **in-process** through `ManagementActorHolder` (`ManagementEndpoints.cs`). Nothing in the repo ever sent to `/user/management` across the boundary, so the registration was deleted — leaving exactly one receptionist registration per cluster role (`CentralCommunicationActor` on central, `SiteCommunicationActor` on sites), both of which serve inter-cluster central↔site messaging and are themselves scheduled for removal by the gRPC transport migration (`docs/plans/2026-07-22-clusterclient-to-grpc-plan.md`).
A ClusterClientReceptionist registration for it existed until 2026-07-22, for an out-of-cluster CLI that was never built. The shipped CLI speaks **HTTP Basic to the central `/management` endpoints**; those endpoints ask the ManagementActor **in-process** through `ManagementActorHolder` (`ManagementEndpoints.cs`). Nothing in the repo ever sent to `/user/management` across the boundary, so the registration was deleted. The two remaining receptionist registrations — `CentralCommunicationActor` on central and `SiteCommunicationActor` on sites — were subsequently **removed entirely in Phase 4 of the ClusterClient→gRPC migration** (2026-07-23): cross-cluster messaging is now gRPC (`CentralControlService` / `SiteCommandService`), reached by dialling the configured endpoints, so `ClusterClientReceptionist` is no longer used anywhere. `Akka.Cluster.Tools` remains a dependency for ClusterSingleton (`docs/plans/2026-07-22-clusterclient-to-grpc-plan.md`).
## Connection Failure Behavior
Disconnect is detected at the **transport layer**, never via an application-level signal from central. There is no `ConnectionStateChanged`-style synchronous notification: the central coordinator does not maintain a model of "this site is up / down" because the two transports already report unavailability at their natural cadence.
- **In-flight command/control messages (ClusterClient + Ask)**: When a connection drops while a request is in flight (e.g., a deployment sent but no response received), the Akka ask pattern times out and the caller receives a failure. There is **no automatic retry or buffering at central** — the engineer sees the failure in the UI and re-initiates the action. This is consistent with the design principle that central does not buffer messages. An in-progress deployment whose round-trip exceeds the Ask timeout (default 120 s at `CommunicationService.DeployInstanceAsync`) surfaces as `DeploymentStatus.Failed` to the caller.
- **In-flight command/control messages (gRPC call + deadline)**: When a connection drops while a request is in flight (e.g., a deployment sent but no response received), the gRPC call fails or hits its deadline and the caller receives a failure. There is **no automatic retry or buffering at central** — the engineer sees the failure in the UI and re-initiates the action. This is consistent with the design principle that central does not buffer messages. An in-progress deployment whose round-trip exceeds the timeout (default 120 s at `CommunicationService.DeployInstanceAsync`) surfaces as `DeploymentStatus.Failed` to the caller.
- **Debug streams (gRPC)**: Any gRPC stream interruption is detected by the HTTP/2 keepalive PING (~25 s) and triggers reconnection logic in the `DebugStreamBridgeActor`. The bridge actor attempts to reconnect to the other site node endpoint (NodeB if NodeA failed, or vice versa), with up to 3 retries and 5-second backoff. If all retries fail, the consumer is notified via `OnStreamTerminated` and the bridge actor is stopped. Events during the reconnection gap are lost (acceptable for real-time debug view). On successful reconnection, the consumer can request a fresh snapshot to re-sync state.
## Failover Behavior
- **Central failover**: The standby node takes over the Akka.NET cluster role. In-progress deployments are treated as failed. Site ClusterClients automatically reconnect to the standby central node via their configured contact points.
- **Site failover**: The standby node takes over. The Deployment Manager singleton restarts and re-creates the Instance Actor hierarchy. Central's per-site ClusterClient automatically reconnects to the surviving site node. Ongoing debug streams are interrupted and must be re-established by the engineer.
- **Central failover**: The standby node takes over the Akka.NET cluster role. In-progress deployments are treated as failed. Each site's `GrpcCentralTransport` flips to the standby central endpoint (`CentralGrpcEndpoints` pair) transparently.
- **Site failover**: The standby node takes over. The Deployment Manager singleton restarts and re-creates the Instance Actor hierarchy. Central's per-site `GrpcSiteTransport` flips to the surviving site node's gRPC endpoint. Ongoing debug streams are interrupted and must be re-established by the engineer.
## Dependencies
- **Akka.NET Remoting + ClusterClient**: Provides the command/control transport layer. ClusterClient/ClusterClientReceptionist used for cross-cluster command/control messaging (deployments, lifecycle, subscribe/unsubscribe handshake, snapshots).
- **gRPC (Grpc.AspNetCore + Grpc.Net.Client)**: Provides the real-time data streaming transport. Site nodes host a gRPC server (SiteStreamGrpcServer); central nodes create per-site gRPC clients (SiteStreamGrpcClient).
- **gRPC (Grpc.AspNetCore + Grpc.Net.Client)**: Provides the **only** cross-cluster transport — command/control (`CentralControlService` on central, `SiteCommandService` on sites), real-time streaming (`SiteStreamService`, site-hosted), and the audit-pull RPCs. Central hosts `CentralControlService` and dials each site's `SiteCommandService` (`GrpcSiteTransport`) + `SiteStreamService`; sites host `SiteCommandService` + `SiteStreamGrpcServer` and dial `CentralControlService` (`GrpcCentralTransport`).
- **Akka.NET Remoting + `Akka.Cluster.Tools`**: Remoting provides **intra-cluster** transport only. `Akka.Cluster.Tools` provides ClusterSingleton; ClusterClient/ClusterClientReceptionist are no longer used (removed in Phase 4).
- **Cluster Infrastructure**: Manages node roles and failover detection.
- **Configuration Database**: Provides site node addresses (NodeAAddress, NodeBAddress for Akka remoting; GrpcNodeAAddress, GrpcNodeBAddress for gRPC streaming) for address resolution.
- **Configuration Database**: Provides site node gRPC endpoints (GrpcNodeAAddress, GrpcNodeBAddress — used for both command/control and streaming) for address resolution; the legacy Akka NodeAAddress/NodeBAddress fields are retained but no longer drive cross-cluster dialling.
- **Site Runtime (SiteStreamManager)**: The SiteStreamGrpcServer subscribes to SiteStreamManager to receive real-time events for gRPC delivery.
- **`ISiteAuditQueue` (site-local)**: Handed to `SiteStreamGrpcServer` (post-construction, on site roles) so the `PullAuditEvents` RPC can read the site's `Pending`/`Forwarded` audit rows to serve the Audit Log (#23) reconciliation pull. Null when not wired (central-only host) — the handler then returns an empty response.
- **`IOperationTrackingStore` (site-local)**: Handed to `SiteStreamGrpcServer` (post-construction, on site roles) so the `PullSiteCalls` RPC can read operation-tracking rows changed since a cursor to serve the Site Call Audit (#22) reconciliation pull. Null when not wired — the handler returns an empty response.
@@ -364,8 +365,8 @@ Disconnect is detected at the **transport layer**, never via an application-leve
- **Site Runtime**: Receives deployments, lifecycle commands, and artifact updates. Provides debug view data.
- **Central UI**: Debug view requests and remote queries flow through communication.
- **Health Monitoring**: Receives periodic health reports from sites.
- **Store-and-Forward Engine (site)**: Parked message queries/commands are routed through communication. Also emits `CachedCallTelemetry` (push, ClusterClient) and serves the `PullSiteCalls` gRPC reconciliation pull from its `IOperationTrackingStore`, and receives relayed `RetryParkedOperation` / `DiscardParkedOperation` commands.
- **Store-and-Forward Engine (site)**: Parked message queries/commands are routed through communication. Also emits `CachedCallTelemetry` (push, gRPC to `CentralControlService`) and serves the `PullSiteCalls` gRPC reconciliation pull from its `IOperationTrackingStore`, and receives relayed `RetryParkedOperation` / `DiscardParkedOperation` commands.
- **Site Call Audit (central)**: Receives cached-call telemetry and issues the `PullSiteCalls` gRPC reconciliation pulls to sites; relays parked-operation Retry/Discard commands to sites through communication.
- **Audit Log (#23)**: Sites forward audit-event telemetry (push) and serve the `PullAuditEvents` gRPC reconciliation pull from their `ISiteAuditQueue`; the central `AuditLogIngestActor` is the ingest target for both the push path and the combined cached-call telemetry packet.
- **Site Event Logging**: Event log queries are routed through communication.
- **Management Service**: The ManagementActor is registered with ClusterClientReceptionist on central nodes. The CLI communicates with the ManagementActor via ClusterClient, which is a separate channel from inter-cluster remoting.
- **Management Service**: The ManagementActor runs at `/user/management` on central nodes and is reached **in-process** via `ManagementActorHolder`. The CLI communicates with it over the central HTTP `/management` endpoints (HTTP Basic), not over any cluster transport. It is no longer registered with ClusterClientReceptionist (that registration, and the whole ClusterClient surface, is gone).
+15 -10
View File
@@ -45,7 +45,7 @@ The Host must bind configuration sections from `appsettings.json` to strongly-ty
| Section | Options Class | Owner | Contents |
|---------|--------------|-------|----------|
| `ScadaBridge:Node` | `NodeOptions` | Host | Role, NodeHostname, SiteId, RemotingPort, GrpcPort (site only, default 8083) |
| `ScadaBridge:Node` | `NodeOptions` | Host | Role, NodeHostname, SiteId, RemotingPort, GrpcPort (site: hosts `SiteCommandService` + `SiteStreamGrpcServer`, default 8083), CentralGrpcPort (central: hosts `CentralControlService`, default 8083) |
| `ScadaBridge:Cluster` | `ClusterOptions` | ClusterInfrastructure | SeedNodes, SplitBrainResolverStrategy, StableAfter, HeartbeatInterval, FailureDetectionThreshold, MinNrOfMembers |
| `ScadaBridge:Database` | `DatabaseOptions` | Host | Central: ConfigurationDb, MachineDataDb connection strings; Site: SQLite paths |
@@ -113,7 +113,7 @@ The Host bootstraps the Akka.NET actor system from a **hand-assembled, injection
- **Coordinated shutdown**: `run-coordinated-shutdown-when-down = on` plus a cluster-leave phase timeout above the singleton drain budget (see REQ-HOST-4a / the down-if-alone recovery contract in Component-ClusterInfrastructure.md).
- **Actor registration**: each component's actors registered conditional on the node's role.
> **Bootstrap-mechanism note.** The `Akka.Hosting` / `Akka.Cluster.Hosting` / `Akka.Remote.Hosting` typed-builder packages were referenced but unused and were dropped (arch-review 01 Task 22). The hand-rolled HOCON path is a single hardened, production-shape code path with full test coverage; migrating to Akka.Hosting's typed options is deliberately **not** done here and is tracked as possible future work rather than carried as an unused dependency. Only `Akka.Cluster.Tools` (ClusterSingleton/ClusterClient, transitively Akka.Cluster) remains.
> **Bootstrap-mechanism note.** The `Akka.Hosting` / `Akka.Cluster.Hosting` / `Akka.Remote.Hosting` typed-builder packages were referenced but unused and were dropped (arch-review 01 Task 22). The hand-rolled HOCON path is a single hardened, production-shape code path with full test coverage; migrating to Akka.Hosting's typed options is deliberately **not** done here and is tracked as possible future work rather than carried as an unused dependency. Only `Akka.Cluster.Tools` (ClusterSingleton, transitively Akka.Cluster) remains — its ClusterClient/ClusterClientReceptionist surface is no longer used since Phase 4 moved cross-cluster messaging to gRPC.
> **Persistence note.** ScadaBridge does not use Akka.Persistence. Durable state
> (store-and-forward buffers, site event logs, static attribute writes,
@@ -123,13 +123,18 @@ The Host bootstraps the Akka.NET actor system from a **hand-assembled, injection
> `akka.persistence` section and references no persistence plugin. There are no
> `PersistentActor` subclasses in the system by design.
### REQ-HOST-6a: ClusterClientReceptionist (Central Only)
### REQ-HOST-6a: gRPC command/control services (no ClusterClientReceptionist)
On central nodes, the Host must configure the Akka.NET **ClusterClientReceptionist** and register the **CentralCommunicationActor** with it, so that site clusters' ClusterClients can reach the central command/control endpoint without joining the central cluster.
As of Phase 4 of the ClusterClient→gRPC migration (2026-07-23), the Host configures **no** `ClusterClientReceptionist` and registers **no** actor with one. Both remaining receptionist registrations were removed:
**The ManagementActor is NOT registered with the receptionist** (removed 2026-07-22). That registration was written for an out-of-cluster CLI that was never built: the shipped CLI speaks HTTP Basic to the central `/management` endpoints, which ask the ManagementActor **in-process** via `ManagementActorHolder` (`ManagementEndpoints.cs`). No sender to `/user/management` existed anywhere in the repo, so the registration only widened the cluster-client surface for nothing. The actor itself still runs at `/user/management`; only its cross-boundary advertisement is gone.
- The central **CentralCommunicationActor** registration is gone. Sites now reach central command/control over gRPC by dialling the central-hosted **`CentralControlService`** (`GrpcCentralTransport`, sticky central-a→central-b failover), not via a ClusterClient locating a receptionist.
- The site **SiteCommunicationActor** registration is gone. Central now reaches site command/control over gRPC by dialling the site-hosted **`SiteCommandService`** (`GrpcSiteTransport`, per-site NodeA→NodeB failover).
> **Migration note.** This receptionist registration — and the `CentralCommunicationActor` one that remains — are scheduled for deletion once the site↔central transport moves to gRPC. See `docs/plans/2026-07-22-clusterclient-to-grpc-plan.md`.
Central hosts `CentralControlService` (site dials in); each site hosts `SiteCommandService` and `SiteStreamGrpcServer` (central dials in). Endpoints are dialled directly from configuration/DB, so there is no cross-cluster actor discovery.
**The ManagementActor was never reachable over the receptionist either** (its registration was removed 2026-07-22, ahead of Phase 4). That registration was written for an out-of-cluster CLI that was never built: the shipped CLI speaks HTTP Basic to the central `/management` endpoints, which ask the ManagementActor **in-process** via `ManagementActorHolder` (`ManagementEndpoints.cs`). The actor itself still runs at `/user/management`.
> **Migration note.** `Akka.Cluster.Tools` stays a Host dependency for ClusterSingleton; only its ClusterClient/ClusterClientReceptionist surface is now unused. See `docs/plans/2026-07-22-clusterclient-to-grpc-plan.md`.
### REQ-HOST-7: ASP.NET Web Endpoints
@@ -138,7 +143,7 @@ On central nodes, the Host must use `WebApplication.CreateBuilder` to produce a
- Central UI (via `MapCentralUI()` extension method).
- Inbound API (via `MapInboundAPI()` extension method).
On site nodes, the Host must also use `WebApplication.CreateBuilder` (not `Host.CreateDefaultBuilder`) to host the **SiteStreamGrpcServer** via Kestrel HTTP/2 on the configured `GrpcPort` (default 8083). Kestrel is configured with `HttpProtocols.Http2` on the gRPC port only — no HTTP/1.1 web endpoints are exposed. The gRPC service is mapped via `MapGrpcService<SiteStreamGrpcServer>()`.
On site nodes, the Host must also use `WebApplication.CreateBuilder` (not `Host.CreateDefaultBuilder`) to host the **SiteStreamGrpcServer** and the **SiteCommandService** (central→site command/control, added in Phase 4) via Kestrel HTTP/2 on the configured `GrpcPort` (default 8083). Kestrel is configured with `HttpProtocols.Http2` on the gRPC port only — no HTTP/1.1 web endpoints are exposed. On central nodes, the Host additionally hosts the **CentralControlService** (site→central command/control) via Kestrel HTTP/2 on the configured `CentralGrpcPort` (default 8083), alongside the HTTP/1.1 Central UI / Inbound API endpoints. The gRPC services are mapped via `MapGrpcService<>()`.
**Startup ordering (site nodes)**:
1. Actor system and SiteStreamManager must be initialized before gRPC begins accepting connections.
@@ -174,7 +179,7 @@ Each component library must expose its services to the Host via a consistent set
- `AkkaConfigurationBuilder.AddXxxActors()` — registers the component's actors with the Akka.NET actor system (for components that have actors).
- `WebApplication.MapXxx()` — maps the component's web endpoints (only for CentralUI and InboundAPI).
The Host's `Program.cs` calls these extension methods; the component libraries own the registration logic. This keeps the Host thin and each component self-contained. The ManagementService component additionally registers the ManagementActor with ClusterClientReceptionist in its `AddManagementServiceActors()` method.
The Host's `Program.cs` calls these extension methods; the component libraries own the registration logic. This keeps the Host thin and each component self-contained. (The ManagementService component no longer registers the ManagementActor with ClusterClientReceptionist — that registration was removed 2026-07-22, and the whole receptionist surface went in Phase 4.)
---
@@ -209,7 +214,7 @@ The Host's `Program.cs` calls these extension methods; the component libraries o
## Dependencies
- **All 19 component libraries**: The Host references every component project to call their extension methods (excludes CLI, which is a separate executable). Audit Log (#23) ships its central+site code in `ZB.MOM.WW.ScadaBridge.AuditLog`; the Host calls `AddAuditLog()` on both roles, M2+ will add `AddAuditLogActors()`.
- **Akka.Cluster.Tools**: ClusterSingleton (singleton manager/proxy) and ClusterClient/ClusterClientReceptionist; transitively pulls Akka.Cluster (SBR provider) and Akka.Remote. The actor system is built from hand-assembled HOCON (REQ-HOST-6), so the `Akka.Hosting`/`Akka.Remote.Hosting`/`Akka.Cluster.Hosting` typed-builder packages are **not** referenced (dropped in arch-review 01 Task 22). No Akka.Persistence plugin — see the Persistence note under REQ-HOST-6.
- **Akka.Cluster.Tools**: ClusterSingleton (singleton manager/proxy); transitively pulls Akka.Cluster (SBR provider) and Akka.Remote. Its ClusterClient/ClusterClientReceptionist surface is no longer used — cross-cluster messaging is gRPC as of Phase 4. The actor system is built from hand-assembled HOCON (REQ-HOST-6), so the `Akka.Hosting`/`Akka.Remote.Hosting`/`Akka.Cluster.Hosting` typed-builder packages are **not** referenced (dropped in arch-review 01 Task 22). No Akka.Persistence plugin — see the Persistence note under REQ-HOST-6.
- **Serilog.AspNetCore**: For structured logging integration.
- **Microsoft.Extensions.Hosting.WindowsServices**: For Windows Service support.
- **ASP.NET Core** (central only): For web endpoint hosting.
@@ -220,5 +225,5 @@ The Host's `Program.cs` calls these extension methods; the component libraries o
- **Configuration Database**: The Host registers the DbContext and wires repository implementations to their interfaces. In development, triggers auto-migration; in production, validates schema version.
- **ClusterInfrastructure**: The Host configures the underlying Akka.NET cluster that ClusterInfrastructure manages at runtime.
- **CentralUI / InboundAPI**: The Host maps their web endpoints into the ASP.NET Core pipeline on central nodes.
- **ManagementService**: The Host registers the ManagementActor and configures ClusterClientReceptionist on central nodes, enabling CLI access.
- **ManagementService**: The Host registers the ManagementActor on central nodes; the CLI reaches it over the HTTP `/management` endpoints (in-process via `ManagementActorHolder`), not via any cluster transport. No ClusterClientReceptionist is configured (removed in Phase 4).
- **HealthMonitoring**: The Host's startup validation and logging configuration provide the foundation for health reporting.
@@ -134,7 +134,7 @@ Each buffered message stores:
## Dependencies
- **SQLite**: Local persistence on each node.
- **Communication Layer**: Application-level replication to standby node; remote query handling from central; carries buffered notifications to the central cluster (ClusterClient) and receives central's acks.
- **Communication Layer**: Application-level replication to standby node; remote query handling from central; carries buffered notifications to the central cluster (gRPC to `CentralControlService`) and receives central's acks.
- **External System Gateway**: Delivers external system API calls.
- **CentralSite Communication**: The delivery target for the notification category — a buffered notification is forwarded to the central cluster over CentralSite Communication and cleared on central's ack. Also carries `CachedCallTelemetry` and reconciliation responses to central, and receives `RetryParkedOperation` / `DiscardParkedOperation` commands.
- **Site Call Audit**: The central audit mirror for cached calls — receives this engine's cached-call telemetry and reconciliation responses, and relays operator Retry/Discard of parked cached calls back as commands.