ScadaBridge/docs/requirements/Component-Communication.md

Component: Central–Site Communication

Purpose

The Communication component manages all messaging between the central cluster and site clusters. It provides the transport layer for deployments, instance lifecycle commands, integration routing, debug streaming, health reporting, notification submission, and remote queries (parked messages, event logs). gRPC is the only inter-cluster transport (ClusterClient/ClusterClientReceptionist for site↔central messaging was removed in Phase 4 of the ClusterClient→gRPC migration, 2026-07-23). It carries three concerns: command/control in both directions — site→central to the central-hosted CentralControlService (heartbeats, health reports, notification submit/status, audit/telemetry ingest, reconcile) and central→site to the site-hosted SiteCommandService (deployments, lifecycle, OPC UA, remote queries, parked, route, failover); gRPC server-streaming for real-time data (attribute values, alarm states) via SiteStreamService; and plain token-gated HTTP for the deployment-config fetch (notify-and-fetch). Akka remoting remains, but only intra-cluster (pair-internal), never across the site↔central boundary.

Location

Both central and site clusters. Each side has communication actors that handle message routing.

Responsibilities

• Resolve site addresses (Akka remoting — intra-cluster only — and gRPC) from the configuration database and maintain a cached address map, including the per-site gRPC command endpoints used by central→site command/control.
• Establish and maintain cross-cluster command/control connections over gRPC: site→central to CentralControlService (GrpcCentralTransport) and central→site to SiteCommandService (GrpcSiteTransport).
• Establish and maintain per-site gRPC streaming connections for real-time data delivery (site→central).
• Route messages between central and site clusters in a hub-and-spoke topology.
• Broker requests from external systems (via central) to sites and return responses.
• Support multiple concurrent message patterns (request/response, fire-and-forget, streaming).
• Detect site connectivity status for health monitoring.
• Host the SiteStreamGrpcServer on site nodes (Kestrel HTTP/2) to serve real-time event streams.
• Manage per-site SiteStreamGrpcClient instances on central nodes via SiteStreamGrpcClientFactory.

Communication Patterns

1. Deployment (Central → Site)

• Pattern: Request/Response.
• Central sends a flattened configuration to a site.
• Site Runtime receives, compiles scripts, creates/updates Instance Actors, and responds with success/failure.
• No buffering at central — if the site is unreachable, the deployment fails immediately.

2. Instance Lifecycle Commands (Central → Site)

• Pattern: Request/Response.
• Central sends disable, enable, or delete commands for specific instances.
• Site Runtime processes the command and responds with success/failure.
• If the site is unreachable, the command fails immediately (no buffering).

3. System-Wide Artifact Deployment (Central → Site(s))

• Pattern: Broadcast with per-site acknowledgment (deploy to all sites), or targeted to a single site (per-site deployment).
• When shared scripts, external system definitions, database connections, or data connections are explicitly deployed, central sends them to the target site(s). (Notification lists and SMTP configuration are central-only and are not deployed to sites.)
• Each site acknowledges receipt and reports success/failure independently.
• Shared script deployment triggers immediate recompilation on the site — the site's SharedScriptLibrary replaces its in-memory compiled code, making updated shared scripts available to all running instances without redeployment. Other artifact types (external systems, database connections, etc.) are stored but do not require recompilation.

4. Integration Routing (External System → Central → Site → Central → External System)

• Pattern: Request/Response (brokered).
• External system sends a request to central (e.g., MES requests machine values).
• Central routes the request to the appropriate site.
• Site reads values from the Instance Actor and responds.
• Central returns the response to the external system.
• Implementation: this brokered round-trip is served by the Inbound API's routed-site-script path — a central-side inbound API method script composes the typed RouteTo* verbs on CommunicationService (RouteToCall, RouteToGetAttributes, RouteToSetAttributes, RouteToWaitForAttribute), driven from InboundAPI/CommunicationServiceInstanceRouter and sharing IntegrationTimeout. An earlier generic IntegrationCallRequest primitive was scaffolded for this pattern but never wired to a producer or a site handler; it was removed as dead code (Gitea #32, 2026-07-23).

5. Recipe/Command Delivery (External System → Central → Site)

• Pattern: Fire-and-forget with acknowledgment.
• External system sends a command to central (e.g., recipe manager sends recipe).
• Central routes to the site.
• Site applies and acknowledges.

6. Debug Streaming (Site → Central)

• Pattern: Subscribe/push with initial snapshot. Both legs ride gRPC: the subscribe/unsubscribe handshake and initial snapshot go over gRPC command/control (SiteCommandService), and ongoing real-time events over gRPC server-streaming (SiteStreamService).
• A DebugStreamBridgeActor (one per active debug session) is created on the central cluster by the DebugStreamService. The bridge actor first opens a gRPC server-streaming subscription to the site via SiteStreamGrpcClient, then sends a SubscribeDebugViewRequest to the site as a central→site command over SiteCommandService (GrpcSiteTransport). The site's InstanceActor replies with an initial snapshot on that gRPC request/response.
• gRPC stream (real-time events): The site's SiteStreamGrpcServer receives the gRPC SubscribeInstance call and creates a StreamRelayActor that subscribes to SiteStreamManager for the requested instance. Events (AttributeValueChanged, AlarmStateChanged) flow from SiteStreamManager → StreamRelayActor → Channel<SiteStreamEvent> (bounded, 1000, DropOldest) → gRPC response stream → SiteStreamGrpcClient on central → DebugStreamBridgeActor.
• The DebugStreamEvent message type no longer exists — events are not routed through a command/control channel. SiteCommunicationActor and CentralCommunicationActor have no role in streaming event delivery.
• The bridge actor forwards received events to the consumer via callbacks (Blazor component or SignalR hub).
• Snapshot-to-stream handoff: The gRPC stream is opened before the snapshot request to avoid missing events. The consumer applies the snapshot as baseline, then replays buffered gRPC events with timestamps newer than the snapshot (timestamp-based dedup).
• Attribute value stream messages: [InstanceUniqueName].[AttributePath].[AttributeName], value, quality, timestamp.
• Alarm state stream messages: [InstanceUniqueName].[AlarmName], state (active/normal), priority, timestamp.
• Central sends an unsubscribe request over SiteCommandService (gRPC command/control) when the debug session ends. The gRPC stream is cancelled. The site's StreamRelayActor is stopped and the SiteStreamManager subscription is removed.
• The stream is session-based and temporary.
• Accepted limitation (central-side session locality): debug sessions are process-local state on the central node hosting the DebugStreamBridgeActor. A central restart or failover drops all active sessions with no OnStreamTerminated signal to the operator — the engineer simply re-establishes the debug session from the UI. Debug streaming is an interactive, transient diagnostic aid, so this is an accepted trade-off rather than a durability gap (arch review 02, U7).
• Backpressure is lossy-by-design and now observable: the per-session Channel<SiteStreamEvent> (bounded 1000, DropOldest) silently evicts the oldest event when a slow consumer falls behind. The eviction is counted via the channel's itemDropped callback and logged at Warning (first eviction, then every 500th) so real event loss on a debug stream is visible instead of silent.

6.1 Aggregated Live Alarm Stream (Site → Central)

Delivered 2026-07-10 (docs/plans/2026-07-10-aggregated-live-alarm-stream-plan.md) for the operator Alarm Summary page (Central UI #9). Where §6 streams one instance for the interactive Debug View, this feeds a transient, per-site central live alarm cache so the Alarm Summary reflects alarm transitions in near-real-time instead of re-polling the whole site every 15s.

• Site-wide, alarm-only stream: a single SubscribeSite gRPC stream per site carries AlarmStateChanged events for all instances (attribute events are dropped — the summary never shows them, and attributes are far higher-volume). On the site this reuses the existing SiteStreamManager broadcast hub via SiteStreamManager.SubscribeSiteAlarms(...) (same wiring as the per-instance Subscribe, minus the InstanceUniqueName filter and filtered to alarms). StreamRelayActor is reused unchanged, so placeholder rows (is_configured_placeholder) are still dropped at the relay.
• Central live cache (ISiteAlarmLiveCache, singleton SiteAlarmLiveCacheService): a DI singleton on the active central node. For each site with ≥1 active viewer it runs ONE shared, reference-counted per-site aggregator (SiteAlarmAggregatorActor); the first Subscribe(siteId, onChanged) starts it, the last subscriber leaving stops it after a short linger to avoid re-seed thrash. GetCurrentAlarms(siteId) returns the current immutable snapshot; IsLive(siteId) reports whether the aggregator has seeded and published at least once.
• Seed-then-stream (copied from DebugStreamBridgeActor ordering): open the SubscribeSite stream first (buffer live deltas), run the snapshot fan-out once via the existing DebugViewSnapshot path (bounded by LiveAlarmCacheSeedConcurrency), flush the buffer with dedup by (InstanceUniqueName, AlarmName, SourceReference), then live pass-through. Placeholders are seeded from the snapshot and never expected on the live stream.
• Failover & drift: NodeA↔NodeB reconnect (client via SiteStreamGrpcClient.SubscribeSiteAsync, factory endpoint failover) re-seeds rather than serving stale; a periodic reconcile snapshot (default 60s) corrects any missed enable/disable so the cache can't drift indefinitely. [PERM] (docs/plans/2026-05-29-native-alarms-design.md): the cache is purely in-memory — no EF entity/table/migration, no persisted central alarm store — so a new active node simply re-seeds from scratch.
• Options (Communication section, CommunicationOptions; eagerly validated by CommunicationOptionsValidator / ValidateOnStart): LiveAlarmCacheLinger (default 30s), LiveAlarmCacheReconcileInterval (default 60s), LiveAlarmCacheSeedConcurrency (default 8), LiveAlarmCacheMaxSubscribersPerSite (default 200), LiveAlarmCachePublishCoalesce (default 250ms; 0 = publish per delta — legacy — batches an alarm storm into one snapshot copy + one viewer fan-out per window; arch review 02 round 2, N6).
• Telemetry (ScadaBridgeTelemetry meter): observable gauge scadabridge.site.alarm_cache.aggregators.active (running per-site aggregators) and counter scadabridge.site.alarm_cache.reconnects (site-wide stream reconnects — a NodeA↔NodeB flip or reconcile-driven reopen; a sustained climb signals a flapping site link).
• Accepted limitations (arch review 02 round 2, N8):
  • Standby-node aggregators: the live cache is per-node and SetActorSystem is wired on every central node, so browsing the standby node directly (diagnostic ports, e.g. 9002) starts a second, fully-functional read-only aggregator + SubscribeSite stream there. Accepted — not gated: the aggregator is read-only, bounded (one stream/site, viewer-capped), and torn down by the viewer linger; gating SetActorSystem behind the active check would break the diagnostic-browse path and buy nothing. The [PERM] "lives only on the active central node" claim is corrected to "per-node; in routine operation only the active node hosts viewers".
  • Site deleted while an Alarm Summary viewer is open: the viewer's aggregator reconciles to an empty snapshot (the deleted site's instances vanish from the fan-out) until its viewers leave, then the linger stop reaps it; the site's cached gRPC channels are disposed at deletion via SiteStreamGrpcClientFactory.RemoveSiteAsync (see ManagementActor.HandleDeleteSite, arch review 02 round 2, N8).

Site-Side gRPC Streaming Components

• SiteStreamGrpcServer: gRPC service (SiteStreamService.SiteStreamServiceBase) hosted on each site node via Kestrel HTTP/2 on a dedicated port (default 8083). Implements the SubscribeInstance RPC. For each subscription, creates a StreamRelayActor that subscribes to SiteStreamManager, bridges events through a Channel<SiteStreamEvent> to the gRPC response stream. Tracks active subscriptions by correlation_id — duplicate IDs cancel the old stream. Enforces a max concurrent stream limit (default 100). Rejects streams with StatusCode.Unavailable before the actor system is ready.
• StreamRelayActor: Short-lived actor created per gRPC subscription. Receives domain events (AttributeValueChanged, AlarmStateChanged) from SiteStreamManager, converts them to protobuf SiteStreamEvent messages, and writes to the Channel<SiteStreamEvent> writer. Stopped when the gRPC stream is cancelled or the client disconnects.

Central-Side Debug Stream Components

• DebugStreamService: Singleton service that manages debug stream sessions. Resolves instance ID to unique name and site, creates and tears down DebugStreamBridgeActor instances, and provides a clean API for both Blazor components and the SignalR hub. Injects SiteStreamGrpcClientFactory for gRPC stream creation.
• DebugStreamBridgeActor: One per active debug session. Opens a gRPC streaming subscription via SiteStreamGrpcClient and receives real-time events via callback. Also receives the initial DebugViewSnapshot over gRPC command/control (SiteCommandService). Forwards all events to the consumer via callbacks. Handles gRPC stream errors with reconnection logic: tries the other site node endpoint, retries with backoff (max 3 retries), terminates the session if all retries fail.
• SiteStreamGrpcClient: Per-site gRPC client that manages GrpcChannel instances and streaming subscriptions. Reads from the gRPC response stream in a background task, converts protobuf messages to domain events, and invokes the onEvent callback.
• SiteStreamGrpcClientFactory: Caches per-site SiteStreamGrpcClient instances. Reads GrpcNodeAAddress / GrpcNodeBAddress from the Site entity (loaded by CentralCommunicationActor). Falls back to NodeB if NodeA connection fails. Disposes clients on site removal or address change.
• DebugStreamHub: SignalR hub at /hubs/debug-stream for external consumers (e.g., CLI). Authenticates via Basic Auth + LDAP and requires the Deployment role. Server-to-client methods: OnSnapshot, OnAttributeChanged, OnAlarmChanged, OnStreamTerminated.

gRPC Proto Definition

The streaming protocol is defined in sitestream.proto (src/ZB.MOM.WW.ScadaBridge.Communication/Protos/sitestream.proto):

• Service: SiteStreamService — hosted on each site node by SiteStreamGrpcServer — exposes six RPCs. Two are real-time server-streaming subscriptions (SubscribeInstance, SubscribeSite); the other four are unary request/response calls added by the Audit Log (#23) and Site Call Audit (#22) components. These are distinct from the command/control gRPC services (CentralControlService on central, SiteCommandService on the site) — SiteStreamService is the streaming + audit-pull surface, not the command/control channel:
  • SubscribeInstance(InstanceStreamRequest) returns (stream SiteStreamEvent) — the per-instance real-time debug stream (§6).
  • SubscribeSite(SiteStreamRequest) returns (stream SiteStreamEvent) — the site-wide, alarm-only aggregated stream (§6.1) added for the operator Alarm Summary live cache. Additive (new RPC + new SiteStreamRequest { correlation_id } message; no field renumbering). The server handler mirrors SubscribeInstance — same bounded Channel(1000, DropOldest), GrpcMaxConcurrentStreams/GrpcMaxStreamLifetime limits, SiteConnectionOpened/Closed telemetry, and StreamRelayActor mapping — but subscribes via SiteStreamManager.SubscribeSiteAlarms (all instances, AlarmStateChanged only; attribute events dropped, no per-instance filter).
  • IngestAuditEvents(AuditEventBatch) returns (IngestAck) — a legacy central-side ingest surface on the site-hosted service; it is dead in the shipped topology because no site ever dials a site for ingest. The production audit-telemetry push path is site→central gRPC to CentralControlService, which routes the batch to the central AuditLogIngestActor proxy and returns the accepted EventIds.
  • IngestCachedTelemetry(CachedTelemetryBatch) returns (IngestAck) — ingest receiving surface for the combined cached-call telemetry packet (audit row + SiteCalls operational upsert written in one transaction).
  • PullAuditEvents(PullAuditEventsRequest) returns (PullAuditEventsResponse) — central→site reconciliation pull for the Audit Log self-heal feed; the site serves Pending/Forwarded rows from its ISiteAuditQueue.
  • PullSiteCalls(PullSiteCallsRequest) returns (PullSiteCallsResponse) — central→site reconciliation pull for the Site Call Audit (#22) self-heal feed; the site serves operation-tracking rows changed since a cursor from its IOperationTrackingStore. A separate RPC from PullAuditEvents because the tracking store is the operational source of truth, distinct from the site audit queue.
• Messages: InstanceStreamRequest (correlation_id, instance_unique_name), SiteStreamRequest (correlation_id only — no instance name; drives the site-wide SubscribeSite stream), SiteStreamEvent (correlation_id, oneof event: AttributeValueUpdate, AlarmStateUpdate); AuditEventDto/AuditEventBatch/IngestAck for ingest; CachedTelemetryPacket/CachedTelemetryBatch (each packet pairing an AuditEventDto with a SiteCallOperationalDto); PullAuditEventsRequest/PullAuditEventsResponse and PullSiteCallsRequest/PullSiteCallsResponse (each request carries since_utc + batch_size; each response carries more_available to signal a saturated batch).
• The oneof event pattern is extensible — future event types (health metrics, connection state changes) are added as new fields without breaking existing consumers.
• Proto field numbers are never reused; new RPCs and message fields are appended additively. Old clients ignore unknown oneof variants.

Authentication (preshared key, 2026-07-22)

Every RPC on this service — streaming and unary alike — requires a preshared key, presented as authorization: Bearer <key> metadata and verified by ControlPlaneAuthInterceptor on the site node. The gate is fail-closed: with no key configured, every call is refused with PermissionDenied, and StartupValidator refuses to boot a site node in that state (an unset key would otherwise leave the node joined and reporting healthy while serving nothing).

Keys are scoped one per site — secret SB-GRPC-PSK-<siteId>, so a compromised site never yields another site's key. The site node reads its own key from ScadaBridge:Communication:GrpcPsk (in production a ${secret:} reference expanded before the host is built); central resolves each site's key at channel-build time via SitePskProvider, because sites are created at runtime and cannot be enumerated in configuration at boot. ControlPlaneCredentials binds the key to the channel as CallCredentials, so no individual call site can omit it.

This is distinct from LocalDb:Replication:ApiKey, which gates /localdb_sync.v1.LocalDbSync/ on the same listener via its own interceptor: that key authenticates the pair partner for database replication, a different trust relationship, and the two are never shared.

The transport remains h2c, so the key is readable and replayable by anyone on the path — the boundary still assumes a trusted network, with the bar raised from "can reach the port" to "can read the traffic". TLS is follow-on hardening and does not change this design.

Enriched AlarmStateUpdate (Native Alarm Mirror)

AlarmStateUpdate carries the read-only native alarm mirror (Computed, native OPC UA, and native MxAccess Gateway alarms) to central over the existing gRPC real-time stream — no new transport, no command/control round-trip. The message was extended additively: existing fields 1–7 are unchanged, and fields 8–23 carry the enriched native-alarm state. Old clients that only read fields 1–7 continue to work; new fields are populated only where the source provides them.

Field	#	Type	Meaning
kind	8	string	Alarm origin: Computed, NativeOpcUa, or NativeMxAccess.
active	9	bool	Alarm condition is active.
acknowledged	10	bool	Alarm has been acknowledged.
confirmed	11	bool	Alarm has been confirmed. The domain Confirmed (bool?) collapses to a definite bool on the wire.
shelve_state	12	string	Unshelved, OneShotShelved, TimedShelved, or PermanentShelved.
suppressed	13	bool	Alarm is suppressed by the source system.
source_reference	14	string	Source node / tag reference.
alarm_type_name	15	string	Native alarm type name.
category	16	string	Alarm category.
operator_user	17	string	User who last acted on the alarm.
operator_comment	18	string	Operator comment from the last action.
original_raise_time	19	Timestamp	First-raise time of the underlying condition (nullable on the wire).
current_value	20	string	Current process value associated with the alarm.
limit_value	21	string	Limit / setpoint value that the alarm evaluates against.
native_source_canonical_name	22	string	Native binding canonical name; empty for computed alarms.
is_configured_placeholder	23	bool	Marks a quiet-binding placeholder row. Snapshot-only — see the relay note below; on the live gRPC stream this is always false.

• Server-side mapping (StreamRelayActor.HandleAlarmStateChanged): maps the enriched domain AlarmStateChanged event — Kind + AlarmConditionState + native metadata — out to the proto AlarmStateUpdate. The nullable original_raise_time is emitted only when present, and shelve_state is mapped from the domain shelve enum to its wire string via a new AlarmShelveStateCodec (string↔enum, defaulting to Unshelved). The domain Confirmed (bool?) is collapsed to a definite bool for field 11.
• Placeholder rows are dropped at the relay: is_configured_placeholder (field 23) is a Debug View snapshot-only concept emitted by InstanceActor.BuildAlarmStatesSnapshot for quiet bindings — it is never a real alarm transition (its timestamp may be DateTimeOffset.MinValue, the Protobuf Timestamp lower boundary). StreamRelayActor.HandleAlarmStateChanged therefore returns early — never relaying a placeholder row to the live gRPC stream — so field 23 is always false on the live stream and only ever carries true in the snapshot path.
• Client-side mapping (SiteStreamGrpcClient.ConvertToDomainEvent): reconstructs the domain AlarmStateChanged from the proto — Kind is parsed via ParseAlarmKind, the Condition is rebuilt with severity taken from the existing wire priority, and native metadata is repopulated from fields 8–23 (native_source_canonical_name → NativeSourceCanonicalName, is_configured_placeholder → IsConfiguredPlaceholder) — so central-side consumers receive the same domain event the site emitted.

Regeneration is manual (macOS-only). sitestream.proto is not auto-compiled: the <Protobuf> include is commented out in the .csproj, and the generated C# is vendored under SiteStreamGrpc/. To regenerate after editing the proto: toggle the <Protobuf> include on, build so Grpc.Tools regenerates the C#, copy the generated files into SiteStreamGrpc/, then re-comment the include. Adding AlarmStateUpdate fields 8–23 and the four unary RPCs (IngestAuditEvents, IngestCachedTelemetry, PullAuditEvents, PullSiteCalls) plus their message types followed this process.

gRPC Connection Keepalive

Three layers of dead-client detection prevent orphan streams on site nodes:

Layer	Detects	Timeline	Mechanism
TCP RST	Clean process death, connection close	1–5s	OS-level TCP, WriteAsync throws
gRPC keepalive PING	Network partition, silent crash, firewall drop	~25s	HTTP/2 PING frames, CancellationToken fires
Session timeout	Misconfigured keepalive, long-lived zombie streams	4 hours	CancellationTokenSource.CancelAfter

Keepalive settings are configurable via CommunicationOptions:

• GrpcKeepAlivePingDelay: 15 seconds (default)
• GrpcKeepAlivePingTimeout: 10 seconds (default)
• GrpcMaxStreamLifetime: 4 hours (default)
• GrpcMaxConcurrentStreams: 100 (default)

6a. Debug Snapshot (Central → Site)

• Pattern: Request/Response (one-shot, no subscription).
• Central sends a DebugSnapshotRequest (identified by instance unique name) to the site.
• Site's Deployment Manager routes to the Instance Actor by unique name.
• Instance Actor builds and returns a DebugViewSnapshot with all current attribute values and alarm states (same payload as the streaming initial snapshot).
• No subscription is created; no stream is established.
• Uses the 30-second QueryTimeout.

7. Health Reporting (Site → Central)

• Pattern: Periodic push.
• Sites periodically send health metrics (connection status, node status, buffer depth, script error rates, alarm evaluation error rates) to central.

8. Remote Queries (Central → Site)

• Pattern: Request/Response.
• Central queries sites for:
  • Parked messages (store-and-forward dead letters).
  • Site event logs.
  • Instance debug snapshots (attribute values and alarm states).
• Central can also send management commands:
  • Retry or discard parked messages and parked cached calls — central sends RetryParkedOperation / DiscardParkedOperation (keyed by TrackedOperationId) to the owning site, which applies the change to its S&F buffer and tracking table.

9. Notification Submission (Site → Central)

• Pattern: Fire-and-forget with acknowledgment.
• The site Store-and-Forward Engine sends a NotificationSubmit message to central carrying the notification — NotificationId, target list name, subject, body, and source provenance.
• Central ingests the submission with an insert-if-not-exists on NotificationId and acknowledges after the row is persisted to the Notifications table in the central configuration database. The site S&F engine clears the buffered message only on that ack.
• The NotificationId GUID — generated at the site — is the idempotency key. The handoff is at-least-once: a re-sent submission after a lost ack is harmless because central's insert-if-not-exists treats the duplicate as a no-op.
• Transport: gRPC to CentralControlService (site→central command/control), consistent with how other site→central messages are sent.

10. Cached Call Telemetry (Site → Central)

• Pattern: Fire-and-forget telemetry with a periodic reconciliation pull.
• The site Store-and-Forward Engine emits a CachedCallTelemetry message to central on every cached-call lifecycle transition (Pending → Retrying → Delivered / Parked / Failed / Discarded). The first telemetry event for an operation carries its initial status — Pending when a transient failure has buffered the call, or directly Delivered/Failed for a cached call that never buffers. The message carries the TrackedOperationId, source site, Kind (the TrackedOperationKind enum), target summary, status, retry count, last error, key timestamps, and source provenance.
• Emission is best-effort and at-least-once, idempotent on TrackedOperationId — central's Site Call Audit component ingests with insert-if-not-exists then upsert-on-newer-status, so a re-sent or out-of-order event is harmless.
• Reconciliation pull: because telemetry is best-effort, the central Site Call Audit component periodically — and on site reconnect — pulls the changed rows back from each site over the PullSiteCalls unary gRPC RPC on SiteStreamService (a distinct surface from the CentralControlService/SiteCommandService command/control path). Central sends a PullSiteCallsRequest (since_utc cursor + batch_size); the site reads its IOperationTrackingStore and replies with a PullSiteCallsResponse carrying the matching operation-tracking rows (as SiteCallOperationalDtos) plus a more_available flag that signals a saturated batch so central advances the cursor and pulls again. Any telemetry missed during a disconnect self-heals through this pull. The Audit Log (#23) reconciliation feed uses the sibling PullAuditEvents RPC the same way.
• Central audit is an eventually-consistent mirror — the site's operation tracking table remains the source of truth for cached-call status (Tracking.Status(id) is always answered site-locally).
• Transport: the push telemetry emission rides gRPC to CentralControlService (site→central command/control), consistent with how other site→central messages are sent; the reconciliation pull rides the gRPC unary PullSiteCalls RPC on SiteStreamService (central→site request/response). The two paths are complementary — push is the fast, best-effort feed; pull is the slower self-heal backfill.

Topology

%%{init: {'theme':'base', 'themeVariables': {'textColor':'#111111','lineColor':'#555555','edgeLabelBackground':'#ffffff','fontSize':'15px'}}}%%
flowchart LR
    subgraph Central["Central Cluster"]
        CCS["CentralControlService<br/>(gRPC server — site→central cmd/ctrl)"]
        SCA["GrpcSiteTransport → SiteCommandService<br/>(gRPC client, per site)"]
        SCB["GrpcSiteTransport → SiteCommandService"]
        SCN["GrpcSiteTransport → SiteCommandService"]
        GRPCC["SiteStreamGrpcClient<br/>(real-time data)"]
    end

    subgraph SiteA["Site A Cluster"]
        SACMD["SiteCommandService<br/>(gRPC server — central→site cmd/ctrl)"]
        SAGRPC["SiteStreamGrpcServer<br/>(Kestrel HTTP/2, port 8083)"]
        SACC["GrpcCentralTransport → CentralControlService<br/>(gRPC client)"]
    end

    subgraph SiteB["Site B Cluster"]
        SBCMD["SiteCommandService"]
        SBGRPC["SiteStreamGrpcServer"]
    end

    subgraph SiteN["Site N Cluster"]
        SNCMD["SiteCommandService"]
        SNGRPC["SiteStreamGrpcServer"]
    end

    SCA -->|"gRPC command/control"| SACMD
    SCB -->|"gRPC command/control"| SBCMD
    SCN -->|"gRPC command/control"| SNCMD

    SAGRPC -->|"gRPC stream (real-time data)"| GRPCC
    SBGRPC -->|gRPC stream| GRPCC
    SNGRPC -->|gRPC stream| GRPCC

    SACC -.->|"gRPC command/control"| CCS

    NOTE["Sites do NOT communicate with each other.<br/>All inter-cluster communication flows through Central."]

    classDef start fill:#d5e8d4,stroke:#82b366,color:#111111;
    classDef proc fill:#dae8fc,stroke:#6c8ebf,color:#111111;
    classDef dec fill:#fff2cc,stroke:#d6b656,color:#111111;
    classDef alt fill:#e1d5e7,stroke:#9673a6,color:#111111;
    classDef muted fill:#f5f5f5,stroke:#999999,color:#666666;
    class CCS,SCA,SCB,SCN,SACMD,SACC,SBCMD,SNCMD dec
    class GRPCC,SAGRPC,SBGRPC,SNGRPC start
    class NOTE muted
    class Central proc
    class SiteA,SiteB,SiteN alt

• Sites do not communicate with each other.
• All inter-cluster communication flows through central.
• Command/control uses gRPC in both directions: central dials each site's SiteCommandService (per-site NodeA→NodeB failover channel pair), and each site dials the central CentralControlService (sticky central-a→central-b failover channel pair). There is no ClusterClientReceptionist and no cross-cluster actor discovery — the endpoints are dialled directly.

Site Address Resolution

Central discovers site addresses through the configuration database, not runtime registration:

• Each site record in the Sites table includes optional GrpcNodeAAddress and GrpcNodeBAddress fields containing the site nodes' gRPC endpoints (e.g., http://host:8083), used for both central→site command/control (SiteCommandService) and site→central streaming (SiteStreamService). The legacy NodeAAddress / NodeBAddress Akka fields are retained but no longer drive cross-cluster dialling — Akka remoting is now intra-cluster only.
• The CentralCommunicationActor loads all site addresses from the database at startup and builds a per-site gRPC-endpoint cache. GrpcSiteTransport uses it (via a SitePairChannelProvider) to dial each site's SiteCommandService as a per-site NodeA→NodeB failover channel pair; the SiteStreamGrpcClientFactory uses the same GrpcNodeAAddress / GrpcNodeBAddress for per-site streaming channels.
• The address cache is refreshed every 60 seconds and on-demand when site records are added, edited, or deleted via the Central UI or CLI. Channels are rebuilt when a site's gRPC endpoints change.
• When routing a command to a site, central calls SiteCommandService on the site's active gRPC endpoint; on error it flips to the other node endpoint. The failover is an explicit NodeA→NodeB channel-pair flip, not internal transport routing.
• Heartbeats from sites serve health monitoring only — they do not serve as a registration or address discovery mechanism.
• If no gRPC endpoints are configured for a site, commands to that site fail and the caller's call times out.

Site → Central Communication

• Site nodes configure a list of CentralGrpcEndpoints — the gRPC h2c URLs of the central nodes (e.g., http://scadabridge-central-a:8083, on the central's CentralGrpcPort, default 8083) — replacing the former CentralContactPoints list of Akka addresses. A site node must list at least one; central nodes leave it empty (they host CentralControlService, they do not dial it). Note these are the central nodes' direct gRPC endpoints, not Traefik (which is HTTP/1 only).
• GrpcCentralTransport dials CentralControlService over these endpoints as a sticky central-a→central-b failover channel pair, and the site sends heartbeats, health reports, notification submit/status, audit/telemetry ingest, and reconcile over it.
• If the active central node goes down, the site's transport flips to the standby central endpoint transparently.

Message Timeouts

Each request/response pattern has a default timeout that can be overridden in configuration:

Pattern	Default Timeout	Rationale
1. Deployment	120 seconds	Script compilation at the site can be slow
2. Instance Lifecycle	30 seconds	Stop/start actors is fast
3. System-Wide Artifacts	120 seconds per site	Includes shared script recompilation
4. Integration Routing	30 seconds	External system waiting for response; Inbound API per-method timeout may cap this further
5. Recipe/Command Delivery	30 seconds	Fire-and-forget with ack
8. Remote Queries	30 seconds	Querying parked messages or event logs
9. Notification Submission	30 seconds	Fire-and-forget with ack; central acks after persisting the row
10. Cached Call Telemetry	30 seconds	Telemetry emission (gRPC to CentralControlService) is fire-and-forget; the reconciliation pull is the unary gRPC PullSiteCalls request/response (its deadline is the gRPC call timeout)

Command/control request/response now uses gRPC calls with a deadline in place of the Akka ask; the per-pattern values above are applied as the gRPC call timeout. If no response is received within the timeout, the caller receives a timeout failure.

Transport Configuration

Akka.NET remoting provides the underlying transport for intra-cluster (pair-internal) communication only; it no longer crosses the site↔central boundary. Cross-cluster command/control, streaming, and audit pulls all ride gRPC. The following settings apply to each concern:

• Transport heartbeat interval (Akka remoting, intra-cluster): Configurable interval at which heartbeat messages are sent over remoting connections within a cluster (e.g., every 5 seconds).
• Failure detection threshold (Akka remoting, intra-cluster): Number of missed heartbeats before an intra-cluster connection is considered lost.
• gRPC reconnection/failover: Both command/control transports handle reconnection and node failover via their channel pairs — GrpcCentralTransport (site→central, central-a→central-b) and GrpcSiteTransport (central→site, NodeA→NodeB). No ClusterClient reconnection is involved.

These settings should be tuned for the expected network conditions between central and site clusters.

Application-Level Correlation

All request/response messages include an application-level correlation ID to ensure correct pairing of requests and responses, even across reconnection events:

• Deployments include a deployment ID and revision hash for idempotency (see Deployment Manager).
• Lifecycle commands include a command ID for deduplication.
• Remote queries include a query ID for response correlation.

This provides protocol-level safety beyond Akka.NET's transport guarantees, which may not hold across disconnect/reconnect cycles.

Message Ordering

Within a cluster, Akka.NET guarantees message ordering between a specific sender/receiver actor pair. Across the site↔central boundary, ordering is preserved by the single gRPC channel each transport holds to a given endpoint — commands to a given site over SiteCommandService are issued in order over that channel. Callers do not need to handle out-of-order delivery.

ManagementActor and cross-cluster access

The ManagementActor runs at the well-known path /user/management on central nodes. It is not advertised for cross-cluster access, and nothing outside the central cluster reaches it directly.

A ClusterClientReceptionist registration for it existed until 2026-07-22, for an out-of-cluster CLI that was never built. The shipped CLI speaks HTTP Basic to the central /management endpoints; those endpoints ask the ManagementActor in-process through ManagementActorHolder (ManagementEndpoints.cs). Nothing in the repo ever sent to /user/management across the boundary, so the registration was deleted. The two remaining receptionist registrations — CentralCommunicationActor on central and SiteCommunicationActor on sites — were subsequently removed entirely in Phase 4 of the ClusterClient→gRPC migration (2026-07-23): cross-cluster messaging is now gRPC (CentralControlService / SiteCommandService), reached by dialling the configured endpoints, so ClusterClientReceptionist is no longer used anywhere. Akka.Cluster.Tools remains a dependency for ClusterSingleton (docs/plans/2026-07-22-clusterclient-to-grpc-plan.md).

Connection Failure Behavior

Disconnect is detected at the transport layer, never via an application-level signal from central. There is no ConnectionStateChanged-style synchronous notification: the central coordinator does not maintain a model of "this site is up / down" because the two transports already report unavailability at their natural cadence.

• In-flight command/control messages (gRPC call + deadline): When a connection drops while a request is in flight (e.g., a deployment sent but no response received), the gRPC call fails or hits its deadline and the caller receives a failure. There is no automatic retry or buffering at central — the engineer sees the failure in the UI and re-initiates the action. This is consistent with the design principle that central does not buffer messages. An in-progress deployment whose round-trip exceeds the timeout (default 120 s at CommunicationService.DeployInstanceAsync) surfaces as DeploymentStatus.Failed to the caller.
• Debug streams (gRPC): Any gRPC stream interruption is detected by the HTTP/2 keepalive PING (~25 s) and triggers reconnection logic in the DebugStreamBridgeActor. The bridge actor attempts to reconnect to the other site node endpoint (NodeB if NodeA failed, or vice versa), with up to 3 retries and 5-second backoff. If all retries fail, the consumer is notified via OnStreamTerminated and the bridge actor is stopped. Events during the reconnection gap are lost (acceptable for real-time debug view). On successful reconnection, the consumer can request a fresh snapshot to re-sync state.

Failover Behavior

• Central failover: The standby node takes over the Akka.NET cluster role. In-progress deployments are treated as failed. Each site's GrpcCentralTransport flips to the standby central endpoint (CentralGrpcEndpoints pair) transparently.
• Site failover: The standby node takes over. The Deployment Manager singleton restarts and re-creates the Instance Actor hierarchy. Central's per-site GrpcSiteTransport flips to the surviving site node's gRPC endpoint. Ongoing debug streams are interrupted and must be re-established by the engineer.

Dependencies

• gRPC (Grpc.AspNetCore + Grpc.Net.Client): Provides the only cross-cluster transport — command/control (CentralControlService on central, SiteCommandService on sites), real-time streaming (SiteStreamService, site-hosted), and the audit-pull RPCs. Central hosts CentralControlService and dials each site's SiteCommandService (GrpcSiteTransport) + SiteStreamService; sites host SiteCommandService + SiteStreamGrpcServer and dial CentralControlService (GrpcCentralTransport).
• Akka.NET Remoting + Akka.Cluster.Tools: Remoting provides intra-cluster transport only. Akka.Cluster.Tools provides ClusterSingleton; ClusterClient/ClusterClientReceptionist are no longer used (removed in Phase 4).
• Cluster Infrastructure: Manages node roles and failover detection.
• Configuration Database: Provides site node gRPC endpoints (GrpcNodeAAddress, GrpcNodeBAddress — used for both command/control and streaming) for address resolution; the legacy Akka NodeAAddress/NodeBAddress fields are retained but no longer drive cross-cluster dialling.
• Site Runtime (SiteStreamManager): The SiteStreamGrpcServer subscribes to SiteStreamManager to receive real-time events for gRPC delivery.
• ISiteAuditQueue (site-local): Handed to SiteStreamGrpcServer (post-construction, on site roles) so the PullAuditEvents RPC can read the site's Pending/Forwarded audit rows to serve the Audit Log (#23) reconciliation pull. Null when not wired (central-only host) — the handler then returns an empty response.
• IOperationTrackingStore (site-local): Handed to SiteStreamGrpcServer (post-construction, on site roles) so the PullSiteCalls RPC can read operation-tracking rows changed since a cursor to serve the Site Call Audit (#22) reconciliation pull. Null when not wired — the handler returns an empty response.
• AuditLogIngestActor proxy (central): Handed to SiteStreamGrpcServer after the central cluster singleton starts; the IngestAuditEvents / IngestCachedTelemetry RPCs route ingested batches to it. Null when not yet wired — the handler returns an empty IngestAck so the caller treats it as transient and retries.

Interactions

• Deployment Manager (central): Uses communication to deliver configurations, lifecycle commands, and system-wide artifacts, and receive status.
• Site Runtime: Receives deployments, lifecycle commands, and artifact updates. Provides debug view data.
• Central UI: Debug view requests and remote queries flow through communication.
• Health Monitoring: Receives periodic health reports from sites.
• Store-and-Forward Engine (site): Parked message queries/commands are routed through communication. Also emits CachedCallTelemetry (push, gRPC to CentralControlService) and serves the PullSiteCalls gRPC reconciliation pull from its IOperationTrackingStore, and receives relayed RetryParkedOperation / DiscardParkedOperation commands.
• Site Call Audit (central): Receives cached-call telemetry and issues the PullSiteCalls gRPC reconciliation pulls to sites; relays parked-operation Retry/Discard commands to sites through communication.
• Audit Log (#23): Sites forward audit-event telemetry (push) and serve the PullAuditEvents gRPC reconciliation pull from their ISiteAuditQueue; the central AuditLogIngestActor is the ingest target for both the push path and the combined cached-call telemetry packet.
• Site Event Logging: Event log queries are routed through communication.
• Management Service: The ManagementActor runs at /user/management on central nodes and is reached in-process via ManagementActorHolder. The CLI communicates with it over the central HTTP /management endpoints (HTTP Basic), not over any cluster transport. It is no longer registered with ClusterClientReceptionist (that registration, and the whole ClusterClient surface, is gone).