17 Commits

Author SHA1 Message Date
Joseph Doherty 9f91e84d83 docs(grpc): Phase 5 live gate — PASS (8/8), migration complete
Full eight-check gate on the Phase 4 deletion build (main @ 7fd5cb2b),
rig rebuilt from main. All 8 checks PASS: PSK negatives; site->central
matrix (notif no-loss/dupe, both audit paths, reconcile self-heal);
central->site matrix (Query/Parked/Lifecycle); active-central kill ->
sticky flip 1s + central-b active 26s; mid-drain total==distinct;
297,574-byte gRPC reply (128KB frame-class retired); cluster membership
pair-only (no cross-boundary Akka association); full-rig restart 0
receptionist/ClusterClient lines on any of 8 nodes.

Check-1 clarification recorded: site-side interceptor uses the node's
single GrpcPsk and ignores x-scadabridge-site (a central-side routing
hint) -- per-site isolation proven by wrong-key reject. Instance-
dependent central->site RPCs (OpcUa/Route/standby-parked/TriggerFailover)
carried forward, unit-proven.
2026-07-23 13:35:16 -04:00
Joseph Doherty 7fd5cb2b56 feat(comm): Phase 4 — delete Akka ClusterClient site↔central transport, gRPC-only
ClusterClient→gRPC migration Phase 4 (docs/plans/2026-07-22-clusterclient-to-grpc-plan.md).
Phases 2/3 proved both directions on gRPC; this removes the Akka transport underneath.

Deleted:
- AkkaCentralTransport, AkkaSiteTransport (+ their dedicated tests)
- ISiteClientFactory + DefaultSiteClientFactory; CentralCommunicationActor legacy
  ctor + SelectTransport (Host now builds GrpcSiteTransport and injects it)
- ClusterClient creation + both ClusterClientReceptionist.RegisterService calls in
  AkkaHostedService; the RegisterCentralClient message + receive block
- CommunicationOptions.CentralContactPoints; the CentralTransport/SiteTransport
  coexistence flags; the CentralTransportMode/SiteTransportKind enums

gRPC is now the only site↔central transport (site→central CentralControlService via
GrpcCentralTransport; central→site SiteCommandService via GrpcSiteTransport), both
built unconditionally by the Host. NoOpCentralTransport is the fail-loud null-default
so TestKit command-dispatch suites still construct the site actor without a wired
transport; production always injects GrpcCentralTransport.

Config: CentralGrpcEndpoints is now unconditional — CommunicationOptionsValidator
rejects blank entries (role-agnostic), and StartupValidator requires a Site node to
list >=1 endpoint (fail-fast, mirrors GrpcPsk). Rig configs moved
CentralContactPoints -> CentralGrpcEndpoints (docker x6, docker-env2 x2, Host default,
deploy/wonder-app-vd03). Kept Akka.Cluster.Tools (ClusterSingleton still used).

Tests: build 0/0; Communication.Tests 640, Host.Tests 421 green. Removed the
ClusterClient.Send per-site-routing tests (covered by the transport suites), swapped
the ISiteClientFactory-based ctors to a substitute ISiteCommandTransport, converted
the audit-push integration relay to an in-process bridge transport.

Docs: Component-Communication/Host/StoreAndForward, components/Communication,
topology-guide, grpc_streams (SUPERSEDED note), the frame-size known-issue (retired
amendment), and CLAUDE.md transport decisions.

Not included: the dead IntegrationCallRequest path (#32) is a separate user-owned
behavioral decision — SiteEnvelope routing is transport-agnostic so it still compiles.
2026-07-23 12:54:32 -04:00
Joseph Doherty 3a8ddb7087 docs(known-issues): link IntegrationCallRequest dead-code note to Gitea #32 2026-07-23 11:52:52 -04:00
Joseph Doherty 54da10dc00 docs(grpc): Phase 3 live gate — PASS; central→site command cutover
Central flipped to SiteTransport=Grpc (both nodes, central-wide flag), sites
kept on gRPC from Phase 2 → both directions on gRPC at once. Central logs
'central→site command transport: gRPC (SiteCommandService)'; 0 ClusterClient-to-
site on either central.

Command matrix over SiteCommandService (all 200):
- ExecuteQuery (event-log) + ExecuteParked (parked) — all 3 sites
- ExecuteLifecycle disable/enable #95 on site-a — NEW live coverage vs 1B/P2
  (SoakNotify instances survived the recreate as Enabled)

Resilience:
- hard-kill the ACTIVE site-a node mid-query-loop → in-flight call returned
  TIMEOUT at exactly the 30s QueryTimeout deadline (bounded, no hang; deadline≠
  retry — an in-flight call can't be safely re-sent)
- next + all subsequent queries recovered automatically via site-a-b
  (SitePairChannelProvider NodeA→NodeB); site→central S&F never stopped
  (notif 619→715, still no dupes)
- 0 PermissionDenied across all 8 nodes

ExecuteOpcUa/ExecuteRoute/TriggerFailover deferred (no OPC-bound instance / no
CLI verb; unit-proven). Rig left both-gRPC; git reverted to Akka default.
2026-07-23 11:45:12 -04:00
Joseph Doherty a54602c14a docs(grpc): Phase 2 live gate — PASS; site→central S&F cutover soak
All 6 site nodes flipped to CentralTransport=Grpc; whole control plane
(heartbeat/health/notification S&F/audit) rides gRPC CentralControlService,
196 RPCs/90s 0 non-200, 0 PermissionDenied, 0 health-sequence regressions.

Soak driven by a live SoakNotify workload (3 instances, 3 notifs/5s):
- single-node active-kill (central-a): sticky failover central-a:8083→b:8083
  logged instantly, central-b active in 29s, buffer drained 72→101, every 5s
  bucket = exactly 3 through the gap, 101==101 distinct
- failback: central-a rejoined ready ~5s as standby, central-b kept active
  (oldest-Up, no flap), traffic uninterrupted
- full outage (both central down ~59s): count frozen, cold re-form central-b
  active ~14s, ~42 buffered drained, every 5s bucket = exactly 3 across the
  whole dead window, 216==216 distinct — zero loss, zero dupes

Also previews Phase 5 checks 4 (failover/failback) and 5 (mid-drain kill).
Rig config reverted to Akka default (defaults stay Akka until Phase 4).
2026-07-23 11:31:09 -04:00
Joseph Doherty 2fa5e93c73 docs(plans): tick 1B DoD (proportionate rig gate PASS) 2026-07-22 22:38:38 -04:00
Joseph Doherty 01693b13db docs(grpc): Phase 1B live gate — PASS (proportionate); central→site rides authenticated gRPC for 3 sites; records the central-wide SiteTransport finding 2026-07-22 22:38:21 -04:00
Joseph Doherty 86ad4d5c8e feat(comm): T1B.3 — central-side gRPC site-command transport seam (default Akka)
Extract the central→site send path in CentralCommunicationActor behind a new
ISiteCommandTransport, selected by ScadaBridge:Communication:SiteTransport
(Akka | Grpc, default Akka — rollback = flip the flag). CommunicationService's
27 commands, SiteCallAuditActor's 2 parked relays and DebugStreamBridgeActor's
subscribe/unsubscribe are untouched; the seam sits below SiteEnvelope.

- AkkaSiteTransport: today's per-site ClusterClient path extracted verbatim
  (the _siteClients lookup + ClusterClient.Send with the reply-to sender
  preserved, and the "no client ⇒ warn + drop, caller's Ask times out" path).
- GrpcSiteTransport: dials the site SiteCommandService (T1B.1 proto client) via
  SiteCommandDtoMapper, PSK + x-scadabridge-site on the channel through
  ControlPlaneCredentials, per-command deadlines set EQUAL to today's
  CommunicationService Ask timeouts (per-command, not per-group: DeploymentState
  query and TriggerSiteFailover use QueryTimeout; the two parked relays map to
  QueryTimeout so SiteCallAudit's inner RelayTimeout 10s < 30s ordering holds;
  WaitForAttribute keeps its dynamic Timeout + IntegrationTimeout).
- SitePairChannelProvider: per-site A/B channel pair with sticky failover
  (flip only on Unavailable — NEVER on DeadlineExceeded, a write/deploy/failover
  may have run), background failback probe to the preferred node with 1s→60s
  doubling backoff, PSK invalidation on site removal. Fed by the SAME DB refresh
  loop (extended to carry GrpcNodeA/GrpcNodeBAddress) — no second poll.

Tests: actor-with-substitute-transport (routing, Ask-reply plumbing, per-site
lifecycle across refreshes), ResolveDeadline pinned to each command's current
Ask timeout, and GrpcSiteTransport/SitePairChannelProvider over dual in-process
TestServers (PSK+header+deadline attached, Unavailable failover + stickiness,
failback to preferred, no-retry-on-DeadlineExceeded). Proto csproj untouched
(no active <Protobuf> item). Full solution builds 0 warnings; Communication.Tests
607 green with Akka default.
2026-07-22 20:09:43 -04:00
Joseph Doherty 518c699b90 feat(comm): extract SiteCommandDispatcher; site serves commands over gRPC too (T1B.2)
Refactor SiteCommunicationActor's central→site routing table into one
SiteCommandDispatcher — the single routing truth for the 28 migrated commands
(IntegrationCallRequest, the dead 29th, stays on the actor and out of the
dispatcher). The Akka actor and the new SiteCommandGrpcService both route through
one dispatcher instance so the two transports can never drift on where a command
goes. Server-side only: nothing central flips to gRPC yet (that is T1B.3);
ClusterClient remains the live path.

Decisions worth recording:

- Targets preserved byte-for-byte. Lifecycle/OPC UA/query/route → the Deployment
  Manager singleton proxy; DeployArtifacts/EventLog/parked → their null-guarded
  handlers with the exact same "handler not available" replies; the parked
  handler stays NODE-LOCAL (per-node replicated-store owner), never the singleton
  proxy — pinned by a dispatcher test that asserts the target is the parked probe
  and NOT the dm proxy.

- Sender preservation intact. The actor's command handlers became thin
  DispatchCommand delegations that still Forward (central Ask → reply routes
  straight back); the existing SiteCommunicationActorTests pass unchanged, which
  is the regression guard for that plumbing. UnsubscribeDebugView keeps its
  fire-and-forget shape: the actor Forwards, the gRPC service Tells + returns the
  synthetic UnsubscribeDebugViewAck so a unary RPC still answers.

- Ack-before-Leave on failover. The dispatcher's PrepareFailover resolves the
  standby with a DRY-RUN (no leave) to build the ack, and hands back a deferred
  CommitLeave; the gRPC service returns the ack, then schedules the real
  Cluster.Leave — so a caller reaching the very node about to leave still gets its
  ack instead of a broken stream. The actor path keeps today's coupled
  resolve-and-leave (over ClusterClient the ack Tell only enqueues, so order is
  immaterial). Proven at both levels: a dispatcher test asserts the ack is built
  before CommitLeave runs, and a TestServer test asserts the recorded seam order
  is resolve-then-leave.

- ControlPlaneAuthInterceptor gates SiteCommandService by EXTENDING
  DefaultGatedPrefixes (descriptor-derived), not by adding a constructor — the
  one-public-ctor invariant and its test stay green.

Tests: SiteCommandDispatcherTests (28-command routing incl. parked node-locality
and both failover paths) and SiteCommandGrpcService TestServer tests (auth,
readiness→Unavailable, one command per oneof group, failover ordering). Full
solution build 0/0; Communication.Tests 574 and Host.Tests 377 green. No active
<Protobuf> item.
2026-07-22 20:07:58 -04:00
Joseph Doherty 59b13d317b feat(grpc): T1B.1 — site_command.proto + SiteCommandDtoMapper + round-trip goldens
Phase 1B's contract slice: the wire shape and the canonical translation for the
28 central→site commands that leave ClusterClient. No behaviour changes yet —
SiteCommunicationActor and CentralCommunicationActor are untouched; the
dispatcher refactor (T1B.2) and the central transport seam (T1B.3) consume this.

Protos/site_command.proto (package scadabridge.sitecommand.v1, service
SiteCommandService): six domain RPCs, each with a `oneof` request/reply
envelope. The grouping is what carries deadline policy — every command inside a
group shares a CommunicationOptions timeout class today, so one RPC per group
keeps the deadline choice in one place on the client and one dispatch switch on
the server, while the oneof keeps each command individually typed:
ExecuteLifecycle(6) · ExecuteOpcUa(8) · ExecuteQuery(4) · ExecuteParked(5) ·
ExecuteRoute(4) · TriggerFailover(1). IntegrationCallRequest — the 29th entry on
SiteCommunicationActor's receive table — is deliberately excluded as dead code
(2026-07-22-integration-call-routing-is-dead-code.md).

Contract decisions worth knowing:

- Nullable COLLECTIONS ride in per-collection wrapper messages
  (DeployArtifactsCommand's six artifact lists, CertTrustResult.Certs,
  RouteToCallRequest.Parameters). proto3 repeated/map collapses null into empty,
  and that distinction is live at the site — the same silent-data-loss class the
  transport round-trip guard exposed in PLAN-05 T8. Goldens cover null, empty
  and populated for each.
- Nullable strings use the empty-string-means-null convention already set by
  AuditEventDtoMapper, with ONE exception: RouteToWaitForAttributeRequest's
  TargetValueEncoded, where "wait for the empty string" is a real target, so it
  carries a StringValue wrapper. Both behaviours are asserted, not assumed.
- Nullable enums ride in one-field messages (proto3 enums have no presence and
  no stock wrapper). Enum translation is an explicit switch in both directions —
  never by ordinal — so reordering a C# enum cannot re-map the wire; every wire
  enum reserves 0 for _UNSPECIFIED and decodes to a documented safe default
  rather than faulting a command from a version-skewed peer.
- New LooseValueCodec carries the surviving `object?` members (script params and
  return values, attribute values, tag read/write values) as a type-tagged union
  so a boxed value keeps its runtime CLR type, as it does today under Akka's
  type-preserving JSON serializer. Dates ride as invariant round-trip strings,
  not Timestamp, which would silently normalise away DateTime.Kind and
  DateTimeOffset.Offset. Lists/maps recurse; anything outside the tagged set
  falls back to JSON and is documented as CLR-type-lossy.
- DebugViewSnapshot gets its own full-fidelity alarm/attribute messages rather
  than reusing sitestream's AlarmStateUpdate, which flattens values to display
  strings — right for a live stream, lossy for a snapshot the UI treats as
  authoritative. The encoder omits an AlarmStateChanged.Condition that already
  equals the record's derived default, so computed alarms round-trip exactly
  (record equality compares the nullable backing field, not the property).

Tests are reflection-driven so the coverage cannot drift: the round-trip theory
enumerates the mapper's own ToProto overloads, the envelope guards enumerate the
generated oneof descriptors, and a missing golden fails the build. 216 new tests
green (Communication 532 total, Commons 684 total, solution build 0/0).

Codegen is checked in under SiteCommandGrpc/ per the sitestream recipe; the
<Protobuf> ItemGroup stays commented out (an active one segfaults protoc in the
linux_arm64 Docker image). docker/regen-proto.sh now handles every proto in that
ItemGroup instead of just sitestream, and re-comments idempotently.
2026-07-22 20:04:23 -04:00
Joseph Doherty aa60f43866 Merge Phase 1A: site→central control plane over gRPC (PR #26) 2026-07-22 20:01:42 -04:00
Joseph Doherty aa49a1d078 docs(plans): tick T1B.3/T1B.4 — central site-command transport seam complete (3be85f19) 2026-07-22 19:43:58 -04:00
Joseph Doherty 81ced76654 docs(plans): tick T1A.3/T1A.4 — site ICentralTransport seam complete (33b15f10) 2026-07-22 19:31:05 -04:00
Joseph Doherty 9f2c96f486 docs(plans): tick T1B.2 — SiteCommandDispatcher + site gRPC command service (cd6c20e1) 2026-07-22 19:12:45 -04:00
Joseph Doherty fc398b47d3 docs(plans): tick T1A.2 — central CentralControlService hosting + per-site auth (780bb9c3) 2026-07-22 18:55:28 -04:00
Joseph Doherty c90b353820 docs(plans): tick T1B.1 — site_command.proto + mapper + 194 goldens (a7481174) 2026-07-22 18:41:22 -04:00
Joseph Doherty c615ba5f78 docs(plans): tick T1A.1 — central_control.proto + mapper + 32 goldens (d7455577) 2026-07-22 18:31:18 -04:00
67 changed files with 41111 additions and 1462 deletions
+2 -2
View File
@@ -134,7 +134,7 @@ Other peers in the `scadaproj` family (see `scadaproj/CLAUDE.md` for details): `
- **Operational constraints (read before upgrading a site pair):** stop and start both nodes TOGETHER — rolling one at a time is no longer supported, since the legacy `SfBufferSnapshot` compatibility handler went with the replicator. And a node offline longer than `LocalDb:Replication:TombstoneRetention` (default 7 days) can resurrect deleted rows on rejoin. See `docs/deployment/topology-guide.md`.
- `LocalDb:Replication:MaxBatchSize` batches by ROW COUNT, not bytes, against a 4 MB gRPC cap — the rig pins it to **16** (~70 KB worst-case `config_json` x 16 ~= 1.1 MB). The 500 default would allow ~35 MB.
- All timestamps are UTC throughout the system.
- Inter-cluster communication uses **three** transports, not two: **ClusterClient** for command/control (deployments, lifecycle, subscribe/unsubscribe handshake, snapshots); **gRPC** server-streaming for real-time data (attribute values, alarm states); and **plain token-gated HTTP** for the deployment config itself — notify-and-fetch, the site pulls the config from `DeploymentConfigEndpoints` (`ManagementService/DeploymentConfigEndpoints.cs`) with an `X-Deployment-Token` header, `AllowAnonymous` with the per-deployment token as the entire security boundary. Both CentralCommunicationActor and SiteCommunicationActor registered with receptionist (**per node, not as a singleton** — contact rotation reaches whichever node answers). Central creates one ClusterClient per site using NodeA/NodeB as contact points. Sites configure multiple central contact points for failover. Addresses cached in CentralCommunicationActor, refreshed periodically (60s) and on admin changes. Heartbeats serve health monitoring only. **Discovery is asymmetric by design:** central discovers sites from the *database* (`Site.NodeAAddress`/`NodeBAddress`, refreshable at runtime), sites discover central from *appsettings* (`ScadaBridge:Communication:CentralContactPoints`, static — restart required). **Central never buffers for an unreachable site** — the send is dropped with a warning and the caller's Ask times out; a `ConnectionStateChanged` mechanism built for this was deleted as dead code.
- Inter-cluster communication uses **three** transports, not two **all cross-cluster command/control and data now rides gRPC** after the ClusterClient→gRPC migration's Phase 4 (`docs/plans/2026-07-22-clusterclient-to-grpc-plan.md`) deleted Akka `ClusterClient`/`ClusterClientReceptionist`: (1) **gRPC command/control** — site→central over the central-hosted `CentralControlService` (`GrpcCentralTransport`, sticky central-a→central-b channel pair; deployments/notifications/health/heartbeat/audit-ingest/reconcile), and central→site over the site-hosted `SiteCommandService` (`GrpcSiteTransport`, per-site NodeA→NodeB channel pair; the 28 lifecycle/OPC-UA/query/parked/route/failover commands); (2) **gRPC** server-streaming for real-time data (attribute values, alarm states, `SiteStreamService`); and (3) **plain token-gated HTTP** for the deployment config itself — notify-and-fetch, the site pulls the config from `DeploymentConfigEndpoints` (`ManagementService/DeploymentConfigEndpoints.cs`) with an `X-Deployment-Token` header, `AllowAnonymous` with the per-deployment token as the entire security boundary. The gRPC boundary is per-site PSK-authenticated (`ControlPlaneAuthInterceptor`, unchanged). There is **no receptionist registration**discovery is by dialling configured endpoints; central builds one `SitePairChannelProvider` per site (addresses from `Site.GrpcNodeAAddress`/`GrpcNodeBAddress`, refreshed from the DB every 60s and on admin changes), sites dial `ScadaBridge:Communication:CentralGrpcEndpoints` (both central nodes, h2c on `CentralGrpcPort` 8083, **NOT** via Traefik). **Discovery is asymmetric by design:** central discovers site gRPC addresses from the *database* (refreshable at runtime), sites discover central from *appsettings* (`CentralGrpcEndpoints`, static — restart required; `StartupValidator` requires a Site node to list at least one). `Akka.Cluster.Tools` stays for ClusterSingleton; only the ClusterClient part is gone. **Central never buffers for an unreachable site** — the send fails with the caller's Ask/deadline timing out; a `ConnectionStateChanged` mechanism built for this was deleted as dead code.
- **All clusters share ONE ActorSystem name**, `"scadabridge"` — hardcoded at `AkkaHostedService.cs:191`. Central and each site are separate clusters *only* by seed-node partitioning. This is required, not incidental: Akka.Remote address matching means a ClusterClient could not reach a differently-named system.
- **`ActiveNodeEvaluator.SelfIsOldestUp` is THE single definition of "active node"** (`Communication/ClusterState/ActiveNodeEvaluator.cs`) — the **oldest Up member** in a role scope, and explicitly **never `cluster.State.Leader`**: leadership (lowest address) is an Akka-internal concept that diverges from singleton placement permanently once the original first node restarts and rejoins, and both sides claim it during a partition. The equivalence *oldest-Up == where `ClusterSingletonManager` places singletons* **is** the design. `ClusterActivityEvaluator.SelfIsOldest`, the S&F delivery gate, `/health/active` and the heartbeat `IsActive` stamp all delegate here.
- Site nodes carry **two Akka roles**: the base `Site` plus a site-specific `site-{SiteId}` (`BuildRoles`, `AkkaHostedService.cs:386`). Singletons scope to the **site-specific** role.
@@ -247,7 +247,7 @@ Other peers in the `scadaproj` family (see `scadaproj/CLAUDE.md` for details): `
### Akka.NET Conventions
- Tell for hot-path internal communication; Ask reserved for system boundaries.
- ClusterClient for cross-cluster communication; ClusterClientReceptionist for service discovery across cluster boundaries.
- Cross-cluster communication is gRPC (per-site PSK-authenticated): site→central `CentralControlService`, central→site `SiteCommandService`, plus the `SiteStreamService` data stream. ClusterClient/ClusterClientReceptionist were removed in the migration's Phase 4 — service discovery is by dialling configured endpoints, not the receptionist. (Akka.Cluster.Tools remains for ClusterSingleton.)
- Script trust model: forbidden APIs (System.IO, Process, Threading, Reflection, raw network). The trust boundary is centralized in the Script Analysis component (#25) — `ScriptTrustPolicy` is the single source of truth; all four call sites (Template Engine, Site Runtime, Inbound API, Central UI) delegate to `ScriptTrustValidator`. The design-time deploy gate in Template Engine is authoritative (real semantic compile), not advisory.
- Application-level correlation IDs on all request/response messages.
@@ -47,9 +47,9 @@
// node fails StartupValidator: the gate is fail-closed, so an unset key would
// refuse every SiteStream call while the node still looked healthy.
"GrpcPsk": "dev-grpc-psk-docker-env2-site-x",
"CentralContactPoints": [
"akka.tcp://scadabridge@scadabridge-env2-central-a:8081",
"akka.tcp://scadabridge@scadabridge-env2-central-b:8081"
"CentralGrpcEndpoints": [
"http://scadabridge-env2-central-a:8083",
"http://scadabridge-env2-central-b:8083"
],
"DeploymentTimeout": "00:02:00",
"LifecycleTimeout": "00:00:30",
@@ -47,9 +47,9 @@
// node fails StartupValidator: the gate is fail-closed, so an unset key would
// refuse every SiteStream call while the node still looked healthy.
"GrpcPsk": "dev-grpc-psk-docker-env2-site-x",
"CentralContactPoints": [
"akka.tcp://scadabridge@scadabridge-env2-central-a:8081",
"akka.tcp://scadabridge@scadabridge-env2-central-b:8081"
"CentralGrpcEndpoints": [
"http://scadabridge-env2-central-a:8083",
"http://scadabridge-env2-central-b:8083"
],
"DeploymentTimeout": "00:02:00",
"LifecycleTimeout": "00:00:30",
+20 -7
View File
@@ -3,14 +3,15 @@
# Regenerates the gRPC C# files from the Communication project's .proto files.
#
# Background: protoc (linux/arm64) segfaults inside our Docker build container
# (Grpc.Tools 2.71.0). As a workaround the generated C# is checked into
# src/ZB.MOM.WW.ScadaBridge.Communication/ — SiteStreamGrpc/ for sitestream.proto
# and CentralControlGrpc/ for central_control.proto and the Protobuf ItemGroup
# in the .csproj is commented out, so Docker just compiles the checked-in files.
# (Grpc.Tools). As a workaround the generated C# is checked into
# src/ZB.MOM.WW.ScadaBridge.Communication/ — SiteStreamGrpc/ for sitestream.proto,
# CentralControlGrpc/ for central_control.proto, and SiteCommandGrpc/ for
# site_command.proto — and the Protobuf ItemGroup in the .csproj is commented out,
# so Docker just compiles the checked-in files.
#
# Run this script ON YOUR DEV MACHINE whenever a .proto changes:
#
# docker/regen-proto.sh [sitestream|centralcontrol|all] (default: all)
# docker/regen-proto.sh [sitestream|centralcontrol|sitecommand|all] (default: all)
#
# 1. Injects a Protobuf ItemGroup for the selected proto(s) so Grpc.Tools runs.
# 2. Deletes the stale checked-in C# so a failed regen is obvious.
@@ -32,8 +33,8 @@ set -euo pipefail
TARGET="${1:-all}"
case "$TARGET" in
sitestream|centralcontrol|all) ;;
*) echo "usage: $0 [sitestream|centralcontrol|all]" >&2; exit 2 ;;
sitestream|centralcontrol|sitecommand|all) ;;
*) echo "usage: $0 [sitestream|centralcontrol|sitecommand|all]" >&2; exit 2 ;;
esac
SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)"
@@ -68,6 +69,8 @@ if target in ("sitestream", "all"):
protos.append("sitestream.proto")
if target in ("centralcontrol", "all"):
protos.append("central_control.proto")
if target in ("sitecommand", "all"):
protos.append("site_command.proto")
items = "\n".join(
f' <Protobuf Include="Protos\\{p}" GrpcServices="Both" />' for p in protos)
@@ -87,6 +90,10 @@ if [[ "$TARGET" == "centralcontrol" || "$TARGET" == "all" ]]; then
rm -f "$COMM_DIR/CentralControlGrpc/CentralControl.cs" \
"$COMM_DIR/CentralControlGrpc/CentralControlGrpc.cs"
fi
if [[ "$TARGET" == "sitecommand" || "$TARGET" == "all" ]]; then
rm -f "$COMM_DIR/SiteCommandGrpc/SiteCommand.cs" \
"$COMM_DIR/SiteCommandGrpc/SiteCommandGrpc.cs"
fi
# 3. Regenerate by building.
echo "Building Communication project (regen)..."
@@ -103,6 +110,11 @@ if [[ "$TARGET" == "centralcontrol" || "$TARGET" == "all" ]]; then
cp "$GEN/CentralControl.cs" "$GEN/CentralControlGrpc.cs" "$COMM_DIR/CentralControlGrpc/"
echo "Copied regenerated files to CentralControlGrpc/"
fi
if [[ "$TARGET" == "sitecommand" || "$TARGET" == "all" ]]; then
mkdir -p "$COMM_DIR/SiteCommandGrpc"
cp "$GEN/SiteCommand.cs" "$GEN/SiteCommandGrpc.cs" "$COMM_DIR/SiteCommandGrpc/"
echo "Copied regenerated files to SiteCommandGrpc/"
fi
# 5. Restore the backed-up csproj — i.e. drop the injected ItemGroup — so Docker
# builds keep working.
@@ -115,4 +127,5 @@ echo "Done. Review and commit:"
echo " git diff src/ZB.MOM.WW.ScadaBridge.Communication/Protos/"
echo " git diff src/ZB.MOM.WW.ScadaBridge.Communication/SiteStreamGrpc/"
echo " git diff src/ZB.MOM.WW.ScadaBridge.Communication/CentralControlGrpc/"
echo " git diff src/ZB.MOM.WW.ScadaBridge.Communication/SiteCommandGrpc/"
echo " git diff -- src/ZB.MOM.WW.ScadaBridge.Communication/*.csproj # must be EMPTY"
+3 -3
View File
@@ -48,9 +48,9 @@
// node fails StartupValidator: the gate is fail-closed, so an unset key would
// refuse every SiteStream call while the node still looked healthy.
"GrpcPsk": "dev-grpc-psk-docker-site-a",
"CentralContactPoints": [
"akka.tcp://scadabridge@scadabridge-central-a:8081",
"akka.tcp://scadabridge@scadabridge-central-b:8081"
"CentralGrpcEndpoints": [
"http://scadabridge-central-a:8083",
"http://scadabridge-central-b:8083"
],
"DeploymentTimeout": "00:02:00",
"LifecycleTimeout": "00:00:30",
+3 -3
View File
@@ -48,9 +48,9 @@
// node fails StartupValidator: the gate is fail-closed, so an unset key would
// refuse every SiteStream call while the node still looked healthy.
"GrpcPsk": "dev-grpc-psk-docker-site-a",
"CentralContactPoints": [
"akka.tcp://scadabridge@scadabridge-central-a:8081",
"akka.tcp://scadabridge@scadabridge-central-b:8081"
"CentralGrpcEndpoints": [
"http://scadabridge-central-a:8083",
"http://scadabridge-central-b:8083"
],
"DeploymentTimeout": "00:02:00",
"LifecycleTimeout": "00:00:30",
+3 -3
View File
@@ -48,9 +48,9 @@
// node fails StartupValidator: the gate is fail-closed, so an unset key would
// refuse every SiteStream call while the node still looked healthy.
"GrpcPsk": "dev-grpc-psk-docker-site-b",
"CentralContactPoints": [
"akka.tcp://scadabridge@scadabridge-central-a:8081",
"akka.tcp://scadabridge@scadabridge-central-b:8081"
"CentralGrpcEndpoints": [
"http://scadabridge-central-a:8083",
"http://scadabridge-central-b:8083"
],
"DeploymentTimeout": "00:02:00",
"LifecycleTimeout": "00:00:30",
+3 -3
View File
@@ -48,9 +48,9 @@
// node fails StartupValidator: the gate is fail-closed, so an unset key would
// refuse every SiteStream call while the node still looked healthy.
"GrpcPsk": "dev-grpc-psk-docker-site-b",
"CentralContactPoints": [
"akka.tcp://scadabridge@scadabridge-central-a:8081",
"akka.tcp://scadabridge@scadabridge-central-b:8081"
"CentralGrpcEndpoints": [
"http://scadabridge-central-a:8083",
"http://scadabridge-central-b:8083"
],
"DeploymentTimeout": "00:02:00",
"LifecycleTimeout": "00:00:30",
+3 -3
View File
@@ -48,9 +48,9 @@
// node fails StartupValidator: the gate is fail-closed, so an unset key would
// refuse every SiteStream call while the node still looked healthy.
"GrpcPsk": "dev-grpc-psk-docker-site-c",
"CentralContactPoints": [
"akka.tcp://scadabridge@scadabridge-central-a:8081",
"akka.tcp://scadabridge@scadabridge-central-b:8081"
"CentralGrpcEndpoints": [
"http://scadabridge-central-a:8083",
"http://scadabridge-central-b:8083"
],
"DeploymentTimeout": "00:02:00",
"LifecycleTimeout": "00:00:30",
+3 -3
View File
@@ -48,9 +48,9 @@
// node fails StartupValidator: the gate is fail-closed, so an unset key would
// refuse every SiteStream call while the node still looked healthy.
"GrpcPsk": "dev-grpc-psk-docker-site-c",
"CentralContactPoints": [
"akka.tcp://scadabridge@scadabridge-central-a:8081",
"akka.tcp://scadabridge@scadabridge-central-b:8081"
"CentralGrpcEndpoints": [
"http://scadabridge-central-a:8083",
"http://scadabridge-central-b:8083"
],
"DeploymentTimeout": "00:02:00",
"LifecycleTimeout": "00:00:30",
+20 -17
View File
@@ -1,6 +1,6 @@
# CentralSite Communication
The CentralSite Communication component is the transport layer that connects the central cluster to every site cluster. It provides three independent transports — Akka.NET `ClusterClient` for command/control, gRPC server-streaming for real-time data, and plain token-gated HTTP for the deployment-config fetch — anchored by a pair of actors that each cluster registers with the `ClusterClientReceptionist`.
The CentralSite Communication component is the transport layer that connects the central cluster to every site cluster. It provides three independent transports, **all now gRPC-based across the boundary** (Akka `ClusterClient` was removed in Phase 4 of the ClusterClient→gRPC migration, 2026-07-23): gRPC command/control in both directions (`CentralControlService` on central via `GrpcCentralTransport`; `SiteCommandService` on the site via `GrpcSiteTransport`), gRPC server-streaming for real-time data (`SiteStreamService`), and plain token-gated HTTP for the deployment-config fetch. Cross-cluster Akka remoting and `ClusterClientReceptionist` are gone; Akka remoting is now intra-cluster only.
## Overview
@@ -22,25 +22,26 @@ DI registration is called from the Host composition root via `AddCommunication`.
| Transport | Who dials | Data direction | Purpose |
|-----------|-----------|----------------|---------|
| Akka.NET `ClusterClient` | both (central → site per site; site → central) | bidirectional | Deploy notifies, lifecycle, subscribe/unsubscribe handshake, snapshots, heartbeats, health reports, telemetry, notifications |
| gRPC command/control (`CentralControlService`) | **site dials central** | site → central | Heartbeats, health reports, notification submit/status, audit + cached-telemetry ingest, reconcile |
| gRPC command/control (`SiteCommandService`) | **central dials the site** | central → site | Deploy notifies, lifecycle, OPC UA, remote queries, subscribe/unsubscribe handshake, snapshots, parked, route, failover |
| gRPC (`SiteStreamService`) | **central dials the site** | mostly site → central | Real-time attribute value and alarm state changes (server-streaming), plus the audit ingest/pull unary RPCs |
| HTTP `GET` (`/api/internal/deployments/{id}/config`) | **site dials central** | central → site | The flattened deployment config itself (notify-and-fetch), gated by a per-deployment `X-Deployment-Token` |
The transports are independent. A gRPC stream interruption does not affect in-flight `ClusterClient` commands, and vice versa.
The transports are independent. A gRPC stream interruption does not affect in-flight command/control calls, and vice versa.
**The gRPC dial direction is inverted from its data direction.** Values flow site → central, but each **site node hosts the gRPC server** and **central is the client**. `MapGrpcService<SiteStreamGrpcServer>()` appears exactly once in the tree, inside the Site branch of `Program.cs` (`Host/Program.cs:542`); there is **no gRPC server on a central node at all**. That is why the two `Ingest*` unary RPCs — nominally a central-side ingest surface — are dead in the shipped topology (acknowledged in `AkkaHostedService.cs:505-508`: "when the gRPC server is not registered (current central topology)"); sites push audit telemetry to central over `ClusterClient` instead, and central pulls with `PullAuditEvents` / `PullSiteCalls` by dialling the site.
**`SiteStreamService`'s gRPC dial direction is inverted from its data direction.** Values flow site → central, but each **site node hosts the `SiteStreamService` server** and **central is the client**. Command/control is different: it runs on its own two services — `CentralControlService` (central-hosted, site dials in) and `SiteCommandService` (site-hosted, central dials in). So a central node now DOES host a gRPC server (`CentralControlService`), unlike before Phase 4. The two legacy `Ingest*` unary RPCs on the site-hosted `SiteStreamService` remain dead in the shipped topology (no site dials a site for ingest); audit telemetry is pushed site → central over `CentralControlService`, and central pulls with `PullAuditEvents` / `PullSiteCalls` by dialling the site's `SiteStreamService`.
**No transport carries transport encryption; two of the three now carry authentication.**
- **Akka remoting / `ClusterClient` — unauthenticated.** `BuildHocon` emits no `enable-ssl`, no secure cookie and no `trusted-selection-paths`, so the command/control path is plaintext and open to anything that can reach the remoting port.
- **gRPC — authenticated by preshared key since 2026-07-22.** The listener is still **h2c**`ListenAnyIP(grpcPort, o => o.Protocols = HttpProtocols.Http2)` with no `UseHttps` — but `ControlPlaneAuthInterceptor` gates every method under `/sitestream.SiteStreamService/`, including the `PullAuditEvents` / `PullSiteCalls` RPCs that return audit rows. It is fail-closed (no key ⇒ everything refused, and `StartupValidator` will not boot a site node in that state), compares with `CryptographicOperations.FixedTimeEquals`, and rejects with `PermissionDenied`. Central attaches the key via `ControlPlaneCredentials`, which binds `CallCredentials` to each channel so unary and streaming calls are covered uniformly`SiteStreamGrpcClient`, `GrpcPullAuditEventsInvoker` and `GrpcPullSiteCallsInvoker` all build their channels through it. Keys are **per site** (`SB-GRPC-PSK-<siteId>`), so a compromised site yields only its own. `LocalDbSyncAuthInterceptor` shares the listener and keeps its own separate key on `/localdb_sync.v1.LocalDbSync/` — the two authenticate different peers (central vs. the pair partner) and are never shared.
- **Akka remoting — unauthenticated, but intra-cluster only.** `BuildHocon` emits no `enable-ssl`, no secure cookie and no `trusted-selection-paths`, so remoting is plaintext and open to anything that can reach the remoting port — but as of Phase 4 remoting no longer crosses the site↔central boundary, so this exposure is pair-internal.
- **gRPC — authenticated by preshared key since 2026-07-22.** The listeners are still **h2c**`ListenAnyIP(grpcPort, o => o.Protocols = HttpProtocols.Http2)` with no `UseHttps` — but `ControlPlaneAuthInterceptor` gates the command/control and streaming services alike: every method under `/sitestream.SiteStreamService/` (including the `PullAuditEvents` / `PullSiteCalls` RPCs that return audit rows) **and** the Phase-4 `CentralControlService` (site→central) / `SiteCommandService` (central→site). It is fail-closed (no key ⇒ everything refused, and `StartupValidator` will not boot a site node in that state), compares with `CryptographicOperations.FixedTimeEquals`, and rejects with `PermissionDenied`. The caller attaches the key via `ControlPlaneCredentials`, which binds `CallCredentials` to each channel so unary and streaming calls are covered uniformly. Keys are **per site** (`SB-GRPC-PSK-<siteId>`), so a compromised site yields only its own. `LocalDbSyncAuthInterceptor` shares the listener and keeps its own separate key on `/localdb_sync.v1.LocalDbSync/` — the two authenticate different peers (central vs. the pair partner) and are never shared.
- **HTTP config fetch — token-authenticated.** Its per-deployment token is the *entire* security boundary (the endpoint is `AllowAnonymous`).
A bearer PSK over plaintext h2c is readable and replayable by anyone on the path, so the design still assumes a trusted network between central and sites — but the bar is now "read the traffic" rather than "reach the port". TLS on these listeners is the follow-on hardening and would not change the key design. Operational detail: [`docs/deployment/topology-guide.md`](../deployment/topology-guide.md).
### Notify-and-fetch: the deployment-config HTTP path
An instance deployment does not carry its flattened configuration inside the Akka message. Central stages a `PendingDeployment` row (config JSON + a freshly generated `DeploymentFetchToken` + a TTL) and sends only a small `RefreshDeploymentCommand` over `ClusterClient`, carrying the deployment id, revision hash, `CentralFetchBaseUrl` and the fetch token. The site's Deployment Manager singleton then calls back to central over plain HTTP:
An instance deployment does not carry its flattened configuration inside the command message. Central stages a `PendingDeployment` row (config JSON + a freshly generated `DeploymentFetchToken` + a TTL) and sends only a small `RefreshDeploymentCommand` over gRPC command/control (`SiteCommandService`), carrying the deployment id, revision hash, `CentralFetchBaseUrl` and the fetch token. The site's Deployment Manager singleton then calls back to central over plain HTTP:
```csharp
// SiteRuntime/Deployment/HttpDeploymentConfigFetcher.cs
@@ -49,11 +50,11 @@ using var req = new HttpRequestMessage(HttpMethod.Get, url);
req.Headers.Add("X-Deployment-Token", token);
```
`DeploymentConfigEndpoints.Resolve` (`ManagementService/DeploymentConfigEndpoints.cs:101`) checks existence and TTL *before* the token, so unknown, superseded and expired deployments are all indistinguishable `404`s; a live row with a wrong or missing token is `401`. The token comparison is constant-time. This exists because a flattened config can exceed the default 128 KB Akka frame size, which drops the single oversized message without tearing down the association — heartbeats keep flowing, the site still reports healthy, and the deploy just hangs to its Ask timeout. See `docs/known-issues/2026-06-26-deploy-config-exceeds-akka-frame-size.md`. `DeployArtifactsCommand` was **not** moved to this path and still carries its payload inline.
`DeploymentConfigEndpoints.Resolve` (`ManagementService/DeploymentConfigEndpoints.cs:101`) checks existence and TTL *before* the token, so unknown, superseded and expired deployments are all indistinguishable `404`s; a live row with a wrong or missing token is `401`. The token comparison is constant-time. This path was originally introduced because a flattened config could exceed the default 128 KB Akka frame size when command/control rode ClusterClient — the oversized message was silently dropped and the deploy hung to its timeout (see `docs/known-issues/2026-06-26-deploy-config-exceeds-akka-frame-size.md`). That Akka frame limit no longer applies now that command/control rides gRPC (`SiteCommandService`, 4 MB cap), but notify-and-fetch remains the deploy path. `DeployArtifactsCommand` was **not** moved to this path and still carries its payload inline — now over gRPC, so it is no longer at risk of the 128 KB drop.
### Hub-and-spoke topology
Sites do not communicate with each other. All inter-cluster traffic flows through central. Central maintains one `ClusterClient` per site; each site maintains a single `ClusterClient` pointed at both central nodes.
Sites do not communicate with each other. All inter-cluster traffic flows through central. Central maintains one `GrpcSiteTransport` channel pair per site (dialling `SiteCommandService`); each site maintains a single `GrpcCentralTransport` channel pair pointed at both central nodes (dialling `CentralControlService`).
### `SiteEnvelope` routing
@@ -94,12 +95,14 @@ If a site is unreachable when a command arrives, the caller's Ask times out. Cen
## Architecture
> **Phase 4 note (2026-07-23).** The `ClusterClient` mechanics described in this section — `SiteEnvelope` routing via `ClusterClient.Send("/user/site-communication", …)`, one `ClusterClient` per site on `CentralCommunicationActor`, the site's outbound `_centralClient`, and `ClusterClientReceptionist` registration — were **replaced by gRPC command/control** in Phase 4 of the ClusterClient→gRPC migration. Central→site commands now go over `GrpcSiteTransport` → the site-hosted `SiteCommandService` (per-site NodeA→NodeB failover channel pair), and site→central messages over `GrpcCentralTransport` → the central-hosted `CentralControlService` (sticky central-a→central-b pair). Endpoints are dialled directly (no receptionist, no cross-cluster actor discovery). The per-site address load below still runs, but it now feeds a **per-site gRPC-endpoint cache** (from `GrpcNodeAAddress`/`GrpcNodeBAddress`) consumed by a `SitePairChannelProvider`, not a `ClusterClient` contact set. See [Component-Communication.md](../requirements/Component-Communication.md) for the current design.
### Central-side: `CentralCommunicationActor`
`CentralCommunicationActor` is a `ReceiveActor` created at `/user/central-communication` and registered with `ClusterClientReceptionist` so the site's `ClusterClient` can locate it. It owns:
- A `Dictionary<string, (IActorRef Client, ImmutableHashSet<string> ContactAddresses)>` keyed by site identifier — one `ClusterClient` per site.
- A `RefreshSiteAddresses` periodic timer (60-second cadence, starting immediately). Each tick fires `LoadSiteAddressesFromDb`, which reads every `Site` row from the database, parses `NodeAAddress` and `NodeBAddress` into Akka receptionist paths (`{addr}/system/receptionist`), and pipes a `SiteAddressCacheLoaded` message back to Self. `HandleSiteAddressCacheLoaded` creates, updates, or stops `ClusterClient` actors based on the diff.
- A `RefreshSiteAddresses` periodic timer (60-second cadence, starting immediately). Each tick fires `LoadSiteAddressesFromDb`, which reads every `Site` row from the database and (as of Phase 4) builds a **per-site gRPC-endpoint cache** from `GrpcNodeAAddress`/`GrpcNodeBAddress`, piping a `SiteAddressCacheLoaded` message back to Self. `HandleSiteAddressCacheLoaded` reconciles the diff, updating the `SitePairChannelProvider` that `GrpcSiteTransport` uses to dial each site's `SiteCommandService`. (Before Phase 4 this parsed the Akka `NodeAAddress`/`NodeBAddress` into receptionist paths and created/updated/stopped a `ClusterClient` per site.)
- Proxy references to `NotificationOutboxActor` and `AuditLogIngestActor` cluster singletons, injected post-construction via `RegisterNotificationOutbox` / `RegisterAuditIngest` messages from the Host. Messages that arrive before the proxy is registered are answered with a non-accepted ack (notifications) or an empty reply (audit), so the site retries without data loss.
- Fanout of `SiteHealthReport` to the peer central node via `DistributedPubSub`, keyed on the `site-health-replica` topic, so both central nodes' aggregators stay in sync regardless of which central node the site's `ClusterClient` load-balanced the report to.
@@ -236,7 +239,7 @@ All options are bound from the `ScadaBridge:Communication` section via `Communic
| `IntegrationTimeout` | `00:00:30` | Ask timeout for integration routing and Inbound API routing. |
| `DebugViewTimeout` | `00:00:10` | Ask timeout for debug subscribe/unsubscribe handshake. |
| `NotificationForwardTimeout` | `00:00:30` | Ask timeout for notification submission forwarding. |
| `CentralContactPoints` | `[]` | Site-side: Akka addresses of central nodes, e.g. `akka.tcp://scadabridge@central-a:8081`. |
| `CentralGrpcEndpoints` | `[]` | Site-side: gRPC h2c endpoints of the central nodes' `CentralControlService`, e.g. `http://scadabridge-central-a:8083` (central's `CentralGrpcPort`, default 8083 — direct, not via Traefik, which is HTTP/1 only). **Required on Site nodes** (at least one); empty on Central nodes, which host `CentralControlService` and do not dial it. Replaces the former `CentralContactPoints` Akka-address list. |
| `GrpcKeepAlivePingDelay` | `00:00:15` | HTTP/2 keepalive PING interval on `SiteStreamGrpcClient`. |
| `GrpcKeepAlivePingTimeout` | `00:00:10` | HTTP/2 keepalive PING timeout. |
| `GrpcMaxStreamLifetime` | `04:00:00` | Per-stream session timeout; forces reconnect of zombie streams. |
@@ -259,27 +262,27 @@ Three layers of dead-client detection protect the gRPC stream path:
## Dependencies & Interactions
- [Commons (#16)](./Commons.md) — owns all message contracts used by this component: `DeployInstanceCommand`, `SiteEnvelope`, `HeartbeatMessage`, `SiteHealthReport`, `SiteHealthReportReplica`, `RegisterNotificationOutbox`, `RegisterAuditIngest`, `IngestAuditEventsCommand`, `IngestCachedTelemetryCommand`, and all other request/response records. Commons does not hold an Akka package reference, so `RegisterAuditIngest` (which carries an `IActorRef`) lives in this project.
- [Cluster Infrastructure (#13)](./ClusterInfrastructure.md) — provides `ClusterClientReceptionist` registration and the oldest-`Up` active/standby model that `SiteCommunicationActor`'s `IsActive` stamp depends on, plus the single `"scadabridge"` `ActorSystem` name that makes cross-cluster `ClusterClient` addressing possible at all. `CentralCommunicationActor`'s `DistributedPubSub` fanout keeps both central nodes in sync regardless of which one a site's report landed on.
- [Cluster Infrastructure (#13)](./ClusterInfrastructure.md) — provides ClusterSingleton and the oldest-`Up` active/standby model that `SiteCommunicationActor`'s `IsActive` stamp depends on, plus the single `"scadabridge"` `ActorSystem` name for intra-cluster remoting. (`ClusterClientReceptionist` is no longer used — cross-cluster messaging is gRPC as of Phase 4.) `CentralCommunicationActor`'s `DistributedPubSub` fanout keeps both central nodes in sync regardless of which one a site's report landed on.
- [Configuration Database (#17)](./ConfigurationDatabase.md) — provides `ISiteRepository.GetAllSitesAsync` for address loading; site records carry `NodeAAddress`, `NodeBAddress`, `GrpcNodeAAddress`, `GrpcNodeBAddress`.
- [Deployment Manager (#2)](./DeploymentManager.md) — the primary consumer of command/control patterns 13. `CommunicationService` is injected into the Deployment Manager actor to send deploy notifies, lifecycle commands, and artifact deployments to sites. It also owns the staging half of the notify-and-fetch HTTP path (`PendingDeployment` rows + fetch tokens); the endpoint itself is served by the Management Service.
- [Site Runtime (#3)](./SiteRuntime.md) — `SiteCommunicationActor` forwards inbound commands to the `DeploymentManager` singleton proxy. `SiteStreamManager` (in Site Runtime) implements `ISiteStreamSubscriber` so `SiteStreamGrpcServer` can subscribe relay actors to instance event feeds without referencing Site Runtime directly.
- [Health Monitoring (#11)](./HealthMonitoring.md) — `CentralCommunicationActor` calls `ICentralHealthAggregator.MarkHeartbeat` and `ProcessReport` for every inbound heartbeat and health report. `DistributedPubSub` fanout keeps both central nodes' aggregators in sync.
- [Audit Log (#23)](./AuditLog.md) — `SiteStreamGrpcServer` hosts the `IngestAuditEvents`, `IngestCachedTelemetry`, `PullAuditEvents` and `PullSiteCalls` RPCs. Because there is no central gRPC server, the `Ingest*` pair is unused in the shipped topology: sites push audit telemetry over ClusterClient, and `CentralCommunicationActor` routes `IngestAuditEventsCommand` / `IngestCachedTelemetryCommand` to the `AuditLogIngestActor` proxy. The `Pull*` reconciliation RPCs run the other way, with the central `site-audit-reconciliation` singleton dialling each site.
- [Audit Log (#23)](./AuditLog.md) — `SiteStreamGrpcServer` hosts the `IngestAuditEvents`, `IngestCachedTelemetry`, `PullAuditEvents` and `PullSiteCalls` RPCs. The `Ingest*` pair on the site-hosted service stays unused in the shipped topology; sites push audit telemetry site→central over gRPC to `CentralControlService`, which routes `IngestAuditEventsCommand` / `IngestCachedTelemetryCommand` to the `AuditLogIngestActor` proxy. The `Pull*` reconciliation RPCs run the other way, with the central `site-audit-reconciliation` singleton dialling each site's `SiteStreamService`.
- [Notification Outbox (#21)](./NotificationOutbox.md) — `CentralCommunicationActor` routes `NotificationSubmit` / `NotificationStatusQuery` messages from sites to the `NotificationOutboxActor` proxy. `CommunicationService` Asks the proxy directly for central-UI outbox management calls.
- [Site Call Audit (#22)](./SiteCallAudit.md) — `CommunicationService` Asks the `SiteCallAuditActor` proxy directly for query and relay operations. `SiteCallAuditActor` issues `RetryParkedOperation` / `DiscardParkedOperation` relay commands to sites via `SiteEnvelope`; `SiteCommunicationActor` dispatches them to `_parkedMessageHandler`.
- [Store-and-Forward Engine (#6)](./StoreAndForward.md) — the site S&F Engine drives `NotificationSubmit` forwarding and cached-call telemetry emission through `SiteCommunicationActor`. Parked-message queries and retry/discard relay commands flow back the other way.
- [Management Service (#18)](./ManagementService.md) — `ManagementActor` runs at `/user/management` on central and is reached **in-process** through `ManagementActorHolder`; the CLI connects over HTTP, not `ClusterClient`. (It was `ClusterClientReceptionist`-registered until 2026-07-22, for a CLI that was never built that way.) So this component's `ClusterClient` usage is exclusively the inter-cluster hub-and-spoke connections. Management Service also hosts `DeploymentConfigEndpoints` — the `GET /api/internal/deployments/{id}/config` route that terminates the third (HTTP) transport, mapped in the central-role block alongside `/api/audit/*` and `/management`.
- [Management Service (#18)](./ManagementService.md) — `ManagementActor` runs at `/user/management` on central and is reached **in-process** through `ManagementActorHolder`; the CLI connects over HTTP, not any cluster transport. (It was `ClusterClientReceptionist`-registered until 2026-07-22, for a CLI that was never built that way.) After Phase 4 this component uses no `ClusterClient` at all — its cross-cluster connections are the gRPC `CentralControlService` / `SiteCommandService` transports. Management Service also hosts `DeploymentConfigEndpoints` — the `GET /api/internal/deployments/{id}/config` route that terminates the HTTP deploy-config transport, mapped in the central-role block alongside `/api/audit/*` and `/management`.
- Design spec: [Component-Communication.md](../requirements/Component-Communication.md).
## Troubleshooting
### A site's commands fail immediately
Check that `NodeAAddress` and `NodeBAddress` are populated in the site configuration — if both are empty, `CentralCommunicationActor` logs a warning and skips that site on every refresh, so no `ClusterClient` is created and all commands timeout. `CommunicationService.RefreshSiteAddresses()` triggers an on-demand refresh after an address is added.
Check that `GrpcNodeAAddress` and `GrpcNodeBAddress` are populated in the site configuration — if both are empty, `CentralCommunicationActor` logs a warning and skips that site on every refresh, so no `SiteCommandService` channel is built and all commands timeout. `CommunicationService.RefreshSiteAddresses()` triggers an on-demand refresh after an address is added.
### Commands are timing out but the site is reachable
A single malformed address string for one site can silently prevent `ClusterClient` creation for that site while other sites are unaffected. Check the logs for a `Warning` line from `HandleSiteAddressCacheLoaded` naming the offending site. The actor parse-guard catches the `ActorPath.Parse` exception per-site so the rest of the refresh proceeds.
A single malformed gRPC endpoint string for one site can silently prevent channel creation for that site while other sites are unaffected. Check the logs for a `Warning` line from `HandleSiteAddressCacheLoaded` naming the offending site. The per-site parse-guard skips the bad entry so the rest of the refresh proceeds.
A `Warning` at the `Status.Failure` handler in `CentralCommunicationActor` means `LoadSiteAddressesFromDb` itself threw (typically a SQL connection error); the cache is left stale until the next successful refresh.
@@ -291,7 +294,7 @@ After a site node failover, the `DebugStreamBridgeActor` attempts to reconnect t
### Deployments fail immediately with a config-fetch error
The site received the `RefreshDeploymentCommand` over ClusterClient but could not complete the HTTP leg. Check `CentralFetchBaseUrl` first — it must be reachable *from the site*, so a value that only resolves inside the central network fails every deploy. A `404` from the fetch means the staged row was unknown, superseded, or past `PendingDeploymentTtl`; a `401` means the row is live but the token did not match. Because the endpoint hides existence, a `404` cannot distinguish "wrong id" from "expired".
The site received the `RefreshDeploymentCommand` over gRPC command/control (`SiteCommandService`) but could not complete the HTTP leg. Check `CentralFetchBaseUrl` first — it must be reachable *from the site*, so a value that only resolves inside the central network fails every deploy. A `404` from the fetch means the staged row was unknown, superseded, or past `PendingDeploymentTtl`; a `401` means the row is live but the token did not match. Because the endpoint hides existence, a `404` cannot distinguish "wrong id" from "expired".
### Heartbeats arrive but health reports do not
+15 -7
View File
@@ -178,14 +178,22 @@ database from its peer rather than letting it rejoin.
### Central-Site Communication
Three transports cross the boundary, not one:
Three transports cross the boundary, not one — **all now gRPC or HTTP; Akka ClusterClient was removed
in Phase 4 of the ClusterClient→gRPC migration (2026-07-23), and Akka remoting no longer crosses the
boundary at all:**
- **Akka ClusterClient** — command/control. Sites list every central node in
`ScadaBridge:Communication:CentralContactPoints`; contact rotation reaches whichever node
answers, so no "active central" needs to be identified. (There is no `Communication:CentralSeedNode`
setting — earlier revisions of this guide named one that never existed in the code.)
- **gRPC** — real-time data and audit pull. Note the direction is inverted from the data flow:
each **site node hosts the gRPC server** on `GrpcPort` (default 8083, h2c) and central dials in.
- **gRPC command/control** — both directions, on sticky-failover channel pairs, dialled directly (no
receptionist, no "active central" to identify — each side dials both of the peer's node endpoints):
- *Site → central* to the central-hosted **`CentralControlService`** (`GrpcCentralTransport`): the
site lists the central nodes' gRPC endpoints in `ScadaBridge:Communication:CentralGrpcEndpoints`
(e.g. `http://scadabridge-central-a:8083`, the central's `CentralGrpcPort`, default 8083 — direct
h2c, **not** via Traefik, which is HTTP/1 only). A Site node must list at least one; central nodes
leave it empty.
- *Central → site* to the site-hosted **`SiteCommandService`** (`GrpcSiteTransport`): central dials
the site's `GrpcNodeAAddress` / `GrpcNodeBAddress` (from the Site entity), NodeA→NodeB failover.
- **gRPC streaming + audit pull** — real-time data and audit/telemetry pull on the site-hosted
**`SiteStreamService`**. Note the direction is inverted from the data flow: each **site node hosts
the server** on `GrpcPort` (default 8083, h2c) and central dials in.
- **Plain HTTP** — the deploy config itself, fetched by the site with a per-deployment token.
#### gRPC control-plane preshared key (required)
@@ -31,6 +31,25 @@ Any deployment replicating wide rows must size that key deliberately; see the Ph
Note the failure mode differs from the one documented below: an oversized gRPC message is
**rejected**, not silently dropped.
## Amendment (2026-07-23) — Akka frame-size class RETIRED for site↔central command/control
Phase 4 of the ClusterClient→gRPC migration deleted the Akka `ClusterClient` site↔central transport
entirely. **All site↔central command/control now rides gRPC** — central→site over `SiteCommandService`
(`GrpcSiteTransport`) and site→central over `CentralControlService` (`GrpcCentralTransport`), each with
the gRPC default 4 MB message cap and **no per-message Akka frame limit**. Consequently:
- The 128 KB Akka `maximum-frame-size` constraint **no longer applies to any site↔central command
message**, including `DeployArtifactsCommand`, which still carries its payload inline but now travels
over gRPC.
- The **silent frame-drop failure mode** described below — the transport dropping one oversized message
while heartbeats keep flowing and the deploy hangs to its timeout — **cannot occur on that path any
more.** An over-cap gRPC message is *rejected* with an error, not silently discarded.
- The notify-and-fetch deploy path (the 2026-06-26 resolution) still stands and remains the deploy
mechanism; it is simply no longer the *only* thing keeping large payloads off a frame-limited hop.
- Akka remoting is now intra-cluster only, so its frame size governs only pair-internal traffic.
See `docs/plans/2026-07-22-clusterclient-to-grpc-plan.md`.
The diagnosis below is retained as the historical record of how the bug was found and reasoned about.
## Summary
@@ -1,7 +1,8 @@
# Integration call routing (`IntegrationCallRequest`) is dead on both ends
**Date:** 2026-07-22 · **Status:** OPEN (decision needed: wire or delete) · **Severity:** Low (no
runtime impact — the path cannot be reached) · **Area:** CentralSite Communication
**Date:** 2026-07-22 · **Status:** OPEN (decision needed: wire or delete) · **Tracked:** Gitea
[#32](https://gitea.dohertylan.com/dohertj2/ScadaBridge/issues/32) (filed 2026-07-23) · **Severity:**
Low (no runtime impact — the path cannot be reached) · **Area:** CentralSite Communication
## What
@@ -196,3 +196,284 @@ the one `ConfigureKestrel` call — `Program.ParseHttpBindPorts` + `CentralHttpB
for a fuller 1A proof if a deployed-instance rig is set up before then.
- Cross-node failover/failback of the site→central channel under a central-node kill (unit-proven
via TestServer; not exercised on the rig at 1A).
---
## Phase 1B — site command plane (central→site over gRPC) — **PASS (proportionate)** (2026-07-23)
Branch `feat/grpc-sitecommand` (rebased onto 1A-merged main). Rig rebuilt with **central
flipped to `SiteTransport=Grpc`** — a DoD-test-only edit to both central appsettings, reverted
from the branch (default stays `Akka`).
`SiteTransport` is a **central-wide** flag (`CentralCommunicationActor.SelectTransport` picks one
transport for all sites), so the plan's "flip for site-a only" is not achievable — the flip
routes central→site commands for **all three sites** to gRPC. Command-plane per-site coexistence
therefore cannot be shown (unlike the site→central plane in 1A, which is per-site). This is a
plan-vs-code finding, recorded rather than worked around.
### Checks
| # | Check | Result |
|---|---|---|
| 1 | central→site commands ride authenticated gRPC `SiteCommandService` | **PASS**`ExecuteQuery` (event-log) and `ExecuteParked` (parked query) HTTP/2 → 200 |
| 2 | Per-site PSK resolution across all sites | **PASS** — site-a, site-b, site-c each answered `ExecuteQuery` → 200 under its own `SB-GRPC-PSK-{site}`; **0** auth failures on any site node |
| 3 | Central HTTP surface intact under the central-wide gRPC flip | **PASS** — central `:5000` **and** `:8083` both listening; 9001 ready `200`, LB `200` (the 1A Kestrel fix carried through the merge) |
| 4 | Query round-trips return correct data | **PASS** — every `health event-log`/`parked-messages` returned `success:true` with the right `siteId` and empty result sets (bare rig) |
### Not driven on this proportionate gate
- **`TriggerSiteFailover`** — unit-proven (two ordering tests pin ack-before-`Leave` via the
dispatcher's dry-run resolve + deferred `CommitLeave`), but not live-driven here: it is
destructive (forces the active node to leave) and has no CLI verb (UI/management-only).
- **Tag commands (`BrowseNode`/`ReadTagValues`/`WriteTag`) and the lifecycle enable/disable
matrix** — need a deployed instance + data connection the bare rig lacks (same blocker as 1A).
- **Parked retry against the STANDBY node** — needs a parked operation to exist, which needs a
deployed instance.
- **Command-plane coexistence (site-b/c on Akka while site-a on gRPC)** — not expressible; the
flag is central-wide (check-1/2 instead prove all three sites over gRPC with distinct keys).
The instance-dependent matrix (tag ops, lifecycle, standby parked retry) and `TriggerSiteFailover`
get their live exercise in Phase 3's full UI command matrix; the transport itself is proven here.
---
## Phase 2 — full site→central cutover + S&F soak — **PASS** (2026-07-23)
All three sites (**all 6 nodes**) flipped to `CentralTransport=Grpc` (edit to
`docker/site-*/appsettings.Site.json`, reverted in git after the gate — defaults stay Akka),
rebuilt from `main` + `--force-recreate`. Central left on `SiteTransport=Akka` (Phase 3 owns
that direction). The gate was driven by a **live S&F workload**, closing the notification/audit
path that 1A/1B could only unit-prove.
### S&F driver
A minimal dependency-free template `SoakNotify` (one 5 s `Interval` script:
`Notify.To("Engineering Alerts").Send(...)`), 3 instances deployed+enabled on site-a → a steady
**3 notifications per 5 s bucket** (the pre-existing `Motor Controller` "soak-motor" instances
need 30 OPC UA bindings and were unusable). The central `dbo.Notifications` table (one row per
`NotificationId`, insert-if-not-exists) is the no-loss/no-dupes source of truth.
### Checks
| # | Check | Result |
|---|---|---|
| 1 | All 6 site nodes on `CentralTransport=Grpc` | **PASS** — each logs `Site→central transport: gRPC to 2 central endpoint(s)`; **0** `PermissionDenied` across all six for the whole run |
| 2 | Full control plane rides gRPC `CentralControlService` | **PASS** — central sees `Heartbeat`, `ReportSiteHealth`, `SubmitNotification` (the S&F path), `IngestAuditEvents`**196 RPCs / 90 s, 0 non-200** |
| 3 | Health/heartbeat cadence unchanged, no sequence regressions | **PASS**`CentralHealthAggregator` logged **0** sequence-regression/out-of-order lines; heartbeat steady at ~144/30 s across 3 sites |
| 4 | **Single-node failover** — hard-kill the **active** central (central-a) | **PASS**`CentralChannelProvider` logged sticky failover `central-a:8083 → central-b:8083` at the instant of kill; central-b active in **29 s** (auto-down); notif count froze at 72 during the gap |
| 5 | Buffer drains, no loss/dupes (single-node) | **PASS** — count resumed 72→101; **every 5 s bucket through the outage = exactly 3**, no gap; 101 total == 101 distinct |
| 6 | **Failback** — restart central-a | **PASS** — rejoined **ready in ~5 s as standby** (`active=503`); central-b **retained active** (oldest-Up, no role flap); traffic uninterrupted (uniform 3/bucket across failback); central-a singletons → `Younger` |
| 7 | **Full central outage** — hard-kill **both** central (~59 s) | **PASS** — count frozen at 155 for the entire outage; sites buffered continuously |
| 8 | Cold re-form + drain, no loss/dupes (both-down) | **PASS** — cold cluster re-formed, central-b active in **~14 s**; ~42 buffered notifications drained; **every 5 s bucket across the whole ~59 s both-dead window = exactly 3**, no gap; **216 total == 216 distinct** |
Checks 48 also stand as a live preview of Phase 5 checks 4 (failover/failback) and 5 (mid-drain
kill, zero loss/zero dupes).
### Notes
- **Audit telemetry also rides the new gRPC `CentralControlService`** (`IngestAuditEvents`) — the
`SiteAuditTelemetryActor` "ClusterClientSiteAuditClient" label is legacy naming, not the wire
path. So `CentralTransport=Grpc` moves heartbeat, health, notification S&F **and** audit off
ClusterClient in one flip.
- Sites settled on **central-a** as the gRPC endpoint after the cold both-restart while **central-b**
held the active/singleton role — the gRPC endpoint node and the singleton host can differ; central-a
receives the forward and Akka-routes to the `NotificationOutboxActor` singleton on central-b. Both
are correct and independent.
- Some notifications land `Parked` at central (no SMTP config on the bare rig) — irrelevant to the
transport proof: the `Notifications` **row** is written on forward regardless of downstream SMTP
delivery, so the count is a faithful no-loss/no-dupes measure.
- Rig left running on the gRPC build; git config reverted to Akka default (a redeploy from `main`
resets to all-Akka).
---
## Phase 3 — full central→site cutover + command matrix — **PASS** (2026-07-23)
Central flipped to `SiteTransport=Grpc` (both central nodes; the flag is **central-wide**
`CentralCommunicationActor.SelectTransport`). Sites kept on `CentralTransport=Grpc` from Phase 2,
so the rig ran **both directions on gRPC simultaneously** — the eventual all-gRPC end state.
Config bind-mounted (`:ro`), so a container `--force-recreate` (no image rebuild — code unchanged
since Phase 2) applied it. Central logged the selection at startup:
`central→site command transport: gRPC (SiteCommandService)`.
The Phase 2 `SoakNotify` instances (#9597) **survived the recreate as Enabled** (site volume
persisted `deployed_configurations` this time), so the notification workload kept flowing —
and lifecycle commands became drivable, closing the 1B/Phase-2 gap.
### Checks
| # | Check | Result |
|---|---|---|
| 1 | Central selects the gRPC command transport | **PASS**`central→site command transport: gRPC (SiteCommandService)`; **0** `Created ClusterClient to` site lines on either central |
| 2 | `ExecuteQuery` (event-log) over gRPC, all 3 sites | **PASS** — all 3 returned correlationIds; site logs `SiteCommandService/ExecuteQuery - 200` (site-a-a/b-a/c-a each served the calls; standby nodes 0 — work lands on the active site node via the singleton proxy) |
| 3 | `ExecuteParked` (parked query) over gRPC, all 3 sites | **PASS** — all 3 returned `success` payloads over `SiteCommandService/ExecuteParked` |
| 4 | `ExecuteLifecycle` (disable→enable) over gRPC, site-a | **PASS**`instance disable`/`enable #95``success:true`; site served `SiteCommandService/ExecuteLifecycle` (×4), instance state toggled — **NEW live coverage vs 1B/Phase-2 (needed a deployed instance)** |
| 5 | **Site-node kill mid-command → clean error, no hang** | **PASS** — killed the **active** site-a node (site-a-a) during a 1 s query loop: the in-flight call returned `TIMEOUT` at **dur=30.1 s = the `QueryTimeout` deadline** — bounded, not an indefinite hang (see note) |
| 6 | **Site-pair failover mid-stream** | **PASS** — the very next query (~32 s after kill) and all subsequent ones succeeded automatically via **site-a-b**; `SitePairChannelProvider` failed the gRPC channel NodeA→NodeB and the site singleton migrated; site→central S&F never stopped (notif count climbed 619→715 through the kill, still no dupes) |
| 7 | No PSK drift | **PASS****0** `PermissionDenied`/`Unauthenticated` across all 8 nodes for the whole run |
| 8 | Zero ClusterClient activity on the flipped path | **PASS** — central built no site ClusterClients; command routing is entirely `SiteCommandService` gRPC |
### Note on check 5 (deadline vs fast-fail)
The command in flight when site-a-a was **hard-killed** (SIGKILL) waited the full 30 s
`QueryTimeout` rather than failing fast on connect-refused: an already-dispatched gRPC call on a
dropped connection isn't observed as unsent, so it correctly cannot be auto-retried on the peer
node (it might have executed) and returns the deadline error to the caller — exactly the plan's
"deadline ≠ retry" rule. The deadline is the backstop; "no hang beyond deadline" is satisfied
(30.1 s). Only **provably-unsent** connect failures fail over fast, which is why every *subsequent*
call recovered immediately via site-a-b.
### Not driven on this gate (unchanged from 1B/Phase 2)
- **`ExecuteOpcUa`** (BrowseNode/ReadTagValues/WriteTag) — needs an OPC-bound deployed instance;
the bare rig's only deployable template (`SoakNotify`) has no data connection, and `Motor
Controller` needs 30 OPC UA bindings. Unit-proven (dispatcher routing ×28).
- **`ExecuteRoute`** (inbound-API → routed site script) — needs an inbound method + routing target
the rig lacks.
- **`TriggerFailover`** — no CLI verb (UI/management-only), destructive; unit-proven (ack-before-
`Leave` ordering tests). The hard-kill in check 6 is the live equivalent of a site-pair failover.
Rig left running with **both** transports on gRPC; git config reverted to Akka default (a redeploy
from `main` resets to all-Akka).
---
## Phase 5 — full eight-check gate on the deletion build — **PASS** (2026-07-23)
The migration's terminal gate, run on `main` **after Phase 4** (`7fd5cb2b`) — the build where the
Akka `ClusterClient`/`ClusterClientReceptionist` transport is **physically deleted**, the
`CentralTransport`/`SiteTransport` flags are gone, and gRPC is the *only* site↔central transport
(no flags to flip). Rig rebuilt from `main` via `bash docker/deploy.sh`, all 9 containers recreated
from the new image; central MS SQL and the S&F driver (`SoakNotify` template #2147 / instances
#9597 / list "Engineering Alerts" #28) persisted from prior phases.
**Provenance guard:** the pre-existing rig image was built `11:17`, *before* the Phase 4 commit
(`12:54`); it still carried ClusterClient code and would have invalidated checks 78. Confirmed
by rebuild timestamp, then by **0** ClusterClient/receptionist log lines across all 8 nodes on the
fresh boot. (The lone `ClusterClientReceptionist` symbol still present is inside
`Akka.Cluster.Tools.dll` — the library we deliberately keep for `ClusterSingleton` — not our code.)
Header names for the negative probes: `authorization: Bearer <psk>` + `x-scadabridge-site: <siteId>`
(`ISitePskProvider.cs`). Docker rig PSKs: `dev-grpc-psk-docker-site-{a,b,c}`
(`ScadaBridge__Communication__SitePsks__site-*`, central override; sites read their own
`GrpcPsk` from mounted `appsettings.Site.json`).
### Checks
| # | Check | Result |
|---|---|---|
| 1 | **PSK negatives** | **PASS (with a contract clarification)** — see below |
| 2 | **Site→central matrix** | **PASS** — notifications, both audit paths, health, heartbeat, reconcile all live over gRPC |
| 3 | **Central→site matrix** | **PASS (proportionate)**`ExecuteQuery`/`ExecuteParked`/`ExecuteLifecycle` live; instance-dependent RPCs carried forward (see below) |
| 4 | **Failover / failback** | **PASS** — active-central kill → sticky flip in 1 s, central-b active in 26 s; failback kept central-b active (oldest-Up, no flap) |
| 5 | **Mid-drain kill, zero loss/dupes** | **PASS** — buffered notifications flushed; **total == distinct** across the outage |
| 6 | **Frame-class retirement (>128 KB)** | **PASS** — a single **297,574-byte** gRPC reply succeeded (Akka's 128 KB frame would have dropped it) |
| 7 | **No cross-boundary Akka association** | **PASS** — each Akka cluster's membership is strictly its own pair; **0** cross-boundary association lines both directions |
| 8 | **Full-rig restart discipline** | **PASS** — all 9 restarted together came up clean; **0** receptionist/ClusterClient lines on any node; sites reconnected over gRPC; S&F resumed with no dupes |
### Check 1 — PSK negatives (and the contract clarification)
Against the two gated services on site-a (`:9023`), using the local proto descriptors
(`-proto sitestream.proto` / `site_command.proto`; server reflection is off in this build):
```
SiteStreamService/PullAuditEvents:
no credentials -> PermissionDenied "Control plane authentication failed."
wrong key (site-b's key) -> PermissionDenied
correct key -> success (audit events returned)
SiteCommandService/ExecuteQuery:
no credentials -> PermissionDenied
wrong key (site-b's key) -> PermissionDenied
```
site-a-a's log recorded **exactly 4** control-plane rejections — the four deliberate negative
probes above, nothing else.
**Clarification the gate produced:** the plan listed "missing `x-scadabridge-site`
`PermissionDenied`" as a negative case. It does **not** hold on the *site* side, and that is
correct by design. `ControlPlaneAuthInterceptor.Authorize` compares the presented bearer against
the node's **own single** `GrpcPsk` (`_options.Value.GrpcPsk`) and never reads
`x-scadabridge-site`. That header is a **central-side routing hint**`SitePskProvider` uses it to
pick *which* site's key to expect, because central holds many. A single-key site node needs no such
hint. Per-site key isolation — the actual security property — is proven by the **wrong-key**
rejection (site-b's key refused at site-a), not by the header. Recorded rather than "fixed": adding
a header requirement to the site interceptor would be theatre.
**LocalDb sync unaffected:** the replicated node (site-a) logged one boot-order
`faulted; reconnecting` (peer node-b not yet up), the passive peer accepted the
`/localdb_sync.v1.LocalDbSync/Sync` POST 1 s later, and the health report settled to
`localDbReplicationConnected:true, localDbOplogBacklog:0`. **0** LocalDb sync auth failures — the
control-plane interceptor and the separate `LocalDbSyncAuthInterceptor` don't interfere. site-b/c
report `localDbReplicationConnected:false` — the intended rig posture (only site-a replicated).
### Check 2 — site→central matrix
- **Notifications e2e:** `dbo.Notifications` climbed continuously at the driver's ~3 rows / 5 s,
**total == distinct** at every sample (insert-if-not-exists no-loss/no-dupe truth).
- **Both audit paths live over gRPC:** in a 3-minute window, `dbo.AuditLog` carried `node-a`/`node-b`
rows (105 each — **site-originated**, forwarded over the gRPC `IngestAuditEvents` RPC) *and*
`central-a` rows (420 — central-direct-write from the outbox dispatcher). Row count climbing live.
- **Health live per site:** `health summary` shows all three sites `isOnline:true`, sequence numbers
advancing, fresh heartbeats; site-a `enabledInstanceCount:3` with a live `Notification` S&F buffer.
- **Heartbeat → active flag:** central-a `active=200`, central-b `active=503`; each site's report
drives its online flag.
- **Reconcile self-heal:** restarting `site-a-a` produced a fresh (17:11:24) `Site→central transport:
gRPC to 2 central endpoint(s)` then `Reconcile pass … complete: 0 fetched, 0 failed, 0 orphan(s)`;
S&F never dropped across the restart (active node kept emitting), no dupes.
- **Not driven (carry-forward, bare rig):** cached-call telemetry (`SoakNotify` only calls
`Notify.Send`, no `CachedCall`/`CachedWrite`); a notification reaching **Delivered** rather than
`Parked` (no SMTP on the rig — the transport truth is the row-count, per Phase 2).
### Check 3 — central→site matrix
`ExecuteQuery` (event-log, all 3 sites → correlationIds + entries), `ExecuteParked` (parked-messages,
all 3 sites → empty, no parked ops on a bare rig), and `ExecuteLifecycle` (disable→enable #95 →
`success:true` both) all rode `SiteCommandService` gRPC; site active nodes logged them
(site-a-a: 8 `ExecuteLifecycle` + 4 `ExecuteParked` + 4 `ExecuteQuery`; site-b-a/c-a:
`ExecuteParked` + `ExecuteQuery`). **Carried forward, unchanged from Phase 3** (needs a deployed
OPC-bound instance / inbound method / a parked op / the destructive UI-only failover verb the bare
rig lacks): `ExecuteOpcUa`, `ExecuteRoute`, standby parked retry/discard, `TriggerFailover`
(the hard-kill in Check 4 is its live equivalent). All four are unit-proven (dispatcher routing ×28,
ack-before-`Leave` ordering).
### Checks 4 & 5 — failover / failback / mid-drain kill
Hard-killed the **active** central (central-a) mid-drain: site-a-b logged the sticky endpoint flip
`central-a:8083 → central-b:8083` **1 s** after the kill; central-b reached active in **26 s**
(auto-down). The notification count froze during the gap, then **flushed the buffered rows and
resumed** at the steady 3 / 5 s. Post-drain **total == distinct (4395 == 4395)** — zero loss, zero
duplicates. Failback: restarting central-a, it rejoined and central-b **retained** active
(`a=503` standby / `b=200` active) — oldest-Up, no role flap.
### Check 6 — frame-class retirement
`PullAuditEvents{since_utc: 2020-01-01, batch_size: 5000}` against the active site node returned a
single **297,574-byte** reply successfully. Over Akka's default 128 KB frame
(`log-frame-size-exceeding` off) this class of payload was silently dropped and the caller's Ask
timed out (`docs/known-issues/2026-06-26-…`). On gRPC (4 MB default cap) it is a normal reply.
### Check 7 — no cross-boundary Akka association
Each Akka cluster's membership is strictly its own pair: central `[central-a, central-b]`, site-a
`[site-a-a, site-a-b]`, site-c `[site-c-a, site-c-b]` — **no site node ever appears in central's
membership and vice versa**. Central's log holds **0** `Association with remote system …site-*`
lines and **0** references to any site address; site-a's log holds **0** references to any central
address. The two boundaries are wired only by per-pair Akka remoting (8082, internal) and the gRPC
control/data planes (8083); there is no Akka association across the site↔central line.
### Check 8 — full-rig restart discipline
Restarted all 9 rig containers together (pairs together, honouring the LocalDb-replication constraint).
Central re-formed with central-a active; **0** receptionist/ClusterClient lines across all 8 nodes;
all three sites logged `Site→central transport: gRPC to 2 central endpoint(s)`; S&F resumed
(4407 → 4483, **distinct == total**, no dupes).
### End state
The migration is complete and proven on the deletion build. The rig runs the committed all-gRPC
default (Phase 4 flipped the shipped appsettings to `CentralGrpcEndpoints` — unlike Phases 2/3, a
redeploy from `main` now yields all-gRPC, not all-Akka). No ClusterClient remains anywhere in the
running system or the source tree. Deferred, unchanged from earlier phases: the instance-dependent
central→site RPC matrix (`ExecuteOpcUa`/`ExecuteRoute`/standby parked retry) and live
`TriggerFailover`, all unit-proven; PSK **rotation** on a live pair; `docker-env2` was updated with
its own key but not gated here.
@@ -309,30 +309,31 @@ Critical path ≈ 1B: **~46 weeks total**, matching the design estimate.
- [x] Phase 0 DoD: suite green; rig unauthenticated ⇒ `PermissionDenied`, authenticated paths work; PR merged (#25, ff to `main` @ `3fa95555`; gate PASS in `2026-07-22-clusterclient-to-grpc-live-gate.md`)
**Phase 1A — central control plane** (worktree, `feat/grpc-central-control`)
- [ ] T1A.1 `central_control.proto` (7 RPCs; checked-in codegen) + `CentralControlDtoMapper` + round-trip golden tests
- [ ] T1A.2 Central hosting: `AddGrpc` + per-site-PSK interceptor (`x-scadabridge-site`), `CentralGrpcPort` h2c listener (8083), `CentralControlGrpcService` (Ask existing handlers), readiness gate
- [ ] T1A.3 `ICentralTransport` (Akka extract + Grpc impl), `CentralChannelProvider` (sticky failover/failback, backoff, deadlines, PSK), `CentralTransport` flag default `Akka`, `CentralGrpcEndpoints` option + validator
- [ ] T1A.4 Tests: actor-with-fake-transport ×7, TestServer transport tests, S&F/audit/health suites pass unmodified
- [ ] 1A DoD: rig site-a on `Grpc` proves all 5 site→central paths while site-b/c stay Akka; PR merged (before 1B)
- [x] T1A.1 `central_control.proto` (7 RPCs; checked-in codegen) + `CentralControlDtoMapper` + round-trip golden tests
- [x] T1A.2 Central hosting: `AddGrpc` + per-site-PSK interceptor (`x-scadabridge-site`), `CentralGrpcPort` h2c listener (8083), `CentralControlGrpcService` (Ask existing handlers), readiness gate
- [x] T1A.3 `ICentralTransport` (Akka extract + Grpc impl), `CentralChannelProvider` (sticky failover/failback, backoff, deadlines, PSK), `CentralTransport` flag default `Akka`, `CentralGrpcEndpoints` option + validator
- [x] T1A.4 Tests: actor-with-fake-transport ×7, TestServer transport tests, S&F/audit/health suites pass unmodified
- [x] 1A DoD: rig site-a on `Grpc` proves site→central paths (heartbeat/health/reconcile + coexistence) while site-b/c stay Akka; PR #26 merged (`aa60f438`). Notification/audit deferred to Phase 2 soak (no deployed instance); rig caught + fixed a central `:5000` HTTP-drop regression (`0e162cb2`)
**Phase 1B — site command plane** (worktree, `feat/grpc-sitecommand`)
- [ ] T1B.1 `site_command.proto` (6 oneof RPCs / 28 commands) + `SiteCommandDtoMapper` + round-trip golden tests (all 28 + replies)
- [ ] T1B.2 `SiteCommandDispatcher` refactor (actor + new `SiteCommandGrpcService` share it; parked stays node-local)
- [ ] T1B.3 `ISiteCommandTransport` in `CentralCommunicationActor` (Akka extract + Grpc impl), `SitePairChannelProvider` (Site entity Grpc columns + DB refresh loop), `SiteTransport` flag default `Akka`
- [ ] T1B.4 Tests: dispatcher routing ×28, actor envelope/reply plumbing, TestServer service tests, existing Communication suites green
- [ ] 1B DoD: rig central on `Grpc` for site-a proves full command matrix incl. standby parked retry; rebased on 1A; PR merged
- [x] T1B.1 `site_command.proto` (6 oneof RPCs / 28 commands) + `SiteCommandDtoMapper` + round-trip golden tests (all 28 commands + 22 reply shapes + 18 nested types; reflection-driven coverage guard over the mapper surface and the generated oneof descriptors)
- [x] T1B.2 `SiteCommandDispatcher` refactor (actor + new `SiteCommandGrpcService` share it; parked stays node-local; failover ack-before-Leave via dry-run resolve + deferred `CommitLeave`; interceptor `DefaultGatedPrefixes` extended to `SiteCommandService`)
- [x] T1B.3 `ISiteCommandTransport` in `CentralCommunicationActor` (Akka extract + Grpc impl), `SitePairChannelProvider` (Site entity Grpc columns + DB refresh loop), `SiteTransport` flag default `Akka`
- [x] T1B.4 Tests: dispatcher routing ×28, actor envelope/reply plumbing, TestServer service tests, existing Communication suites green
- [x] 1B DoD (proportionate): rig central on `SiteTransport=Grpc` proves central→site rides authenticated gRPC `SiteCommandService` for all 3 sites (`ExecuteQuery`/`ExecuteParked` → 200, per-site PSK, 0 auth failures); rebased on 1A. Instance-dependent commands (tag ops/lifecycle/standby parked retry) + `TriggerSiteFailover` deferred to Phase 3 (no deployed instance / destructive UI-only); command-plane coexistence not expressible (`SiteTransport` is central-wide). Gate: `2026-07-22-clusterclient-to-grpc-live-gate.md`
**Phase 2 ∥ 3 — cutover + soak**
- [ ] P2 All sites `CentralTransport=Grpc`; central-kill S&F soak (no loss/dupes), failback observed, health sequences clean
- [ ] P3 Central `SiteTransport=Grpc` all sites; full UI command matrix per site; site-kill mid-command clean; no PSK noise; zero ClusterClient log activity on flipped paths
- [x] P2 All sites `CentralTransport=Grpc`; central-kill S&F soak (no loss/dupes), failback observed, health sequences clean**PASS 2026-07-23** (live gate: single-node active-kill failover 29s + full-outage ~59s both-down drain; every 5s bucket = exactly 3 through both outages, 216 total == 216 distinct; 0 auth failures / 0 seq regressions; whole control plane — heartbeat/health/notification/audit — on gRPC `CentralControlService`)
- [x] P3 Central `SiteTransport=Grpc` all sites; full UI command matrix per site; site-kill mid-command clean; no PSK noise; zero ClusterClient log activity on flipped paths**PASS 2026-07-23** (live gate: `ExecuteQuery`/`ExecuteParked` all 3 sites + `ExecuteLifecycle` disable/enable on site-a, all 200 over `SiteCommandService`; active-site-node hard-kill mid-command → clean `TIMEOUT` at the 30s deadline, next call failed over to site-a-b automatically; 0 PermissionDenied / 0 ClusterClient-to-site on both central. `ExecuteOpcUa`/`ExecuteRoute`/`TriggerFailover` deferred — no OPC-bound instance / no CLI verb, unit-proven)
**Phase 4 — deletion**
- [ ] Defaults flip to `Grpc` + soak; then delete Akka transports, ClusterClient creation, `DefaultSiteClientFactory`, receptionist registrations, `CentralContactPoints`, then the flags
- [ ] Grep-gates pass (`clusterclient|receptionist` → historical docs only; `CentralContactPoints` → empty)
- [ ] Docs updated: `grpc_streams.md`, `Component-Host.md`, `Component-StoreAndForward.md:137`, known-issues cross-ref
- [x] gRPC made the only transport (flags DELETED, not flipped — end state is identical) + deleted `AkkaCentralTransport`/`AkkaSiteTransport`, ClusterClient creation (`AkkaHostedService`), `DefaultSiteClientFactory`+`ISiteClientFactory`, both receptionist registrations (`:436`/`:1001`), `CentralContactPoints` + the `CentralTransport`/`SiteTransport` flags + `CentralTransportMode`/`SiteTransportKind` enums. `NoOpCentralTransport` added as the fail-loud null-default; `CentralGrpcEndpoints` now unconditional (StartupValidator requires ≥1 on Site). Kept `Akka.Cluster.Tools` (ClusterSingleton). Rig configs (docker ×6, docker-env2 ×2, Host default, wonder-app-vd03) moved `CentralContactPoints``CentralGrpcEndpoints`. **Full solution build 0/0; Communication.Tests 640, Host.Tests + StartupValidator + SiteActorPath green, audit-push integration green** (2026-07-23)
- [x] Grep-gates pass (`CentralContactPoints` → only "replaces the former" doc refs + plan trackers; deleted symbols → 0 live refs; remaining `clusterclient` in src = the misleadingly-named `ClusterClientSiteAuditClient` [transport-agnostic, works unchanged] + stale inline doc-comments, noted as follow-up)
- [x] Docs updated: `Component-Communication.md`, `components/Communication.md`, `Component-Host.md`, `Component-StoreAndForward.md`, `topology-guide.md`, `grpc_streams.md`, known-issues frame-size amendment, **CLAUDE.md** transport decisions
**Phase 5 — live gate** (record in `2026-07-22-clusterclient-to-grpc-live-gate.md`)
- [ ] 1 PSK negatives · [ ] 2 site→central matrix · [ ] 3 central→site matrix · [ ] 4 failover/failback both directions · [ ] 5 mid-drain kill · [ ] 6 >128 KB frame-class proof · [ ] 7 no cross-boundary Akka association · [ ] 8 full-rig restart clean
**Phase 5 — live gate** (record in `2026-07-22-clusterclient-to-grpc-live-gate.md`)**PASS 2026-07-23** (deletion build, `main` @ `7fd5cb2b`)
- [x] 1 PSK negatives · [x] 2 site→central matrix · [x] 3 central→site matrix · [x] 4 failover/failback both directions · [x] 5 mid-drain kill · [x] 6 >128 KB frame-class proof · [x] 7 no cross-boundary Akka association · [x] 8 full-rig restart clean
- All 8 PASS. Notables: check 1 clarified the site-side gate uses the node's single `GrpcPsk` and ignores `x-scadabridge-site` (central-side routing hint) — per-site isolation proven by wrong-key reject, not the header; check 6 = a **297,574-byte** single gRPC reply (a payload Akka's 128 KB frame dropped); check 7 = Akka cluster membership strictly pair-only, 0 cross-boundary association; check 8 = full-rig restart, 0 receptionist/ClusterClient lines on any of the 8 nodes. Instance-dependent central→site RPCs (`ExecuteOpcUa`/`ExecuteRoute`/standby parked retry/`TriggerFailover`) carried forward unit-proven — bare rig has no OPC-bound instance / inbound method / parked op / CLI failover verb.
## Gotchas for the executor (will bite; read twice)
@@ -87,42 +87,46 @@
"id": "T1A.1",
"phase": "1A",
"subject": "central_control.proto (7 RPCs, checked-in codegen) + CentralControlDtoMapper + round-trip golden tests",
"status": "in_progress",
"status": "completed",
"activeForm": "Authoring central_control.proto and its mappers",
"blockedBy": [
"P0.DoD"
]
],
"notes": "d7455577 on feat/grpc-central-control. 7 RPCs; ingest RPCs reuse sitestream AuditEventBatch/CachedTelemetryBatch/IngestAck by import. 32 goldens, verified to have teeth by mutation. PLAN CORRECTIONS FOUND: (1) actor sends IngestAuditEventsCommand/-Reply (IReadOnlyList<Guid>), NOT the batch/IngestAck types the plan's table claims - mapper bridges; (2) CachedTelemetryEntry carries SiteCall not SiteCallOperational, needed a new SiteCallDtoMapper.ToDto(SiteCall); (3) SiteHealthReport is ~33 members and 5 are INIT-ONLY props not ctor params - FromDto needs an object initializer or they silently drop; (4) 3 collections are nullable with load-bearing null, proto3 cannot express presence on repeated/map => wrapper messages; (5) ConnectionHealth has no Unspecified member, so naive mapping puts Connected on proto3 zero - reserved 0 and unknown decodes to Error, never Connected."
},
{
"id": "T1A.2",
"phase": "1A",
"subject": "Central hosting: AddGrpc + per-site-PSK interceptor, CentralGrpcPort h2c listener, CentralControlGrpcService, readiness gate",
"status": "pending",
"status": "completed",
"activeForm": "Hosting CentralControlService on central",
"blockedBy": [
"T1A.1"
]
],
"notes": "780bb9c3 on feat/grpc-central-control. NEW class CentralControlAuthInterceptor (one public ctor, 3-arg internal, pinned by reflection test) - central verifies per-site PSK via ISitePskProvider keyed by required x-scadabridge-site header, fail-closed on missing/blank/unknown/mismatch. CentralControlGrpcService Asks existing CentralCommunicationActor (0 handler changes), readiness via SetReady mirror, heartbeat Tell/never-gated, ingest reuses AuditIngestAskTimeout. Central branch had NO AddGrpc/Kestrel before - added h2c listener on CentralGrpcPort default 8083, :5000 untouched. Rig ports 9013/9014:8083 published. PLAN GAPS: (1) mappers throw on unset WKT fields - test DTOs must carry a timestamp (no prod impact); (2) plan gave no deadline for Submit/QueryNotification - used NotificationForwardTimeout, check T1A.3 client sets same. 17 CentralControl tests, Host.Tests 384, Communication.Tests 356."
},
{
"id": "T1A.3",
"phase": "1A",
"subject": "ICentralTransport (Akka extract + Grpc impl), CentralChannelProvider, CentralTransport flag, CentralGrpcEndpoints option",
"status": "pending",
"status": "completed",
"activeForm": "Building the site->central transport seam",
"blockedBy": [
"T1A.1"
]
],
"notes": "33b15f10 on feat/grpc-central-control. ICentralTransport: 7 methods, actor delegates all 7; optional ctor param (null->actor self-builds AkkaCentralTransport in PreStart, byte-identical Akka path). CentralChannelProvider = sticky failover/failback, 1s-double-cap-60s backoff, per-site PSK not needed (site's own GrpcPsk + site header). Flag CentralTransport default Akka; on Grpc the ClusterClient is not created at all. Deadlines matched per-RPC to today's Ask timeouts (table in report). PLAN CORRECTION (important): connect-refused surfaces as StatusCode.Internal + 'Error starting gRPC call'/HttpRequestException, NOT Unavailable, in this Grpc.Net version - IsConnectFailure covers Unavailable OR that Internal class, still excludes DeadlineExceeded (verified in code + DeadlineExceeded_IsNotRetriedOnThePeer test). Above-seam suites (NotificationForwarder/HealthReportSender/SiteAuditTelemetry) pass UNMODIFIED. Communication.Tests 371, Host.Tests 391."
},
{
"id": "T1A.4",
"phase": "1A",
"subject": "Tests: actor-with-fake-transport x7, TestServer transport tests, S&F/audit/health suites pass unmodified",
"status": "pending",
"status": "completed",
"activeForm": "Testing the central control plane",
"blockedBy": [
"T1A.2",
"T1A.3"
]
],
"notes": "Tests shipped inside T1A.1/1A.2/1A.3 per the plan folding T1A.4 into each task. Actor-with-fake-transport x7, TestServer transport tests, above-seam suites pass unmodified."
},
{
"id": "P1A.DoD",
@@ -138,42 +142,46 @@
"id": "T1B.1",
"phase": "1B",
"subject": "site_command.proto (6 oneof RPCs / 28 commands) + SiteCommandDtoMapper + round-trip golden tests",
"status": "in_progress",
"status": "completed",
"activeForm": "Authoring site_command.proto and its mappers",
"blockedBy": [
"P0.DoD"
]
],
"notes": "a7481174 on feat/grpc-sitecommand. 6 oneof RPCs / 28 commands / 22 replies / 18 nested, verified by reflection guards not a hand list. PLAN CORRECTIONS: (1) generic Guid?->empty-string-means-null is UNSAFE for RouteToWaitForAttributeRequest.TargetValueEncoded, a nullable STRING where '' is a real wait target distinct from null - gave it a StringValue wrapper; (2) RouteToGetAttributesResponse.Values is NON-nullable dict but RouteToCallRequest.Parameters is nullable - two decode paths (absent->empty vs absent->null); (3) DeployArtifactsCommand has 6 NULLABLE artifact collections, proto3 repeated collapses null<->empty => wrapper messages (PLAN-05 T8 class); (4) AlarmStateChanged.Condition is derived-on-read over a nullable backing field - encoding unconditionally breaks record equality, mapper omits condition==computed default. Added seam types for T1B.2/3: SiteCommandGroup, GroupOf/GroupOfReply, UnsubscribeDebugViewAck marker (unsubscribe is Tell today, unary RPC must answer). REBASE NOTE: union-conflict expected in Communication.csproj commented Protobuf ItemGroup + docker/regen-proto.sh vs 1A."
},
{
"id": "T1B.2",
"phase": "1B",
"subject": "SiteCommandDispatcher refactor (actor + SiteCommandGrpcService share it; parked stays node-local)",
"status": "pending",
"status": "completed",
"activeForm": "Extracting the site command dispatcher",
"blockedBy": [
"T1B.1"
]
],
"notes": "cd6c20e1 on feat/grpc-sitecommand. SiteCommandDispatcher = pure routing (ResolveRoute -> Route{Disposition,Target,Reply}); actor delegates all 27 non-failover Receives to it, gRPC SiteCommandGrpcService calls the SAME dispatcher; shared via SetReady(dispatcher) hand-off like SiteStreamGrpcServer. Parked stays node-local (proven NotSame(proxy)). Interceptor: SiteCommandService added to DefaultGatedPrefixes via descriptor - NO new ctor, one-public-ctor test green. PLAN CORRECTION (important): existing actor code issues cluster Leave BEFORE the ack (fine over ClusterClient Tell, WRONG for gRPC ack-before-Leave). Solved via the pre-existing dryRun param on ClusterFailoverCoordinator.FailOverOldest: seam widened to Func<role,dryRun,addr?>, actor commits immediately (dryRun:false, byte-identical today), gRPC resolves dry-run then defers CommitLeave until after ack. Two ordering tests pin resolve-then-leave. Actor got an OPTIONAL dispatcher param so existing SiteCommunicationActorTests pass with ZERO edits. Server derives local-Ask timeout from ServerCallContext.Deadline (remaining), 2min fallback. Communication.Tests 574, Host.Tests 377."
},
{
"id": "T1B.3",
"phase": "1B",
"subject": "ISiteCommandTransport in CentralCommunicationActor (Akka extract + Grpc impl), SitePairChannelProvider, SiteTransport flag",
"status": "pending",
"status": "completed",
"activeForm": "Building the central->site transport seam",
"blockedBy": [
"T1B.1"
]
],
"notes": "3be85f19 on feat/grpc-sitecommand. ISiteCommandTransport injected into CentralCommunicationActor; HandleSiteEnvelope->_transport.Send(env,Sender), HandleSiteAddressCacheLoaded->_transport.ReconcileSites. AkkaSiteTransport extracted verbatim (incl no-client->drop path). GrpcSiteTransport + SitePairChannelProvider (A/B from GrpcNode*Address, per-site PSK via ISitePskProvider, Invalidate on removal). Rode the EXISTING LoadSiteAddressesFromDb loop (extended SiteAddressCacheLoaded to carry gRPC cols too) - ONE poll loop. Flag SiteTransport default Akka. PLAN CORRECTIONS: (1) LoadSiteAddressesFromDb DOES exist (known); (2) plan deadline table wrong TWICE - DeploymentStateQuery uses QueryTimeout not LifecycleTimeout, TriggerSiteFailover uses QueryTimeout not LifecycleTimeout - resolver is per-command-type not per-group, verified in code; (3) SiteCommandGroup XML doc's 'shared deadline class' claim is false; (4) THIRD SiteEnvelope producer DebugStreamBridgeActor (not just CommunicationService+SiteCallAudit) - reply plumbing routes to a real actor sender, not only Ask temp actors. RetryParkedOperation/DiscardParkedOperation->QueryTimeout(30s) keeps SiteCallAudit RelayTimeout(10s)<30s. Communication.Tests 607, gRPC Host.Tests 34. Existing suites: 1 trivial helper edit (SiteAddressCacheLoaded internal->public + new dict arg)."
},
{
"id": "T1B.4",
"phase": "1B",
"subject": "Tests: dispatcher routing x28, actor envelope/reply plumbing, TestServer service tests, existing suites green",
"status": "pending",
"status": "completed",
"activeForm": "Testing the site command plane",
"blockedBy": [
"T1B.2",
"T1B.3"
]
],
"notes": "Tests shipped inside T1B.1/1B.2/1B.3 per the plan folding T1B.4 into each task. Dispatcher routing x28, actor envelope/reply plumbing, TestServer service tests, deadline theory, failover/failback, existing Communication suites green with Akka default."
},
{
"id": "P1B.DoD",
@@ -190,7 +198,7 @@
"id": "P2",
"phase": "2",
"subject": "All sites CentralTransport=Grpc; central-kill S&F soak, failback observed, health sequences clean",
"status": "pending",
"status": "completed",
"activeForm": "Running the site->central cutover soak",
"blockedBy": [
"P1A.DoD"
@@ -200,7 +208,7 @@
"id": "P3",
"phase": "3",
"subject": "Central SiteTransport=Grpc all sites; full UI command matrix; site-kill mid-command clean; zero ClusterClient activity",
"status": "pending",
"status": "completed",
"activeForm": "Running the central->site cutover soak",
"blockedBy": [
"P1B.DoD"
@@ -210,32 +218,35 @@
"id": "P4.1",
"phase": "4",
"subject": "Flip both flag defaults to Grpc + soak; delete Akka transports, ClusterClient creation, DefaultSiteClientFactory, receptionist registrations, CentralContactPoints, then the flags",
"status": "pending",
"status": "completed",
"activeForm": "Deleting the ClusterClient transport",
"blockedBy": [
"P2",
"P3"
]
],
"notes": "Branch feat/grpc-phase4-deletion (2026-07-23). Flags DELETED rather than flipped \u2014 end state is gRPC-only, identical. Deleted: AkkaCentralTransport, AkkaSiteTransport, ISiteClientFactory+DefaultSiteClientFactory, CentralCommunicationActor legacy ctor + SelectTransport, ClusterClient creation + both ClusterClientReceptionist.RegisterService in AkkaHostedService, CommunicationOptions.CentralContactPoints + CentralTransport/SiteTransport flags + CentralTransportMode/SiteTransportKind enums, RegisterCentralClient message + receive block, AkkaCentralTransportTests + DefaultSiteClientFactoryTests. Added NoOpCentralTransport (fail-loud null-default so command-dispatch TestKit suites keep constructing without a wired transport; production always injects GrpcCentralTransport). CommunicationOptionsValidator: CentralGrpcEndpoints now unconditional (no-blank-entries, role-agnostic); StartupValidator: Site-only requires \u22651 CentralGrpcEndpoint (fail-fast, mirrors GrpcPsk). Host builds GrpcSiteTransport (central) + GrpcCentralTransport (site) unconditionally. Kept Akka.Cluster.Tools (ClusterSingleton). Test rework: deleted the ClusterClient.Send per-site-routing tests (covered by CentralCommunicationActorTransportTests + GrpcSiteTransport suites), swapped ISiteClientFactory\u2192substitute ISiteCommandTransport across 5 files, converted SiteAuditPushFlow's ClusterClientRelay\u2192BridgeCentralTransport, converted Heartbeat_StampsIsActive to a substitute ICentralTransport, repurposed HealthReportAck no-client test to the NoOp fail-loud path, removed SiteActors_CentralClusterClient_Exists. Rig: docker \u00d76 + docker-env2 \u00d72 + Host appsettings.Site.json + deploy/wonder-app-vd03 moved CentralContactPoints\u2192CentralGrpcEndpoints. Full solution build 0/0; Communication.Tests 640, StartupValidator 59, SiteActorPath 5, audit-push integration 1 all green. DID NOT fold in the dead IntegrationCallRequest deletion (#32) \u2014 SiteEnvelope routing is transport-agnostic so it still compiles; user-owned behavioral decision, left OUT of this PR."
},
{
"id": "P4.2",
"phase": "4",
"subject": "Grep-gates pass + docs updated (grpc_streams.md, Component-Host.md, Component-StoreAndForward.md, known-issues cross-ref)",
"status": "pending",
"status": "completed",
"activeForm": "Running the deletion grep-gates and doc sweep",
"blockedBy": [
"P4.1"
]
],
"notes": "Grep-gates: CentralContactPoints \u2192 only intentional 'replaces the former' doc refs + plan trackers + stale bin/ artifacts (regenerate on build); deleted symbols (DefaultSiteClientFactory/ISiteClientFactory/RegisterCentralClient/AkkaCentralTransport/AkkaSiteTransport/CentralTransportMode/SiteTransportKind/the flags) \u2192 0 live refs. Residual 'clusterclient' in src = ClusterClientSiteAuditClient (misleadingly named but transport-agnostic \u2014 it Asks SiteCommunicationActor, holds no ClusterClient; works unchanged) + ~25 stale inline XML-doc comments across SiteRuntime/Commons/SiteCallAudit \u2014 NOT scrubbed (out of plan scope, no behavior impact); FOLLOW-UP noted. Docs updated (subagent + me): Component-Communication.md, components/Communication.md, Component-Host.md, Component-StoreAndForward.md, deployment/topology-guide.md, plans/grpc_streams.md (SUPERSEDED note + LoadSiteAddressesFromDb correction), known-issues/2026-06-26-deploy-config-exceeds-akka-frame-size.md (2026-07-23 frame-size-retired amendment), and CLAUDE.md (2 transport-decision passages). Phase 5 (live gate) is next and requires a rig redeploy."
},
{
"id": "P5",
"phase": "5",
"subject": "Live gate, 8 checks, recorded in docs/plans/2026-07-22-clusterclient-to-grpc-live-gate.md",
"status": "pending",
"status": "completed",
"activeForm": "Running the live gate",
"blockedBy": [
"P4.2"
]
],
"notes": "Live gate PASS 2026-07-23 on Phase 4 deletion build (main @ 7fd5cb2b). All 8 checks PASS: PSK negatives (site-header contract clarified \u2014 site node has one key, header is central-side routing hint; per-site isolation proven by wrong-key reject); site->central matrix (notif no-loss/dupe, both audit paths node-a/b via IngestAuditEvents + central-a direct, health/heartbeat/active, reconcile self-heal over gRPC); central->site matrix ExecuteQuery/Parked/Lifecycle (OpcUa/Route/standby-parked/TriggerFailover carried forward, unit-proven); active-central kill sticky-flip 1s + central-b active 26s; mid-drain total==distinct zero loss/dupes; 297574-byte gRPC reply (frame-class retired); 0 cross-boundary Akka association (cluster membership pair-only); full-rig restart 0 clusterclient lines all 8 nodes. Recorded in docs/plans/2026-07-22-clusterclient-to-grpc-live-gate.md."
}
]
}
+15 -4
View File
@@ -1,10 +1,21 @@
# gRPC Streaming Channel: Site → Central Real-Time Data
> **Status update (2026-07-23) — the "ClusterClient keeps command/control" framing is SUPERSEDED.**
> This plan introduced gRPC for *streaming only*, leaving command/control on Akka ClusterClient. That
> split no longer holds: **Phase 4 of the ClusterClient→gRPC migration moved command/control to gRPC
> too and deleted the ClusterClient site↔central transport entirely.** Command/control now rides gRPC
> in both directions — site→central to the central-hosted `CentralControlService` (`GrpcCentralTransport`)
> and central→site to the site-hosted `SiteCommandService` (`GrpcSiteTransport`) — alongside the
> streaming `SiteStreamService` this plan describes. Wherever the text below says "ClusterClient handles
> command/control", read it as "gRPC command/control (`CentralControlService` / `SiteCommandService`)".
> See `docs/requirements/Component-Communication.md` and
> `docs/plans/2026-07-22-clusterclient-to-grpc-plan.md`.
## Context
Debug streaming events currently flow through Akka.NET ClusterClient (`InstanceActor → SiteCommunicationActor → ClusterClient.Send → CentralCommunicationActor → bridge actor`). ClusterClient wasn't built for high-throughput value streaming — it's a cluster coordination tool with gossip-based routing. As we scale beyond debug view to health streaming, alarm feeds, or future live dashboards, pushing all real-time data through ClusterClient will become a bottleneck.
**Goal**: Add a dedicated gRPC server-streaming channel on each site node. Central subscribes to sites over gRPC for real-time data. ClusterClient continues to handle command/control (subscribe, unsubscribe, deploy, lifecycle) but all streaming values flow through the gRPC channel.
**Goal**: Add a dedicated gRPC server-streaming channel on each site node. Central subscribes to sites over gRPC for real-time data. Command/control (subscribe, unsubscribe, deploy, lifecycle) and streaming values flow over gRPC. *(As originally written, command/control stayed on ClusterClient; Phase 4 later moved it to gRPC — see the status note above.)*
**Scope**: General-purpose site→central streaming transport. Debug view is the first consumer, but the proto and server are designed so future features (health streaming, alarm feeds, live dashboards) can subscribe with different event types and filters.
@@ -57,7 +68,7 @@ flowchart TD
class PB dec
```
**Key separation**: ClusterClient handles subscribe/unsubscribe/snapshot (request-response). gRPC handles the ongoing value stream (server-streaming).
**Key separation**: command/control handles subscribe/unsubscribe/snapshot (request-response); the ongoing value stream is server-streaming. Both ride gRPC after Phase 4 (subscribe/unsubscribe/snapshot over `SiteCommandService`, the value stream over `SiteStreamService`); as originally drawn, the request-response leg was ClusterClient.
## Port & Address Configuration
@@ -164,7 +175,7 @@ Add corresponding columns to the sites list table. Wire `_formGrpcNodeAAddress`
### SiteStreamGrpcClientFactory
Reads `GrpcNodeAAddress` / `GrpcNodeBAddress` from the `Site` entity (loaded by `CentralCommunicationActor.LoadSiteAddressesFromDb()`) when creating per-site gRPC channels. Falls back to NodeB if NodeA connection fails (same pattern as ClusterClient dual-contact-point failover).
Reads `GrpcNodeAAddress` / `GrpcNodeBAddress` from the `Site` entity when creating per-site gRPC channels. As of Phase 4, `CentralCommunicationActor.LoadSiteAddressesFromDb()` no longer builds Akka ClusterClient contacts at all — it builds a **per-site gRPC-endpoint cache** from those same `GrpcNodeAAddress` / `GrpcNodeBAddress` columns, feeding a `SitePairChannelProvider` that both this streaming factory and the command/control `GrpcSiteTransport` (`SiteCommandService`) dial. Falls back to NodeB if NodeA connection fails.
### Docker Compose Port Allocation
@@ -364,7 +375,7 @@ public async Task<StreamSubscription> SubscribeAsync(
### Client Factory
`SiteStreamGrpcClientFactory` caches per-site `GrpcChannel` instances (same pattern as `CentralCommunicationActor._siteClients` caching per-site ClusterClient instances).
`SiteStreamGrpcClientFactory` caches per-site `GrpcChannel` instances. (As originally written this mirrored `CentralCommunicationActor._siteClients` caching per-site ClusterClient instances; after Phase 4 that dictionary caches per-site gRPC endpoints/channels, not ClusterClients.)
## Failover & Reconnection
+53 -52
View File
@@ -2,7 +2,7 @@
## Purpose
The Communication component manages all messaging between the central cluster and site clusters. It provides the transport layer for deployments, instance lifecycle commands, integration routing, debug streaming, health reporting, notification submission, and remote queries (parked messages, event logs). Two transports are used: **Akka.NET ClusterClient** for command/control messaging and **gRPC server-streaming** for real-time data (attribute values, alarm states).
The Communication component manages all messaging between the central cluster and site clusters. It provides the transport layer for deployments, instance lifecycle commands, integration routing, debug streaming, health reporting, notification submission, and remote queries (parked messages, event logs). **gRPC is the only inter-cluster transport** (ClusterClient/ClusterClientReceptionist for site↔central messaging was removed in Phase 4 of the ClusterClient→gRPC migration, 2026-07-23). It carries three concerns: **command/control** in both directions — site→central to the central-hosted `CentralControlService` (heartbeats, health reports, notification submit/status, audit/telemetry ingest, reconcile) and central→site to the site-hosted `SiteCommandService` (deployments, lifecycle, OPC UA, remote queries, parked, route, failover); **gRPC server-streaming** for real-time data (attribute values, alarm states) via `SiteStreamService`; and **plain token-gated HTTP** for the deployment-config fetch (notify-and-fetch). Akka remoting remains, but only *intra-cluster* (pair-internal), never across the site↔central boundary.
## Location
@@ -10,8 +10,8 @@ Both central and site clusters. Each side has communication actors that handle m
## Responsibilities
- Resolve site addresses (Akka remoting and gRPC) from the configuration database and maintain a cached address map.
- Establish and maintain cross-cluster connections using Akka.NET ClusterClient/ClusterClientReceptionist for command/control.
- Resolve site addresses (Akka remoting — intra-cluster only — and gRPC) from the configuration database and maintain a cached address map, including the per-site gRPC command endpoints used by central→site command/control.
- Establish and maintain cross-cluster command/control connections over gRPC: site→central to `CentralControlService` (`GrpcCentralTransport`) and central→site to `SiteCommandService` (`GrpcSiteTransport`).
- Establish and maintain per-site gRPC streaming connections for real-time data delivery (site→central).
- Route messages between central and site clusters in a hub-and-spoke topology.
- Broker requests from external systems (via central) to sites and return responses.
@@ -54,15 +54,15 @@ Both central and site clusters. Each side has communication actors that handle m
- Site applies and acknowledges.
### 6. Debug Streaming (Site → Central)
- **Pattern**: Subscribe/push with initial snapshot. Two transports: **ClusterClient** for the subscribe/unsubscribe handshake and initial snapshot, **gRPC server-streaming** for ongoing real-time events.
- A **DebugStreamBridgeActor** (one per active debug session) is created on the central cluster by the **DebugStreamService**. The bridge actor first opens a **gRPC server-streaming subscription** to the site via `SiteStreamGrpcClient`, then sends a `SubscribeDebugViewRequest` to the site via `CentralCommunicationActor` (ClusterClient). The site's `InstanceActor` replies with an initial snapshot via the ClusterClient reply path.
- **Pattern**: Subscribe/push with initial snapshot. Both legs ride gRPC: the subscribe/unsubscribe handshake and initial snapshot go over **gRPC command/control** (`SiteCommandService`), and ongoing real-time events over **gRPC server-streaming** (`SiteStreamService`).
- A **DebugStreamBridgeActor** (one per active debug session) is created on the central cluster by the **DebugStreamService**. The bridge actor first opens a **gRPC server-streaming subscription** to the site via `SiteStreamGrpcClient`, then sends a `SubscribeDebugViewRequest` to the site as a central→site command over `SiteCommandService` (`GrpcSiteTransport`). The site's `InstanceActor` replies with an initial snapshot on that gRPC request/response.
- **gRPC stream (real-time events)**: The site's **SiteStreamGrpcServer** receives the gRPC `SubscribeInstance` call and creates a **StreamRelayActor** that subscribes to **SiteStreamManager** for the requested instance. Events (`AttributeValueChanged`, `AlarmStateChanged`) flow from `SiteStreamManager``StreamRelayActor``Channel<SiteStreamEvent>` (bounded, 1000, DropOldest) → gRPC response stream → `SiteStreamGrpcClient` on central → `DebugStreamBridgeActor`.
- The `DebugStreamEvent` message type no longer exists — events are not routed through ClusterClient. `SiteCommunicationActor` and `CentralCommunicationActor` have no role in streaming event delivery.
- The `DebugStreamEvent` message type no longer exists — events are not routed through a command/control channel. `SiteCommunicationActor` and `CentralCommunicationActor` have no role in streaming event delivery.
- The bridge actor forwards received events to the consumer via callbacks (Blazor component or SignalR hub).
- **Snapshot-to-stream handoff**: The gRPC stream is opened **before** the snapshot request to avoid missing events. The consumer applies the snapshot as baseline, then replays buffered gRPC events with timestamps newer than the snapshot (timestamp-based dedup).
- Attribute value stream messages: `[InstanceUniqueName].[AttributePath].[AttributeName]`, value, quality, timestamp.
- Alarm state stream messages: `[InstanceUniqueName].[AlarmName]`, state (active/normal), priority, timestamp.
- Central sends an unsubscribe request via ClusterClient when the debug session ends. The gRPC stream is cancelled. The site's `StreamRelayActor` is stopped and the SiteStreamManager subscription is removed.
- Central sends an unsubscribe request over `SiteCommandService` (gRPC command/control) when the debug session ends. The gRPC stream is cancelled. The site's `StreamRelayActor` is stopped and the SiteStreamManager subscription is removed.
- The stream is session-based and temporary.
- **Accepted limitation (central-side session locality):** debug sessions are process-local state on the central node hosting the `DebugStreamBridgeActor`. A central **restart or failover drops all active sessions** with no `OnStreamTerminated` signal to the operator — the engineer simply re-establishes the debug session from the UI. Debug streaming is an interactive, transient diagnostic aid, so this is an accepted trade-off rather than a durability gap (arch review 02, U7).
- **Backpressure is lossy-by-design and now observable:** the per-session `Channel<SiteStreamEvent>` (bounded 1000, `DropOldest`) silently evicts the oldest event when a slow consumer falls behind. The eviction is counted via the channel's `itemDropped` callback and logged at Warning (first eviction, then every 500th) so real event loss on a debug stream is visible instead of silent.
@@ -89,7 +89,7 @@ Delivered 2026-07-10 (`docs/plans/2026-07-10-aggregated-live-alarm-stream-plan.m
#### Central-Side Debug Stream Components
- **DebugStreamService**: Singleton service that manages debug stream sessions. Resolves instance ID to unique name and site, creates and tears down `DebugStreamBridgeActor` instances, and provides a clean API for both Blazor components and the SignalR hub. Injects `SiteStreamGrpcClientFactory` for gRPC stream creation.
- **DebugStreamBridgeActor**: One per active debug session. Opens a gRPC streaming subscription via `SiteStreamGrpcClient` and receives real-time events via callback. Also receives the initial `DebugViewSnapshot` via ClusterClient. Forwards all events to the consumer via callbacks. Handles gRPC stream errors with reconnection logic: tries the other site node endpoint, retries with backoff (max 3 retries), terminates the session if all retries fail.
- **DebugStreamBridgeActor**: One per active debug session. Opens a gRPC streaming subscription via `SiteStreamGrpcClient` and receives real-time events via callback. Also receives the initial `DebugViewSnapshot` over gRPC command/control (`SiteCommandService`). Forwards all events to the consumer via callbacks. Handles gRPC stream errors with reconnection logic: tries the other site node endpoint, retries with backoff (max 3 retries), terminates the session if all retries fail.
- **SiteStreamGrpcClient**: Per-site gRPC client that manages `GrpcChannel` instances and streaming subscriptions. Reads from the gRPC response stream in a background task, converts protobuf messages to domain events, and invokes the `onEvent` callback.
- **SiteStreamGrpcClientFactory**: Caches per-site `SiteStreamGrpcClient` instances. Reads `GrpcNodeAAddress` / `GrpcNodeBAddress` from the `Site` entity (loaded by `CentralCommunicationActor`). Falls back to NodeB if NodeA connection fails. Disposes clients on site removal or address change.
- **DebugStreamHub**: SignalR hub at `/hubs/debug-stream` for external consumers (e.g., CLI). Authenticates via Basic Auth + LDAP and requires the **Deployment** role. Server-to-client methods: `OnSnapshot`, `OnAttributeChanged`, `OnAlarmChanged`, `OnStreamTerminated`.
@@ -98,10 +98,10 @@ Delivered 2026-07-10 (`docs/plans/2026-07-10-aggregated-live-alarm-stream-plan.m
The streaming protocol is defined in `sitestream.proto` (`src/ZB.MOM.WW.ScadaBridge.Communication/Protos/sitestream.proto`):
- **Service**: `SiteStreamService` — hosted on each site node by `SiteStreamGrpcServer` — exposes six RPCs. Two are real-time **server-streaming** subscriptions (`SubscribeInstance`, `SubscribeSite`); the other four are **unary request/response** calls added by the Audit Log (#23) and Site Call Audit (#22) components. A unary call is request/response and is distinct from the command/control ClusterClient channel — gRPC on this service is no longer real-time-stream-only:
- **Service**: `SiteStreamService` — hosted on each site node by `SiteStreamGrpcServer` — exposes six RPCs. Two are real-time **server-streaming** subscriptions (`SubscribeInstance`, `SubscribeSite`); the other four are **unary request/response** calls added by the Audit Log (#23) and Site Call Audit (#22) components. These are distinct from the command/control gRPC services (`CentralControlService` on central, `SiteCommandService` on the site) — `SiteStreamService` is the streaming + audit-pull surface, not the command/control channel:
- `SubscribeInstance(InstanceStreamRequest) returns (stream SiteStreamEvent)` — the per-instance real-time debug stream (§6).
- `SubscribeSite(SiteStreamRequest) returns (stream SiteStreamEvent)` — the **site-wide, alarm-only** aggregated stream (§6.1) added for the operator Alarm Summary live cache. Additive (new RPC + new `SiteStreamRequest { correlation_id }` message; no field renumbering). The server handler mirrors `SubscribeInstance` — same bounded `Channel(1000, DropOldest)`, `GrpcMaxConcurrentStreams`/`GrpcMaxStreamLifetime` limits, `SiteConnectionOpened/Closed` telemetry, and `StreamRelayActor` mapping — but subscribes via `SiteStreamManager.SubscribeSiteAlarms` (all instances, `AlarmStateChanged` only; attribute events dropped, no per-instance filter).
- `IngestAuditEvents(AuditEventBatch) returns (IngestAck)` — central-side **ingest** receiving surface for Audit Log (#23) telemetry; routes the batch to the central `AuditLogIngestActor` proxy and returns the accepted `EventId`s. (The production *push* path is still ClusterClient via `ClusterClientSiteAuditClient`; this RPC is the gRPC-receiving counterpart.)
- `IngestAuditEvents(AuditEventBatch) returns (IngestAck)` a legacy central-side ingest surface on the *site*-hosted service; it is **dead in the shipped topology** because no site ever dials a site for ingest. The production audit-telemetry *push* path is site→central gRPC to `CentralControlService`, which routes the batch to the central `AuditLogIngestActor` proxy and returns the accepted `EventId`s.
- `IngestCachedTelemetry(CachedTelemetryBatch) returns (IngestAck)` — ingest receiving surface for the combined cached-call telemetry packet (audit row + `SiteCalls` operational upsert written in one transaction).
- `PullAuditEvents(PullAuditEventsRequest) returns (PullAuditEventsResponse)` — central→site **reconciliation pull** for the Audit Log self-heal feed; the site serves `Pending`/`Forwarded` rows from its `ISiteAuditQueue`.
- `PullSiteCalls(PullSiteCallsRequest) returns (PullSiteCallsResponse)` — central→site reconciliation pull for the Site Call Audit (#22) self-heal feed; the site serves operation-tracking rows changed since a cursor from its `IOperationTrackingStore`. A separate RPC from `PullAuditEvents` because the tracking store is the operational source of truth, distinct from the site audit queue.
@@ -205,15 +205,15 @@ Keepalive settings are configurable via `CommunicationOptions`:
- The site **Store-and-Forward Engine** sends a `NotificationSubmit` message to central carrying the notification — `NotificationId`, target list name, subject, body, and source provenance.
- Central ingests the submission with an insert-if-not-exists on `NotificationId` and acknowledges **after the row is persisted** to the `Notifications` table in the central configuration database. The site S&F engine clears the buffered message only on that ack.
- The `NotificationId` GUID — generated at the site — is the **idempotency key**. The handoff is at-least-once: a re-sent submission after a lost ack is harmless because central's insert-if-not-exists treats the duplicate as a no-op.
- **Transport**: ClusterClient (site→central command/control), consistent with how other site→central messages are sent.
- **Transport**: gRPC to `CentralControlService` (site→central command/control), consistent with how other site→central messages are sent.
### 10. Cached Call Telemetry (Site → Central)
- **Pattern**: Fire-and-forget telemetry with a periodic reconciliation pull.
- The site **Store-and-Forward Engine** emits a `CachedCallTelemetry` message to central on **every** cached-call lifecycle transition (`Pending → Retrying → Delivered / Parked / Failed / Discarded`). The first telemetry event for an operation carries its initial status — `Pending` when a transient failure has buffered the call, or directly `Delivered`/`Failed` for a cached call that never buffers. The message carries the `TrackedOperationId`, source site, `Kind` (the `TrackedOperationKind` enum), target summary, status, retry count, last error, key timestamps, and source provenance.
- Emission is **best-effort and at-least-once**, **idempotent on `TrackedOperationId`** — central's Site Call Audit component ingests with insert-if-not-exists then upsert-on-newer-status, so a re-sent or out-of-order event is harmless.
- **Reconciliation pull**: because telemetry is best-effort, the central **Site Call Audit** component periodically — and on site reconnect — pulls the changed rows back from each site over the **`PullSiteCalls` unary gRPC RPC** on `SiteStreamService` (not a ClusterClient round-trip). Central sends a `PullSiteCallsRequest` (`since_utc` cursor + `batch_size`); the site reads its `IOperationTrackingStore` and replies with a `PullSiteCallsResponse` carrying the matching operation-tracking rows (as `SiteCallOperationalDto`s) plus a `more_available` flag that signals a saturated batch so central advances the cursor and pulls again. Any telemetry missed during a disconnect self-heals through this pull. The Audit Log (#23) reconciliation feed uses the sibling `PullAuditEvents` RPC the same way.
- **Reconciliation pull**: because telemetry is best-effort, the central **Site Call Audit** component periodically — and on site reconnect — pulls the changed rows back from each site over the **`PullSiteCalls` unary gRPC RPC** on `SiteStreamService` (a distinct surface from the `CentralControlService`/`SiteCommandService` command/control path). Central sends a `PullSiteCallsRequest` (`since_utc` cursor + `batch_size`); the site reads its `IOperationTrackingStore` and replies with a `PullSiteCallsResponse` carrying the matching operation-tracking rows (as `SiteCallOperationalDto`s) plus a `more_available` flag that signals a saturated batch so central advances the cursor and pulls again. Any telemetry missed during a disconnect self-heals through this pull. The Audit Log (#23) reconciliation feed uses the sibling `PullAuditEvents` RPC the same way.
- Central audit is an **eventually-consistent mirror** — the site's operation tracking table remains the source of truth for cached-call status (`Tracking.Status(id)` is always answered site-locally).
- **Transport**: the *push* telemetry emission rides **ClusterClient** (site→central command/control), consistent with how other site→central messages are sent; the *reconciliation pull* rides the **gRPC** unary `PullSiteCalls` RPC (central→site request/response). The two paths are complementary — push is the fast, best-effort feed; pull is the slower self-heal backfill.
- **Transport**: the *push* telemetry emission rides **gRPC to `CentralControlService`** (site→central command/control), consistent with how other site→central messages are sent; the *reconciliation pull* rides the **gRPC** unary `PullSiteCalls` RPC on `SiteStreamService` (central→site request/response). The two paths are complementary — push is the fast, best-effort feed; pull is the slower self-heal backfill.
## Topology
@@ -221,37 +221,38 @@ Keepalive settings are configurable via `CommunicationOptions`:
%%{init: {'theme':'base', 'themeVariables': {'textColor':'#111111','lineColor':'#555555','edgeLabelBackground':'#ffffff','fontSize':'15px'}}}%%
flowchart LR
subgraph Central["Central Cluster"]
CCA["ClusterClient<br/>(command/control)"]
CCB["ClusterClient<br/>(command/control)"]
CCN["ClusterClient<br/>(command/control)"]
CCS["CentralControlService<br/>(gRPC server — site→central cmd/ctrl)"]
SCA["GrpcSiteTransport → SiteCommandService<br/>(gRPC client, per site)"]
SCB["GrpcSiteTransport → SiteCommandService"]
SCN["GrpcSiteTransport → SiteCommandService"]
GRPCC["SiteStreamGrpcClient<br/>(real-time data)"]
end
subgraph SiteA["Site A Cluster"]
SACOMM["SiteCommunicationActor<br/>(via Receptionist)"]
SACMD["SiteCommandService<br/>(gRPC server — central→site cmd/ctrl)"]
SAGRPC["SiteStreamGrpcServer<br/>(Kestrel HTTP/2, port 8083)"]
SACC["ClusterClient to Central<br/>(CentralCommunicationActor)"]
SACC["GrpcCentralTransport → CentralControlService<br/>(gRPC client)"]
end
subgraph SiteB["Site B Cluster"]
SBCOMM["SiteCommunicationActor<br/>(via Receptionist)"]
SBCMD["SiteCommandService"]
SBGRPC["SiteStreamGrpcServer"]
end
subgraph SiteN["Site N Cluster"]
SNCOMM["SiteCommunicationActor<br/>(via Receptionist)"]
SNCMD["SiteCommandService"]
SNGRPC["SiteStreamGrpcServer"]
end
CCA -->|command/control| SACOMM
CCB -->|command/control| SBCOMM
CCN -->|command/control| SNCOMM
SCA -->|"gRPC command/control"| SACMD
SCB -->|"gRPC command/control"| SBCMD
SCN -->|"gRPC command/control"| SNCMD
SAGRPC -->|"gRPC stream (real-time data)"| GRPCC
SBGRPC -->|gRPC stream| GRPCC
SNGRPC -->|gRPC stream| GRPCC
SACC -.->|command/control| Central
SACC -.->|"gRPC command/control"| CCS
NOTE["Sites do NOT communicate with each other.<br/>All inter-cluster communication flows through Central."]
@@ -260,7 +261,7 @@ flowchart LR
classDef dec fill:#fff2cc,stroke:#d6b656,color:#111111;
classDef alt fill:#e1d5e7,stroke:#9673a6,color:#111111;
classDef muted fill:#f5f5f5,stroke:#999999,color:#666666;
class CCA,CCB,CCN,SACOMM,SACC,SBCOMM,SNCOMM dec
class CCS,SCA,SCB,SCN,SACMD,SACC,SBCMD,SNCMD dec
class GRPCC,SAGRPC,SBGRPC,SNGRPC start
class NOTE muted
class Central proc
@@ -269,24 +270,24 @@ flowchart LR
- Sites do **not** communicate with each other.
- All inter-cluster communication flows through central.
- Both **CentralCommunicationActor** and **SiteCommunicationActor** are registered with their cluster's **ClusterClientReceptionist** for cross-cluster discovery.
- Command/control uses gRPC in both directions: central dials each site's **SiteCommandService** (per-site NodeA→NodeB failover channel pair), and each site dials the central **CentralControlService** (sticky central-a→central-b failover channel pair). There is no ClusterClientReceptionist and no cross-cluster actor discovery — the endpoints are dialled directly.
## Site Address Resolution
Central discovers site addresses through the **configuration database**, not runtime registration:
- Each site record in the Sites table includes optional **NodeAAddress** and **NodeBAddress** fields containing base Akka addresses of the site's cluster nodes (e.g., `akka.tcp://scadabridge@host:port`), and optional **GrpcNodeAAddress** and **GrpcNodeBAddress** fields containing gRPC endpoints (e.g., `http://host:8083`).
- The **CentralCommunicationActor** loads all site addresses from the database at startup and creates one **ClusterClient per site**, configured with both NodeA and NodeB as contact points. The **SiteStreamGrpcClientFactory** uses `GrpcNodeAAddress` / `GrpcNodeBAddress` to create per-site gRPC channels for streaming.
- The address cache is **refreshed every 60 seconds** and **on-demand** when site records are added, edited, or deleted via the Central UI or CLI. ClusterClient instances are recreated when contact points change.
- When routing a message to a site, central sends via `ClusterClient.Send("/user/site-communication", msg)`. **ClusterClient handles failover between NodeA and NodeB internally** — there is no application-level NodeA preference/NodeB fallback logic.
- Each site record in the Sites table includes optional **GrpcNodeAAddress** and **GrpcNodeBAddress** fields containing the site nodes' gRPC endpoints (e.g., `http://host:8083`), used for **both** central→site command/control (`SiteCommandService`) and site→central streaming (`SiteStreamService`). The legacy **NodeAAddress** / **NodeBAddress** Akka fields are retained but no longer drive cross-cluster dialling — Akka remoting is now intra-cluster only.
- The **CentralCommunicationActor** loads all site addresses from the database at startup and builds a per-site gRPC-endpoint cache. `GrpcSiteTransport` uses it (via a `SitePairChannelProvider`) to dial each site's `SiteCommandService` as a **per-site NodeANodeB failover channel pair**; the **SiteStreamGrpcClientFactory** uses the same `GrpcNodeAAddress` / `GrpcNodeBAddress` for per-site streaming channels.
- The address cache is **refreshed every 60 seconds** and **on-demand** when site records are added, edited, or deleted via the Central UI or CLI. Channels are rebuilt when a site's gRPC endpoints change.
- When routing a command to a site, central calls `SiteCommandService` on the site's active gRPC endpoint; on error it flips to the other node endpoint. The failover is an explicit NodeA→NodeB channel-pair flip, not internal transport routing.
- **Heartbeats** from sites serve **health monitoring only** — they do not serve as a registration or address discovery mechanism.
- If no addresses are configured for a site, messages to that site are **dropped** and the caller's Ask times out.
- If no gRPC endpoints are configured for a site, commands to that site fail and the caller's call times out.
### Site → Central Communication
- Site nodes configure a list of **CentralContactPoints** (both central node addresses) instead of a single `CentralActorPath`.
- The site creates a **ClusterClient** using the central contact points and sends heartbeats, health reports, and other messages via `ClusterClient.Send("/user/central-communication", msg)`.
- ClusterClient handles automatic failover between central nodes — if the active central node goes down, the site's ClusterClient reconnects to the standby node transparently.
- Site nodes configure a list of **CentralGrpcEndpoints** — the gRPC h2c URLs of the central nodes (e.g., `http://scadabridge-central-a:8083`, on the central's `CentralGrpcPort`, default 8083) — replacing the former `CentralContactPoints` list of Akka addresses. A site node must list at least one; central nodes leave it empty (they host `CentralControlService`, they do not dial it). Note these are the central nodes' **direct** gRPC endpoints, not Traefik (which is HTTP/1 only).
- `GrpcCentralTransport` dials `CentralControlService` over these endpoints as a **sticky central-a→central-b failover channel pair**, and the site sends heartbeats, health reports, notification submit/status, audit/telemetry ingest, and reconcile over it.
- If the active central node goes down, the site's transport flips to the standby central endpoint transparently.
## Message Timeouts
@@ -301,17 +302,17 @@ Each request/response pattern has a default timeout that can be overridden in co
| 5. Recipe/Command Delivery | 30 seconds | Fire-and-forget with ack |
| 8. Remote Queries | 30 seconds | Querying parked messages or event logs |
| 9. Notification Submission | 30 seconds | Fire-and-forget with ack; central acks after persisting the row |
| 10. Cached Call Telemetry | 30 seconds | Telemetry emission (ClusterClient) is fire-and-forget; the reconciliation pull is the unary gRPC `PullSiteCalls` request/response (its deadline is the gRPC call timeout, not the Akka ask) |
| 10. Cached Call Telemetry | 30 seconds | Telemetry emission (gRPC to `CentralControlService`) is fire-and-forget; the reconciliation pull is the unary gRPC `PullSiteCalls` request/response (its deadline is the gRPC call timeout) |
Timeouts use the Akka.NET **ask pattern**. If no response is received within the timeout, the caller receives a timeout failure.
Command/control request/response now uses **gRPC calls with a deadline** in place of the Akka ask; the per-pattern values above are applied as the gRPC call timeout. If no response is received within the timeout, the caller receives a timeout failure.
## Transport Configuration
Akka.NET remoting provides the underlying transport for both intra-cluster communication and ClusterClient connections. The following transport-level settings are **explicitly configured** (not left to framework defaults) for predictable behavior:
Akka.NET remoting provides the underlying transport for **intra-cluster** (pair-internal) communication only; it no longer crosses the site↔central boundary. Cross-cluster command/control, streaming, and audit pulls all ride gRPC. The following settings apply to each concern:
- **Transport heartbeat interval**: Configurable interval at which heartbeat messages are sent over remoting connections (e.g., every 5 seconds).
- **Failure detection threshold**: Number of missed heartbeats before the connection is considered lost (e.g., 3 missed heartbeats = 15 seconds with a 5-second interval).
- **Reconnection**: ClusterClient handles reconnection and failover between contact points automatically for cross-cluster communication. No custom reconnection logic is required.
- **Transport heartbeat interval** (Akka remoting, intra-cluster): Configurable interval at which heartbeat messages are sent over remoting connections within a cluster (e.g., every 5 seconds).
- **Failure detection threshold** (Akka remoting, intra-cluster): Number of missed heartbeats before an intra-cluster connection is considered lost.
- **gRPC reconnection/failover**: Both command/control transports handle reconnection and node failover via their channel pairs — `GrpcCentralTransport` (site→central, central-a→central-b) and `GrpcSiteTransport` (central→site, NodeA→NodeB). No ClusterClient reconnection is involved.
These settings should be tuned for the expected network conditions between central and site clusters.
@@ -327,32 +328,32 @@ This provides protocol-level safety beyond Akka.NET's transport guarantees, whic
## Message Ordering
Akka.NET guarantees message ordering between a specific sender/receiver actor pair. The Communication Layer relies on this guarantee — messages to a given site are processed in the order they are sent. Callers do not need to handle out-of-order delivery.
Within a cluster, Akka.NET guarantees message ordering between a specific sender/receiver actor pair. Across the site↔central boundary, ordering is preserved by the single gRPC channel each transport holds to a given endpoint — commands to a given site over `SiteCommandService` are issued in order over that channel. Callers do not need to handle out-of-order delivery.
## ManagementActor and ClusterClient
## ManagementActor and cross-cluster access
The ManagementActor runs at the well-known path `/user/management` on central nodes. It is **not** advertised via ClusterClientReceptionist, and no ClusterClient reaches it.
The ManagementActor runs at the well-known path `/user/management` on central nodes. It is **not** advertised for cross-cluster access, and nothing outside the central cluster reaches it directly.
That registration existed until 2026-07-22 for an out-of-cluster CLI that was never built. The shipped CLI speaks **HTTP Basic to the central `/management` endpoints**; those endpoints ask the ManagementActor **in-process** through `ManagementActorHolder` (`ManagementEndpoints.cs`). Nothing in the repo ever sent to `/user/management` across the boundary, so the registration was deleted — leaving exactly one receptionist registration per cluster role (`CentralCommunicationActor` on central, `SiteCommunicationActor` on sites), both of which serve inter-cluster central↔site messaging and are themselves scheduled for removal by the gRPC transport migration (`docs/plans/2026-07-22-clusterclient-to-grpc-plan.md`).
A ClusterClientReceptionist registration for it existed until 2026-07-22, for an out-of-cluster CLI that was never built. The shipped CLI speaks **HTTP Basic to the central `/management` endpoints**; those endpoints ask the ManagementActor **in-process** through `ManagementActorHolder` (`ManagementEndpoints.cs`). Nothing in the repo ever sent to `/user/management` across the boundary, so the registration was deleted. The two remaining receptionist registrations — `CentralCommunicationActor` on central and `SiteCommunicationActor` on sites — were subsequently **removed entirely in Phase 4 of the ClusterClient→gRPC migration** (2026-07-23): cross-cluster messaging is now gRPC (`CentralControlService` / `SiteCommandService`), reached by dialling the configured endpoints, so `ClusterClientReceptionist` is no longer used anywhere. `Akka.Cluster.Tools` remains a dependency for ClusterSingleton (`docs/plans/2026-07-22-clusterclient-to-grpc-plan.md`).
## Connection Failure Behavior
Disconnect is detected at the **transport layer**, never via an application-level signal from central. There is no `ConnectionStateChanged`-style synchronous notification: the central coordinator does not maintain a model of "this site is up / down" because the two transports already report unavailability at their natural cadence.
- **In-flight command/control messages (ClusterClient + Ask)**: When a connection drops while a request is in flight (e.g., a deployment sent but no response received), the Akka ask pattern times out and the caller receives a failure. There is **no automatic retry or buffering at central** — the engineer sees the failure in the UI and re-initiates the action. This is consistent with the design principle that central does not buffer messages. An in-progress deployment whose round-trip exceeds the Ask timeout (default 120 s at `CommunicationService.DeployInstanceAsync`) surfaces as `DeploymentStatus.Failed` to the caller.
- **In-flight command/control messages (gRPC call + deadline)**: When a connection drops while a request is in flight (e.g., a deployment sent but no response received), the gRPC call fails or hits its deadline and the caller receives a failure. There is **no automatic retry or buffering at central** — the engineer sees the failure in the UI and re-initiates the action. This is consistent with the design principle that central does not buffer messages. An in-progress deployment whose round-trip exceeds the timeout (default 120 s at `CommunicationService.DeployInstanceAsync`) surfaces as `DeploymentStatus.Failed` to the caller.
- **Debug streams (gRPC)**: Any gRPC stream interruption is detected by the HTTP/2 keepalive PING (~25 s) and triggers reconnection logic in the `DebugStreamBridgeActor`. The bridge actor attempts to reconnect to the other site node endpoint (NodeB if NodeA failed, or vice versa), with up to 3 retries and 5-second backoff. If all retries fail, the consumer is notified via `OnStreamTerminated` and the bridge actor is stopped. Events during the reconnection gap are lost (acceptable for real-time debug view). On successful reconnection, the consumer can request a fresh snapshot to re-sync state.
## Failover Behavior
- **Central failover**: The standby node takes over the Akka.NET cluster role. In-progress deployments are treated as failed. Site ClusterClients automatically reconnect to the standby central node via their configured contact points.
- **Site failover**: The standby node takes over. The Deployment Manager singleton restarts and re-creates the Instance Actor hierarchy. Central's per-site ClusterClient automatically reconnects to the surviving site node. Ongoing debug streams are interrupted and must be re-established by the engineer.
- **Central failover**: The standby node takes over the Akka.NET cluster role. In-progress deployments are treated as failed. Each site's `GrpcCentralTransport` flips to the standby central endpoint (`CentralGrpcEndpoints` pair) transparently.
- **Site failover**: The standby node takes over. The Deployment Manager singleton restarts and re-creates the Instance Actor hierarchy. Central's per-site `GrpcSiteTransport` flips to the surviving site node's gRPC endpoint. Ongoing debug streams are interrupted and must be re-established by the engineer.
## Dependencies
- **Akka.NET Remoting + ClusterClient**: Provides the command/control transport layer. ClusterClient/ClusterClientReceptionist used for cross-cluster command/control messaging (deployments, lifecycle, subscribe/unsubscribe handshake, snapshots).
- **gRPC (Grpc.AspNetCore + Grpc.Net.Client)**: Provides the real-time data streaming transport. Site nodes host a gRPC server (SiteStreamGrpcServer); central nodes create per-site gRPC clients (SiteStreamGrpcClient).
- **gRPC (Grpc.AspNetCore + Grpc.Net.Client)**: Provides the **only** cross-cluster transport — command/control (`CentralControlService` on central, `SiteCommandService` on sites), real-time streaming (`SiteStreamService`, site-hosted), and the audit-pull RPCs. Central hosts `CentralControlService` and dials each site's `SiteCommandService` (`GrpcSiteTransport`) + `SiteStreamService`; sites host `SiteCommandService` + `SiteStreamGrpcServer` and dial `CentralControlService` (`GrpcCentralTransport`).
- **Akka.NET Remoting + `Akka.Cluster.Tools`**: Remoting provides **intra-cluster** transport only. `Akka.Cluster.Tools` provides ClusterSingleton; ClusterClient/ClusterClientReceptionist are no longer used (removed in Phase 4).
- **Cluster Infrastructure**: Manages node roles and failover detection.
- **Configuration Database**: Provides site node addresses (NodeAAddress, NodeBAddress for Akka remoting; GrpcNodeAAddress, GrpcNodeBAddress for gRPC streaming) for address resolution.
- **Configuration Database**: Provides site node gRPC endpoints (GrpcNodeAAddress, GrpcNodeBAddress — used for both command/control and streaming) for address resolution; the legacy Akka NodeAAddress/NodeBAddress fields are retained but no longer drive cross-cluster dialling.
- **Site Runtime (SiteStreamManager)**: The SiteStreamGrpcServer subscribes to SiteStreamManager to receive real-time events for gRPC delivery.
- **`ISiteAuditQueue` (site-local)**: Handed to `SiteStreamGrpcServer` (post-construction, on site roles) so the `PullAuditEvents` RPC can read the site's `Pending`/`Forwarded` audit rows to serve the Audit Log (#23) reconciliation pull. Null when not wired (central-only host) — the handler then returns an empty response.
- **`IOperationTrackingStore` (site-local)**: Handed to `SiteStreamGrpcServer` (post-construction, on site roles) so the `PullSiteCalls` RPC can read operation-tracking rows changed since a cursor to serve the Site Call Audit (#22) reconciliation pull. Null when not wired — the handler returns an empty response.
@@ -364,8 +365,8 @@ Disconnect is detected at the **transport layer**, never via an application-leve
- **Site Runtime**: Receives deployments, lifecycle commands, and artifact updates. Provides debug view data.
- **Central UI**: Debug view requests and remote queries flow through communication.
- **Health Monitoring**: Receives periodic health reports from sites.
- **Store-and-Forward Engine (site)**: Parked message queries/commands are routed through communication. Also emits `CachedCallTelemetry` (push, ClusterClient) and serves the `PullSiteCalls` gRPC reconciliation pull from its `IOperationTrackingStore`, and receives relayed `RetryParkedOperation` / `DiscardParkedOperation` commands.
- **Store-and-Forward Engine (site)**: Parked message queries/commands are routed through communication. Also emits `CachedCallTelemetry` (push, gRPC to `CentralControlService`) and serves the `PullSiteCalls` gRPC reconciliation pull from its `IOperationTrackingStore`, and receives relayed `RetryParkedOperation` / `DiscardParkedOperation` commands.
- **Site Call Audit (central)**: Receives cached-call telemetry and issues the `PullSiteCalls` gRPC reconciliation pulls to sites; relays parked-operation Retry/Discard commands to sites through communication.
- **Audit Log (#23)**: Sites forward audit-event telemetry (push) and serve the `PullAuditEvents` gRPC reconciliation pull from their `ISiteAuditQueue`; the central `AuditLogIngestActor` is the ingest target for both the push path and the combined cached-call telemetry packet.
- **Site Event Logging**: Event log queries are routed through communication.
- **Management Service**: The ManagementActor is registered with ClusterClientReceptionist on central nodes. The CLI communicates with the ManagementActor via ClusterClient, which is a separate channel from inter-cluster remoting.
- **Management Service**: The ManagementActor runs at `/user/management` on central nodes and is reached **in-process** via `ManagementActorHolder`. The CLI communicates with it over the central HTTP `/management` endpoints (HTTP Basic), not over any cluster transport. It is no longer registered with ClusterClientReceptionist (that registration, and the whole ClusterClient surface, is gone).
+15 -10
View File
@@ -45,7 +45,7 @@ The Host must bind configuration sections from `appsettings.json` to strongly-ty
| Section | Options Class | Owner | Contents |
|---------|--------------|-------|----------|
| `ScadaBridge:Node` | `NodeOptions` | Host | Role, NodeHostname, SiteId, RemotingPort, GrpcPort (site only, default 8083) |
| `ScadaBridge:Node` | `NodeOptions` | Host | Role, NodeHostname, SiteId, RemotingPort, GrpcPort (site: hosts `SiteCommandService` + `SiteStreamGrpcServer`, default 8083), CentralGrpcPort (central: hosts `CentralControlService`, default 8083) |
| `ScadaBridge:Cluster` | `ClusterOptions` | ClusterInfrastructure | SeedNodes, SplitBrainResolverStrategy, StableAfter, HeartbeatInterval, FailureDetectionThreshold, MinNrOfMembers |
| `ScadaBridge:Database` | `DatabaseOptions` | Host | Central: ConfigurationDb, MachineDataDb connection strings; Site: SQLite paths |
@@ -113,7 +113,7 @@ The Host bootstraps the Akka.NET actor system from a **hand-assembled, injection
- **Coordinated shutdown**: `run-coordinated-shutdown-when-down = on` plus a cluster-leave phase timeout above the singleton drain budget (see REQ-HOST-4a / the down-if-alone recovery contract in Component-ClusterInfrastructure.md).
- **Actor registration**: each component's actors registered conditional on the node's role.
> **Bootstrap-mechanism note.** The `Akka.Hosting` / `Akka.Cluster.Hosting` / `Akka.Remote.Hosting` typed-builder packages were referenced but unused and were dropped (arch-review 01 Task 22). The hand-rolled HOCON path is a single hardened, production-shape code path with full test coverage; migrating to Akka.Hosting's typed options is deliberately **not** done here and is tracked as possible future work rather than carried as an unused dependency. Only `Akka.Cluster.Tools` (ClusterSingleton/ClusterClient, transitively Akka.Cluster) remains.
> **Bootstrap-mechanism note.** The `Akka.Hosting` / `Akka.Cluster.Hosting` / `Akka.Remote.Hosting` typed-builder packages were referenced but unused and were dropped (arch-review 01 Task 22). The hand-rolled HOCON path is a single hardened, production-shape code path with full test coverage; migrating to Akka.Hosting's typed options is deliberately **not** done here and is tracked as possible future work rather than carried as an unused dependency. Only `Akka.Cluster.Tools` (ClusterSingleton, transitively Akka.Cluster) remains — its ClusterClient/ClusterClientReceptionist surface is no longer used since Phase 4 moved cross-cluster messaging to gRPC.
> **Persistence note.** ScadaBridge does not use Akka.Persistence. Durable state
> (store-and-forward buffers, site event logs, static attribute writes,
@@ -123,13 +123,18 @@ The Host bootstraps the Akka.NET actor system from a **hand-assembled, injection
> `akka.persistence` section and references no persistence plugin. There are no
> `PersistentActor` subclasses in the system by design.
### REQ-HOST-6a: ClusterClientReceptionist (Central Only)
### REQ-HOST-6a: gRPC command/control services (no ClusterClientReceptionist)
On central nodes, the Host must configure the Akka.NET **ClusterClientReceptionist** and register the **CentralCommunicationActor** with it, so that site clusters' ClusterClients can reach the central command/control endpoint without joining the central cluster.
As of Phase 4 of the ClusterClient→gRPC migration (2026-07-23), the Host configures **no** `ClusterClientReceptionist` and registers **no** actor with one. Both remaining receptionist registrations were removed:
**The ManagementActor is NOT registered with the receptionist** (removed 2026-07-22). That registration was written for an out-of-cluster CLI that was never built: the shipped CLI speaks HTTP Basic to the central `/management` endpoints, which ask the ManagementActor **in-process** via `ManagementActorHolder` (`ManagementEndpoints.cs`). No sender to `/user/management` existed anywhere in the repo, so the registration only widened the cluster-client surface for nothing. The actor itself still runs at `/user/management`; only its cross-boundary advertisement is gone.
- The central **CentralCommunicationActor** registration is gone. Sites now reach central command/control over gRPC by dialling the central-hosted **`CentralControlService`** (`GrpcCentralTransport`, sticky central-a→central-b failover), not via a ClusterClient locating a receptionist.
- The site **SiteCommunicationActor** registration is gone. Central now reaches site command/control over gRPC by dialling the site-hosted **`SiteCommandService`** (`GrpcSiteTransport`, per-site NodeA→NodeB failover).
> **Migration note.** This receptionist registration — and the `CentralCommunicationActor` one that remains — are scheduled for deletion once the site↔central transport moves to gRPC. See `docs/plans/2026-07-22-clusterclient-to-grpc-plan.md`.
Central hosts `CentralControlService` (site dials in); each site hosts `SiteCommandService` and `SiteStreamGrpcServer` (central dials in). Endpoints are dialled directly from configuration/DB, so there is no cross-cluster actor discovery.
**The ManagementActor was never reachable over the receptionist either** (its registration was removed 2026-07-22, ahead of Phase 4). That registration was written for an out-of-cluster CLI that was never built: the shipped CLI speaks HTTP Basic to the central `/management` endpoints, which ask the ManagementActor **in-process** via `ManagementActorHolder` (`ManagementEndpoints.cs`). The actor itself still runs at `/user/management`.
> **Migration note.** `Akka.Cluster.Tools` stays a Host dependency for ClusterSingleton; only its ClusterClient/ClusterClientReceptionist surface is now unused. See `docs/plans/2026-07-22-clusterclient-to-grpc-plan.md`.
### REQ-HOST-7: ASP.NET Web Endpoints
@@ -138,7 +143,7 @@ On central nodes, the Host must use `WebApplication.CreateBuilder` to produce a
- Central UI (via `MapCentralUI()` extension method).
- Inbound API (via `MapInboundAPI()` extension method).
On site nodes, the Host must also use `WebApplication.CreateBuilder` (not `Host.CreateDefaultBuilder`) to host the **SiteStreamGrpcServer** via Kestrel HTTP/2 on the configured `GrpcPort` (default 8083). Kestrel is configured with `HttpProtocols.Http2` on the gRPC port only — no HTTP/1.1 web endpoints are exposed. The gRPC service is mapped via `MapGrpcService<SiteStreamGrpcServer>()`.
On site nodes, the Host must also use `WebApplication.CreateBuilder` (not `Host.CreateDefaultBuilder`) to host the **SiteStreamGrpcServer** and the **SiteCommandService** (central→site command/control, added in Phase 4) via Kestrel HTTP/2 on the configured `GrpcPort` (default 8083). Kestrel is configured with `HttpProtocols.Http2` on the gRPC port only — no HTTP/1.1 web endpoints are exposed. On central nodes, the Host additionally hosts the **CentralControlService** (site→central command/control) via Kestrel HTTP/2 on the configured `CentralGrpcPort` (default 8083), alongside the HTTP/1.1 Central UI / Inbound API endpoints. The gRPC services are mapped via `MapGrpcService<>()`.
**Startup ordering (site nodes)**:
1. Actor system and SiteStreamManager must be initialized before gRPC begins accepting connections.
@@ -174,7 +179,7 @@ Each component library must expose its services to the Host via a consistent set
- `AkkaConfigurationBuilder.AddXxxActors()` — registers the component's actors with the Akka.NET actor system (for components that have actors).
- `WebApplication.MapXxx()` — maps the component's web endpoints (only for CentralUI and InboundAPI).
The Host's `Program.cs` calls these extension methods; the component libraries own the registration logic. This keeps the Host thin and each component self-contained. The ManagementService component additionally registers the ManagementActor with ClusterClientReceptionist in its `AddManagementServiceActors()` method.
The Host's `Program.cs` calls these extension methods; the component libraries own the registration logic. This keeps the Host thin and each component self-contained. (The ManagementService component no longer registers the ManagementActor with ClusterClientReceptionist — that registration was removed 2026-07-22, and the whole receptionist surface went in Phase 4.)
---
@@ -209,7 +214,7 @@ The Host's `Program.cs` calls these extension methods; the component libraries o
## Dependencies
- **All 19 component libraries**: The Host references every component project to call their extension methods (excludes CLI, which is a separate executable). Audit Log (#23) ships its central+site code in `ZB.MOM.WW.ScadaBridge.AuditLog`; the Host calls `AddAuditLog()` on both roles, M2+ will add `AddAuditLogActors()`.
- **Akka.Cluster.Tools**: ClusterSingleton (singleton manager/proxy) and ClusterClient/ClusterClientReceptionist; transitively pulls Akka.Cluster (SBR provider) and Akka.Remote. The actor system is built from hand-assembled HOCON (REQ-HOST-6), so the `Akka.Hosting`/`Akka.Remote.Hosting`/`Akka.Cluster.Hosting` typed-builder packages are **not** referenced (dropped in arch-review 01 Task 22). No Akka.Persistence plugin — see the Persistence note under REQ-HOST-6.
- **Akka.Cluster.Tools**: ClusterSingleton (singleton manager/proxy); transitively pulls Akka.Cluster (SBR provider) and Akka.Remote. Its ClusterClient/ClusterClientReceptionist surface is no longer used — cross-cluster messaging is gRPC as of Phase 4. The actor system is built from hand-assembled HOCON (REQ-HOST-6), so the `Akka.Hosting`/`Akka.Remote.Hosting`/`Akka.Cluster.Hosting` typed-builder packages are **not** referenced (dropped in arch-review 01 Task 22). No Akka.Persistence plugin — see the Persistence note under REQ-HOST-6.
- **Serilog.AspNetCore**: For structured logging integration.
- **Microsoft.Extensions.Hosting.WindowsServices**: For Windows Service support.
- **ASP.NET Core** (central only): For web endpoint hosting.
@@ -220,5 +225,5 @@ The Host's `Program.cs` calls these extension methods; the component libraries o
- **Configuration Database**: The Host registers the DbContext and wires repository implementations to their interfaces. In development, triggers auto-migration; in production, validates schema version.
- **ClusterInfrastructure**: The Host configures the underlying Akka.NET cluster that ClusterInfrastructure manages at runtime.
- **CentralUI / InboundAPI**: The Host maps their web endpoints into the ASP.NET Core pipeline on central nodes.
- **ManagementService**: The Host registers the ManagementActor and configures ClusterClientReceptionist on central nodes, enabling CLI access.
- **ManagementService**: The Host registers the ManagementActor on central nodes; the CLI reaches it over the HTTP `/management` endpoints (in-process via `ManagementActorHolder`), not via any cluster transport. No ClusterClientReceptionist is configured (removed in Phase 4).
- **HealthMonitoring**: The Host's startup validation and logging configuration provide the foundation for health reporting.
@@ -134,7 +134,7 @@ Each buffered message stores:
## Dependencies
- **SQLite**: Local persistence on each node.
- **Communication Layer**: Application-level replication to standby node; remote query handling from central; carries buffered notifications to the central cluster (ClusterClient) and receives central's acks.
- **Communication Layer**: Application-level replication to standby node; remote query handling from central; carries buffered notifications to the central cluster (gRPC to `CentralControlService`) and receives central's acks.
- **External System Gateway**: Delivers external system API calls.
- **CentralSite Communication**: The delivery target for the notification category — a buffered notification is forwarded to the central cluster over CentralSite Communication and cleared on central's ack. Also carries `CachedCallTelemetry` and reconciliation responses to central, and receives `RetryParkedOperation` / `DiscardParkedOperation` commands.
- **Site Call Audit**: The central audit mirror for cached calls — receives this engine's cached-call telemetry and reconciliation responses, and relays operator Retry/Discard of parked cached calls back as commands.
@@ -1,185 +0,0 @@
using Akka.Actor;
using Akka.Cluster.Tools.Client;
using Akka.Event;
using ZB.MOM.WW.ScadaBridge.Commons.Messages.Audit;
using ZB.MOM.WW.ScadaBridge.Commons.Messages.Deployment;
using ZB.MOM.WW.ScadaBridge.Commons.Messages.Health;
using ZB.MOM.WW.ScadaBridge.Commons.Messages.Notification;
namespace ZB.MOM.WW.ScadaBridge.Communication.Actors;
/// <summary>
/// The <see cref="ICentralTransport"/> that carries the seven site→central sends over Akka
/// <c>ClusterClient</c> — the transport in production today, and the default. Every method is a
/// verbatim lift of the corresponding <c>SiteCommunicationActor</c> send block: it forwards a
/// <see cref="ClusterClient.Send"/> to <c>/user/central-communication</c> with the
/// <paramref name="replyTo"/> as the send's sender, so central's reply routes straight back to the
/// waiting Ask rather than through the site communication actor.
/// </summary>
/// <remarks>
/// The <c>ClusterClient</c> reference arrives after construction via
/// <see cref="SetCentralClient"/> (the actor forwards the <c>RegisterCentralClient</c> message it
/// receives once the Host builds the client). Until then — and if central contact points are not
/// configured at all — the client is null and each method answers the same transient-failure reply
/// the old inline handlers did.
/// </remarks>
public sealed class AkkaCentralTransport : ICentralTransport
{
/// <summary>The receptionist-registered path of the central communication actor.</summary>
private const string CentralPath = "/user/central-communication";
private readonly ILoggingAdapter? _log;
private IActorRef? _centralClient;
/// <summary>Creates the transport with no logging adapter (behaviourally identical; warnings are dropped).</summary>
public AkkaCentralTransport()
{
}
/// <summary>Creates the transport bound to the site communication actor's logging adapter.</summary>
/// <param name="log">Logging adapter used for the "no ClusterClient registered" warnings.</param>
public AkkaCentralTransport(ILoggingAdapter log)
{
_log = log;
}
/// <summary>
/// Registers the central <c>ClusterClient</c> once the Host has built it. Called from the site
/// communication actor's <c>RegisterCentralClient</c> handler.
/// </summary>
/// <param name="centralClient">The ClusterClient reaching the central cluster.</param>
public void SetCentralClient(IActorRef centralClient)
{
_centralClient = centralClient;
_log?.Info("Registered central ClusterClient");
}
/// <inheritdoc />
public void SubmitNotification(NotificationSubmit message, IActorRef replyTo)
{
if (_centralClient == null)
{
// No ClusterClient registered yet (e.g. central contact points not
// configured, or registration not yet completed). A non-accepted ack
// makes the S&F forwarder treat this as transient and retry later.
_log?.Warning(
"Cannot forward NotificationSubmit {0} — no central ClusterClient registered",
message.NotificationId);
replyTo.Tell(new NotificationSubmitAck(
message.NotificationId, Accepted: false, Error: "Central ClusterClient not registered"));
return;
}
_log?.Debug("Forwarding NotificationSubmit {0} to central", message.NotificationId);
_centralClient.Tell(new ClusterClient.Send(CentralPath, message), replyTo);
}
/// <inheritdoc />
public void QueryNotificationStatus(NotificationStatusQuery message, IActorRef replyTo)
{
if (_centralClient == null)
{
// No ClusterClient registered yet. Reply Found: false so Notify.Status
// falls back to the site S&F buffer to decide Forwarding vs Unknown.
_log?.Warning(
"Cannot forward NotificationStatusQuery {0} — no central ClusterClient registered",
message.NotificationId);
replyTo.Tell(new NotificationStatusResponse(
message.CorrelationId, Found: false, Status: "Unknown",
RetryCount: 0, LastError: null, DeliveredAt: null));
return;
}
_log?.Debug("Forwarding NotificationStatusQuery {0} to central", message.NotificationId);
_centralClient.Tell(new ClusterClient.Send(CentralPath, message), replyTo);
}
/// <inheritdoc />
public void IngestAuditEvents(IngestAuditEventsCommand message, IActorRef replyTo)
{
if (_centralClient == null)
{
// No ClusterClient registered yet. Faulting the Ask makes the
// SiteAuditTelemetryActor drain loop treat this as transient and keep
// the rows Pending for the next tick.
_log?.Warning(
"Cannot forward IngestAuditEventsCommand ({0} events) — no central ClusterClient registered",
message.Events.Count);
replyTo.Tell(new Status.Failure(
new InvalidOperationException("Central ClusterClient not registered")));
return;
}
_log?.Debug("Forwarding IngestAuditEventsCommand ({0} events) to central", message.Events.Count);
_centralClient.Tell(new ClusterClient.Send(CentralPath, message), replyTo);
}
/// <inheritdoc />
public void IngestCachedTelemetry(IngestCachedTelemetryCommand message, IActorRef replyTo)
{
if (_centralClient == null)
{
_log?.Warning(
"Cannot forward IngestCachedTelemetryCommand ({0} entries) — no central ClusterClient registered",
message.Entries.Count);
replyTo.Tell(new Status.Failure(
new InvalidOperationException("Central ClusterClient not registered")));
return;
}
_log?.Debug("Forwarding IngestCachedTelemetryCommand ({0} entries) to central", message.Entries.Count);
_centralClient.Tell(new ClusterClient.Send(CentralPath, message), replyTo);
}
/// <inheritdoc />
public void ReconcileSite(ReconcileSiteRequest message, IActorRef replyTo)
{
if (_centralClient == null)
{
// No ClusterClient registered yet. Faulting the Ask makes the
// SiteReconciliationActor treat the pass as best-effort-failed; it
// logs a warning and retries reconcile on the next node startup.
_log?.Warning(
"Cannot forward ReconcileSiteRequest for site {0} node {1} — no central ClusterClient registered",
message.SiteIdentifier, message.NodeId);
replyTo.Tell(new Status.Failure(
new InvalidOperationException("Central ClusterClient not registered")));
return;
}
_log?.Debug(
"Forwarding ReconcileSiteRequest for site {0} node {1} ({2} local instance(s)) to central",
message.SiteIdentifier, message.NodeId, message.LocalNameToRevisionHash.Count);
_centralClient.Tell(new ClusterClient.Send(CentralPath, message), replyTo);
}
/// <inheritdoc />
public void ReportSiteHealth(SiteHealthReport message, IActorRef replyTo)
{
if (_centralClient == null)
{
// No ClusterClient registered yet. A non-accepted ack makes the
// sender's counter-restore path treat this tick as a loss.
_log?.Warning(
"Cannot forward SiteHealthReport #{0} — no central ClusterClient registered",
message.SequenceNumber);
replyTo.Tell(new SiteHealthReportAck(
message.SiteId, message.SequenceNumber, Accepted: false,
Error: "Central ClusterClient not registered"));
return;
}
_centralClient.Tell(new ClusterClient.Send(CentralPath, message), replyTo);
}
/// <inheritdoc />
public void SendHeartbeat(HeartbeatMessage message, IActorRef self)
{
if (_centralClient == null)
{
return;
}
_centralClient.Tell(new ClusterClient.Send(CentralPath, message), self);
}
}
@@ -1,9 +1,8 @@
using System.Collections.Immutable;
using Akka.Actor;
using Akka.Cluster.Tools.Client;
using Akka.Cluster.Tools.PublishSubscribe;
using Akka.Event;
using Microsoft.Extensions.DependencyInjection;
using Microsoft.Extensions.Logging;
using ZB.MOM.WW.ScadaBridge.Commons.Interfaces.Repositories;
using ZB.MOM.WW.ScadaBridge.Commons.Messages.Audit;
using ZB.MOM.WW.ScadaBridge.Commons.Messages.Communication;
@@ -15,61 +14,10 @@ using ZB.MOM.WW.ScadaBridge.HealthMonitoring;
namespace ZB.MOM.WW.ScadaBridge.Communication.Actors;
/// <summary>
/// Abstraction for creating ClusterClient instances per site, enabling testability.
/// </summary>
public interface ISiteClientFactory
{
/// <summary>Creates a ClusterClient actor for the given site with the specified contact points.</summary>
/// <param name="system">The actor system in which to create the client.</param>
/// <param name="siteId">The site identifier, used to name the actor.</param>
/// <param name="contacts">The set of receptionist actor paths to use as initial contacts.</param>
/// <returns>An actor reference for the new ClusterClient.</returns>
IActorRef Create(ActorSystem system, string siteId, ImmutableHashSet<ActorPath> contacts);
}
/// <summary>
/// Default implementation that creates a real ClusterClient for each site.
/// </summary>
public class DefaultSiteClientFactory : ISiteClientFactory
{
/// <summary>
/// Per-incarnation generation counter. Context.Stop of the previous client is
/// asynchronous — its actor name stays reserved until termination completes —
/// so a same-named recreate in the same message handling throws
/// InvalidActorNameException. A generation suffix makes every incarnation's
/// name unique by construction (mirrors SiteStreamGrpcServer._actorCounter).
/// </summary>
private long _generation;
/// <inheritdoc />
public IActorRef Create(ActorSystem system, string siteId, ImmutableHashSet<ActorPath> contacts)
{
var settings = ClusterClientSettings.Create(system).WithInitialContacts(contacts);
var name = $"site-client-{SanitizeForActorName(siteId)}-{System.Threading.Interlocked.Increment(ref _generation)}";
return system.ActorOf(ClusterClient.Props(settings), name);
}
/// <summary>
/// Maps an arbitrary SiteIdentifier onto a valid Akka actor-path element:
/// letters/digits/'-'/'_' pass through, everything else becomes '_'. Uniqueness
/// is NOT required here (the generation suffix guarantees it); only validity is.
/// </summary>
/// <param name="siteId">The SiteIdentifier to sanitize.</param>
/// <returns>A valid Akka actor-path element derived from <paramref name="siteId"/>.</returns>
internal static string SanitizeForActorName(string siteId)
{
if (string.IsNullOrEmpty(siteId)) return "site";
var sanitized = new string(siteId
.Select(c => char.IsAsciiLetterOrDigit(c) || c is '-' or '_' ? c : '_')
.ToArray());
return ActorPath.IsValidPathElement(sanitized) ? sanitized : "site";
}
}
/// <summary>
/// Central-side actor that routes messages from central to site clusters via ClusterClient.
/// Resolves site addresses from the database on a periodic refresh cycle and manages
/// per-site ClusterClient instances.
/// Central-side actor that routes messages from central to site clusters over the gRPC
/// <c>SiteCommandService</c> command plane (<see cref="Grpc.GrpcSiteTransport"/>). Resolves site
/// addresses from the database on a periodic refresh cycle and reconciles the transport's per-site
/// channel pairs.
///
/// All 8 message patterns routed through this actor.
/// Ask timeout on connection drop (no central buffering). Debug streams killed on interruption.
@@ -78,14 +26,14 @@ public class CentralCommunicationActor : ReceiveActor
{
private readonly ILoggingAdapter _log = Context.GetLogger();
private readonly IServiceProvider _serviceProvider;
private readonly ISiteClientFactory _siteClientFactory;
/// <summary>
/// Per-site ClusterClient instances and their contact addresses.
/// Maps SiteIdentifier → (ClusterClient actor, set of contact address strings).
/// Refreshed periodically via RefreshSiteAddresses.
/// The central→site command transport (the gRPC <see cref="Grpc.GrpcSiteTransport"/>, built by
/// the Host and injected). The <see cref="SiteEnvelope"/> handler delegates every send here, and
/// each DB refresh tick reconciles its per-site channel pairs. Assigned in the constructor body
/// before any message can arrive.
/// </summary>
private readonly Dictionary<string, (IActorRef Client, ImmutableHashSet<string> ContactAddresses)> _siteClients = new();
private ISiteCommandTransport _transport = null!;
// The previous _debugSubscriptions / _inProgressDeployments
// dictionaries existed solely to support a documented "synchronous kill streams +
@@ -156,21 +104,33 @@ public class CentralCommunicationActor : ReceiveActor
/// </summary>
private const string HealthReportTopic = "site-health-replica";
/// <summary>Initializes the <see cref="CentralCommunicationActor"/> and wires all message handlers.</summary>
/// <summary>
/// Constructs the actor over the given central→site command transport (production injects the
/// gRPC <see cref="Grpc.GrpcSiteTransport"/>, built by the Host from the DI
/// <see cref="Grpc.SitePairChannelProvider"/>; tests substitute a fake).
/// </summary>
/// <param name="serviceProvider">DI service provider for scoped repository and aggregator access.</param>
/// <param name="siteClientFactory">Factory used to create per-site ClusterClient actors.</param>
/// <param name="auditIngestAskTimeout">
/// Optional override for the audit-ingest Ask timeout; defaults to
/// <see cref="Grpc.SiteStreamGrpcServer.AuditIngestAskTimeout"/> (30 s). Exists only so tests can
/// exercise the timeout/fault path quickly — production always uses the default.
/// </param>
/// <param name="transport">The central→site command transport to route every <see cref="SiteEnvelope"/> through.</param>
/// <param name="auditIngestAskTimeout">Optional override for the audit-ingest Ask timeout (test hook).</param>
public CentralCommunicationActor(
IServiceProvider serviceProvider,
ISiteClientFactory siteClientFactory,
ISiteCommandTransport transport,
TimeSpan? auditIngestAskTimeout = null)
: this(serviceProvider, auditIngestAskTimeout)
{
_transport = transport ?? throw new ArgumentNullException(nameof(transport));
}
/// <summary>Shared wiring: sets scoped state and registers every message handler. The
/// <see cref="_transport"/> is assigned by the delegating public constructor before any message
/// is dispatched.</summary>
/// <param name="serviceProvider">DI service provider.</param>
/// <param name="auditIngestAskTimeout">Optional audit-ingest Ask timeout override.</param>
private CentralCommunicationActor(
IServiceProvider serviceProvider,
TimeSpan? auditIngestAskTimeout)
{
_serviceProvider = serviceProvider;
_siteClientFactory = siteClientFactory;
_auditIngestAskTimeout = auditIngestAskTimeout ?? Grpc.SiteStreamGrpcServer.AuditIngestAskTimeout;
// Site address cache loaded from database
@@ -185,7 +145,7 @@ public class CentralCommunicationActor : ReceiveActor
// distinguish "no sites configured" from "database is down". Log at Warning.
Receive<Status.Failure>(failure =>
_log.Warning(failure.Cause,
"Failed to load site addresses from the database; the site ClusterClient "
"Failed to load site addresses from the database; the per-site gRPC channel "
+ "cache was not refreshed and may be stale or empty"));
// Health monitoring: heartbeats and health reports from sites
@@ -451,19 +411,10 @@ public class CentralCommunicationActor : ReceiveActor
private void HandleSiteEnvelope(SiteEnvelope envelope)
{
if (!_siteClients.TryGetValue(envelope.SiteId, out var entry))
{
_log.Warning("No ClusterClient for site {0}, cannot route message {1}",
envelope.SiteId, envelope.Message.GetType().Name);
// The Ask will timeout on the caller side — no central buffering
return;
}
// Route via ClusterClient — Sender is preserved for Ask response routing
entry.Client.Tell(
new ClusterClient.Send("/user/site-communication", envelope.Message),
Sender);
// Below-the-seam routing: the gRPC transport owns the "no route for this site ⇒ warn +
// drop, caller's Ask times out" contract. Sender is the temporary Ask actor (or the
// debug-bridge actor) and is preserved for reply routing.
_transport.Send(envelope, Sender);
}
private void LoadSiteAddressesFromDb()
@@ -491,6 +442,10 @@ public class CentralCommunicationActor : ReceiveActor
var sites = await repo.GetAllSitesAsync(ct).ConfigureAwait(false);
var contacts = new Dictionary<string, List<string>>();
// Parallel gRPC-endpoint cache fed by the SAME DB read (the streaming path's
// GrpcNodeA/BAddress columns, NOT the Akka NodeA/BAddress ones). No second poll loop —
// the gRPC transport rides this one, and the Akka transport simply ignores the field.
var grpcContacts = new Dictionary<string, SiteGrpcEndpoints>();
foreach (var site in sites)
{
var addrs = new List<string>();
@@ -511,6 +466,11 @@ public class CentralCommunicationActor : ReceiveActor
}
if (addrs.Count > 0)
contacts[site.SiteIdentifier] = addrs;
var grpcA = string.IsNullOrWhiteSpace(site.GrpcNodeAAddress) ? null : site.GrpcNodeAAddress;
var grpcB = string.IsNullOrWhiteSpace(site.GrpcNodeBAddress) ? null : site.GrpcNodeBAddress;
if (grpcA is not null || grpcB is not null)
grpcContacts[site.SiteIdentifier] = new SiteGrpcEndpoints(grpcA, grpcB);
}
// Freeze the cross-task payload before piping to
@@ -525,82 +485,20 @@ public class CentralCommunicationActor : ReceiveActor
// address-bearing subset in `frozen`) so the aggregator prunes only
// genuinely-deleted sites and never an addressless-but-configured one.
var knownSiteIds = sites.Select(s => s.SiteIdentifier).ToList();
return new SiteAddressCacheLoaded(frozen, knownSiteIds);
return new SiteAddressCacheLoaded(frozen, knownSiteIds, grpcContacts);
}).PipeTo(self);
}
private void HandleSiteAddressCacheLoaded(SiteAddressCacheLoaded msg)
{
var newSiteIds = msg.SiteContacts.Keys.ToHashSet();
var existingSiteIds = _siteClients.Keys.ToHashSet();
// Stop ClusterClients for removed sites
foreach (var removed in existingSiteIds.Except(newSiteIds))
{
_log.Info("Stopping ClusterClient for removed site {0}", removed);
Context.Stop(_siteClients[removed].Client);
_siteClients.Remove(removed);
}
// Add or update
foreach (var (siteId, addresses) in msg.SiteContacts)
{
// Parse all addresses up front inside a try/catch so a
// single malformed site row cannot abort the whole refresh loop and leave
// the cache half-updated. A bad site is logged and skipped; others proceed.
ImmutableHashSet<ActorPath> contactPaths;
try
{
contactPaths = addresses
.Select(a => ActorPath.Parse($"{a}/system/receptionist"))
.ToImmutableHashSet();
}
catch (Exception ex)
{
_log.Warning(ex,
"Malformed contact address for site {0}; skipping this site in the refresh "
+ "(other sites are unaffected)", siteId);
continue;
}
var contactStrings = addresses.ToImmutableHashSet();
// Skip if unchanged
if (_siteClients.TryGetValue(siteId, out var existing) && existing.ContactAddresses.SetEquals(contactStrings))
continue;
// Stop old client if addresses changed
if (_siteClients.ContainsKey(siteId))
{
_log.Info("Updating ClusterClient for site {0} (addresses changed)", siteId);
Context.Stop(_siteClients[siteId].Client);
// Remove now: if the replacement create below fails, a stale entry
// would route envelopes to a stopping actor.
_siteClients.Remove(siteId);
}
IActorRef client;
try
{
client = _siteClientFactory.Create(Context.System, siteId, contactPaths);
}
catch (Exception ex)
{
_log.Error(ex,
"Failed to create ClusterClient for site {0}; site is unroutable until the next refresh",
siteId);
continue;
}
_siteClients[siteId] = (client, contactStrings);
_log.Info("Created ClusterClient for site {0} with {1} contact(s)", siteId, addresses.Count);
}
_log.Info("Site ClusterClient cache refreshed with {0} site(s)", _siteClients.Count);
// Per-transport per-site resource reconciliation (create/stop ClusterClients, or
// build/drop gRPC channel pairs). Runs on the actor thread each refresh tick.
_transport.ReconcileSites(msg);
// Self-healing eviction: a site deleted from configuration would otherwise
// linger in the aggregator as a permanently-offline tile (and live KPI
// sample source) forever. Prune on every refresh so it disappears within
// one refresh interval without needing a dedicated deletion event.
// one refresh interval without needing a dedicated deletion event. Transport-agnostic.
_serviceProvider.GetService<ICentralHealthAggregator>()?.PruneUnknownSites(msg.KnownSiteIds);
}
@@ -673,8 +571,9 @@ public class CentralCommunicationActor : ReceiveActor
public record RefreshSiteAddresses;
/// <summary>
/// Internal message carrying the loaded site contact data from the database.
/// ClusterClient creation happens on the actor thread in HandleSiteAddressCacheLoaded.
/// Message carrying the loaded site contact data from the database. Per-transport resource
/// reconciliation happens on the actor thread in HandleSiteAddressCacheLoaded, which hands this
/// straight to <see cref="ISiteCommandTransport.ReconcileSites"/>.
///
/// The payload is exposed as <see cref="IReadOnlyDictionary{TKey,TValue}"/>
/// of <see cref="IReadOnlyList{T}"/> so the Akka.NET "messages are immutable"
@@ -682,9 +581,24 @@ public record RefreshSiteAddresses;
/// discipline. The producer wraps the constructed buckets with
/// <c>List&lt;T&gt;.AsReadOnly()</c> before piping to Self.
/// </summary>
internal record SiteAddressCacheLoaded(
/// <param name="SiteContacts">Akka ClusterClient contact addresses per site (from NodeA/NodeBAddress).</param>
/// <param name="KnownSiteIds">Every configured site id, address-bearing or not, for aggregator pruning.</param>
/// <param name="GrpcContacts">
/// gRPC endpoint pairs per site (from GrpcNodeA/GrpcNodeBAddress) — the streaming path's columns,
/// consumed by the gRPC transport and ignored by the Akka one.
/// </param>
public sealed record SiteAddressCacheLoaded(
IReadOnlyDictionary<string, IReadOnlyList<string>> SiteContacts,
IReadOnlyCollection<string> KnownSiteIds);
IReadOnlyCollection<string> KnownSiteIds,
IReadOnlyDictionary<string, SiteGrpcEndpoints> GrpcContacts);
/// <summary>
/// A site's gRPC node-pair endpoints, as loaded from <c>Site.GrpcNodeAAddress</c>/
/// <c>GrpcNodeBAddress</c>. Either may be null when only one node has a gRPC address configured.
/// </summary>
/// <param name="NodeA">NodeA gRPC base address (e.g. <c>http://scadabridge-site-a-node-a:8083</c>), or null.</param>
/// <param name="NodeB">NodeB gRPC base address, or null.</param>
public readonly record struct SiteGrpcEndpoints(string? NodeA, string? NodeB);
/// <summary>
/// Peer-replication envelope for a site heartbeat, fanned out over the same
@@ -13,21 +13,20 @@ namespace ZB.MOM.WW.ScadaBridge.Communication.Actors;
/// <remarks>
/// <para>
/// The actor's receive handlers no longer own the wire plumbing — they capture the current
/// <c>Sender</c> and hand it to the transport as <paramref name="replyTo"/>. Two implementations
/// exist behind the <c>ScadaBridge:Communication:CentralTransport</c> flag: the default
/// <see cref="AkkaCentralTransport"/> (verbatim of the old <c>ClusterClient.Send</c> path,
/// including the exact sender-forwarding that routes central's reply straight back to the waiting
/// Ask) and <see cref="Grpc.GrpcCentralTransport"/> (a gRPC dial of <c>CentralControlService</c>).
/// <c>Sender</c> and hand it to the transport as <paramref name="replyTo"/>. The production
/// implementation is <see cref="Grpc.GrpcCentralTransport"/> (a gRPC dial of
/// <c>CentralControlService</c> with sticky central-a→central-b failover) — the only site→central
/// transport since the ClusterClient→gRPC migration removed the Akka path in Phase 4.
/// <see cref="NoOpCentralTransport"/> is the fail-loud placeholder used only when the Host injects
/// nothing (a wiring bug, and in TestKit suites that only exercise command dispatch).
/// </para>
/// <para>
/// <b>Reply/fault contract, identical on both transports.</b> Each Ask-returning method (all but
/// the heartbeat) guarantees exactly one reply eventually lands at <paramref name="replyTo"/>:
/// either the real reply type the caller Asks for, or a transient-failure signal. The Akka path
/// sends a not-accepted ack / <see cref="Status.Failure"/> when no ClusterClient is registered;
/// the gRPC path sends <see cref="Status.Failure"/> on any non-OK status (a timeout or an
/// <c>Unavailable</c> that could not be failed over). Both are what the S&amp;F / audit / health
/// layers above the seam already treat as transient — rows stay buffered, counters restore, the
/// pass re-runs.
/// <b>Reply/fault contract.</b> Each Ask-returning method (all but the heartbeat) guarantees exactly
/// one reply eventually lands at <paramref name="replyTo"/>: either the real reply type the caller
/// Asks for, or a transient-failure signal. The gRPC path sends <see cref="Status.Failure"/> on any
/// non-OK status (a timeout or an <c>Unavailable</c> that could not be failed over) — what the
/// S&amp;F / audit / health layers above the seam already treat as transient (rows stay buffered,
/// counters restore, the pass re-runs).
/// </para>
/// <para>
/// <b>The heartbeat stays fire-and-forget end-to-end.</b> <see cref="SendHeartbeat"/> takes no
@@ -0,0 +1,43 @@
using Akka.Actor;
namespace ZB.MOM.WW.ScadaBridge.Communication.Actors;
/// <summary>
/// The central→site command-send seam, injected into <see cref="CentralCommunicationActor"/>
/// below the <see cref="SiteEnvelope"/> handler. Exactly one implementation is active per node,
/// chosen by <c>ScadaBridge:Communication:SiteTransport</c>:
/// <list type="bullet">
/// <item><see cref="AkkaSiteTransport"/> — today's per-site <c>ClusterClient</c> path (default).</item>
/// <item><see cref="ZB.MOM.WW.ScadaBridge.Communication.Grpc.GrpcSiteTransport"/> — the site
/// <c>SiteCommandService</c> gRPC plane.</item>
/// </list>
/// The producers above the seam (<c>CommunicationService</c>'s 27 commands, <c>SiteCallAuditActor</c>'s
/// 2 parked relays, and <c>DebugStreamBridgeActor</c>'s subscribe/unsubscribe) are unchanged — they
/// still <c>Ask</c>/<c>Tell</c> a <see cref="SiteEnvelope"/> to the actor, which delegates here.
/// </summary>
/// <remarks>
/// Both members run on the actor thread (from the <see cref="SiteEnvelope"/> and
/// <c>SiteAddressCacheLoaded</c> handlers), so implementations need no internal synchronisation for
/// their own per-site bookkeeping beyond what a background failback loop requires.
/// </remarks>
public interface ISiteCommandTransport
{
/// <summary>
/// Routes <paramref name="envelope"/>'s message to its site. Any reply the site produces is
/// delivered to <paramref name="replyTo"/> — for an <c>Ask</c> that is the temporary ask actor
/// (completing the caller's task); for a <c>Tell</c>-with-sender (the debug bridge) that is the
/// originating actor. A message with no route (an unknown site) is warned and dropped so the
/// caller's <c>Ask</c> times out, exactly as today's ClusterClient path behaves.
/// </summary>
/// <param name="envelope">The site-addressed command envelope.</param>
/// <param name="replyTo">Where a reply (or a <see cref="Status.Failure"/>) is delivered.</param>
void Send(SiteEnvelope envelope, IActorRef replyTo);
/// <summary>
/// Reconciles per-site transport resources (ClusterClients for Akka, channel pairs for gRPC)
/// against the freshly loaded site set. Called once per DB refresh tick with the same cache
/// message the actor already receives — the ONE DB-poll loop feeds both transports.
/// </summary>
/// <param name="cache">The loaded site address cache (Akka contacts + gRPC endpoints + known ids).</param>
void ReconcileSites(SiteAddressCacheLoaded cache);
}
@@ -0,0 +1,62 @@
using Akka.Actor;
using Akka.Event;
using ZB.MOM.WW.ScadaBridge.Commons.Messages.Audit;
using ZB.MOM.WW.ScadaBridge.Commons.Messages.Deployment;
using ZB.MOM.WW.ScadaBridge.Commons.Messages.Health;
using ZB.MOM.WW.ScadaBridge.Commons.Messages.Notification;
namespace ZB.MOM.WW.ScadaBridge.Communication.Actors;
/// <summary>
/// The <see cref="ICentralTransport"/> used only when the Host injects none — a fail-loud placeholder,
/// not a working transport. Production always injects <see cref="Grpc.GrpcCentralTransport"/>; a null
/// here is a wiring bug, so every send warns and each Ask-returning method answers a
/// <see cref="Status.Failure"/> so the caller sees the same transient failure the old ClusterClient
/// path produced when no client was registered (rows stay buffered, the pass re-runs). The heartbeat
/// stays fire-and-forget: swallowed and logged, never faulting the heartbeat timer.
/// </summary>
/// <remarks>
/// This replaced the previous default (<c>AkkaCentralTransport</c> with no registered ClusterClient),
/// deleted when the site→central path became gRPC-only in the ClusterClient→gRPC migration's Phase 4.
/// It exists mostly so TestKit suites that only exercise central→site command dispatch — and never
/// inject a transport — construct the actor without wiring a real channel.
/// </remarks>
public sealed class NoOpCentralTransport : ICentralTransport
{
private readonly ILoggingAdapter _log;
/// <summary>Creates the placeholder transport with the owning actor's logging adapter.</summary>
/// <param name="log">The actor's logging adapter, used for the "no transport configured" warnings.</param>
public NoOpCentralTransport(ILoggingAdapter log) => _log = log;
private void Fail(string what, IActorRef replyTo)
{
_log.Warning(
"No site→central transport is configured; dropping {0} as a transient failure. This is a "
+ "wiring bug — the Host must inject a GrpcCentralTransport.", what);
replyTo.Tell(new Status.Failure(new InvalidOperationException(
$"No site→central transport configured (dropping {what}).")));
}
/// <inheritdoc />
public void SubmitNotification(NotificationSubmit message, IActorRef replyTo) => Fail(nameof(SubmitNotification), replyTo);
/// <inheritdoc />
public void QueryNotificationStatus(NotificationStatusQuery message, IActorRef replyTo) => Fail(nameof(QueryNotificationStatus), replyTo);
/// <inheritdoc />
public void IngestAuditEvents(IngestAuditEventsCommand message, IActorRef replyTo) => Fail(nameof(IngestAuditEvents), replyTo);
/// <inheritdoc />
public void IngestCachedTelemetry(IngestCachedTelemetryCommand message, IActorRef replyTo) => Fail(nameof(IngestCachedTelemetry), replyTo);
/// <inheritdoc />
public void ReconcileSite(ReconcileSiteRequest message, IActorRef replyTo) => Fail(nameof(ReconcileSite), replyTo);
/// <inheritdoc />
public void ReportSiteHealth(SiteHealthReport message, IActorRef replyTo) => Fail(nameof(ReportSiteHealth), replyTo);
/// <inheritdoc />
public void SendHeartbeat(HeartbeatMessage message, IActorRef self) =>
_log.Warning("No site→central transport is configured; heartbeat dropped (wiring bug).");
}
@@ -0,0 +1,303 @@
using Akka.Actor;
using ZB.MOM.WW.ScadaBridge.Commons.Messages.Artifacts;
using ZB.MOM.WW.ScadaBridge.Commons.Messages.DataConnection;
using ZB.MOM.WW.ScadaBridge.Commons.Messages.DebugView;
using ZB.MOM.WW.ScadaBridge.Commons.Messages.Deployment;
using ZB.MOM.WW.ScadaBridge.Commons.Messages.InboundApi;
using ZB.MOM.WW.ScadaBridge.Commons.Messages.Lifecycle;
using ZB.MOM.WW.ScadaBridge.Commons.Messages.Management;
using ZB.MOM.WW.ScadaBridge.Commons.Messages.RemoteQuery;
using ZB.MOM.WW.ScadaBridge.Communication.Grpc;
namespace ZB.MOM.WW.ScadaBridge.Communication.Actors;
/// <summary>
/// The single routing truth for the 28 migrated central→site commands. Given one
/// of those command records it decides which local target answers it —
/// the Deployment Manager singleton proxy, the artifact/event-log/parked-message
/// handlers, or the node-local failover path — preserving EXACTLY the targets and
/// null-guard semantics <see cref="SiteCommunicationActor"/> used when this routing
/// lived inline.
/// </summary>
/// <remarks>
/// <para>
/// Two transports call this one unit: the Akka <see cref="SiteCommunicationActor"/>
/// (ClusterClient) and the new <c>SiteCommandGrpcService</c> (gRPC). Centralising the
/// table means the two can never drift on where a command goes. Each transport keeps
/// its own send mechanics — the actor <c>Forward</c>s (preserving the Ask sender), the
/// gRPC service <c>Ask</c>s and encodes the reply — but both read the same
/// <see cref="Route"/> here.
/// </para>
/// <para>
/// <b>The parked-message handler stays node-local on purpose.</b> A parked retry or
/// discard must run on the node that holds the replicated store row, so parked
/// commands route to the per-node <c>_parkedMessageHandler</c> — never onto the
/// singleton proxy (design §7.3). This is the same target the actor used; the
/// extraction does not "fix" it.
/// </para>
/// <para>
/// <b>Excluded by design:</b> <c>IntegrationCallRequest</c> — the 29th command, dead at
/// both ends. It never enters this dispatcher; the actor keeps its own vestigial handler
/// for it (28 of 29 migrate).
/// </para>
/// </remarks>
public sealed class SiteCommandDispatcher
{
/// <summary>How a resolved command is delivered to its target.</summary>
public enum RouteDisposition
{
/// <summary>Forward/Ask <see cref="Route.Target"/> and route its reply back.</summary>
Forward,
/// <summary>
/// Tell <see cref="Route.Target"/> and answer immediately with
/// <see cref="Route.Reply"/> — the fire-and-forget path (only
/// <see cref="UnsubscribeDebugViewRequest"/>, which the downstream never acks).
/// The actor <c>Forward</c>s it and sends nothing back; the gRPC transport Tells and
/// returns the synthetic ack so a unary RPC still answers.
/// </summary>
TellFireAndForget,
/// <summary>
/// No target is available (a null-guarded handler is unregistered); answer with the
/// synthetic <see cref="Route.Reply"/> — the "handler not available" reply the actor
/// produced inline.
/// </summary>
ImmediateReply
}
/// <summary>A resolved routing decision for one command.</summary>
/// <param name="Disposition">How the command is delivered.</param>
/// <param name="Target">The target actor for <see cref="RouteDisposition.Forward"/>/<see cref="RouteDisposition.TellFireAndForget"/>; <c>null</c> for an immediate reply.</param>
/// <param name="Reply">The synthetic reply for <see cref="RouteDisposition.ImmediateReply"/>/<see cref="RouteDisposition.TellFireAndForget"/>; <c>null</c> for a forward.</param>
public readonly record struct Route(RouteDisposition Disposition, IActorRef? Target, object? Reply);
/// <summary>The outcome of preparing a failover: the ack to send now, plus the leave to run after.</summary>
/// <param name="Ack">The ack the caller must send back before the node leaves.</param>
/// <param name="CommitLeave">
/// When non-<c>null</c> (accepted only), invoking it performs the real graceful
/// <c>Cluster.Leave</c>. It is deliberately deferred so the transport can flush the
/// <see cref="Ack"/> first — a caller reaching the very node that is about to leave still
/// receives the ack rather than a broken stream.
/// </param>
public readonly record struct FailoverOutcome(SiteFailoverAck Ack, Action? CommitLeave);
private readonly string _siteId;
private readonly IActorRef _deploymentManagerProxy;
// (role, dryRun) -> leaving node address, or null when there is no standby.
// dryRun:true resolves the target WITHOUT leaving; dryRun:false performs the Leave.
private readonly Func<string, bool, string?> _resolveFailover;
// Registered at runtime via the actor's RegisterLocalHandler flow. Written on the actor
// thread, read by both the actor and the gRPC service (a Kestrel thread), so volatile.
private volatile IActorRef? _eventLogHandler;
private volatile IActorRef? _parkedMessageHandler;
private volatile IActorRef? _artifactHandler;
/// <summary>Creates the dispatcher.</summary>
/// <param name="siteId">This site's identifier, stamped into synthetic replies and matched by the failover guard.</param>
/// <param name="deploymentManagerProxy">The local Deployment Manager singleton proxy — the target for all lifecycle/OPC UA/query/route commands.</param>
/// <param name="resolveFailover">
/// Resolves (and, when <c>dryRun</c> is false, performs) the graceful leave of the oldest
/// Up member in a role scope. Injected so tests need no real cluster.
/// </param>
public SiteCommandDispatcher(
string siteId,
IActorRef deploymentManagerProxy,
Func<string, bool, string?> resolveFailover)
{
ArgumentNullException.ThrowIfNull(siteId);
ArgumentNullException.ThrowIfNull(deploymentManagerProxy);
ArgumentNullException.ThrowIfNull(resolveFailover);
_siteId = siteId;
_deploymentManagerProxy = deploymentManagerProxy;
_resolveFailover = resolveFailover;
}
/// <summary>Registers the site event-log query handler (a cluster singleton proxy).</summary>
/// <param name="handler">The event-log handler.</param>
public void RegisterEventLogHandler(IActorRef handler) => _eventLogHandler = handler;
/// <summary>Registers the node-local parked-message handler (the replicated-store owner on this node).</summary>
/// <param name="handler">The parked-message handler.</param>
public void RegisterParkedMessageHandler(IActorRef handler) => _parkedMessageHandler = handler;
/// <summary>Registers the artifact-deployment handler.</summary>
/// <param name="handler">The artifact handler.</param>
public void RegisterArtifactHandler(IActorRef handler) => _artifactHandler = handler;
/// <summary>
/// Resolves how one of the 27 non-failover commands is delivered. <see cref="TriggerSiteFailover"/>
/// is handled separately via <see cref="PrepareFailover"/>/<see cref="HandleFailover"/> because its
/// leave is deferred; passing it here throws.
/// </summary>
/// <param name="command">The command to route.</param>
/// <returns>The routing decision.</returns>
/// <exception cref="ArgumentException">The command is not a migrated site command (or is failover).</exception>
public Route ResolveRoute(object command)
{
ArgumentNullException.ThrowIfNull(command);
return command switch
{
// ── Deployment + instance lifecycle → Deployment Manager singleton proxy ──
RefreshDeploymentCommand => ToProxy(),
EnableInstanceCommand => ToProxy(),
DisableInstanceCommand => ToProxy(),
DeleteInstanceCommand => ToProxy(),
DeploymentStateQueryRequest => ToProxy(),
// ── Artifact deployment → artifact handler (null-guarded) ──
DeployArtifactsCommand c => _artifactHandler is { } h
? Forwarded(h)
: Immediate(new ArtifactDeploymentResponse(
c.DeploymentId, _siteId, false, "Artifact handler not available", DateTimeOffset.UtcNow)),
// ── Interactive OPC UA / MxGateway → Deployment Manager singleton proxy ──
// The singleton always lands on the active node, which owns the live sessions.
BrowseNodeCommand => ToProxy(),
SearchAddressSpaceCommand => ToProxy(),
ReadTagValuesCommand => ToProxy(),
VerifyEndpointCommand => ToProxy(),
TrustServerCertCommand => ToProxy(),
ListServerCertsCommand => ToProxy(),
RemoveServerCertCommand => ToProxy(),
WriteTagRequest => ToProxy(),
// ── Remote queries: event log (null-guarded) + debug view (proxy) ──
EventLogQueryRequest r => _eventLogHandler is { } h
? Forwarded(h)
: Immediate(new EventLogQueryResponse(
r.CorrelationId, _siteId, [], null, false, false,
"Event log handler not available", DateTimeOffset.UtcNow)),
DebugSnapshotRequest => ToProxy(),
SubscribeDebugViewRequest => ToProxy(),
// Fire-and-forget: the Deployment Manager never acks an unsubscribe, so the gRPC
// transport Tells it and returns the synthetic ack; the actor just Forwards.
UnsubscribeDebugViewRequest => new Route(
RouteDisposition.TellFireAndForget, _deploymentManagerProxy, UnsubscribeDebugViewAck.Instance),
// ── Parked store-and-forward actions → node-local parked handler (null-guarded) ──
ParkedMessageQueryRequest r => _parkedMessageHandler is { } h
? Forwarded(h)
: Immediate(new ParkedMessageQueryResponse(
r.CorrelationId, _siteId, [], 0, r.PageNumber, r.PageSize, false,
"Parked message handler not available", DateTimeOffset.UtcNow)),
ParkedMessageRetryRequest r => _parkedMessageHandler is { } h
? Forwarded(h)
: Immediate(new ParkedMessageRetryResponse(
r.CorrelationId, false, "Parked message handler not available")),
ParkedMessageDiscardRequest r => _parkedMessageHandler is { } h
? Forwarded(h)
: Immediate(new ParkedMessageDiscardResponse(
r.CorrelationId, false, "Parked message handler not available")),
RetryParkedOperation r => _parkedMessageHandler is { } h
? Forwarded(h)
: Immediate(new ParkedOperationActionAck(
r.CorrelationId, Applied: false, "Parked message handler not available")),
DiscardParkedOperation r => _parkedMessageHandler is { } h
? Forwarded(h)
: Immediate(new ParkedOperationActionAck(
r.CorrelationId, Applied: false, "Parked message handler not available")),
// ── Inbound-API Route.To() relays → Deployment Manager singleton proxy ──
RouteToCallRequest => ToProxy(),
RouteToGetAttributesRequest => ToProxy(),
RouteToSetAttributesRequest => ToProxy(),
RouteToWaitForAttributeRequest => ToProxy(),
TriggerSiteFailover => throw new ArgumentException(
"TriggerSiteFailover is handled by PrepareFailover/HandleFailover, not ResolveRoute.",
nameof(command)),
_ => throw new ArgumentException(
$"'{command.GetType().Name}' is not a migrated site command.", nameof(command))
};
Route ToProxy() => Forwarded(_deploymentManagerProxy);
}
private static Route Forwarded(IActorRef target) => new(RouteDisposition.Forward, target, null);
private static Route Immediate(object reply) => new(RouteDisposition.ImmediateReply, null, reply);
/// <summary>
/// The actor's failover path: resolve the standby AND issue the leave in one step (today's
/// coupled behaviour over ClusterClient, where <c>Tell</c> merely enqueues the ack so leave
/// order is immaterial), then return the ack.
/// </summary>
/// <param name="msg">The failover command.</param>
/// <returns>The ack to send back.</returns>
public SiteFailoverAck HandleFailover(TriggerSiteFailover msg)
=> FailoverCore(msg, commitLeaveImmediately: true).Ack;
/// <summary>
/// The gRPC failover path: resolve the standby WITHOUT leaving, build the ack, and hand back a
/// deferred <see cref="FailoverOutcome.CommitLeave"/>. The caller must send the ack before
/// invoking the leave — otherwise a caller reaching the leaving node sees a broken stream
/// instead of its ack (the ack-before-Leave rule).
/// </summary>
/// <param name="msg">The failover command.</param>
/// <returns>The ack and the deferred leave (leave is <c>null</c> when refused).</returns>
public FailoverOutcome PrepareFailover(TriggerSiteFailover msg)
=> FailoverCore(msg, commitLeaveImmediately: false);
private FailoverOutcome FailoverCore(TriggerSiteFailover msg, bool commitLeaveImmediately)
{
ArgumentNullException.ThrowIfNull(msg);
// A misrouted command must be refused, not silently acted on — acting would fail over a
// site the operator never selected. Checked before resolving so the resolver is untouched.
if (!string.Equals(msg.SiteId, _siteId, StringComparison.Ordinal))
{
return new FailoverOutcome(
new SiteFailoverAck(
msg.CorrelationId, Accepted: false, TargetAddress: null,
ErrorMessage: $"Command addressed to site '{msg.SiteId}' but this node serves '{_siteId}'."),
CommitLeave: null);
}
// Site singletons are scoped to the site-specific role, so failover must target that role.
var role = $"site-{_siteId}";
try
{
if (commitLeaveImmediately)
{
var target = _resolveFailover(role, false);
if (target is null)
{
return new FailoverOutcome(NoPeerAck(msg), CommitLeave: null);
}
return new FailoverOutcome(
new SiteFailoverAck(msg.CorrelationId, Accepted: true, target, ErrorMessage: null),
CommitLeave: null);
}
var resolved = _resolveFailover(role, true);
if (resolved is null)
{
return new FailoverOutcome(NoPeerAck(msg), CommitLeave: null);
}
return new FailoverOutcome(
new SiteFailoverAck(msg.CorrelationId, Accepted: true, resolved, ErrorMessage: null),
CommitLeave: () => _resolveFailover(role, false));
}
catch (Exception ex)
{
// A fault must be reported to the operator, never thrown into supervision (over
// ClusterClient) or surfaced as a broken stream (over gRPC).
return new FailoverOutcome(
new SiteFailoverAck(
msg.CorrelationId, Accepted: false, TargetAddress: null, ErrorMessage: ex.Message),
CommitLeave: null);
}
}
private static SiteFailoverAck NoPeerAck(TriggerSiteFailover msg) => new(
msg.CorrelationId, Accepted: false, TargetAddress: null,
ErrorMessage: "No standby available — failing over a lone node would be an outage.");
}
@@ -36,7 +36,6 @@ public class SiteCommunicationActor : ReceiveActor, IWithTimers
/// do not need a real cluster.
/// </summary>
private readonly Func<bool> _isActiveCheck;
private readonly Func<string, string?> _failOverRole;
/// <summary>
/// Reference to the local Deployment Manager singleton proxy.
@@ -44,23 +43,29 @@ public class SiteCommunicationActor : ReceiveActor, IWithTimers
private readonly IActorRef _deploymentManagerProxy;
/// <summary>
/// The site→central transport. Finalized in <see cref="PreStart"/> to the injected instance,
/// or a default <see cref="AkkaCentralTransport"/> (ClusterClient) when none is supplied — so
/// the seven site→central sends delegate here rather than owning the wire plumbing inline.
/// The single routing truth for the 28 migrated central→site commands, shared with
/// the gRPC <c>SiteCommandGrpcService</c> so the two transports cannot drift. In
/// production it is created by the Host and passed in (so the gRPC server holds the
/// same instance); when unset (tests) the actor builds its own from the failover seam.
/// </summary>
private readonly SiteCommandDispatcher _dispatcher;
/// <summary>
/// The site→central transport. Finalized in <see cref="PreStart"/> to the injected instance, or a
/// fail-loud <see cref="NoOpCentralTransport"/> when none is supplied — so the seven site→central
/// sends delegate here rather than owning the wire plumbing inline. Production injects the gRPC
/// <see cref="Grpc.GrpcCentralTransport"/>.
/// </summary>
private ICentralTransport _transport;
/// <summary>The transport supplied by the Host (null selects the default Akka transport).</summary>
/// <summary>The transport supplied by the Host (null falls back to <see cref="NoOpCentralTransport"/>).</summary>
private readonly ICentralTransport? _injectedTransport;
/// <summary>
/// Local actor references for routing specific message patterns.
/// Populated via registration messages.
/// Handler for the vestigial <see cref="IntegrationCallRequest"/> — the one command NOT
/// migrated to the dispatcher (dead at both ends), so it still routes on the actor.
/// </summary>
private IActorRef? _eventLogHandler;
private IActorRef? _parkedMessageHandler;
private IActorRef? _integrationHandler;
private IActorRef? _artifactHandler;
/// <summary>Akka timer scheduler injected by the framework via <see cref="IWithTimers"/>.</summary>
public ITimerScheduler Timers { get; set; } = null!;
@@ -77,10 +82,15 @@ public class SiteCommunicationActor : ReceiveActor, IWithTimers
/// ActorSystem.
/// </param>
/// <param name="transport">
/// The site→central transport. <c>null</c> (the default, used by every existing test and by
/// the Host's Akka path) selects an <see cref="AkkaCentralTransport"/> over ClusterClient; the
/// Host injects a <see cref="Grpc.GrpcCentralTransport"/> when
/// <c>ScadaBridge:Communication:CentralTransport</c> is <c>Grpc</c>.
/// The site→central transport. Production always passes the Host-built gRPC
/// <see cref="Grpc.GrpcCentralTransport"/>. <c>null</c> (used by TestKit suites that only
/// exercise command dispatch) falls back to a fail-loud <see cref="NoOpCentralTransport"/>.
/// </param>
/// <param name="dispatcher">
/// The shared <see cref="SiteCommandDispatcher"/> (production: created by the Host and also
/// handed to the gRPC command service, so both transports route through one instance).
/// <c>null</c> makes the actor build its own from <paramref name="failOverRole"/> — the shape
/// the existing tests use.
/// </param>
public SiteCommunicationActor(
string siteId,
@@ -88,58 +98,69 @@ public class SiteCommunicationActor : ReceiveActor, IWithTimers
IActorRef deploymentManagerProxy,
Func<bool>? isActiveCheck = null,
Func<string, string?>? failOverRole = null,
ICentralTransport? transport = null)
ICentralTransport? transport = null,
SiteCommandDispatcher? dispatcher = null)
{
_siteId = siteId;
_options = options;
_deploymentManagerProxy = deploymentManagerProxy;
_isActiveCheck = isActiveCheck ?? DefaultIsActiveCheck;
_failOverRole = failOverRole ?? DefaultFailOverRole;
_injectedTransport = transport;
// Finalized in PreStart (where _log is usable for the default transport); assigned here
// too so the field is definitely-assigned for the constructor's Receive closures.
_transport = transport!;
// Registration. Feeding the ClusterClient into the transport is a no-op unless the
// default/Akka transport is in use — the gRPC transport dials configured endpoints and
// never receives this message (the Host does not create a ClusterClient for it).
Receive<RegisterCentralClient>(msg =>
{
(_transport as AkkaCentralTransport)?.SetCentralClient(msg.Client);
});
// When no shared dispatcher is supplied, build one over the same failover seam the
// actor used before extraction: an injected Func (tests) that both resolves and leaves,
// or the shared ClusterFailoverCoordinator. The actor only ever commits the leave
// immediately (dryRun:false), so an injected coupled Func fits unchanged.
var system = Context.System;
Func<string, bool, string?> resolveFailover = failOverRole is not null
? (role, _) => failOverRole(role)
: (role, dryRun) =>
ClusterState.ClusterFailoverCoordinator.FailOverOldest(system, role, dryRun)?.ToString();
_dispatcher = dispatcher ?? new SiteCommandDispatcher(siteId, deploymentManagerProxy, resolveFailover);
Receive<RegisterLocalHandler>(HandleRegisterLocalHandler);
// Pattern 1: Instance Deployment — forward to Deployment Manager
Receive<RefreshDeploymentCommand>(msg =>
{
_log.Debug("Routing RefreshDeploymentCommand for {0} to DeploymentManager", msg.InstanceUniqueName);
_deploymentManagerProxy.Forward(msg);
});
// ── The 27 migrated central→site commands (28th is failover, below) all route
// through the shared SiteCommandDispatcher — the single routing truth also used by
// the gRPC SiteCommandGrpcService. The actor's job per command is unchanged: Forward
// to the resolved target (preserving the central Ask sender so replies route straight
// back to the waiting Ask), or Tell the caller the dispatcher's synthetic reply when a
// null-guarded handler is unregistered. See SiteCommandDispatcher for the target of
// each command and why the parked handler stays node-local.
Receive<RefreshDeploymentCommand>(cmd => DispatchCommand(cmd));
Receive<DisableInstanceCommand>(cmd => DispatchCommand(cmd));
Receive<EnableInstanceCommand>(cmd => DispatchCommand(cmd));
Receive<DeleteInstanceCommand>(cmd => DispatchCommand(cmd));
Receive<DeploymentStateQueryRequest>(cmd => DispatchCommand(cmd));
Receive<DeployArtifactsCommand>(cmd => DispatchCommand(cmd));
Receive<SubscribeDebugViewRequest>(cmd => DispatchCommand(cmd));
Receive<UnsubscribeDebugViewRequest>(cmd => DispatchCommand(cmd));
Receive<DebugSnapshotRequest>(cmd => DispatchCommand(cmd));
Receive<RouteToCallRequest>(cmd => DispatchCommand(cmd));
Receive<RouteToGetAttributesRequest>(cmd => DispatchCommand(cmd));
Receive<RouteToSetAttributesRequest>(cmd => DispatchCommand(cmd));
Receive<RouteToWaitForAttributeRequest>(cmd => DispatchCommand(cmd));
Receive<BrowseNodeCommand>(cmd => DispatchCommand(cmd));
Receive<ReadTagValuesCommand>(cmd => DispatchCommand(cmd));
Receive<SearchAddressSpaceCommand>(cmd => DispatchCommand(cmd));
Receive<Commons.Messages.DataConnection.WriteTagRequest>(cmd => DispatchCommand(cmd));
Receive<VerifyEndpointCommand>(cmd => DispatchCommand(cmd));
Receive<TrustServerCertCommand>(cmd => DispatchCommand(cmd));
Receive<ListServerCertsCommand>(cmd => DispatchCommand(cmd));
Receive<RemoveServerCertCommand>(cmd => DispatchCommand(cmd));
Receive<EventLogQueryRequest>(cmd => DispatchCommand(cmd));
Receive<ParkedMessageQueryRequest>(cmd => DispatchCommand(cmd));
Receive<ParkedMessageRetryRequest>(cmd => DispatchCommand(cmd));
Receive<ParkedMessageDiscardRequest>(cmd => DispatchCommand(cmd));
Receive<RetryParkedOperation>(cmd => DispatchCommand(cmd));
Receive<DiscardParkedOperation>(cmd => DispatchCommand(cmd));
// Pattern 2: Lifecycle — forward to Deployment Manager
Receive<DisableInstanceCommand>(msg => _deploymentManagerProxy.Forward(msg));
Receive<EnableInstanceCommand>(msg => _deploymentManagerProxy.Forward(msg));
Receive<DeleteInstanceCommand>(msg => _deploymentManagerProxy.Forward(msg));
// Query-the-site-before-redeploy — forward to
// the Deployment Manager, which owns the deployed-config store and
// answers with the instance's currently-applied deployment identity.
Receive<DeploymentStateQueryRequest>(msg => _deploymentManagerProxy.Forward(msg));
// Pattern 3: Artifact Deployment — forward to artifact handler if registered
Receive<DeployArtifactsCommand>(msg =>
{
if (_artifactHandler != null)
_artifactHandler.Forward(msg);
else
{
_log.Warning("No artifact handler registered, replying with failure");
Sender.Tell(new ArtifactDeploymentResponse(
msg.DeploymentId, _siteId, false, "Artifact handler not available", DateTimeOffset.UtcNow));
}
});
// Pattern 4: Integration Routing — forward to integration handler
// Integration Routing — the 29th command, NOT migrated to the dispatcher (dead at
// both ends; no production code registers the handler). Kept on the actor so the
// dispatcher's command surface stays the 28 that actually migrate.
Receive<IntegrationCallRequest>(msg =>
{
if (_integrationHandler != null)
@@ -151,143 +172,15 @@ public class SiteCommunicationActor : ReceiveActor, IWithTimers
}
});
// Pattern 5: Debug View — forward to Deployment Manager (which routes to Instance Actor)
Receive<SubscribeDebugViewRequest>(msg => _deploymentManagerProxy.Forward(msg));
Receive<UnsubscribeDebugViewRequest>(msg => _deploymentManagerProxy.Forward(msg));
// Pattern 6a: Debug Snapshot (one-shot) — forward to Deployment Manager
Receive<DebugSnapshotRequest>(msg => _deploymentManagerProxy.Forward(msg));
// Inbound API Route.To() — forward to Deployment Manager for instance routing
Receive<RouteToCallRequest>(msg => _deploymentManagerProxy.Forward(msg));
Receive<RouteToGetAttributesRequest>(msg => _deploymentManagerProxy.Forward(msg));
Receive<RouteToSetAttributesRequest>(msg => _deploymentManagerProxy.Forward(msg));
Receive<RouteToWaitForAttributeRequest>(msg => _deploymentManagerProxy.Forward(msg));
// OPC UA Tag Browser (interactive design-time query) — forward to the
// Deployment Manager singleton, which always lands on the active site
// node. Routing to the site-local /user/dcl-manager directly is wrong
// because the standby node has a dcl-manager too, but its
// DataConnectionActor children (which own the live OPC UA sessions)
// only exist on the singleton's node. The singleton then re-forwards
// to its own /user/dcl-manager, which DOES have the connection.
Receive<BrowseNodeCommand>(msg => _deploymentManagerProxy.Forward(msg));
// Test Bindings (interactive design-time read) — same routing rationale
// as BrowseNodeCommand above: the singleton always lands on the
// active site node, which is the node that owns the DataConnectionActor
// children holding the live OPC UA sessions.
Receive<ReadTagValuesCommand>(msg => _deploymentManagerProxy.Forward(msg));
// OPC UA tag-picker address-space search and secured-write execute
// — same singleton routing rationale as BrowseNodeCommand above: the
// DataConnectionActor children that own the live OPC UA sessions exist only
// on the singleton's (active) node, so these must hop through the Deployment
// Manager proxy too. Without these forwards the commands dead-letter and the
// central Ask times out. Forward preserves the central Ask sender so the
// result routes straight back to the waiting Ask.
Receive<SearchAddressSpaceCommand>(msg => _deploymentManagerProxy.Forward(msg));
Receive<Commons.Messages.DataConnection.WriteTagRequest>(msg => _deploymentManagerProxy.Forward(msg));
// OPC UA endpoint Verify — probes a (possibly unsaved) endpoint config
// WITHOUT persisting it. The Deployment Manager singleton's dcl-manager runs
// the probe directly (no existing connection required), so — like the
// commands above — Verify routes through the singleton's active node.
Receive<VerifyEndpointCommand>(msg => _deploymentManagerProxy.Forward(msg));
// OPC UA server-certificate trust management — forward to the
// Deployment Manager singleton, which owns the cross-node trust broadcast.
// The trusted-peer PKI store is node-wide per site node, so a trust/remove
// decision must reach BOTH nodes' CertStoreActor; the singleton broadcasts
// to every site node (list answers from the singleton's own node). The
// singleton always lands on the active node, the same routing rationale as
// BrowseNodeCommand above. Forward preserves the central Ask sender so the
// CertTrustResult routes straight back to the waiting Ask.
Receive<TrustServerCertCommand>(msg => _deploymentManagerProxy.Forward(msg));
Receive<ListServerCertsCommand>(msg => _deploymentManagerProxy.Forward(msg));
Receive<RemoveServerCertCommand>(msg => _deploymentManagerProxy.Forward(msg));
// Pattern 7: Remote Queries
Receive<EventLogQueryRequest>(msg =>
{
if (_eventLogHandler != null)
_eventLogHandler.Forward(msg);
else
{
Sender.Tell(new EventLogQueryResponse(
msg.CorrelationId, _siteId, [], null, false, false,
"Event log handler not available", DateTimeOffset.UtcNow));
}
});
Receive<ParkedMessageQueryRequest>(msg =>
{
if (_parkedMessageHandler != null)
_parkedMessageHandler.Forward(msg);
else
{
Sender.Tell(new ParkedMessageQueryResponse(
msg.CorrelationId, _siteId, [], 0, msg.PageNumber, msg.PageSize, false,
"Parked message handler not available", DateTimeOffset.UtcNow));
}
});
Receive<ParkedMessageRetryRequest>(msg =>
{
if (_parkedMessageHandler != null)
_parkedMessageHandler.Forward(msg);
else
{
Sender.Tell(new ParkedMessageRetryResponse(
msg.CorrelationId, false, "Parked message handler not available"));
}
});
Receive<ParkedMessageDiscardRequest>(msg =>
{
if (_parkedMessageHandler != null)
_parkedMessageHandler.Forward(msg);
else
{
Sender.Tell(new ParkedMessageDiscardResponse(
msg.CorrelationId, false, "Parked message handler not available"));
}
});
// Central→site Retry/Discard relay for parked cached
// operations. SiteCallAuditActor relays these over the command/control
// channel; the parked-message handler executes them against the local
// S&F buffer and replies a ParkedOperationActionAck that routes back to
// the relaying SiteCallAuditActor's Ask.
Receive<RetryParkedOperation>(msg =>
{
if (_parkedMessageHandler != null)
_parkedMessageHandler.Forward(msg);
else
{
Sender.Tell(new ParkedOperationActionAck(
msg.CorrelationId, Applied: false, "Parked message handler not available"));
}
});
Receive<DiscardParkedOperation>(msg =>
{
if (_parkedMessageHandler != null)
_parkedMessageHandler.Forward(msg);
else
{
Sender.Tell(new ParkedOperationActionAck(
msg.CorrelationId, Applied: false, "Parked message handler not available"));
}
});
// Central→site manual failover relay. Central and the site are separate clusters,
// so central can only ask — this node performs the graceful Leave locally, scoped to
// the SITE-SPECIFIC role, because that is what site singletons (the Deployment
// Manager) are placed on. Either node may receive this (the actor is per-node, not a
// singleton, and contact rotation picks whichever answers); the target is resolved
// from cluster state, not from who received the message.
Receive<TriggerSiteFailover>(HandleTriggerSiteFailover);
// from cluster state, not from who received the message. Over ClusterClient the ack
// Tell merely enqueues, so the dispatcher resolves-and-leaves in one step (the gRPC
// transport defers the leave to keep ack-before-Leave — see PrepareFailover).
Receive<TriggerSiteFailover>(msg => Sender.Tell(_dispatcher.HandleFailover(msg)));
// The seven site→central sends now delegate to the injected transport (ClusterClient by
// default, gRPC when configured). Each handler captures the current Sender as the reply
@@ -334,11 +227,11 @@ public class SiteCommunicationActor : ReceiveActor, IWithTimers
/// <inheritdoc />
protected override void PreStart()
{
// Finalize the transport now that the actor context (and _log) exist. The default Akka
// transport is given this actor's logging adapter so the "no ClusterClient registered"
// warnings are preserved exactly. PreStart always runs before any message, so the Receive
// closures above see a non-null _transport.
_transport = _injectedTransport ?? new AkkaCentralTransport(_log);
// Finalize the transport now that the actor context (and _log) exist. When the Host injects
// none, fall back to a fail-loud NoOpCentralTransport (given this actor's logging adapter so
// "no transport configured" warnings surface). PreStart always runs before any message, so
// the Receive closures above see a non-null _transport.
_transport = _injectedTransport ?? new NoOpCentralTransport(_log);
_log.Info("SiteCommunicationActor started for site {0}", _siteId);
@@ -352,21 +245,47 @@ public class SiteCommunicationActor : ReceiveActor, IWithTimers
_options.ApplicationHeartbeatInterval);
}
/// <summary>
/// Executes a resolved command route within the actor: Forward to the target (preserving the
/// central Ask sender so the reply routes straight back to the waiting Ask), or — when a
/// null-guarded handler is unregistered — Tell the caller the dispatcher's synthetic reply.
/// The fire-and-forget disposition (UnsubscribeDebugView) is a plain Forward here, exactly as
/// before: over ClusterClient the site never acked it, so the synthetic ack is a gRPC-only
/// concern.
/// </summary>
/// <param name="command">The migrated central→site command to route.</param>
private void DispatchCommand(object command)
{
var route = _dispatcher.ResolveRoute(command);
switch (route.Disposition)
{
case SiteCommandDispatcher.RouteDisposition.ImmediateReply:
Sender.Tell(route.Reply!);
break;
default:
// Forward and TellFireAndForget both Forward on the actor path.
route.Target!.Forward(command);
break;
}
}
private void HandleRegisterLocalHandler(RegisterLocalHandler msg)
{
// The migrated handlers live on the shared dispatcher so the gRPC command service sees the
// same registrations. Integration is the one command kept on the actor (see the receive).
switch (msg.HandlerType)
{
case LocalHandlerType.EventLog:
_eventLogHandler = msg.Handler;
_dispatcher.RegisterEventLogHandler(msg.Handler);
break;
case LocalHandlerType.ParkedMessages:
_parkedMessageHandler = msg.Handler;
_dispatcher.RegisterParkedMessageHandler(msg.Handler);
break;
case LocalHandlerType.Integration:
_integrationHandler = msg.Handler;
break;
case LocalHandlerType.Artifacts:
_artifactHandler = msg.Handler;
_dispatcher.RegisterArtifactHandler(msg.Handler);
break;
}
@@ -431,79 +350,11 @@ public class SiteCommunicationActor : ReceiveActor, IWithTimers
private bool DefaultIsActiveCheck() =>
ClusterState.ActiveNodeEvaluator.SelfIsOldestUp(Cluster.Get(Context.System));
/// <summary>
/// Handles a central-initiated site failover. Refuses a command addressed to a different
/// site (a misroute must never silently fail over a site the operator did not select) and
/// refuses when the site pair has no peer to take over. The ack is sent BEFORE the Leave
/// takes effect on the wire, so it still reaches central even when this node is the one
/// leaving.
/// </summary>
private void HandleTriggerSiteFailover(TriggerSiteFailover msg)
{
if (!string.Equals(msg.SiteId, _siteId, StringComparison.Ordinal))
{
_log.Warning(
"Refusing TriggerSiteFailover addressed to site {Requested}; this node serves {Actual}",
msg.SiteId, _siteId);
Sender.Tell(new SiteFailoverAck(
msg.CorrelationId, Accepted: false, TargetAddress: null,
ErrorMessage: $"Command addressed to site '{msg.SiteId}' but this node serves '{_siteId}'."));
return;
}
var role = $"site-{_siteId}";
try
{
var target = _failOverRole(role);
if (target is null)
{
_log.Warning(
"Refusing TriggerSiteFailover for {SiteId}: fewer than 2 Up members in role {Role}, "
+ "so there is no standby to take over", _siteId, role);
Sender.Tell(new SiteFailoverAck(
msg.CorrelationId, Accepted: false, TargetAddress: null,
ErrorMessage: "No standby available — failing over a lone node would be an outage."));
return;
}
_log.Warning(
"Manual failover requested by central for site {SiteId}: {Target} is leaving the "
+ "site cluster gracefully; its singletons hand over to the standby.", _siteId, target);
Sender.Tell(new SiteFailoverAck(msg.CorrelationId, Accepted: true, target, ErrorMessage: null));
}
catch (Exception ex)
{
// A fault here must be reported to the operator, not thrown into supervision —
// restarting the communication actor would drop central's Ask into a timeout and
// lose the reason.
_log.Error(ex, "TriggerSiteFailover for {SiteId} faulted", _siteId);
Sender.Tell(new SiteFailoverAck(
msg.CorrelationId, Accepted: false, TargetAddress: null, ErrorMessage: ex.Message));
}
}
/// <summary>
/// Production failover action: gracefully Leave the oldest Up member carrying
/// <paramref name="role"/>, via the shared <see cref="ClusterState.ClusterFailoverCoordinator"/>
/// so the central and site paths cannot drift. Injected in tests for the same reason
/// <see cref="DefaultIsActiveCheck"/> is — a real Leave needs Akka.Cluster in the
/// ActorSystem, which the TestKit system does not load.
/// </summary>
/// <param name="role">Site-specific role scope.</param>
/// <returns>Address of the leaving node, or null when there is no peer.</returns>
private string? DefaultFailOverRole(string role) =>
ClusterState.ClusterFailoverCoordinator.FailOverOldest(Context.System, role)?.ToString();
// ── Internal messages ──
internal record SendHeartbeat;
}
/// <summary>
/// Command to register a ClusterClient for communicating with the central cluster.
/// </summary>
public record RegisterCentralClient(IActorRef Client);
/// <summary>
/// Command to register a local actor as a handler for a specific message pattern.
/// </summary>
@@ -1,37 +1,16 @@
namespace ZB.MOM.WW.ScadaBridge.Communication;
/// <summary>
/// Which transport carries the seven site→central control messages. Selected per node by
/// <c>ScadaBridge:Communication:CentralTransport</c>; the migration ships with
/// <see cref="Akka"/> as the default so nothing flips until a node opts in.
/// </summary>
public enum CentralTransportMode
{
/// <summary>Akka <c>ClusterClient</c> — the transport in production today, and the default.</summary>
Akka = 0,
/// <summary>gRPC dial of the central <c>CentralControlService</c> (Phase 1A migration target).</summary>
Grpc = 1,
}
/// <summary>
/// Configuration options for central-site communication, including per-pattern
/// timeouts and transport heartbeat settings.
/// </summary>
public class CommunicationOptions
{
/// <summary>
/// Which transport carries the site→central control messages. Default <see cref="CentralTransportMode.Akka"/>
/// (ClusterClient) — coexistence rule: a node flips to gRPC only by setting this to <c>Grpc</c>,
/// and rollback is flipping it back. Selecting <c>Grpc</c> requires <see cref="CentralGrpcEndpoints"/>.
/// </summary>
public CentralTransportMode CentralTransport { get; set; } = CentralTransportMode.Akka;
/// <summary>
/// Central control-plane gRPC endpoints (preferred first), e.g.
/// <c>["http://scadabridge-central-a:8083", "http://scadabridge-central-b:8083"]</c>. Dialled by
/// <see cref="Grpc.CentralChannelProvider"/> with sticky failover/failback. Required when
/// <see cref="CentralTransport"/> is <see cref="CentralTransportMode.Grpc"/>, ignored otherwise.
/// <see cref="Grpc.CentralChannelProvider"/> with sticky failover/failback. Required on every site
/// node — gRPC (<c>CentralControlService</c>) is the only site→central transport.
/// </summary>
/// <remarks>
/// Sites reach central by container/host name, NOT via Traefik (which is HTTP/1 only; gRPC is
@@ -68,12 +47,6 @@ public class CommunicationOptions
/// </summary>
public TimeSpan NotificationForwardTimeout { get; set; } = TimeSpan.FromSeconds(30);
/// <summary>
/// Contact point addresses for the central cluster (e.g. "akka.tcp://scadabridge@central-a:8081").
/// Used by site nodes to create a ClusterClient for reaching central.
/// </summary>
public List<string> CentralContactPoints { get; set; } = new();
/// <summary>
/// Preshared key authenticating this node's gRPC control plane — the site↔central
/// boundary. On a site node this is the key its inbound gate
@@ -66,18 +66,15 @@ public sealed class CommunicationOptionsValidator : OptionsValidatorBase<Communi
builder.RequireThat(options.GrpcMaxConcurrentStreams > 0,
$"ScadaBridge:Communication:GrpcMaxConcurrentStreams must be positive (was {options.GrpcMaxConcurrentStreams}).");
// The gRPC site→central transport needs at least one central endpoint to dial. Only
// enforced when that transport is selected the default Akka path ignores the list, so a
// node on ClusterClient must not be forced to declare gRPC endpoints it never uses.
if (options.CentralTransport == CentralTransportMode.Grpc)
{
builder.RequireThat(
options.CentralGrpcEndpoints.Count > 0
&& options.CentralGrpcEndpoints.All(e => !string.IsNullOrWhiteSpace(e)),
"ScadaBridge:Communication:CentralGrpcEndpoints must list at least one non-empty "
+ "central gRPC endpoint when CentralTransport is Grpc "
+ $"(was {options.CentralGrpcEndpoints.Count} entr{(options.CentralGrpcEndpoints.Count == 1 ? "y" : "ies")}).");
}
// The gRPC site→central transport needs at least one central endpoint to dial. gRPC is now
// the only site→central transport (ClusterClient was removed in the migration's Phase 4), so
// every site node must declare its central endpoints — there is no Akka fallback to ignore
// the list. Central nodes leave it empty (they host CentralControlService, they don't dial it).
builder.RequireThat(
options.CentralGrpcEndpoints.All(e => !string.IsNullOrWhiteSpace(e)),
"ScadaBridge:Communication:CentralGrpcEndpoints must not contain empty or whitespace "
+ "entries (a site node must list at least one central gRPC endpoint; central nodes "
+ "leave it empty).");
// ── Aggregated live alarm cache (plan #10, Task 6) ───────────────────────
// Linger drives a Timer dueTime; TimeSpan.Zero is valid (stop the aggregator
@@ -0,0 +1,244 @@
using Akka.Actor;
using Grpc.Net.Client;
using Microsoft.Extensions.Logging;
using ZB.MOM.WW.ScadaBridge.Commons.Messages.Artifacts;
using ZB.MOM.WW.ScadaBridge.Commons.Messages.DataConnection;
using ZB.MOM.WW.ScadaBridge.Commons.Messages.DebugView;
using ZB.MOM.WW.ScadaBridge.Commons.Messages.Deployment;
using ZB.MOM.WW.ScadaBridge.Commons.Messages.InboundApi;
using ZB.MOM.WW.ScadaBridge.Commons.Messages.Lifecycle;
using ZB.MOM.WW.ScadaBridge.Commons.Messages.Management;
using ZB.MOM.WW.ScadaBridge.Commons.Messages.RemoteQuery;
using ZB.MOM.WW.ScadaBridge.Communication.Actors;
namespace ZB.MOM.WW.ScadaBridge.Communication.Grpc;
/// <summary>
/// The gRPC central→site command transport: encodes each <see cref="SiteEnvelope"/> command with
/// <see cref="SiteCommandDtoMapper"/>, dials the site's <c>SiteCommandService</c> through the sticky
/// <see cref="SitePairChannelProvider"/> (PSK + <c>x-scadabridge-site</c> already on the channel),
/// and routes the decoded reply back to the waiting <c>Ask</c> (or the debug-bridge actor).
/// </summary>
/// <remarks>
/// <para>
/// <b>Per-call deadlines match today's Ask timeouts exactly</b> — see <see cref="ResolveDeadline"/>.
/// Behaviour is otherwise unchanged from the Akka path: an unknown/unconfigured site is warned and
/// dropped (the caller's Ask times out), and a transport fault surfaces to the caller as a
/// <see cref="Status.Failure"/>, which the S&amp;F/audit layers already treat as transient.
/// </para>
/// <para>
/// <b>Cross-node retry is the channel provider's job</b> and happens only on
/// <c>Unavailable</c> — never on <c>DeadlineExceeded</c> (a write/deploy/failover may have run).
/// </para>
/// </remarks>
public sealed class GrpcSiteTransport : ISiteCommandTransport
{
private readonly SitePairChannelProvider _channels;
private readonly CommunicationOptions _options;
private readonly ILogger<GrpcSiteTransport> _logger;
// Sites this transport has pushed into the channel provider — used to diff removals per refresh,
// the gRPC analogue of the Akka transport's _siteClients key set. Touched only on the actor thread.
private readonly HashSet<string> _knownSites = new(StringComparer.Ordinal);
/// <summary>Creates the gRPC transport and ensures the provider's failback loop is running.</summary>
/// <param name="channels">The shared per-site channel-pair provider.</param>
/// <param name="options">Communication options supplying the per-command deadlines.</param>
/// <param name="logger">Logger for drop/fault diagnostics.</param>
public GrpcSiteTransport(
SitePairChannelProvider channels,
CommunicationOptions options,
ILogger<GrpcSiteTransport> logger)
{
_channels = channels ?? throw new ArgumentNullException(nameof(channels));
_options = options ?? throw new ArgumentNullException(nameof(options));
_logger = logger ?? throw new ArgumentNullException(nameof(logger));
_channels.EnsureFailbackLoop();
}
/// <inheritdoc />
public void Send(SiteEnvelope envelope, IActorRef replyTo)
{
ArgumentNullException.ThrowIfNull(envelope);
// UnsubscribeDebugView is the one fire-and-forget command: the site never acks it over Akka
// (the debug bridge Tells and expects nothing), so we run the RPC but do not deliver its
// synthetic ack — mirroring today's no-reply behaviour and avoiding a dead-lettered ack.
var fireAndForget = envelope.Message is UnsubscribeDebugViewRequest;
_ = RunAsync(envelope, replyTo, fireAndForget);
}
private async Task RunAsync(SiteEnvelope envelope, IActorRef replyTo, bool fireAndForget)
{
try
{
var reply = await SendCoreAsync(envelope, CancellationToken.None).ConfigureAwait(false);
if (!fireAndForget && !replyTo.IsNobody())
{
replyTo.Tell(reply, ActorRefs.NoSender);
}
}
catch (SiteChannelUnavailableException)
{
// Parity with the Akka "no ClusterClient for site" path: warn and drop, so the caller's
// Ask times out. Central never buffers.
_logger.LogWarning(
"No gRPC channel for site {SiteId}; dropping {Message} (caller's Ask will time out)",
envelope.SiteId, envelope.Message.GetType().Name);
}
catch (Exception ex)
{
if (!fireAndForget && !replyTo.IsNobody())
{
// A timeout or non-OK status faults the caller's Ask exactly as an Akka Ask timeout
// did — the S&F/audit layers already treat that as transient.
replyTo.Tell(new Status.Failure(ex), ActorRefs.NoSender);
}
else
{
_logger.LogWarning(ex,
"Fire-and-forget {Message} to site {SiteId} faulted",
envelope.Message.GetType().Name, envelope.SiteId);
}
}
}
private Task<object> SendCoreAsync(SiteEnvelope envelope, CancellationToken ct)
{
var message = envelope.Message;
// Compute the absolute deadline ONCE so a cross-node failover retry shares the overall
// budget rather than restarting it.
var deadline = DateTime.UtcNow + ResolveDeadline(message);
var group = SiteCommandDtoMapper.GroupOf(message);
return _channels.ExecuteAsync(
envelope.SiteId,
(channel, callCt) => InvokeAsync(channel, group, message, deadline, callCt),
ct);
}
private static async Task<object> InvokeAsync(
GrpcChannel channel, SiteCommandGroup group, object message, DateTime deadline, CancellationToken ct)
{
var client = new SiteCommandService.SiteCommandServiceClient(channel);
switch (group)
{
case SiteCommandGroup.Lifecycle:
{
var reply = await client.ExecuteLifecycleAsync(
SiteCommandDtoMapper.ToLifecycleRequest(message),
deadline: deadline, cancellationToken: ct);
return SiteCommandDtoMapper.FromLifecycleReply(reply);
}
case SiteCommandGroup.OpcUa:
{
var reply = await client.ExecuteOpcUaAsync(
SiteCommandDtoMapper.ToOpcUaRequest(message),
deadline: deadline, cancellationToken: ct);
return SiteCommandDtoMapper.FromOpcUaReply(reply);
}
case SiteCommandGroup.Query:
{
var reply = await client.ExecuteQueryAsync(
SiteCommandDtoMapper.ToQueryRequest(message),
deadline: deadline, cancellationToken: ct);
return SiteCommandDtoMapper.FromQueryReply(reply);
}
case SiteCommandGroup.Parked:
{
var reply = await client.ExecuteParkedAsync(
SiteCommandDtoMapper.ToParkedRequest(message),
deadline: deadline, cancellationToken: ct);
return SiteCommandDtoMapper.FromParkedReply(reply);
}
case SiteCommandGroup.Route:
{
var reply = await client.ExecuteRouteAsync(
SiteCommandDtoMapper.ToRouteRequest(message),
deadline: deadline, cancellationToken: ct);
return SiteCommandDtoMapper.FromRouteReply(reply);
}
case SiteCommandGroup.Failover:
{
var ack = await client.TriggerFailoverAsync(
SiteCommandDtoMapper.ToProto((TriggerSiteFailover)message),
deadline: deadline, cancellationToken: ct);
return SiteCommandDtoMapper.FromProto(ack);
}
default:
throw new ArgumentOutOfRangeException(nameof(group), group, "Unknown site command group.");
}
}
/// <summary>
/// The per-command deadline, set EQUAL to the Ask timeout <c>CommunicationService</c> uses for
/// that command today, so flipping the transport changes nothing about how long a call waits.
/// Note the group is NOT a uniform deadline class: within Lifecycle, <c>DeploymentStateQuery</c>
/// uses <c>QueryTimeout</c> (not <c>LifecycleTimeout</c>), and <c>TriggerSiteFailover</c> uses
/// <c>QueryTimeout</c> (not <c>LifecycleTimeout</c>) — matching the real Ask sites, not the
/// plan's per-group table. The two parked relays (<c>RetryParkedOperation</c>/
/// <c>DiscardParkedOperation</c>) map to <c>QueryTimeout</c> (30s), preserving the
/// <c>SiteCallAuditActor</c> inner <c>RelayTimeout</c> (10s) &lt; 30s ordering. WaitForAttribute
/// keeps its dynamic <c>request.Timeout + IntegrationTimeout</c> budget.
/// </summary>
/// <param name="message">The command being sent.</param>
/// <returns>The deadline duration for that command.</returns>
internal TimeSpan ResolveDeadline(object message) => message switch
{
RefreshDeploymentCommand => _options.DeploymentTimeout,
EnableInstanceCommand or DisableInstanceCommand or DeleteInstanceCommand => _options.LifecycleTimeout,
DeploymentStateQueryRequest => _options.QueryTimeout,
DeployArtifactsCommand => _options.ArtifactDeploymentTimeout,
BrowseNodeCommand or SearchAddressSpaceCommand or ReadTagValuesCommand or VerifyEndpointCommand
or TrustServerCertCommand or ListServerCertsCommand or RemoveServerCertCommand or WriteTagRequest
=> _options.QueryTimeout,
EventLogQueryRequest or DebugSnapshotRequest => _options.QueryTimeout,
SubscribeDebugViewRequest or UnsubscribeDebugViewRequest => _options.DebugViewTimeout,
ParkedMessageQueryRequest or ParkedMessageRetryRequest or ParkedMessageDiscardRequest
or RetryParkedOperation or DiscardParkedOperation
=> _options.QueryTimeout,
RouteToCallRequest or RouteToGetAttributesRequest or RouteToSetAttributesRequest
=> _options.IntegrationTimeout,
RouteToWaitForAttributeRequest r => r.Timeout + _options.IntegrationTimeout,
TriggerSiteFailover => _options.QueryTimeout,
_ => throw new ArgumentException(
$"'{message.GetType().Name}' is not a migrated site command.", nameof(message))
};
/// <inheritdoc />
public void ReconcileSites(SiteAddressCacheLoaded cache)
{
ArgumentNullException.ThrowIfNull(cache);
var desired = cache.GrpcContacts;
// Drop sites that lost their gRPC endpoints (or were deleted).
foreach (var removed in _knownSites.Where(s => !desired.ContainsKey(s)).ToList())
{
_channels.RemoveSite(removed);
_knownSites.Remove(removed);
}
// Create/refresh the rest.
foreach (var (siteId, endpoints) in desired)
{
_channels.UpdateSite(siteId, endpoints.NodeA, endpoints.NodeB);
_knownSites.Add(siteId);
}
}
}
@@ -0,0 +1,219 @@
using System.Collections;
using System.Globalization;
using System.Text.Json;
using Google.Protobuf;
namespace ZB.MOM.WW.ScadaBridge.Communication.Grpc;
/// <summary>
/// Codec for the loosely-typed <c>object?</c> members that survive on the site
/// command plane — script parameters and return values, attribute values, and
/// OPC UA tag read/write values — mapping them to and from the type-tagged
/// <see cref="LooseValue"/> proto carrier.
/// </summary>
/// <remarks>
/// <para>
/// <b>Why a tagged union rather than a string or JSON blob.</b> These values
/// reach an operator's screen (Test Bindings, Debug View) and a device write
/// (<c>WriteTag</c>), so collapsing them to text would change behaviour: today
/// the Akka JSON serializer runs with <c>TypeNameHandling</c> on and preserves
/// the boxed CLR type end-to-end. The tags below cover every CLR type these
/// fields actually carry, so those values keep their runtime type across the
/// wire exactly as they do over Akka remoting.
/// </para>
/// <para>
/// <b>The one documented lossy path.</b> Anything outside the tagged set —
/// an exotic numeric (<see cref="byte"/>, <see cref="uint"/>, …), an enum, a
/// POCO — falls back to <see cref="LooseValue.JsonValue"/> and decodes as a
/// <see cref="JsonElement"/> rather than its original CLR type. The value
/// itself is preserved; its CLR identity is not. Collections and string-keyed
/// dictionaries do NOT take that path: they encode recursively (see
/// <see cref="LooseValueList"/>/<see cref="LooseValueMap"/>) and decode to
/// <c>List&lt;object?&gt;</c> / <c>Dictionary&lt;string, object?&gt;</c>, so
/// their ELEMENTS keep their types while the container type widens.
/// </para>
/// <para>
/// <b>Why dates ride as strings.</b> <c>google.protobuf.Timestamp</c>
/// normalises everything to UTC, which silently discards
/// <see cref="DateTime.Kind"/> and <see cref="DateTimeOffset.Offset"/>. For a
/// timestamped tag value that is data loss, not normalisation, so
/// <see cref="DateTime"/>/<see cref="DateTimeOffset"/>/<see cref="TimeSpan"/>
/// use invariant round-trip formats instead. (Message FIELDS that are declared
/// <c>DateTimeOffset</c> in the DTO are a different case and do use
/// <c>Timestamp</c> — see <see cref="SiteCommandDtoMapper"/>.)
/// </para>
/// </remarks>
public static class LooseValueCodec
{
private static readonly JsonSerializerOptions JsonOpts = new() { WriteIndented = false };
/// <summary>
/// Encodes a boxed value for a NULLABLE message field: <c>null</c> returns
/// <c>null</c> so the field is simply left unset on the wire.
/// </summary>
/// <param name="value">The boxed value to encode, or <c>null</c>.</param>
/// <returns>The encoded carrier, or <c>null</c> when <paramref name="value"/> is <c>null</c>.</returns>
public static LooseValue? ToProtoOrNull(object? value) =>
value is null ? null : ToProto(value);
/// <summary>
/// Encodes a boxed value, representing <c>null</c> explicitly as
/// <see cref="LooseNull"/>. Used inside maps and lists, where an absent
/// entry means "no such key/element" rather than "a null value".
/// </summary>
/// <param name="value">The boxed value to encode, or <c>null</c>.</param>
/// <returns>A populated <see cref="LooseValue"/>; never <c>null</c>.</returns>
public static LooseValue ToProto(object? value) => value switch
{
null => new LooseValue { NullValue = new LooseNull() },
bool b => new LooseValue { BoolValue = b },
int i => new LooseValue { Int32Value = i },
long l => new LooseValue { Int64Value = l },
double d => new LooseValue { DoubleValue = d },
float f => new LooseValue { FloatValue = f },
string s => new LooseValue { StringValue = s },
decimal m => new LooseValue { DecimalValue = m.ToString(CultureInfo.InvariantCulture) },
DateTime dt => new LooseValue { DateTimeValue = dt.ToString("O", CultureInfo.InvariantCulture) },
DateTimeOffset dto => new LooseValue { DateTimeOffsetValue = dto.ToString("O", CultureInfo.InvariantCulture) },
Guid g => new LooseValue { GuidValue = g.ToString("D") },
TimeSpan ts => new LooseValue { TimeSpanValue = ts.ToString("c", CultureInfo.InvariantCulture) },
byte[] bytes => new LooseValue { BytesValue = ByteString.CopyFrom(bytes) },
IDictionary dict => new LooseValue { MapValue = ToMap(dict) },
IEnumerable seq => new LooseValue { ListValue = ToList(seq) },
// Escape hatch. Preserves the value, not the CLR type — see the remarks.
_ => new LooseValue { JsonValue = JsonSerializer.Serialize(value, value.GetType(), JsonOpts) }
};
/// <summary>
/// Decodes a carrier back to a boxed value. An unset field
/// (<c>null</c>) and an explicit <see cref="LooseNull"/> both decode to
/// <c>null</c>, so the two encoders above are interchangeable on read.
/// </summary>
/// <param name="value">The carrier to decode, or <c>null</c> when the field was unset.</param>
/// <returns>The decoded boxed value; <c>null</c> for an unset or explicitly-null carrier.</returns>
public static object? FromProto(LooseValue? value)
{
if (value is null)
{
return null;
}
return value.KindCase switch
{
LooseValue.KindOneofCase.None => null,
LooseValue.KindOneofCase.NullValue => null,
LooseValue.KindOneofCase.BoolValue => value.BoolValue,
LooseValue.KindOneofCase.Int32Value => value.Int32Value,
LooseValue.KindOneofCase.Int64Value => value.Int64Value,
LooseValue.KindOneofCase.DoubleValue => value.DoubleValue,
LooseValue.KindOneofCase.FloatValue => value.FloatValue,
LooseValue.KindOneofCase.StringValue => value.StringValue,
LooseValue.KindOneofCase.DecimalValue =>
decimal.Parse(value.DecimalValue, CultureInfo.InvariantCulture),
LooseValue.KindOneofCase.DateTimeValue =>
DateTime.Parse(value.DateTimeValue, CultureInfo.InvariantCulture, DateTimeStyles.RoundtripKind),
LooseValue.KindOneofCase.DateTimeOffsetValue =>
DateTimeOffset.Parse(value.DateTimeOffsetValue, CultureInfo.InvariantCulture, DateTimeStyles.RoundtripKind),
LooseValue.KindOneofCase.GuidValue => Guid.Parse(value.GuidValue),
LooseValue.KindOneofCase.TimeSpanValue =>
TimeSpan.ParseExact(value.TimeSpanValue, "c", CultureInfo.InvariantCulture),
LooseValue.KindOneofCase.BytesValue => value.BytesValue.ToByteArray(),
LooseValue.KindOneofCase.ListValue => FromList(value.ListValue),
LooseValue.KindOneofCase.MapValue => FromMap(value.MapValue),
LooseValue.KindOneofCase.JsonValue => JsonSerializer.Deserialize<JsonElement>(value.JsonValue),
_ => null
};
}
/// <summary>
/// Encodes a string-keyed dictionary onto the wire. Null values are carried
/// explicitly (<see cref="LooseNull"/>), so a key present with a null value
/// stays distinct from an absent key.
/// </summary>
/// <param name="values">The dictionary to encode.</param>
/// <returns>A populated <see cref="LooseValueMap"/>.</returns>
public static LooseValueMap ToProtoMap(IReadOnlyDictionary<string, object?> values)
{
ArgumentNullException.ThrowIfNull(values);
var map = new LooseValueMap();
foreach (var (key, value) in values)
{
map.Entries[key] = ToProto(value);
}
return map;
}
/// <summary>
/// Encodes a NULLABLE string-keyed dictionary: <c>null</c> returns
/// <c>null</c> so the field is left unset and the null/empty distinction
/// survives (proto3 <c>map</c> alone cannot express it).
/// </summary>
/// <param name="values">The dictionary to encode, or <c>null</c>.</param>
/// <returns>The encoded map, or <c>null</c> when <paramref name="values"/> is <c>null</c>.</returns>
public static LooseValueMap? ToProtoMapOrNull(IReadOnlyDictionary<string, object?>? values) =>
values is null ? null : ToProtoMap(values);
/// <summary>Decodes a wire map back to a dictionary; an unset field decodes to <c>null</c>.</summary>
/// <param name="map">The wire map, or <c>null</c> when the field was unset.</param>
/// <returns>The decoded dictionary, or <c>null</c>.</returns>
public static IReadOnlyDictionary<string, object?>? FromProtoMapOrNull(LooseValueMap? map) =>
map is null ? null : FromMap(map);
/// <summary>Decodes a wire map back to a dictionary; an unset field decodes to an EMPTY dictionary.</summary>
/// <remarks>For DTO members that are declared non-nullable, so "absent" can only mean "empty".</remarks>
/// <param name="map">The wire map, or <c>null</c> when the field was unset.</param>
/// <returns>The decoded dictionary; empty when <paramref name="map"/> is <c>null</c>.</returns>
public static IReadOnlyDictionary<string, object?> FromProtoMap(LooseValueMap? map) =>
map is null ? new Dictionary<string, object?>() : FromMap(map);
private static LooseValueList ToList(IEnumerable source)
{
var list = new LooseValueList();
foreach (var element in source)
{
list.Items.Add(ToProto(element));
}
return list;
}
private static LooseValueMap ToMap(IDictionary source)
{
var map = new LooseValueMap();
foreach (DictionaryEntry entry in source)
{
// Non-string keys are outside the wire contract; the invariant-culture
// rendering keeps the entry readable rather than dropping it silently.
var key = entry.Key as string
?? Convert.ToString(entry.Key, CultureInfo.InvariantCulture)
?? string.Empty;
map.Entries[key] = ToProto(entry.Value);
}
return map;
}
private static List<object?> FromList(LooseValueList list)
{
var result = new List<object?>(list.Items.Count);
foreach (var item in list.Items)
{
result.Add(FromProto(item));
}
return result;
}
private static Dictionary<string, object?> FromMap(LooseValueMap map)
{
var result = new Dictionary<string, object?>(map.Entries.Count);
foreach (var (key, value) in map.Entries)
{
result[key] = FromProto(value);
}
return result;
}
}
File diff suppressed because it is too large Load Diff
@@ -0,0 +1,33 @@
namespace ZB.MOM.WW.ScadaBridge.Communication.Grpc;
/// <summary>
/// The six domain groups the 28 migrated central→site commands are partitioned
/// into — one group per <c>SiteCommandService</c> RPC.
/// </summary>
/// <remarks>
/// The partition is not cosmetic: every command inside a group shares a
/// DEADLINE class today (the <c>CommunicationOptions</c> timeout the central
/// <c>Ask</c> uses), so one RPC per group keeps the deadline decision in one
/// place on the client and one dispatch switch on the server, while the
/// <c>oneof</c> envelope keeps each command individually typed.
/// </remarks>
public enum SiteCommandGroup
{
/// <summary>Deployment refresh, instance enable/disable/delete, deployment-state query, artifact deployment.</summary>
Lifecycle,
/// <summary>Interactive OPC UA / MxGateway design-time commands: browse, search, read, verify, cert trust, write tag.</summary>
OpcUa,
/// <summary>Read-only remote queries: site event log and debug view snapshot/subscribe/unsubscribe.</summary>
Query,
/// <summary>Parked store-and-forward message actions and parked cached-operation retry/discard relays.</summary>
Parked,
/// <summary>Inbound-API <c>Route.To()</c> relays: call, get/set attributes, wait for attribute.</summary>
Route,
/// <summary>Operator-initiated manual site-pair failover.</summary>
Failover
}
@@ -0,0 +1,227 @@
using Akka.Actor;
using Grpc.Core;
using Microsoft.Extensions.Logging;
using ZB.MOM.WW.ScadaBridge.Commons.Messages.RemoteQuery;
using ZB.MOM.WW.ScadaBridge.Communication.Actors;
using GrpcStatus = Grpc.Core.Status;
namespace ZB.MOM.WW.ScadaBridge.Communication.Grpc;
/// <summary>
/// gRPC front door for the central→site command plane: decodes a proto command (via
/// <see cref="SiteCommandDtoMapper"/>), routes it through the ONE
/// <see cref="SiteCommandDispatcher"/> the Akka <c>SiteCommunicationActor</c> also uses, and
/// encodes the reply. One routing truth, two transports.
/// </summary>
/// <remarks>
/// <para>
/// Mapped in the site branch next to <c>SiteStreamGrpcServer</c>, behind the same
/// <c>ControlPlaneAuthInterceptor</c> PSK gate and the same readiness convention: calls are
/// rejected with <see cref="StatusCode.Unavailable"/> until <see cref="SetReady"/> is called
/// once the site actor system is up (mirrors <c>SiteStreamGrpcServer.SetReady</c>).
/// </para>
/// <para>
/// <b>Coexistence — server-side only.</b> This makes the site ALSO listen on gRPC for commands;
/// nothing central flips to gRPC here (that is T1B.3). Central still dials sites via ClusterClient.
/// </para>
/// <para>
/// <b>Fire-and-forget.</b> The Akka path never acks <c>UnsubscribeDebugView</c>, but a unary RPC
/// must answer, so the dispatcher marks it <see cref="SiteCommandDispatcher.RouteDisposition.TellFireAndForget"/>
/// and this service Tells the target then returns the synthetic ack.
/// </para>
/// <para>
/// <b>Ack-before-Leave.</b> <c>TriggerFailover</c> resolves the standby WITHOUT leaving
/// (<see cref="SiteCommandDispatcher.PrepareFailover"/>), returns the ack, and only THEN schedules
/// the real <c>Cluster.Leave</c> — so a caller reaching the very node that is about to leave still
/// receives its ack instead of a broken stream.
/// </para>
/// </remarks>
public sealed class SiteCommandGrpcService : SiteCommandService.SiteCommandServiceBase
{
// A local Ask has no client deadline of its own; fall back to a generous ceiling when the
// caller set none (a deadline-less client is a test or an internal caller). When the caller
// DID set a gRPC deadline, honour the remaining time instead.
private static readonly TimeSpan DefaultAskTimeout = TimeSpan.FromMinutes(2);
private readonly ILogger<SiteCommandGrpcService> _logger;
private readonly Action<Action> _leaveScheduler;
// Set once by SetReady after the site actor system and dispatcher exist. Read on Kestrel
// threads, written on the host bring-up thread — volatile.
private volatile SiteCommandDispatcher? _dispatcher;
private volatile bool _ready;
/// <summary>DI constructor.</summary>
/// <param name="logger">Logger for denial/dispatch diagnostics.</param>
public SiteCommandGrpcService(ILogger<SiteCommandGrpcService> logger)
: this(logger, DeferLeaveUntilAfterReply)
{
}
/// <summary>
/// Test constructor letting a test observe when the deferred leave runs. Internal so DI sees a
/// single public constructor.
/// </summary>
/// <param name="logger">Logger.</param>
/// <param name="leaveScheduler">Runs the deferred <c>Cluster.Leave</c> after the ack is returned.</param>
internal SiteCommandGrpcService(ILogger<SiteCommandGrpcService> logger, Action<Action> leaveScheduler)
{
ArgumentNullException.ThrowIfNull(logger);
ArgumentNullException.ThrowIfNull(leaveScheduler);
_logger = logger;
_leaveScheduler = leaveScheduler;
}
/// <summary>
/// Marks the service ready and injects the shared routing table. Called once the site actor
/// system, the Deployment Manager singleton, and the local handlers are up — the same point
/// <c>SiteStreamGrpcServer.SetReady</c> is called.
/// </summary>
/// <param name="dispatcher">The shared dispatcher (the same instance the actor routes through).</param>
public void SetReady(SiteCommandDispatcher dispatcher)
{
ArgumentNullException.ThrowIfNull(dispatcher);
_dispatcher = dispatcher;
_ready = true;
}
/// <summary>Whether the service is accepting commands. Exposed for tests.</summary>
internal bool IsReady => _ready;
/// <inheritdoc />
public override async Task<LifecycleReply> ExecuteLifecycle(LifecycleRequest request, ServerCallContext context)
{
var command = SiteCommandDtoMapper.FromLifecycleRequest(request);
var reply = await DispatchAsync(command, context).ConfigureAwait(false);
return SiteCommandDtoMapper.ToLifecycleReply(reply);
}
/// <inheritdoc />
public override async Task<OpcUaReply> ExecuteOpcUa(OpcUaRequest request, ServerCallContext context)
{
var command = SiteCommandDtoMapper.FromOpcUaRequest(request);
var reply = await DispatchAsync(command, context).ConfigureAwait(false);
return SiteCommandDtoMapper.ToOpcUaReply(reply);
}
/// <inheritdoc />
public override async Task<QueryReply> ExecuteQuery(QueryRequest request, ServerCallContext context)
{
var command = SiteCommandDtoMapper.FromQueryRequest(request);
var reply = await DispatchAsync(command, context).ConfigureAwait(false);
return SiteCommandDtoMapper.ToQueryReply(reply);
}
/// <inheritdoc />
public override async Task<ParkedReply> ExecuteParked(ParkedRequest request, ServerCallContext context)
{
var command = SiteCommandDtoMapper.FromParkedRequest(request);
var reply = await DispatchAsync(command, context).ConfigureAwait(false);
return SiteCommandDtoMapper.ToParkedReply(reply);
}
/// <inheritdoc />
public override async Task<RouteReply> ExecuteRoute(RouteRequest request, ServerCallContext context)
{
var command = SiteCommandDtoMapper.FromRouteRequest(request);
var reply = await DispatchAsync(command, context).ConfigureAwait(false);
return SiteCommandDtoMapper.ToRouteReply(reply);
}
/// <inheritdoc />
public override Task<SiteFailoverAckDto> TriggerFailover(TriggerSiteFailoverDto request, ServerCallContext context)
{
var dispatcher = EnsureReady();
var msg = SiteCommandDtoMapper.FromProto(request);
// Resolve the standby and build the ack WITHOUT leaving. The real leave is deferred so the
// ack is on the wire first (ack-before-Leave).
var outcome = dispatcher.PrepareFailover(msg);
var dto = SiteCommandDtoMapper.ToProto(outcome.Ack);
if (outcome.CommitLeave is not null)
{
_leaveScheduler(outcome.CommitLeave);
}
return Task.FromResult(dto);
}
/// <summary>
/// Routes a decoded command through the shared dispatcher and returns the reply record.
/// Forward → local <c>Ask</c>; fire-and-forget → local <c>Tell</c> + synthetic ack; no target →
/// the dispatcher's synthetic "handler not available" reply.
/// </summary>
private async Task<object> DispatchAsync(object command, ServerCallContext context)
{
var dispatcher = EnsureReady();
var route = dispatcher.ResolveRoute(command);
switch (route.Disposition)
{
case SiteCommandDispatcher.RouteDisposition.ImmediateReply:
return route.Reply!;
case SiteCommandDispatcher.RouteDisposition.TellFireAndForget:
route.Target!.Tell(command);
return route.Reply!;
default:
try
{
return await route.Target!.Ask<object>(
command, AskTimeout(context), context.CancellationToken).ConfigureAwait(false);
}
catch (RpcException)
{
throw;
}
catch (OperationCanceledException)
{
// Client cancelled or the deadline elapsed.
throw new RpcException(new GrpcStatus(
StatusCode.DeadlineExceeded, "Site did not answer within the deadline."));
}
catch (Exception ex)
{
_logger.LogWarning(ex,
"Local dispatch of {Command} faulted", command.GetType().Name);
throw new RpcException(new GrpcStatus(StatusCode.Internal, ex.Message));
}
}
}
private SiteCommandDispatcher EnsureReady()
{
var dispatcher = _dispatcher;
if (!_ready || dispatcher is null)
{
throw new RpcException(new GrpcStatus(
StatusCode.Unavailable, "Site command plane not ready."));
}
return dispatcher;
}
private static TimeSpan AskTimeout(ServerCallContext context)
{
var deadline = context.Deadline;
if (deadline == DateTime.MaxValue)
{
return DefaultAskTimeout;
}
var remaining = deadline - DateTime.UtcNow;
return remaining > TimeSpan.Zero ? remaining : TimeSpan.FromMilliseconds(1);
}
// Default deferred-leave: return control (so the ack serialises) before the graceful Leave
// runs. A yield hands the current continuation back before the leave begins; the leave itself
// is slow-async (member marked Leaving, CoordinatedShutdown over seconds), so the ack is long
// gone by the time Kestrel actually stops.
private static void DeferLeaveUntilAfterReply(Action commitLeave)
=> _ = Task.Run(async () =>
{
await Task.Yield();
commitLeave();
});
}
@@ -0,0 +1,436 @@
using System.Collections.Concurrent;
using Grpc.Core;
using Grpc.Net.Client;
using Microsoft.Extensions.Logging;
using Microsoft.Extensions.Options;
namespace ZB.MOM.WW.ScadaBridge.Communication.Grpc;
/// <summary>
/// Raised when a site has no usable gRPC channel — an unknown site, or a site with neither
/// <c>GrpcNodeAAddress</c> nor <c>GrpcNodeBAddress</c> configured. The gRPC transport treats this
/// the way the Akka path treats "no ClusterClient for site": warn and drop, so the caller's Ask
/// times out (central never buffers).
/// </summary>
public sealed class SiteChannelUnavailableException(string siteId)
: Exception($"No gRPC channel is configured for site '{siteId}'.")
{
/// <summary>The site with no configured channel.</summary>
public string SiteId { get; } = siteId;
}
/// <summary>
/// Shared, per-site gRPC channel PAIR provider with sticky failover + failback (design §3.5).
/// Central holds one <see cref="GrpcChannel"/> per site node (NodeA/NodeB) rather than one channel
/// with re-dial logic; NodeA is the preferred node (config order). Every call goes to the current
/// sticky channel; on connect failure / <see cref="StatusCode.Unavailable"/> / readiness rejection
/// it flips to the other node and stays there. A background probe returns to the preferred node
/// once it answers again. Reconnect probing backs off 1 s → doubling → 60 s cap.
/// </summary>
/// <remarks>
/// <para>
/// <b>Retry safety.</b> <see cref="ExecuteAsync{T}"/> retries on the other node ONLY when the first
/// attempt failed with <see cref="StatusCode.Unavailable"/> (provably never reached a server —
/// connect refused or readiness-rejected before response headers). A
/// <see cref="StatusCode.DeadlineExceeded"/>, or any other status, is rethrown WITHOUT a cross-node
/// retry: a <c>WriteTag</c>/<c>DeployArtifacts</c>/<c>TriggerSiteFailover</c> may already have
/// executed, and the layers above tolerate the one-shot ambiguity exactly as ClusterClient's Ask
/// did — the migration must not turn one write into two.
/// </para>
/// <para>
/// <b>Addresses.</b> Fed by <see cref="UpdateSite"/>/<see cref="RemoveSite"/> from the single DB
/// refresh loop (the <c>GrpcNodeAAddress</c>/<c>GrpcNodeBAddress</c> columns). PSK credentials and
/// the <c>x-scadabridge-site</c> header ride every channel via
/// <see cref="ControlPlaneCredentials.WithSiteCredentials"/>; removing a site invalidates its
/// cached key.
/// </para>
/// </remarks>
public sealed class SitePairChannelProvider : IDisposable
{
private static readonly TimeSpan InitialBackoff = TimeSpan.FromSeconds(1);
private static readonly TimeSpan MaxBackoff = TimeSpan.FromSeconds(60);
private static readonly TimeSpan FailbackSweepInterval = TimeSpan.FromSeconds(1);
private readonly ISitePskProvider _pskProvider;
private readonly CommunicationOptions _options;
private readonly ILogger<SitePairChannelProvider> _logger;
private readonly Func<string, HttpMessageHandler>? _handlerFactory;
private readonly Func<GrpcChannel, CancellationToken, Task<bool>> _reachabilityProbe;
private readonly ConcurrentDictionary<string, SitePair> _sites = new(StringComparer.Ordinal);
private readonly CancellationTokenSource _shutdown = new();
private Task? _failbackLoop;
private readonly object _loopGate = new();
private bool _disposed;
/// <summary>Creates the provider.</summary>
/// <param name="pskProvider">Resolves each site's preshared key for channel credentials.</param>
/// <param name="options">Communication options (keepalive settings applied to production channels).</param>
/// <param name="logger">Logger for failover/failback diagnostics.</param>
/// <param name="handlerFactory">
/// Test seam mapping an endpoint to the <see cref="HttpMessageHandler"/> its channel should use
/// (e.g. an in-process <c>TestServer</c> handler). Null in production, where each channel builds
/// a keepalive-configured <see cref="SocketsHttpHandler"/>.
/// </param>
/// <param name="reachabilityProbe">
/// Test seam deciding whether a preferred node is reachable during a failback sweep. Null in
/// production, where <see cref="GrpcChannel.ConnectAsync"/> is used.
/// </param>
public SitePairChannelProvider(
ISitePskProvider pskProvider,
IOptions<CommunicationOptions> options,
ILogger<SitePairChannelProvider> logger,
Func<string, HttpMessageHandler>? handlerFactory = null,
Func<GrpcChannel, CancellationToken, Task<bool>>? reachabilityProbe = null)
{
ArgumentNullException.ThrowIfNull(pskProvider);
ArgumentNullException.ThrowIfNull(options);
ArgumentNullException.ThrowIfNull(logger);
_pskProvider = pskProvider;
_options = options.Value;
_logger = logger;
_handlerFactory = handlerFactory;
_reachabilityProbe = reachabilityProbe ?? DefaultReachabilityProbe;
}
/// <summary>
/// (Re)builds a site's channel pair from its gRPC node endpoints. A no-op when neither endpoint
/// changed. Disposes and rebuilds a channel whose endpoint changed, and resets stickiness to the
/// preferred node when the pair is (re)created. Called on the DB refresh tick.
/// </summary>
/// <param name="siteId">Site identifier.</param>
/// <param name="nodeAEndpoint">NodeA gRPC base address, or null when unconfigured.</param>
/// <param name="nodeBEndpoint">NodeB gRPC base address, or null when unconfigured.</param>
public void UpdateSite(string siteId, string? nodeAEndpoint, string? nodeBEndpoint)
{
ArgumentException.ThrowIfNullOrWhiteSpace(siteId);
var pair = _sites.GetOrAdd(siteId, id => new SitePair(id));
lock (pair.Gate)
{
var changedA = !string.Equals(pair.NodeAEndpoint, nodeAEndpoint, StringComparison.Ordinal);
var changedB = !string.Equals(pair.NodeBEndpoint, nodeBEndpoint, StringComparison.Ordinal);
if (!changedA && !changedB)
{
return;
}
if (changedA)
{
pair.ChannelA?.Dispose();
pair.ChannelA = string.IsNullOrWhiteSpace(nodeAEndpoint) ? null : BuildChannel(nodeAEndpoint, siteId);
pair.NodeAEndpoint = nodeAEndpoint;
}
if (changedB)
{
pair.ChannelB?.Dispose();
pair.ChannelB = string.IsNullOrWhiteSpace(nodeBEndpoint) ? null : BuildChannel(nodeBEndpoint, siteId);
pair.NodeBEndpoint = nodeBEndpoint;
}
// Addresses changed — return to the preferred node and reset failback backoff.
pair.CurrentIsA = pair.ChannelA is not null;
pair.ResetFailback();
_logger.LogInformation(
"gRPC channel pair for site {SiteId} refreshed (nodeA={HasA}, nodeB={HasB})",
siteId, pair.ChannelA is not null, pair.ChannelB is not null);
}
}
/// <summary>Disposes a removed site's channels and invalidates its cached preshared key.</summary>
/// <param name="siteId">Site identifier to remove.</param>
public void RemoveSite(string siteId)
{
if (string.IsNullOrWhiteSpace(siteId))
{
return;
}
if (_sites.TryRemove(siteId, out var pair))
{
lock (pair.Gate)
{
pair.ChannelA?.Dispose();
pair.ChannelB?.Dispose();
pair.ChannelA = null;
pair.ChannelB = null;
}
_pskProvider.Invalidate(siteId);
_logger.LogInformation("gRPC channel pair for site {SiteId} removed", siteId);
}
}
/// <summary>
/// Runs <paramref name="call"/> against the site's current sticky channel, failing over to the
/// other node once on <see cref="StatusCode.Unavailable"/> (never on
/// <see cref="StatusCode.DeadlineExceeded"/>). Ensures the background failback loop is running.
/// </summary>
/// <typeparam name="T">The RPC reply type.</typeparam>
/// <param name="siteId">Target site.</param>
/// <param name="call">The RPC to run against a given channel.</param>
/// <param name="ct">Cancellation token.</param>
/// <returns>The RPC reply.</returns>
/// <exception cref="SiteChannelUnavailableException">The site has no configured channel.</exception>
public async Task<T> ExecuteAsync<T>(
string siteId, Func<GrpcChannel, CancellationToken, Task<T>> call, CancellationToken ct)
{
ArgumentException.ThrowIfNullOrWhiteSpace(siteId);
ArgumentNullException.ThrowIfNull(call);
EnsureFailbackLoop();
if (!_sites.TryGetValue(siteId, out var pair))
{
throw new SiteChannelUnavailableException(siteId);
}
GrpcChannel primary;
GrpcChannel? secondary;
bool primaryIsA;
lock (pair.Gate)
{
primaryIsA = pair.CurrentIsA;
primary = (primaryIsA ? pair.ChannelA : pair.ChannelB)
?? pair.ChannelA ?? pair.ChannelB
?? throw new SiteChannelUnavailableException(siteId);
// Recompute which node `primary` actually is (the current side may be null).
primaryIsA = ReferenceEquals(primary, pair.ChannelA);
secondary = primaryIsA ? pair.ChannelB : pair.ChannelA;
}
try
{
return await call(primary, ct).ConfigureAwait(false);
}
catch (RpcException ex) when (ex.StatusCode == StatusCode.Unavailable && secondary is not null)
{
// Provably never reached a server → safe to try the other node, even for a write.
_logger.LogWarning(
"Site {SiteId} node {FromNode} unavailable; failing over to node {ToNode}",
siteId, primaryIsA ? "A" : "B", primaryIsA ? "B" : "A");
FlipTo(pair, toIsA: !primaryIsA);
return await call(secondary, ct).ConfigureAwait(false);
}
}
/// <summary>
/// Probes a site's preferred node (NodeA) and, if reachable, returns stickiness to it. The
/// background loop calls this per due site; tests call it to drive failback deterministically.
/// </summary>
/// <param name="siteId">Site identifier.</param>
/// <param name="ct">Cancellation token.</param>
/// <returns>True when the site is now (or already) pointed at its preferred node.</returns>
internal async Task<bool> TryFailbackAsync(string siteId, CancellationToken ct)
{
if (!_sites.TryGetValue(siteId, out var pair))
{
return false;
}
GrpcChannel? preferred;
lock (pair.Gate)
{
if (pair.CurrentIsA || pair.ChannelA is null)
{
return true; // already on preferred (or no preferred channel to fail back to)
}
preferred = pair.ChannelA;
}
var reachable = await _reachabilityProbe(preferred, ct).ConfigureAwait(false);
lock (pair.Gate)
{
if (reachable)
{
pair.CurrentIsA = true;
pair.ResetFailback();
_logger.LogInformation("Site {SiteId} preferred node A reachable again; failing back", siteId);
return true;
}
pair.BumpFailbackBackoff();
return false;
}
}
/// <summary>Test/diagnostic accessor: true when the site currently targets its preferred node (A).</summary>
/// <param name="siteId">Site identifier.</param>
/// <returns>True when on NodeA (or when the site is unknown).</returns>
internal bool IsOnPreferredNode(string siteId)
=> !_sites.TryGetValue(siteId, out var pair) || pair.CurrentIsA;
/// <summary>Starts the background failback loop if not already running. Idempotent.</summary>
public void EnsureFailbackLoop()
{
if (_failbackLoop is not null || _disposed)
{
return;
}
lock (_loopGate)
{
if (_failbackLoop is null && !_disposed)
{
_failbackLoop = Task.Run(() => FailbackLoopAsync(_shutdown.Token));
}
}
}
private async Task FailbackLoopAsync(CancellationToken ct)
{
using var timer = new PeriodicTimer(FailbackSweepInterval);
try
{
while (await timer.WaitForNextTickAsync(ct).ConfigureAwait(false))
{
var now = DateTime.UtcNow;
foreach (var kvp in _sites)
{
var pair = kvp.Value;
bool due;
lock (pair.Gate)
{
due = !pair.CurrentIsA && pair.ChannelA is not null && now >= pair.NextProbeUtc;
}
if (due)
{
try
{
await TryFailbackAsync(kvp.Key, ct).ConfigureAwait(false);
}
catch (OperationCanceledException)
{
throw;
}
catch (Exception ex)
{
_logger.LogDebug(ex, "Failback probe for site {SiteId} faulted", kvp.Key);
}
}
}
}
}
catch (OperationCanceledException)
{
// Shutting down.
}
}
private void FlipTo(SitePair pair, bool toIsA)
{
lock (pair.Gate)
{
pair.CurrentIsA = toIsA;
if (!toIsA)
{
// Failed over off the preferred node → schedule the first failback probe.
pair.ScheduleFailback(InitialBackoff);
}
else
{
pair.ResetFailback();
}
}
}
private GrpcChannel BuildChannel(string endpoint, string siteId)
{
var channelOptions = new GrpcChannelOptions
{
HttpHandler = _handlerFactory is not null
? _handlerFactory(endpoint)
: new SocketsHttpHandler
{
KeepAlivePingDelay = _options.GrpcKeepAlivePingDelay,
KeepAlivePingTimeout = _options.GrpcKeepAlivePingTimeout,
KeepAlivePingPolicy = HttpKeepAlivePingPolicy.Always,
EnableMultipleHttp2Connections = true
}
}.WithSiteCredentials(_pskProvider, siteId);
return GrpcChannel.ForAddress(endpoint, channelOptions);
}
private static async Task<bool> DefaultReachabilityProbe(GrpcChannel channel, CancellationToken ct)
{
try
{
using var timeout = CancellationTokenSource.CreateLinkedTokenSource(ct);
timeout.CancelAfter(TimeSpan.FromSeconds(5));
await channel.ConnectAsync(timeout.Token).ConfigureAwait(false);
return true;
}
catch
{
return false;
}
}
/// <inheritdoc />
public void Dispose()
{
if (_disposed)
{
return;
}
_disposed = true;
_shutdown.Cancel();
try
{
_failbackLoop?.Wait(TimeSpan.FromSeconds(2));
}
catch
{
// Best-effort loop drain on shutdown.
}
_shutdown.Dispose();
foreach (var pair in _sites.Values)
{
lock (pair.Gate)
{
pair.ChannelA?.Dispose();
pair.ChannelB?.Dispose();
}
}
_sites.Clear();
}
/// <summary>Per-site mutable channel state, guarded by <see cref="Gate"/>.</summary>
private sealed class SitePair(string siteId)
{
public string SiteId { get; } = siteId;
public readonly object Gate = new();
public string? NodeAEndpoint;
public string? NodeBEndpoint;
public GrpcChannel? ChannelA;
public GrpcChannel? ChannelB;
/// <summary>Sticky current node; NodeA is preferred. True = pointing at NodeA.</summary>
public bool CurrentIsA = true;
/// <summary>Next time a failback probe is due (while failed over). MaxValue = not scheduled.</summary>
public DateTime NextProbeUtc = DateTime.MaxValue;
private TimeSpan _backoff = InitialBackoff;
public void ScheduleFailback(TimeSpan delay)
{
_backoff = delay;
NextProbeUtc = DateTime.UtcNow + _backoff;
}
public void BumpFailbackBackoff()
{
var next = TimeSpan.FromTicks(Math.Min(_backoff.Ticks * 2, MaxBackoff.Ticks));
_backoff = next;
NextProbeUtc = DateTime.UtcNow + _backoff;
}
public void ResetFailback()
{
_backoff = InitialBackoff;
NextProbeUtc = DateTime.MaxValue;
}
}
}
@@ -0,0 +1,857 @@
syntax = "proto3";
option csharp_namespace = "ZB.MOM.WW.ScadaBridge.Communication.Grpc";
package scadabridge.sitecommand.v1;
import "google/protobuf/timestamp.proto";
import "google/protobuf/duration.proto";
import "google/protobuf/wrappers.proto";
//
// Site command plane (central site) the gRPC replacement for the per-site
// ClusterClient command/control channel handled by SiteCommunicationActor.
//
// The 28 migrated commands are grouped into six domain RPCs rather than 28
// individual RPCs, because the grouping is what carries the DEADLINE policy:
// every command inside one group shares a timeout class today (see
// CommunicationOptions), so one RPC per group keeps the deadline choice in one
// place on the client and one dispatch switch on the server. A `oneof` request
// / reply envelope preserves per-command typing inside the group.
//
// IntegrationCallRequest (the 29th command) is deliberately NOT here it is
// dead code at both ends; see
// docs/known-issues/2026-07-22-integration-call-routing-is-dead-code.md.
//
// EVOLUTION RULE (same as sitestream.proto): field numbers are never reused and
// changes are additive only. New commands take the next free oneof tag; an
// older peer simply sees an unset oneof and answers with a clean error.
//
// The generated C# is CHECKED IN under Communication/SiteCommandGrpc/ because
// protoc segfaults inside the linux_arm64 Docker build image. Regenerate with
// docker/regen-proto.sh (or by hand per the recipe in the .csproj) never by
// leaving an active <Protobuf> item in the project file.
//
service SiteCommandService {
// Deployment + instance lifecycle (6 commands).
rpc ExecuteLifecycle(LifecycleRequest) returns (LifecycleReply);
// Interactive OPC UA / MxGateway design-time commands (8 commands).
rpc ExecuteOpcUa(OpcUaRequest) returns (OpcUaReply);
// Remote read-only queries: event log + debug view (4 commands).
rpc ExecuteQuery(QueryRequest) returns (QueryReply);
// Parked store-and-forward message + cached-operation actions (5 commands).
rpc ExecuteParked(ParkedRequest) returns (ParkedReply);
// Inbound-API Route.To() relays (4 commands).
rpc ExecuteRoute(RouteRequest) returns (RouteReply);
// Operator-initiated manual site-pair failover (1 command).
rpc TriggerFailover(TriggerSiteFailoverDto) returns (SiteFailoverAckDto);
}
//
// Shared value carrier
//
// Explicit null marker. Needed because a map<string, LooseValue> entry always
// has a value message present, so "the dictionary holds a null for this key"
// cannot be expressed by absence the way a nullable message FIELD can.
message LooseNull {}
message LooseValueList { repeated LooseValue items = 1; }
message LooseValueMap { map<string, LooseValue> entries = 1; }
// Type-tagged carrier for the `object?` members the command plane still uses
// (script parameters and return values, attribute values, tag read/write
// values). The tags cover every CLR type these fields actually carry, so those
// values round-trip with their runtime type intact; anything else falls back to
// `json_value` (see the mapper's documented lossiness note).
//
// Date/time and decimal ride as invariant round-trip STRINGS rather than
// google.protobuf.Timestamp: Timestamp normalises everything to UTC and would
// silently drop DateTime.Kind / DateTimeOffset.Offset, which for an operator's
// tag value is data loss, not normalisation.
message LooseValue {
oneof kind {
LooseNull null_value = 1;
bool bool_value = 2;
int32 int32_value = 3;
int64 int64_value = 4;
double double_value = 5;
float float_value = 6;
string string_value = 7;
string decimal_value = 8; // invariant-culture round-trip
string date_time_value = 9; // DateTime, "O" (preserves Kind)
string date_time_offset_value = 10; // DateTimeOffset, "O" (preserves Offset)
string guid_value = 11; // "D"
string time_span_value = 12; // "c" (constant/round-trip)
bytes bytes_value = 13;
LooseValueList list_value = 14;
LooseValueMap map_value = 15;
string json_value = 16; // fallback; decodes to JsonElement
}
}
//
// Enums. Every enum reserves 0 for _UNSPECIFIED so a value that is absent on
// the wire is never mistaken for the first CLR member. The mapper translates
// explicitly in both directions (never by ordinal), so reordering a C# enum
// cannot silently re-map the wire.
//
enum DeploymentStatusDto {
DEPLOYMENT_STATUS_DTO_UNSPECIFIED = 0;
DEPLOYMENT_STATUS_DTO_PENDING = 1;
DEPLOYMENT_STATUS_DTO_IN_PROGRESS = 2;
DEPLOYMENT_STATUS_DTO_SUCCESS = 3;
DEPLOYMENT_STATUS_DTO_FAILED = 4;
}
enum BrowseNodeClassDto {
BROWSE_NODE_CLASS_DTO_UNSPECIFIED = 0;
BROWSE_NODE_CLASS_DTO_OBJECT = 1;
BROWSE_NODE_CLASS_DTO_VARIABLE = 2;
BROWSE_NODE_CLASS_DTO_METHOD = 3;
BROWSE_NODE_CLASS_DTO_OTHER = 4;
}
enum BrowseFailureKindDto {
BROWSE_FAILURE_KIND_DTO_UNSPECIFIED = 0;
BROWSE_FAILURE_KIND_DTO_CONNECTION_NOT_FOUND = 1;
BROWSE_FAILURE_KIND_DTO_CONNECTION_NOT_CONNECTED = 2;
BROWSE_FAILURE_KIND_DTO_NOT_BROWSABLE = 3;
BROWSE_FAILURE_KIND_DTO_TIMEOUT = 4;
BROWSE_FAILURE_KIND_DTO_SERVER_ERROR = 5;
}
enum ReadTagValuesFailureKindDto {
READ_TAG_VALUES_FAILURE_KIND_DTO_UNSPECIFIED = 0;
READ_TAG_VALUES_FAILURE_KIND_DTO_CONNECTION_NOT_FOUND = 1;
READ_TAG_VALUES_FAILURE_KIND_DTO_CONNECTION_NOT_CONNECTED = 2;
READ_TAG_VALUES_FAILURE_KIND_DTO_TIMEOUT = 3;
READ_TAG_VALUES_FAILURE_KIND_DTO_SERVER_ERROR = 4;
}
enum VerifyFailureKindDto {
VERIFY_FAILURE_KIND_DTO_UNSPECIFIED = 0;
VERIFY_FAILURE_KIND_DTO_UNREACHABLE = 1;
VERIFY_FAILURE_KIND_DTO_AUTH_FAILED = 2;
VERIFY_FAILURE_KIND_DTO_UNTRUSTED_CERTIFICATE = 3;
VERIFY_FAILURE_KIND_DTO_TIMEOUT = 4;
VERIFY_FAILURE_KIND_DTO_SERVER_ERROR = 5;
}
enum StoreAndForwardCategoryDto {
STORE_AND_FORWARD_CATEGORY_DTO_UNSPECIFIED = 0;
STORE_AND_FORWARD_CATEGORY_DTO_EXTERNAL_SYSTEM = 1;
STORE_AND_FORWARD_CATEGORY_DTO_NOTIFICATION = 2;
STORE_AND_FORWARD_CATEGORY_DTO_CACHED_DB_WRITE = 3;
}
enum AlarmStateDto {
ALARM_STATE_DTO_UNSPECIFIED = 0;
ALARM_STATE_DTO_ACTIVE = 1;
ALARM_STATE_DTO_NORMAL = 2;
}
enum AlarmLevelDto {
ALARM_LEVEL_DTO_UNSPECIFIED = 0;
ALARM_LEVEL_DTO_NONE = 1;
ALARM_LEVEL_DTO_LOW = 2;
ALARM_LEVEL_DTO_LOW_LOW = 3;
ALARM_LEVEL_DTO_HIGH = 4;
ALARM_LEVEL_DTO_HIGH_HIGH = 5;
}
enum AlarmKindDto {
ALARM_KIND_DTO_UNSPECIFIED = 0;
ALARM_KIND_DTO_COMPUTED = 1;
ALARM_KIND_DTO_NATIVE_OPC_UA = 2;
ALARM_KIND_DTO_NATIVE_MX_ACCESS = 3;
}
enum AlarmShelveStateDto {
ALARM_SHELVE_STATE_DTO_UNSPECIFIED = 0;
ALARM_SHELVE_STATE_DTO_UNSHELVED = 1;
ALARM_SHELVE_STATE_DTO_ONE_SHOT_SHELVED = 2;
ALARM_SHELVE_STATE_DTO_TIMED_SHELVED = 3;
ALARM_SHELVE_STATE_DTO_PERMANENT_SHELVED = 4;
}
//
// RPC 1 ExecuteLifecycle
//
message RefreshDeploymentCommandDto {
string deployment_id = 1;
string instance_unique_name = 2;
string revision_hash = 3;
string deployed_by = 4;
google.protobuf.Timestamp timestamp = 5;
string central_fetch_base_url = 6;
string fetch_token = 7;
}
message EnableInstanceCommandDto {
string command_id = 1;
string instance_unique_name = 2;
google.protobuf.Timestamp timestamp = 3;
}
message DisableInstanceCommandDto {
string command_id = 1;
string instance_unique_name = 2;
google.protobuf.Timestamp timestamp = 3;
}
message DeleteInstanceCommandDto {
string command_id = 1;
string instance_unique_name = 2;
google.protobuf.Timestamp timestamp = 3;
}
message DeploymentStateQueryRequestDto {
string correlation_id = 1;
string instance_unique_name = 2;
google.protobuf.Timestamp timestamp = 3;
}
message SharedScriptArtifactDto {
string name = 1;
string code = 2;
string parameter_definitions = 3; // empty string represents null
string return_definition = 4; // empty string represents null
}
message ExternalSystemArtifactDto {
string name = 1;
string endpoint_url = 2;
string auth_type = 3;
string auth_configuration = 4; // empty string represents null
string method_definitions_json = 5; // empty string represents null
int32 timeout_seconds = 6;
}
message DatabaseConnectionArtifactDto {
string name = 1;
string connection_string = 2;
int32 max_retries = 3;
google.protobuf.Duration retry_delay = 4;
}
message NotificationListArtifactDto {
string name = 1;
repeated string recipient_emails = 2;
}
message DataConnectionArtifactDto {
string name = 1;
string protocol = 2;
string primary_configuration_json = 3; // empty string represents null
string backup_configuration_json = 4; // empty string represents null
int32 failover_retry_count = 5;
}
message SmtpConfigurationArtifactDto {
string name = 1;
string server = 2;
int32 port = 3;
string auth_mode = 4;
string from_address = 5;
string username = 6; // empty string represents null
string password = 7; // empty string represents null
string oauth_config = 8; // empty string represents null
}
// Each artifact collection on DeployArtifactsCommand is a NULLABLE list, and
// proto3 `repeated` cannot distinguish null from empty. Wrapping each in its
// own message makes the null/empty distinction a message-presence question,
// which proto3 does model the same silent-data-loss class the transport
// round-trip guard caught in PLAN-05 T8.
message SharedScriptArtifactListDto { repeated SharedScriptArtifactDto items = 1; }
message ExternalSystemArtifactListDto { repeated ExternalSystemArtifactDto items = 1; }
message DatabaseConnectionArtifactListDto { repeated DatabaseConnectionArtifactDto items = 1; }
message NotificationListArtifactListDto { repeated NotificationListArtifactDto items = 1; }
message DataConnectionArtifactListDto { repeated DataConnectionArtifactDto items = 1; }
message SmtpConfigurationArtifactListDto { repeated SmtpConfigurationArtifactDto items = 1; }
message DeployArtifactsCommandDto {
string deployment_id = 1;
SharedScriptArtifactListDto shared_scripts = 2; // absent => null
ExternalSystemArtifactListDto external_systems = 3; // absent => null
DatabaseConnectionArtifactListDto database_connections = 4; // absent => null
NotificationListArtifactListDto notification_lists = 5; // absent => null
DataConnectionArtifactListDto data_connections = 6; // absent => null
SmtpConfigurationArtifactListDto smtp_configurations = 7; // absent => null
google.protobuf.Timestamp timestamp = 8;
}
message DeploymentStatusResponseDto {
string deployment_id = 1;
string instance_unique_name = 2;
DeploymentStatusDto status = 3;
string error_message = 4; // empty string represents null
google.protobuf.Timestamp timestamp = 5;
}
message InstanceLifecycleResponseDto {
string command_id = 1;
string instance_unique_name = 2;
bool success = 3;
string error_message = 4; // empty string represents null
google.protobuf.Timestamp timestamp = 5;
}
message DeploymentStateQueryResponseDto {
string correlation_id = 1;
string instance_unique_name = 2;
bool is_deployed = 3;
string applied_deployment_id = 4; // empty string represents null
string applied_revision_hash = 5; // empty string represents null
google.protobuf.Timestamp timestamp = 6;
}
message ArtifactDeploymentResponseDto {
string deployment_id = 1;
string site_id = 2;
bool success = 3;
string error_message = 4; // empty string represents null
google.protobuf.Timestamp timestamp = 5;
}
message LifecycleRequest {
oneof command {
RefreshDeploymentCommandDto refresh_deployment = 1;
EnableInstanceCommandDto enable_instance = 2;
DisableInstanceCommandDto disable_instance = 3;
DeleteInstanceCommandDto delete_instance = 4;
DeploymentStateQueryRequestDto deployment_state_query = 5;
DeployArtifactsCommandDto deploy_artifacts = 6;
}
}
message LifecycleReply {
oneof reply {
DeploymentStatusResponseDto deployment_status = 1;
InstanceLifecycleResponseDto instance_lifecycle = 2;
DeploymentStateQueryResponseDto deployment_state_query = 3;
ArtifactDeploymentResponseDto artifact_deployment = 4;
}
}
//
// RPC 2 ExecuteOpcUa
//
message BrowseNodeCommandDto {
string connection_name = 1;
string parent_node_id = 2; // empty string represents null (browse root)
string continuation_token = 3; // empty string represents null (first page)
string site_identifier = 4; // empty string represents null
}
message BrowseNodeDto {
string node_id = 1;
string display_name = 2;
BrowseNodeClassDto node_class = 3;
bool has_children = 4;
string data_type = 5; // empty string represents null
google.protobuf.Int32Value value_rank = 6; // absent => null
google.protobuf.BoolValue writable = 7; // absent => null
}
message BrowseFailureDto {
BrowseFailureKindDto kind = 1;
string message = 2;
}
message BrowseNodeResultDto {
repeated BrowseNodeDto children = 1;
bool truncated = 2;
BrowseFailureDto failure = 3; // absent => null (success)
string continuation_token = 4; // empty string represents null (final page)
}
message SearchAddressSpaceCommandDto {
string connection_name = 1;
string query = 2;
int32 max_depth = 3;
int32 max_results = 4;
string site_identifier = 5; // empty string represents null
}
message AddressSpaceMatchDto {
BrowseNodeDto node = 1;
string path = 2;
}
message SearchAddressSpaceResultDto {
repeated AddressSpaceMatchDto matches = 1;
bool cap_reached = 2;
BrowseFailureDto failure = 3; // absent => null (success)
}
message ReadTagValuesCommandDto {
string connection_name = 1;
repeated string tag_paths = 2;
}
message TagReadOutcomeDto {
string tag_path = 1;
bool success = 2;
LooseValue value = 3; // absent => null
string quality = 4;
google.protobuf.Timestamp timestamp = 5;
string error_message = 6; // empty string represents null
}
message ReadTagValuesFailureDto {
ReadTagValuesFailureKindDto kind = 1;
string message = 2;
}
message ReadTagValuesResultDto {
repeated TagReadOutcomeDto outcomes = 1;
ReadTagValuesFailureDto failure = 2; // absent => null (success)
}
message VerifyEndpointCommandDto {
string connection_name = 1;
string protocol = 2;
string config_json = 3;
string site_identifier = 4; // empty string represents null
}
message ServerCertInfoDto {
string thumbprint = 1;
string subject = 2;
string issuer = 3;
google.protobuf.Timestamp not_before_utc = 4;
google.protobuf.Timestamp not_after_utc = 5;
string der_base64 = 6;
}
// failure_kind is a NULLABLE enum on the CLR side, and proto3 enums have no
// presence, so it rides inside a one-field message exactly like Int32Value.
message VerifyFailureKindValue { VerifyFailureKindDto value = 1; }
message VerifyEndpointResultDto {
bool success = 1;
VerifyFailureKindValue failure_kind = 2; // absent => null (success)
string error = 3; // empty string represents null
ServerCertInfoDto cert = 4; // absent => null
}
message TrustServerCertCommandDto {
string connection_name = 1;
string der_base64 = 2;
string thumbprint = 3;
string site_identifier = 4; // empty string represents null
}
message ListServerCertsCommandDto {
string site_identifier = 1; // empty string represents null
}
message RemoveServerCertCommandDto {
string thumbprint = 1;
string site_identifier = 2; // empty string represents null
}
message TrustedCertInfoDto {
string thumbprint = 1;
string subject = 2;
string issuer = 3;
google.protobuf.Timestamp not_before_utc = 4;
google.protobuf.Timestamp not_after_utc = 5;
bool rejected = 6;
}
// CertTrustResult.Certs is a nullable list (null for trust/remove, populated
// for list) same null-vs-empty problem as the artifact collections.
message TrustedCertInfoListDto { repeated TrustedCertInfoDto items = 1; }
message CertTrustResultDto {
bool success = 1;
string error = 2; // empty string represents null
TrustedCertInfoListDto certs = 3; // absent => null
}
message WriteTagRequestDto {
string correlation_id = 1;
string connection_name = 2;
string tag_path = 3;
LooseValue value = 4; // absent => null
google.protobuf.Timestamp timestamp = 5;
}
message WriteTagResponseDto {
string correlation_id = 1;
bool success = 2;
string error_message = 3; // empty string represents null
google.protobuf.Timestamp timestamp = 4;
}
message OpcUaRequest {
oneof command {
BrowseNodeCommandDto browse_node = 1;
SearchAddressSpaceCommandDto search_address_space = 2;
ReadTagValuesCommandDto read_tag_values = 3;
VerifyEndpointCommandDto verify_endpoint = 4;
TrustServerCertCommandDto trust_server_cert = 5;
ListServerCertsCommandDto list_server_certs = 6;
RemoveServerCertCommandDto remove_server_cert = 7;
WriteTagRequestDto write_tag = 8;
}
}
message OpcUaReply {
oneof reply {
BrowseNodeResultDto browse_node = 1;
SearchAddressSpaceResultDto search_address_space = 2;
ReadTagValuesResultDto read_tag_values = 3;
VerifyEndpointResultDto verify_endpoint = 4;
CertTrustResultDto cert_trust = 5;
WriteTagResponseDto write_tag = 6;
}
}
//
// RPC 3 ExecuteQuery
//
message EventLogQueryRequestDto {
string correlation_id = 1;
string site_id = 2;
google.protobuf.Timestamp from = 3; // absent => null
google.protobuf.Timestamp to = 4; // absent => null
string event_type = 5; // empty string represents null
string severity = 6; // empty string represents null
string instance_id = 7; // empty string represents null
string keyword_filter = 8; // empty string represents null
string continuation_token = 9; // empty string represents null
int32 page_size = 10;
google.protobuf.Timestamp timestamp = 11;
}
message EventLogEntryDto {
string id = 1;
google.protobuf.Timestamp timestamp = 2;
string event_type = 3;
string severity = 4;
string instance_id = 5; // empty string represents null
string source = 6;
string message = 7;
string details = 8; // empty string represents null
}
message EventLogQueryResponseDto {
string correlation_id = 1;
string site_id = 2;
repeated EventLogEntryDto entries = 3;
string continuation_token = 4; // empty string represents null
bool has_more = 5;
bool success = 6;
string error_message = 7; // empty string represents null
google.protobuf.Timestamp timestamp = 8;
}
message DebugSnapshotRequestDto {
string instance_unique_name = 1;
string correlation_id = 2;
}
message SubscribeDebugViewRequestDto {
string instance_unique_name = 1;
string correlation_id = 2;
}
message UnsubscribeDebugViewRequestDto {
string instance_unique_name = 1;
string correlation_id = 2;
}
// UnsubscribeDebugView is a Tell today (CommunicationService.UnsubscribeDebugView
// fires and forgets). A unary RPC must still answer something, so the site sends
// this empty ack; the central transport ignores it to keep the caller-visible
// fire-and-forget semantics identical.
message UnsubscribeDebugViewAckDto {}
message AlarmConditionStateDto {
bool active = 1;
bool acknowledged = 2;
google.protobuf.BoolValue confirmed = 3; // absent => null (not confirmable)
AlarmShelveStateDto shelve = 4;
bool suppressed = 5;
int32 severity = 6;
}
message DebugAttributeValueDto {
string instance_unique_name = 1;
string attribute_path = 2;
string attribute_name = 3;
LooseValue value = 4; // absent => null
string quality = 5;
google.protobuf.Timestamp timestamp = 6;
}
// Full-fidelity projection of Commons AlarmStateChanged. Deliberately NOT the
// sitestream AlarmStateUpdate: that one flattens the value to a display string
// and the shelve state to free text, which is right for a live stream but would
// lose data on a snapshot the central UI renders as authoritative state.
message DebugAlarmStateDto {
string instance_unique_name = 1;
string alarm_name = 2;
AlarmStateDto state = 3;
int32 priority = 4;
google.protobuf.Timestamp timestamp = 5;
AlarmLevelDto level = 6;
string message = 7;
AlarmKindDto kind = 8;
// Absent means "not explicitly set" the CLR record then derives the
// computed default from state + priority. See the mapper's note on why the
// encoder omits a condition that already equals that derived default.
AlarmConditionStateDto condition = 9;
string source_reference = 10;
string alarm_type_name = 11;
string category = 12;
string operator_user = 13;
string operator_comment = 14;
google.protobuf.Timestamp original_raise_time = 15; // absent => null
string current_value = 16;
string limit_value = 17;
string native_source_canonical_name = 18;
bool is_configured_placeholder = 19;
}
message DebugViewSnapshotDto {
string instance_unique_name = 1;
repeated DebugAttributeValueDto attribute_values = 2;
repeated DebugAlarmStateDto alarm_states = 3;
google.protobuf.Timestamp snapshot_timestamp = 4;
bool instance_not_found = 5;
}
message QueryRequest {
oneof command {
EventLogQueryRequestDto event_log_query = 1;
DebugSnapshotRequestDto debug_snapshot = 2;
SubscribeDebugViewRequestDto subscribe_debug_view = 3;
UnsubscribeDebugViewRequestDto unsubscribe_debug_view = 4;
}
}
message QueryReply {
oneof reply {
EventLogQueryResponseDto event_log_query = 1;
DebugViewSnapshotDto debug_view_snapshot = 2;
UnsubscribeDebugViewAckDto unsubscribe_debug_view = 3;
}
}
//
// RPC 4 ExecuteParked
//
message ParkedMessageQueryRequestDto {
string correlation_id = 1;
string site_id = 2;
int32 page_number = 3;
int32 page_size = 4;
google.protobuf.Timestamp timestamp = 5;
}
message ParkedMessageEntryDto {
string message_id = 1;
string target_system = 2;
string method_name = 3;
string error_message = 4;
int32 attempt_count = 5;
google.protobuf.Timestamp original_timestamp = 6;
google.protobuf.Timestamp last_attempt_timestamp = 7;
int32 max_attempts = 8;
StoreAndForwardCategoryDto category = 9;
string origin_instance = 10; // empty string represents null
}
message ParkedMessageQueryResponseDto {
string correlation_id = 1;
string site_id = 2;
repeated ParkedMessageEntryDto messages = 3;
int32 total_count = 4;
int32 page_number = 5;
int32 page_size = 6;
bool success = 7;
string error_message = 8; // empty string represents null
google.protobuf.Timestamp timestamp = 9;
}
message ParkedMessageRetryRequestDto {
string correlation_id = 1;
string site_id = 2;
string message_id = 3;
google.protobuf.Timestamp timestamp = 4;
}
message ParkedMessageRetryResponseDto {
string correlation_id = 1;
bool success = 2;
string error_message = 3; // empty string represents null
}
message ParkedMessageDiscardRequestDto {
string correlation_id = 1;
string site_id = 2;
string message_id = 3;
google.protobuf.Timestamp timestamp = 4;
}
message ParkedMessageDiscardResponseDto {
string correlation_id = 1;
bool success = 2;
string error_message = 3; // empty string represents null
}
message RetryParkedOperationDto {
string correlation_id = 1;
string tracked_operation_id = 2; // TrackedOperationId GUID, "D" format
}
message DiscardParkedOperationDto {
string correlation_id = 1;
string tracked_operation_id = 2; // TrackedOperationId GUID, "D" format
}
message ParkedOperationActionAckDto {
string correlation_id = 1;
bool applied = 2;
string error_message = 3; // empty string represents null
}
message ParkedRequest {
oneof command {
ParkedMessageQueryRequestDto parked_message_query = 1;
ParkedMessageRetryRequestDto parked_message_retry = 2;
ParkedMessageDiscardRequestDto parked_message_discard = 3;
RetryParkedOperationDto retry_parked_operation = 4;
DiscardParkedOperationDto discard_parked_operation = 5;
}
}
message ParkedReply {
oneof reply {
ParkedMessageQueryResponseDto parked_message_query = 1;
ParkedMessageRetryResponseDto parked_message_retry = 2;
ParkedMessageDiscardResponseDto parked_message_discard = 3;
ParkedOperationActionAckDto parked_operation_action = 4;
}
}
//
// RPC 5 ExecuteRoute
//
message RouteToCallRequestDto {
string correlation_id = 1;
string instance_unique_name = 2;
string script_name = 3;
LooseValueMap parameters = 4; // absent => null (distinct from empty)
google.protobuf.Timestamp timestamp = 5;
string parent_execution_id = 6; // empty string represents null
}
message RouteToCallResponseDto {
string correlation_id = 1;
bool success = 2;
LooseValue return_value = 3; // absent => null
string error_message = 4; // empty string represents null
google.protobuf.Timestamp timestamp = 5;
}
message RouteToGetAttributesRequestDto {
string correlation_id = 1;
string instance_unique_name = 2;
repeated string attribute_names = 3;
google.protobuf.Timestamp timestamp = 4;
string parent_execution_id = 5; // empty string represents null
}
message RouteToGetAttributesResponseDto {
string correlation_id = 1;
LooseValueMap values = 2; // non-nullable on the CLR side; absent => empty
bool success = 3;
string error_message = 4; // empty string represents null
google.protobuf.Timestamp timestamp = 5;
}
message RouteToSetAttributesRequestDto {
string correlation_id = 1;
string instance_unique_name = 2;
map<string, string> attribute_values = 3;
google.protobuf.Timestamp timestamp = 4;
string parent_execution_id = 5; // empty string represents null
}
message RouteToSetAttributesResponseDto {
string correlation_id = 1;
bool success = 2;
string error_message = 3; // empty string represents null
google.protobuf.Timestamp timestamp = 4;
}
message RouteToWaitForAttributeRequestDto {
string correlation_id = 1;
string instance_unique_name = 2;
string attribute_name = 3;
// NOT the empty-string-means-null convention: "wait for this attribute to
// become the empty string" is a legitimate target that must stay distinct
// from "no target supplied", so this one nullable string rides a wrapper.
google.protobuf.StringValue target_value_encoded = 4;
google.protobuf.Duration timeout = 5;
google.protobuf.Timestamp timestamp = 6;
string parent_execution_id = 7; // empty string represents null
bool require_good_quality = 8;
}
message RouteToWaitForAttributeResponseDto {
string correlation_id = 1;
bool matched = 2;
LooseValue value = 3; // absent => null
string quality = 4; // empty string represents null
bool timed_out = 5;
bool success = 6;
string error_message = 7; // empty string represents null
google.protobuf.Timestamp timestamp = 8;
}
message RouteRequest {
oneof command {
RouteToCallRequestDto route_to_call = 1;
RouteToGetAttributesRequestDto route_to_get_attributes = 2;
RouteToSetAttributesRequestDto route_to_set_attributes = 3;
RouteToWaitForAttributeRequestDto route_to_wait_for_attribute = 4;
}
}
message RouteReply {
oneof reply {
RouteToCallResponseDto route_to_call = 1;
RouteToGetAttributesResponseDto route_to_get_attributes = 2;
RouteToSetAttributesResponseDto route_to_set_attributes = 3;
RouteToWaitForAttributeResponseDto route_to_wait_for_attribute = 4;
}
}
//
// RPC 6 TriggerFailover
//
message TriggerSiteFailoverDto {
string correlation_id = 1;
string site_id = 2;
}
message SiteFailoverAckDto {
string correlation_id = 1;
bool accepted = 2;
string target_address = 3; // empty string represents null
string error_message = 4; // empty string represents null
}
File diff suppressed because it is too large Load Diff
@@ -0,0 +1,559 @@
// <auto-generated>
// Generated by the protocol buffer compiler. DO NOT EDIT!
// source: Protos/site_command.proto
// </auto-generated>
#pragma warning disable 0414, 1591, 8981, 0612
#region Designer generated code
using grpc = global::Grpc.Core;
namespace ZB.MOM.WW.ScadaBridge.Communication.Grpc {
public static partial class SiteCommandService
{
static readonly string __ServiceName = "scadabridge.sitecommand.v1.SiteCommandService";
[global::System.CodeDom.Compiler.GeneratedCode("grpc_csharp_plugin", null)]
static void __Helper_SerializeMessage(global::Google.Protobuf.IMessage message, grpc::SerializationContext context)
{
#if !GRPC_DISABLE_PROTOBUF_BUFFER_SERIALIZATION
if (message is global::Google.Protobuf.IBufferMessage)
{
context.SetPayloadLength(message.CalculateSize());
global::Google.Protobuf.MessageExtensions.WriteTo(message, context.GetBufferWriter());
context.Complete();
return;
}
#endif
context.Complete(global::Google.Protobuf.MessageExtensions.ToByteArray(message));
}
[global::System.CodeDom.Compiler.GeneratedCode("grpc_csharp_plugin", null)]
static class __Helper_MessageCache<T>
{
public static readonly bool IsBufferMessage = global::System.Reflection.IntrospectionExtensions.GetTypeInfo(typeof(global::Google.Protobuf.IBufferMessage)).IsAssignableFrom(typeof(T));
}
[global::System.CodeDom.Compiler.GeneratedCode("grpc_csharp_plugin", null)]
static T __Helper_DeserializeMessage<T>(grpc::DeserializationContext context, global::Google.Protobuf.MessageParser<T> parser) where T : global::Google.Protobuf.IMessage<T>
{
#if !GRPC_DISABLE_PROTOBUF_BUFFER_SERIALIZATION
if (__Helper_MessageCache<T>.IsBufferMessage)
{
return parser.ParseFrom(context.PayloadAsReadOnlySequence());
}
#endif
return parser.ParseFrom(context.PayloadAsNewBuffer());
}
[global::System.CodeDom.Compiler.GeneratedCode("grpc_csharp_plugin", null)]
static readonly grpc::Marshaller<global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.LifecycleRequest> __Marshaller_scadabridge_sitecommand_v1_LifecycleRequest = grpc::Marshallers.Create(__Helper_SerializeMessage, context => __Helper_DeserializeMessage(context, global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.LifecycleRequest.Parser));
[global::System.CodeDom.Compiler.GeneratedCode("grpc_csharp_plugin", null)]
static readonly grpc::Marshaller<global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.LifecycleReply> __Marshaller_scadabridge_sitecommand_v1_LifecycleReply = grpc::Marshallers.Create(__Helper_SerializeMessage, context => __Helper_DeserializeMessage(context, global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.LifecycleReply.Parser));
[global::System.CodeDom.Compiler.GeneratedCode("grpc_csharp_plugin", null)]
static readonly grpc::Marshaller<global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.OpcUaRequest> __Marshaller_scadabridge_sitecommand_v1_OpcUaRequest = grpc::Marshallers.Create(__Helper_SerializeMessage, context => __Helper_DeserializeMessage(context, global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.OpcUaRequest.Parser));
[global::System.CodeDom.Compiler.GeneratedCode("grpc_csharp_plugin", null)]
static readonly grpc::Marshaller<global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.OpcUaReply> __Marshaller_scadabridge_sitecommand_v1_OpcUaReply = grpc::Marshallers.Create(__Helper_SerializeMessage, context => __Helper_DeserializeMessage(context, global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.OpcUaReply.Parser));
[global::System.CodeDom.Compiler.GeneratedCode("grpc_csharp_plugin", null)]
static readonly grpc::Marshaller<global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.QueryRequest> __Marshaller_scadabridge_sitecommand_v1_QueryRequest = grpc::Marshallers.Create(__Helper_SerializeMessage, context => __Helper_DeserializeMessage(context, global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.QueryRequest.Parser));
[global::System.CodeDom.Compiler.GeneratedCode("grpc_csharp_plugin", null)]
static readonly grpc::Marshaller<global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.QueryReply> __Marshaller_scadabridge_sitecommand_v1_QueryReply = grpc::Marshallers.Create(__Helper_SerializeMessage, context => __Helper_DeserializeMessage(context, global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.QueryReply.Parser));
[global::System.CodeDom.Compiler.GeneratedCode("grpc_csharp_plugin", null)]
static readonly grpc::Marshaller<global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.ParkedRequest> __Marshaller_scadabridge_sitecommand_v1_ParkedRequest = grpc::Marshallers.Create(__Helper_SerializeMessage, context => __Helper_DeserializeMessage(context, global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.ParkedRequest.Parser));
[global::System.CodeDom.Compiler.GeneratedCode("grpc_csharp_plugin", null)]
static readonly grpc::Marshaller<global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.ParkedReply> __Marshaller_scadabridge_sitecommand_v1_ParkedReply = grpc::Marshallers.Create(__Helper_SerializeMessage, context => __Helper_DeserializeMessage(context, global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.ParkedReply.Parser));
[global::System.CodeDom.Compiler.GeneratedCode("grpc_csharp_plugin", null)]
static readonly grpc::Marshaller<global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.RouteRequest> __Marshaller_scadabridge_sitecommand_v1_RouteRequest = grpc::Marshallers.Create(__Helper_SerializeMessage, context => __Helper_DeserializeMessage(context, global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.RouteRequest.Parser));
[global::System.CodeDom.Compiler.GeneratedCode("grpc_csharp_plugin", null)]
static readonly grpc::Marshaller<global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.RouteReply> __Marshaller_scadabridge_sitecommand_v1_RouteReply = grpc::Marshallers.Create(__Helper_SerializeMessage, context => __Helper_DeserializeMessage(context, global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.RouteReply.Parser));
[global::System.CodeDom.Compiler.GeneratedCode("grpc_csharp_plugin", null)]
static readonly grpc::Marshaller<global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.TriggerSiteFailoverDto> __Marshaller_scadabridge_sitecommand_v1_TriggerSiteFailoverDto = grpc::Marshallers.Create(__Helper_SerializeMessage, context => __Helper_DeserializeMessage(context, global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.TriggerSiteFailoverDto.Parser));
[global::System.CodeDom.Compiler.GeneratedCode("grpc_csharp_plugin", null)]
static readonly grpc::Marshaller<global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.SiteFailoverAckDto> __Marshaller_scadabridge_sitecommand_v1_SiteFailoverAckDto = grpc::Marshallers.Create(__Helper_SerializeMessage, context => __Helper_DeserializeMessage(context, global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.SiteFailoverAckDto.Parser));
[global::System.CodeDom.Compiler.GeneratedCode("grpc_csharp_plugin", null)]
static readonly grpc::Method<global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.LifecycleRequest, global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.LifecycleReply> __Method_ExecuteLifecycle = new grpc::Method<global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.LifecycleRequest, global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.LifecycleReply>(
grpc::MethodType.Unary,
__ServiceName,
"ExecuteLifecycle",
__Marshaller_scadabridge_sitecommand_v1_LifecycleRequest,
__Marshaller_scadabridge_sitecommand_v1_LifecycleReply);
[global::System.CodeDom.Compiler.GeneratedCode("grpc_csharp_plugin", null)]
static readonly grpc::Method<global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.OpcUaRequest, global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.OpcUaReply> __Method_ExecuteOpcUa = new grpc::Method<global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.OpcUaRequest, global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.OpcUaReply>(
grpc::MethodType.Unary,
__ServiceName,
"ExecuteOpcUa",
__Marshaller_scadabridge_sitecommand_v1_OpcUaRequest,
__Marshaller_scadabridge_sitecommand_v1_OpcUaReply);
[global::System.CodeDom.Compiler.GeneratedCode("grpc_csharp_plugin", null)]
static readonly grpc::Method<global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.QueryRequest, global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.QueryReply> __Method_ExecuteQuery = new grpc::Method<global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.QueryRequest, global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.QueryReply>(
grpc::MethodType.Unary,
__ServiceName,
"ExecuteQuery",
__Marshaller_scadabridge_sitecommand_v1_QueryRequest,
__Marshaller_scadabridge_sitecommand_v1_QueryReply);
[global::System.CodeDom.Compiler.GeneratedCode("grpc_csharp_plugin", null)]
static readonly grpc::Method<global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.ParkedRequest, global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.ParkedReply> __Method_ExecuteParked = new grpc::Method<global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.ParkedRequest, global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.ParkedReply>(
grpc::MethodType.Unary,
__ServiceName,
"ExecuteParked",
__Marshaller_scadabridge_sitecommand_v1_ParkedRequest,
__Marshaller_scadabridge_sitecommand_v1_ParkedReply);
[global::System.CodeDom.Compiler.GeneratedCode("grpc_csharp_plugin", null)]
static readonly grpc::Method<global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.RouteRequest, global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.RouteReply> __Method_ExecuteRoute = new grpc::Method<global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.RouteRequest, global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.RouteReply>(
grpc::MethodType.Unary,
__ServiceName,
"ExecuteRoute",
__Marshaller_scadabridge_sitecommand_v1_RouteRequest,
__Marshaller_scadabridge_sitecommand_v1_RouteReply);
[global::System.CodeDom.Compiler.GeneratedCode("grpc_csharp_plugin", null)]
static readonly grpc::Method<global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.TriggerSiteFailoverDto, global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.SiteFailoverAckDto> __Method_TriggerFailover = new grpc::Method<global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.TriggerSiteFailoverDto, global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.SiteFailoverAckDto>(
grpc::MethodType.Unary,
__ServiceName,
"TriggerFailover",
__Marshaller_scadabridge_sitecommand_v1_TriggerSiteFailoverDto,
__Marshaller_scadabridge_sitecommand_v1_SiteFailoverAckDto);
/// <summary>Service descriptor</summary>
public static global::Google.Protobuf.Reflection.ServiceDescriptor Descriptor
{
get { return global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.SiteCommandReflection.Descriptor.Services[0]; }
}
/// <summary>Base class for server-side implementations of SiteCommandService</summary>
[grpc::BindServiceMethod(typeof(SiteCommandService), "BindService")]
public abstract partial class SiteCommandServiceBase
{
/// <summary>
/// Deployment + instance lifecycle (6 commands).
/// </summary>
/// <param name="request">The request received from the client.</param>
/// <param name="context">The context of the server-side call handler being invoked.</param>
/// <returns>The response to send back to the client (wrapped by a task).</returns>
[global::System.CodeDom.Compiler.GeneratedCode("grpc_csharp_plugin", null)]
public virtual global::System.Threading.Tasks.Task<global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.LifecycleReply> ExecuteLifecycle(global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.LifecycleRequest request, grpc::ServerCallContext context)
{
throw new grpc::RpcException(new grpc::Status(grpc::StatusCode.Unimplemented, ""));
}
/// <summary>
/// Interactive OPC UA / MxGateway design-time commands (8 commands).
/// </summary>
/// <param name="request">The request received from the client.</param>
/// <param name="context">The context of the server-side call handler being invoked.</param>
/// <returns>The response to send back to the client (wrapped by a task).</returns>
[global::System.CodeDom.Compiler.GeneratedCode("grpc_csharp_plugin", null)]
public virtual global::System.Threading.Tasks.Task<global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.OpcUaReply> ExecuteOpcUa(global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.OpcUaRequest request, grpc::ServerCallContext context)
{
throw new grpc::RpcException(new grpc::Status(grpc::StatusCode.Unimplemented, ""));
}
/// <summary>
/// Remote read-only queries: event log + debug view (4 commands).
/// </summary>
/// <param name="request">The request received from the client.</param>
/// <param name="context">The context of the server-side call handler being invoked.</param>
/// <returns>The response to send back to the client (wrapped by a task).</returns>
[global::System.CodeDom.Compiler.GeneratedCode("grpc_csharp_plugin", null)]
public virtual global::System.Threading.Tasks.Task<global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.QueryReply> ExecuteQuery(global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.QueryRequest request, grpc::ServerCallContext context)
{
throw new grpc::RpcException(new grpc::Status(grpc::StatusCode.Unimplemented, ""));
}
/// <summary>
/// Parked store-and-forward message + cached-operation actions (5 commands).
/// </summary>
/// <param name="request">The request received from the client.</param>
/// <param name="context">The context of the server-side call handler being invoked.</param>
/// <returns>The response to send back to the client (wrapped by a task).</returns>
[global::System.CodeDom.Compiler.GeneratedCode("grpc_csharp_plugin", null)]
public virtual global::System.Threading.Tasks.Task<global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.ParkedReply> ExecuteParked(global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.ParkedRequest request, grpc::ServerCallContext context)
{
throw new grpc::RpcException(new grpc::Status(grpc::StatusCode.Unimplemented, ""));
}
/// <summary>
/// Inbound-API Route.To() relays (4 commands).
/// </summary>
/// <param name="request">The request received from the client.</param>
/// <param name="context">The context of the server-side call handler being invoked.</param>
/// <returns>The response to send back to the client (wrapped by a task).</returns>
[global::System.CodeDom.Compiler.GeneratedCode("grpc_csharp_plugin", null)]
public virtual global::System.Threading.Tasks.Task<global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.RouteReply> ExecuteRoute(global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.RouteRequest request, grpc::ServerCallContext context)
{
throw new grpc::RpcException(new grpc::Status(grpc::StatusCode.Unimplemented, ""));
}
/// <summary>
/// Operator-initiated manual site-pair failover (1 command).
/// </summary>
/// <param name="request">The request received from the client.</param>
/// <param name="context">The context of the server-side call handler being invoked.</param>
/// <returns>The response to send back to the client (wrapped by a task).</returns>
[global::System.CodeDom.Compiler.GeneratedCode("grpc_csharp_plugin", null)]
public virtual global::System.Threading.Tasks.Task<global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.SiteFailoverAckDto> TriggerFailover(global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.TriggerSiteFailoverDto request, grpc::ServerCallContext context)
{
throw new grpc::RpcException(new grpc::Status(grpc::StatusCode.Unimplemented, ""));
}
}
/// <summary>Client for SiteCommandService</summary>
public partial class SiteCommandServiceClient : grpc::ClientBase<SiteCommandServiceClient>
{
/// <summary>Creates a new client for SiteCommandService</summary>
/// <param name="channel">The channel to use to make remote calls.</param>
[global::System.CodeDom.Compiler.GeneratedCode("grpc_csharp_plugin", null)]
public SiteCommandServiceClient(grpc::ChannelBase channel) : base(channel)
{
}
/// <summary>Creates a new client for SiteCommandService that uses a custom <c>CallInvoker</c>.</summary>
/// <param name="callInvoker">The callInvoker to use to make remote calls.</param>
[global::System.CodeDom.Compiler.GeneratedCode("grpc_csharp_plugin", null)]
public SiteCommandServiceClient(grpc::CallInvoker callInvoker) : base(callInvoker)
{
}
/// <summary>Protected parameterless constructor to allow creation of test doubles.</summary>
[global::System.CodeDom.Compiler.GeneratedCode("grpc_csharp_plugin", null)]
protected SiteCommandServiceClient() : base()
{
}
/// <summary>Protected constructor to allow creation of configured clients.</summary>
/// <param name="configuration">The client configuration.</param>
[global::System.CodeDom.Compiler.GeneratedCode("grpc_csharp_plugin", null)]
protected SiteCommandServiceClient(ClientBaseConfiguration configuration) : base(configuration)
{
}
/// <summary>
/// Deployment + instance lifecycle (6 commands).
/// </summary>
/// <param name="request">The request to send to the server.</param>
/// <param name="headers">The initial metadata to send with the call. This parameter is optional.</param>
/// <param name="deadline">An optional deadline for the call. The call will be cancelled if deadline is hit.</param>
/// <param name="cancellationToken">An optional token for canceling the call.</param>
/// <returns>The response received from the server.</returns>
[global::System.CodeDom.Compiler.GeneratedCode("grpc_csharp_plugin", null)]
public virtual global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.LifecycleReply ExecuteLifecycle(global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.LifecycleRequest request, grpc::Metadata headers = null, global::System.DateTime? deadline = null, global::System.Threading.CancellationToken cancellationToken = default(global::System.Threading.CancellationToken))
{
return ExecuteLifecycle(request, new grpc::CallOptions(headers, deadline, cancellationToken));
}
/// <summary>
/// Deployment + instance lifecycle (6 commands).
/// </summary>
/// <param name="request">The request to send to the server.</param>
/// <param name="options">The options for the call.</param>
/// <returns>The response received from the server.</returns>
[global::System.CodeDom.Compiler.GeneratedCode("grpc_csharp_plugin", null)]
public virtual global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.LifecycleReply ExecuteLifecycle(global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.LifecycleRequest request, grpc::CallOptions options)
{
return CallInvoker.BlockingUnaryCall(__Method_ExecuteLifecycle, null, options, request);
}
/// <summary>
/// Deployment + instance lifecycle (6 commands).
/// </summary>
/// <param name="request">The request to send to the server.</param>
/// <param name="headers">The initial metadata to send with the call. This parameter is optional.</param>
/// <param name="deadline">An optional deadline for the call. The call will be cancelled if deadline is hit.</param>
/// <param name="cancellationToken">An optional token for canceling the call.</param>
/// <returns>The call object.</returns>
[global::System.CodeDom.Compiler.GeneratedCode("grpc_csharp_plugin", null)]
public virtual grpc::AsyncUnaryCall<global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.LifecycleReply> ExecuteLifecycleAsync(global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.LifecycleRequest request, grpc::Metadata headers = null, global::System.DateTime? deadline = null, global::System.Threading.CancellationToken cancellationToken = default(global::System.Threading.CancellationToken))
{
return ExecuteLifecycleAsync(request, new grpc::CallOptions(headers, deadline, cancellationToken));
}
/// <summary>
/// Deployment + instance lifecycle (6 commands).
/// </summary>
/// <param name="request">The request to send to the server.</param>
/// <param name="options">The options for the call.</param>
/// <returns>The call object.</returns>
[global::System.CodeDom.Compiler.GeneratedCode("grpc_csharp_plugin", null)]
public virtual grpc::AsyncUnaryCall<global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.LifecycleReply> ExecuteLifecycleAsync(global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.LifecycleRequest request, grpc::CallOptions options)
{
return CallInvoker.AsyncUnaryCall(__Method_ExecuteLifecycle, null, options, request);
}
/// <summary>
/// Interactive OPC UA / MxGateway design-time commands (8 commands).
/// </summary>
/// <param name="request">The request to send to the server.</param>
/// <param name="headers">The initial metadata to send with the call. This parameter is optional.</param>
/// <param name="deadline">An optional deadline for the call. The call will be cancelled if deadline is hit.</param>
/// <param name="cancellationToken">An optional token for canceling the call.</param>
/// <returns>The response received from the server.</returns>
[global::System.CodeDom.Compiler.GeneratedCode("grpc_csharp_plugin", null)]
public virtual global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.OpcUaReply ExecuteOpcUa(global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.OpcUaRequest request, grpc::Metadata headers = null, global::System.DateTime? deadline = null, global::System.Threading.CancellationToken cancellationToken = default(global::System.Threading.CancellationToken))
{
return ExecuteOpcUa(request, new grpc::CallOptions(headers, deadline, cancellationToken));
}
/// <summary>
/// Interactive OPC UA / MxGateway design-time commands (8 commands).
/// </summary>
/// <param name="request">The request to send to the server.</param>
/// <param name="options">The options for the call.</param>
/// <returns>The response received from the server.</returns>
[global::System.CodeDom.Compiler.GeneratedCode("grpc_csharp_plugin", null)]
public virtual global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.OpcUaReply ExecuteOpcUa(global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.OpcUaRequest request, grpc::CallOptions options)
{
return CallInvoker.BlockingUnaryCall(__Method_ExecuteOpcUa, null, options, request);
}
/// <summary>
/// Interactive OPC UA / MxGateway design-time commands (8 commands).
/// </summary>
/// <param name="request">The request to send to the server.</param>
/// <param name="headers">The initial metadata to send with the call. This parameter is optional.</param>
/// <param name="deadline">An optional deadline for the call. The call will be cancelled if deadline is hit.</param>
/// <param name="cancellationToken">An optional token for canceling the call.</param>
/// <returns>The call object.</returns>
[global::System.CodeDom.Compiler.GeneratedCode("grpc_csharp_plugin", null)]
public virtual grpc::AsyncUnaryCall<global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.OpcUaReply> ExecuteOpcUaAsync(global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.OpcUaRequest request, grpc::Metadata headers = null, global::System.DateTime? deadline = null, global::System.Threading.CancellationToken cancellationToken = default(global::System.Threading.CancellationToken))
{
return ExecuteOpcUaAsync(request, new grpc::CallOptions(headers, deadline, cancellationToken));
}
/// <summary>
/// Interactive OPC UA / MxGateway design-time commands (8 commands).
/// </summary>
/// <param name="request">The request to send to the server.</param>
/// <param name="options">The options for the call.</param>
/// <returns>The call object.</returns>
[global::System.CodeDom.Compiler.GeneratedCode("grpc_csharp_plugin", null)]
public virtual grpc::AsyncUnaryCall<global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.OpcUaReply> ExecuteOpcUaAsync(global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.OpcUaRequest request, grpc::CallOptions options)
{
return CallInvoker.AsyncUnaryCall(__Method_ExecuteOpcUa, null, options, request);
}
/// <summary>
/// Remote read-only queries: event log + debug view (4 commands).
/// </summary>
/// <param name="request">The request to send to the server.</param>
/// <param name="headers">The initial metadata to send with the call. This parameter is optional.</param>
/// <param name="deadline">An optional deadline for the call. The call will be cancelled if deadline is hit.</param>
/// <param name="cancellationToken">An optional token for canceling the call.</param>
/// <returns>The response received from the server.</returns>
[global::System.CodeDom.Compiler.GeneratedCode("grpc_csharp_plugin", null)]
public virtual global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.QueryReply ExecuteQuery(global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.QueryRequest request, grpc::Metadata headers = null, global::System.DateTime? deadline = null, global::System.Threading.CancellationToken cancellationToken = default(global::System.Threading.CancellationToken))
{
return ExecuteQuery(request, new grpc::CallOptions(headers, deadline, cancellationToken));
}
/// <summary>
/// Remote read-only queries: event log + debug view (4 commands).
/// </summary>
/// <param name="request">The request to send to the server.</param>
/// <param name="options">The options for the call.</param>
/// <returns>The response received from the server.</returns>
[global::System.CodeDom.Compiler.GeneratedCode("grpc_csharp_plugin", null)]
public virtual global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.QueryReply ExecuteQuery(global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.QueryRequest request, grpc::CallOptions options)
{
return CallInvoker.BlockingUnaryCall(__Method_ExecuteQuery, null, options, request);
}
/// <summary>
/// Remote read-only queries: event log + debug view (4 commands).
/// </summary>
/// <param name="request">The request to send to the server.</param>
/// <param name="headers">The initial metadata to send with the call. This parameter is optional.</param>
/// <param name="deadline">An optional deadline for the call. The call will be cancelled if deadline is hit.</param>
/// <param name="cancellationToken">An optional token for canceling the call.</param>
/// <returns>The call object.</returns>
[global::System.CodeDom.Compiler.GeneratedCode("grpc_csharp_plugin", null)]
public virtual grpc::AsyncUnaryCall<global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.QueryReply> ExecuteQueryAsync(global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.QueryRequest request, grpc::Metadata headers = null, global::System.DateTime? deadline = null, global::System.Threading.CancellationToken cancellationToken = default(global::System.Threading.CancellationToken))
{
return ExecuteQueryAsync(request, new grpc::CallOptions(headers, deadline, cancellationToken));
}
/// <summary>
/// Remote read-only queries: event log + debug view (4 commands).
/// </summary>
/// <param name="request">The request to send to the server.</param>
/// <param name="options">The options for the call.</param>
/// <returns>The call object.</returns>
[global::System.CodeDom.Compiler.GeneratedCode("grpc_csharp_plugin", null)]
public virtual grpc::AsyncUnaryCall<global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.QueryReply> ExecuteQueryAsync(global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.QueryRequest request, grpc::CallOptions options)
{
return CallInvoker.AsyncUnaryCall(__Method_ExecuteQuery, null, options, request);
}
/// <summary>
/// Parked store-and-forward message + cached-operation actions (5 commands).
/// </summary>
/// <param name="request">The request to send to the server.</param>
/// <param name="headers">The initial metadata to send with the call. This parameter is optional.</param>
/// <param name="deadline">An optional deadline for the call. The call will be cancelled if deadline is hit.</param>
/// <param name="cancellationToken">An optional token for canceling the call.</param>
/// <returns>The response received from the server.</returns>
[global::System.CodeDom.Compiler.GeneratedCode("grpc_csharp_plugin", null)]
public virtual global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.ParkedReply ExecuteParked(global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.ParkedRequest request, grpc::Metadata headers = null, global::System.DateTime? deadline = null, global::System.Threading.CancellationToken cancellationToken = default(global::System.Threading.CancellationToken))
{
return ExecuteParked(request, new grpc::CallOptions(headers, deadline, cancellationToken));
}
/// <summary>
/// Parked store-and-forward message + cached-operation actions (5 commands).
/// </summary>
/// <param name="request">The request to send to the server.</param>
/// <param name="options">The options for the call.</param>
/// <returns>The response received from the server.</returns>
[global::System.CodeDom.Compiler.GeneratedCode("grpc_csharp_plugin", null)]
public virtual global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.ParkedReply ExecuteParked(global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.ParkedRequest request, grpc::CallOptions options)
{
return CallInvoker.BlockingUnaryCall(__Method_ExecuteParked, null, options, request);
}
/// <summary>
/// Parked store-and-forward message + cached-operation actions (5 commands).
/// </summary>
/// <param name="request">The request to send to the server.</param>
/// <param name="headers">The initial metadata to send with the call. This parameter is optional.</param>
/// <param name="deadline">An optional deadline for the call. The call will be cancelled if deadline is hit.</param>
/// <param name="cancellationToken">An optional token for canceling the call.</param>
/// <returns>The call object.</returns>
[global::System.CodeDom.Compiler.GeneratedCode("grpc_csharp_plugin", null)]
public virtual grpc::AsyncUnaryCall<global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.ParkedReply> ExecuteParkedAsync(global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.ParkedRequest request, grpc::Metadata headers = null, global::System.DateTime? deadline = null, global::System.Threading.CancellationToken cancellationToken = default(global::System.Threading.CancellationToken))
{
return ExecuteParkedAsync(request, new grpc::CallOptions(headers, deadline, cancellationToken));
}
/// <summary>
/// Parked store-and-forward message + cached-operation actions (5 commands).
/// </summary>
/// <param name="request">The request to send to the server.</param>
/// <param name="options">The options for the call.</param>
/// <returns>The call object.</returns>
[global::System.CodeDom.Compiler.GeneratedCode("grpc_csharp_plugin", null)]
public virtual grpc::AsyncUnaryCall<global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.ParkedReply> ExecuteParkedAsync(global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.ParkedRequest request, grpc::CallOptions options)
{
return CallInvoker.AsyncUnaryCall(__Method_ExecuteParked, null, options, request);
}
/// <summary>
/// Inbound-API Route.To() relays (4 commands).
/// </summary>
/// <param name="request">The request to send to the server.</param>
/// <param name="headers">The initial metadata to send with the call. This parameter is optional.</param>
/// <param name="deadline">An optional deadline for the call. The call will be cancelled if deadline is hit.</param>
/// <param name="cancellationToken">An optional token for canceling the call.</param>
/// <returns>The response received from the server.</returns>
[global::System.CodeDom.Compiler.GeneratedCode("grpc_csharp_plugin", null)]
public virtual global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.RouteReply ExecuteRoute(global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.RouteRequest request, grpc::Metadata headers = null, global::System.DateTime? deadline = null, global::System.Threading.CancellationToken cancellationToken = default(global::System.Threading.CancellationToken))
{
return ExecuteRoute(request, new grpc::CallOptions(headers, deadline, cancellationToken));
}
/// <summary>
/// Inbound-API Route.To() relays (4 commands).
/// </summary>
/// <param name="request">The request to send to the server.</param>
/// <param name="options">The options for the call.</param>
/// <returns>The response received from the server.</returns>
[global::System.CodeDom.Compiler.GeneratedCode("grpc_csharp_plugin", null)]
public virtual global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.RouteReply ExecuteRoute(global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.RouteRequest request, grpc::CallOptions options)
{
return CallInvoker.BlockingUnaryCall(__Method_ExecuteRoute, null, options, request);
}
/// <summary>
/// Inbound-API Route.To() relays (4 commands).
/// </summary>
/// <param name="request">The request to send to the server.</param>
/// <param name="headers">The initial metadata to send with the call. This parameter is optional.</param>
/// <param name="deadline">An optional deadline for the call. The call will be cancelled if deadline is hit.</param>
/// <param name="cancellationToken">An optional token for canceling the call.</param>
/// <returns>The call object.</returns>
[global::System.CodeDom.Compiler.GeneratedCode("grpc_csharp_plugin", null)]
public virtual grpc::AsyncUnaryCall<global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.RouteReply> ExecuteRouteAsync(global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.RouteRequest request, grpc::Metadata headers = null, global::System.DateTime? deadline = null, global::System.Threading.CancellationToken cancellationToken = default(global::System.Threading.CancellationToken))
{
return ExecuteRouteAsync(request, new grpc::CallOptions(headers, deadline, cancellationToken));
}
/// <summary>
/// Inbound-API Route.To() relays (4 commands).
/// </summary>
/// <param name="request">The request to send to the server.</param>
/// <param name="options">The options for the call.</param>
/// <returns>The call object.</returns>
[global::System.CodeDom.Compiler.GeneratedCode("grpc_csharp_plugin", null)]
public virtual grpc::AsyncUnaryCall<global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.RouteReply> ExecuteRouteAsync(global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.RouteRequest request, grpc::CallOptions options)
{
return CallInvoker.AsyncUnaryCall(__Method_ExecuteRoute, null, options, request);
}
/// <summary>
/// Operator-initiated manual site-pair failover (1 command).
/// </summary>
/// <param name="request">The request to send to the server.</param>
/// <param name="headers">The initial metadata to send with the call. This parameter is optional.</param>
/// <param name="deadline">An optional deadline for the call. The call will be cancelled if deadline is hit.</param>
/// <param name="cancellationToken">An optional token for canceling the call.</param>
/// <returns>The response received from the server.</returns>
[global::System.CodeDom.Compiler.GeneratedCode("grpc_csharp_plugin", null)]
public virtual global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.SiteFailoverAckDto TriggerFailover(global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.TriggerSiteFailoverDto request, grpc::Metadata headers = null, global::System.DateTime? deadline = null, global::System.Threading.CancellationToken cancellationToken = default(global::System.Threading.CancellationToken))
{
return TriggerFailover(request, new grpc::CallOptions(headers, deadline, cancellationToken));
}
/// <summary>
/// Operator-initiated manual site-pair failover (1 command).
/// </summary>
/// <param name="request">The request to send to the server.</param>
/// <param name="options">The options for the call.</param>
/// <returns>The response received from the server.</returns>
[global::System.CodeDom.Compiler.GeneratedCode("grpc_csharp_plugin", null)]
public virtual global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.SiteFailoverAckDto TriggerFailover(global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.TriggerSiteFailoverDto request, grpc::CallOptions options)
{
return CallInvoker.BlockingUnaryCall(__Method_TriggerFailover, null, options, request);
}
/// <summary>
/// Operator-initiated manual site-pair failover (1 command).
/// </summary>
/// <param name="request">The request to send to the server.</param>
/// <param name="headers">The initial metadata to send with the call. This parameter is optional.</param>
/// <param name="deadline">An optional deadline for the call. The call will be cancelled if deadline is hit.</param>
/// <param name="cancellationToken">An optional token for canceling the call.</param>
/// <returns>The call object.</returns>
[global::System.CodeDom.Compiler.GeneratedCode("grpc_csharp_plugin", null)]
public virtual grpc::AsyncUnaryCall<global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.SiteFailoverAckDto> TriggerFailoverAsync(global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.TriggerSiteFailoverDto request, grpc::Metadata headers = null, global::System.DateTime? deadline = null, global::System.Threading.CancellationToken cancellationToken = default(global::System.Threading.CancellationToken))
{
return TriggerFailoverAsync(request, new grpc::CallOptions(headers, deadline, cancellationToken));
}
/// <summary>
/// Operator-initiated manual site-pair failover (1 command).
/// </summary>
/// <param name="request">The request to send to the server.</param>
/// <param name="options">The options for the call.</param>
/// <returns>The call object.</returns>
[global::System.CodeDom.Compiler.GeneratedCode("grpc_csharp_plugin", null)]
public virtual grpc::AsyncUnaryCall<global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.SiteFailoverAckDto> TriggerFailoverAsync(global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.TriggerSiteFailoverDto request, grpc::CallOptions options)
{
return CallInvoker.AsyncUnaryCall(__Method_TriggerFailover, null, options, request);
}
/// <summary>Creates a new instance of client from given <c>ClientBaseConfiguration</c>.</summary>
[global::System.CodeDom.Compiler.GeneratedCode("grpc_csharp_plugin", null)]
protected override SiteCommandServiceClient NewInstance(ClientBaseConfiguration configuration)
{
return new SiteCommandServiceClient(configuration);
}
}
/// <summary>Creates service definition that can be registered with a server</summary>
/// <param name="serviceImpl">An object implementing the server-side handling logic.</param>
[global::System.CodeDom.Compiler.GeneratedCode("grpc_csharp_plugin", null)]
public static grpc::ServerServiceDefinition BindService(SiteCommandServiceBase serviceImpl)
{
return grpc::ServerServiceDefinition.CreateBuilder()
.AddMethod(__Method_ExecuteLifecycle, serviceImpl.ExecuteLifecycle)
.AddMethod(__Method_ExecuteOpcUa, serviceImpl.ExecuteOpcUa)
.AddMethod(__Method_ExecuteQuery, serviceImpl.ExecuteQuery)
.AddMethod(__Method_ExecuteParked, serviceImpl.ExecuteParked)
.AddMethod(__Method_ExecuteRoute, serviceImpl.ExecuteRoute)
.AddMethod(__Method_TriggerFailover, serviceImpl.TriggerFailover).Build();
}
/// <summary>Register service method with a service binder with or without implementation. Useful when customizing the service binding logic.
/// Note: this method is part of an experimental API that can change or be removed without any prior notice.</summary>
/// <param name="serviceBinder">Service methods will be bound by calling <c>AddMethod</c> on this object.</param>
/// <param name="serviceImpl">An object implementing the server-side handling logic.</param>
[global::System.CodeDom.Compiler.GeneratedCode("grpc_csharp_plugin", null)]
public static void BindService(grpc::ServiceBinderBase serviceBinder, SiteCommandServiceBase serviceImpl)
{
serviceBinder.AddMethod(__Method_ExecuteLifecycle, serviceImpl == null ? null : new grpc::UnaryServerMethod<global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.LifecycleRequest, global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.LifecycleReply>(serviceImpl.ExecuteLifecycle));
serviceBinder.AddMethod(__Method_ExecuteOpcUa, serviceImpl == null ? null : new grpc::UnaryServerMethod<global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.OpcUaRequest, global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.OpcUaReply>(serviceImpl.ExecuteOpcUa));
serviceBinder.AddMethod(__Method_ExecuteQuery, serviceImpl == null ? null : new grpc::UnaryServerMethod<global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.QueryRequest, global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.QueryReply>(serviceImpl.ExecuteQuery));
serviceBinder.AddMethod(__Method_ExecuteParked, serviceImpl == null ? null : new grpc::UnaryServerMethod<global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.ParkedRequest, global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.ParkedReply>(serviceImpl.ExecuteParked));
serviceBinder.AddMethod(__Method_ExecuteRoute, serviceImpl == null ? null : new grpc::UnaryServerMethod<global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.RouteRequest, global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.RouteReply>(serviceImpl.ExecuteRoute));
serviceBinder.AddMethod(__Method_TriggerFailover, serviceImpl == null ? null : new grpc::UnaryServerMethod<global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.TriggerSiteFailoverDto, global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.SiteFailoverAckDto>(serviceImpl.TriggerFailover));
}
}
}
#endregion
@@ -10,6 +10,9 @@
<ItemGroup>
<InternalsVisibleTo Include="ZB.MOM.WW.ScadaBridge.Communication.Tests" />
<InternalsVisibleTo Include="ZB.MOM.WW.ScadaBridge.IntegrationTests" />
<!-- GrpcSiteTransport failover/failback TestServer coverage (T1B.3) lives in Host.Tests,
which owns the ASP.NET TestHost stack; it drives the provider's internal failback seams. -->
<InternalsVisibleTo Include="ZB.MOM.WW.ScadaBridge.Host.Tests" />
</ItemGroup>
<ItemGroup>
@@ -32,30 +35,31 @@
<ProjectReference Include="../ZB.MOM.WW.ScadaBridge.HealthMonitoring/ZB.MOM.WW.ScadaBridge.HealthMonitoring.csproj" />
</ItemGroup>
<!-- gRPC proto generation. The compiled C# is checked in — SiteStreamGrpc/
(Sitestream.cs + SitestreamGrpc.cs) for Protos/sitestream.proto, and
<!-- gRPC proto generation. The compiled C# is checked in — under SiteStreamGrpc/
(Sitestream.cs + SitestreamGrpc.cs) for Protos/sitestream.proto,
CentralControlGrpc/ (CentralControl.cs + CentralControlGrpc.cs) for
Protos/central_control.proto — because protoc segfaults inside our
linux_arm64 Docker build image. To regenerate after schema changes run
`docker/regen-proto.sh [sitestream|centralcontrol|all]`, which does all
of the following and always leaves this file as it found it:
1. Temporarily uncomment the Protobuf ItemGroup below (just the line
for the proto you changed — the other file's checked-in C# is
already compiled, so enabling both at once duplicates types).
Protos/central_control.proto, and SiteCommandGrpc/ (SiteCommand.cs +
SiteCommandGrpc.cs) for Protos/site_command.proto — because protoc segfaults
inside our linux_arm64 Docker build image. To regenerate after schema changes
run `docker/regen-proto.sh [sitestream|centralcontrol|sitecommand|all]`, which
does all of the following and always leaves this file as it found it:
1. Temporarily uncomment the Protobuf ItemGroup below (just the line for
the proto you changed — the other files' checked-in C# is already
compiled, so enabling several at once duplicates types).
2. Delete the matching checked-in *.cs.
3. `dotnet build` (on macOS) — Grpc.Tools writes fresh files to obj/.
4. Copy obj/Debug/net10.0/Protos/*.cs into the matching folder.
5. Re-comment the ItemGroup.
central_control.proto imports sitestream.proto, so protoc resolves it
from the project-relative path without sitestream.proto needing its own
Protobuf item.
An ACTIVE Protobuf item must never be committed — it breaks the Docker
image build. Eventually we should switch the Docker build image to one
with a working protoc on arm64. -->
central_control.proto imports sitestream.proto, so protoc resolves it from the
project-relative path without sitestream.proto needing its own Protobuf item.
An ACTIVE Protobuf item must never be committed — it breaks the Docker image
build. Eventually we should switch the Docker build image to one with a working
protoc on arm64. -->
<!--
<ItemGroup>
<Protobuf Include="Protos\sitestream.proto" GrpcServices="Both" />
<Protobuf Include="Protos\central_control.proto" GrpcServices="Both" />
<Protobuf Include="Protos\site_command.proto" GrpcServices="Both" />
</ItemGroup>
-->
@@ -1,7 +1,5 @@
using System.Collections.Immutable;
using Akka.Actor;
using Akka.Cluster;
using Akka.Cluster.Tools.Client;
using Akka.Cluster.Tools.Singleton;
using Akka.Configuration;
using Microsoft.Extensions.Options;
@@ -428,15 +426,21 @@ akka {{
var centralHealthCollector = _serviceProvider.GetService<ZB.MOM.WW.ScadaBridge.HealthMonitoring.ISiteHealthCollector>();
centralHealthCollector?.SetNodeHostname(_nodeOptions.NodeHostname);
var siteClientFactory = new DefaultSiteClientFactory();
// Central→site command transport: gRPC SiteCommandService. This is the ONLY transport since
// the ClusterClient→gRPC migration's Phase 4 removed the Akka path; the actor no longer
// chooses a transport, the Host builds it. Constructed once from the DI
// SitePairChannelProvider and shared across CentralCommunicationActor incarnations (it holds
// only the shared provider + options); the actor reconciles its per-site channel pairs from
// the DB refresh loop.
var siteCommandTransport = new GrpcSiteTransport(
_serviceProvider.GetRequiredService<SitePairChannelProvider>(),
_communicationOptions,
_serviceProvider.GetRequiredService<ILoggerFactory>().CreateLogger<GrpcSiteTransport>());
_logger.LogInformation("central→site command transport: gRPC (SiteCommandService)");
var centralCommActor = _actorSystem!.ActorOf(
Props.Create(() => new CentralCommunicationActor(_serviceProvider, siteClientFactory)),
Props.Create(() => new CentralCommunicationActor(_serviceProvider, siteCommandTransport)),
"central-communication");
// Register CentralCommunicationActor with ClusterClientReceptionist so site ClusterClients can reach it
ClusterClientReceptionist.Get(_actorSystem).RegisterService(centralCommActor);
_logger.LogInformation("CentralCommunicationActor registered with ClusterClientReceptionist");
// Hand the same actor to the central-hosted gRPC control plane (T1A.2) and open its
// readiness gate — the gRPC face Asks this exact actor, so both transports resolve to
// one handler implementation. Mirrors SiteStreamGrpcServer.SetReady on the site side:
@@ -833,37 +837,40 @@ akka {{
_logger, role: siteRole);
var dmProxy = dm.Proxy;
// Select the site→central transport behind the coexistence flag (default Akka
// ClusterClient). When gRPC is chosen the site dials CentralControlService directly with
// a sticky-failover channel pair, presenting its own preshared key; the ClusterClient
// below is then not created at all.
ICentralTransport? centralTransport = null;
if (_communicationOptions.CentralTransport == CentralTransportMode.Grpc)
{
var loggerFactory = _serviceProvider.GetRequiredService<ILoggerFactory>();
var channelProvider = new CentralChannelProvider(
_communicationOptions.CentralGrpcEndpoints,
new StaticSitePskProvider(_communicationOptions.GrpcPsk),
_nodeOptions.SiteId!,
_communicationOptions,
loggerFactory.CreateLogger<CentralChannelProvider>());
_trackedDisposables.Add(channelProvider);
centralTransport = new GrpcCentralTransport(
channelProvider,
_communicationOptions,
loggerFactory.CreateLogger<GrpcCentralTransport>());
_logger.LogInformation(
"Site→central transport: gRPC to {Count} central endpoint(s) for site {SiteId}",
_communicationOptions.CentralGrpcEndpoints.Count, _nodeOptions.SiteId);
}
else
{
_logger.LogInformation(
"Site→central transport: Akka ClusterClient (default) for site {SiteId}",
_nodeOptions.SiteId);
}
// Site→central transport: a gRPC dial of CentralControlService with a sticky
// central-a→central-b channel pair, presenting this site's preshared key. This is the ONLY
// transport since the ClusterClient→gRPC migration's Phase 4 removed the Akka path;
// StartupValidator guarantees a Site node lists at least one CentralGrpcEndpoint.
var loggerFactory = _serviceProvider.GetRequiredService<ILoggerFactory>();
var channelProvider = new CentralChannelProvider(
_communicationOptions.CentralGrpcEndpoints,
new StaticSitePskProvider(_communicationOptions.GrpcPsk),
_nodeOptions.SiteId!,
_communicationOptions,
loggerFactory.CreateLogger<CentralChannelProvider>());
_trackedDisposables.Add(channelProvider);
ICentralTransport centralTransport = new GrpcCentralTransport(
channelProvider,
_communicationOptions,
loggerFactory.CreateLogger<GrpcCentralTransport>());
_logger.LogInformation(
"Site→central transport: gRPC to {Count} central endpoint(s) for site {SiteId}",
_communicationOptions.CentralGrpcEndpoints.Count, _nodeOptions.SiteId);
// Create SiteCommunicationActor for receiving messages from central
// The ONE routing table for central→site commands, shared by the Akka
// SiteCommunicationActor (below) and the gRPC SiteCommandGrpcService (SetReady at the end
// of this method). Its failover seam resolves (dryRun) or performs the graceful Leave via
// the shared ClusterFailoverCoordinator so the central/site paths cannot drift.
var siteCommandDispatcher = new SiteCommandDispatcher(
_nodeOptions.SiteId!,
dmProxy,
(role, dryRun) => ZB.MOM.WW.ScadaBridge.Communication.ClusterState.ClusterFailoverCoordinator
.FailOverOldest(_actorSystem!, role, dryRun)?.ToString());
// Create SiteCommunicationActor for receiving messages from central. It routes commands
// through the shared dispatcher; RegisterLocalHandler (below) registers the handlers INTO
// that dispatcher, so the gRPC command service sees the same registrations. The site→central
// transport is selected above (default Akka); both are passed in.
var siteCommActor = _actorSystem.ActorOf(
Props.Create(() => new SiteCommunicationActor(
_nodeOptions.SiteId!,
@@ -871,7 +878,8 @@ akka {{
dmProxy,
activeNodeCheck,
null,
centralTransport)),
centralTransport,
siteCommandDispatcher)),
"site-communication");
// Register local handlers with SiteCommunicationActor
@@ -983,36 +991,10 @@ akka {{
siteCommActor.Tell(new RegisterLocalHandler(LocalHandlerType.ParkedMessages, parkedMessageHandler));
}
// Register SiteCommunicationActor with ClusterClientReceptionist so central ClusterClients can reach it
ClusterClientReceptionist.Get(_actorSystem).RegisterService(siteCommActor);
_logger.LogInformation(
"Site actors registered. DeploymentManager singleton scoped to role={SiteRole}, SiteCommunicationActor created.",
siteRole);
// Create ClusterClient to central if contact points are configured — but only on the Akka
// transport. On the gRPC transport the SiteCommunicationActor already holds a
// GrpcCentralTransport and never receives RegisterCentralClient, so a ClusterClient here
// would be dead weight (and keep an unwanted cross-cluster Akka association alive).
if (_communicationOptions.CentralTransport == CentralTransportMode.Akka
&& _communicationOptions.CentralContactPoints.Count > 0)
{
var contacts = _communicationOptions.CentralContactPoints
.Select(cp => ActorPath.Parse($"{cp}/system/receptionist"))
.ToImmutableHashSet();
var clientSettings = ClusterClientSettings.Create(_actorSystem)
.WithInitialContacts(contacts);
var centralClient = _actorSystem.ActorOf(
ClusterClient.Props(clientSettings), "central-cluster-client");
var siteCommSelection = _actorSystem.ActorSelection("/user/site-communication");
siteCommSelection.Tell(new RegisterCentralClient(centralClient));
_logger.LogInformation(
"Created ClusterClient to central with {Count} contact point(s) for site {SiteId}",
contacts.Count, _nodeOptions.SiteId);
}
// Per-node startup reconciliation. Created on EVERY site node (NOT a
// singleton) so a standby that was DOWN during a deploy self-heals on its next
// restart: it reports its local deployed inventory to central via the
@@ -1141,5 +1123,12 @@ akka {{
grpcServer?.SetOperationTrackingStore(siteTrackingStore);
}
grpcServer?.SetReady(_actorSystem!);
// Site command plane: hand the shared dispatcher to the gRPC command service and flip it
// ready at the same point as the streaming server — the actor graph exists and the local
// handlers have been registered into the dispatcher, so it can route commands now.
var commandService = _serviceProvider
.GetService<ZB.MOM.WW.ScadaBridge.Communication.Grpc.SiteCommandGrpcService>();
commandService?.SetReady(siteCommandDispatcher);
}
}
@@ -51,12 +51,18 @@ namespace ZB.MOM.WW.ScadaBridge.Host;
public sealed class ControlPlaneAuthInterceptor : Interceptor
{
/// <summary>
/// Service prefixes gated by default. Read from the generated <c>sitestream.proto</c>
/// package/service names — <c>package sitestream; service SiteStreamService</c>.
/// Later phases append their own services here.
/// Service prefixes gated by default. Read from the generated service descriptors —
/// <c>package sitestream; service SiteStreamService</c> (real-time data + audit pull) and
/// <c>package scadabridge.sitecommand.v1; service SiteCommandService</c> (the T1B command
/// plane). Later phases append their own services here rather than adding a second
/// interceptor or constructor (see the public constructor's remarks).
/// </summary>
public static readonly IReadOnlyList<string> DefaultGatedPrefixes =
new[] { "/sitestream.SiteStreamService/" };
new[]
{
$"/{SiteStreamService.Descriptor.FullName}/",
$"/{SiteCommandService.Descriptor.FullName}/",
};
private readonly IReadOnlyList<string> _gatedPrefixes;
private readonly IOptions<CommunicationOptions> _options;
+15
View File
@@ -167,6 +167,14 @@ try
});
builder.Services.AddSingleton<
ZB.MOM.WW.ScadaBridge.Communication.Grpc.CentralControlGrpcService>();
// Shared per-site gRPC channel-pair provider (sticky failover/failback) backing the
// GrpcSiteTransport when ScadaBridge:Communication:SiteTransport=Grpc. Central-only, and a
// no-op until CentralCommunicationActor builds the gRPC transport (its failback loop starts
// on first use), so registering it unconditionally is harmless under the default Akka path.
builder.Services.AddSingleton<
ZB.MOM.WW.ScadaBridge.Communication.Grpc.SitePairChannelProvider>();
builder.Services.AddHealthMonitoring();
builder.Services.AddCentralHealthAggregation();
builder.Services.AddExternalSystemGateway();
@@ -610,6 +618,10 @@ try
options.Interceptors.Add<ControlPlaneAuthInterceptor>();
});
builder.Services.AddSingleton<ZB.MOM.WW.ScadaBridge.Communication.Grpc.SiteStreamGrpcServer>();
// Site command plane (central→site) — the gRPC peer of SiteStreamGrpcServer. Both share
// the h2c listener and the ControlPlaneAuthInterceptor PSK gate; both are readiness-gated
// (SetReady, below). Central still dials over ClusterClient until T1B.3.
builder.Services.AddSingleton<ZB.MOM.WW.ScadaBridge.Communication.Grpc.SiteCommandGrpcService>();
// Existing site service registrations (this is also where LocalDb and its
// replication engine are registered — see SiteServiceRegistration)
@@ -631,6 +643,9 @@ try
// Map gRPC service — resolves the singleton SiteStreamGrpcServer from DI
app.MapGrpcService<ZB.MOM.WW.ScadaBridge.Communication.Grpc.SiteStreamGrpcServer>();
// Map the site command plane (central→site) alongside it, on the same gated listener.
app.MapGrpcService<ZB.MOM.WW.ScadaBridge.Communication.Grpc.SiteCommandGrpcService>();
// The passive half of LocalDb replication: the peer node dials THIS endpoint.
// It shares the Kestrel h2c listener the site gRPC server already uses, so no
// listener or port changes are needed. Mapping it is harmless with no peer
@@ -143,6 +143,22 @@ public static class StartupValidator
+ "production as ${secret:SB-GRPC-PSK-<siteId>}) and under the secret "
+ "name SB-GRPC-PSK-<siteId> in central's secret store");
// gRPC (CentralControlService) is the only site→central transport after the
// ClusterClient→gRPC migration's Phase 4 — the Akka ClusterClient path and its
// CentralContactPoints option are gone. A site with no central gRPC endpoint has
// nothing to dial: heartbeats, health reports, notification forwards and audit
// ingest all silently fail. The shared CommunicationOptionsValidator only rejects
// BLANK entries (it is role-agnostic, and central nodes legitimately leave the list
// empty), so the "a Site must have at least one" rule lives here, where the role is
// known. The predicate reads index :0 directly, so its non-empty presence proves the
// list has a usable first endpoint.
p.Require("ScadaBridge:Communication:CentralGrpcEndpoints:0",
value => !string.IsNullOrWhiteSpace(value),
"is required for Site nodes: gRPC (CentralControlService) is the only "
+ "site→central transport, so each site must list at least one central gRPC "
+ "endpoint under ScadaBridge:Communication:CentralGrpcEndpoints "
+ "(e.g. http://scadabridge-central-a:8083). Central nodes leave it empty.");
// ScadaBridge:Database:SiteDbPath was required here until LocalDb
// Phase 2. The site's tables now live in the consolidated LocalDb
// database (LocalDb:Path, which SiteServiceRegistration requires),
@@ -44,9 +44,9 @@
"Communication": {
"_grpcPsk": "REQUIRED on Site nodes (StartupValidator fails the boot without it). The preshared key the gRPC control plane authenticates with: ControlPlaneAuthInterceptor is fail-closed, so an unset key refuses every SiteStream call — live subscriptions, audit pulls, cached-telemetry ingest — while the node still reports healthy. Supply it as ${secret:SB-GRPC-PSK-<siteId>} so the plaintext never sits in this file, and seed the SAME value on central: either as the secret SB-GRPC-PSK-<siteId> in its store, or as ScadaBridge:Communication:SitePsks:<siteId>. BOTH nodes of the pair carry the same key. Distinct from LocalDb:Replication:ApiKey, which authenticates the pair partner, not central — never share the two.",
"GrpcPsk": "${secret:SB-GRPC-PSK-site-1}",
"_centralContactPoints": "Host-016: each entry MUST be a central node's remoting endpoint, NOT this site's own remoting port. The single dev-loopback default below points only at central-a (localhost:8081). In a multi-central deployment add the second central node here (e.g. 'akka.tcp://scadabridge@central-b-host:8081') so ClusterClient can fail over when central-a is down. The previous template listed localhost:8082 as the second contact — that is THIS site's own RemotingPort and is a permanent failure in the initial-contact rotation.",
"CentralContactPoints": [
"akka.tcp://scadabridge@localhost:8081"
"_centralGrpcEndpoints": "gRPC (CentralControlService) is the only site→central transport since the ClusterClient→gRPC migration's Phase 4. Each entry MUST be a central node's gRPC (h2c) endpoint on its CentralGrpcPort (default 8083) — NOT this site's own gRPC port, and NOT via Traefik (HTTP/1 only). The single dev-loopback default below points only at central-a (localhost:8083). In a multi-central deployment add the second central node here (e.g. 'http://central-b-host:8083') so the channel pair can fail over when central-a is down. StartupValidator requires a Site node to list at least one endpoint.",
"CentralGrpcEndpoints": [
"http://localhost:8083"
],
"DeploymentTimeout": "00:02:00",
"LifecycleTimeout": "00:00:30",
@@ -1,67 +0,0 @@
using Akka.Actor;
using Akka.Cluster.Tools.Client;
using Akka.TestKit.Xunit2;
using ZB.MOM.WW.ScadaBridge.Commons.Messages.Notification;
using ZB.MOM.WW.ScadaBridge.Communication.Actors;
namespace ZB.MOM.WW.ScadaBridge.Communication.Tests.Actors;
/// <summary>
/// T1A.3: the regression-prone bit of the Akka transport — that a forwarded
/// <see cref="ClusterClient.Send"/> carries the caller's sender, so central's reply routes
/// straight back to the waiting Ask rather than to the site communication actor — plus the
/// no-ClusterClient-yet fallback that keeps the S&amp;F layer treating the send as transient.
/// </summary>
public class AkkaCentralTransportTests : TestKit
{
[Fact]
public void SubmitNotification_ForwardsToClusterClient_WithReplyToAsSender()
{
var transport = new AkkaCentralTransport();
var clusterClient = CreateTestProbe();
var replyTo = CreateTestProbe();
transport.SetCentralClient(clusterClient.Ref);
var submit = new NotificationSubmit(
"notif-1", "Operators", "S", "B", "site1", null, null, DateTimeOffset.UtcNow);
transport.SubmitNotification(submit, replyTo.Ref);
// The ClusterClient receives a Send addressed to the central actor...
var send = clusterClient.ExpectMsg<ClusterClient.Send>();
Assert.Equal("/user/central-communication", send.Path);
Assert.IsType<NotificationSubmit>(send.Message);
// ...and replying to it lands at replyTo, proving the sender was forwarded (not the
// transport / actor). This is the routing the waiting Ask relies on.
clusterClient.Reply(new NotificationSubmitAck("notif-1", Accepted: true, Error: null));
replyTo.ExpectMsg<NotificationSubmitAck>(ack => ack.NotificationId == "notif-1" && ack.Accepted);
}
[Fact]
public void SubmitNotification_WithNoClusterClient_RepliesNonAcceptedToReplyTo()
{
var transport = new AkkaCentralTransport();
var replyTo = CreateTestProbe();
transport.SubmitNotification(
new NotificationSubmit("notif-2", "Operators", "S", "B", "site1", null, null, DateTimeOffset.UtcNow),
replyTo.Ref);
replyTo.ExpectMsg<NotificationSubmitAck>(ack => ack.NotificationId == "notif-2" && !ack.Accepted);
}
[Fact]
public void IngestAuditEvents_WithNoClusterClient_FaultsTheReplyTo()
{
// The audit drain treats a faulted Ask as transient and keeps rows Pending — so the
// no-client path must be a Status.Failure, not a silent drop.
var transport = new AkkaCentralTransport();
var replyTo = CreateTestProbe();
transport.IngestAuditEvents(
new Commons.Messages.Audit.IngestAuditEventsCommand(new List<ZB.MOM.WW.Audit.AuditEvent>()),
replyTo.Ref);
replyTo.ExpectMsg<Status.Failure>();
}
}
@@ -36,9 +36,9 @@ public class CentralCommunicationActorAuditTests : TestKit
services.AddScoped(_ => mockRepo);
var sp = services.BuildServiceProvider();
var mockFactory = Substitute.For<ISiteClientFactory>();
var transport = Substitute.For<ISiteCommandTransport>();
return Sys.ActorOf(Props.Create(() =>
new CentralCommunicationActor(sp, mockFactory, auditIngestAskTimeout)));
new CentralCommunicationActor(sp, transport, auditIngestAskTimeout)));
}
// C3 (Task 2.5): canonical ZB.MOM.WW.Audit.AuditEvent via the shared factory.
@@ -1,46 +1,25 @@
using System.Collections.Immutable;
using Akka.Actor;
using Akka.Cluster.Tools.Client;
using Akka.Configuration;
using Akka.TestKit.Xunit2;
using Microsoft.Extensions.DependencyInjection;
using NSubstitute;
using Xunit;
using ZB.MOM.WW.ScadaBridge.Commons.Entities.Sites;
using ZB.MOM.WW.ScadaBridge.Commons.Interfaces.Repositories;
using ZB.MOM.WW.ScadaBridge.Communication;
using ZB.MOM.WW.ScadaBridge.Communication.Actors;
using ZB.MOM.WW.ScadaBridge.HealthMonitoring;
namespace ZB.MOM.WW.ScadaBridge.Communication.Tests;
/// <summary>
/// Transport-agnostic refresh behaviour of <see cref="CentralCommunicationActor"/>: the periodic
/// DB refresh evicts deleted sites from the health aggregator. The two former tests here —
/// per-site ClusterClient recreate-without-restart-loop and factory-throw resilience — asserted
/// Akka <c>AkkaSiteTransport</c> internals (InvalidActorNameException, "No ClusterClient for site")
/// and were removed with that transport in the ClusterClient→gRPC migration's Phase 4; the
/// equivalent gRPC channel reconcile is covered by the <c>GrpcSiteTransport</c> suites.
/// </summary>
public class CentralCommunicationActorClientLifecycleTests : TestKit
{
private static readonly Config TestConfig = ConfigurationFactory.ParseString(@"
akka.actor.provider = cluster
akka.remote.dot-netty.tcp.port = 0
akka.remote.dot-netty.tcp.hostname = localhost")
.WithFallback(ClusterClientReceptionist.DefaultConfig());
public CentralCommunicationActorClientLifecycleTests() : base(TestConfig) { }
// Empty ServiceProvider with a no-op ISiteRepository so the actor's periodic
// db-refresh (fired at PreStart) resolves and returns no sites, keeping the
// logs clean; the tests drive SiteAddressCacheLoaded directly.
private static IServiceProvider EmptyProvider()
{
var siteRepo = Substitute.For<ISiteRepository>();
siteRepo.GetAllSitesAsync(Arg.Any<CancellationToken>()).Returns(new List<Site>());
var services = new ServiceCollection();
services.AddScoped(_ => siteRepo);
return services.BuildServiceProvider();
}
private static SiteAddressCacheLoaded Load(string siteId, params string[] addrs) =>
new(new Dictionary<string, IReadOnlyList<string>>
{ [siteId] = addrs.ToList().AsReadOnly() },
new[] { siteId });
[Fact]
public void PeriodicRefresh_PrunesDeletedSites_FromHealthAggregator()
{
@@ -56,8 +35,9 @@ public class CentralCommunicationActorClientLifecycleTests : TestKit
services.AddSingleton(aggregator);
var provider = services.BuildServiceProvider();
var transport = Substitute.For<ISiteCommandTransport>();
var actor = Sys.ActorOf(Props.Create(() => new CentralCommunicationActor(
provider, new DefaultSiteClientFactory(), null)));
provider, transport, (TimeSpan?)null)));
// Trigger the refresh (also fires at PreStart, but drive it explicitly so
// the assertion is deterministic). The load runs on a detached task and
@@ -69,45 +49,4 @@ public class CentralCommunicationActorClientLifecycleTests : TestKit
Arg.Is<IReadOnlyCollection<string>>(ids => ids.Contains("site-a"))),
TimeSpan.FromSeconds(3));
}
[Fact]
public void AddressEdit_RecreatesClient_WithoutRestartLoop()
{
var actor = Sys.ActorOf(Props.Create(() => new CentralCommunicationActor(
EmptyProvider(), new DefaultSiteClientFactory(), null)));
// First load creates the client; second load (edited NodeA address) stops
// the old one and creates a replacement in the same message handling.
// Pre-fix this throws InvalidActorNameException and restarts the actor.
EventFilter.Exception<InvalidActorNameException>().Expect(0, () =>
{
actor.Tell(Load("site-a", "akka.tcp://scadabridge@node-a:8081"));
actor.Tell(Load("site-a", "akka.tcp://scadabridge@node-a-edited:8081"));
actor.Tell(Load("site-a", "akka.tcp://scadabridge@node-a:8081")); // and back — third generation
});
// The actor must still be alive and routing (not crash-looping):
// an envelope for an unknown site produces the "No ClusterClient" warning,
// proving the Receive pipeline is healthy.
EventFilter.Warning(contains: "No ClusterClient for site").ExpectOne(() =>
actor.Tell(new SiteEnvelope("unknown-site", new object())));
}
[Fact]
public void FactoryThrow_SkipsSite_DoesNotCrashActor()
{
var throwingFactory = Substitute.For<ISiteClientFactory>();
throwingFactory.Create(Arg.Any<ActorSystem>(), Arg.Any<string>(), Arg.Any<ImmutableHashSet<ActorPath>>())
.Returns(_ => throw new InvalidOperationException("boom"));
var actor = Sys.ActorOf(Props.Create(() => new CentralCommunicationActor(
EmptyProvider(), throwingFactory, null)));
EventFilter.Error(contains: "Failed to create ClusterClient").ExpectOne(() =>
actor.Tell(Load("site-a", "akka.tcp://scadabridge@node-a:8081")));
// Actor survived — a subsequent envelope for that (unrouted) site still
// produces the healthy "No ClusterClient" warning rather than a dead actor.
EventFilter.Warning(contains: "No ClusterClient for site").ExpectOne(() =>
actor.Tell(new SiteEnvelope("site-a", new object())));
}
}
@@ -62,8 +62,8 @@ public class CentralCommunicationActorReconcileTests : TestKit
services.AddScoped<ReconcileService>();
var sp = services.BuildServiceProvider();
var factory = Substitute.For<ISiteClientFactory>();
var actor = Sys.ActorOf(Props.Create(() => new CentralCommunicationActor(sp, factory, null)));
var transport = Substitute.For<ISiteCommandTransport>();
var actor = Sys.ActorOf(Props.Create(() => new CentralCommunicationActor(sp, transport, (TimeSpan?)null)));
// Node B is missing inst-B entirely → it should come back as a gap item.
actor.Tell(new ReconcileSiteRequest(
@@ -1,6 +1,4 @@
using System.Collections.Immutable;
using Akka.Actor;
using Akka.Cluster.Tools.Client;
using Akka.TestKit.Xunit2;
using Microsoft.Extensions.DependencyInjection;
using NSubstitute;
@@ -18,15 +16,19 @@ using Akka.TestKit;
namespace ZB.MOM.WW.ScadaBridge.Communication.Tests;
/// <summary>
/// Tests for CentralCommunicationActor with per-site ClusterClient routing.
/// WP-4: Message routing via ClusterClient instances created per site.
/// WP-5: Connection failure and failover handling.
/// Tests for <see cref="CentralCommunicationActor"/> behaviour that is independent of the
/// central→site command transport: heartbeat/replica aggregation, notification-outbox proxying,
/// and the DB-refresh failure path. The per-site routing itself (envelope → transport → the right
/// site) is the transport's job and is covered by
/// <see cref="CentralCommunicationActorTransportTests"/> and the <c>GrpcSiteTransport</c> suites;
/// the old ClusterClient-routing tests here were removed with the Akka transport in the
/// ClusterClient→gRPC migration's Phase 4.
/// </summary>
public class CentralCommunicationActorTests : TestKit
{
public CentralCommunicationActorTests() : base(@"akka.loglevel = DEBUG") { }
private (IActorRef actor, ISiteRepository mockRepo, Dictionary<string, TestProbe> siteProbes) CreateActorWithMockRepo(
private (IActorRef actor, ISiteRepository mockRepo) CreateActorWithMockRepo(
IEnumerable<Site>? sites = null)
{
var mockRepo = Substitute.For<ISiteRepository>();
@@ -37,93 +39,11 @@ public class CentralCommunicationActorTests : TestKit
services.AddScoped(_ => mockRepo);
var sp = services.BuildServiceProvider();
var siteProbes = new Dictionary<string, TestProbe>();
var mockFactory = Substitute.For<ISiteClientFactory>();
mockFactory.Create(Arg.Any<ActorSystem>(), Arg.Any<string>(), Arg.Any<ImmutableHashSet<ActorPath>>())
.Returns(callInfo =>
{
var siteId = callInfo.ArgAt<string>(1);
var probe = CreateTestProbe();
siteProbes[siteId] = probe;
return probe.Ref;
});
var actor = Sys.ActorOf(Props.Create(() => new CentralCommunicationActor(sp, mockFactory)));
return (actor, mockRepo, siteProbes);
var transport = Substitute.For<ISiteCommandTransport>();
var actor = Sys.ActorOf(Props.Create(() => new CentralCommunicationActor(sp, transport, (TimeSpan?)null)));
return (actor, mockRepo);
}
private (IActorRef actor, ISiteRepository mockRepo, Dictionary<string, TestProbe> siteProbes, ISiteClientFactory mockFactory) CreateActorWithFactory(
IEnumerable<Site>? sites = null)
{
var mockRepo = Substitute.For<ISiteRepository>();
mockRepo.GetAllSitesAsync(Arg.Any<CancellationToken>())
.Returns(sites?.ToList() ?? new List<Site>());
var services = new ServiceCollection();
services.AddScoped(_ => mockRepo);
var sp = services.BuildServiceProvider();
var siteProbes = new Dictionary<string, TestProbe>();
var mockFactory = Substitute.For<ISiteClientFactory>();
mockFactory.Create(Arg.Any<ActorSystem>(), Arg.Any<string>(), Arg.Any<ImmutableHashSet<ActorPath>>())
.Returns(callInfo =>
{
var siteId = callInfo.ArgAt<string>(1);
var probe = CreateTestProbe();
siteProbes[siteId] = probe;
return probe.Ref;
});
var actor = Sys.ActorOf(Props.Create(() => new CentralCommunicationActor(sp, mockFactory)));
return (actor, mockRepo, siteProbes, mockFactory);
}
private Site CreateSite(string identifier, string? nodeAAddress, string? nodeBAddress = null) =>
new("Test Site", identifier) { NodeAAddress = nodeAAddress, NodeBAddress = nodeBAddress };
[Fact]
public void ClusterClientRouting_RefreshDeploymentCommand_RoutesToSite()
{
var site = CreateSite("site1", "akka.tcp://scadabridge@host:8082");
var (actor, _, siteProbes) = CreateActorWithMockRepo(new[] { site });
Thread.Sleep(1000);
var command = new RefreshDeploymentCommand(
"dep1", "inst1", "rev1", "admin", DateTimeOffset.UtcNow,
"https://central:9000", "tok1");
actor.Tell(new SiteEnvelope("site1", command));
var msg = siteProbes["site1"].ExpectMsg<ClusterClient.Send>();
Assert.Equal("/user/site-communication", msg.Path);
Assert.IsType<RefreshDeploymentCommand>(msg.Message);
Assert.Equal("dep1", ((RefreshDeploymentCommand)msg.Message).DeploymentId);
}
[Fact]
public void UnconfiguredSite_MessageIsDropped()
{
var (actor, _, _) = CreateActorWithMockRepo();
// Wait for auto-refresh
Thread.Sleep(1000);
var command = new RefreshDeploymentCommand(
"dep1", "inst1", "hash1", "admin", DateTimeOffset.UtcNow,
"https://central:9000", "tok1");
actor.Tell(new SiteEnvelope("unknown-site", command));
ExpectNoMsg(TimeSpan.FromMilliseconds(200));
}
// Communication-016: the prior `ConnectionLost_DebugStreamsKilled` test was
// removed alongside the dead HandleConnectionStateChanged handler. No
// production code ever emitted ConnectionStateChanged, so the test was
// exercising a workflow that never ran. Disconnect detection is owned by
// the gRPC keepalive (DebugStreamBridgeActor self-terminates) and by the
// Ask-timeout path at the CommunicationService layer (deploy callers see
// a failure).
[Fact]
public void Heartbeat_BumpsAggregatorTimestamp()
{
@@ -138,9 +58,9 @@ public class CentralCommunicationActorTests : TestKit
services.AddSingleton(aggregator);
var sp = services.BuildServiceProvider();
var siteClientFactory = Substitute.For<ISiteClientFactory>();
var transport = Substitute.For<ISiteCommandTransport>();
var centralActor = Sys.ActorOf(
Props.Create(() => new CentralCommunicationActor(sp, siteClientFactory)));
Props.Create(() => new CentralCommunicationActor(sp, transport, (TimeSpan?)null)));
var timestamp = DateTimeOffset.UtcNow;
centralActor.Tell(new HeartbeatMessage("site1", "host1", true, timestamp));
@@ -165,9 +85,9 @@ public class CentralCommunicationActorTests : TestKit
services.AddSingleton(aggregator);
var sp = services.BuildServiceProvider();
var siteClientFactory = Substitute.For<ISiteClientFactory>();
var transport = Substitute.For<ISiteCommandTransport>();
var centralActor = Sys.ActorOf(
Props.Create(() => new CentralCommunicationActor(sp, siteClientFactory)));
Props.Create(() => new CentralCommunicationActor(sp, transport, (TimeSpan?)null)));
var ts = DateTimeOffset.UtcNow;
centralActor.Tell(new SiteHeartbeatReplica(new HeartbeatMessage("site-1", "host-a", true, ts)));
@@ -175,41 +95,6 @@ public class CentralCommunicationActorTests : TestKit
AwaitAssert(() => aggregator.Received(1).MarkHeartbeat("site-1", ts));
}
[Fact]
public void RefreshSiteAddresses_UpdatesCache()
{
var site1 = CreateSite("site1", "akka.tcp://scadabridge@host1:8082");
var (actor, mockRepo, siteProbes) = CreateActorWithMockRepo(new[] { site1 });
// Wait for initial load
Thread.Sleep(1000);
// Verify routing to site1 works
var cmd1 = new RefreshDeploymentCommand(
"dep1", "inst1", "hash1", "admin", DateTimeOffset.UtcNow,
"https://central:9000", "tok1");
actor.Tell(new SiteEnvelope("site1", cmd1));
var msg1 = siteProbes["site1"].ExpectMsg<ClusterClient.Send>();
Assert.Equal("dep1", ((RefreshDeploymentCommand)msg1.Message).DeploymentId);
// Update mock repo to return both sites
var site2 = CreateSite("site2", "akka.tcp://scadabridge@host2:8082");
mockRepo.GetAllSitesAsync(Arg.Any<CancellationToken>())
.Returns(new List<Site> { site1, site2 });
// Refresh again
actor.Tell(new RefreshSiteAddresses());
Thread.Sleep(1000);
// Verify routing to site2 now works
var cmd2 = new RefreshDeploymentCommand(
"dep2", "inst2", "hash2", "admin", DateTimeOffset.UtcNow,
"https://central:9000", "tok2");
actor.Tell(new SiteEnvelope("site2", cmd2));
var msg2 = siteProbes["site2"].ExpectMsg<ClusterClient.Send>();
Assert.Equal("dep2", ((RefreshDeploymentCommand)msg2.Message).DeploymentId);
}
[Fact]
public void LoadSiteAddressesFailure_IsLoggedNotSilentlySwallowed()
{
@@ -226,56 +111,26 @@ public class CentralCommunicationActorTests : TestKit
services.AddScoped(_ => mockRepo);
var sp = services.BuildServiceProvider();
var mockFactory = Substitute.For<ISiteClientFactory>();
var transport = Substitute.For<ISiteCommandTransport>();
// The fix logs a Warning carrying the InvalidOperationException as the cause.
EventFilter.Warning(contains: "Failed to load site addresses from the database").ExpectOne(() =>
{
Sys.ActorOf(Props.Create(() => new CentralCommunicationActor(sp, mockFactory)));
Sys.ActorOf(Props.Create(() => new CentralCommunicationActor(sp, transport, (TimeSpan?)null)));
});
}
[Fact]
public void MalformedSiteAddress_DoesNotAbortRefresh_OtherSitesStillRegistered()
{
// Regression test for Communication-009. HandleSiteAddressCacheLoaded calls
// ActorPath.Parse for every site in a single loop. A malformed NodeAAddress
// throws inside that loop; before the fix the whole refresh aborted partway
// through, leaving the cache half-updated (some sites registered, others not).
// The fix wraps the parse in a try/catch that logs and skips the bad site so
// a single garbage row cannot starve every other site of its ClusterClient.
var goodSite = CreateSite("good-site", "akka.tcp://scadabridge@host1:8082");
// A garbage address that ActorPath.Parse rejects.
var badSite = CreateSite("bad-site", "this is not a valid actor path !!!");
// Order the bad site first so a non-resilient loop aborts before reaching good-site.
var (actor, _, siteProbes) = CreateActorWithMockRepo(new[] { badSite, goodSite });
Thread.Sleep(1000);
// good-site must still be registered and routable despite bad-site failing to parse.
var cmd = new RefreshDeploymentCommand(
"dep1", "inst1", "hash1", "admin", DateTimeOffset.UtcNow,
"https://central:9000", "tok1");
actor.Tell(new SiteEnvelope("good-site", cmd));
Assert.True(siteProbes.ContainsKey("good-site"),
"good-site should have a ClusterClient even though bad-site's address is malformed");
var msg = siteProbes["good-site"].ExpectMsg<ClusterClient.Send>();
Assert.Equal("dep1", ((RefreshDeploymentCommand)msg.Message).DeploymentId);
}
private NotificationSubmit CreateSubmit(string id = "notif1") =>
new(id, "ops-list", "Subject", "Body", "site1", "inst1", "script.cs", DateTimeOffset.UtcNow);
[Fact]
public void NotificationSubmit_ForwardedToOutboxProxy_AckRoutesBackToSite()
{
var (actor, _, _) = CreateActorWithMockRepo();
var (actor, _) = CreateActorWithMockRepo();
var outboxProbe = CreateTestProbe();
actor.Tell(new RegisterNotificationOutbox(outboxProbe.Ref));
// A second probe stands in for the site's ClusterClient (the original Sender).
// A second probe stands in for the site (the original Sender).
var siteProbe = CreateTestProbe();
var submit = CreateSubmit();
actor.Tell(submit, siteProbe.Ref);
@@ -290,7 +145,7 @@ public class CentralCommunicationActorTests : TestKit
[Fact]
public void NotificationStatusQuery_ForwardedToOutboxProxy_ResponseRoutesBackToSite()
{
var (actor, _, _) = CreateActorWithMockRepo();
var (actor, _) = CreateActorWithMockRepo();
var outboxProbe = CreateTestProbe();
actor.Tell(new RegisterNotificationOutbox(outboxProbe.Ref));
@@ -308,7 +163,7 @@ public class CentralCommunicationActorTests : TestKit
[Fact]
public void NotificationSubmit_NoOutboxConfigured_RepliesNonAccepted()
{
var (actor, _, _) = CreateActorWithMockRepo();
var (actor, _) = CreateActorWithMockRepo();
// No RegisterNotificationOutbox sent — the proxy is null.
var submit = CreateSubmit();
@@ -323,7 +178,7 @@ public class CentralCommunicationActorTests : TestKit
[Fact]
public void NotificationStatusQuery_NoOutboxConfigured_RepliesNotFound()
{
var (actor, _, _) = CreateActorWithMockRepo();
var (actor, _) = CreateActorWithMockRepo();
// No RegisterNotificationOutbox sent — the proxy is null.
var query = new NotificationStatusQuery("corr1", "notif1");
@@ -333,23 +188,4 @@ public class CentralCommunicationActorTests : TestKit
Assert.Equal("corr1", response.CorrelationId);
Assert.False(response.Found);
}
[Fact]
public void BothContactPoints_UsedInSingleClient()
{
var site = CreateSite("site1",
"akka.tcp://scadabridge@host1:8082",
"akka.tcp://scadabridge@host2:8082");
var (actor, _, siteProbes, mockFactory) = CreateActorWithFactory(new[] { site });
// Wait for auto-refresh
Thread.Sleep(1000);
// Verify the factory was called with 2 contact paths
mockFactory.Received(1).Create(
Arg.Any<ActorSystem>(),
Arg.Is("site1"),
Arg.Is<ImmutableHashSet<ActorPath>>(paths => paths.Count == 2));
}
}
@@ -0,0 +1,136 @@
using Akka.Actor;
using Akka.TestKit.Xunit2;
using Microsoft.Extensions.DependencyInjection;
using NSubstitute;
using ZB.MOM.WW.ScadaBridge.Commons.Entities.Sites;
using ZB.MOM.WW.ScadaBridge.Commons.Interfaces.Repositories;
using ZB.MOM.WW.ScadaBridge.Commons.Messages.Deployment;
using ZB.MOM.WW.ScadaBridge.Commons.Messages.Lifecycle;
using ZB.MOM.WW.ScadaBridge.Commons.Types.Enums;
using ZB.MOM.WW.ScadaBridge.Communication.Actors;
namespace ZB.MOM.WW.ScadaBridge.Communication.Tests;
/// <summary>
/// T1B.3 seam tests: <see cref="CentralCommunicationActor"/> routing a <see cref="SiteEnvelope"/>
/// through an injected <see cref="ISiteCommandTransport"/> (transport-agnostic), proving the
/// envelope reaches the transport, the reply routes back to the waiting Ask, and the DB refresh
/// tick reconciles per-site transport resources.
/// </summary>
public class CentralCommunicationActorTransportTests : TestKit
{
public CentralCommunicationActorTransportTests() : base(@"akka.loglevel = WARNING") { }
private static Site GrpcSite(string id, string? grpcA = "http://a:8083", string? grpcB = "http://b:8083") =>
new("Test " + id, id)
{
NodeAAddress = $"akka.tcp://scadabridge@{id}-a:8081",
GrpcNodeAAddress = grpcA,
GrpcNodeBAddress = grpcB
};
private (IActorRef actor, ISiteCommandTransport transport, ISiteRepository repo) CreateActor(
IEnumerable<Site>? sites = null)
{
var repo = Substitute.For<ISiteRepository>();
repo.GetAllSitesAsync(Arg.Any<CancellationToken>()).Returns(sites?.ToList() ?? new List<Site>());
var services = new ServiceCollection();
services.AddScoped(_ => repo);
var sp = services.BuildServiceProvider();
var transport = Substitute.For<ISiteCommandTransport>();
var actor = Sys.ActorOf(Props.Create(() => new CentralCommunicationActor(sp, transport, (TimeSpan?)null)));
return (actor, transport, repo);
}
[Fact]
public void SiteEnvelope_IsRoutedToTheTransport_WithTheSenderAsReplyTo()
{
var (actor, transport, _) = CreateActor();
var probe = CreateTestProbe();
var cmd = new EnableInstanceCommand("cmd-1", "Site1.Pump1", DateTimeOffset.UtcNow);
actor.Tell(new SiteEnvelope("site-a", cmd), probe.Ref);
AwaitAssert(() => transport.Received(1).Send(
Arg.Is<SiteEnvelope>(e => e.SiteId == "site-a" && ReferenceEquals(e.Message, cmd)),
probe.Ref));
}
[Fact]
public async Task AskReply_FromTransport_RoutesBackToTheWaitingAsk()
{
var (actor, transport, _) = CreateActor();
// The transport substitute stands in for the site: when handed the envelope it delivers a
// reply to the captured replyTo, exactly as a real transport pipes the site's reply back.
var reply = new DeploymentStatusResponse(
"dep-1", "Site1.Pump1", DeploymentStatus.Success, null, DateTimeOffset.UtcNow);
transport
.When(t => t.Send(Arg.Any<SiteEnvelope>(), Arg.Any<IActorRef>()))
.Do(ci => ci.Arg<IActorRef>().Tell(reply));
var cmd = new RefreshDeploymentCommand(
"dep-1", "Site1.Pump1", "hash", "multi-role", DateTimeOffset.UtcNow, "http://c:5000", "tok");
var result = await actor.Ask<DeploymentStatusResponse>(
new SiteEnvelope("site-a", cmd), TimeSpan.FromSeconds(3));
Assert.Equal("dep-1", result.DeploymentId);
Assert.Equal(DeploymentStatus.Success, result.Status);
}
[Fact]
public void DbRefresh_ReconcilesTheTransport_WithTheLoadedGrpcEndpoints()
{
var (_, transport, _) = CreateActor(new[] { GrpcSite("site-a") });
// PreStart fires the refresh at Zero; the loaded cache is handed to the transport.
AwaitAssert(() => transport.Received().ReconcileSites(
Arg.Is<SiteAddressCacheLoaded>(c =>
c.KnownSiteIds.Contains("site-a")
&& c.GrpcContacts.ContainsKey("site-a")
&& c.GrpcContacts["site-a"].NodeA == "http://a:8083"
&& c.GrpcContacts["site-a"].NodeB == "http://b:8083")),
TimeSpan.FromSeconds(3));
}
[Fact]
public void SiteAddedAndRemoved_AcrossRefreshes_FlowsThroughToTheTransport()
{
var (actor, transport, repo) = CreateActor(new[] { GrpcSite("site-a") });
AwaitAssert(() => transport.Received().ReconcileSites(
Arg.Is<SiteAddressCacheLoaded>(c => c.GrpcContacts.ContainsKey("site-a"))),
TimeSpan.FromSeconds(3));
// site-a removed, site-b added.
repo.GetAllSitesAsync(Arg.Any<CancellationToken>()).Returns(new List<Site> { GrpcSite("site-b") });
actor.Tell(new RefreshSiteAddresses());
AwaitAssert(() => transport.Received().ReconcileSites(
Arg.Is<SiteAddressCacheLoaded>(c =>
c.GrpcContacts.ContainsKey("site-b")
&& !c.GrpcContacts.ContainsKey("site-a")
&& c.KnownSiteIds.Contains("site-b")
&& !c.KnownSiteIds.Contains("site-a"))),
TimeSpan.FromSeconds(3));
}
[Fact]
public void SiteWithoutGrpcAddresses_IsAbsentFromGrpcContacts_ButStillKnown()
{
// A site with only Akka addresses (no gRPC columns) is a known site but carries no gRPC
// endpoint — the gRPC transport must not try to dial it, the Akka one still can.
var akkaOnly = new Site("Akka only", "site-x")
{
NodeAAddress = "akka.tcp://scadabridge@site-x-a:8081"
};
var (_, transport, _) = CreateActor(new[] { akkaOnly });
AwaitAssert(() => transport.Received().ReconcileSites(
Arg.Is<SiteAddressCacheLoaded>(c =>
c.KnownSiteIds.Contains("site-x") && !c.GrpcContacts.ContainsKey("site-x"))),
TimeSpan.FromSeconds(3));
}
}
@@ -105,26 +105,28 @@ public class CommunicationOptionsValidatorTests
Assert.Contains("LiveAlarmCachePublishCoalesce", result.FailureMessage);
}
// ── T1A.3: gRPC central transport endpoints (required only when selected) ────
// ── gRPC central transport endpoints ─────────────────────────────────────────
// gRPC (CentralControlService) is the only site→central transport after the
// ClusterClient→gRPC migration's Phase 4. This role-agnostic validator only rejects BLANK
// entries; an EMPTY list is valid (central nodes legitimately declare none). The role-aware
// "a Site must list at least one endpoint" rule lives in StartupValidator, tested there.
[Fact]
public void GrpcTransport_WithNoEndpoints_IsRejected()
public void EmptyEndpoints_IsValid()
{
// A central node hosts CentralControlService; it does not dial it, so it declares none.
var result = Validate(new CommunicationOptions
{
CentralTransport = CentralTransportMode.Grpc,
CentralGrpcEndpoints = new List<string>(),
});
Assert.True(result.Failed);
Assert.Contains("CentralGrpcEndpoints", result.FailureMessage);
Assert.True(result.Succeeded, result.FailureMessage);
}
[Fact]
public void GrpcTransport_WithBlankEndpoint_IsRejected()
public void BlankEndpoint_IsRejected()
{
var result = Validate(new CommunicationOptions
{
CentralTransport = CentralTransportMode.Grpc,
CentralGrpcEndpoints = new List<string> { " " },
});
Assert.True(result.Failed);
@@ -132,11 +134,10 @@ public class CommunicationOptionsValidatorTests
}
[Fact]
public void GrpcTransport_WithEndpoints_IsValid()
public void Endpoints_IsValid()
{
var result = Validate(new CommunicationOptions
{
CentralTransport = CentralTransportMode.Grpc,
CentralGrpcEndpoints = new List<string>
{
"http://scadabridge-central-a:8083",
@@ -145,16 +146,4 @@ public class CommunicationOptionsValidatorTests
});
Assert.True(result.Succeeded, result.FailureMessage);
}
[Fact]
public void AkkaTransport_IgnoresMissingGrpcEndpoints()
{
// The default transport must not be forced to declare gRPC endpoints it never dials.
var result = Validate(new CommunicationOptions
{
CentralTransport = CentralTransportMode.Akka,
CentralGrpcEndpoints = new List<string>(),
});
Assert.True(result.Succeeded, result.FailureMessage);
}
}
@@ -12,7 +12,7 @@ namespace ZB.MOM.WW.ScadaBridge.Communication.Tests;
/// explicit <c>Resume</c> supervision strategy per the CLAUDE.md decision
/// ("Resume for coordinator actors"). A child fault under the default
/// (Restart) strategy would wipe a child's in-memory state; the long-lived
/// coordinators own per-site ClusterClients and must not silently discard
/// coordinators own per-site transport resources and must not silently discard
/// their children on a transient fault.
/// </summary>
public class CoordinatorSupervisionTests : TestKit
@@ -23,8 +23,8 @@ public class CoordinatorSupervisionTests : TestKit
/// </summary>
private sealed class CentralCommunicationActorProbe : CentralCommunicationActor
{
public CentralCommunicationActorProbe(IServiceProvider sp, ISiteClientFactory factory)
: base(sp, factory) { }
public CentralCommunicationActorProbe(IServiceProvider sp, ISiteCommandTransport transport)
: base(sp, transport) { }
public SupervisorStrategy GetSupervisorStrategy() => SupervisorStrategy();
}
@@ -54,10 +54,10 @@ public class CoordinatorSupervisionTests : TestKit
public void CentralCommunicationActor_SupervisorStrategy_IsResume()
{
var sp = EmptyServiceProvider();
var factory = Substitute.For<ISiteClientFactory>();
var transport = Substitute.For<ISiteCommandTransport>();
var actorRef = new Akka.TestKit.TestActorRef<CentralCommunicationActorProbe>(
Sys, Props.Create(() => new CentralCommunicationActorProbe(sp, factory)));
Sys, Props.Create(() => new CentralCommunicationActorProbe(sp, transport)));
var strategy = actorRef.UnderlyingActor.GetSupervisorStrategy();
@@ -1,54 +0,0 @@
using System.Collections.Immutable;
using Akka.Actor;
using Akka.Cluster.Tools.Client;
using Akka.Configuration;
using Akka.TestKit.Xunit2;
using Xunit;
using ZB.MOM.WW.ScadaBridge.Communication.Actors;
namespace ZB.MOM.WW.ScadaBridge.Communication.Tests;
public class DefaultSiteClientFactoryTests : TestKit
{
private static readonly Config TestConfig = ConfigurationFactory.ParseString(@"
akka.actor.provider = cluster
akka.remote.dot-netty.tcp.port = 0
akka.remote.dot-netty.tcp.hostname = localhost")
.WithFallback(ClusterClientReceptionist.DefaultConfig());
public DefaultSiteClientFactoryTests() : base(TestConfig) { }
private static ImmutableHashSet<ActorPath> Contacts() =>
ImmutableHashSet.Create(ActorPath.Parse("akka.tcp://other@localhost:2552/system/receptionist"));
[Fact]
public void Create_TwiceForSameSite_DoesNotCollide()
{
var factory = new DefaultSiteClientFactory();
var first = factory.Create(Sys, "site-a", Contacts());
var second = factory.Create(Sys, "site-a", Contacts()); // pre-fix: InvalidActorNameException
Assert.NotEqual(first.Path, second.Path);
}
[Theory]
[InlineData("site/with/slashes")]
[InlineData("site with spaces")]
[InlineData("näme#!")]
public void Create_WithUnsafeSiteId_SanitizesAndSucceeds(string siteId)
{
var factory = new DefaultSiteClientFactory();
var client = factory.Create(Sys, siteId, Contacts()); // pre-fix: InvalidActorNameException
Assert.NotNull(client);
}
[Theory]
[InlineData("plant-01", "plant-01")]
[InlineData("a/b c", "a_b_c")]
[InlineData("", "site")]
public void SanitizeForActorName_ProducesValidPathElement(string input, string expected)
{
Assert.Equal(expected, DefaultSiteClientFactory.SanitizeForActorName(input));
Assert.True(ActorPath.IsValidPathElement(
DefaultSiteClientFactory.SanitizeForActorName(input)));
}
}
@@ -0,0 +1,107 @@
using Microsoft.Extensions.Logging.Abstractions;
using Microsoft.Extensions.Options;
using ZB.MOM.WW.ScadaBridge.Commons.Messages.Artifacts;
using ZB.MOM.WW.ScadaBridge.Commons.Messages.DataConnection;
using ZB.MOM.WW.ScadaBridge.Commons.Messages.DebugView;
using ZB.MOM.WW.ScadaBridge.Commons.Messages.Deployment;
using ZB.MOM.WW.ScadaBridge.Commons.Messages.InboundApi;
using ZB.MOM.WW.ScadaBridge.Commons.Messages.Lifecycle;
using ZB.MOM.WW.ScadaBridge.Commons.Messages.Management;
using ZB.MOM.WW.ScadaBridge.Commons.Messages.RemoteQuery;
using ZB.MOM.WW.ScadaBridge.Commons.Types;
using ZB.MOM.WW.ScadaBridge.Communication.Grpc;
namespace ZB.MOM.WW.ScadaBridge.Communication.Tests;
/// <summary>
/// Pins <see cref="GrpcSiteTransport.ResolveDeadline"/> to the EXACT Ask timeout
/// <see cref="CommunicationService"/> uses for each command today, so flipping the transport cannot
/// change how long any call waits. Every timeout is given a distinct value so a wrong mapping is
/// caught, not masked by two defaults that happen to be equal (QueryTimeout == IntegrationTimeout in
/// production).
/// </summary>
public class GrpcSiteTransportDeadlineTests
{
private static readonly CommunicationOptions Opts = new()
{
DeploymentTimeout = TimeSpan.FromSeconds(120),
LifecycleTimeout = TimeSpan.FromSeconds(30),
ArtifactDeploymentTimeout = TimeSpan.FromSeconds(60),
QueryTimeout = TimeSpan.FromSeconds(31),
IntegrationTimeout = TimeSpan.FromSeconds(32),
DebugViewTimeout = TimeSpan.FromSeconds(10)
};
private static readonly GrpcSiteTransport Transport = new(
new SitePairChannelProvider(
new NoKeyProvider(), Options.Create(Opts), NullLogger<SitePairChannelProvider>.Instance),
Opts,
NullLogger<GrpcSiteTransport>.Instance);
private static readonly DateTimeOffset T = DateTimeOffset.UtcNow;
public static IEnumerable<object[]> Cases()
{
// Lifecycle group — note it is NOT one deadline class.
yield return [new RefreshDeploymentCommand("d", "i", "h", "by", T, "u", "t"), Opts.DeploymentTimeout];
yield return [new EnableInstanceCommand("c", "i", T), Opts.LifecycleTimeout];
yield return [new DisableInstanceCommand("c", "i", T), Opts.LifecycleTimeout];
yield return [new DeleteInstanceCommand("c", "i", T), Opts.LifecycleTimeout];
// DeploymentStateQuery uses QueryTimeout in CommunicationService, NOT LifecycleTimeout.
yield return [new DeploymentStateQueryRequest("c", "i", T), Opts.QueryTimeout];
yield return [new DeployArtifactsCommand("d", null, null, null, null, null, null, T), Opts.ArtifactDeploymentTimeout];
// OPC UA group — all QueryTimeout.
yield return [new BrowseNodeCommand("conn", null, null, null), Opts.QueryTimeout];
yield return [new SearchAddressSpaceCommand("conn", "q", 1, 1, null), Opts.QueryTimeout];
yield return [new ReadTagValuesCommand("conn", []), Opts.QueryTimeout];
yield return [new VerifyEndpointCommand("conn", "OpcUa", "{}", null), Opts.QueryTimeout];
yield return [new TrustServerCertCommand("conn", "ZGVy", "AA", null), Opts.QueryTimeout];
yield return [new ListServerCertsCommand(null), Opts.QueryTimeout];
yield return [new RemoveServerCertCommand("AA", null), Opts.QueryTimeout];
yield return [new WriteTagRequest("c", "conn", "tag", 1, T), Opts.QueryTimeout];
// Query group.
yield return [new EventLogQueryRequest("c", "s", null, null, null, null, null, null, null, 10, T), Opts.QueryTimeout];
yield return [new DebugSnapshotRequest("i", "c"), Opts.QueryTimeout];
yield return [new SubscribeDebugViewRequest("i", "c"), Opts.DebugViewTimeout];
yield return [new UnsubscribeDebugViewRequest("i", "c"), Opts.DebugViewTimeout];
// Parked group — the two relays keep QueryTimeout (30s) so SiteCallAudit's inner
// RelayTimeout (10s) still expires first.
yield return [new ParkedMessageQueryRequest("c", "s", 1, 10, T), Opts.QueryTimeout];
yield return [new ParkedMessageRetryRequest("c", "s", "m", T), Opts.QueryTimeout];
yield return [new ParkedMessageDiscardRequest("c", "s", "m", T), Opts.QueryTimeout];
yield return [new RetryParkedOperation("c", new TrackedOperationId(Guid.NewGuid())), Opts.QueryTimeout];
yield return [new DiscardParkedOperation("c", new TrackedOperationId(Guid.NewGuid())), Opts.QueryTimeout];
// Route group.
yield return [new RouteToCallRequest("c", "i", "m", null, T), Opts.IntegrationTimeout];
yield return [new RouteToGetAttributesRequest("c", "i", [], T), Opts.IntegrationTimeout];
yield return [new RouteToSetAttributesRequest("c", "i", new Dictionary<string, string>(), T), Opts.IntegrationTimeout];
// Failover — QueryTimeout in CommunicationService, NOT LifecycleTimeout.
yield return [new TriggerSiteFailover("c", "s"), Opts.QueryTimeout];
}
[Theory]
[MemberData(nameof(Cases))]
public void ResolveDeadline_MatchesTodaysAskTimeout(object command, TimeSpan expected)
{
Assert.Equal(expected, Transport.ResolveDeadline(command));
}
[Fact]
public void WaitForAttribute_UsesItsDynamicTimeoutPlusIntegrationSlack()
{
// CommunicationService.RouteToWaitForAttributeAsync uses request.Timeout + IntegrationTimeout.
var wait = new RouteToWaitForAttributeRequest("c", "i", "attr", "10", TimeSpan.FromSeconds(45), T);
Assert.Equal(TimeSpan.FromSeconds(45) + Opts.IntegrationTimeout, Transport.ResolveDeadline(wait));
}
private sealed class NoKeyProvider : ISitePskProvider
{
public ValueTask<string> GetAsync(string siteId, CancellationToken ct) => new("k");
public void Invalidate(string siteId) { }
}
}
@@ -14,9 +14,9 @@ namespace ZB.MOM.WW.ScadaBridge.Communication.Tests;
/// <summary>
/// Review 01 [Medium]: site→central health reports were fire-and-forget, so a lost
/// report was invisible to the sender. These tests pin the new end-to-end ack:
/// the site actor replies not-accepted when it has no central ClusterClient, and the
/// central actor processes + acks a report it receives.
/// report was invisible to the sender. These tests pin the end-to-end ack: the site actor
/// always replies to the sender (even when the transport cannot forward — a fail-loud failure,
/// not silence), and the central actor processes + acks a report it receives.
/// </summary>
public class HealthReportAckTests : TestKit
{
@@ -29,17 +29,17 @@ public class HealthReportAckTests : TestKit
0, 0, new Dictionary<string, int>(), 0, 0, 0, 0);
[Fact]
public void SiteCommunicationActor_NoCentralClient_RepliesNotAccepted()
public void SiteCommunicationActor_NoTransport_RepliesFailure_NotSilence()
{
// With no transport injected the actor falls back to the fail-loud
// NoOpCentralTransport, which answers a Status.Failure so the health transport's
// Ask sees a transient failure rather than hanging. (Production always injects the
// gRPC GrpcCentralTransport; this pins the wiring-guard fallback.)
var dmProbe = CreateTestProbe();
var siteComm = Sys.ActorOf(Props.Create(() =>
new SiteCommunicationActor("site-a", _options, dmProbe.Ref)));
// No RegisterCentralClient sent => _centralClient is null.
siteComm.Tell(SampleReport(seq: 7));
var ack = ExpectMsg<SiteHealthReportAck>();
Assert.False(ack.Accepted);
Assert.Equal(7, ack.SequenceNumber);
Assert.Equal("site-a", ack.SiteId);
ExpectMsg<Status.Failure>();
}
[Fact]
@@ -54,8 +54,8 @@ public class HealthReportAckTests : TestKit
services.AddSingleton(aggregator);
var sp = services.BuildServiceProvider();
var siteClientFactory = Substitute.For<ISiteClientFactory>();
var actor = Sys.ActorOf(Props.Create(() => new CentralCommunicationActor(sp, siteClientFactory)));
var transport = Substitute.For<ISiteCommandTransport>();
var actor = Sys.ActorOf(Props.Create(() => new CentralCommunicationActor(sp, transport, (TimeSpan?)null)));
actor.Tell(SampleReport(seq: 3));
var ack = ExpectMsg<SiteHealthReportAck>();
@@ -0,0 +1,153 @@
using System.Text.Json;
using ZB.MOM.WW.ScadaBridge.Communication.Grpc;
namespace ZB.MOM.WW.ScadaBridge.Communication.Tests;
/// <summary>
/// Goldens for <see cref="LooseValueCodec"/> — the type-tagged carrier for the
/// <c>object?</c> members of the site command plane (script parameters and
/// return values, attribute values, tag reads/writes).
/// </summary>
/// <remarks>
/// The contract these tests pin down is that a boxed value keeps its RUNTIME
/// CLR TYPE across the wire, not merely its printed form. Today's Akka JSON
/// serializer preserves it, and an operator-facing tag value that silently
/// turns from <c>int</c> into <c>long</c> (or from <c>DateTime</c> into a
/// UTC-normalised copy) is a behaviour change, not a refactor.
/// </remarks>
public class LooseValueCodecTests
{
public static IEnumerable<object[]> TaggedScalars() =>
[
[true],
[false],
[42],
[-1],
[long.MaxValue],
[3.5d],
[1.5f],
["a string"],
[string.Empty],
[12.3456789m],
[new DateTime(2026, 7, 22, 1, 2, 3, DateTimeKind.Utc)],
[new DateTimeOffset(2026, 7, 22, 1, 2, 3, TimeSpan.FromHours(-5))],
[Guid.Parse("11111111-2222-3333-4444-555555555555")],
[TimeSpan.FromMinutes(90)],
[new byte[] { 1, 2, 3 }]
];
[Theory]
[MemberData(nameof(TaggedScalars))]
public void TaggedValues_KeepTheirClrType(object value)
{
var restored = LooseValueCodec.FromProto(LooseValueCodec.ToProto(value));
Assert.NotNull(restored);
Assert.Equal(value.GetType(), restored.GetType());
StructuralEquality.AssertDeepEqual(value, restored, value.GetType().Name);
}
/// <summary>
/// A local <see cref="DateTime"/> must come back local, and an offset
/// <see cref="DateTimeOffset"/> must come back with the same offset — the
/// reason these ride as round-trip strings instead of proto timestamps.
/// </summary>
[Fact]
public void DateValues_KeepKindAndOffset()
{
var local = new DateTime(2026, 7, 22, 1, 2, 3, DateTimeKind.Local);
var offset = new DateTimeOffset(2026, 7, 22, 1, 2, 3, TimeSpan.FromHours(5.5));
var restoredLocal = Assert.IsType<DateTime>(LooseValueCodec.FromProto(LooseValueCodec.ToProto(local)));
var restoredOffset =
Assert.IsType<DateTimeOffset>(LooseValueCodec.FromProto(LooseValueCodec.ToProto(offset)));
Assert.Equal(DateTimeKind.Local, restoredLocal.Kind);
Assert.Equal(local, restoredLocal);
Assert.Equal(TimeSpan.FromHours(5.5), restoredOffset.Offset);
}
/// <summary>An unset carrier and an explicit null marker both decode to null.</summary>
[Fact]
public void NullIsCarriedTwoWays()
{
Assert.Null(LooseValueCodec.FromProto(null));
Assert.Null(LooseValueCodec.FromProto(LooseValueCodec.ToProto(null)));
Assert.Null(LooseValueCodec.ToProtoOrNull(null));
}
/// <summary>
/// A dictionary entry whose value is null stays distinct from an absent key —
/// the reason <c>LooseNull</c> exists at all.
/// </summary>
[Fact]
public void NullMapValue_SurvivesAsAPresentKey()
{
var original = new Dictionary<string, object?> { ["present"] = null };
var restored = LooseValueCodec.FromProtoMap(LooseValueCodec.ToProtoMap(original));
Assert.True(restored.ContainsKey("present"));
Assert.Null(restored["present"]);
}
/// <summary>A null dictionary stays null; an empty one stays empty.</summary>
[Fact]
public void NullMap_StaysDistinctFromEmptyMap()
{
Assert.Null(LooseValueCodec.ToProtoMapOrNull(null));
Assert.Null(LooseValueCodec.FromProtoMapOrNull(null));
var empty = LooseValueCodec.FromProtoMapOrNull(
LooseValueCodec.ToProtoMapOrNull(new Dictionary<string, object?>()));
Assert.NotNull(empty);
Assert.Empty(empty);
}
/// <summary>Nested lists and maps recurse, so element values keep their types too.</summary>
[Fact]
public void NestedCollections_KeepElementTypes()
{
object original = new List<object?>
{
1,
"two",
null,
new Dictionary<string, object?> { ["deep"] = 3.5d }
};
var restored = LooseValueCodec.FromProto(LooseValueCodec.ToProto(original));
StructuralEquality.AssertDeepEqual(original, restored, "list");
}
/// <summary>
/// The documented widening: a typed collection keeps its ELEMENTS' types but
/// the container comes back as <c>List&lt;object?&gt;</c>. Recorded here so
/// the behaviour is a decision rather than a surprise.
/// </summary>
[Fact]
public void TypedList_WidensToObjectList()
{
var restored = LooseValueCodec.FromProto(LooseValueCodec.ToProto(new List<int> { 1, 2, 3 }));
var list = Assert.IsType<List<object?>>(restored);
Assert.Equal([1, 2, 3], list.Cast<int>());
}
/// <summary>
/// The documented lossy escape hatch: a value outside the tagged set keeps its
/// DATA but not its CLR identity — it decodes as a <see cref="JsonElement"/>.
/// Nothing on the command plane carries such a value today; the fallback
/// exists so an unexpected one cannot fault a command.
/// </summary>
[Fact]
public void UntaggedValue_FallsBackToJsonAndLosesItsClrType()
{
var restored = LooseValueCodec.FromProto(LooseValueCodec.ToProto((byte)7));
var element = Assert.IsType<JsonElement>(restored);
Assert.Equal("7", element.GetRawText());
}
}
@@ -0,0 +1,360 @@
using Akka.Actor;
using Akka.TestKit.Xunit2;
using ZB.MOM.WW.ScadaBridge.Commons.Messages.Artifacts;
using ZB.MOM.WW.ScadaBridge.Commons.Messages.Integration;
using ZB.MOM.WW.ScadaBridge.Commons.Messages.RemoteQuery;
using ZB.MOM.WW.ScadaBridge.Commons.Types;
using ZB.MOM.WW.ScadaBridge.Communication.Actors;
using ZB.MOM.WW.ScadaBridge.Communication.Grpc;
namespace ZB.MOM.WW.ScadaBridge.Communication.Tests;
/// <summary>
/// The single routing truth for the 28 migrated central→site commands. Proves each command
/// resolves to EXACTLY the target the actor used inline before the extraction — including the
/// node-local parked handler (never the singleton proxy) and the local failover path — so the
/// Akka actor and the gRPC service can share one table without drift.
/// </summary>
public class SiteCommandDispatcherTests : TestKit
{
private const string SiteId = "site1";
private SiteCommandDispatcher Build(IActorRef dmProxy, Func<string, bool, string?>? failover = null)
=> new(SiteId, dmProxy, failover ?? ((_, _) => null));
// ── The 19 commands that forward to the Deployment Manager singleton proxy ──
/// <summary>The proxy-routed commands (lifecycle, OPC UA, debug snapshot/subscribe, route).</summary>
public static IEnumerable<object[]> ProxyCommandTypes() => new[]
{
typeof(Commons.Messages.Deployment.RefreshDeploymentCommand),
typeof(Commons.Messages.Lifecycle.EnableInstanceCommand),
typeof(Commons.Messages.Lifecycle.DisableInstanceCommand),
typeof(Commons.Messages.Lifecycle.DeleteInstanceCommand),
typeof(Commons.Messages.Deployment.DeploymentStateQueryRequest),
typeof(Commons.Messages.Management.BrowseNodeCommand),
typeof(Commons.Messages.Management.SearchAddressSpaceCommand),
typeof(Commons.Messages.Management.ReadTagValuesCommand),
typeof(Commons.Messages.Management.VerifyEndpointCommand),
typeof(Commons.Messages.Management.TrustServerCertCommand),
typeof(Commons.Messages.Management.ListServerCertsCommand),
typeof(Commons.Messages.Management.RemoveServerCertCommand),
typeof(Commons.Messages.DataConnection.WriteTagRequest),
typeof(Commons.Messages.DebugView.DebugSnapshotRequest),
typeof(Commons.Messages.DebugView.SubscribeDebugViewRequest),
typeof(Commons.Messages.InboundApi.RouteToCallRequest),
typeof(Commons.Messages.InboundApi.RouteToGetAttributesRequest),
typeof(Commons.Messages.InboundApi.RouteToSetAttributesRequest),
typeof(Commons.Messages.InboundApi.RouteToWaitForAttributeRequest),
}.Select(t => new object[] { t });
[Theory]
[MemberData(nameof(ProxyCommandTypes))]
public void ProxyCommands_ForwardToDeploymentManager(Type commandType)
{
var dm = CreateTestProbe();
var dispatcher = Build(dm.Ref);
var command = SiteCommandSamples.All[commandType][0];
var route = dispatcher.ResolveRoute(command);
Assert.Equal(SiteCommandDispatcher.RouteDisposition.Forward, route.Disposition);
Assert.Same(dm.Ref, route.Target);
}
[Fact]
public void UnsubscribeDebugView_IsFireAndForget_ToDeploymentManager_WithSyntheticAck()
{
// Fire-and-forget: the Deployment Manager never acks an unsubscribe. The actor Forwards it;
// the gRPC transport Tells it and returns this synthetic ack so a unary RPC still answers.
var dm = CreateTestProbe();
var dispatcher = Build(dm.Ref);
var command = SiteCommandSamples.All[typeof(Commons.Messages.DebugView.UnsubscribeDebugViewRequest)][0];
var route = dispatcher.ResolveRoute(command);
Assert.Equal(SiteCommandDispatcher.RouteDisposition.TellFireAndForget, route.Disposition);
Assert.Same(dm.Ref, route.Target);
Assert.Same(UnsubscribeDebugViewAck.Instance, route.Reply);
}
// ── Artifact handler (null-guarded) ──
[Fact]
public void DeployArtifacts_WithHandler_ForwardsToArtifactHandler_NotTheProxy()
{
var dm = CreateTestProbe();
var artifact = CreateTestProbe();
var dispatcher = Build(dm.Ref);
dispatcher.RegisterArtifactHandler(artifact.Ref);
var route = dispatcher.ResolveRoute(SiteCommandSamples.All[typeof(DeployArtifactsCommand)][0]);
Assert.Equal(SiteCommandDispatcher.RouteDisposition.Forward, route.Disposition);
Assert.Same(artifact.Ref, route.Target);
Assert.NotSame(dm.Ref, route.Target);
}
[Fact]
public void DeployArtifacts_WithoutHandler_RepliesHandlerNotAvailable()
{
var dm = CreateTestProbe();
var dispatcher = Build(dm.Ref);
var command = (DeployArtifactsCommand)SiteCommandSamples.All[typeof(DeployArtifactsCommand)][0];
var route = dispatcher.ResolveRoute(command);
Assert.Equal(SiteCommandDispatcher.RouteDisposition.ImmediateReply, route.Disposition);
var reply = Assert.IsType<ArtifactDeploymentResponse>(route.Reply);
Assert.False(reply.Success);
Assert.Equal("Artifact handler not available", reply.ErrorMessage);
Assert.Equal(command.DeploymentId, reply.DeploymentId);
Assert.Equal(SiteId, reply.SiteId);
}
// ── Event-log handler (null-guarded) ──
[Fact]
public void EventLogQuery_WithHandler_ForwardsToEventLogHandler()
{
var dm = CreateTestProbe();
var eventLog = CreateTestProbe();
var dispatcher = Build(dm.Ref);
dispatcher.RegisterEventLogHandler(eventLog.Ref);
var route = dispatcher.ResolveRoute(SiteCommandSamples.All[typeof(EventLogQueryRequest)][0]);
Assert.Equal(SiteCommandDispatcher.RouteDisposition.Forward, route.Disposition);
Assert.Same(eventLog.Ref, route.Target);
}
[Fact]
public void EventLogQuery_WithoutHandler_RepliesHandlerNotAvailable()
{
var dm = CreateTestProbe();
var dispatcher = Build(dm.Ref);
var command = (EventLogQueryRequest)SiteCommandSamples.All[typeof(EventLogQueryRequest)][0];
var route = dispatcher.ResolveRoute(command);
Assert.Equal(SiteCommandDispatcher.RouteDisposition.ImmediateReply, route.Disposition);
var reply = Assert.IsType<EventLogQueryResponse>(route.Reply);
Assert.False(reply.Success);
Assert.Equal(command.CorrelationId, reply.CorrelationId);
}
// ── Parked handler (null-guarded) — stays NODE-LOCAL on purpose (replicated store) ──
/// <summary>The five parked commands, each of which routes to the per-node parked handler.</summary>
public static IEnumerable<object[]> ParkedCommandTypes() => new[]
{
typeof(ParkedMessageQueryRequest),
typeof(ParkedMessageRetryRequest),
typeof(ParkedMessageDiscardRequest),
typeof(RetryParkedOperation),
typeof(DiscardParkedOperation),
}.Select(t => new object[] { t });
[Theory]
[MemberData(nameof(ParkedCommandTypes))]
public void ParkedCommands_WithHandler_RouteToNodeLocalParkedHandler_NeverTheProxy(Type commandType)
{
// Node-locality proof: a parked retry/discard must run on the node holding the replicated
// store row, so it goes to the per-node parked handler — NEVER the singleton proxy. This is
// the constraint the extraction must not "fix".
var dm = CreateTestProbe();
var parked = CreateTestProbe();
var dispatcher = Build(dm.Ref);
dispatcher.RegisterParkedMessageHandler(parked.Ref);
var route = dispatcher.ResolveRoute(SiteCommandSamples.All[commandType][0]);
Assert.Equal(SiteCommandDispatcher.RouteDisposition.Forward, route.Disposition);
Assert.Same(parked.Ref, route.Target);
Assert.NotSame(dm.Ref, route.Target);
}
[Fact]
public void ParkedMessageQuery_WithoutHandler_RepliesHandlerNotAvailable()
{
var dispatcher = Build(CreateTestProbe().Ref);
var command = (ParkedMessageQueryRequest)SiteCommandSamples.All[typeof(ParkedMessageQueryRequest)][0];
var route = dispatcher.ResolveRoute(command);
var reply = Assert.IsType<ParkedMessageQueryResponse>(route.Reply);
Assert.False(reply.Success);
Assert.Equal(command.CorrelationId, reply.CorrelationId);
Assert.Equal(command.PageNumber, reply.PageNumber);
}
[Fact]
public void ParkedMessageRetry_WithoutHandler_RepliesHandlerNotAvailable()
{
var dispatcher = Build(CreateTestProbe().Ref);
var command = (ParkedMessageRetryRequest)SiteCommandSamples.All[typeof(ParkedMessageRetryRequest)][0];
var reply = Assert.IsType<ParkedMessageRetryResponse>(dispatcher.ResolveRoute(command).Reply);
Assert.False(reply.Success);
Assert.Equal(command.CorrelationId, reply.CorrelationId);
}
[Fact]
public void ParkedMessageDiscard_WithoutHandler_RepliesHandlerNotAvailable()
{
var dispatcher = Build(CreateTestProbe().Ref);
var command = (ParkedMessageDiscardRequest)SiteCommandSamples.All[typeof(ParkedMessageDiscardRequest)][0];
var reply = Assert.IsType<ParkedMessageDiscardResponse>(dispatcher.ResolveRoute(command).Reply);
Assert.False(reply.Success);
Assert.Equal(command.CorrelationId, reply.CorrelationId);
}
[Fact]
public void RetryParkedOperation_WithoutHandler_RepliesNotAppliedAck()
{
var dispatcher = Build(CreateTestProbe().Ref);
var command = new RetryParkedOperation("corr-x", TrackedOperationId.New());
var reply = Assert.IsType<ParkedOperationActionAck>(dispatcher.ResolveRoute(command).Reply);
Assert.False(reply.Applied);
Assert.Equal("corr-x", reply.CorrelationId);
Assert.NotNull(reply.ErrorMessage);
}
[Fact]
public void DiscardParkedOperation_WithoutHandler_RepliesNotAppliedAck()
{
var dispatcher = Build(CreateTestProbe().Ref);
var command = new DiscardParkedOperation("corr-y", TrackedOperationId.New());
var reply = Assert.IsType<ParkedOperationActionAck>(dispatcher.ResolveRoute(command).Reply);
Assert.False(reply.Applied);
Assert.Equal("corr-y", reply.CorrelationId);
}
// ── Commands that must NOT enter ResolveRoute ──
[Fact]
public void ResolveRoute_RejectsFailover_ItGoesThroughPrepareFailover()
{
var dispatcher = Build(CreateTestProbe().Ref);
Assert.Throws<ArgumentException>(() =>
dispatcher.ResolveRoute(new TriggerSiteFailover("c", SiteId)));
}
[Fact]
public void ResolveRoute_RejectsTheExcludedIntegrationCommand()
{
// IntegrationCallRequest is the 29th command, dead at both ends and deliberately excluded
// (28 of 29 migrate). It never enters the dispatcher.
var dispatcher = Build(CreateTestProbe().Ref);
var command = new IntegrationCallRequest(
"c", SiteId, "inst", "es", "m", new Dictionary<string, object?>(), DateTimeOffset.UtcNow);
Assert.Throws<ArgumentException>(() => dispatcher.ResolveRoute(command));
}
// ── Failover (local path) ──
[Fact]
public void HandleFailover_ResolvesAndLeaves_ThenAcksWithTheTarget()
{
string? roleAsked = null;
var leaveIssued = false;
Func<string, bool, string?> resolve = (role, dryRun) =>
{
roleAsked = role;
leaveIssued = !dryRun;
return "akka.tcp://scadabridge@site1-a:8082";
};
var dispatcher = Build(CreateTestProbe().Ref, resolve);
var ack = dispatcher.HandleFailover(new TriggerSiteFailover("corr-1", SiteId));
Assert.True(ack.Accepted);
Assert.Equal("corr-1", ack.CorrelationId);
Assert.Equal("akka.tcp://scadabridge@site1-a:8082", ack.TargetAddress);
Assert.Null(ack.ErrorMessage);
// Site singletons are scoped to the site-specific role.
Assert.Equal("site-site1", roleAsked);
// The actor path leaves in one step (dryRun:false).
Assert.True(leaveIssued);
}
[Fact]
public void HandleFailover_RefusesWhenThereIsNoPeer()
{
var dispatcher = Build(CreateTestProbe().Ref, (_, _) => null);
var ack = dispatcher.HandleFailover(new TriggerSiteFailover("corr-2", SiteId));
Assert.False(ack.Accepted);
Assert.Null(ack.TargetAddress);
Assert.NotNull(ack.ErrorMessage);
}
[Fact]
public void HandleFailover_RefusesACommandAddressedToAnotherSite_WithoutTouchingTheResolver()
{
var invoked = false;
var dispatcher = Build(CreateTestProbe().Ref, (_, _) => { invoked = true; return "addr"; });
var ack = dispatcher.HandleFailover(new TriggerSiteFailover("corr-3", "site2"));
Assert.False(ack.Accepted);
Assert.Contains("site2", ack.ErrorMessage);
Assert.False(invoked);
}
[Fact]
public void HandleFailover_FaultInTheLeave_IsReportedNotThrown()
{
var dispatcher = Build(CreateTestProbe().Ref,
(_, _) => throw new InvalidOperationException("cluster unavailable"));
var ack = dispatcher.HandleFailover(new TriggerSiteFailover("corr-4", SiteId));
Assert.False(ack.Accepted);
Assert.Contains("cluster unavailable", ack.ErrorMessage);
}
[Fact]
public void PrepareFailover_BuildsTheAckFromADryRun_WithoutLeaving_UntilCommitLeaveRuns()
{
// The gRPC ack-before-Leave proof at the routing-truth level: PrepareFailover resolves the
// standby with a DRY-RUN only (so the ack is built without leaving), and hands back a
// deferred CommitLeave. Only invoking CommitLeave — what the gRPC service does AFTER the
// ack is on the wire — performs the real leave.
var events = new List<string>();
Func<string, bool, string?> resolve = (_, dryRun) =>
{
events.Add(dryRun ? "resolve" : "leave");
return "akka.tcp://scadabridge@site1-a:8082";
};
var dispatcher = Build(CreateTestProbe().Ref, resolve);
var outcome = dispatcher.PrepareFailover(new TriggerSiteFailover("corr-5", SiteId));
Assert.True(outcome.Ack.Accepted);
Assert.Equal("akka.tcp://scadabridge@site1-a:8082", outcome.Ack.TargetAddress);
Assert.NotNull(outcome.CommitLeave);
// Building the ack did NOT leave.
Assert.Equal(new[] { "resolve" }, events);
outcome.CommitLeave!();
// The leave runs strictly after — the caller sends the ack first.
Assert.Equal(new[] { "resolve", "leave" }, events);
}
[Fact]
public void PrepareFailover_WhenRefused_HasNoDeferredLeave()
{
var dispatcher = Build(CreateTestProbe().Ref, (_, _) => null);
var outcome = dispatcher.PrepareFailover(new TriggerSiteFailover("corr-6", SiteId));
Assert.False(outcome.Ack.Accepted);
Assert.Null(outcome.CommitLeave);
}
}
@@ -0,0 +1,569 @@
using System.Reflection;
using Google.Protobuf;
using Google.Protobuf.Reflection;
using ZB.MOM.WW.ScadaBridge.Commons.Interfaces.Protocol;
using ZB.MOM.WW.ScadaBridge.Commons.Messages.DebugView;
using ZB.MOM.WW.ScadaBridge.Commons.Messages.InboundApi;
using ZB.MOM.WW.ScadaBridge.Commons.Messages.Management;
using ZB.MOM.WW.ScadaBridge.Commons.Messages.RemoteQuery;
using ZB.MOM.WW.ScadaBridge.Commons.Messages.Streaming;
using ZB.MOM.WW.ScadaBridge.Commons.Types.Alarms;
using ZB.MOM.WW.ScadaBridge.Commons.Types.Enums;
using ZB.MOM.WW.ScadaBridge.Communication.Grpc;
namespace ZB.MOM.WW.ScadaBridge.Communication.Tests;
/// <summary>
/// Round-trip goldens for <see cref="SiteCommandDtoMapper"/> — the 28 migrated
/// central→site commands, all 22 reply shapes, and every nested type they
/// carry, each proven to survive record → proto → record unchanged.
/// </summary>
/// <remarks>
/// <para>
/// The suite is REFLECTION-DRIVEN on purpose. Both the per-type theory and the
/// coverage guards enumerate the mapper's real surface (its <c>ToProto</c>
/// overloads) and the real proto contract (the generated <c>oneof</c>
/// descriptors), so adding a command without a golden fails the build rather
/// than quietly shipping an untested field. A hand-maintained list across 50
/// types would drift on the first change.
/// </para>
/// <para>
/// Where a round trip is not bit-exact, the mapper is the thing that gets fixed
/// — never the fixture. The two deliberate normalisations
/// (empty-string-means-null, and the implicit alarm <c>Condition</c>) are
/// asserted explicitly below rather than papered over.
/// </para>
/// </remarks>
public class SiteCommandDtoMapperGoldenTests
{
private static readonly IReadOnlyDictionary<string, Type> TypesByName =
SiteCommandSamples.All.Keys.ToDictionary(t => t.FullName!, t => t);
/// <summary>One theory case per (mapped type, golden sample).</summary>
/// <returns>Type name and sample index pairs covering every golden fixture.</returns>
public static IEnumerable<object[]> AllSamples() =>
SiteCommandSamples.All
.OrderBy(kv => kv.Key.FullName, StringComparer.Ordinal)
.SelectMany(kv => Enumerable.Range(0, kv.Value.Length)
.Select(i => new object[] { kv.Key.FullName!, i }));
[Theory]
[MemberData(nameof(AllSamples))]
public void RoundTrips_Unchanged(string typeName, int sampleIndex)
{
var type = TypesByName[typeName];
var original = SiteCommandSamples.All[type][sampleIndex];
var restored = RoundTrip(original, type);
StructuralEquality.AssertDeepEqual(original, restored, type.Name);
}
// ─────────────────────────────────────────────────────────────────────
// Coverage guards
// ─────────────────────────────────────────────────────────────────────
/// <summary>
/// Every non-enum type the mapper can project must have goldens. This is the
/// guard that makes the suite self-maintaining.
/// </summary>
[Fact]
public void EveryMappedType_HasGoldenSamples()
{
var missing = MappedRecordTypes()
.Where(t => !SiteCommandSamples.All.ContainsKey(t))
.Select(t => t.Name)
.OrderBy(n => n, StringComparer.Ordinal)
.ToList();
Assert.True(
missing.Count == 0,
"SiteCommandDtoMapper projects types with no golden sample: " + string.Join(", ", missing));
}
/// <summary>Every golden sample type must have at least two samples (populated + minimal).</summary>
[Fact]
public void EveryGoldenType_HasAtLeastOneSample()
{
var empty = SiteCommandSamples.All
.Where(kv => kv.Value.Length == 0)
.Select(kv => kv.Key.Name)
.ToList();
Assert.True(empty.Count == 0, "Golden types with no samples: " + string.Join(", ", empty));
}
/// <summary>
/// The contract carries exactly 28 commands — 29 on
/// <c>SiteCommunicationActor</c>'s receive table minus the dead
/// <c>IntegrationCallRequest</c>. If the site gains a command, this count
/// moves deliberately, not silently.
/// </summary>
[Fact]
public void CommandInventory_Is28_AcrossSixGroups()
{
var commands = CommandSampleTypes().ToList();
Assert.Equal(28, commands.Count);
Assert.Equal(
[6, 8, 4, 5, 4, 1],
new[]
{
SiteCommandGroup.Lifecycle, SiteCommandGroup.OpcUa, SiteCommandGroup.Query,
SiteCommandGroup.Parked, SiteCommandGroup.Route, SiteCommandGroup.Failover
}.Select(g => commands.Count(t => SiteCommandDtoMapper.GroupOf(Sample(t)) == g)).ToArray());
}
/// <summary>The contract carries 22 reply shapes across the same six groups.</summary>
[Fact]
public void ReplyInventory_Is22_AcrossSixGroups()
{
var replies = ReplyTypes().ToList();
Assert.Equal(22, replies.Count);
Assert.Equal(
[4, 6, 3, 4, 4, 1],
new[]
{
SiteCommandGroup.Lifecycle, SiteCommandGroup.OpcUa, SiteCommandGroup.Query,
SiteCommandGroup.Parked, SiteCommandGroup.Route, SiteCommandGroup.Failover
}.Select(g => replies.Count(t => SiteCommandDtoMapper.GroupOfReply(SampleReply(t)) == g)).ToArray());
}
/// <summary>
/// Packing every command golden must exercise EVERY <c>oneof</c> case
/// declared in the four multi-command request envelopes. A new proto case
/// with no producing command fails here.
/// </summary>
[Theory]
[MemberData(nameof(RequestEnvelopes))]
public void EveryRequestOneofCase_IsProducedByACommand(string envelopeName)
{
var produced = CommandSampleTypes()
.Select(t => PackCommand(Sample(t)))
.OfType<IMessage>()
.Where(m => m.Descriptor.Name == envelopeName)
.Select(OneofCaseName)
.ToHashSet(StringComparer.Ordinal);
var declared = DescriptorFor(envelopeName).Oneofs[0].Fields
.Select(f => f.PropertyName)
.ToHashSet(StringComparer.Ordinal);
Assert.Equal(declared.OrderBy(n => n, StringComparer.Ordinal), produced.OrderBy(n => n, StringComparer.Ordinal));
}
/// <summary>The reply-envelope mirror of <see cref="EveryRequestOneofCase_IsProducedByACommand"/>.</summary>
[Theory]
[MemberData(nameof(ReplyEnvelopes))]
public void EveryReplyOneofCase_IsProducedByAReply(string envelopeName)
{
var produced = ReplyTypes()
.Select(t => PackReply(SampleReply(t)))
.OfType<IMessage>()
.Where(m => m.Descriptor.Name == envelopeName)
.Select(OneofCaseName)
.ToHashSet(StringComparer.Ordinal);
var declared = DescriptorFor(envelopeName).Oneofs[0].Fields
.Select(f => f.PropertyName)
.ToHashSet(StringComparer.Ordinal);
Assert.Equal(declared.OrderBy(n => n, StringComparer.Ordinal), produced.OrderBy(n => n, StringComparer.Ordinal));
}
/// <summary>Names of the four multi-command request envelopes.</summary>
/// <returns>Envelope message names.</returns>
public static IEnumerable<object[]> RequestEnvelopes() =>
[["LifecycleRequest"], ["OpcUaRequest"], ["QueryRequest"], ["ParkedRequest"], ["RouteRequest"]];
/// <summary>Names of the four multi-reply reply envelopes.</summary>
/// <returns>Envelope message names.</returns>
public static IEnumerable<object[]> ReplyEnvelopes() =>
[["LifecycleReply"], ["OpcUaReply"], ["QueryReply"], ["ParkedReply"], ["RouteReply"]];
/// <summary>Every command golden survives the full envelope pack/unpack, not just its own message.</summary>
[Fact]
public void EveryCommandGolden_SurvivesItsEnvelope()
{
foreach (var (type, samples) in SiteCommandSamples.All)
{
if (!IsCommand(type))
{
continue;
}
foreach (var sample in samples)
{
var restored = UnpackCommand(PackCommand(sample));
StructuralEquality.AssertDeepEqual(sample, restored, $"{type.Name}(envelope)");
}
}
}
/// <summary>Every reply golden survives the full envelope pack/unpack.</summary>
[Fact]
public void EveryReplyGolden_SurvivesItsEnvelope()
{
foreach (var (type, samples) in SiteCommandSamples.All)
{
if (!IsReply(type))
{
continue;
}
foreach (var sample in samples)
{
var restored = UnpackReply(PackReply(sample));
StructuralEquality.AssertDeepEqual(sample, restored, $"{type.Name}(envelope)");
}
}
}
/// <summary>The fire-and-forget unsubscribe ack is a real envelope case, not a special path.</summary>
[Fact]
public void UnsubscribeDebugViewAck_RoundTripsThroughQueryReply()
{
var envelope = SiteCommandDtoMapper.ToQueryReply(UnsubscribeDebugViewAck.Instance);
Assert.Equal(QueryReply.ReplyOneofCase.UnsubscribeDebugView, envelope.ReplyCase);
Assert.Same(UnsubscribeDebugViewAck.Instance, SiteCommandDtoMapper.FromQueryReply(envelope));
}
// ─────────────────────────────────────────────────────────────────────
// Enum exhaustiveness
// ─────────────────────────────────────────────────────────────────────
[Theory]
[InlineData(DeploymentStatus.Pending)]
[InlineData(DeploymentStatus.InProgress)]
[InlineData(DeploymentStatus.Success)]
[InlineData(DeploymentStatus.Failed)]
public void DeploymentStatus_RoundTrips(DeploymentStatus value) =>
Assert.Equal(value, SiteCommandDtoMapper.FromProto(SiteCommandDtoMapper.ToProto(value)));
[Theory]
[InlineData(BrowseNodeClass.Object)]
[InlineData(BrowseNodeClass.Variable)]
[InlineData(BrowseNodeClass.Method)]
[InlineData(BrowseNodeClass.Other)]
public void BrowseNodeClass_RoundTrips(BrowseNodeClass value) =>
Assert.Equal(value, SiteCommandDtoMapper.FromProto(SiteCommandDtoMapper.ToProto(value)));
[Theory]
[InlineData(BrowseFailureKind.ConnectionNotFound)]
[InlineData(BrowseFailureKind.ConnectionNotConnected)]
[InlineData(BrowseFailureKind.NotBrowsable)]
[InlineData(BrowseFailureKind.Timeout)]
[InlineData(BrowseFailureKind.ServerError)]
public void BrowseFailureKind_RoundTrips(BrowseFailureKind value) =>
Assert.Equal(value, SiteCommandDtoMapper.FromProto(SiteCommandDtoMapper.ToProto(value)));
[Theory]
[InlineData(ReadTagValuesFailureKind.ConnectionNotFound)]
[InlineData(ReadTagValuesFailureKind.ConnectionNotConnected)]
[InlineData(ReadTagValuesFailureKind.Timeout)]
[InlineData(ReadTagValuesFailureKind.ServerError)]
public void ReadTagValuesFailureKind_RoundTrips(ReadTagValuesFailureKind value) =>
Assert.Equal(value, SiteCommandDtoMapper.FromProto(SiteCommandDtoMapper.ToProto(value)));
[Theory]
[InlineData(VerifyFailureKind.Unreachable)]
[InlineData(VerifyFailureKind.AuthFailed)]
[InlineData(VerifyFailureKind.UntrustedCertificate)]
[InlineData(VerifyFailureKind.Timeout)]
[InlineData(VerifyFailureKind.ServerError)]
public void VerifyFailureKind_RoundTrips(VerifyFailureKind value) =>
Assert.Equal(value, SiteCommandDtoMapper.FromProto(SiteCommandDtoMapper.ToProto(value)));
[Theory]
[InlineData(StoreAndForwardCategory.ExternalSystem)]
[InlineData(StoreAndForwardCategory.Notification)]
[InlineData(StoreAndForwardCategory.CachedDbWrite)]
public void StoreAndForwardCategory_RoundTrips(StoreAndForwardCategory value) =>
Assert.Equal(value, SiteCommandDtoMapper.FromProto(SiteCommandDtoMapper.ToProto(value)));
[Theory]
[InlineData(AlarmState.Active)]
[InlineData(AlarmState.Normal)]
public void AlarmState_RoundTrips(AlarmState value) =>
Assert.Equal(value, SiteCommandDtoMapper.FromProto(SiteCommandDtoMapper.ToProto(value)));
[Theory]
[InlineData(AlarmLevel.None)]
[InlineData(AlarmLevel.Low)]
[InlineData(AlarmLevel.LowLow)]
[InlineData(AlarmLevel.High)]
[InlineData(AlarmLevel.HighHigh)]
public void AlarmLevel_RoundTrips(AlarmLevel value) =>
Assert.Equal(value, SiteCommandDtoMapper.FromProto(SiteCommandDtoMapper.ToProto(value)));
[Theory]
[InlineData(AlarmKind.Computed)]
[InlineData(AlarmKind.NativeOpcUa)]
[InlineData(AlarmKind.NativeMxAccess)]
public void AlarmKind_RoundTrips(AlarmKind value) =>
Assert.Equal(value, SiteCommandDtoMapper.FromProto(SiteCommandDtoMapper.ToProto(value)));
[Theory]
[InlineData(AlarmShelveState.Unshelved)]
[InlineData(AlarmShelveState.OneShotShelved)]
[InlineData(AlarmShelveState.TimedShelved)]
[InlineData(AlarmShelveState.PermanentShelved)]
public void AlarmShelveState_RoundTrips(AlarmShelveState value) =>
Assert.Equal(value, SiteCommandDtoMapper.FromProto(SiteCommandDtoMapper.ToProto(value)));
/// <summary>
/// An unspecified wire enum — what a peer on an older contract sends — must
/// decode to the documented safe default instead of faulting the command.
/// </summary>
[Fact]
public void UnspecifiedWireEnums_DecodeToSafeDefaults()
{
Assert.Equal(DeploymentStatus.Failed, SiteCommandDtoMapper.FromProto(DeploymentStatusDto.Unspecified));
Assert.Equal(BrowseNodeClass.Other, SiteCommandDtoMapper.FromProto(BrowseNodeClassDto.Unspecified));
Assert.Equal(BrowseFailureKind.ServerError, SiteCommandDtoMapper.FromProto(BrowseFailureKindDto.Unspecified));
Assert.Equal(
ReadTagValuesFailureKind.ServerError,
SiteCommandDtoMapper.FromProto(ReadTagValuesFailureKindDto.Unspecified));
Assert.Equal(VerifyFailureKind.ServerError, SiteCommandDtoMapper.FromProto(VerifyFailureKindDto.Unspecified));
Assert.Equal(
StoreAndForwardCategory.ExternalSystem,
SiteCommandDtoMapper.FromProto(StoreAndForwardCategoryDto.Unspecified));
Assert.Equal(AlarmState.Normal, SiteCommandDtoMapper.FromProto(AlarmStateDto.Unspecified));
Assert.Equal(AlarmLevel.None, SiteCommandDtoMapper.FromProto(AlarmLevelDto.Unspecified));
Assert.Equal(AlarmKind.Computed, SiteCommandDtoMapper.FromProto(AlarmKindDto.Unspecified));
Assert.Equal(AlarmShelveState.Unshelved, SiteCommandDtoMapper.FromProto(AlarmShelveStateDto.Unspecified));
}
// ─────────────────────────────────────────────────────────────────────
// The two deliberate normalisations, asserted rather than hidden
// ─────────────────────────────────────────────────────────────────────
/// <summary>
/// Nullable strings ride as plain proto3 strings, so an empty one comes back
/// as null. Documented in the mapper; asserted here so it stays a choice.
/// </summary>
[Fact]
public void EmptyNullableString_NormalisesToNull()
{
var original = new SiteFailoverAck("corr", Accepted: false, TargetAddress: "", ErrorMessage: "");
var restored = SiteCommandDtoMapper.FromProto(SiteCommandDtoMapper.ToProto(original));
Assert.Null(restored.TargetAddress);
Assert.Null(restored.ErrorMessage);
}
/// <summary>
/// …but NOT for a wait target, where the empty string is a real value. This
/// is why that one field carries a <c>StringValue</c> wrapper.
/// </summary>
[Fact]
public void EmptyWaitTarget_StaysDistinctFromNull()
{
var empty = SiteCommandDtoMapper.FromProto(SiteCommandDtoMapper.ToProto(
new RouteToWaitForAttributeRequest("c", "i", "a", "", TimeSpan.FromSeconds(1), DateTimeOffset.UtcNow)));
var missing = SiteCommandDtoMapper.FromProto(SiteCommandDtoMapper.ToProto(
new RouteToWaitForAttributeRequest("c", "i", "a", null, TimeSpan.FromSeconds(1), DateTimeOffset.UtcNow)));
Assert.Equal(string.Empty, empty.TargetValueEncoded);
Assert.Null(missing.TargetValueEncoded);
}
/// <summary>
/// A computed alarm leaves <see cref="AlarmStateChanged.Condition"/> implicit
/// (the record derives it from State + Priority). The encoder omits a
/// condition that already equals that derived value, so the record comes back
/// byte-for-byte equal — including under record equality, which compares the
/// nullable backing field, not the property.
/// </summary>
[Fact]
public void ComputedAlarm_KeepsItsImplicitCondition()
{
var original = new AlarmStateChanged("inst", "alarm", AlarmState.Active, 500, DateTimeOffset.UtcNow);
var dto = SiteCommandDtoMapper.ToProto(original);
var restored = SiteCommandDtoMapper.FromProto(dto);
Assert.Null(dto.Condition);
Assert.Equal(original, restored);
}
/// <summary>
/// The flip side of the normalisation: a condition set EXPLICITLY to the value
/// the record would have derived comes back implicit. The
/// <see cref="AlarmStateChanged.Condition"/> value is identical — only the
/// record's private "was it set?" bit differs, which no consumer can observe.
/// </summary>
[Fact]
public void ExplicitConditionEqualToTheDerivedDefault_NormalisesToImplicit()
{
var original = new AlarmStateChanged("inst", "alarm", AlarmState.Normal, 100, DateTimeOffset.UtcNow)
{
Condition = AlarmConditionStateFactory.ForComputed(AlarmState.Normal, 100)
};
var restored = SiteCommandDtoMapper.FromProto(SiteCommandDtoMapper.ToProto(original));
Assert.Equal(original.Condition, restored.Condition);
}
/// <summary>A native alarm's explicit, non-derived condition is carried verbatim.</summary>
[Fact]
public void NativeAlarm_KeepsItsExplicitCondition()
{
var condition = new AlarmConditionState(true, false, true, AlarmShelveState.PermanentShelved, true, 999);
var original = new AlarmStateChanged("inst", "alarm", AlarmState.Active, 500, DateTimeOffset.UtcNow)
{
Kind = AlarmKind.NativeMxAccess,
Condition = condition
};
var dto = SiteCommandDtoMapper.ToProto(original);
var restored = SiteCommandDtoMapper.FromProto(dto);
Assert.NotNull(dto.Condition);
Assert.Equal(condition, restored.Condition);
}
// ─────────────────────────────────────────────────────────────────────
// Group classification + rejection
// ─────────────────────────────────────────────────────────────────────
/// <summary><c>IntegrationCallRequest</c> is excluded by design and must be rejected, not silently dropped.</summary>
[Fact]
public void IntegrationCallRequest_IsRejected()
{
var dead = new Commons.Messages.Integration.IntegrationCallRequest(
"corr", "site-a", "Instance", "System", "Method",
new Dictionary<string, object?>(), DateTimeOffset.UtcNow);
Assert.Throws<ArgumentException>(() => SiteCommandDtoMapper.GroupOf(dead));
}
/// <summary>Packing a command into the wrong group's envelope is a hard error, not a silent no-op.</summary>
[Fact]
public void PackingIntoTheWrongGroup_Throws() =>
Assert.Throws<ArgumentException>(() =>
SiteCommandDtoMapper.ToLifecycleRequest(new DebugSnapshotRequest("inst", "corr")));
/// <summary>An envelope with no oneof set (a newer peer's unknown case) surfaces as a clear failure.</summary>
[Fact]
public void UnsetOneof_ThrowsNotSupported() =>
Assert.Throws<NotSupportedException>(() => SiteCommandDtoMapper.FromLifecycleRequest(new LifecycleRequest()));
// ─────────────────────────────────────────────────────────────────────
// Reflection plumbing
// ─────────────────────────────────────────────────────────────────────
private static object RoundTrip(object original, Type type)
{
var toProto = typeof(SiteCommandDtoMapper)
.GetMethods(BindingFlags.Public | BindingFlags.Static)
.Single(m => m.Name == "ToProto" && m.GetParameters() is [{ } p] && p.ParameterType == type);
var wire = toProto.Invoke(null, [original])!;
var fromProto = typeof(SiteCommandDtoMapper)
.GetMethods(BindingFlags.Public | BindingFlags.Static)
.Single(m => m.Name == "FromProto"
&& m.GetParameters() is [{ } p]
&& p.ParameterType == wire.GetType());
return fromProto.Invoke(null, [wire])!;
}
private static IEnumerable<Type> MappedRecordTypes() =>
typeof(SiteCommandDtoMapper)
.GetMethods(BindingFlags.Public | BindingFlags.Static)
.Where(m => m.Name == "ToProto" && m.GetParameters().Length == 1)
.Select(m => m.GetParameters()[0].ParameterType)
.Where(t => !t.IsEnum)
.Distinct();
private static IEnumerable<Type> CommandSampleTypes() =>
SiteCommandSamples.All.Keys.Where(IsCommand);
/// <summary>
/// Reply shapes = the mapper's own classification, plus the synthetic
/// unsubscribe ack, which has no <c>ToProto</c> overload of its own.
/// </summary>
private static IEnumerable<Type> ReplyTypes() =>
SiteCommandSamples.All.Keys.Where(IsReply).Append(typeof(UnsubscribeDebugViewAck));
private static bool IsCommand(Type type) => Classifies(type, SiteCommandDtoMapper.GroupOf);
private static bool IsReply(Type type) => Classifies(type, SiteCommandDtoMapper.GroupOfReply);
private static bool Classifies(Type type, Func<object, SiteCommandGroup> classify)
{
try
{
classify(SiteCommandSamples.All[type][0]);
return true;
}
catch (ArgumentException)
{
return false;
}
}
private static object Sample(Type type) => SiteCommandSamples.All[type][0];
private static object SampleReply(Type type) =>
type == typeof(UnsubscribeDebugViewAck) ? UnsubscribeDebugViewAck.Instance : Sample(type);
private static object PackCommand(object command) => SiteCommandDtoMapper.GroupOf(command) switch
{
SiteCommandGroup.Lifecycle => SiteCommandDtoMapper.ToLifecycleRequest(command),
SiteCommandGroup.OpcUa => SiteCommandDtoMapper.ToOpcUaRequest(command),
SiteCommandGroup.Query => SiteCommandDtoMapper.ToQueryRequest(command),
SiteCommandGroup.Parked => SiteCommandDtoMapper.ToParkedRequest(command),
SiteCommandGroup.Route => SiteCommandDtoMapper.ToRouteRequest(command),
_ => SiteCommandDtoMapper.ToProto((TriggerSiteFailover)command)
};
private static object UnpackCommand(object envelope) => envelope switch
{
LifecycleRequest r => SiteCommandDtoMapper.FromLifecycleRequest(r),
OpcUaRequest r => SiteCommandDtoMapper.FromOpcUaRequest(r),
QueryRequest r => SiteCommandDtoMapper.FromQueryRequest(r),
ParkedRequest r => SiteCommandDtoMapper.FromParkedRequest(r),
RouteRequest r => SiteCommandDtoMapper.FromRouteRequest(r),
TriggerSiteFailoverDto d => SiteCommandDtoMapper.FromProto(d),
_ => throw new InvalidOperationException($"Unknown request envelope {envelope.GetType().Name}.")
};
private static object PackReply(object reply) => SiteCommandDtoMapper.GroupOfReply(reply) switch
{
SiteCommandGroup.Lifecycle => SiteCommandDtoMapper.ToLifecycleReply(reply),
SiteCommandGroup.OpcUa => SiteCommandDtoMapper.ToOpcUaReply(reply),
SiteCommandGroup.Query => SiteCommandDtoMapper.ToQueryReply(reply),
SiteCommandGroup.Parked => SiteCommandDtoMapper.ToParkedReply(reply),
SiteCommandGroup.Route => SiteCommandDtoMapper.ToRouteReply(reply),
_ => SiteCommandDtoMapper.ToProto((SiteFailoverAck)reply)
};
private static object UnpackReply(object envelope) => envelope switch
{
LifecycleReply r => SiteCommandDtoMapper.FromLifecycleReply(r),
OpcUaReply r => SiteCommandDtoMapper.FromOpcUaReply(r),
QueryReply r => SiteCommandDtoMapper.FromQueryReply(r),
ParkedReply r => SiteCommandDtoMapper.FromParkedReply(r),
RouteReply r => SiteCommandDtoMapper.FromRouteReply(r),
SiteFailoverAckDto d => SiteCommandDtoMapper.FromProto(d),
_ => throw new InvalidOperationException($"Unknown reply envelope {envelope.GetType().Name}.")
};
private static MessageDescriptor DescriptorFor(string name) =>
SiteCommandReflection.Descriptor.MessageTypes.Single(m => m.Name == name);
private static string OneofCaseName(IMessage envelope)
{
var oneof = envelope.Descriptor.Oneofs[0];
var field = oneof.Accessor.GetCaseFieldDescriptor(envelope);
Assert.NotNull(field);
return field.PropertyName;
}
}
@@ -0,0 +1,304 @@
using ZB.MOM.WW.ScadaBridge.Commons.Interfaces.Protocol;
using ZB.MOM.WW.ScadaBridge.Commons.Messages.Artifacts;
using ZB.MOM.WW.ScadaBridge.Commons.Messages.DataConnection;
using ZB.MOM.WW.ScadaBridge.Commons.Messages.DebugView;
using ZB.MOM.WW.ScadaBridge.Commons.Messages.Deployment;
using ZB.MOM.WW.ScadaBridge.Commons.Messages.InboundApi;
using ZB.MOM.WW.ScadaBridge.Commons.Messages.Lifecycle;
using ZB.MOM.WW.ScadaBridge.Commons.Messages.Management;
using ZB.MOM.WW.ScadaBridge.Commons.Messages.RemoteQuery;
using ZB.MOM.WW.ScadaBridge.Commons.Messages.Streaming;
using ZB.MOM.WW.ScadaBridge.Commons.Types;
using ZB.MOM.WW.ScadaBridge.Commons.Types.Alarms;
using ZB.MOM.WW.ScadaBridge.Commons.Types.Enums;
namespace ZB.MOM.WW.ScadaBridge.Communication.Tests;
/// <summary>
/// Golden fixtures for the site-command round-trip suite: for every command,
/// reply and nested type the mapper handles, at least one MAXIMALLY populated
/// sample and one MINIMAL sample (every nullable null, every optional
/// collection empty or absent).
/// </summary>
/// <remarks>
/// <para>
/// The two-sample rule is the point. A single "typical" sample proves only that
/// the happy path survives; the pairs are what catch a dropped optional field or
/// a null/empty collapse — the class of silent data loss the transport
/// round-trip guard exposed in PLAN-05 T8, which per-field unit tests had all
/// missed.
/// </para>
/// <para>
/// <see cref="SiteCommandDtoMapperGoldenTests"/> drives this table by
/// reflection and FAILS when the mapper gains a type that has no sample here, so
/// the coverage cannot silently drift as commands are added.
/// </para>
/// </remarks>
internal static class SiteCommandSamples
{
private static readonly DateTimeOffset T1 = new(2026, 7, 22, 13, 45, 12, 345, TimeSpan.Zero);
private static readonly DateTimeOffset T2 = new(2026, 1, 2, 3, 4, 5, TimeSpan.Zero);
private static readonly DateTime D1 = new(2026, 3, 4, 5, 6, 7, DateTimeKind.Utc);
private static readonly DateTime D2 = new(2027, 3, 4, 5, 6, 7, DateTimeKind.Utc);
private static readonly Guid G1 = Guid.Parse("11111111-2222-3333-4444-555555555555");
/// <summary>
/// Every CLR type the mapper round-trips, mapped to its golden samples.
/// Enums are excluded — they get their own exhaustive test.
/// </summary>
public static IReadOnlyDictionary<Type, object[]> All { get; } = Build();
private static Dictionary<Type, object[]> Build()
{
var samples = new Dictionary<Type, object[]>();
void Add<T>(params T[] values) where T : notnull =>
samples[typeof(T)] = values.Cast<object>().ToArray();
// ── Lifecycle commands ──
Add(new RefreshDeploymentCommand(
"dep-1", "Site1.Pump1", "hash-abc", "multi-role", T1, "http://central:5000", "tok-1"));
Add(new EnableInstanceCommand("cmd-1", "Site1.Pump1", T1));
Add(new DisableInstanceCommand("cmd-2", "Site1.Pump1", T2));
Add(new DeleteInstanceCommand("cmd-3", "Site1.Pump1", T1));
Add(new DeploymentStateQueryRequest("corr-1", "Site1.Pump1", T1));
Add(
// Full: every artifact collection populated.
new DeployArtifactsCommand(
"dep-2",
[SharedScript(), new SharedScriptArtifact("s2", "return 2;", null, null)],
[ExternalSystem()],
[new DatabaseConnectionArtifact("db", "Server=x;", 3, TimeSpan.FromSeconds(7.5))],
[new NotificationListArtifact("ops", ["a@x.com", "b@x.com"])],
[DataConnection()],
[Smtp()],
T1),
// Minimal: every collection NULL — must not come back as empty lists.
new DeployArtifactsCommand("dep-3", null, null, null, null, null, null, T2),
// Boundary: every collection present but EMPTY — must not come back null.
new DeployArtifactsCommand("dep-4", [], [], [], [], [], [], T1));
Add(SharedScript(), new SharedScriptArtifact("bare", "", null, null));
Add(ExternalSystem(), new ExternalSystemArtifact("bare", "http://x", "None", null, null, 0));
Add(new DatabaseConnectionArtifact("db", "Server=x;", 3, TimeSpan.FromMilliseconds(1500)),
new DatabaseConnectionArtifact("db2", "", 0, TimeSpan.Zero));
Add(new NotificationListArtifact("ops", ["a@x.com"]), new NotificationListArtifact("empty", []));
Add(DataConnection(), new DataConnectionArtifact("bare", "OpcUa", null, null, 0));
Add(Smtp(), new SmtpConfigurationArtifact("bare", "smtp", 25, "None", "f@x.com", null, null, null));
// ── Lifecycle replies ──
Add(new DeploymentStatusResponse("dep-1", "Site1.Pump1", DeploymentStatus.Success, null, T1),
new DeploymentStatusResponse("dep-1", "Site1.Pump1", DeploymentStatus.Failed, "boom", T2));
Add(new InstanceLifecycleResponse("cmd-1", "Site1.Pump1", true, null, T1),
new InstanceLifecycleResponse("cmd-1", "Site1.Pump1", false, "nope", T2));
Add(new DeploymentStateQueryResponse("corr-1", "Site1.Pump1", true, "dep-1", "hash-abc", T1),
new DeploymentStateQueryResponse("corr-1", "Site1.Pump1", false, null, null, T2));
Add(new ArtifactDeploymentResponse("dep-2", "site-a", true, null, T1),
new ArtifactDeploymentResponse("dep-2", "site-a", false, "handler missing", T2));
// ── OPC UA commands ──
Add(new BrowseNodeCommand("conn", "ns=2;s=Root", "cursor-1", "site-a"),
new BrowseNodeCommand("conn", null, null, null));
Add(new SearchAddressSpaceCommand("conn", "pump", 4, 200, "site-a"),
new SearchAddressSpaceCommand("conn", "", 0, 0, null));
Add(new ReadTagValuesCommand("conn", ["a", "b"]), new ReadTagValuesCommand("conn", []));
Add(new VerifyEndpointCommand("conn", "OpcUa", "{\"url\":\"opc.tcp://x\"}", "site-a"),
new VerifyEndpointCommand("conn", "OpcUa", "{}", null));
Add(new TrustServerCertCommand("conn", "ZGVy", "AABB", "site-a"),
new TrustServerCertCommand("conn", "ZGVy", "AABB", null));
Add(new ListServerCertsCommand("site-a"), new ListServerCertsCommand(null));
Add(new RemoveServerCertCommand("AABB", "site-a"), new RemoveServerCertCommand("AABB", null));
Add(new WriteTagRequest("corr-2", "conn", "ns=2;s=Speed", 42.5d, T1),
new WriteTagRequest("corr-2", "conn", "ns=2;s=Speed", null, T2),
new WriteTagRequest("corr-2", "conn", "ns=2;s=Flag", true, T1),
new WriteTagRequest("corr-2", "conn", "ns=2;s=Name", "manual", T1));
// ── OPC UA replies ──
Add(new BrowseNodeResult([Node(), NodeBare()], true, null, "cursor-2"),
new BrowseNodeResult([], false, new BrowseFailure(BrowseFailureKind.Timeout, "timed out"), null));
Add(Node(), NodeBare());
Add(new BrowseFailure(BrowseFailureKind.ConnectionNotConnected, "not connected"),
new BrowseFailure(BrowseFailureKind.ServerError, ""));
Add(new SearchAddressSpaceResult([new AddressSpaceMatch(Node(), "/Root/Pump1")], true, null),
new SearchAddressSpaceResult([], false, new BrowseFailure(BrowseFailureKind.NotBrowsable, "no")));
Add(new AddressSpaceMatch(Node(), "/Root/Pump1"), new AddressSpaceMatch(NodeBare(), ""));
Add(new ReadTagValuesResult([Outcome(), OutcomeFailed()], null),
new ReadTagValuesResult([], new ReadTagValuesFailure(ReadTagValuesFailureKind.ConnectionNotFound, "gone")));
Add(Outcome(), OutcomeFailed());
Add(new ReadTagValuesFailure(ReadTagValuesFailureKind.Timeout, "slow"),
new ReadTagValuesFailure(ReadTagValuesFailureKind.ServerError, ""));
Add(new VerifyEndpointResult(true, null, null, null),
new VerifyEndpointResult(false, VerifyFailureKind.UntrustedCertificate, "untrusted", Cert()),
new VerifyEndpointResult(false, VerifyFailureKind.Unreachable, "refused", null));
Add(Cert());
Add(new CertTrustResult(true, null, [Trusted(), TrustedRejected()]),
new CertTrustResult(true, null, null),
new CertTrustResult(false, "partial failure", []));
Add(Trusted(), TrustedRejected());
Add(new WriteTagResponse("corr-2", true, null, T1),
new WriteTagResponse("corr-2", false, "denied", T2));
// ── Query commands ──
Add(new EventLogQueryRequest(
"corr-3", "site-a", T1, T2, "Lifecycle", "Warning", "Site1.Pump1", "restart", "cursor-3", 50, T1),
new EventLogQueryRequest("corr-3", "site-a", null, null, null, null, null, null, null, 25, T2));
Add(new DebugSnapshotRequest("Site1.Pump1", "corr-4"));
Add(new SubscribeDebugViewRequest("Site1.Pump1", "corr-5"));
Add(new UnsubscribeDebugViewRequest("Site1.Pump1", "corr-6"));
// ── Query replies ──
Add(new EventLogQueryResponse("corr-3", "site-a", [Entry(), EntryBare()], "cursor-4", true, true, null, T1),
new EventLogQueryResponse("corr-3", "site-a", [], null, false, false, "handler missing", T2));
Add(Entry(), EntryBare());
Add(new DebugViewSnapshot("Site1.Pump1", [AttrValue(), AttrValueNull()], [ComputedAlarm(), NativeAlarm()], T1),
new DebugViewSnapshot("Site1.Pump1", [], [], T2, InstanceNotFound: true));
Add(AttrValue(), AttrValueNull(), AttrValueList());
Add(ComputedAlarm(), NativeAlarm());
Add(new AlarmConditionState(true, false, true, AlarmShelveState.TimedShelved, true, 900),
new AlarmConditionState(false, true, null, AlarmShelveState.Unshelved, false, 0));
// ── Parked commands ──
Add(new ParkedMessageQueryRequest("corr-7", "site-a", 2, 25, T1));
Add(new ParkedMessageRetryRequest("corr-8", "site-a", "msg-1", T1));
Add(new ParkedMessageDiscardRequest("corr-9", "site-a", "msg-1", T2));
Add(new RetryParkedOperation("corr-10", new TrackedOperationId(G1)),
new RetryParkedOperation("corr-10", default));
Add(new DiscardParkedOperation("corr-11", new TrackedOperationId(G1)),
new DiscardParkedOperation("corr-11", default));
// ── Parked replies ──
Add(new ParkedMessageQueryResponse("corr-7", "site-a", [Parked(), ParkedBare()], 2, 1, 25, true, null, T1),
new ParkedMessageQueryResponse("corr-7", "site-a", [], 0, 1, 25, false, "handler missing", T2));
Add(Parked(), ParkedBare());
Add(new ParkedMessageRetryResponse("corr-8", true),
new ParkedMessageRetryResponse("corr-8", false, "not parked"));
Add(new ParkedMessageDiscardResponse("corr-9", true),
new ParkedMessageDiscardResponse("corr-9", false, "not parked"));
Add(new ParkedOperationActionAck("corr-10", true),
new ParkedOperationActionAck("corr-10", false, "handler missing"));
// ── Route commands ──
Add(new RouteToCallRequest("corr-12", "Site1.Pump1", "Start", Parameters(), T1, G1),
new RouteToCallRequest("corr-12", "Site1.Pump1", "Start", null, T2),
new RouteToCallRequest("corr-12", "Site1.Pump1", "Start", new Dictionary<string, object?>(), T1));
Add(new RouteToGetAttributesRequest("corr-13", "Site1.Pump1", ["Speed", "Flow"], T1, G1),
new RouteToGetAttributesRequest("corr-13", "Site1.Pump1", [], T2));
Add(new RouteToSetAttributesRequest(
"corr-14", "Site1.Pump1", new Dictionary<string, string> { ["Speed"] = "10" }, T1, G1),
new RouteToSetAttributesRequest("corr-14", "Site1.Pump1", new Dictionary<string, string>(), T2));
Add(new RouteToWaitForAttributeRequest(
"corr-15", "Site1.Pump1", "Speed", "10", TimeSpan.FromSeconds(30), T1, G1, true),
new RouteToWaitForAttributeRequest(
"corr-15", "Site1.Pump1", "Speed", null, TimeSpan.Zero, T2),
// "" is a legitimate wait target and must NOT collapse to null.
new RouteToWaitForAttributeRequest(
"corr-15", "Site1.Pump1", "Speed", "", TimeSpan.FromMinutes(1), T1));
// ── Route replies ──
Add(new RouteToCallResponse("corr-12", true, 17L, null, T1),
new RouteToCallResponse("corr-12", false, null, "script faulted", T2));
Add(new RouteToGetAttributesResponse("corr-13", Parameters(), true, null, T1),
new RouteToGetAttributesResponse("corr-13", new Dictionary<string, object?>(), false, "no instance", T2));
Add(new RouteToSetAttributesResponse("corr-14", true, null, T1),
new RouteToSetAttributesResponse("corr-14", false, "locked", T2));
Add(new RouteToWaitForAttributeResponse("corr-15", true, 10, "Good", false, true, null, T1),
new RouteToWaitForAttributeResponse("corr-15", false, null, null, true, true, null, T2));
// ── Failover ──
Add(new TriggerSiteFailover("corr-16", "site-a"));
Add(new SiteFailoverAck("corr-16", true, "akka.tcp://scadabridge@node-b:8081", null),
new SiteFailoverAck("corr-16", false, null, "no standby available"));
return samples;
}
private static SharedScriptArtifact SharedScript() =>
new("Calc", "return 1;", "{\"p\":\"int\"}", "{\"r\":\"int\"}");
private static ExternalSystemArtifact ExternalSystem() =>
new("Mes", "https://mes/api", "ApiKey", "{\"key\":\"x\"}", "[{\"name\":\"Post\"}]", 45);
private static DataConnectionArtifact DataConnection() =>
new("Plc1", "OpcUa", "{\"url\":\"opc.tcp://a\"}", "{\"url\":\"opc.tcp://b\"}", 5);
private static SmtpConfigurationArtifact Smtp() =>
new("Default", "smtp.host", 587, "Basic", "from@x.com", "user", "secret", "{\"tenant\":\"t\"}");
private static BrowseNode Node() => new("ns=2;s=Pump1.Speed", "Speed", BrowseNodeClass.Variable, false, "Double", -1, true);
private static BrowseNode NodeBare() => new("ns=2;s=Root", "Root", BrowseNodeClass.Object, true);
private static TagReadOutcome Outcome() => new("ns=2;s=Speed", true, 12.5d, "Good", T1, null);
private static TagReadOutcome OutcomeFailed() => new("ns=2;s=Bad", false, null, "Bad", T2, "no such node");
private static ServerCertInfo Cert() => new("AABB", "CN=server", "CN=issuer", D1, D2, "ZGVy");
private static TrustedCertInfo Trusted() => new("AABB", "CN=server", "CN=issuer", D1, D2, false);
private static TrustedCertInfo TrustedRejected() => new("CCDD", "CN=other", "CN=issuer", D1, D2, true);
private static EventLogEntry Entry() =>
new(G1.ToString("D"), T1, "Lifecycle", "Warning", "Site1.Pump1", "InstanceActor", "restarted", "{\"n\":1}");
private static EventLogEntry EntryBare() =>
new(Guid.Empty.ToString("D"), T2, "System", "Info", null, "Host", "started", null);
private static ParkedMessageEntry Parked() =>
new("msg-1", "Mes", "Post", "500 from server", 7, T1, T2, 10, StoreAndForwardCategory.CachedDbWrite, "Site1.Pump1");
private static ParkedMessageEntry ParkedBare() =>
new("msg-2", "Mes", "Post", "", 0, T2, T2);
private static AttributeValueChanged AttrValue() =>
new("Site1.Pump1", "Pump1.Speed", "Speed", 12.5d, "Good", T1);
private static AttributeValueChanged AttrValueNull() =>
new("Site1.Pump1", "Pump1.Speed", "Speed", null, "Bad", T2);
private static AttributeValueChanged AttrValueList() =>
new("Site1.Pump1", "Pump1.Trend", "Trend", new List<object?> { 1, 2.5d, "x", null }, "Good", T1);
/// <summary>A computed alarm: <c>Condition</c> left implicit, which is the common case.</summary>
private static AlarmStateChanged ComputedAlarm() =>
new("Site1.Pump1", "HighSpeed", AlarmState.Active, 700, T1)
{
Level = AlarmLevel.HighHigh,
Message = "Speed critically high"
};
/// <summary>A mirrored native alarm: every native enrichment field populated, <c>Condition</c> explicit.</summary>
private static AlarmStateChanged NativeAlarm() =>
new("Site1.Pump1", "Tank01.Level.HiHi", AlarmState.Active, 850, T2)
{
Level = AlarmLevel.None,
Message = "Level high",
Kind = AlarmKind.NativeOpcUa,
Condition = new AlarmConditionState(true, false, false, AlarmShelveState.OneShotShelved, true, 850),
SourceReference = "Tank01.Level.HiHi",
AlarmTypeName = "AnalogLimitAlarm.HiHi",
Category = "Process",
OperatorUser = "multi-role",
OperatorComment = "ack'd at panel",
OriginalRaiseTime = T1,
CurrentValue = "91.2",
LimitValue = "90.0",
NativeSourceCanonicalName = "Tank01.TankAlarms",
IsConfiguredPlaceholder = true
};
private static Dictionary<string, object?> Parameters() => new()
{
["count"] = 3,
["ratio"] = 1.25d,
["name"] = "batch-1",
["enabled"] = true,
["missing"] = null,
["when"] = T1,
["id"] = G1,
["items"] = new List<object?> { 1L, "two", null },
["nested"] = new Dictionary<string, object?> { ["inner"] = 5f }
};
}
@@ -1,6 +1,6 @@
using Akka.Actor;
using Akka.Cluster.Tools.Client;
using Akka.TestKit.Xunit2;
using NSubstitute;
using ZB.MOM.WW.ScadaBridge.Commons.Messages.DataConnection;
using ZB.MOM.WW.ScadaBridge.Commons.Messages.Deployment;
using ZB.MOM.WW.ScadaBridge.Commons.Messages.Health;
@@ -109,100 +109,11 @@ public class SiteCommunicationActorTests : TestKit
handlerProbe.ExpectMsg<IntegrationCallRequest>(msg => msg.CorrelationId == "corr1");
}
[Fact]
public void NotificationSubmit_WithCentralClient_ForwardedToCentralAndAckRoutedBack()
{
// The site forwards a buffered notification to central over the ClusterClient
// command/control transport; the central ack must route back to the original
// sender (the S&F forwarder's Ask), not to the SiteCommunicationActor.
var dmProbe = CreateTestProbe();
var centralClientProbe = CreateTestProbe();
var siteActor = Sys.ActorOf(Props.Create(() =>
new SiteCommunicationActor("site1", _options, dmProbe.Ref)));
siteActor.Tell(new RegisterCentralClient(centralClientProbe.Ref));
var submit = new NotificationSubmit(
"notif-1", "Operators", "Subj", "Body", "site1", "inst1", "alarmScript",
DateTimeOffset.UtcNow);
siteActor.Tell(submit);
// Central client (acting as ClusterClient) receives a ClusterClient.Send wrapping
// the NotificationSubmit, addressed to the central communication actor. Fish past
// any periodic HeartbeatMessage the actor's timer may interleave.
var send = centralClientProbe.FishForMessage<ClusterClient.Send>(
s => s.Message is NotificationSubmit);
Assert.Equal("/user/central-communication", send.Path);
var forwarded = Assert.IsType<NotificationSubmit>(send.Message);
Assert.Equal("notif-1", forwarded.NotificationId);
// The ack is sent to the ClusterClient.Send's Sender — replying as that probe
// must land back at the test actor (the original Tell sender).
centralClientProbe.Reply(new NotificationSubmitAck("notif-1", Accepted: true, Error: null));
ExpectMsg<NotificationSubmitAck>(ack => ack.NotificationId == "notif-1" && ack.Accepted);
}
[Fact]
public void NotificationSubmit_WithoutCentralClient_RepliesWithNonAccepted()
{
// No ClusterClient registered yet: the submit cannot be forwarded, so the actor
// replies with a non-accepted ack and the S&F forwarder treats it as transient.
var dmProbe = CreateTestProbe();
var siteActor = Sys.ActorOf(Props.Create(() =>
new SiteCommunicationActor("site1", _options, dmProbe.Ref)));
var submit = new NotificationSubmit(
"notif-2", "Operators", "Subj", "Body", "site1", null, null,
DateTimeOffset.UtcNow);
siteActor.Tell(submit);
ExpectMsg<NotificationSubmitAck>(ack => ack.NotificationId == "notif-2" && !ack.Accepted);
}
[Fact]
public void NotificationStatusQuery_WithCentralClient_ForwardedToCentralAndResponseRoutedBack()
{
// Notify.Status(id) issues a NotificationStatusQuery; the site actor forwards it
// to central over the ClusterClient command/control transport and the central
// response must route back to the original sender (the helper's Ask).
var dmProbe = CreateTestProbe();
var centralClientProbe = CreateTestProbe();
var siteActor = Sys.ActorOf(Props.Create(() =>
new SiteCommunicationActor("site1", _options, dmProbe.Ref)));
siteActor.Tell(new RegisterCentralClient(centralClientProbe.Ref));
var query = new NotificationStatusQuery("corr-99", "notif-1");
siteActor.Tell(query);
var send = centralClientProbe.FishForMessage<ClusterClient.Send>(
s => s.Message is NotificationStatusQuery);
Assert.Equal("/user/central-communication", send.Path);
var forwarded = Assert.IsType<NotificationStatusQuery>(send.Message);
Assert.Equal("notif-1", forwarded.NotificationId);
// The response is sent to the ClusterClient.Send's Sender — replying as that
// probe must land back at the test actor (the original Tell sender).
centralClientProbe.Reply(new NotificationStatusResponse(
"corr-99", Found: true, Status: "Delivered", RetryCount: 0,
LastError: null, DeliveredAt: DateTimeOffset.UtcNow));
ExpectMsg<NotificationStatusResponse>(r => r.CorrelationId == "corr-99" && r.Found);
}
[Fact]
public void NotificationStatusQuery_WithoutCentralClient_RepliesWithNotFound()
{
// No ClusterClient registered yet: the query cannot reach central, so the actor
// replies Found: false. Notify.Status then falls back to the site S&F buffer.
var dmProbe = CreateTestProbe();
var siteActor = Sys.ActorOf(Props.Create(() =>
new SiteCommunicationActor("site1", _options, dmProbe.Ref)));
siteActor.Tell(new NotificationStatusQuery("corr-100", "notif-2"));
ExpectMsg<NotificationStatusResponse>(
r => r.CorrelationId == "corr-100" && !r.Found);
}
// The site→central send-forwarding tests that used to live here (NotificationSubmit /
// NotificationStatusQuery, with and without a registered central ClusterClient) were removed
// with the Akka transport in the ClusterClient→gRPC migration's Phase 4: the actor no longer
// owns the wire, it delegates to an injected ICentralTransport, and those seven sends (plus the
// reply routing and the fail-loud fallback) are now covered by SiteCommunicationActorTransportTests.
[Fact]
public void EventLogQuery_WithoutHandler_ReturnsFailure()
@@ -355,22 +266,22 @@ public class SiteCommunicationActorTests : TestKit
// leader check in production); tests inject a stub so they do not need
// to bring up a full cluster in the TestKit ActorSystem.
var dmProbe = CreateTestProbe();
var centralClientProbe = CreateTestProbe();
var transport = Substitute.For<ICentralTransport>();
var fastHeartbeatOptions = new CommunicationOptions
{
TransportHeartbeatInterval = TimeSpan.FromMilliseconds(50)
ApplicationHeartbeatInterval = TimeSpan.FromMilliseconds(100)
};
var siteActor = Sys.ActorOf(Props.Create(() =>
new SiteCommunicationActor("site1", fastHeartbeatOptions, dmProbe.Ref, () => isActive)));
Sys.ActorOf(Props.Create(() =>
new SiteCommunicationActor("site1", fastHeartbeatOptions, dmProbe.Ref, () => isActive, null, transport)));
siteActor.Tell(new RegisterCentralClient(centralClientProbe.Ref));
var send = centralClientProbe.FishForMessage<ClusterClient.Send>(
s => s.Message is HeartbeatMessage, TimeSpan.FromSeconds(3));
var heartbeat = Assert.IsType<HeartbeatMessage>(send.Message);
Assert.Equal(isActive, heartbeat.IsActive);
Assert.Equal("site1", heartbeat.SiteId);
// The heartbeat now rides the injected ICentralTransport (SendHeartbeat), not a
// ClusterClient. The captured message must carry the injected active/standby flag.
AwaitAssert(
() => transport.Received().SendHeartbeat(
Arg.Is<HeartbeatMessage>(h => h.IsActive == isActive && h.SiteId == "site1"),
Arg.Any<IActorRef>()),
TimeSpan.FromSeconds(3));
}
// ---- Central→site failover relay (Task 10) ----------------------------------
@@ -0,0 +1,157 @@
using System.Collections;
using System.Reflection;
using System.Text.Json;
namespace ZB.MOM.WW.ScadaBridge.Communication.Tests;
/// <summary>
/// Structural deep-equality assertion for the site-command round-trip goldens.
/// </summary>
/// <remarks>
/// <para>
/// Record equality is NOT usable here. A C# record's generated <c>Equals</c>
/// compares members with <c>EqualityComparer&lt;T&gt;.Default</c>, which for
/// <c>IReadOnlyList&lt;T&gt;</c> / <c>IReadOnlyDictionary&lt;,&gt;</c> members
/// degrades to reference equality — so <c>Assert.Equal(dto, roundTripped)</c>
/// would fail on every collection-bearing message even when the mapper is
/// perfect, and (worse) would pass vacuously nowhere useful. This walker
/// compares by shape instead: dictionaries by key, sequences element-wise, and
/// everything else property-by-property down to leaf values.
/// </para>
/// <para>
/// The failure message carries the full property path, so a dropped field
/// reports as e.g. <c>DeployArtifactsCommand.ExternalSystems[0].TimeoutSeconds</c>
/// rather than an opaque "objects differ".
/// </para>
/// </remarks>
internal static class StructuralEquality
{
/// <summary>Asserts two object graphs are structurally identical.</summary>
/// <param name="expected">The original value.</param>
/// <param name="actual">The value that came back through the mapper.</param>
/// <param name="rootName">Name used as the root of the reported property path.</param>
public static void AssertDeepEqual(object? expected, object? actual, string rootName)
=> Compare(expected, actual, rootName);
private static void Compare(object? expected, object? actual, string path)
{
if (expected is null && actual is null)
{
return;
}
if (expected is null || actual is null)
{
Assert.Fail($"{path}: expected {Describe(expected)} but got {Describe(actual)}.");
return;
}
// JsonElement is the documented lossy escape hatch of LooseValueCodec; it
// has no useful Equals, so compare the canonical JSON text.
if (expected is JsonElement expectedJson && actual is JsonElement actualJson)
{
Assert.Equal(expectedJson.GetRawText(), actualJson.GetRawText());
return;
}
// Collections are compared by CONTENT, not by concrete container type.
// Members are declared IReadOnlyList<T>/IReadOnlyDictionary<,>, so the
// backing type (a collection-expression array here, a List<T> out of the
// mapper) is not part of the contract. Element and value types below are
// still compared strictly.
if (expected is not string && actual is not string)
{
if (expected is IDictionary expectedMap && actual is IDictionary actualMap)
{
CompareDictionaries(expectedMap, actualMap, path);
return;
}
if (expected is IEnumerable expectedSeq && actual is IEnumerable actualSeq)
{
CompareSequences(expectedSeq, actualSeq, path);
return;
}
}
var expectedType = expected.GetType();
var actualType = actual.GetType();
if (expectedType != actualType)
{
Assert.Fail(
$"{path}: type changed across the round trip — expected {expectedType.Name}, got {actualType.Name}.");
return;
}
if (IsLeaf(expectedType))
{
Assert.True(
Equals(expected, actual),
$"{path}: expected '{expected}' but got '{actual}'.");
return;
}
CompareProperties(expected, actual, expectedType, path);
}
private static void CompareDictionaries(IDictionary expected, IDictionary actual, string path)
{
Assert.True(
expected.Count == actual.Count,
$"{path}: entry count changed — expected {expected.Count}, got {actual.Count}.");
foreach (DictionaryEntry entry in expected)
{
Assert.True(
actual.Contains(entry.Key),
$"{path}: key '{entry.Key}' is missing after the round trip.");
Compare(entry.Value, actual[entry.Key], $"{path}['{entry.Key}']");
}
}
private static void CompareSequences(IEnumerable expected, IEnumerable actual, string path)
{
var expectedItems = expected.Cast<object?>().ToList();
var actualItems = actual.Cast<object?>().ToList();
Assert.True(
expectedItems.Count == actualItems.Count,
$"{path}: element count changed — expected {expectedItems.Count}, got {actualItems.Count}.");
for (var i = 0; i < expectedItems.Count; i++)
{
Compare(expectedItems[i], actualItems[i], $"{path}[{i}]");
}
}
private static void CompareProperties(object expected, object actual, Type type, string path)
{
var properties = type
.GetProperties(BindingFlags.Public | BindingFlags.Instance)
.Where(p => p.CanRead && p.GetIndexParameters().Length == 0)
.ToList();
Assert.True(properties.Count > 0, $"{path}: {type.Name} exposes no readable properties to compare.");
foreach (var property in properties)
{
Compare(property.GetValue(expected), property.GetValue(actual), $"{path}.{property.Name}");
}
}
private static bool IsLeaf(Type type) =>
type.IsPrimitive
|| type.IsEnum
|| type == typeof(string)
|| type == typeof(decimal)
|| type == typeof(DateTime)
|| type == typeof(DateTimeOffset)
|| type == typeof(TimeSpan)
|| type == typeof(Guid)
// Value types with no collection members (e.g. TrackedOperationId) have a
// correct structural Equals of their own.
|| (type.IsValueType && !type.IsGenericType);
private static string Describe(object? value) => value is null ? "<null>" : $"{value.GetType().Name}('{value}')";
}
@@ -189,8 +189,9 @@ public class SiteActorPathTests : IAsyncLifetime
// fixture actually starts the host — a site node with no configured
// consolidated-database path fails fast rather than booting half-working.
["LocalDb:Path"] = _tempLocalDbPath,
// Configure a dummy central contact point to trigger ClusterClient creation
["ScadaBridge:Communication:CentralContactPoints:0"] = "akka.tcp://scadabridge@localhost:25510",
// A central gRPC endpoint so the Site branch builds its GrpcCentralTransport
// (the only site→central transport since Phase 4). Never dialled by this test.
["ScadaBridge:Communication:CentralGrpcEndpoints:0"] = "http://localhost:8083",
});
});
builder.ConfigureServices((context, services) =>
@@ -236,9 +237,10 @@ public class SiteActorPathTests : IAsyncLifetime
public async Task SiteActors_SiteCommunication_Exists()
=> await AssertActorExists("/user/site-communication");
[Fact]
public async Task SiteActors_CentralClusterClient_Exists()
=> await AssertActorExists("/user/central-cluster-client");
// SiteActors_CentralClusterClient_Exists was removed with the site→central Akka ClusterClient
// (the `central-cluster-client` actor) in the ClusterClient→gRPC migration's Phase 4. The site
// now reaches central over gRPC (GrpcCentralTransport dialling CentralControlService), which is
// not an actor in the local system, so there is no actor path to assert here.
private async Task AssertActorExists(string path)
{
@@ -0,0 +1,218 @@
using Grpc.Core;
using Grpc.Net.Client;
using Microsoft.AspNetCore.Builder;
using Microsoft.AspNetCore.Hosting;
using Microsoft.AspNetCore.TestHost;
using Microsoft.Extensions.DependencyInjection;
using Microsoft.Extensions.Hosting;
using Microsoft.Extensions.Logging.Abstractions;
using Microsoft.Extensions.Options;
using ZB.MOM.WW.ScadaBridge.Commons.Messages.Lifecycle;
using ZB.MOM.WW.ScadaBridge.Communication;
using ZB.MOM.WW.ScadaBridge.Communication.Grpc;
namespace ZB.MOM.WW.ScadaBridge.Host.Tests;
/// <summary>
/// T1B.3: <see cref="SitePairChannelProvider"/> failover/failback + credential/deadline proof over
/// two in-process gRPC <see cref="TestServer"/>s (NodeA preferred, NodeB standby). Exercises the
/// real client stack — PSK call credentials, per-call deadline, sticky failover, no-retry on
/// <see cref="StatusCode.DeadlineExceeded"/>, and failback to the preferred node.
/// </summary>
public sealed class GrpcSiteTransportFailoverTests : IAsyncLifetime
{
private const string SiteId = "site-1";
private const string SiteKey = "the-site-1-key";
private const string EndpointA = "http://node-a";
private const string EndpointB = "http://node-b";
private IHost _hostA = null!;
private IHost _hostB = null!;
private StubSiteCommandService _stubA = null!;
private StubSiteCommandService _stubB = null!;
private SitePairChannelProvider _provider = null!;
private volatile bool _preferredReachable;
/// <inheritdoc />
public async Task InitializeAsync()
{
(_hostA, _stubA) = await StartServerAsync();
(_hostB, _stubB) = await StartServerAsync();
var handlerA = _hostA.GetTestServer().CreateHandler();
var handlerB = _hostB.GetTestServer().CreateHandler();
_provider = new SitePairChannelProvider(
new FixedPskProvider(SiteKey),
Options.Create(new CommunicationOptions()),
NullLogger<SitePairChannelProvider>.Instance,
handlerFactory: endpoint => endpoint == EndpointA ? handlerA : handlerB,
reachabilityProbe: (_, _) => Task.FromResult(_preferredReachable));
_provider.UpdateSite(SiteId, EndpointA, EndpointB);
}
/// <inheritdoc />
public async Task DisposeAsync()
{
_provider.Dispose();
await _hostA.StopAsync();
await _hostB.StopAsync();
_hostA.Dispose();
_hostB.Dispose();
}
private static async Task<(IHost, StubSiteCommandService)> StartServerAsync()
{
var stub = new StubSiteCommandService();
var host = await new HostBuilder()
.ConfigureWebHost(web => web
.UseTestServer()
.ConfigureServices(services =>
{
services.AddGrpc();
services.AddSingleton(stub);
})
.Configure(app =>
{
app.UseRouting();
app.UseEndpoints(e => e.MapGrpcService<StubSiteCommandService>());
}))
.StartAsync();
return (host, stub);
}
private static LifecycleRequest EnableReq() =>
SiteCommandDtoMapper.ToLifecycleRequest(
new EnableInstanceCommand("cmd-1", "Site1.Pump1", DateTimeOffset.UtcNow));
private Task<LifecycleReply> CallAsync(TimeSpan? deadline = null) =>
_provider.ExecuteAsync(
SiteId,
(channel, ct) =>
{
var client = new SiteCommandService.SiteCommandServiceClient(channel);
return client.ExecuteLifecycleAsync(
EnableReq(),
deadline: DateTime.UtcNow + (deadline ?? TimeSpan.FromSeconds(10)),
cancellationToken: ct).ResponseAsync;
},
CancellationToken.None);
[Fact]
public async Task HealthyCall_HitsPreferredNodeA_WithPskAndSiteHeaderAndDeadline()
{
var reply = await CallAsync();
Assert.Equal(LifecycleReply.ReplyOneofCase.InstanceLifecycle, reply.ReplyCase);
Assert.Equal(1, _stubA.LifecycleCalls);
Assert.Equal(0, _stubB.LifecycleCalls);
Assert.Equal($"Bearer {SiteKey}", _stubA.LastAuthHeader);
Assert.Equal(SiteId, _stubA.LastSiteHeader);
Assert.True(_stubA.DeadlineWasSet, "the client must set a per-call deadline");
Assert.True(_provider.IsOnPreferredNode(SiteId));
}
[Fact]
public async Task Unavailable_OnNodeA_FailsOverToNodeB_AndStaysSticky()
{
_stubA.ThrowUnavailable = true;
var reply = await CallAsync();
// Failed over: A was tried (and threw), B answered.
Assert.Equal(LifecycleReply.ReplyOneofCase.InstanceLifecycle, reply.ReplyCase);
Assert.Equal(1, _stubA.LifecycleCalls);
Assert.Equal(1, _stubB.LifecycleCalls);
Assert.False(_provider.IsOnPreferredNode(SiteId));
// Sticky: the next call goes straight to B without re-touching A.
await CallAsync();
Assert.Equal(1, _stubA.LifecycleCalls);
Assert.Equal(2, _stubB.LifecycleCalls);
}
[Fact]
public async Task Failback_ReturnsToPreferredNodeA_OncePreferredIsReachableAgain()
{
_stubA.ThrowUnavailable = true;
await CallAsync(); // flips to B
Assert.False(_provider.IsOnPreferredNode(SiteId));
// NodeA recovers; the failback probe reports it reachable.
_stubA.ThrowUnavailable = false;
_preferredReachable = true;
var back = await _provider.TryFailbackAsync(SiteId, CancellationToken.None);
Assert.True(back);
Assert.True(_provider.IsOnPreferredNode(SiteId));
var beforeA = _stubA.LifecycleCalls;
await CallAsync();
Assert.Equal(beforeA + 1, _stubA.LifecycleCalls); // next call back on A
}
[Fact]
public async Task DeadlineExceeded_OnNodeA_IsNotRetriedOnNodeB()
{
// A DeadlineExceeded is ambiguous (a WriteTag/Deploy/Failover may already have executed), so
// — unlike Unavailable — it must NOT fail over to B. Modelled by A returning the status
// directly, isolating the retry-decision from TestServer's own timeout mechanics.
_stubA.StatusToThrow = StatusCode.DeadlineExceeded;
var ex = await Assert.ThrowsAsync<RpcException>(() => CallAsync());
Assert.Equal(StatusCode.DeadlineExceeded, ex.StatusCode);
Assert.Equal(1, _stubA.LifecycleCalls);
Assert.Equal(0, _stubB.LifecycleCalls); // B was never tried
Assert.True(_provider.IsOnPreferredNode(SiteId), "a deadline must not flip stickiness");
}
[Fact]
public async Task UnknownSite_Throws_SiteChannelUnavailable()
{
await Assert.ThrowsAsync<SiteChannelUnavailableException>(
() => _provider.ExecuteAsync<LifecycleReply>(
"not-configured", (_, _) => Task.FromResult(new LifecycleReply()), CancellationToken.None));
}
private sealed class FixedPskProvider(string key) : ISitePskProvider
{
public ValueTask<string> GetAsync(string siteId, CancellationToken ct) => new(key);
public void Invalidate(string siteId) { }
}
/// <summary>Stub SiteCommandService recording what the client sent and letting a test steer faults.</summary>
private sealed class StubSiteCommandService : SiteCommandService.SiteCommandServiceBase
{
private int _lifecycleCalls;
public int LifecycleCalls => Volatile.Read(ref _lifecycleCalls);
public string? LastAuthHeader { get; private set; }
public string? LastSiteHeader { get; private set; }
public bool DeadlineWasSet { get; private set; }
public volatile bool ThrowUnavailable;
public StatusCode? StatusToThrow;
public override Task<LifecycleReply> ExecuteLifecycle(LifecycleRequest request, ServerCallContext context)
{
Interlocked.Increment(ref _lifecycleCalls);
LastAuthHeader = context.RequestHeaders
.FirstOrDefault(h => h.Key == ControlPlaneCredentials.AuthorizationHeader)?.Value;
LastSiteHeader = context.RequestHeaders
.FirstOrDefault(h => h.Key == ControlPlaneCredentials.SiteHeader)?.Value;
DeadlineWasSet = context.Deadline != DateTime.MaxValue;
if (ThrowUnavailable)
{
throw new RpcException(new Status(StatusCode.Unavailable, "node down"));
}
if (StatusToThrow is { } status)
{
throw new RpcException(new Status(status, "modelled fault"));
}
return Task.FromResult(SiteCommandDtoMapper.ToLifecycleReply(
new InstanceLifecycleResponse("cmd-1", "Site1.Pump1", true, null, DateTimeOffset.UtcNow)));
}
}
}
@@ -0,0 +1,285 @@
using Akka.Actor;
using Grpc.Core;
using Grpc.Net.Client;
using Microsoft.AspNetCore.Builder;
using Microsoft.AspNetCore.Hosting;
using Microsoft.AspNetCore.TestHost;
using Microsoft.Extensions.DependencyInjection;
using Microsoft.Extensions.Hosting;
using Microsoft.Extensions.Logging.Abstractions;
using Microsoft.Extensions.Options;
using ZB.MOM.WW.ScadaBridge.Commons.Messages.DebugView;
using ZB.MOM.WW.ScadaBridge.Commons.Messages.InboundApi;
using ZB.MOM.WW.ScadaBridge.Commons.Messages.Lifecycle;
using ZB.MOM.WW.ScadaBridge.Commons.Messages.Management;
using ZB.MOM.WW.ScadaBridge.Commons.Messages.RemoteQuery;
using ZB.MOM.WW.ScadaBridge.Communication;
using ZB.MOM.WW.ScadaBridge.Communication.Actors;
using ZB.MOM.WW.ScadaBridge.Communication.Grpc;
namespace ZB.MOM.WW.ScadaBridge.Host.Tests;
/// <summary>
/// The site command plane's gRPC front door (T1B.2): the same PSK gate and readiness convention
/// as <c>SiteStreamGrpcServer</c>, and a decode→dispatch→encode round trip through the ONE
/// <see cref="SiteCommandDispatcher"/> for one command in each of the six oneof groups. Runs
/// in-process over <see cref="TestServer"/> — no ports, no containers — with the interceptor
/// registered BY TYPE on <c>AddGrpc</c> (never pre-registered in DI), the shape that keeps
/// <c>Grpc.AspNetCore</c>'s own activation in the picture. See
/// <see cref="ControlPlaneAuthEndToEndTests"/> for why that matters.
/// </summary>
public sealed class SiteCommandGrpcServiceTests : IDisposable
{
private const string SiteKey = "the-site-a-command-key";
private const string SiteId = "site-a";
private readonly ActorSystem _system = ActorSystem.Create("sitecmd-tests");
/// <inheritdoc />
public void Dispose() => _system.Dispose();
/// <summary>A dispatcher whose Deployment Manager proxy is a canned-reply responder.</summary>
private SiteCommandDispatcher DispatcherWithResponder(out IActorRef responder)
{
responder = _system.ActorOf(Props.Create(() => new Responder()));
var dispatcher = new SiteCommandDispatcher(SiteId, responder, (_, _) => null);
// The parked handler is node-local; point it at the responder too so the Parked group can
// be exercised end-to-end (the null-guard path is covered by the dispatcher unit tests).
dispatcher.RegisterParkedMessageHandler(responder);
return dispatcher;
}
private SiteCommandGrpcService ReadyService()
{
var service = new SiteCommandGrpcService(NullLogger<SiteCommandGrpcService>.Instance);
service.SetReady(DispatcherWithResponder(out _));
return service;
}
private static async Task<IHost> StartHost(SiteCommandGrpcService service)
=> await new HostBuilder()
.ConfigureWebHost(web => web
.UseTestServer()
.ConfigureServices(services =>
{
// BY TYPE on AddGrpc — never AddSingleton the interceptor (see the sibling test).
services.AddGrpc(o => o.Interceptors.Add<ControlPlaneAuthInterceptor>());
services.AddSingleton(Options.Create(new CommunicationOptions { GrpcPsk = SiteKey }));
services.AddSingleton(service);
})
.Configure(app =>
{
app.UseRouting();
app.UseEndpoints(e => e.MapGrpcService<SiteCommandGrpcService>());
}))
.StartAsync();
private static SiteCommandService.SiteCommandServiceClient Client(IHost host, string? key)
{
var server = host.GetTestServer();
var options = new GrpcChannelOptions { HttpHandler = server.CreateHandler() };
if (key is not null)
{
options.WithSiteCredentials(new FixedPskProvider(key), SiteId);
}
var channel = GrpcChannel.ForAddress(server.BaseAddress, options);
return new SiteCommandService.SiteCommandServiceClient(channel);
}
private sealed class FixedPskProvider(string key) : ISitePskProvider
{
public ValueTask<string> GetAsync(string siteId, CancellationToken ct) => new(key);
public void Invalidate(string siteId) { }
}
// ── Auth gating (delegated to ControlPlaneAuthInterceptor, proven wired here) ──
[Fact]
public async Task NoCredentials_IsRejected_WithPermissionDenied()
{
using var host = await StartHost(ReadyService());
var client = Client(host, key: null);
var ex = await Assert.ThrowsAsync<RpcException>(() => client.ExecuteQueryAsync(
new QueryRequest { UnsubscribeDebugView = new UnsubscribeDebugViewRequestDto() }).ResponseAsync);
Assert.Equal(StatusCode.PermissionDenied, ex.StatusCode);
}
[Fact]
public async Task WrongKey_IsRejected_WithPermissionDenied()
{
using var host = await StartHost(ReadyService());
var client = Client(host, "some-other-key");
var ex = await Assert.ThrowsAsync<RpcException>(() => client.ExecuteQueryAsync(
new QueryRequest { UnsubscribeDebugView = new UnsubscribeDebugViewRequestDto() }).ResponseAsync);
Assert.Equal(StatusCode.PermissionDenied, ex.StatusCode);
}
// ── Readiness ──
[Fact]
public async Task BeforeSetReady_IsRejected_WithUnavailable_EvenWithACorrectKey()
{
// Auth passes (correct key); the readiness gate then rejects until the site actor graph is up.
var unready = new SiteCommandGrpcService(NullLogger<SiteCommandGrpcService>.Instance);
using var host = await StartHost(unready);
var client = Client(host, SiteKey);
var ex = await Assert.ThrowsAsync<RpcException>(() => client.ExecuteLifecycleAsync(
new LifecycleRequest { EnableInstance = new EnableInstanceCommandDto { CommandId = "c" } }).ResponseAsync);
Assert.Equal(StatusCode.Unavailable, ex.StatusCode);
}
// ── decode → dispatch → encode, one command per oneof group ──
[Fact]
public async Task ExecuteLifecycle_RoundTripsThroughTheDispatcher()
{
using var host = await StartHost(ReadyService());
var client = Client(host, SiteKey);
var command = new EnableInstanceCommand("cmd-1", "Site1.Pump1", DateTimeOffset.UtcNow);
var reply = await client.ExecuteLifecycleAsync(
new LifecycleRequest { EnableInstance = SiteCommandDtoMapper.ToProto(command) });
var decoded = Assert.IsType<InstanceLifecycleResponse>(SiteCommandDtoMapper.FromLifecycleReply(reply));
Assert.True(decoded.Success);
Assert.Equal("cmd-1", decoded.CommandId);
}
[Fact]
public async Task ExecuteOpcUa_RoundTripsThroughTheDispatcher()
{
using var host = await StartHost(ReadyService());
var client = Client(host, SiteKey);
var command = new BrowseNodeCommand("conn", null, null, null);
var reply = await client.ExecuteOpcUaAsync(
new OpcUaRequest { BrowseNode = SiteCommandDtoMapper.ToProto(command) });
Assert.IsType<BrowseNodeResult>(SiteCommandDtoMapper.FromOpcUaReply(reply));
}
[Fact]
public async Task ExecuteQuery_DebugSnapshot_RoundTripsThroughTheDispatcher()
{
using var host = await StartHost(ReadyService());
var client = Client(host, SiteKey);
var command = new DebugSnapshotRequest("Site1.Pump1", "corr-4");
var reply = await client.ExecuteQueryAsync(
new QueryRequest { DebugSnapshot = SiteCommandDtoMapper.ToProto(command) });
var decoded = Assert.IsType<DebugViewSnapshot>(SiteCommandDtoMapper.FromQueryReply(reply));
Assert.Equal("Site1.Pump1", decoded.InstanceUniqueName);
}
[Fact]
public async Task ExecuteQuery_UnsubscribeDebugView_ReturnsTheFireAndForgetAck()
{
// Fire-and-forget: the service Tells the target and returns the synthetic ack so a unary
// RPC still answers, keeping the caller-visible fire-and-forget semantics.
using var host = await StartHost(ReadyService());
var client = Client(host, SiteKey);
var reply = await client.ExecuteQueryAsync(new QueryRequest
{
UnsubscribeDebugView = SiteCommandDtoMapper.ToProto(
new UnsubscribeDebugViewRequest("Site1.Pump1", "corr-6")),
});
Assert.Equal(QueryReply.ReplyOneofCase.UnsubscribeDebugView, reply.ReplyCase);
}
[Fact]
public async Task ExecuteParked_RoundTripsThroughTheNodeLocalHandler()
{
using var host = await StartHost(ReadyService());
var client = Client(host, SiteKey);
var command = new ParkedMessageQueryRequest("corr-7", SiteId, 2, 25, DateTimeOffset.UtcNow);
var reply = await client.ExecuteParkedAsync(
new ParkedRequest { ParkedMessageQuery = SiteCommandDtoMapper.ToProto(command) });
var decoded = Assert.IsType<ParkedMessageQueryResponse>(SiteCommandDtoMapper.FromParkedReply(reply));
Assert.True(decoded.Success);
Assert.Equal("corr-7", decoded.CorrelationId);
}
[Fact]
public async Task ExecuteRoute_RoundTripsThroughTheDispatcher()
{
using var host = await StartHost(ReadyService());
var client = Client(host, SiteKey);
var command = new RouteToCallRequest(
"corr-12", "Site1.Pump1", "Start", new Dictionary<string, object?>(), DateTimeOffset.UtcNow, null);
var reply = await client.ExecuteRouteAsync(
new RouteRequest { RouteToCall = SiteCommandDtoMapper.ToProto(command) });
var decoded = Assert.IsType<RouteToCallResponse>(SiteCommandDtoMapper.FromRouteReply(reply));
Assert.True(decoded.Success);
Assert.Equal("corr-12", decoded.CorrelationId);
}
// ── Failover: the reply completes before the leave is initiated ──
[Fact]
public async Task TriggerFailover_ReturnsTheAck_BeforeTheLeaveIsInitiated()
{
// The resolver records the order of its calls: PrepareFailover does a DRY-RUN resolve to
// build the ack (recorded synchronously, before the RPC returns), and the real leave runs
// only on the deferred CommitLeave — so the recorded order is always resolve-then-leave,
// i.e. the ack is on the wire before the node begins leaving.
var events = new List<string>();
var leaveHappened = new ManualResetEventSlim(false);
Func<string, bool, string?> resolve = (_, dryRun) =>
{
lock (events) { events.Add(dryRun ? "resolve" : "leave"); }
if (!dryRun) leaveHappened.Set();
return "akka.tcp://scadabridge@site-a-node-a:8082";
};
var service = new SiteCommandGrpcService(NullLogger<SiteCommandGrpcService>.Instance);
var dispatcher = new SiteCommandDispatcher(SiteId, _system.ActorOf(Props.Create(() => new Responder())), resolve);
service.SetReady(dispatcher);
using var host = await StartHost(service);
var client = Client(host, SiteKey);
var reply = await client.TriggerFailoverAsync(
new TriggerSiteFailoverDto { CorrelationId = "corr-16", SiteId = SiteId });
Assert.True(reply.Accepted);
Assert.Equal("akka.tcp://scadabridge@site-a-node-a:8082", reply.TargetAddress);
Assert.True(leaveHappened.Wait(TimeSpan.FromSeconds(5)), "the deferred leave never ran");
lock (events)
{
Assert.Equal(new[] { "resolve", "leave" }, events);
}
}
/// <summary>Deployment Manager stand-in: replies each command with a canned reply of the right group.</summary>
private sealed class Responder : ReceiveActor
{
public Responder() => ReceiveAny(msg => Sender.Tell(ReplyFor(msg)));
private static object ReplyFor(object m) => m switch
{
EnableInstanceCommand e =>
new InstanceLifecycleResponse(e.CommandId, e.InstanceUniqueName, true, null, DateTimeOffset.UtcNow),
BrowseNodeCommand => new BrowseNodeResult([], false, null, null),
DebugSnapshotRequest d => new DebugViewSnapshot(d.InstanceUniqueName, [], [], DateTimeOffset.UtcNow),
RouteToCallRequest r =>
new RouteToCallResponse(r.CorrelationId, true, null, null, DateTimeOffset.UtcNow),
ParkedMessageQueryRequest p => new ParkedMessageQueryResponse(
p.CorrelationId, p.SiteId, [], 0, p.PageNumber, p.PageSize, true, null, DateTimeOffset.UtcNow),
_ => new Akka.Actor.Status.Failure(new InvalidOperationException($"no canned reply for {m.GetType().Name}")),
};
}
}
@@ -41,6 +41,9 @@ public class StartupValidatorTests
// T0.3: the gRPC control plane is fail-closed, so a Site node without a preshared
// key serves nothing while still looking healthy. Required at boot for that reason.
["ScadaBridge:Communication:GrpcPsk"] = "test-site-control-plane-key",
// Phase 4: gRPC (CentralControlService) is the only site→central transport, so a Site
// node must list at least one central gRPC endpoint to dial (no Akka fallback remains).
["ScadaBridge:Communication:CentralGrpcEndpoints:0"] = "http://central-a:8083",
};
[Fact]
@@ -97,6 +100,30 @@ public class StartupValidatorTests
Assert.Null(Record.Exception(() => StartupValidator.Validate(config)));
}
[Fact]
public void SiteWithoutCentralGrpcEndpoints_FailsValidation()
{
// Phase 4: gRPC (CentralControlService) is the only site→central transport, so a Site
// node with no central gRPC endpoint has nothing to dial — heartbeats, health, notification
// forwards and audit ingest all silently fail. Fail fast at boot instead.
var values = ValidSiteConfig();
values.Remove("ScadaBridge:Communication:CentralGrpcEndpoints:0");
var config = BuildConfig(values);
var ex = Assert.Throws<InvalidOperationException>(() => StartupValidator.Validate(config));
Assert.Contains("CentralGrpcEndpoints", ex.Message);
}
[Fact]
public void CentralWithoutCentralGrpcEndpoints_PassesValidation()
{
// A central node hosts CentralControlService; it does not dial it, so it declares no
// endpoints. The site-only requirement must not fire for Central.
var config = BuildConfig(ValidCentralConfig());
Assert.Null(Record.Exception(() => StartupValidator.Validate(config)));
}
[Fact]
public void MissingRole_FailsValidation()
{
@@ -1,7 +1,5 @@
using System.Collections.Concurrent;
using System.Collections.Immutable;
using Akka.Actor;
using Akka.Cluster.Tools.Client;
using Akka.TestKit.Xunit2;
using Microsoft.Extensions.DependencyInjection;
using Microsoft.Extensions.Logging.Abstractions;
@@ -12,6 +10,10 @@ using ZB.MOM.WW.ScadaBridge.AuditLog.Site.Telemetry;
using ZB.MOM.WW.Audit;
using ZB.MOM.WW.ScadaBridge.Commons.Entities.Sites;
using ZB.MOM.WW.ScadaBridge.Commons.Interfaces.Repositories;
using ZB.MOM.WW.ScadaBridge.Commons.Messages.Audit;
using ZB.MOM.WW.ScadaBridge.Commons.Messages.Deployment;
using ZB.MOM.WW.ScadaBridge.Commons.Messages.Health;
using ZB.MOM.WW.ScadaBridge.Commons.Messages.Notification;
using ZB.MOM.WW.ScadaBridge.Commons.Types;
using ZB.MOM.WW.ScadaBridge.Commons.Types.Audit;
using ZB.MOM.WW.ScadaBridge.Commons.Types.Enums;
@@ -32,10 +34,10 @@ namespace ZB.MOM.WW.ScadaBridge.IntegrationTests.AuditLog;
/// <see cref="ClusterClientSiteAuditClient"/>, the real
/// <see cref="SiteCommunicationActor"/> forward, the real
/// <see cref="CentralCommunicationActor"/> routing, and the real
/// <c>AuditLogIngestActor</c> ingest — only the cross-cluster ClusterClient
/// transport itself is substituted by an in-process <see cref="ClusterClientRelay"/>
/// that unwraps <see cref="ClusterClient.Send"/> exactly as a real ClusterClient
/// would (a multi-node cluster is out of scope for an in-process test).
/// <c>AuditLogIngestActor</c> ingest — only the cross-cluster gRPC transport itself is
/// substituted by an in-process <see cref="BridgeCentralTransport"/> that Tells the central
/// actor exactly as the real <c>GrpcCentralTransport</c> would (a multi-node cluster is out of
/// scope for an in-process test).
/// </para>
/// <para>
/// The central audit store is an in-memory <see cref="IAuditLogRepository"/> —
@@ -49,18 +51,23 @@ namespace ZB.MOM.WW.ScadaBridge.IntegrationTests.AuditLog;
public class SiteAuditPushFlowTests : TestKit
{
/// <summary>
/// In-process stand-in for a real Akka ClusterClient: unwraps a
/// <see cref="ClusterClient.Send"/> and forwards the inner message to the
/// central actor, preserving the original sender so the reply routes back to
/// the site's Ask. A real ClusterClient does exactly this across the cluster
/// boundary; the in-process relay keeps the test free of a multi-node setup.
/// In-process stand-in for the site→central gRPC transport (<c>GrpcCentralTransport</c>):
/// each site→central send is Tell'd straight to the central actor, preserving the reply-to so
/// the central reply routes back to the site's Ask. A real transport does exactly this across
/// the cluster boundary; the in-process bridge keeps the test free of a multi-node/gRPC setup.
/// </summary>
private sealed class ClusterClientRelay : ReceiveActor
private sealed class BridgeCentralTransport : ICentralTransport
{
public ClusterClientRelay(IActorRef central)
{
Receive<ClusterClient.Send>(send => central.Forward(send.Message));
}
private readonly IActorRef _central;
public BridgeCentralTransport(IActorRef central) => _central = central;
public void SubmitNotification(NotificationSubmit message, IActorRef replyTo) => _central.Tell(message, replyTo);
public void QueryNotificationStatus(NotificationStatusQuery message, IActorRef replyTo) => _central.Tell(message, replyTo);
public void IngestAuditEvents(IngestAuditEventsCommand message, IActorRef replyTo) => _central.Tell(message, replyTo);
public void IngestCachedTelemetry(IngestCachedTelemetryCommand message, IActorRef replyTo) => _central.Tell(message, replyTo);
public void ReconcileSite(ReconcileSiteRequest message, IActorRef replyTo) => _central.Tell(message, replyTo);
public void ReportSiteHealth(SiteHealthReport message, IActorRef replyTo) => _central.Tell(message, replyTo);
public void SendHeartbeat(HeartbeatMessage message, IActorRef self) => _central.Tell(message, self);
}
/// <summary>
@@ -148,7 +155,7 @@ public class SiteAuditPushFlowTests : TestKit
var centralCommActor = Sys.ActorOf(Props.Create(() => new CentralCommunicationActor(
centralProvider,
new DefaultSiteClientFactory(),
Substitute.For<ISiteCommandTransport>(),
TimeSpan.FromSeconds(5))));
centralCommActor.Tell(new RegisterAuditIngest(ingestActor));
@@ -162,14 +169,15 @@ public class SiteAuditPushFlowTests : TestKit
await using var writer = new SqliteAuditWriter(
writerOptions, NullLogger<SqliteAuditWriter>.Instance, nodeIdentity);
// Real SiteCommunicationActor. RegisterCentralClient is given the relay
// standing in for the central ClusterClient.
// Real SiteCommunicationActor. Its site→central transport is the in-process bridge that
// Tells the central actor (standing in for GrpcCentralTransport dialling CentralControlService).
var siteCommActor = Sys.ActorOf(Props.Create(() => new SiteCommunicationActor(
"site-1",
new CommunicationOptions(),
CreateTestProbe().Ref))); // deployment-manager proxy is unused here
var relay = Sys.ActorOf(Props.Create(() => new ClusterClientRelay(centralCommActor)));
siteCommActor.Tell(new RegisterCentralClient(relay));
CreateTestProbe().Ref, // deployment-manager proxy is unused here
null,
null,
new BridgeCentralTransport(centralCommActor))));
// The production site audit push client — the unit under integration.
var auditClient = new ClusterClientSiteAuditClient(