18 Commits

Author SHA1 Message Date
Joseph Doherty 2fa5e93c73 docs(plans): tick 1B DoD (proportionate rig gate PASS) 2026-07-22 22:38:38 -04:00
Joseph Doherty 01693b13db docs(grpc): Phase 1B live gate — PASS (proportionate); central→site rides authenticated gRPC for 3 sites; records the central-wide SiteTransport finding 2026-07-22 22:38:21 -04:00
Joseph Doherty 86ad4d5c8e feat(comm): T1B.3 — central-side gRPC site-command transport seam (default Akka)
Extract the central→site send path in CentralCommunicationActor behind a new
ISiteCommandTransport, selected by ScadaBridge:Communication:SiteTransport
(Akka | Grpc, default Akka — rollback = flip the flag). CommunicationService's
27 commands, SiteCallAuditActor's 2 parked relays and DebugStreamBridgeActor's
subscribe/unsubscribe are untouched; the seam sits below SiteEnvelope.

- AkkaSiteTransport: today's per-site ClusterClient path extracted verbatim
  (the _siteClients lookup + ClusterClient.Send with the reply-to sender
  preserved, and the "no client ⇒ warn + drop, caller's Ask times out" path).
- GrpcSiteTransport: dials the site SiteCommandService (T1B.1 proto client) via
  SiteCommandDtoMapper, PSK + x-scadabridge-site on the channel through
  ControlPlaneCredentials, per-command deadlines set EQUAL to today's
  CommunicationService Ask timeouts (per-command, not per-group: DeploymentState
  query and TriggerSiteFailover use QueryTimeout; the two parked relays map to
  QueryTimeout so SiteCallAudit's inner RelayTimeout 10s < 30s ordering holds;
  WaitForAttribute keeps its dynamic Timeout + IntegrationTimeout).
- SitePairChannelProvider: per-site A/B channel pair with sticky failover
  (flip only on Unavailable — NEVER on DeadlineExceeded, a write/deploy/failover
  may have run), background failback probe to the preferred node with 1s→60s
  doubling backoff, PSK invalidation on site removal. Fed by the SAME DB refresh
  loop (extended to carry GrpcNodeA/GrpcNodeBAddress) — no second poll.

Tests: actor-with-substitute-transport (routing, Ask-reply plumbing, per-site
lifecycle across refreshes), ResolveDeadline pinned to each command's current
Ask timeout, and GrpcSiteTransport/SitePairChannelProvider over dual in-process
TestServers (PSK+header+deadline attached, Unavailable failover + stickiness,
failback to preferred, no-retry-on-DeadlineExceeded). Proto csproj untouched
(no active <Protobuf> item). Full solution builds 0 warnings; Communication.Tests
607 green with Akka default.
2026-07-22 20:09:43 -04:00
Joseph Doherty 518c699b90 feat(comm): extract SiteCommandDispatcher; site serves commands over gRPC too (T1B.2)
Refactor SiteCommunicationActor's central→site routing table into one
SiteCommandDispatcher — the single routing truth for the 28 migrated commands
(IntegrationCallRequest, the dead 29th, stays on the actor and out of the
dispatcher). The Akka actor and the new SiteCommandGrpcService both route through
one dispatcher instance so the two transports can never drift on where a command
goes. Server-side only: nothing central flips to gRPC yet (that is T1B.3);
ClusterClient remains the live path.

Decisions worth recording:

- Targets preserved byte-for-byte. Lifecycle/OPC UA/query/route → the Deployment
  Manager singleton proxy; DeployArtifacts/EventLog/parked → their null-guarded
  handlers with the exact same "handler not available" replies; the parked
  handler stays NODE-LOCAL (per-node replicated-store owner), never the singleton
  proxy — pinned by a dispatcher test that asserts the target is the parked probe
  and NOT the dm proxy.

- Sender preservation intact. The actor's command handlers became thin
  DispatchCommand delegations that still Forward (central Ask → reply routes
  straight back); the existing SiteCommunicationActorTests pass unchanged, which
  is the regression guard for that plumbing. UnsubscribeDebugView keeps its
  fire-and-forget shape: the actor Forwards, the gRPC service Tells + returns the
  synthetic UnsubscribeDebugViewAck so a unary RPC still answers.

- Ack-before-Leave on failover. The dispatcher's PrepareFailover resolves the
  standby with a DRY-RUN (no leave) to build the ack, and hands back a deferred
  CommitLeave; the gRPC service returns the ack, then schedules the real
  Cluster.Leave — so a caller reaching the very node about to leave still gets its
  ack instead of a broken stream. The actor path keeps today's coupled
  resolve-and-leave (over ClusterClient the ack Tell only enqueues, so order is
  immaterial). Proven at both levels: a dispatcher test asserts the ack is built
  before CommitLeave runs, and a TestServer test asserts the recorded seam order
  is resolve-then-leave.

- ControlPlaneAuthInterceptor gates SiteCommandService by EXTENDING
  DefaultGatedPrefixes (descriptor-derived), not by adding a constructor — the
  one-public-ctor invariant and its test stay green.

Tests: SiteCommandDispatcherTests (28-command routing incl. parked node-locality
and both failover paths) and SiteCommandGrpcService TestServer tests (auth,
readiness→Unavailable, one command per oneof group, failover ordering). Full
solution build 0/0; Communication.Tests 574 and Host.Tests 377 green. No active
<Protobuf> item.
2026-07-22 20:07:58 -04:00
Joseph Doherty 59b13d317b feat(grpc): T1B.1 — site_command.proto + SiteCommandDtoMapper + round-trip goldens
Phase 1B's contract slice: the wire shape and the canonical translation for the
28 central→site commands that leave ClusterClient. No behaviour changes yet —
SiteCommunicationActor and CentralCommunicationActor are untouched; the
dispatcher refactor (T1B.2) and the central transport seam (T1B.3) consume this.

Protos/site_command.proto (package scadabridge.sitecommand.v1, service
SiteCommandService): six domain RPCs, each with a `oneof` request/reply
envelope. The grouping is what carries deadline policy — every command inside a
group shares a CommunicationOptions timeout class today, so one RPC per group
keeps the deadline choice in one place on the client and one dispatch switch on
the server, while the oneof keeps each command individually typed:
ExecuteLifecycle(6) · ExecuteOpcUa(8) · ExecuteQuery(4) · ExecuteParked(5) ·
ExecuteRoute(4) · TriggerFailover(1). IntegrationCallRequest — the 29th entry on
SiteCommunicationActor's receive table — is deliberately excluded as dead code
(2026-07-22-integration-call-routing-is-dead-code.md).

Contract decisions worth knowing:

- Nullable COLLECTIONS ride in per-collection wrapper messages
  (DeployArtifactsCommand's six artifact lists, CertTrustResult.Certs,
  RouteToCallRequest.Parameters). proto3 repeated/map collapses null into empty,
  and that distinction is live at the site — the same silent-data-loss class the
  transport round-trip guard exposed in PLAN-05 T8. Goldens cover null, empty
  and populated for each.
- Nullable strings use the empty-string-means-null convention already set by
  AuditEventDtoMapper, with ONE exception: RouteToWaitForAttributeRequest's
  TargetValueEncoded, where "wait for the empty string" is a real target, so it
  carries a StringValue wrapper. Both behaviours are asserted, not assumed.
- Nullable enums ride in one-field messages (proto3 enums have no presence and
  no stock wrapper). Enum translation is an explicit switch in both directions —
  never by ordinal — so reordering a C# enum cannot re-map the wire; every wire
  enum reserves 0 for _UNSPECIFIED and decodes to a documented safe default
  rather than faulting a command from a version-skewed peer.
- New LooseValueCodec carries the surviving `object?` members (script params and
  return values, attribute values, tag read/write values) as a type-tagged union
  so a boxed value keeps its runtime CLR type, as it does today under Akka's
  type-preserving JSON serializer. Dates ride as invariant round-trip strings,
  not Timestamp, which would silently normalise away DateTime.Kind and
  DateTimeOffset.Offset. Lists/maps recurse; anything outside the tagged set
  falls back to JSON and is documented as CLR-type-lossy.
- DebugViewSnapshot gets its own full-fidelity alarm/attribute messages rather
  than reusing sitestream's AlarmStateUpdate, which flattens values to display
  strings — right for a live stream, lossy for a snapshot the UI treats as
  authoritative. The encoder omits an AlarmStateChanged.Condition that already
  equals the record's derived default, so computed alarms round-trip exactly
  (record equality compares the nullable backing field, not the property).

Tests are reflection-driven so the coverage cannot drift: the round-trip theory
enumerates the mapper's own ToProto overloads, the envelope guards enumerate the
generated oneof descriptors, and a missing golden fails the build. 216 new tests
green (Communication 532 total, Commons 684 total, solution build 0/0).

Codegen is checked in under SiteCommandGrpc/ per the sitestream recipe; the
<Protobuf> ItemGroup stays commented out (an active one segfaults protoc in the
linux_arm64 Docker image). docker/regen-proto.sh now handles every proto in that
ItemGroup instead of just sitestream, and re-comments idempotently.
2026-07-22 20:04:23 -04:00
Joseph Doherty aa60f43866 Merge Phase 1A: site→central control plane over gRPC (PR #26) 2026-07-22 20:01:42 -04:00
Joseph Doherty f7c7811940 docs(grpc): Phase 1A live gate — PASS (3 RPCs + restart-reconcile + coexistence); records the Kestrel-drop defect 2026-07-22 19:55:09 -04:00
Joseph Doherty 0e162cb250 fix(grpc): central Kestrel gRPC listener was dropping the :5000 HTTP surface (T1A.2 regression)
Caught on the Phase 1A rig proof: central-a logged only "Now listening on:
http://[::]:8083" and nothing else. Central UI, the Management + Inbound API, and
the /health/* endpoints Traefik + IActiveNodeGate depend on were all gone, with no
startup error — the node booted, joined the cluster and served gRPC fine.

Cause: calling options.ListenAnyIP in ConfigureKestrel puts Kestrel into
explicit-endpoints mode, which SUPPRESSES the URLs from ASPNETCORE_URLS/--urls
entirely — it is not additive, contrary to the comment T1A.2 shipped. Central's whole
HTTP/1 surface lives on that URL (http://+:5000 on the rig, a different port in prod),
so binding only the gRPC port silently deleted it. The site branch has the same shape
but no ASPNETCORE_URLS surface to lose — it binds every port it needs explicitly.

Fix: parse the port(s) from the configured URLs and re-declare them (Http1AndHttp2)
alongside the gRPC port (Http2) in the one ConfigureKestrel call. New
Program.ParseHttpBindPorts + CentralHttpBindPortsTests (14 cases: wildcard/ipv6/
hostname hosts, multi-URL, de-dupe, scheme-default, null/blank, unparseable-skipped).

Why no test caught it originally: unit/E2E tests use TestServer, which never binds
real Kestrel. Only a live node exposes the missing listener — which is exactly what
the rig proof is for.
2026-07-22 19:50:32 -04:00
Joseph Doherty aa49a1d078 docs(plans): tick T1B.3/T1B.4 — central site-command transport seam complete (3be85f19) 2026-07-22 19:43:58 -04:00
Joseph Doherty 81ced76654 docs(plans): tick T1A.3/T1A.4 — site ICentralTransport seam complete (33b15f10) 2026-07-22 19:31:05 -04:00
Joseph Doherty 33b15f10a4 feat(grpc): site-side ICentralTransport seam + gRPC transport (T1A.3)
Introduce ICentralTransport as the site->central choke point inside
SiteCommunicationActor. The seven site->central sends (notification submit/
status, audit + cached-telemetry ingest, reconcile, health, heartbeat) now
delegate to an injected transport instead of owning ClusterClient.Send inline.

- AkkaCentralTransport: verbatim extraction of today's ClusterClient.Send path,
  including the exact sender-forwarding that routes central's reply straight
  back to the waiting Ask. Default when no transport is injected -> behaviour
  unchanged, existing SiteCommunicationActorTests pass as-is.
- GrpcCentralTransport + CentralChannelProvider: dial CentralControlService with
  sticky failover + background failback (1s-doubling-cap-60s), PSK +
  x-scadabridge-site via ControlPlaneCredentials, per-call deadlines mirroring
  today's Ask timeouts. Cross-node retry ONLY on provably-unsent
  connect failures; never on DeadlineExceeded. Heartbeat stays fire-and-forget.
- StaticSitePskProvider: site's single own-key provider (fail-closed).
- CommunicationOptions: CentralTransport flag (default Akka) + CentralGrpcEndpoints
  (validator: required when transport=Grpc). Host selects the impl; the
  ClusterClient is created only on the Akka path.

Tests: actor-with-fake-transport (7 delegations + fault routing + heartbeat
no-fault), AkkaCentralTransport sender-forwarding, GrpcCentralTransport over
in-process TestServer (failover flip, sticky, failback, PSK+header, deadline,
no-retry-on-deadline), validator. Communication.Tests 371 green, Host.Tests 391
green; the three above-seam suites pass unmodified.
2026-07-22 19:29:14 -04:00
Joseph Doherty 9f2c96f486 docs(plans): tick T1B.2 — SiteCommandDispatcher + site gRPC command service (cd6c20e1) 2026-07-22 19:12:45 -04:00
Joseph Doherty fc398b47d3 docs(plans): tick T1A.2 — central CentralControlService hosting + per-site auth (780bb9c3) 2026-07-22 18:55:28 -04:00
Joseph Doherty 780bb9c369 feat(grpc): host CentralControlService on the central node (T1A.2)
Central now ALSO listens for the seven site→central control messages over
gRPC, alongside the existing ClusterClient path. Nothing flips to gRPC yet —
sites keep CentralTransport=Akka (T1A.3's job); central simply starts also
accepting.

- CentralControlGrpcService (Communication.Grpc): decodes each RPC onto the
  SAME in-process message the ClusterClient path carries, Asks the existing
  CentralCommunicationActor (zero handler-logic changes), encodes the reply via
  the T1A.1 mapper. Readiness-gated like SiteStreamGrpcServer.SetReady —
  Unavailable until AkkaHostedService hands the actor over. Heartbeat stays
  fire-and-forget (Tell, always-OK, never gated on readiness). Ingest reuses the
  shared SiteStreamGrpcServer.AuditIngestAskTimeout constant. Fault→status
  mapping is retry-aware: Unavailable (never dispatched, safe to cross-node
  retry) vs DeadlineExceeded/Internal (it ran, do not re-send elsewhere).

- CentralControlAuthInterceptor (Host): a SEPARATE interceptor class, not a
  variant constructor on ControlPlaneAuthInterceptor. Central's model is per-site
  (verify the Bearer token against the key for the site in the required
  x-scadabridge-site header, via ISitePskProvider) where a site verifies its one
  own-key — a genuinely different model. Fail-closed on every branch: missing or
  blank header, unresolvable key, and mismatched token all → PermissionDenied,
  never pass-through. One public constructor only (the explicit-prefix ctor is
  internal), pinned by a reflection test — a second public ctor makes
  Grpc.AspNetCore's GetFactory() throw per-call and silently disables the gate.

- Explicit Kestrel h2c listener on new option ScadaBridge:Node:CentralGrpcPort
  (default 8083, symmetric with sites), mirroring the Site branch. Additive to
  central's :5000 HTTP/1 surface, which is untouched — gRPC does NOT go through
  Traefik (HTTP/1 only). Registered by type on AddGrpc; service mapped with
  MapGrpcService. Port range-validated by NodeOptionsValidator.

- Rig: publish the central gRPC port 9013:8083 / 9014:8083 on both central nodes
  so a later task can exercise it.

Tests: CentralControlEndToEndTests (Host.Tests, TestServer + real interceptor +
real service over a stub actor) proves auth positives/negatives are
distinguishable and covers unary + the ingest bridge shapes; the interceptor is
registered BY TYPE, never in DI. CentralControlAuthInterceptorTests pins the
per-site gate + one-public-ctor invariant. CentralControlGrpcServiceTests
(Communication.Tests, TestKit) covers the readiness gate, fire-and-forget
heartbeat, and the DeadlineExceeded-vs-Unavailable status mapping. No active
<Protobuf> item. Communication.Tests (356) + Host.Tests (384) green.
2026-07-22 18:53:59 -04:00
Joseph Doherty c90b353820 docs(plans): tick T1B.1 — site_command.proto + mapper + 194 goldens (a7481174) 2026-07-22 18:41:22 -04:00
Joseph Doherty c615ba5f78 docs(plans): tick T1A.1 — central_control.proto + mapper + 32 goldens (d7455577) 2026-07-22 18:31:18 -04:00
Joseph Doherty d7455577a8 feat(grpc): central_control.proto + DTO mapper for the 7 site→central control RPCs (T1A.1)
Phase 1A of the ClusterClient→gRPC migration needs a wire contract for the
seven messages SiteCommunicationActor forwards to /user/central-communication.
This lands the contract and its mapper only — hosting (T1A.2) and the site-side
transport seam (T1A.3) follow.

`Protos/central_control.proto` (package scadabridge.centralcontrol.v1, service
CentralControlService) declares SubmitNotification, QueryNotificationStatus,
IngestAuditEvents, IngestCachedTelemetry, ReconcileSite, ReportSiteHealth and
Heartbeat. Note the direction is the inverse of SiteStreamService: here the site
dials and central serves.

Decisions worth recording:

- The two ingest RPCs IMPORT sitestream.proto and reuse AuditEventBatch /
  CachedTelemetryBatch / IngestAck rather than redeclaring them. The site
  telemetry actor already builds those messages, so a second copy would fork one
  wire contract into two kept in lockstep by hand. ForwardState / IngestedAtUtc
  stay off-wire exactly as they are today.
- Heartbeat replies google.protobuf.Empty — it is fire-and-forget and must never
  surface a fault onto the heartbeat timer path.
- The three NULLABLE SiteHealthReport collections travel inside single-field
  wrapper messages (ConnectionEndpointMapDto / TagQualityMapDto /
  NodeStatusListDto). proto3 cannot express presence on repeated/map fields, but
  null and empty genuinely differ here — SiteHealthCollector emits
  `ClusterNodes: _clusterNodes?.ToList()` and the central health surface reads
  null as "not reported", not as "reported empty". Same reasoning drives the
  BoolValue/Int64Value/DoubleValue wrappers on LocalDbReplicationConnected,
  LocalDbOplogBacklog and the two age gauges, whose docs are explicit that null
  is not zero/false.
- ConnectionHealthEnum reserves 0 for UNSPECIFIED instead of mapping Connected
  onto it, and the decoder resolves anything unknown to ConnectionHealth.Error.
  An unrecognised connection state must not render as "healthy".
- Guid? execution ids travel as "D" strings with empty meaning null; a malformed
  non-empty value throws rather than being laundered into "no correlation".
- DateTimeOffset normalizes to a UTC instant (a protobuf Timestamp has no
  offset). Lossless in practice — every producer stamps UTC — and documented +
  asserted rather than left implicit.

SiteCallDtoMapper gains a ToDto(SiteCall) overload. Its doc comment previously
asserted such a method "would be dead code"; that held only while ClusterClient
was the sole path from IngestCachedTelemetryCommand (which carries SiteCall, not
SiteCallOperational) to central. Comment corrected alongside.

Golden tests round-trip every message through a real protobuf encode/decode —
DTO → proto → bytes → proto → DTO — with a fully-populated case and a
null/empty/minimal case for every optional member. Verified to have teeth by
mutation: dropping a scalar, collapsing an empty nullable collection to absent,
and nulling a gauge each fail a test.

Codegen stays CHECKED IN under CentralControlGrpc/ (protoc segfaults in the
linux_arm64 Docker image); no active <Protobuf> item is committed.
docker/regen-proto.sh is generalized to `regen-proto.sh [sitestream|
centralcontrol|all]` — it now injects the ItemGroup rather than unwrapping a
comment, so it no longer depends on there being exactly one Protobuf line, and
it restores the csproj verbatim on every exit path.
2026-07-22 18:28:27 -04:00
Joseph Doherty aa7c5cd138 docs(plans): tick Phase 0 DoD — PR #25 merged to main @ 3fa95555 2026-07-22 18:12:01 -04:00
58 changed files with 52272 additions and 584 deletions
+2
View File
@@ -39,6 +39,7 @@ services:
ports:
- "9001:5000" # Web UI + Inbound API
- "9011:8081" # Akka remoting (host access for CLI/debugging)
- "9013:8083" # gRPC control plane (CentralControlService, T1A.2)
volumes:
- ./central-node-a/appsettings.Central.json:/app/appsettings.Central.json:ro
- ./central-node-a/logs:/app/logs
@@ -86,6 +87,7 @@ services:
ports:
- "9002:5000" # Web UI + Inbound API
- "9012:8081" # Akka remoting
- "9014:8083" # gRPC control plane (CentralControlService, T1A.2)
volumes:
- ./central-node-b/appsettings.Central.json:/app/appsettings.Central.json:ro
- ./central-node-b/logs:/app/logs
+88 -49
View File
@@ -1,92 +1,131 @@
#!/usr/bin/env bash
#
# Regenerates the gRPC C# files from sitestream.proto.
# Regenerates the gRPC C# files from the Communication project's .proto files.
#
# Background: protoc (linux/arm64) segfaults inside our Docker build container
# (Grpc.Tools 2.71.0). As a workaround the generated Sitestream.cs +
# SitestreamGrpc.cs are checked into src/ZB.MOM.WW.ScadaBridge.Communication/SiteStreamGrpc/
# and the Protobuf ItemGroup in the .csproj is commented out — Docker just
# compiles the checked-in C# files.
# (Grpc.Tools). As a workaround the generated C# is checked into
# src/ZB.MOM.WW.ScadaBridge.Communication/SiteStreamGrpc/ for sitestream.proto,
# CentralControlGrpc/ for central_control.proto, and SiteCommandGrpc/ for
# site_command.proto — and the Protobuf ItemGroup in the .csproj is commented out,
# so Docker just compiles the checked-in files.
#
# Run this script ON YOUR DEV MACHINE whenever Protos/sitestream.proto changes:
# Run this script ON YOUR DEV MACHINE whenever a .proto changes:
#
# 1. Temporarily uncomments the Protobuf ItemGroup so Grpc.Tools runs.
# 2. dotnet build (regen writes fresh files to obj/).
# 3. Copies the regenerated files back into SiteStreamGrpc/.
# 4. Re-comments the Protobuf ItemGroup so Docker builds stay safe.
# docker/regen-proto.sh [sitestream|centralcontrol|sitecommand|all] (default: all)
#
# 1. Injects a Protobuf ItemGroup for the selected proto(s) so Grpc.Tools runs.
# 2. Deletes the stale checked-in C# so a failed regen is obvious.
# 3. dotnet build (regen writes fresh files to obj/).
# 4. Copies the regenerated files back into the source tree.
# 5. Restores the original csproj so no active Protobuf item is left behind.
#
# Only the SELECTED protos get a Protobuf item. Enabling one whose generated C#
# is still checked in would define every generated type twice, which is why the
# per-proto selection exists. central_control.proto imports sitestream.proto,
# but protoc resolves that from the project-relative path — the import needs no
# Protobuf item of its own.
#
# Once we move to a Dockerfile base image that ships a working linux/arm64
# protoc, this script can be retired and Docker can regen the proto on every
# protoc, this script can be retired and Docker can regen the protos on every
# build like every other normal .NET project.
set -euo pipefail
TARGET="${1:-all}"
case "$TARGET" in
sitestream|centralcontrol|sitecommand|all) ;;
*) echo "usage: $0 [sitestream|centralcontrol|sitecommand|all]" >&2; exit 2 ;;
esac
SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)"
REPO_ROOT="$(cd "$SCRIPT_DIR/.." && pwd)"
COMM_DIR="$REPO_ROOT/src/ZB.MOM.WW.ScadaBridge.Communication"
CSPROJ="$COMM_DIR/ZB.MOM.WW.ScadaBridge.Communication.csproj"
GEN_DIR="$COMM_DIR/SiteStreamGrpc"
GEN="$COMM_DIR/obj/Debug/net10.0/Protos"
echo "=== Regenerating gRPC files from sitestream.proto ==="
echo "=== Regenerating gRPC files ($TARGET) ==="
if [[ ! -f "$CSPROJ" ]]; then
echo "ERROR: csproj not found at $CSPROJ" >&2
exit 1
fi
# Backup so we can always restore the comment state on failure.
# Backup so we can always restore the comment state on failure. Leaving the
# csproj with an active Protobuf item is the one outcome that breaks Docker, so
# every exit path restores this copy.
BACKUP="$(mktemp)"
cp "$CSPROJ" "$BACKUP"
trap 'cp "$BACKUP" "$CSPROJ"; rm -f "$BACKUP"; echo "Restored csproj from backup."' ERR
# 1. Uncomment the Protobuf ItemGroup (strip the surrounding <!-- ... --> wrapper).
python3 - <<PY
import re, pathlib
p = pathlib.Path("$CSPROJ")
src = p.read_text()
# Find the commented Protobuf block and unwrap it.
new = re.sub(
r"<!--\s*\n(\s*<ItemGroup>\s*\n\s*<Protobuf [^>]*/>\s*\n\s*</ItemGroup>)\s*\n\s*-->",
r"\1",
src,
count=1,
)
if new == src:
raise SystemExit("Couldn't find commented Protobuf ItemGroup to enable.")
p.write_text(new)
# 1. Inject an ItemGroup holding just the selected protos, immediately before
# the closing </Project>. The documented commented-out block is left alone.
python3 - "$CSPROJ" "$TARGET" <<'PY'
import pathlib, sys
csproj, target = pathlib.Path(sys.argv[1]), sys.argv[2]
protos = []
if target in ("sitestream", "all"):
protos.append("sitestream.proto")
if target in ("centralcontrol", "all"):
protos.append("central_control.proto")
if target in ("sitecommand", "all"):
protos.append("site_command.proto")
items = "\n".join(
f' <Protobuf Include="Protos\\{p}" GrpcServices="Both" />' for p in protos)
block = f" <ItemGroup>\n{items}\n </ItemGroup>\n\n</Project>"
src = csproj.read_text()
if "</Project>" not in src:
raise SystemExit("Couldn't find </Project> to inject the Protobuf ItemGroup before.")
csproj.write_text(src.replace("</Project>", block, 1))
PY
# 2. Delete the stale files so any failure to regen is obvious.
rm -f "$GEN_DIR/Sitestream.cs" "$GEN_DIR/SitestreamGrpc.cs"
if [[ "$TARGET" == "sitestream" || "$TARGET" == "all" ]]; then
rm -f "$COMM_DIR/SiteStreamGrpc/Sitestream.cs" "$COMM_DIR/SiteStreamGrpc/SitestreamGrpc.cs"
fi
if [[ "$TARGET" == "centralcontrol" || "$TARGET" == "all" ]]; then
rm -f "$COMM_DIR/CentralControlGrpc/CentralControl.cs" \
"$COMM_DIR/CentralControlGrpc/CentralControlGrpc.cs"
fi
if [[ "$TARGET" == "sitecommand" || "$TARGET" == "all" ]]; then
rm -f "$COMM_DIR/SiteCommandGrpc/SiteCommand.cs" \
"$COMM_DIR/SiteCommandGrpc/SiteCommandGrpc.cs"
fi
# 3. Regenerate by building.
echo "Building Communication project (regen)..."
dotnet build "$CSPROJ" --nologo -v minimal | tail -5
# 4. Copy generated files back into the source tree.
mkdir -p "$GEN_DIR"
cp "$COMM_DIR/obj/Debug/net10.0/Protos/Sitestream.cs" "$GEN_DIR/Sitestream.cs"
cp "$COMM_DIR/obj/Debug/net10.0/Protos/SitestreamGrpc.cs" "$GEN_DIR/SitestreamGrpc.cs"
echo "Copied regenerated files to $GEN_DIR/"
# 5. Re-comment the Protobuf ItemGroup so Docker builds keep working.
python3 - <<PY
import re, pathlib
p = pathlib.Path("$CSPROJ")
src = p.read_text()
new = re.sub(
r"(\s*<ItemGroup>\s*\n\s*<Protobuf [^>]*/>\s*\n\s*</ItemGroup>)",
r"\n <!--\1\n -->",
src,
count=1,
)
p.write_text(new)
PY
if [[ "$TARGET" == "sitestream" || "$TARGET" == "all" ]]; then
mkdir -p "$COMM_DIR/SiteStreamGrpc"
cp "$GEN/Sitestream.cs" "$GEN/SitestreamGrpc.cs" "$COMM_DIR/SiteStreamGrpc/"
echo "Copied regenerated files to SiteStreamGrpc/"
fi
if [[ "$TARGET" == "centralcontrol" || "$TARGET" == "all" ]]; then
mkdir -p "$COMM_DIR/CentralControlGrpc"
cp "$GEN/CentralControl.cs" "$GEN/CentralControlGrpc.cs" "$COMM_DIR/CentralControlGrpc/"
echo "Copied regenerated files to CentralControlGrpc/"
fi
if [[ "$TARGET" == "sitecommand" || "$TARGET" == "all" ]]; then
mkdir -p "$COMM_DIR/SiteCommandGrpc"
cp "$GEN/SiteCommand.cs" "$GEN/SiteCommandGrpc.cs" "$COMM_DIR/SiteCommandGrpc/"
echo "Copied regenerated files to SiteCommandGrpc/"
fi
# 5. Restore the backed-up csproj — i.e. drop the injected ItemGroup — so Docker
# builds keep working.
cp "$BACKUP" "$CSPROJ"
rm -f "$BACKUP"
trap - ERR
echo ""
echo "Done. Review and commit:"
echo " git diff src/ZB.MOM.WW.ScadaBridge.Communication/Protos/sitestream.proto"
echo " git diff src/ZB.MOM.WW.ScadaBridge.Communication/Protos/"
echo " git diff src/ZB.MOM.WW.ScadaBridge.Communication/SiteStreamGrpc/"
echo " git diff src/ZB.MOM.WW.ScadaBridge.Communication/CentralControlGrpc/"
echo " git diff src/ZB.MOM.WW.ScadaBridge.Communication/SiteCommandGrpc/"
echo " git diff -- src/ZB.MOM.WW.ScadaBridge.Communication/*.csproj # must be EMPTY"
@@ -141,3 +141,96 @@ reaches the page HTML** — has not executed since. Fix is a valid 32-hex SID in
the same code path — but a live `SubscribeInstance` under load is untested here.
- Key **rotation** on a live pair.
- `docker-env2` was updated with its own key but not redeployed or gated.
---
## Phase 1A — central control plane (site→central over gRPC) — **PASS** (2026-07-22)
Branch `feat/grpc-central-control` @ `0e162cb2`. Rig rebuilt; **site-a flipped to
`CentralTransport=Grpc`** with `CentralGrpcEndpoints=[central-a:8083, central-b:8083]`,
**site-b/c left on Akka** (default) to prove coexistence. The site-a flag flip was a
DoD-test-only rig edit — reverted from the branch, never committed (the plan keeps the default
`Akka` until Phase 4).
### The defect this gate caught (T1A.2 shipped it; fixed in `0e162cb2`)
**First rebuild: central's entire HTTP surface was gone.** central-a logged only
`Now listening on: http://[::]:8083` — no `:5000`. Central UI, the Management + Inbound API,
and every `/health/*` endpoint (Traefik routing + `IActiveNodeGate` both depend on them) were
dead. The node booted, joined the cluster and served gRPC fine; **no startup error.**
Cause: `builder.WebHost.ConfigureKestrel(o => o.ListenAnyIP(8083, Http2))` puts Kestrel into
explicit-endpoints mode, which **suppresses the URLs from `ASPNETCORE_URLS`/`--urls`** — it is
not additive, contrary to the comment T1A.2 shipped. Central's whole HTTP/1 surface lives on
that URL (`http://+:5000` on the rig; a different port in production). The site branch has the
same `ConfigureKestrel` shape but nothing on `ASPNETCORE_URLS` to lose — it binds every port it
needs explicitly — which is why the pattern looked safe.
Every unit + E2E test uses `TestServer`, which never binds real Kestrel, so the whole suite
(6,872) stayed green. Only a live node exposes a missing listener. Fix: parse the port(s) from
the configured URLs and re-declare them (`Http1AndHttp2`) alongside the gRPC port (`Http2`) in
the one `ConfigureKestrel` call — `Program.ParseHttpBindPorts` + `CentralHttpBindPortsTests`
(14 cases). After the fix: `Now listening on: http://[::]:5000` **and** `:8083`; 9001 ready
`200`/active Healthy, 9002 standby, Traefik LB `200`, CLI over the LB works.
### Checks (post-fix rebuild)
| # | Check | Result |
|---|---|---|
| 1 | site-a rides authenticated gRPC to `CentralControlService` | **PASS** — Heartbeat, `ReportSiteHealth`, `ReconcileSite` all HTTP/2 → 200; **0** auth failures on either central |
| 2 | Heartbeat drives the active flag | **PASS** — 144+ heartbeats over gRPC; site-a `online=True` at central |
| 3 | Health page live | **PASS**`ReportSiteHealth` lands; central shows site-a `online=True`, sequence advancing, alongside Akka site-b/c |
| 4 | Reconcile works after site restart | **PASS** — restarted `site-a-a`; it logged `Site→central transport: gRPC to 2 central endpoint(s)`, then `Reconcile pass … complete: 0 fetched, 0 failed, 0 orphan(s)` |
| 5 | Coexistence | **PASS** — site-b/c log `Created ClusterClient to central`; both `online=True` — Akka and gRPC sites side by side |
| 6 | Central HTTP surface intact under the new gRPC listener | **PASS** — after the fix (see above) |
### Not independently exercised on this gate
- **Notification e2e (`SubmitNotification`/`QueryNotificationStatus`) and audit ingest
(`IngestAuditEvents`/`IngestCachedTelemetry`)** were NOT driven live: the rig has templates
but **no deployed instance**, so nothing emits site→central notifications or audit rows on its
own. These four RPCs traverse the identical `CentralControlGrpcService`
`CentralCommunicationActor` Ask path that checks 13 proved live under real auth, and their
payload mappers carry 32 round-trip goldens — but the payloads themselves were not put over
the wire here. **Phase 2's central-kill S&F soak is where they get their live workout;** flag
for a fuller 1A proof if a deployed-instance rig is set up before then.
- Cross-node failover/failback of the site→central channel under a central-node kill (unit-proven
via TestServer; not exercised on the rig at 1A).
---
## Phase 1B — site command plane (central→site over gRPC) — **PASS (proportionate)** (2026-07-23)
Branch `feat/grpc-sitecommand` (rebased onto 1A-merged main). Rig rebuilt with **central
flipped to `SiteTransport=Grpc`** — a DoD-test-only edit to both central appsettings, reverted
from the branch (default stays `Akka`).
`SiteTransport` is a **central-wide** flag (`CentralCommunicationActor.SelectTransport` picks one
transport for all sites), so the plan's "flip for site-a only" is not achievable — the flip
routes central→site commands for **all three sites** to gRPC. Command-plane per-site coexistence
therefore cannot be shown (unlike the site→central plane in 1A, which is per-site). This is a
plan-vs-code finding, recorded rather than worked around.
### Checks
| # | Check | Result |
|---|---|---|
| 1 | central→site commands ride authenticated gRPC `SiteCommandService` | **PASS**`ExecuteQuery` (event-log) and `ExecuteParked` (parked query) HTTP/2 → 200 |
| 2 | Per-site PSK resolution across all sites | **PASS** — site-a, site-b, site-c each answered `ExecuteQuery` → 200 under its own `SB-GRPC-PSK-{site}`; **0** auth failures on any site node |
| 3 | Central HTTP surface intact under the central-wide gRPC flip | **PASS** — central `:5000` **and** `:8083` both listening; 9001 ready `200`, LB `200` (the 1A Kestrel fix carried through the merge) |
| 4 | Query round-trips return correct data | **PASS** — every `health event-log`/`parked-messages` returned `success:true` with the right `siteId` and empty result sets (bare rig) |
### Not driven on this proportionate gate
- **`TriggerSiteFailover`** — unit-proven (two ordering tests pin ack-before-`Leave` via the
dispatcher's dry-run resolve + deferred `CommitLeave`), but not live-driven here: it is
destructive (forces the active node to leave) and has no CLI verb (UI/management-only).
- **Tag commands (`BrowseNode`/`ReadTagValues`/`WriteTag`) and the lifecycle enable/disable
matrix** — need a deployed instance + data connection the bare rig lacks (same blocker as 1A).
- **Parked retry against the STANDBY node** — needs a parked operation to exist, which needs a
deployed instance.
- **Command-plane coexistence (site-b/c on Akka while site-a on gRPC)** — not expressible; the
flag is central-wide (check-1/2 instead prove all three sites over gRPC with distinct keys).
The instance-dependent matrix (tag ops, lifecycle, standby parked retry) and `TriggerSiteFailover`
get their live exercise in Phase 3's full UI command matrix; the transport itself is proven here.
@@ -306,21 +306,21 @@ Critical path ≈ 1B: **~46 weeks total**, matching the design estimate.
- [x] T0.2 File dead-`IntegrationCallRequest` issue; record exclusion (28 of 29 migrate)
- [x] T0.3 `ControlPlaneAuthInterceptor` + `CommunicationOptions.GrpcPsk` + `SitePskProvider`; gate SiteStream; PSK attached on central's streaming + pull clients
- [x] T0.4 Rig dev keys (all 3 sites + central store seeds) + interceptor/provider/wiring tests
- [ ] Phase 0 DoD: suite green; rig unauthenticated ⇒ `PermissionDenied`, authenticated paths work; PR merged
- [x] Phase 0 DoD: suite green; rig unauthenticated ⇒ `PermissionDenied`, authenticated paths work; PR merged (#25, ff to `main` @ `3fa95555`; gate PASS in `2026-07-22-clusterclient-to-grpc-live-gate.md`)
**Phase 1A — central control plane** (worktree, `feat/grpc-central-control`)
- [ ] T1A.1 `central_control.proto` (7 RPCs; checked-in codegen) + `CentralControlDtoMapper` + round-trip golden tests
- [ ] T1A.2 Central hosting: `AddGrpc` + per-site-PSK interceptor (`x-scadabridge-site`), `CentralGrpcPort` h2c listener (8083), `CentralControlGrpcService` (Ask existing handlers), readiness gate
- [ ] T1A.3 `ICentralTransport` (Akka extract + Grpc impl), `CentralChannelProvider` (sticky failover/failback, backoff, deadlines, PSK), `CentralTransport` flag default `Akka`, `CentralGrpcEndpoints` option + validator
- [ ] T1A.4 Tests: actor-with-fake-transport ×7, TestServer transport tests, S&F/audit/health suites pass unmodified
- [ ] 1A DoD: rig site-a on `Grpc` proves all 5 site→central paths while site-b/c stay Akka; PR merged (before 1B)
- [x] T1A.1 `central_control.proto` (7 RPCs; checked-in codegen) + `CentralControlDtoMapper` + round-trip golden tests
- [x] T1A.2 Central hosting: `AddGrpc` + per-site-PSK interceptor (`x-scadabridge-site`), `CentralGrpcPort` h2c listener (8083), `CentralControlGrpcService` (Ask existing handlers), readiness gate
- [x] T1A.3 `ICentralTransport` (Akka extract + Grpc impl), `CentralChannelProvider` (sticky failover/failback, backoff, deadlines, PSK), `CentralTransport` flag default `Akka`, `CentralGrpcEndpoints` option + validator
- [x] T1A.4 Tests: actor-with-fake-transport ×7, TestServer transport tests, S&F/audit/health suites pass unmodified
- [x] 1A DoD: rig site-a on `Grpc` proves site→central paths (heartbeat/health/reconcile + coexistence) while site-b/c stay Akka; PR #26 merged (`aa60f438`). Notification/audit deferred to Phase 2 soak (no deployed instance); rig caught + fixed a central `:5000` HTTP-drop regression (`0e162cb2`)
**Phase 1B — site command plane** (worktree, `feat/grpc-sitecommand`)
- [ ] T1B.1 `site_command.proto` (6 oneof RPCs / 28 commands) + `SiteCommandDtoMapper` + round-trip golden tests (all 28 + replies)
- [ ] T1B.2 `SiteCommandDispatcher` refactor (actor + new `SiteCommandGrpcService` share it; parked stays node-local)
- [ ] T1B.3 `ISiteCommandTransport` in `CentralCommunicationActor` (Akka extract + Grpc impl), `SitePairChannelProvider` (Site entity Grpc columns + DB refresh loop), `SiteTransport` flag default `Akka`
- [ ] T1B.4 Tests: dispatcher routing ×28, actor envelope/reply plumbing, TestServer service tests, existing Communication suites green
- [ ] 1B DoD: rig central on `Grpc` for site-a proves full command matrix incl. standby parked retry; rebased on 1A; PR merged
- [x] T1B.1 `site_command.proto` (6 oneof RPCs / 28 commands) + `SiteCommandDtoMapper` + round-trip golden tests (all 28 commands + 22 reply shapes + 18 nested types; reflection-driven coverage guard over the mapper surface and the generated oneof descriptors)
- [x] T1B.2 `SiteCommandDispatcher` refactor (actor + new `SiteCommandGrpcService` share it; parked stays node-local; failover ack-before-Leave via dry-run resolve + deferred `CommitLeave`; interceptor `DefaultGatedPrefixes` extended to `SiteCommandService`)
- [x] T1B.3 `ISiteCommandTransport` in `CentralCommunicationActor` (Akka extract + Grpc impl), `SitePairChannelProvider` (Site entity Grpc columns + DB refresh loop), `SiteTransport` flag default `Akka`
- [x] T1B.4 Tests: dispatcher routing ×28, actor envelope/reply plumbing, TestServer service tests, existing Communication suites green
- [x] 1B DoD (proportionate): rig central on `SiteTransport=Grpc` proves central→site rides authenticated gRPC `SiteCommandService` for all 3 sites (`ExecuteQuery`/`ExecuteParked` → 200, per-site PSK, 0 auth failures); rebased on 1A. Instance-dependent commands (tag ops/lifecycle/standby parked retry) + `TriggerSiteFailover` deferred to Phase 3 (no deployed instance / destructive UI-only); command-plane coexistence not expressible (`SiteTransport` is central-wide). Gate: `2026-07-22-clusterclient-to-grpc-live-gate.md`
**Phase 2 ∥ 3 — cutover + soak**
- [ ] P2 All sites `CentralTransport=Grpc`; central-kill S&F soak (no loss/dupes), failback observed, health sequences clean
@@ -73,7 +73,7 @@
"id": "P0.DoD",
"phase": "0",
"subject": "Phase 0 DoD: suite green; rig unauthenticated => PermissionDenied, authenticated paths work; PR merged",
"status": "in_progress",
"status": "completed",
"activeForm": "Verifying the Phase 0 DoD",
"blockedBy": [
"T0.1",
@@ -87,42 +87,46 @@
"id": "T1A.1",
"phase": "1A",
"subject": "central_control.proto (7 RPCs, checked-in codegen) + CentralControlDtoMapper + round-trip golden tests",
"status": "pending",
"status": "completed",
"activeForm": "Authoring central_control.proto and its mappers",
"blockedBy": [
"P0.DoD"
]
],
"notes": "d7455577 on feat/grpc-central-control. 7 RPCs; ingest RPCs reuse sitestream AuditEventBatch/CachedTelemetryBatch/IngestAck by import. 32 goldens, verified to have teeth by mutation. PLAN CORRECTIONS FOUND: (1) actor sends IngestAuditEventsCommand/-Reply (IReadOnlyList<Guid>), NOT the batch/IngestAck types the plan's table claims - mapper bridges; (2) CachedTelemetryEntry carries SiteCall not SiteCallOperational, needed a new SiteCallDtoMapper.ToDto(SiteCall); (3) SiteHealthReport is ~33 members and 5 are INIT-ONLY props not ctor params - FromDto needs an object initializer or they silently drop; (4) 3 collections are nullable with load-bearing null, proto3 cannot express presence on repeated/map => wrapper messages; (5) ConnectionHealth has no Unspecified member, so naive mapping puts Connected on proto3 zero - reserved 0 and unknown decodes to Error, never Connected."
},
{
"id": "T1A.2",
"phase": "1A",
"subject": "Central hosting: AddGrpc + per-site-PSK interceptor, CentralGrpcPort h2c listener, CentralControlGrpcService, readiness gate",
"status": "pending",
"status": "completed",
"activeForm": "Hosting CentralControlService on central",
"blockedBy": [
"T1A.1"
]
],
"notes": "780bb9c3 on feat/grpc-central-control. NEW class CentralControlAuthInterceptor (one public ctor, 3-arg internal, pinned by reflection test) - central verifies per-site PSK via ISitePskProvider keyed by required x-scadabridge-site header, fail-closed on missing/blank/unknown/mismatch. CentralControlGrpcService Asks existing CentralCommunicationActor (0 handler changes), readiness via SetReady mirror, heartbeat Tell/never-gated, ingest reuses AuditIngestAskTimeout. Central branch had NO AddGrpc/Kestrel before - added h2c listener on CentralGrpcPort default 8083, :5000 untouched. Rig ports 9013/9014:8083 published. PLAN GAPS: (1) mappers throw on unset WKT fields - test DTOs must carry a timestamp (no prod impact); (2) plan gave no deadline for Submit/QueryNotification - used NotificationForwardTimeout, check T1A.3 client sets same. 17 CentralControl tests, Host.Tests 384, Communication.Tests 356."
},
{
"id": "T1A.3",
"phase": "1A",
"subject": "ICentralTransport (Akka extract + Grpc impl), CentralChannelProvider, CentralTransport flag, CentralGrpcEndpoints option",
"status": "pending",
"status": "completed",
"activeForm": "Building the site->central transport seam",
"blockedBy": [
"T1A.1"
]
],
"notes": "33b15f10 on feat/grpc-central-control. ICentralTransport: 7 methods, actor delegates all 7; optional ctor param (null->actor self-builds AkkaCentralTransport in PreStart, byte-identical Akka path). CentralChannelProvider = sticky failover/failback, 1s-double-cap-60s backoff, per-site PSK not needed (site's own GrpcPsk + site header). Flag CentralTransport default Akka; on Grpc the ClusterClient is not created at all. Deadlines matched per-RPC to today's Ask timeouts (table in report). PLAN CORRECTION (important): connect-refused surfaces as StatusCode.Internal + 'Error starting gRPC call'/HttpRequestException, NOT Unavailable, in this Grpc.Net version - IsConnectFailure covers Unavailable OR that Internal class, still excludes DeadlineExceeded (verified in code + DeadlineExceeded_IsNotRetriedOnThePeer test). Above-seam suites (NotificationForwarder/HealthReportSender/SiteAuditTelemetry) pass UNMODIFIED. Communication.Tests 371, Host.Tests 391."
},
{
"id": "T1A.4",
"phase": "1A",
"subject": "Tests: actor-with-fake-transport x7, TestServer transport tests, S&F/audit/health suites pass unmodified",
"status": "pending",
"status": "completed",
"activeForm": "Testing the central control plane",
"blockedBy": [
"T1A.2",
"T1A.3"
]
],
"notes": "Tests shipped inside T1A.1/1A.2/1A.3 per the plan folding T1A.4 into each task. Actor-with-fake-transport x7, TestServer transport tests, above-seam suites pass unmodified."
},
{
"id": "P1A.DoD",
@@ -138,42 +142,46 @@
"id": "T1B.1",
"phase": "1B",
"subject": "site_command.proto (6 oneof RPCs / 28 commands) + SiteCommandDtoMapper + round-trip golden tests",
"status": "pending",
"status": "completed",
"activeForm": "Authoring site_command.proto and its mappers",
"blockedBy": [
"P0.DoD"
]
],
"notes": "a7481174 on feat/grpc-sitecommand. 6 oneof RPCs / 28 commands / 22 replies / 18 nested, verified by reflection guards not a hand list. PLAN CORRECTIONS: (1) generic Guid?->empty-string-means-null is UNSAFE for RouteToWaitForAttributeRequest.TargetValueEncoded, a nullable STRING where '' is a real wait target distinct from null - gave it a StringValue wrapper; (2) RouteToGetAttributesResponse.Values is NON-nullable dict but RouteToCallRequest.Parameters is nullable - two decode paths (absent->empty vs absent->null); (3) DeployArtifactsCommand has 6 NULLABLE artifact collections, proto3 repeated collapses null<->empty => wrapper messages (PLAN-05 T8 class); (4) AlarmStateChanged.Condition is derived-on-read over a nullable backing field - encoding unconditionally breaks record equality, mapper omits condition==computed default. Added seam types for T1B.2/3: SiteCommandGroup, GroupOf/GroupOfReply, UnsubscribeDebugViewAck marker (unsubscribe is Tell today, unary RPC must answer). REBASE NOTE: union-conflict expected in Communication.csproj commented Protobuf ItemGroup + docker/regen-proto.sh vs 1A."
},
{
"id": "T1B.2",
"phase": "1B",
"subject": "SiteCommandDispatcher refactor (actor + SiteCommandGrpcService share it; parked stays node-local)",
"status": "pending",
"status": "completed",
"activeForm": "Extracting the site command dispatcher",
"blockedBy": [
"T1B.1"
]
],
"notes": "cd6c20e1 on feat/grpc-sitecommand. SiteCommandDispatcher = pure routing (ResolveRoute -> Route{Disposition,Target,Reply}); actor delegates all 27 non-failover Receives to it, gRPC SiteCommandGrpcService calls the SAME dispatcher; shared via SetReady(dispatcher) hand-off like SiteStreamGrpcServer. Parked stays node-local (proven NotSame(proxy)). Interceptor: SiteCommandService added to DefaultGatedPrefixes via descriptor - NO new ctor, one-public-ctor test green. PLAN CORRECTION (important): existing actor code issues cluster Leave BEFORE the ack (fine over ClusterClient Tell, WRONG for gRPC ack-before-Leave). Solved via the pre-existing dryRun param on ClusterFailoverCoordinator.FailOverOldest: seam widened to Func<role,dryRun,addr?>, actor commits immediately (dryRun:false, byte-identical today), gRPC resolves dry-run then defers CommitLeave until after ack. Two ordering tests pin resolve-then-leave. Actor got an OPTIONAL dispatcher param so existing SiteCommunicationActorTests pass with ZERO edits. Server derives local-Ask timeout from ServerCallContext.Deadline (remaining), 2min fallback. Communication.Tests 574, Host.Tests 377."
},
{
"id": "T1B.3",
"phase": "1B",
"subject": "ISiteCommandTransport in CentralCommunicationActor (Akka extract + Grpc impl), SitePairChannelProvider, SiteTransport flag",
"status": "pending",
"status": "completed",
"activeForm": "Building the central->site transport seam",
"blockedBy": [
"T1B.1"
]
],
"notes": "3be85f19 on feat/grpc-sitecommand. ISiteCommandTransport injected into CentralCommunicationActor; HandleSiteEnvelope->_transport.Send(env,Sender), HandleSiteAddressCacheLoaded->_transport.ReconcileSites. AkkaSiteTransport extracted verbatim (incl no-client->drop path). GrpcSiteTransport + SitePairChannelProvider (A/B from GrpcNode*Address, per-site PSK via ISitePskProvider, Invalidate on removal). Rode the EXISTING LoadSiteAddressesFromDb loop (extended SiteAddressCacheLoaded to carry gRPC cols too) - ONE poll loop. Flag SiteTransport default Akka. PLAN CORRECTIONS: (1) LoadSiteAddressesFromDb DOES exist (known); (2) plan deadline table wrong TWICE - DeploymentStateQuery uses QueryTimeout not LifecycleTimeout, TriggerSiteFailover uses QueryTimeout not LifecycleTimeout - resolver is per-command-type not per-group, verified in code; (3) SiteCommandGroup XML doc's 'shared deadline class' claim is false; (4) THIRD SiteEnvelope producer DebugStreamBridgeActor (not just CommunicationService+SiteCallAudit) - reply plumbing routes to a real actor sender, not only Ask temp actors. RetryParkedOperation/DiscardParkedOperation->QueryTimeout(30s) keeps SiteCallAudit RelayTimeout(10s)<30s. Communication.Tests 607, gRPC Host.Tests 34. Existing suites: 1 trivial helper edit (SiteAddressCacheLoaded internal->public + new dict arg)."
},
{
"id": "T1B.4",
"phase": "1B",
"subject": "Tests: dispatcher routing x28, actor envelope/reply plumbing, TestServer service tests, existing suites green",
"status": "pending",
"status": "completed",
"activeForm": "Testing the site command plane",
"blockedBy": [
"T1B.2",
"T1B.3"
]
],
"notes": "Tests shipped inside T1B.1/1B.2/1B.3 per the plan folding T1B.4 into each task. Dispatcher routing x28, actor envelope/reply plumbing, TestServer service tests, deadline theory, failover/failback, existing Communication suites green with Akka default."
},
{
"id": "P1B.DoD",
@@ -0,0 +1,185 @@
using Akka.Actor;
using Akka.Cluster.Tools.Client;
using Akka.Event;
using ZB.MOM.WW.ScadaBridge.Commons.Messages.Audit;
using ZB.MOM.WW.ScadaBridge.Commons.Messages.Deployment;
using ZB.MOM.WW.ScadaBridge.Commons.Messages.Health;
using ZB.MOM.WW.ScadaBridge.Commons.Messages.Notification;
namespace ZB.MOM.WW.ScadaBridge.Communication.Actors;
/// <summary>
/// The <see cref="ICentralTransport"/> that carries the seven site→central sends over Akka
/// <c>ClusterClient</c> — the transport in production today, and the default. Every method is a
/// verbatim lift of the corresponding <c>SiteCommunicationActor</c> send block: it forwards a
/// <see cref="ClusterClient.Send"/> to <c>/user/central-communication</c> with the
/// <paramref name="replyTo"/> as the send's sender, so central's reply routes straight back to the
/// waiting Ask rather than through the site communication actor.
/// </summary>
/// <remarks>
/// The <c>ClusterClient</c> reference arrives after construction via
/// <see cref="SetCentralClient"/> (the actor forwards the <c>RegisterCentralClient</c> message it
/// receives once the Host builds the client). Until then — and if central contact points are not
/// configured at all — the client is null and each method answers the same transient-failure reply
/// the old inline handlers did.
/// </remarks>
public sealed class AkkaCentralTransport : ICentralTransport
{
/// <summary>The receptionist-registered path of the central communication actor.</summary>
private const string CentralPath = "/user/central-communication";
private readonly ILoggingAdapter? _log;
private IActorRef? _centralClient;
/// <summary>Creates the transport with no logging adapter (behaviourally identical; warnings are dropped).</summary>
public AkkaCentralTransport()
{
}
/// <summary>Creates the transport bound to the site communication actor's logging adapter.</summary>
/// <param name="log">Logging adapter used for the "no ClusterClient registered" warnings.</param>
public AkkaCentralTransport(ILoggingAdapter log)
{
_log = log;
}
/// <summary>
/// Registers the central <c>ClusterClient</c> once the Host has built it. Called from the site
/// communication actor's <c>RegisterCentralClient</c> handler.
/// </summary>
/// <param name="centralClient">The ClusterClient reaching the central cluster.</param>
public void SetCentralClient(IActorRef centralClient)
{
_centralClient = centralClient;
_log?.Info("Registered central ClusterClient");
}
/// <inheritdoc />
public void SubmitNotification(NotificationSubmit message, IActorRef replyTo)
{
if (_centralClient == null)
{
// No ClusterClient registered yet (e.g. central contact points not
// configured, or registration not yet completed). A non-accepted ack
// makes the S&F forwarder treat this as transient and retry later.
_log?.Warning(
"Cannot forward NotificationSubmit {0} — no central ClusterClient registered",
message.NotificationId);
replyTo.Tell(new NotificationSubmitAck(
message.NotificationId, Accepted: false, Error: "Central ClusterClient not registered"));
return;
}
_log?.Debug("Forwarding NotificationSubmit {0} to central", message.NotificationId);
_centralClient.Tell(new ClusterClient.Send(CentralPath, message), replyTo);
}
/// <inheritdoc />
public void QueryNotificationStatus(NotificationStatusQuery message, IActorRef replyTo)
{
if (_centralClient == null)
{
// No ClusterClient registered yet. Reply Found: false so Notify.Status
// falls back to the site S&F buffer to decide Forwarding vs Unknown.
_log?.Warning(
"Cannot forward NotificationStatusQuery {0} — no central ClusterClient registered",
message.NotificationId);
replyTo.Tell(new NotificationStatusResponse(
message.CorrelationId, Found: false, Status: "Unknown",
RetryCount: 0, LastError: null, DeliveredAt: null));
return;
}
_log?.Debug("Forwarding NotificationStatusQuery {0} to central", message.NotificationId);
_centralClient.Tell(new ClusterClient.Send(CentralPath, message), replyTo);
}
/// <inheritdoc />
public void IngestAuditEvents(IngestAuditEventsCommand message, IActorRef replyTo)
{
if (_centralClient == null)
{
// No ClusterClient registered yet. Faulting the Ask makes the
// SiteAuditTelemetryActor drain loop treat this as transient and keep
// the rows Pending for the next tick.
_log?.Warning(
"Cannot forward IngestAuditEventsCommand ({0} events) — no central ClusterClient registered",
message.Events.Count);
replyTo.Tell(new Status.Failure(
new InvalidOperationException("Central ClusterClient not registered")));
return;
}
_log?.Debug("Forwarding IngestAuditEventsCommand ({0} events) to central", message.Events.Count);
_centralClient.Tell(new ClusterClient.Send(CentralPath, message), replyTo);
}
/// <inheritdoc />
public void IngestCachedTelemetry(IngestCachedTelemetryCommand message, IActorRef replyTo)
{
if (_centralClient == null)
{
_log?.Warning(
"Cannot forward IngestCachedTelemetryCommand ({0} entries) — no central ClusterClient registered",
message.Entries.Count);
replyTo.Tell(new Status.Failure(
new InvalidOperationException("Central ClusterClient not registered")));
return;
}
_log?.Debug("Forwarding IngestCachedTelemetryCommand ({0} entries) to central", message.Entries.Count);
_centralClient.Tell(new ClusterClient.Send(CentralPath, message), replyTo);
}
/// <inheritdoc />
public void ReconcileSite(ReconcileSiteRequest message, IActorRef replyTo)
{
if (_centralClient == null)
{
// No ClusterClient registered yet. Faulting the Ask makes the
// SiteReconciliationActor treat the pass as best-effort-failed; it
// logs a warning and retries reconcile on the next node startup.
_log?.Warning(
"Cannot forward ReconcileSiteRequest for site {0} node {1} — no central ClusterClient registered",
message.SiteIdentifier, message.NodeId);
replyTo.Tell(new Status.Failure(
new InvalidOperationException("Central ClusterClient not registered")));
return;
}
_log?.Debug(
"Forwarding ReconcileSiteRequest for site {0} node {1} ({2} local instance(s)) to central",
message.SiteIdentifier, message.NodeId, message.LocalNameToRevisionHash.Count);
_centralClient.Tell(new ClusterClient.Send(CentralPath, message), replyTo);
}
/// <inheritdoc />
public void ReportSiteHealth(SiteHealthReport message, IActorRef replyTo)
{
if (_centralClient == null)
{
// No ClusterClient registered yet. A non-accepted ack makes the
// sender's counter-restore path treat this tick as a loss.
_log?.Warning(
"Cannot forward SiteHealthReport #{0} — no central ClusterClient registered",
message.SequenceNumber);
replyTo.Tell(new SiteHealthReportAck(
message.SiteId, message.SequenceNumber, Accepted: false,
Error: "Central ClusterClient not registered"));
return;
}
_centralClient.Tell(new ClusterClient.Send(CentralPath, message), replyTo);
}
/// <inheritdoc />
public void SendHeartbeat(HeartbeatMessage message, IActorRef self)
{
if (_centralClient == null)
{
return;
}
_centralClient.Tell(new ClusterClient.Send(CentralPath, message), self);
}
}
@@ -0,0 +1,141 @@
using System.Collections.Immutable;
using Akka.Actor;
using Akka.Cluster.Tools.Client;
using Akka.Event;
namespace ZB.MOM.WW.ScadaBridge.Communication.Actors;
/// <summary>
/// The default (and, until the Phase 1B cutover, the shipping) central→site transport: routes each
/// <see cref="SiteEnvelope"/> through a per-site Akka <see cref="ClusterClient"/>, exactly as
/// <c>CentralCommunicationActor</c> did inline before the seam was extracted.
/// </summary>
/// <remarks>
/// <para>
/// Behaviour is identical to the pre-seam code: the per-site client map is (re)built from the DB
/// refresh cache; a send to a site with no client is warned and dropped (the caller's Ask times
/// out — central never buffers); a send preserves the reply-to sender so the site's reply routes
/// straight back to the waiting Ask (or the debug-bridge actor).
/// </para>
/// <para>
/// Both members run on the owning actor's thread, so the client map needs no synchronisation and
/// the stored <see cref="IActorContext"/> (used for <c>Context.Stop</c> and <c>Context.System</c>)
/// is only ever touched there.
/// </para>
/// </remarks>
public sealed class AkkaSiteTransport : ISiteCommandTransport
{
private readonly ISiteClientFactory _siteClientFactory;
private readonly IActorContext _context;
private readonly ILoggingAdapter _log;
/// <summary>
/// Per-site ClusterClient instances and their contact addresses.
/// Maps SiteIdentifier → (ClusterClient actor, set of contact address strings).
/// Refreshed by <see cref="ReconcileSites"/>.
/// </summary>
private readonly Dictionary<string, (IActorRef Client, ImmutableHashSet<string> ContactAddresses)> _siteClients = new();
/// <summary>Creates the Akka transport bound to the owning actor's context.</summary>
/// <param name="siteClientFactory">Factory that creates a ClusterClient per site.</param>
/// <param name="context">The owning actor's context (for <c>Stop</c>/<c>System</c>); calls stay on the actor thread.</param>
/// <param name="log">The owning actor's logger, so warnings keep the actor's log source.</param>
public AkkaSiteTransport(ISiteClientFactory siteClientFactory, IActorContext context, ILoggingAdapter log)
{
_siteClientFactory = siteClientFactory ?? throw new ArgumentNullException(nameof(siteClientFactory));
_context = context ?? throw new ArgumentNullException(nameof(context));
_log = log ?? throw new ArgumentNullException(nameof(log));
}
/// <inheritdoc />
public void Send(SiteEnvelope envelope, IActorRef replyTo)
{
ArgumentNullException.ThrowIfNull(envelope);
if (!_siteClients.TryGetValue(envelope.SiteId, out var entry))
{
_log.Warning("No ClusterClient for site {0}, cannot route message {1}",
envelope.SiteId, envelope.Message.GetType().Name);
// The Ask will timeout on the caller side — no central buffering
return;
}
// Route via ClusterClient — replyTo is preserved for Ask response routing
entry.Client.Tell(
new ClusterClient.Send("/user/site-communication", envelope.Message),
replyTo);
}
/// <inheritdoc />
public void ReconcileSites(SiteAddressCacheLoaded cache)
{
ArgumentNullException.ThrowIfNull(cache);
var newSiteIds = cache.SiteContacts.Keys.ToHashSet();
var existingSiteIds = _siteClients.Keys.ToHashSet();
// Stop ClusterClients for removed sites
foreach (var removed in existingSiteIds.Except(newSiteIds))
{
_log.Info("Stopping ClusterClient for removed site {0}", removed);
_context.Stop(_siteClients[removed].Client);
_siteClients.Remove(removed);
}
// Add or update
foreach (var (siteId, addresses) in cache.SiteContacts)
{
// Parse all addresses up front inside a try/catch so a
// single malformed site row cannot abort the whole refresh loop and leave
// the cache half-updated. A bad site is logged and skipped; others proceed.
ImmutableHashSet<ActorPath> contactPaths;
try
{
contactPaths = addresses
.Select(a => ActorPath.Parse($"{a}/system/receptionist"))
.ToImmutableHashSet();
}
catch (Exception ex)
{
_log.Warning(ex,
"Malformed contact address for site {0}; skipping this site in the refresh "
+ "(other sites are unaffected)", siteId);
continue;
}
var contactStrings = addresses.ToImmutableHashSet();
// Skip if unchanged
if (_siteClients.TryGetValue(siteId, out var existing) && existing.ContactAddresses.SetEquals(contactStrings))
continue;
// Stop old client if addresses changed
if (_siteClients.ContainsKey(siteId))
{
_log.Info("Updating ClusterClient for site {0} (addresses changed)", siteId);
_context.Stop(_siteClients[siteId].Client);
// Remove now: if the replacement create below fails, a stale entry
// would route envelopes to a stopping actor.
_siteClients.Remove(siteId);
}
IActorRef client;
try
{
client = _siteClientFactory.Create(_context.System, siteId, contactPaths);
}
catch (Exception ex)
{
_log.Error(ex,
"Failed to create ClusterClient for site {0}; site is unroutable until the next refresh",
siteId);
continue;
}
_siteClients[siteId] = (client, contactStrings);
_log.Info("Created ClusterClient for site {0} with {1} contact(s)", siteId, addresses.Count);
}
_log.Info("Site ClusterClient cache refreshed with {0} site(s)", _siteClients.Count);
}
}
@@ -4,6 +4,8 @@ using Akka.Cluster.Tools.Client;
using Akka.Cluster.Tools.PublishSubscribe;
using Akka.Event;
using Microsoft.Extensions.DependencyInjection;
using Microsoft.Extensions.Logging;
using Microsoft.Extensions.Options;
using ZB.MOM.WW.ScadaBridge.Commons.Interfaces.Repositories;
using ZB.MOM.WW.ScadaBridge.Commons.Messages.Audit;
using ZB.MOM.WW.ScadaBridge.Commons.Messages.Communication;
@@ -78,14 +80,15 @@ public class CentralCommunicationActor : ReceiveActor
{
private readonly ILoggingAdapter _log = Context.GetLogger();
private readonly IServiceProvider _serviceProvider;
private readonly ISiteClientFactory _siteClientFactory;
/// <summary>
/// Per-site ClusterClient instances and their contact addresses.
/// Maps SiteIdentifier → (ClusterClient actor, set of contact address strings).
/// Refreshed periodically via RefreshSiteAddresses.
/// The active central→site command transport, chosen by
/// <c>ScadaBridge:Communication:SiteTransport</c> (default <see cref="SiteTransportKind.Akka"/>).
/// The <see cref="SiteEnvelope"/> handler delegates every send here, and each DB refresh tick
/// reconciles its per-site resources (ClusterClients for Akka, channel pairs for gRPC). Assigned
/// in the public constructor body before any message can arrive.
/// </summary>
private readonly Dictionary<string, (IActorRef Client, ImmutableHashSet<string> ContactAddresses)> _siteClients = new();
private ISiteCommandTransport _transport = null!;
// The previous _debugSubscriptions / _inProgressDeployments
// dictionaries existed solely to support a documented "synchronous kill streams +
@@ -156,9 +159,15 @@ public class CentralCommunicationActor : ReceiveActor
/// </summary>
private const string HealthReportTopic = "site-health-replica";
/// <summary>Initializes the <see cref="CentralCommunicationActor"/> and wires all message handlers.</summary>
/// <summary>
/// Legacy constructor: builds the transport by reading
/// <c>ScadaBridge:Communication:SiteTransport</c> and wrapping <paramref name="siteClientFactory"/>
/// in an <see cref="AkkaSiteTransport"/> (default) or resolving the gRPC transport from
/// <paramref name="serviceProvider"/>. Kept so the Host and the existing TestKit suites construct
/// the actor exactly as before (the factory is still the Akka seam).
/// </summary>
/// <param name="serviceProvider">DI service provider for scoped repository and aggregator access.</param>
/// <param name="siteClientFactory">Factory used to create per-site ClusterClient actors.</param>
/// <param name="siteClientFactory">Factory used to create per-site ClusterClient actors (Akka transport).</param>
/// <param name="auditIngestAskTimeout">
/// Optional override for the audit-ingest Ask timeout; defaults to
/// <see cref="Grpc.SiteStreamGrpcServer.AuditIngestAskTimeout"/> (30 s). Exists only so tests can
@@ -168,9 +177,38 @@ public class CentralCommunicationActor : ReceiveActor
IServiceProvider serviceProvider,
ISiteClientFactory siteClientFactory,
TimeSpan? auditIngestAskTimeout = null)
: this(serviceProvider, auditIngestAskTimeout)
{
ArgumentNullException.ThrowIfNull(siteClientFactory);
_transport = SelectTransport(serviceProvider, siteClientFactory);
}
/// <summary>
/// Primary constructor: takes the already-selected <see cref="ISiteCommandTransport"/> directly.
/// Used by tests that substitute the transport, and reachable via the legacy constructor.
/// </summary>
/// <param name="serviceProvider">DI service provider for scoped repository and aggregator access.</param>
/// <param name="transport">The central→site command transport to route every <see cref="SiteEnvelope"/> through.</param>
/// <param name="auditIngestAskTimeout">Optional override for the audit-ingest Ask timeout (test hook).</param>
public CentralCommunicationActor(
IServiceProvider serviceProvider,
ISiteCommandTransport transport,
TimeSpan? auditIngestAskTimeout = null)
: this(serviceProvider, auditIngestAskTimeout)
{
_transport = transport ?? throw new ArgumentNullException(nameof(transport));
}
/// <summary>Shared wiring: sets scoped state and registers every message handler. The
/// <see cref="_transport"/> is assigned by the delegating public constructor before any message
/// is dispatched.</summary>
/// <param name="serviceProvider">DI service provider.</param>
/// <param name="auditIngestAskTimeout">Optional audit-ingest Ask timeout override.</param>
private CentralCommunicationActor(
IServiceProvider serviceProvider,
TimeSpan? auditIngestAskTimeout)
{
_serviceProvider = serviceProvider;
_siteClientFactory = siteClientFactory;
_auditIngestAskTimeout = auditIngestAskTimeout ?? Grpc.SiteStreamGrpcServer.AuditIngestAskTimeout;
// Site address cache loaded from database
@@ -451,19 +489,41 @@ public class CentralCommunicationActor : ReceiveActor
private void HandleSiteEnvelope(SiteEnvelope envelope)
{
if (!_siteClients.TryGetValue(envelope.SiteId, out var entry))
{
_log.Warning("No ClusterClient for site {0}, cannot route message {1}",
envelope.SiteId, envelope.Message.GetType().Name);
// The Ask will timeout on the caller side — no central buffering
return;
// Below-the-seam routing: the active transport (Akka ClusterClient or gRPC) owns the
// "no route for this site ⇒ warn + drop, caller's Ask times out" contract. Sender is the
// temporary Ask actor (or the debug-bridge actor) and is preserved for reply routing.
_transport.Send(envelope, Sender);
}
// Route via ClusterClient — Sender is preserved for Ask response routing
entry.Client.Tell(
new ClusterClient.Send("/user/site-communication", envelope.Message),
Sender);
/// <summary>
/// Chooses the transport from <c>ScadaBridge:Communication:SiteTransport</c> (default
/// <see cref="SiteTransportKind.Akka"/>). For Akka, wraps the injected
/// <see cref="ISiteClientFactory"/> in an <see cref="AkkaSiteTransport"/> bound to this actor's
/// context; for gRPC, resolves the shared <see cref="Grpc.SitePairChannelProvider"/> and options.
/// Runs in the constructor body, where <see cref="ActorBase.Context"/> is available.
/// </summary>
/// <param name="serviceProvider">The DI provider carrying options and (for gRPC) the channel provider.</param>
/// <param name="siteClientFactory">The ClusterClient factory used by the Akka transport.</param>
/// <returns>The selected transport.</returns>
private ISiteCommandTransport SelectTransport(
IServiceProvider serviceProvider, ISiteClientFactory siteClientFactory)
{
var options = serviceProvider.GetService<IOptions<CommunicationOptions>>()?.Value;
var kind = options?.SiteTransport ?? SiteTransportKind.Akka;
if (kind == SiteTransportKind.Grpc)
{
var channelProvider = serviceProvider.GetRequiredService<Grpc.SitePairChannelProvider>();
var loggerFactory = serviceProvider.GetRequiredService<ILoggerFactory>();
_log.Info("central→site command transport: gRPC (SiteCommandService)");
return new Grpc.GrpcSiteTransport(
channelProvider,
options!,
loggerFactory.CreateLogger<Grpc.GrpcSiteTransport>());
}
_log.Info("central→site command transport: Akka ClusterClient");
return new AkkaSiteTransport(siteClientFactory, Context, _log);
}
private void LoadSiteAddressesFromDb()
@@ -491,6 +551,10 @@ public class CentralCommunicationActor : ReceiveActor
var sites = await repo.GetAllSitesAsync(ct).ConfigureAwait(false);
var contacts = new Dictionary<string, List<string>>();
// Parallel gRPC-endpoint cache fed by the SAME DB read (the streaming path's
// GrpcNodeA/BAddress columns, NOT the Akka NodeA/BAddress ones). No second poll loop —
// the gRPC transport rides this one, and the Akka transport simply ignores the field.
var grpcContacts = new Dictionary<string, SiteGrpcEndpoints>();
foreach (var site in sites)
{
var addrs = new List<string>();
@@ -511,6 +575,11 @@ public class CentralCommunicationActor : ReceiveActor
}
if (addrs.Count > 0)
contacts[site.SiteIdentifier] = addrs;
var grpcA = string.IsNullOrWhiteSpace(site.GrpcNodeAAddress) ? null : site.GrpcNodeAAddress;
var grpcB = string.IsNullOrWhiteSpace(site.GrpcNodeBAddress) ? null : site.GrpcNodeBAddress;
if (grpcA is not null || grpcB is not null)
grpcContacts[site.SiteIdentifier] = new SiteGrpcEndpoints(grpcA, grpcB);
}
// Freeze the cross-task payload before piping to
@@ -525,82 +594,20 @@ public class CentralCommunicationActor : ReceiveActor
// address-bearing subset in `frozen`) so the aggregator prunes only
// genuinely-deleted sites and never an addressless-but-configured one.
var knownSiteIds = sites.Select(s => s.SiteIdentifier).ToList();
return new SiteAddressCacheLoaded(frozen, knownSiteIds);
return new SiteAddressCacheLoaded(frozen, knownSiteIds, grpcContacts);
}).PipeTo(self);
}
private void HandleSiteAddressCacheLoaded(SiteAddressCacheLoaded msg)
{
var newSiteIds = msg.SiteContacts.Keys.ToHashSet();
var existingSiteIds = _siteClients.Keys.ToHashSet();
// Stop ClusterClients for removed sites
foreach (var removed in existingSiteIds.Except(newSiteIds))
{
_log.Info("Stopping ClusterClient for removed site {0}", removed);
Context.Stop(_siteClients[removed].Client);
_siteClients.Remove(removed);
}
// Add or update
foreach (var (siteId, addresses) in msg.SiteContacts)
{
// Parse all addresses up front inside a try/catch so a
// single malformed site row cannot abort the whole refresh loop and leave
// the cache half-updated. A bad site is logged and skipped; others proceed.
ImmutableHashSet<ActorPath> contactPaths;
try
{
contactPaths = addresses
.Select(a => ActorPath.Parse($"{a}/system/receptionist"))
.ToImmutableHashSet();
}
catch (Exception ex)
{
_log.Warning(ex,
"Malformed contact address for site {0}; skipping this site in the refresh "
+ "(other sites are unaffected)", siteId);
continue;
}
var contactStrings = addresses.ToImmutableHashSet();
// Skip if unchanged
if (_siteClients.TryGetValue(siteId, out var existing) && existing.ContactAddresses.SetEquals(contactStrings))
continue;
// Stop old client if addresses changed
if (_siteClients.ContainsKey(siteId))
{
_log.Info("Updating ClusterClient for site {0} (addresses changed)", siteId);
Context.Stop(_siteClients[siteId].Client);
// Remove now: if the replacement create below fails, a stale entry
// would route envelopes to a stopping actor.
_siteClients.Remove(siteId);
}
IActorRef client;
try
{
client = _siteClientFactory.Create(Context.System, siteId, contactPaths);
}
catch (Exception ex)
{
_log.Error(ex,
"Failed to create ClusterClient for site {0}; site is unroutable until the next refresh",
siteId);
continue;
}
_siteClients[siteId] = (client, contactStrings);
_log.Info("Created ClusterClient for site {0} with {1} contact(s)", siteId, addresses.Count);
}
_log.Info("Site ClusterClient cache refreshed with {0} site(s)", _siteClients.Count);
// Per-transport per-site resource reconciliation (create/stop ClusterClients, or
// build/drop gRPC channel pairs). Runs on the actor thread each refresh tick.
_transport.ReconcileSites(msg);
// Self-healing eviction: a site deleted from configuration would otherwise
// linger in the aggregator as a permanently-offline tile (and live KPI
// sample source) forever. Prune on every refresh so it disappears within
// one refresh interval without needing a dedicated deletion event.
// one refresh interval without needing a dedicated deletion event. Transport-agnostic.
_serviceProvider.GetService<ICentralHealthAggregator>()?.PruneUnknownSites(msg.KnownSiteIds);
}
@@ -673,8 +680,9 @@ public class CentralCommunicationActor : ReceiveActor
public record RefreshSiteAddresses;
/// <summary>
/// Internal message carrying the loaded site contact data from the database.
/// ClusterClient creation happens on the actor thread in HandleSiteAddressCacheLoaded.
/// Message carrying the loaded site contact data from the database. Per-transport resource
/// reconciliation happens on the actor thread in HandleSiteAddressCacheLoaded, which hands this
/// straight to <see cref="ISiteCommandTransport.ReconcileSites"/>.
///
/// The payload is exposed as <see cref="IReadOnlyDictionary{TKey,TValue}"/>
/// of <see cref="IReadOnlyList{T}"/> so the Akka.NET "messages are immutable"
@@ -682,9 +690,24 @@ public record RefreshSiteAddresses;
/// discipline. The producer wraps the constructed buckets with
/// <c>List&lt;T&gt;.AsReadOnly()</c> before piping to Self.
/// </summary>
internal record SiteAddressCacheLoaded(
/// <param name="SiteContacts">Akka ClusterClient contact addresses per site (from NodeA/NodeBAddress).</param>
/// <param name="KnownSiteIds">Every configured site id, address-bearing or not, for aggregator pruning.</param>
/// <param name="GrpcContacts">
/// gRPC endpoint pairs per site (from GrpcNodeA/GrpcNodeBAddress) — the streaming path's columns,
/// consumed by the gRPC transport and ignored by the Akka one.
/// </param>
public sealed record SiteAddressCacheLoaded(
IReadOnlyDictionary<string, IReadOnlyList<string>> SiteContacts,
IReadOnlyCollection<string> KnownSiteIds);
IReadOnlyCollection<string> KnownSiteIds,
IReadOnlyDictionary<string, SiteGrpcEndpoints> GrpcContacts);
/// <summary>
/// A site's gRPC node-pair endpoints, as loaded from <c>Site.GrpcNodeAAddress</c>/
/// <c>GrpcNodeBAddress</c>. Either may be null when only one node has a gRPC address configured.
/// </summary>
/// <param name="NodeA">NodeA gRPC base address (e.g. <c>http://scadabridge-site-a-node-a:8083</c>), or null.</param>
/// <param name="NodeB">NodeB gRPC base address, or null.</param>
public readonly record struct SiteGrpcEndpoints(string? NodeA, string? NodeB);
/// <summary>
/// Peer-replication envelope for a site heartbeat, fanned out over the same
@@ -0,0 +1,78 @@
using Akka.Actor;
using ZB.MOM.WW.ScadaBridge.Commons.Messages.Audit;
using ZB.MOM.WW.ScadaBridge.Commons.Messages.Deployment;
using ZB.MOM.WW.ScadaBridge.Commons.Messages.Health;
using ZB.MOM.WW.ScadaBridge.Commons.Messages.Notification;
namespace ZB.MOM.WW.ScadaBridge.Communication.Actors;
/// <summary>
/// The site→central transport seam: one method per the seven messages
/// <see cref="SiteCommunicationActor"/> sends to <c>/user/central-communication</c> today.
/// </summary>
/// <remarks>
/// <para>
/// The actor's receive handlers no longer own the wire plumbing — they capture the current
/// <c>Sender</c> and hand it to the transport as <paramref name="replyTo"/>. Two implementations
/// exist behind the <c>ScadaBridge:Communication:CentralTransport</c> flag: the default
/// <see cref="AkkaCentralTransport"/> (verbatim of the old <c>ClusterClient.Send</c> path,
/// including the exact sender-forwarding that routes central's reply straight back to the waiting
/// Ask) and <see cref="Grpc.GrpcCentralTransport"/> (a gRPC dial of <c>CentralControlService</c>).
/// </para>
/// <para>
/// <b>Reply/fault contract, identical on both transports.</b> Each Ask-returning method (all but
/// the heartbeat) guarantees exactly one reply eventually lands at <paramref name="replyTo"/>:
/// either the real reply type the caller Asks for, or a transient-failure signal. The Akka path
/// sends a not-accepted ack / <see cref="Status.Failure"/> when no ClusterClient is registered;
/// the gRPC path sends <see cref="Status.Failure"/> on any non-OK status (a timeout or an
/// <c>Unavailable</c> that could not be failed over). Both are what the S&amp;F / audit / health
/// layers above the seam already treat as transient — rows stay buffered, counters restore, the
/// pass re-runs.
/// </para>
/// <para>
/// <b>The heartbeat stays fire-and-forget end-to-end.</b> <see cref="SendHeartbeat"/> takes no
/// <c>replyTo</c> and never surfaces a fault: a transport failure is swallowed and logged, exactly
/// as the old <c>Tell</c> dropped it. A failing heartbeat must never fault the site's heartbeat
/// timer path.
/// </para>
/// </remarks>
public interface ICentralTransport
{
/// <summary>Forwards a buffered notification; central replies <see cref="NotificationSubmitAck"/> to <paramref name="replyTo"/>.</summary>
/// <param name="message">The notification submission.</param>
/// <param name="replyTo">The actor (the S&amp;F forwarder's Ask) the ack routes back to.</param>
void SubmitNotification(NotificationSubmit message, IActorRef replyTo);
/// <summary>Forwards a Notify.Status query; central replies <see cref="NotificationStatusResponse"/> to <paramref name="replyTo"/>.</summary>
/// <param name="message">The status query.</param>
/// <param name="replyTo">The actor (the Notify helper's Ask) the response routes back to.</param>
void QueryNotificationStatus(NotificationStatusQuery message, IActorRef replyTo);
/// <summary>Pushes a batch of audit events; central replies <see cref="IngestAuditEventsReply"/> to <paramref name="replyTo"/>.</summary>
/// <param name="message">The audit-event ingest command.</param>
/// <param name="replyTo">The actor (the telemetry drain's Ask) the reply routes back to.</param>
void IngestAuditEvents(IngestAuditEventsCommand message, IActorRef replyTo);
/// <summary>Pushes a batch of combined cached-call telemetry; central replies <see cref="IngestCachedTelemetryReply"/> to <paramref name="replyTo"/>.</summary>
/// <param name="message">The cached-telemetry ingest command.</param>
/// <param name="replyTo">The actor (the telemetry drain's Ask) the reply routes back to.</param>
void IngestCachedTelemetry(IngestCachedTelemetryCommand message, IActorRef replyTo);
/// <summary>Reports a node's startup inventory; central replies <see cref="ReconcileSiteResponse"/> to <paramref name="replyTo"/>.</summary>
/// <param name="message">The reconcile request.</param>
/// <param name="replyTo">The actor (the reconciliation Ask) the response routes back to.</param>
void ReconcileSite(ReconcileSiteRequest message, IActorRef replyTo);
/// <summary>Reports periodic site health; central replies <see cref="SiteHealthReportAck"/> to <paramref name="replyTo"/>.</summary>
/// <param name="message">The health report.</param>
/// <param name="replyTo">The actor (the health transport's Ask) the ack routes back to.</param>
void ReportSiteHealth(SiteHealthReport message, IActorRef replyTo);
/// <summary>
/// Sends an application heartbeat, fire-and-forget. Never replies and never faults; a failure
/// is swallowed and logged.
/// </summary>
/// <param name="message">The heartbeat.</param>
/// <param name="self">The site communication actor, used as the sender on the Akka path (ignored on gRPC).</param>
void SendHeartbeat(HeartbeatMessage message, IActorRef self);
}
@@ -0,0 +1,43 @@
using Akka.Actor;
namespace ZB.MOM.WW.ScadaBridge.Communication.Actors;
/// <summary>
/// The central→site command-send seam, injected into <see cref="CentralCommunicationActor"/>
/// below the <see cref="SiteEnvelope"/> handler. Exactly one implementation is active per node,
/// chosen by <c>ScadaBridge:Communication:SiteTransport</c>:
/// <list type="bullet">
/// <item><see cref="AkkaSiteTransport"/> — today's per-site <c>ClusterClient</c> path (default).</item>
/// <item><see cref="ZB.MOM.WW.ScadaBridge.Communication.Grpc.GrpcSiteTransport"/> — the site
/// <c>SiteCommandService</c> gRPC plane.</item>
/// </list>
/// The producers above the seam (<c>CommunicationService</c>'s 27 commands, <c>SiteCallAuditActor</c>'s
/// 2 parked relays, and <c>DebugStreamBridgeActor</c>'s subscribe/unsubscribe) are unchanged — they
/// still <c>Ask</c>/<c>Tell</c> a <see cref="SiteEnvelope"/> to the actor, which delegates here.
/// </summary>
/// <remarks>
/// Both members run on the actor thread (from the <see cref="SiteEnvelope"/> and
/// <c>SiteAddressCacheLoaded</c> handlers), so implementations need no internal synchronisation for
/// their own per-site bookkeeping beyond what a background failback loop requires.
/// </remarks>
public interface ISiteCommandTransport
{
/// <summary>
/// Routes <paramref name="envelope"/>'s message to its site. Any reply the site produces is
/// delivered to <paramref name="replyTo"/> — for an <c>Ask</c> that is the temporary ask actor
/// (completing the caller's task); for a <c>Tell</c>-with-sender (the debug bridge) that is the
/// originating actor. A message with no route (an unknown site) is warned and dropped so the
/// caller's <c>Ask</c> times out, exactly as today's ClusterClient path behaves.
/// </summary>
/// <param name="envelope">The site-addressed command envelope.</param>
/// <param name="replyTo">Where a reply (or a <see cref="Status.Failure"/>) is delivered.</param>
void Send(SiteEnvelope envelope, IActorRef replyTo);
/// <summary>
/// Reconciles per-site transport resources (ClusterClients for Akka, channel pairs for gRPC)
/// against the freshly loaded site set. Called once per DB refresh tick with the same cache
/// message the actor already receives — the ONE DB-poll loop feeds both transports.
/// </summary>
/// <param name="cache">The loaded site address cache (Akka contacts + gRPC endpoints + known ids).</param>
void ReconcileSites(SiteAddressCacheLoaded cache);
}
@@ -0,0 +1,303 @@
using Akka.Actor;
using ZB.MOM.WW.ScadaBridge.Commons.Messages.Artifacts;
using ZB.MOM.WW.ScadaBridge.Commons.Messages.DataConnection;
using ZB.MOM.WW.ScadaBridge.Commons.Messages.DebugView;
using ZB.MOM.WW.ScadaBridge.Commons.Messages.Deployment;
using ZB.MOM.WW.ScadaBridge.Commons.Messages.InboundApi;
using ZB.MOM.WW.ScadaBridge.Commons.Messages.Lifecycle;
using ZB.MOM.WW.ScadaBridge.Commons.Messages.Management;
using ZB.MOM.WW.ScadaBridge.Commons.Messages.RemoteQuery;
using ZB.MOM.WW.ScadaBridge.Communication.Grpc;
namespace ZB.MOM.WW.ScadaBridge.Communication.Actors;
/// <summary>
/// The single routing truth for the 28 migrated central→site commands. Given one
/// of those command records it decides which local target answers it —
/// the Deployment Manager singleton proxy, the artifact/event-log/parked-message
/// handlers, or the node-local failover path — preserving EXACTLY the targets and
/// null-guard semantics <see cref="SiteCommunicationActor"/> used when this routing
/// lived inline.
/// </summary>
/// <remarks>
/// <para>
/// Two transports call this one unit: the Akka <see cref="SiteCommunicationActor"/>
/// (ClusterClient) and the new <c>SiteCommandGrpcService</c> (gRPC). Centralising the
/// table means the two can never drift on where a command goes. Each transport keeps
/// its own send mechanics — the actor <c>Forward</c>s (preserving the Ask sender), the
/// gRPC service <c>Ask</c>s and encodes the reply — but both read the same
/// <see cref="Route"/> here.
/// </para>
/// <para>
/// <b>The parked-message handler stays node-local on purpose.</b> A parked retry or
/// discard must run on the node that holds the replicated store row, so parked
/// commands route to the per-node <c>_parkedMessageHandler</c> — never onto the
/// singleton proxy (design §7.3). This is the same target the actor used; the
/// extraction does not "fix" it.
/// </para>
/// <para>
/// <b>Excluded by design:</b> <c>IntegrationCallRequest</c> — the 29th command, dead at
/// both ends. It never enters this dispatcher; the actor keeps its own vestigial handler
/// for it (28 of 29 migrate).
/// </para>
/// </remarks>
public sealed class SiteCommandDispatcher
{
/// <summary>How a resolved command is delivered to its target.</summary>
public enum RouteDisposition
{
/// <summary>Forward/Ask <see cref="Route.Target"/> and route its reply back.</summary>
Forward,
/// <summary>
/// Tell <see cref="Route.Target"/> and answer immediately with
/// <see cref="Route.Reply"/> — the fire-and-forget path (only
/// <see cref="UnsubscribeDebugViewRequest"/>, which the downstream never acks).
/// The actor <c>Forward</c>s it and sends nothing back; the gRPC transport Tells and
/// returns the synthetic ack so a unary RPC still answers.
/// </summary>
TellFireAndForget,
/// <summary>
/// No target is available (a null-guarded handler is unregistered); answer with the
/// synthetic <see cref="Route.Reply"/> — the "handler not available" reply the actor
/// produced inline.
/// </summary>
ImmediateReply
}
/// <summary>A resolved routing decision for one command.</summary>
/// <param name="Disposition">How the command is delivered.</param>
/// <param name="Target">The target actor for <see cref="RouteDisposition.Forward"/>/<see cref="RouteDisposition.TellFireAndForget"/>; <c>null</c> for an immediate reply.</param>
/// <param name="Reply">The synthetic reply for <see cref="RouteDisposition.ImmediateReply"/>/<see cref="RouteDisposition.TellFireAndForget"/>; <c>null</c> for a forward.</param>
public readonly record struct Route(RouteDisposition Disposition, IActorRef? Target, object? Reply);
/// <summary>The outcome of preparing a failover: the ack to send now, plus the leave to run after.</summary>
/// <param name="Ack">The ack the caller must send back before the node leaves.</param>
/// <param name="CommitLeave">
/// When non-<c>null</c> (accepted only), invoking it performs the real graceful
/// <c>Cluster.Leave</c>. It is deliberately deferred so the transport can flush the
/// <see cref="Ack"/> first — a caller reaching the very node that is about to leave still
/// receives the ack rather than a broken stream.
/// </param>
public readonly record struct FailoverOutcome(SiteFailoverAck Ack, Action? CommitLeave);
private readonly string _siteId;
private readonly IActorRef _deploymentManagerProxy;
// (role, dryRun) -> leaving node address, or null when there is no standby.
// dryRun:true resolves the target WITHOUT leaving; dryRun:false performs the Leave.
private readonly Func<string, bool, string?> _resolveFailover;
// Registered at runtime via the actor's RegisterLocalHandler flow. Written on the actor
// thread, read by both the actor and the gRPC service (a Kestrel thread), so volatile.
private volatile IActorRef? _eventLogHandler;
private volatile IActorRef? _parkedMessageHandler;
private volatile IActorRef? _artifactHandler;
/// <summary>Creates the dispatcher.</summary>
/// <param name="siteId">This site's identifier, stamped into synthetic replies and matched by the failover guard.</param>
/// <param name="deploymentManagerProxy">The local Deployment Manager singleton proxy — the target for all lifecycle/OPC UA/query/route commands.</param>
/// <param name="resolveFailover">
/// Resolves (and, when <c>dryRun</c> is false, performs) the graceful leave of the oldest
/// Up member in a role scope. Injected so tests need no real cluster.
/// </param>
public SiteCommandDispatcher(
string siteId,
IActorRef deploymentManagerProxy,
Func<string, bool, string?> resolveFailover)
{
ArgumentNullException.ThrowIfNull(siteId);
ArgumentNullException.ThrowIfNull(deploymentManagerProxy);
ArgumentNullException.ThrowIfNull(resolveFailover);
_siteId = siteId;
_deploymentManagerProxy = deploymentManagerProxy;
_resolveFailover = resolveFailover;
}
/// <summary>Registers the site event-log query handler (a cluster singleton proxy).</summary>
/// <param name="handler">The event-log handler.</param>
public void RegisterEventLogHandler(IActorRef handler) => _eventLogHandler = handler;
/// <summary>Registers the node-local parked-message handler (the replicated-store owner on this node).</summary>
/// <param name="handler">The parked-message handler.</param>
public void RegisterParkedMessageHandler(IActorRef handler) => _parkedMessageHandler = handler;
/// <summary>Registers the artifact-deployment handler.</summary>
/// <param name="handler">The artifact handler.</param>
public void RegisterArtifactHandler(IActorRef handler) => _artifactHandler = handler;
/// <summary>
/// Resolves how one of the 27 non-failover commands is delivered. <see cref="TriggerSiteFailover"/>
/// is handled separately via <see cref="PrepareFailover"/>/<see cref="HandleFailover"/> because its
/// leave is deferred; passing it here throws.
/// </summary>
/// <param name="command">The command to route.</param>
/// <returns>The routing decision.</returns>
/// <exception cref="ArgumentException">The command is not a migrated site command (or is failover).</exception>
public Route ResolveRoute(object command)
{
ArgumentNullException.ThrowIfNull(command);
return command switch
{
// ── Deployment + instance lifecycle → Deployment Manager singleton proxy ──
RefreshDeploymentCommand => ToProxy(),
EnableInstanceCommand => ToProxy(),
DisableInstanceCommand => ToProxy(),
DeleteInstanceCommand => ToProxy(),
DeploymentStateQueryRequest => ToProxy(),
// ── Artifact deployment → artifact handler (null-guarded) ──
DeployArtifactsCommand c => _artifactHandler is { } h
? Forwarded(h)
: Immediate(new ArtifactDeploymentResponse(
c.DeploymentId, _siteId, false, "Artifact handler not available", DateTimeOffset.UtcNow)),
// ── Interactive OPC UA / MxGateway → Deployment Manager singleton proxy ──
// The singleton always lands on the active node, which owns the live sessions.
BrowseNodeCommand => ToProxy(),
SearchAddressSpaceCommand => ToProxy(),
ReadTagValuesCommand => ToProxy(),
VerifyEndpointCommand => ToProxy(),
TrustServerCertCommand => ToProxy(),
ListServerCertsCommand => ToProxy(),
RemoveServerCertCommand => ToProxy(),
WriteTagRequest => ToProxy(),
// ── Remote queries: event log (null-guarded) + debug view (proxy) ──
EventLogQueryRequest r => _eventLogHandler is { } h
? Forwarded(h)
: Immediate(new EventLogQueryResponse(
r.CorrelationId, _siteId, [], null, false, false,
"Event log handler not available", DateTimeOffset.UtcNow)),
DebugSnapshotRequest => ToProxy(),
SubscribeDebugViewRequest => ToProxy(),
// Fire-and-forget: the Deployment Manager never acks an unsubscribe, so the gRPC
// transport Tells it and returns the synthetic ack; the actor just Forwards.
UnsubscribeDebugViewRequest => new Route(
RouteDisposition.TellFireAndForget, _deploymentManagerProxy, UnsubscribeDebugViewAck.Instance),
// ── Parked store-and-forward actions → node-local parked handler (null-guarded) ──
ParkedMessageQueryRequest r => _parkedMessageHandler is { } h
? Forwarded(h)
: Immediate(new ParkedMessageQueryResponse(
r.CorrelationId, _siteId, [], 0, r.PageNumber, r.PageSize, false,
"Parked message handler not available", DateTimeOffset.UtcNow)),
ParkedMessageRetryRequest r => _parkedMessageHandler is { } h
? Forwarded(h)
: Immediate(new ParkedMessageRetryResponse(
r.CorrelationId, false, "Parked message handler not available")),
ParkedMessageDiscardRequest r => _parkedMessageHandler is { } h
? Forwarded(h)
: Immediate(new ParkedMessageDiscardResponse(
r.CorrelationId, false, "Parked message handler not available")),
RetryParkedOperation r => _parkedMessageHandler is { } h
? Forwarded(h)
: Immediate(new ParkedOperationActionAck(
r.CorrelationId, Applied: false, "Parked message handler not available")),
DiscardParkedOperation r => _parkedMessageHandler is { } h
? Forwarded(h)
: Immediate(new ParkedOperationActionAck(
r.CorrelationId, Applied: false, "Parked message handler not available")),
// ── Inbound-API Route.To() relays → Deployment Manager singleton proxy ──
RouteToCallRequest => ToProxy(),
RouteToGetAttributesRequest => ToProxy(),
RouteToSetAttributesRequest => ToProxy(),
RouteToWaitForAttributeRequest => ToProxy(),
TriggerSiteFailover => throw new ArgumentException(
"TriggerSiteFailover is handled by PrepareFailover/HandleFailover, not ResolveRoute.",
nameof(command)),
_ => throw new ArgumentException(
$"'{command.GetType().Name}' is not a migrated site command.", nameof(command))
};
Route ToProxy() => Forwarded(_deploymentManagerProxy);
}
private static Route Forwarded(IActorRef target) => new(RouteDisposition.Forward, target, null);
private static Route Immediate(object reply) => new(RouteDisposition.ImmediateReply, null, reply);
/// <summary>
/// The actor's failover path: resolve the standby AND issue the leave in one step (today's
/// coupled behaviour over ClusterClient, where <c>Tell</c> merely enqueues the ack so leave
/// order is immaterial), then return the ack.
/// </summary>
/// <param name="msg">The failover command.</param>
/// <returns>The ack to send back.</returns>
public SiteFailoverAck HandleFailover(TriggerSiteFailover msg)
=> FailoverCore(msg, commitLeaveImmediately: true).Ack;
/// <summary>
/// The gRPC failover path: resolve the standby WITHOUT leaving, build the ack, and hand back a
/// deferred <see cref="FailoverOutcome.CommitLeave"/>. The caller must send the ack before
/// invoking the leave — otherwise a caller reaching the leaving node sees a broken stream
/// instead of its ack (the ack-before-Leave rule).
/// </summary>
/// <param name="msg">The failover command.</param>
/// <returns>The ack and the deferred leave (leave is <c>null</c> when refused).</returns>
public FailoverOutcome PrepareFailover(TriggerSiteFailover msg)
=> FailoverCore(msg, commitLeaveImmediately: false);
private FailoverOutcome FailoverCore(TriggerSiteFailover msg, bool commitLeaveImmediately)
{
ArgumentNullException.ThrowIfNull(msg);
// A misrouted command must be refused, not silently acted on — acting would fail over a
// site the operator never selected. Checked before resolving so the resolver is untouched.
if (!string.Equals(msg.SiteId, _siteId, StringComparison.Ordinal))
{
return new FailoverOutcome(
new SiteFailoverAck(
msg.CorrelationId, Accepted: false, TargetAddress: null,
ErrorMessage: $"Command addressed to site '{msg.SiteId}' but this node serves '{_siteId}'."),
CommitLeave: null);
}
// Site singletons are scoped to the site-specific role, so failover must target that role.
var role = $"site-{_siteId}";
try
{
if (commitLeaveImmediately)
{
var target = _resolveFailover(role, false);
if (target is null)
{
return new FailoverOutcome(NoPeerAck(msg), CommitLeave: null);
}
return new FailoverOutcome(
new SiteFailoverAck(msg.CorrelationId, Accepted: true, target, ErrorMessage: null),
CommitLeave: null);
}
var resolved = _resolveFailover(role, true);
if (resolved is null)
{
return new FailoverOutcome(NoPeerAck(msg), CommitLeave: null);
}
return new FailoverOutcome(
new SiteFailoverAck(msg.CorrelationId, Accepted: true, resolved, ErrorMessage: null),
CommitLeave: () => _resolveFailover(role, false));
}
catch (Exception ex)
{
// A fault must be reported to the operator, never thrown into supervision (over
// ClusterClient) or surfaced as a broken stream (over gRPC).
return new FailoverOutcome(
new SiteFailoverAck(
msg.CorrelationId, Accepted: false, TargetAddress: null, ErrorMessage: ex.Message),
CommitLeave: null);
}
}
private static SiteFailoverAck NoPeerAck(TriggerSiteFailover msg) => new(
msg.CorrelationId, Accepted: false, TargetAddress: null,
ErrorMessage: "No standby available — failing over a lone node would be an outage.");
}
@@ -1,6 +1,5 @@
using Akka.Actor;
using Akka.Cluster;
using Akka.Cluster.Tools.Client;
using Akka.Event;
using ZB.MOM.WW.ScadaBridge.Commons.Messages.Artifacts;
using ZB.MOM.WW.ScadaBridge.Commons.Messages.Audit;
@@ -37,7 +36,6 @@ public class SiteCommunicationActor : ReceiveActor, IWithTimers
/// do not need a real cluster.
/// </summary>
private readonly Func<bool> _isActiveCheck;
private readonly Func<string, string?> _failOverRole;
/// <summary>
/// Reference to the local Deployment Manager singleton proxy.
@@ -45,19 +43,28 @@ public class SiteCommunicationActor : ReceiveActor, IWithTimers
private readonly IActorRef _deploymentManagerProxy;
/// <summary>
/// ClusterClient reference for sending messages to the central cluster.
/// Set via RegisterCentralClient message.
/// The single routing truth for the 28 migrated central→site commands, shared with
/// the gRPC <c>SiteCommandGrpcService</c> so the two transports cannot drift. In
/// production it is created by the Host and passed in (so the gRPC server holds the
/// same instance); when unset (tests) the actor builds its own from the failover seam.
/// </summary>
private IActorRef? _centralClient;
private readonly SiteCommandDispatcher _dispatcher;
/// <summary>
/// Local actor references for routing specific message patterns.
/// Populated via registration messages.
/// The site→central transport. Finalized in <see cref="PreStart"/> to the injected instance,
/// or a default <see cref="AkkaCentralTransport"/> (ClusterClient) when none is supplied — so
/// the seven site→central sends delegate here rather than owning the wire plumbing inline.
/// </summary>
private ICentralTransport _transport;
/// <summary>The transport supplied by the Host (null selects the default Akka transport).</summary>
private readonly ICentralTransport? _injectedTransport;
/// <summary>
/// Handler for the vestigial <see cref="IntegrationCallRequest"/> — the one command NOT
/// migrated to the dispatcher (dead at both ends), so it still routes on the actor.
/// </summary>
private IActorRef? _eventLogHandler;
private IActorRef? _parkedMessageHandler;
private IActorRef? _integrationHandler;
private IActorRef? _artifactHandler;
/// <summary>Akka timer scheduler injected by the framework via <see cref="IWithTimers"/>.</summary>
public ITimerScheduler Timers { get; set; } = null!;
@@ -73,58 +80,94 @@ public class SiteCommunicationActor : ReceiveActor, IWithTimers
/// pass a stub so they do not need to load Akka.Cluster into the <c>TestKit</c>
/// ActorSystem.
/// </param>
/// <param name="transport">
/// The site→central transport. <c>null</c> (the default, used by every existing test and by
/// the Host's Akka path) selects an <see cref="AkkaCentralTransport"/> over ClusterClient; the
/// Host injects a <see cref="Grpc.GrpcCentralTransport"/> when
/// <c>ScadaBridge:Communication:CentralTransport</c> is <c>Grpc</c>.
/// </param>
/// <param name="dispatcher">
/// The shared <see cref="SiteCommandDispatcher"/> (production: created by the Host and also
/// handed to the gRPC command service, so both transports route through one instance).
/// <c>null</c> makes the actor build its own from <paramref name="failOverRole"/> — the shape
/// the existing tests use.
/// </param>
public SiteCommunicationActor(
string siteId,
CommunicationOptions options,
IActorRef deploymentManagerProxy,
Func<bool>? isActiveCheck = null,
Func<string, string?>? failOverRole = null)
Func<string, string?>? failOverRole = null,
ICentralTransport? transport = null,
SiteCommandDispatcher? dispatcher = null)
{
_siteId = siteId;
_options = options;
_deploymentManagerProxy = deploymentManagerProxy;
_isActiveCheck = isActiveCheck ?? DefaultIsActiveCheck;
_failOverRole = failOverRole ?? DefaultFailOverRole;
_injectedTransport = transport;
// Finalized in PreStart (where _log is usable for the default transport); assigned here
// too so the field is definitely-assigned for the constructor's Receive closures.
_transport = transport!;
// Registration
// When no shared dispatcher is supplied, build one over the same failover seam the
// actor used before extraction: an injected Func (tests) that both resolves and leaves,
// or the shared ClusterFailoverCoordinator. The actor only ever commits the leave
// immediately (dryRun:false), so an injected coupled Func fits unchanged.
var system = Context.System;
Func<string, bool, string?> resolveFailover = failOverRole is not null
? (role, _) => failOverRole(role)
: (role, dryRun) =>
ClusterState.ClusterFailoverCoordinator.FailOverOldest(system, role, dryRun)?.ToString();
_dispatcher = dispatcher ?? new SiteCommandDispatcher(siteId, deploymentManagerProxy, resolveFailover);
// Registration. Feeding the ClusterClient into the transport is a no-op unless the
// default/Akka transport is in use — the gRPC transport dials configured endpoints and
// never receives this message (the Host does not create a ClusterClient for it).
Receive<RegisterCentralClient>(msg =>
{
_centralClient = msg.Client;
_log.Info("Registered central ClusterClient");
(_transport as AkkaCentralTransport)?.SetCentralClient(msg.Client);
});
Receive<RegisterLocalHandler>(HandleRegisterLocalHandler);
// Pattern 1: Instance Deployment — forward to Deployment Manager
Receive<RefreshDeploymentCommand>(msg =>
{
_log.Debug("Routing RefreshDeploymentCommand for {0} to DeploymentManager", msg.InstanceUniqueName);
_deploymentManagerProxy.Forward(msg);
});
// ── The 27 migrated central→site commands (28th is failover, below) all route
// through the shared SiteCommandDispatcher — the single routing truth also used by
// the gRPC SiteCommandGrpcService. The actor's job per command is unchanged: Forward
// to the resolved target (preserving the central Ask sender so replies route straight
// back to the waiting Ask), or Tell the caller the dispatcher's synthetic reply when a
// null-guarded handler is unregistered. See SiteCommandDispatcher for the target of
// each command and why the parked handler stays node-local.
Receive<RefreshDeploymentCommand>(cmd => DispatchCommand(cmd));
Receive<DisableInstanceCommand>(cmd => DispatchCommand(cmd));
Receive<EnableInstanceCommand>(cmd => DispatchCommand(cmd));
Receive<DeleteInstanceCommand>(cmd => DispatchCommand(cmd));
Receive<DeploymentStateQueryRequest>(cmd => DispatchCommand(cmd));
Receive<DeployArtifactsCommand>(cmd => DispatchCommand(cmd));
Receive<SubscribeDebugViewRequest>(cmd => DispatchCommand(cmd));
Receive<UnsubscribeDebugViewRequest>(cmd => DispatchCommand(cmd));
Receive<DebugSnapshotRequest>(cmd => DispatchCommand(cmd));
Receive<RouteToCallRequest>(cmd => DispatchCommand(cmd));
Receive<RouteToGetAttributesRequest>(cmd => DispatchCommand(cmd));
Receive<RouteToSetAttributesRequest>(cmd => DispatchCommand(cmd));
Receive<RouteToWaitForAttributeRequest>(cmd => DispatchCommand(cmd));
Receive<BrowseNodeCommand>(cmd => DispatchCommand(cmd));
Receive<ReadTagValuesCommand>(cmd => DispatchCommand(cmd));
Receive<SearchAddressSpaceCommand>(cmd => DispatchCommand(cmd));
Receive<Commons.Messages.DataConnection.WriteTagRequest>(cmd => DispatchCommand(cmd));
Receive<VerifyEndpointCommand>(cmd => DispatchCommand(cmd));
Receive<TrustServerCertCommand>(cmd => DispatchCommand(cmd));
Receive<ListServerCertsCommand>(cmd => DispatchCommand(cmd));
Receive<RemoveServerCertCommand>(cmd => DispatchCommand(cmd));
Receive<EventLogQueryRequest>(cmd => DispatchCommand(cmd));
Receive<ParkedMessageQueryRequest>(cmd => DispatchCommand(cmd));
Receive<ParkedMessageRetryRequest>(cmd => DispatchCommand(cmd));
Receive<ParkedMessageDiscardRequest>(cmd => DispatchCommand(cmd));
Receive<RetryParkedOperation>(cmd => DispatchCommand(cmd));
Receive<DiscardParkedOperation>(cmd => DispatchCommand(cmd));
// Pattern 2: Lifecycle — forward to Deployment Manager
Receive<DisableInstanceCommand>(msg => _deploymentManagerProxy.Forward(msg));
Receive<EnableInstanceCommand>(msg => _deploymentManagerProxy.Forward(msg));
Receive<DeleteInstanceCommand>(msg => _deploymentManagerProxy.Forward(msg));
// Query-the-site-before-redeploy — forward to
// the Deployment Manager, which owns the deployed-config store and
// answers with the instance's currently-applied deployment identity.
Receive<DeploymentStateQueryRequest>(msg => _deploymentManagerProxy.Forward(msg));
// Pattern 3: Artifact Deployment — forward to artifact handler if registered
Receive<DeployArtifactsCommand>(msg =>
{
if (_artifactHandler != null)
_artifactHandler.Forward(msg);
else
{
_log.Warning("No artifact handler registered, replying with failure");
Sender.Tell(new ArtifactDeploymentResponse(
msg.DeploymentId, _siteId, false, "Artifact handler not available", DateTimeOffset.UtcNow));
}
});
// Pattern 4: Integration Routing — forward to integration handler
// Integration Routing — the 29th command, NOT migrated to the dispatcher (dead at
// both ends; no production code registers the handler). Kept on the actor so the
// dispatcher's command surface stays the 28 that actually migrate.
Receive<IntegrationCallRequest>(msg =>
{
if (_integrationHandler != null)
@@ -136,295 +179,43 @@ public class SiteCommunicationActor : ReceiveActor, IWithTimers
}
});
// Pattern 5: Debug View — forward to Deployment Manager (which routes to Instance Actor)
Receive<SubscribeDebugViewRequest>(msg => _deploymentManagerProxy.Forward(msg));
Receive<UnsubscribeDebugViewRequest>(msg => _deploymentManagerProxy.Forward(msg));
// Pattern 6a: Debug Snapshot (one-shot) — forward to Deployment Manager
Receive<DebugSnapshotRequest>(msg => _deploymentManagerProxy.Forward(msg));
// Inbound API Route.To() — forward to Deployment Manager for instance routing
Receive<RouteToCallRequest>(msg => _deploymentManagerProxy.Forward(msg));
Receive<RouteToGetAttributesRequest>(msg => _deploymentManagerProxy.Forward(msg));
Receive<RouteToSetAttributesRequest>(msg => _deploymentManagerProxy.Forward(msg));
Receive<RouteToWaitForAttributeRequest>(msg => _deploymentManagerProxy.Forward(msg));
// OPC UA Tag Browser (interactive design-time query) — forward to the
// Deployment Manager singleton, which always lands on the active site
// node. Routing to the site-local /user/dcl-manager directly is wrong
// because the standby node has a dcl-manager too, but its
// DataConnectionActor children (which own the live OPC UA sessions)
// only exist on the singleton's node. The singleton then re-forwards
// to its own /user/dcl-manager, which DOES have the connection.
Receive<BrowseNodeCommand>(msg => _deploymentManagerProxy.Forward(msg));
// Test Bindings (interactive design-time read) — same routing rationale
// as BrowseNodeCommand above: the singleton always lands on the
// active site node, which is the node that owns the DataConnectionActor
// children holding the live OPC UA sessions.
Receive<ReadTagValuesCommand>(msg => _deploymentManagerProxy.Forward(msg));
// OPC UA tag-picker address-space search and secured-write execute
// — same singleton routing rationale as BrowseNodeCommand above: the
// DataConnectionActor children that own the live OPC UA sessions exist only
// on the singleton's (active) node, so these must hop through the Deployment
// Manager proxy too. Without these forwards the commands dead-letter and the
// central Ask times out. Forward preserves the central Ask sender so the
// result routes straight back to the waiting Ask.
Receive<SearchAddressSpaceCommand>(msg => _deploymentManagerProxy.Forward(msg));
Receive<Commons.Messages.DataConnection.WriteTagRequest>(msg => _deploymentManagerProxy.Forward(msg));
// OPC UA endpoint Verify — probes a (possibly unsaved) endpoint config
// WITHOUT persisting it. The Deployment Manager singleton's dcl-manager runs
// the probe directly (no existing connection required), so — like the
// commands above — Verify routes through the singleton's active node.
Receive<VerifyEndpointCommand>(msg => _deploymentManagerProxy.Forward(msg));
// OPC UA server-certificate trust management — forward to the
// Deployment Manager singleton, which owns the cross-node trust broadcast.
// The trusted-peer PKI store is node-wide per site node, so a trust/remove
// decision must reach BOTH nodes' CertStoreActor; the singleton broadcasts
// to every site node (list answers from the singleton's own node). The
// singleton always lands on the active node, the same routing rationale as
// BrowseNodeCommand above. Forward preserves the central Ask sender so the
// CertTrustResult routes straight back to the waiting Ask.
Receive<TrustServerCertCommand>(msg => _deploymentManagerProxy.Forward(msg));
Receive<ListServerCertsCommand>(msg => _deploymentManagerProxy.Forward(msg));
Receive<RemoveServerCertCommand>(msg => _deploymentManagerProxy.Forward(msg));
// Pattern 7: Remote Queries
Receive<EventLogQueryRequest>(msg =>
{
if (_eventLogHandler != null)
_eventLogHandler.Forward(msg);
else
{
Sender.Tell(new EventLogQueryResponse(
msg.CorrelationId, _siteId, [], null, false, false,
"Event log handler not available", DateTimeOffset.UtcNow));
}
});
Receive<ParkedMessageQueryRequest>(msg =>
{
if (_parkedMessageHandler != null)
_parkedMessageHandler.Forward(msg);
else
{
Sender.Tell(new ParkedMessageQueryResponse(
msg.CorrelationId, _siteId, [], 0, msg.PageNumber, msg.PageSize, false,
"Parked message handler not available", DateTimeOffset.UtcNow));
}
});
Receive<ParkedMessageRetryRequest>(msg =>
{
if (_parkedMessageHandler != null)
_parkedMessageHandler.Forward(msg);
else
{
Sender.Tell(new ParkedMessageRetryResponse(
msg.CorrelationId, false, "Parked message handler not available"));
}
});
Receive<ParkedMessageDiscardRequest>(msg =>
{
if (_parkedMessageHandler != null)
_parkedMessageHandler.Forward(msg);
else
{
Sender.Tell(new ParkedMessageDiscardResponse(
msg.CorrelationId, false, "Parked message handler not available"));
}
});
// Central→site Retry/Discard relay for parked cached
// operations. SiteCallAuditActor relays these over the command/control
// channel; the parked-message handler executes them against the local
// S&F buffer and replies a ParkedOperationActionAck that routes back to
// the relaying SiteCallAuditActor's Ask.
Receive<RetryParkedOperation>(msg =>
{
if (_parkedMessageHandler != null)
_parkedMessageHandler.Forward(msg);
else
{
Sender.Tell(new ParkedOperationActionAck(
msg.CorrelationId, Applied: false, "Parked message handler not available"));
}
});
Receive<DiscardParkedOperation>(msg =>
{
if (_parkedMessageHandler != null)
_parkedMessageHandler.Forward(msg);
else
{
Sender.Tell(new ParkedOperationActionAck(
msg.CorrelationId, Applied: false, "Parked message handler not available"));
}
});
// Central→site manual failover relay. Central and the site are separate clusters,
// so central can only ask — this node performs the graceful Leave locally, scoped to
// the SITE-SPECIFIC role, because that is what site singletons (the Deployment
// Manager) are placed on. Either node may receive this (the actor is per-node, not a
// singleton, and contact rotation picks whichever answers); the target is resolved
// from cluster state, not from who received the message.
Receive<TriggerSiteFailover>(HandleTriggerSiteFailover);
// from cluster state, not from who received the message. Over ClusterClient the ack
// Tell merely enqueues, so the dispatcher resolves-and-leaves in one step (the gRPC
// transport defers the leave to keep ack-before-Leave — see PrepareFailover).
Receive<TriggerSiteFailover>(msg => Sender.Tell(_dispatcher.HandleFailover(msg)));
// Notification Outbox: forward a buffered notification submitted by the site
// Store-and-Forward Engine to the central cluster. The original Sender (the
// S&F forwarder's Ask) is forwarded as the ClusterClient.Send sender so the
// NotificationSubmitAck routes straight back to the waiting Ask, not here.
Receive<NotificationSubmit>(msg =>
{
if (_centralClient == null)
{
// No ClusterClient registered yet (e.g. central contact points not
// configured, or registration not yet completed). A non-accepted ack
// makes the S&F forwarder treat this as transient and retry later.
_log.Warning(
"Cannot forward NotificationSubmit {0} — no central ClusterClient registered",
msg.NotificationId);
Sender.Tell(new NotificationSubmitAck(
msg.NotificationId, Accepted: false, Error: "Central ClusterClient not registered"));
return;
}
// The seven site→central sends now delegate to the injected transport (ClusterClient by
// default, gRPC when configured). Each handler captures the current Sender as the reply
// target so central's reply routes straight back to the waiting Ask, not through this
// actor — the exact sender-forwarding the ClusterClient path relied on. The per-message
// "no transport / not-accepted" fallbacks live inside the transport now.
_log.Debug("Forwarding NotificationSubmit {0} to central", msg.NotificationId);
_centralClient.Tell(
new ClusterClient.Send("/user/central-communication", msg), Sender);
});
// Notification Outbox: forward a buffered notification (S&F forwarder's Ask → ack back).
Receive<NotificationSubmit>(msg => _transport.SubmitNotification(msg, Sender));
// Notification Outbox: forward a Notify.Status query to the central cluster.
// The original Sender (the Notify helper's Ask) is forwarded as the
// ClusterClient.Send sender so the NotificationStatusResponse routes straight
// back to the waiting Ask, not here.
Receive<NotificationStatusQuery>(msg =>
{
if (_centralClient == null)
{
// No ClusterClient registered yet. Reply Found: false so Notify.Status
// falls back to the site S&F buffer to decide Forwarding vs Unknown.
_log.Warning(
"Cannot forward NotificationStatusQuery {0} — no central ClusterClient registered",
msg.NotificationId);
Sender.Tell(new NotificationStatusResponse(
msg.CorrelationId, Found: false, Status: "Unknown",
RetryCount: 0, LastError: null, DeliveredAt: null));
return;
}
// Notification Outbox: forward a Notify.Status query (Notify helper's Ask → response back).
Receive<NotificationStatusQuery>(msg => _transport.QueryNotificationStatus(msg, Sender));
_log.Debug("Forwarding NotificationStatusQuery {0} to central", msg.NotificationId);
_centralClient.Tell(
new ClusterClient.Send("/user/central-communication", msg), Sender);
});
// Audit Log: forward a batch of site-local audit events (telemetry drain's Ask → reply back).
Receive<IngestAuditEventsCommand>(msg => _transport.IngestAuditEvents(msg, Sender));
// Audit Log: forward a batch of site-local audit events to the
// central cluster. The site SiteAuditTelemetryActor drains its SQLite
// Pending queue through the ClusterClientSiteAuditClient, which Asks
// this actor; the original Sender (that Ask) is passed as the
// ClusterClient.Send sender so the IngestAuditEventsReply routes
// straight back to the waiting Ask, not here. Mirrors NotificationSubmit.
Receive<IngestAuditEventsCommand>(msg =>
{
if (_centralClient == null)
{
// No ClusterClient registered yet (e.g. central contact points
// not configured, or registration not yet completed). Faulting
// the Ask makes the SiteAuditTelemetryActor drain loop treat
// this as transient and keep the rows Pending for the next tick.
_log.Warning(
"Cannot forward IngestAuditEventsCommand ({0} events) — no central ClusterClient registered",
msg.Events.Count);
Sender.Tell(new Status.Failure(
new InvalidOperationException("Central ClusterClient not registered")));
return;
}
// Audit Log: forward a batch of combined cached-call telemetry (telemetry drain's Ask → reply back).
Receive<IngestCachedTelemetryCommand>(msg => _transport.IngestCachedTelemetry(msg, Sender));
_log.Debug("Forwarding IngestAuditEventsCommand ({0} events) to central", msg.Events.Count);
_centralClient.Tell(
new ClusterClient.Send("/user/central-communication", msg), Sender);
});
// Site startup reconciliation: forward the node's local-inventory request (reconcile Ask → response back).
Receive<ReconcileSiteRequest>(msg => _transport.ReconcileSite(msg, Sender));
// Audit Log: forward a batch of combined cached-call telemetry
// packets to the central cluster. Same forward + reply-routing pattern
// as IngestAuditEventsCommand; central replies with an
// IngestCachedTelemetryReply.
Receive<IngestCachedTelemetryCommand>(msg =>
{
if (_centralClient == null)
{
_log.Warning(
"Cannot forward IngestCachedTelemetryCommand ({0} entries) — no central ClusterClient registered",
msg.Entries.Count);
Sender.Tell(new Status.Failure(
new InvalidOperationException("Central ClusterClient not registered")));
return;
}
_log.Debug("Forwarding IngestCachedTelemetryCommand ({0} entries) to central", msg.Entries.Count);
_centralClient.Tell(
new ClusterClient.Send("/user/central-communication", msg), Sender);
});
// Site startup reconciliation: forward the node's local-inventory
// ReconcileSiteRequest to the central cluster. The original Sender (the
// SiteReconciliationActor's Ask) is passed as the ClusterClient.Send sender so
// the ReconcileSiteResponse routes straight back to the waiting Ask, not here.
// Mirrors IngestAuditEventsCommand.
Receive<ReconcileSiteRequest>(msg =>
{
if (_centralClient == null)
{
// No ClusterClient registered yet (e.g. central contact points not
// configured, or registration not yet completed). Faulting the Ask makes
// the SiteReconciliationActor treat the pass as best-effort-failed; it
// logs a warning and retries reconcile on the next node startup.
_log.Warning(
"Cannot forward ReconcileSiteRequest for site {0} node {1} — no central ClusterClient registered",
msg.SiteIdentifier, msg.NodeId);
Sender.Tell(new Status.Failure(
new InvalidOperationException("Central ClusterClient not registered")));
return;
}
_log.Debug(
"Forwarding ReconcileSiteRequest for site {0} node {1} ({2} local instance(s)) to central",
msg.SiteIdentifier, msg.NodeId, msg.LocalNameToRevisionHash.Count);
_centralClient.Tell(
new ClusterClient.Send("/user/central-communication", msg), Sender);
});
// Internal: send heartbeat tick
// Internal: send heartbeat tick.
Receive<SendHeartbeat>(_ => SendHeartbeatToCentral());
// Internal: forward health report to central. The original Sender (the
// AkkaHealthReportTransport's Ask) is forwarded as the ClusterClient.Send
// sender so the central SiteHealthReportAck routes straight back to the
// waiting Ask — making report delivery observable end-to-end (review 01
// [Medium]). Mirrors the NotificationSubmit ack pattern above.
Receive<SiteHealthReport>(msg =>
{
if (_centralClient == null)
{
// No ClusterClient registered yet. A non-accepted ack makes the
// sender's counter-restore path treat this tick as a loss.
_log.Warning(
"Cannot forward SiteHealthReport #{0} — no central ClusterClient registered",
msg.SequenceNumber);
Sender.Tell(new SiteHealthReportAck(
msg.SiteId, msg.SequenceNumber, Accepted: false,
Error: "Central ClusterClient not registered"));
return;
}
_centralClient.Tell(
new ClusterClient.Send("/user/central-communication", msg), Sender);
});
// Internal: forward the periodic health report (health transport's Ask → ack back), so a
// lost report is observable end-to-end and the sender can restore its per-interval counters.
Receive<SiteHealthReport>(msg => _transport.ReportSiteHealth(msg, Sender));
}
/// <inheritdoc />
@@ -443,6 +234,12 @@ public class SiteCommunicationActor : ReceiveActor, IWithTimers
/// <inheritdoc />
protected override void PreStart()
{
// Finalize the transport now that the actor context (and _log) exist. The default Akka
// transport is given this actor's logging adapter so the "no ClusterClient registered"
// warnings are preserved exactly. PreStart always runs before any message, so the Receive
// closures above see a non-null _transport.
_transport = _injectedTransport ?? new AkkaCentralTransport(_log);
_log.Info("SiteCommunicationActor started for site {0}", _siteId);
// Schedule periodic heartbeat to central. Uses the application heartbeat
@@ -455,21 +252,47 @@ public class SiteCommunicationActor : ReceiveActor, IWithTimers
_options.ApplicationHeartbeatInterval);
}
/// <summary>
/// Executes a resolved command route within the actor: Forward to the target (preserving the
/// central Ask sender so the reply routes straight back to the waiting Ask), or — when a
/// null-guarded handler is unregistered — Tell the caller the dispatcher's synthetic reply.
/// The fire-and-forget disposition (UnsubscribeDebugView) is a plain Forward here, exactly as
/// before: over ClusterClient the site never acked it, so the synthetic ack is a gRPC-only
/// concern.
/// </summary>
/// <param name="command">The migrated central→site command to route.</param>
private void DispatchCommand(object command)
{
var route = _dispatcher.ResolveRoute(command);
switch (route.Disposition)
{
case SiteCommandDispatcher.RouteDisposition.ImmediateReply:
Sender.Tell(route.Reply!);
break;
default:
// Forward and TellFireAndForget both Forward on the actor path.
route.Target!.Forward(command);
break;
}
}
private void HandleRegisterLocalHandler(RegisterLocalHandler msg)
{
// The migrated handlers live on the shared dispatcher so the gRPC command service sees the
// same registrations. Integration is the one command kept on the actor (see the receive).
switch (msg.HandlerType)
{
case LocalHandlerType.EventLog:
_eventLogHandler = msg.Handler;
_dispatcher.RegisterEventLogHandler(msg.Handler);
break;
case LocalHandlerType.ParkedMessages:
_parkedMessageHandler = msg.Handler;
_dispatcher.RegisterParkedMessageHandler(msg.Handler);
break;
case LocalHandlerType.Integration:
_integrationHandler = msg.Handler;
break;
case LocalHandlerType.Artifacts:
_artifactHandler = msg.Handler;
_dispatcher.RegisterArtifactHandler(msg.Handler);
break;
}
@@ -478,9 +301,6 @@ public class SiteCommunicationActor : ReceiveActor, IWithTimers
private void SendHeartbeatToCentral()
{
if (_centralClient == null)
return;
var hostname = Environment.MachineName;
// Stamp HeartbeatMessage.IsActive with this node's
@@ -512,8 +332,17 @@ public class SiteCommunicationActor : ReceiveActor, IWithTimers
IsActive: isActive,
DateTimeOffset.UtcNow);
_centralClient.Tell(
new ClusterClient.Send("/user/central-communication", heartbeat), Self);
// Fire-and-forget on both transports: a failure here must never fault the heartbeat timer
// path. Both real transports swallow their own errors; this catch is a belt-and-braces
// guarantee that no transport (including a future one) can turn a heartbeat into a fault.
try
{
_transport.SendHeartbeat(heartbeat, Self);
}
catch (Exception ex)
{
_log.Debug(ex, "Heartbeat send for site {0} failed; swallowed (heartbeats are fire-and-forget)", _siteId);
}
}
/// <summary>
@@ -528,69 +357,6 @@ public class SiteCommunicationActor : ReceiveActor, IWithTimers
private bool DefaultIsActiveCheck() =>
ClusterState.ActiveNodeEvaluator.SelfIsOldestUp(Cluster.Get(Context.System));
/// <summary>
/// Handles a central-initiated site failover. Refuses a command addressed to a different
/// site (a misroute must never silently fail over a site the operator did not select) and
/// refuses when the site pair has no peer to take over. The ack is sent BEFORE the Leave
/// takes effect on the wire, so it still reaches central even when this node is the one
/// leaving.
/// </summary>
private void HandleTriggerSiteFailover(TriggerSiteFailover msg)
{
if (!string.Equals(msg.SiteId, _siteId, StringComparison.Ordinal))
{
_log.Warning(
"Refusing TriggerSiteFailover addressed to site {Requested}; this node serves {Actual}",
msg.SiteId, _siteId);
Sender.Tell(new SiteFailoverAck(
msg.CorrelationId, Accepted: false, TargetAddress: null,
ErrorMessage: $"Command addressed to site '{msg.SiteId}' but this node serves '{_siteId}'."));
return;
}
var role = $"site-{_siteId}";
try
{
var target = _failOverRole(role);
if (target is null)
{
_log.Warning(
"Refusing TriggerSiteFailover for {SiteId}: fewer than 2 Up members in role {Role}, "
+ "so there is no standby to take over", _siteId, role);
Sender.Tell(new SiteFailoverAck(
msg.CorrelationId, Accepted: false, TargetAddress: null,
ErrorMessage: "No standby available — failing over a lone node would be an outage."));
return;
}
_log.Warning(
"Manual failover requested by central for site {SiteId}: {Target} is leaving the "
+ "site cluster gracefully; its singletons hand over to the standby.", _siteId, target);
Sender.Tell(new SiteFailoverAck(msg.CorrelationId, Accepted: true, target, ErrorMessage: null));
}
catch (Exception ex)
{
// A fault here must be reported to the operator, not thrown into supervision —
// restarting the communication actor would drop central's Ask into a timeout and
// lose the reason.
_log.Error(ex, "TriggerSiteFailover for {SiteId} faulted", _siteId);
Sender.Tell(new SiteFailoverAck(
msg.CorrelationId, Accepted: false, TargetAddress: null, ErrorMessage: ex.Message));
}
}
/// <summary>
/// Production failover action: gracefully Leave the oldest Up member carrying
/// <paramref name="role"/>, via the shared <see cref="ClusterState.ClusterFailoverCoordinator"/>
/// so the central and site paths cannot drift. Injected in tests for the same reason
/// <see cref="DefaultIsActiveCheck"/> is — a real Leave needs Akka.Cluster in the
/// ActorSystem, which the TestKit system does not load.
/// </summary>
/// <param name="role">Site-specific role scope.</param>
/// <returns>Address of the leaving node, or null when there is no peer.</returns>
private string? DefaultFailOverRole(string role) =>
ClusterState.ClusterFailoverCoordinator.FailOverOldest(Context.System, role)?.ToString();
// ── Internal messages ──
internal record SendHeartbeat;
File diff suppressed because it is too large Load Diff
@@ -0,0 +1,704 @@
// <auto-generated>
// Generated by the protocol buffer compiler. DO NOT EDIT!
// source: Protos/central_control.proto
// </auto-generated>
#pragma warning disable 0414, 1591, 8981, 0612
#region Designer generated code
using grpc = global::Grpc.Core;
namespace ZB.MOM.WW.ScadaBridge.Communication.Grpc {
/// <summary>
/// Central-hosted control plane (Phase 1A of the ClusterClient→gRPC migration).
///
/// Direction: SITE is the client, CENTRAL is the server — the inverse of
/// SiteStreamService, where central dials the site. That asymmetry is deliberate
/// and mirrors the direction the Akka ClusterClient traffic flows today: these
/// seven calls are exactly the seven messages SiteCommunicationActor sends to
/// /user/central-communication.
///
/// Every call is gated by ControlPlaneAuthInterceptor: `authorization: Bearer &lt;psk>`
/// plus the `x-scadabridge-site` metadata header naming which site's preshared key
/// central must verify against.
/// </summary>
public static partial class CentralControlService
{
static readonly string __ServiceName = "scadabridge.centralcontrol.v1.CentralControlService";
[global::System.CodeDom.Compiler.GeneratedCode("grpc_csharp_plugin", null)]
static void __Helper_SerializeMessage(global::Google.Protobuf.IMessage message, grpc::SerializationContext context)
{
#if !GRPC_DISABLE_PROTOBUF_BUFFER_SERIALIZATION
if (message is global::Google.Protobuf.IBufferMessage)
{
context.SetPayloadLength(message.CalculateSize());
global::Google.Protobuf.MessageExtensions.WriteTo(message, context.GetBufferWriter());
context.Complete();
return;
}
#endif
context.Complete(global::Google.Protobuf.MessageExtensions.ToByteArray(message));
}
[global::System.CodeDom.Compiler.GeneratedCode("grpc_csharp_plugin", null)]
static class __Helper_MessageCache<T>
{
public static readonly bool IsBufferMessage = global::System.Reflection.IntrospectionExtensions.GetTypeInfo(typeof(global::Google.Protobuf.IBufferMessage)).IsAssignableFrom(typeof(T));
}
[global::System.CodeDom.Compiler.GeneratedCode("grpc_csharp_plugin", null)]
static T __Helper_DeserializeMessage<T>(grpc::DeserializationContext context, global::Google.Protobuf.MessageParser<T> parser) where T : global::Google.Protobuf.IMessage<T>
{
#if !GRPC_DISABLE_PROTOBUF_BUFFER_SERIALIZATION
if (__Helper_MessageCache<T>.IsBufferMessage)
{
return parser.ParseFrom(context.PayloadAsReadOnlySequence());
}
#endif
return parser.ParseFrom(context.PayloadAsNewBuffer());
}
[global::System.CodeDom.Compiler.GeneratedCode("grpc_csharp_plugin", null)]
static readonly grpc::Marshaller<global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.NotificationSubmitDto> __Marshaller_scadabridge_centralcontrol_v1_NotificationSubmitDto = grpc::Marshallers.Create(__Helper_SerializeMessage, context => __Helper_DeserializeMessage(context, global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.NotificationSubmitDto.Parser));
[global::System.CodeDom.Compiler.GeneratedCode("grpc_csharp_plugin", null)]
static readonly grpc::Marshaller<global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.NotificationSubmitAckDto> __Marshaller_scadabridge_centralcontrol_v1_NotificationSubmitAckDto = grpc::Marshallers.Create(__Helper_SerializeMessage, context => __Helper_DeserializeMessage(context, global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.NotificationSubmitAckDto.Parser));
[global::System.CodeDom.Compiler.GeneratedCode("grpc_csharp_plugin", null)]
static readonly grpc::Marshaller<global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.NotificationStatusQueryDto> __Marshaller_scadabridge_centralcontrol_v1_NotificationStatusQueryDto = grpc::Marshallers.Create(__Helper_SerializeMessage, context => __Helper_DeserializeMessage(context, global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.NotificationStatusQueryDto.Parser));
[global::System.CodeDom.Compiler.GeneratedCode("grpc_csharp_plugin", null)]
static readonly grpc::Marshaller<global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.NotificationStatusResponseDto> __Marshaller_scadabridge_centralcontrol_v1_NotificationStatusResponseDto = grpc::Marshallers.Create(__Helper_SerializeMessage, context => __Helper_DeserializeMessage(context, global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.NotificationStatusResponseDto.Parser));
[global::System.CodeDom.Compiler.GeneratedCode("grpc_csharp_plugin", null)]
static readonly grpc::Marshaller<global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.AuditEventBatch> __Marshaller_sitestream_AuditEventBatch = grpc::Marshallers.Create(__Helper_SerializeMessage, context => __Helper_DeserializeMessage(context, global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.AuditEventBatch.Parser));
[global::System.CodeDom.Compiler.GeneratedCode("grpc_csharp_plugin", null)]
static readonly grpc::Marshaller<global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.IngestAck> __Marshaller_sitestream_IngestAck = grpc::Marshallers.Create(__Helper_SerializeMessage, context => __Helper_DeserializeMessage(context, global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.IngestAck.Parser));
[global::System.CodeDom.Compiler.GeneratedCode("grpc_csharp_plugin", null)]
static readonly grpc::Marshaller<global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.CachedTelemetryBatch> __Marshaller_sitestream_CachedTelemetryBatch = grpc::Marshallers.Create(__Helper_SerializeMessage, context => __Helper_DeserializeMessage(context, global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.CachedTelemetryBatch.Parser));
[global::System.CodeDom.Compiler.GeneratedCode("grpc_csharp_plugin", null)]
static readonly grpc::Marshaller<global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.ReconcileSiteRequestDto> __Marshaller_scadabridge_centralcontrol_v1_ReconcileSiteRequestDto = grpc::Marshallers.Create(__Helper_SerializeMessage, context => __Helper_DeserializeMessage(context, global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.ReconcileSiteRequestDto.Parser));
[global::System.CodeDom.Compiler.GeneratedCode("grpc_csharp_plugin", null)]
static readonly grpc::Marshaller<global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.ReconcileSiteResponseDto> __Marshaller_scadabridge_centralcontrol_v1_ReconcileSiteResponseDto = grpc::Marshallers.Create(__Helper_SerializeMessage, context => __Helper_DeserializeMessage(context, global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.ReconcileSiteResponseDto.Parser));
[global::System.CodeDom.Compiler.GeneratedCode("grpc_csharp_plugin", null)]
static readonly grpc::Marshaller<global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.SiteHealthReportDto> __Marshaller_scadabridge_centralcontrol_v1_SiteHealthReportDto = grpc::Marshallers.Create(__Helper_SerializeMessage, context => __Helper_DeserializeMessage(context, global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.SiteHealthReportDto.Parser));
[global::System.CodeDom.Compiler.GeneratedCode("grpc_csharp_plugin", null)]
static readonly grpc::Marshaller<global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.SiteHealthReportAckDto> __Marshaller_scadabridge_centralcontrol_v1_SiteHealthReportAckDto = grpc::Marshallers.Create(__Helper_SerializeMessage, context => __Helper_DeserializeMessage(context, global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.SiteHealthReportAckDto.Parser));
[global::System.CodeDom.Compiler.GeneratedCode("grpc_csharp_plugin", null)]
static readonly grpc::Marshaller<global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.HeartbeatDto> __Marshaller_scadabridge_centralcontrol_v1_HeartbeatDto = grpc::Marshallers.Create(__Helper_SerializeMessage, context => __Helper_DeserializeMessage(context, global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.HeartbeatDto.Parser));
[global::System.CodeDom.Compiler.GeneratedCode("grpc_csharp_plugin", null)]
static readonly grpc::Marshaller<global::Google.Protobuf.WellKnownTypes.Empty> __Marshaller_google_protobuf_Empty = grpc::Marshallers.Create(__Helper_SerializeMessage, context => __Helper_DeserializeMessage(context, global::Google.Protobuf.WellKnownTypes.Empty.Parser));
[global::System.CodeDom.Compiler.GeneratedCode("grpc_csharp_plugin", null)]
static readonly grpc::Method<global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.NotificationSubmitDto, global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.NotificationSubmitAckDto> __Method_SubmitNotification = new grpc::Method<global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.NotificationSubmitDto, global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.NotificationSubmitAckDto>(
grpc::MethodType.Unary,
__ServiceName,
"SubmitNotification",
__Marshaller_scadabridge_centralcontrol_v1_NotificationSubmitDto,
__Marshaller_scadabridge_centralcontrol_v1_NotificationSubmitAckDto);
[global::System.CodeDom.Compiler.GeneratedCode("grpc_csharp_plugin", null)]
static readonly grpc::Method<global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.NotificationStatusQueryDto, global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.NotificationStatusResponseDto> __Method_QueryNotificationStatus = new grpc::Method<global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.NotificationStatusQueryDto, global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.NotificationStatusResponseDto>(
grpc::MethodType.Unary,
__ServiceName,
"QueryNotificationStatus",
__Marshaller_scadabridge_centralcontrol_v1_NotificationStatusQueryDto,
__Marshaller_scadabridge_centralcontrol_v1_NotificationStatusResponseDto);
[global::System.CodeDom.Compiler.GeneratedCode("grpc_csharp_plugin", null)]
static readonly grpc::Method<global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.AuditEventBatch, global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.IngestAck> __Method_IngestAuditEvents = new grpc::Method<global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.AuditEventBatch, global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.IngestAck>(
grpc::MethodType.Unary,
__ServiceName,
"IngestAuditEvents",
__Marshaller_sitestream_AuditEventBatch,
__Marshaller_sitestream_IngestAck);
[global::System.CodeDom.Compiler.GeneratedCode("grpc_csharp_plugin", null)]
static readonly grpc::Method<global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.CachedTelemetryBatch, global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.IngestAck> __Method_IngestCachedTelemetry = new grpc::Method<global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.CachedTelemetryBatch, global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.IngestAck>(
grpc::MethodType.Unary,
__ServiceName,
"IngestCachedTelemetry",
__Marshaller_sitestream_CachedTelemetryBatch,
__Marshaller_sitestream_IngestAck);
[global::System.CodeDom.Compiler.GeneratedCode("grpc_csharp_plugin", null)]
static readonly grpc::Method<global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.ReconcileSiteRequestDto, global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.ReconcileSiteResponseDto> __Method_ReconcileSite = new grpc::Method<global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.ReconcileSiteRequestDto, global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.ReconcileSiteResponseDto>(
grpc::MethodType.Unary,
__ServiceName,
"ReconcileSite",
__Marshaller_scadabridge_centralcontrol_v1_ReconcileSiteRequestDto,
__Marshaller_scadabridge_centralcontrol_v1_ReconcileSiteResponseDto);
[global::System.CodeDom.Compiler.GeneratedCode("grpc_csharp_plugin", null)]
static readonly grpc::Method<global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.SiteHealthReportDto, global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.SiteHealthReportAckDto> __Method_ReportSiteHealth = new grpc::Method<global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.SiteHealthReportDto, global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.SiteHealthReportAckDto>(
grpc::MethodType.Unary,
__ServiceName,
"ReportSiteHealth",
__Marshaller_scadabridge_centralcontrol_v1_SiteHealthReportDto,
__Marshaller_scadabridge_centralcontrol_v1_SiteHealthReportAckDto);
[global::System.CodeDom.Compiler.GeneratedCode("grpc_csharp_plugin", null)]
static readonly grpc::Method<global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.HeartbeatDto, global::Google.Protobuf.WellKnownTypes.Empty> __Method_Heartbeat = new grpc::Method<global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.HeartbeatDto, global::Google.Protobuf.WellKnownTypes.Empty>(
grpc::MethodType.Unary,
__ServiceName,
"Heartbeat",
__Marshaller_scadabridge_centralcontrol_v1_HeartbeatDto,
__Marshaller_google_protobuf_Empty);
/// <summary>Service descriptor</summary>
public static global::Google.Protobuf.Reflection.ServiceDescriptor Descriptor
{
get { return global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.CentralControlReflection.Descriptor.Services[0]; }
}
/// <summary>Base class for server-side implementations of CentralControlService</summary>
[grpc::BindServiceMethod(typeof(CentralControlService), "BindService")]
public abstract partial class CentralControlServiceBase
{
/// <summary>
/// Store-and-forward handoff of one notification for central delivery. The
/// ack is idempotent on notification_id — a duplicate submit after a lost ack
/// must not produce a second delivery.
/// </summary>
/// <param name="request">The request received from the client.</param>
/// <param name="context">The context of the server-side call handler being invoked.</param>
/// <returns>The response to send back to the client (wrapped by a task).</returns>
[global::System.CodeDom.Compiler.GeneratedCode("grpc_csharp_plugin", null)]
public virtual global::System.Threading.Tasks.Task<global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.NotificationSubmitAckDto> SubmitNotification(global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.NotificationSubmitDto request, grpc::ServerCallContext context)
{
throw new grpc::RpcException(new grpc::Status(grpc::StatusCode.Unimplemented, ""));
}
/// <summary>
/// Notify.Status(id) round-trip for a notification that has already left the
/// site buffer. `found = false` sends the caller back to the site-local buffer
/// to decide Forwarding vs Unknown.
/// </summary>
/// <param name="request">The request received from the client.</param>
/// <param name="context">The context of the server-side call handler being invoked.</param>
/// <returns>The response to send back to the client (wrapped by a task).</returns>
[global::System.CodeDom.Compiler.GeneratedCode("grpc_csharp_plugin", null)]
public virtual global::System.Threading.Tasks.Task<global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.NotificationStatusResponseDto> QueryNotificationStatus(global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.NotificationStatusQueryDto request, grpc::ServerCallContext context)
{
throw new grpc::RpcException(new grpc::Status(grpc::StatusCode.Unimplemented, ""));
}
/// <summary>
/// Audit Log (#23) push telemetry. Reuses the SiteStreamService messages: the
/// batch a site drains from its SQLite hot path is byte-identical whichever
/// transport carries it.
/// </summary>
/// <param name="request">The request received from the client.</param>
/// <param name="context">The context of the server-side call handler being invoked.</param>
/// <returns>The response to send back to the client (wrapped by a task).</returns>
[global::System.CodeDom.Compiler.GeneratedCode("grpc_csharp_plugin", null)]
public virtual global::System.Threading.Tasks.Task<global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.IngestAck> IngestAuditEvents(global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.AuditEventBatch request, grpc::ServerCallContext context)
{
throw new grpc::RpcException(new grpc::Status(grpc::StatusCode.Unimplemented, ""));
}
/// <summary>
/// Audit Log (#23) M3 combined cached-call telemetry (audit row + SiteCalls
/// operational upsert, written in one central transaction).
/// </summary>
/// <param name="request">The request received from the client.</param>
/// <param name="context">The context of the server-side call handler being invoked.</param>
/// <returns>The response to send back to the client (wrapped by a task).</returns>
[global::System.CodeDom.Compiler.GeneratedCode("grpc_csharp_plugin", null)]
public virtual global::System.Threading.Tasks.Task<global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.IngestAck> IngestCachedTelemetry(global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.CachedTelemetryBatch request, grpc::ServerCallContext context)
{
throw new grpc::RpcException(new grpc::Status(grpc::StatusCode.Unimplemented, ""));
}
/// <summary>
/// Node-startup self-heal: the node's local deployed inventory in, fetch
/// tokens for whatever it is missing or stale out.
/// </summary>
/// <param name="request">The request received from the client.</param>
/// <param name="context">The context of the server-side call handler being invoked.</param>
/// <returns>The response to send back to the client (wrapped by a task).</returns>
[global::System.CodeDom.Compiler.GeneratedCode("grpc_csharp_plugin", null)]
public virtual global::System.Threading.Tasks.Task<global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.ReconcileSiteResponseDto> ReconcileSite(global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.ReconcileSiteRequestDto request, grpc::ServerCallContext context)
{
throw new grpc::RpcException(new grpc::Status(grpc::StatusCode.Unimplemented, ""));
}
/// <summary>
/// Periodic site health report (30 s cadence). The ack makes delivery
/// observable end-to-end so the sender can restore its per-interval counters
/// when a report is lost.
/// </summary>
/// <param name="request">The request received from the client.</param>
/// <param name="context">The context of the server-side call handler being invoked.</param>
/// <returns>The response to send back to the client (wrapped by a task).</returns>
[global::System.CodeDom.Compiler.GeneratedCode("grpc_csharp_plugin", null)]
public virtual global::System.Threading.Tasks.Task<global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.SiteHealthReportAckDto> ReportSiteHealth(global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.SiteHealthReportDto request, grpc::ServerCallContext context)
{
throw new grpc::RpcException(new grpc::Status(grpc::StatusCode.Unimplemented, ""));
}
/// <summary>
/// Application heartbeat. Returns Empty because the message is
/// fire-and-forget: nothing on the site consumes a reply, and a failure here
/// must never surface as a fault on the heartbeat timer path.
/// </summary>
/// <param name="request">The request received from the client.</param>
/// <param name="context">The context of the server-side call handler being invoked.</param>
/// <returns>The response to send back to the client (wrapped by a task).</returns>
[global::System.CodeDom.Compiler.GeneratedCode("grpc_csharp_plugin", null)]
public virtual global::System.Threading.Tasks.Task<global::Google.Protobuf.WellKnownTypes.Empty> Heartbeat(global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.HeartbeatDto request, grpc::ServerCallContext context)
{
throw new grpc::RpcException(new grpc::Status(grpc::StatusCode.Unimplemented, ""));
}
}
/// <summary>Client for CentralControlService</summary>
public partial class CentralControlServiceClient : grpc::ClientBase<CentralControlServiceClient>
{
/// <summary>Creates a new client for CentralControlService</summary>
/// <param name="channel">The channel to use to make remote calls.</param>
[global::System.CodeDom.Compiler.GeneratedCode("grpc_csharp_plugin", null)]
public CentralControlServiceClient(grpc::ChannelBase channel) : base(channel)
{
}
/// <summary>Creates a new client for CentralControlService that uses a custom <c>CallInvoker</c>.</summary>
/// <param name="callInvoker">The callInvoker to use to make remote calls.</param>
[global::System.CodeDom.Compiler.GeneratedCode("grpc_csharp_plugin", null)]
public CentralControlServiceClient(grpc::CallInvoker callInvoker) : base(callInvoker)
{
}
/// <summary>Protected parameterless constructor to allow creation of test doubles.</summary>
[global::System.CodeDom.Compiler.GeneratedCode("grpc_csharp_plugin", null)]
protected CentralControlServiceClient() : base()
{
}
/// <summary>Protected constructor to allow creation of configured clients.</summary>
/// <param name="configuration">The client configuration.</param>
[global::System.CodeDom.Compiler.GeneratedCode("grpc_csharp_plugin", null)]
protected CentralControlServiceClient(ClientBaseConfiguration configuration) : base(configuration)
{
}
/// <summary>
/// Store-and-forward handoff of one notification for central delivery. The
/// ack is idempotent on notification_id — a duplicate submit after a lost ack
/// must not produce a second delivery.
/// </summary>
/// <param name="request">The request to send to the server.</param>
/// <param name="headers">The initial metadata to send with the call. This parameter is optional.</param>
/// <param name="deadline">An optional deadline for the call. The call will be cancelled if deadline is hit.</param>
/// <param name="cancellationToken">An optional token for canceling the call.</param>
/// <returns>The response received from the server.</returns>
[global::System.CodeDom.Compiler.GeneratedCode("grpc_csharp_plugin", null)]
public virtual global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.NotificationSubmitAckDto SubmitNotification(global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.NotificationSubmitDto request, grpc::Metadata headers = null, global::System.DateTime? deadline = null, global::System.Threading.CancellationToken cancellationToken = default(global::System.Threading.CancellationToken))
{
return SubmitNotification(request, new grpc::CallOptions(headers, deadline, cancellationToken));
}
/// <summary>
/// Store-and-forward handoff of one notification for central delivery. The
/// ack is idempotent on notification_id — a duplicate submit after a lost ack
/// must not produce a second delivery.
/// </summary>
/// <param name="request">The request to send to the server.</param>
/// <param name="options">The options for the call.</param>
/// <returns>The response received from the server.</returns>
[global::System.CodeDom.Compiler.GeneratedCode("grpc_csharp_plugin", null)]
public virtual global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.NotificationSubmitAckDto SubmitNotification(global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.NotificationSubmitDto request, grpc::CallOptions options)
{
return CallInvoker.BlockingUnaryCall(__Method_SubmitNotification, null, options, request);
}
/// <summary>
/// Store-and-forward handoff of one notification for central delivery. The
/// ack is idempotent on notification_id — a duplicate submit after a lost ack
/// must not produce a second delivery.
/// </summary>
/// <param name="request">The request to send to the server.</param>
/// <param name="headers">The initial metadata to send with the call. This parameter is optional.</param>
/// <param name="deadline">An optional deadline for the call. The call will be cancelled if deadline is hit.</param>
/// <param name="cancellationToken">An optional token for canceling the call.</param>
/// <returns>The call object.</returns>
[global::System.CodeDom.Compiler.GeneratedCode("grpc_csharp_plugin", null)]
public virtual grpc::AsyncUnaryCall<global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.NotificationSubmitAckDto> SubmitNotificationAsync(global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.NotificationSubmitDto request, grpc::Metadata headers = null, global::System.DateTime? deadline = null, global::System.Threading.CancellationToken cancellationToken = default(global::System.Threading.CancellationToken))
{
return SubmitNotificationAsync(request, new grpc::CallOptions(headers, deadline, cancellationToken));
}
/// <summary>
/// Store-and-forward handoff of one notification for central delivery. The
/// ack is idempotent on notification_id — a duplicate submit after a lost ack
/// must not produce a second delivery.
/// </summary>
/// <param name="request">The request to send to the server.</param>
/// <param name="options">The options for the call.</param>
/// <returns>The call object.</returns>
[global::System.CodeDom.Compiler.GeneratedCode("grpc_csharp_plugin", null)]
public virtual grpc::AsyncUnaryCall<global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.NotificationSubmitAckDto> SubmitNotificationAsync(global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.NotificationSubmitDto request, grpc::CallOptions options)
{
return CallInvoker.AsyncUnaryCall(__Method_SubmitNotification, null, options, request);
}
/// <summary>
/// Notify.Status(id) round-trip for a notification that has already left the
/// site buffer. `found = false` sends the caller back to the site-local buffer
/// to decide Forwarding vs Unknown.
/// </summary>
/// <param name="request">The request to send to the server.</param>
/// <param name="headers">The initial metadata to send with the call. This parameter is optional.</param>
/// <param name="deadline">An optional deadline for the call. The call will be cancelled if deadline is hit.</param>
/// <param name="cancellationToken">An optional token for canceling the call.</param>
/// <returns>The response received from the server.</returns>
[global::System.CodeDom.Compiler.GeneratedCode("grpc_csharp_plugin", null)]
public virtual global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.NotificationStatusResponseDto QueryNotificationStatus(global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.NotificationStatusQueryDto request, grpc::Metadata headers = null, global::System.DateTime? deadline = null, global::System.Threading.CancellationToken cancellationToken = default(global::System.Threading.CancellationToken))
{
return QueryNotificationStatus(request, new grpc::CallOptions(headers, deadline, cancellationToken));
}
/// <summary>
/// Notify.Status(id) round-trip for a notification that has already left the
/// site buffer. `found = false` sends the caller back to the site-local buffer
/// to decide Forwarding vs Unknown.
/// </summary>
/// <param name="request">The request to send to the server.</param>
/// <param name="options">The options for the call.</param>
/// <returns>The response received from the server.</returns>
[global::System.CodeDom.Compiler.GeneratedCode("grpc_csharp_plugin", null)]
public virtual global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.NotificationStatusResponseDto QueryNotificationStatus(global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.NotificationStatusQueryDto request, grpc::CallOptions options)
{
return CallInvoker.BlockingUnaryCall(__Method_QueryNotificationStatus, null, options, request);
}
/// <summary>
/// Notify.Status(id) round-trip for a notification that has already left the
/// site buffer. `found = false` sends the caller back to the site-local buffer
/// to decide Forwarding vs Unknown.
/// </summary>
/// <param name="request">The request to send to the server.</param>
/// <param name="headers">The initial metadata to send with the call. This parameter is optional.</param>
/// <param name="deadline">An optional deadline for the call. The call will be cancelled if deadline is hit.</param>
/// <param name="cancellationToken">An optional token for canceling the call.</param>
/// <returns>The call object.</returns>
[global::System.CodeDom.Compiler.GeneratedCode("grpc_csharp_plugin", null)]
public virtual grpc::AsyncUnaryCall<global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.NotificationStatusResponseDto> QueryNotificationStatusAsync(global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.NotificationStatusQueryDto request, grpc::Metadata headers = null, global::System.DateTime? deadline = null, global::System.Threading.CancellationToken cancellationToken = default(global::System.Threading.CancellationToken))
{
return QueryNotificationStatusAsync(request, new grpc::CallOptions(headers, deadline, cancellationToken));
}
/// <summary>
/// Notify.Status(id) round-trip for a notification that has already left the
/// site buffer. `found = false` sends the caller back to the site-local buffer
/// to decide Forwarding vs Unknown.
/// </summary>
/// <param name="request">The request to send to the server.</param>
/// <param name="options">The options for the call.</param>
/// <returns>The call object.</returns>
[global::System.CodeDom.Compiler.GeneratedCode("grpc_csharp_plugin", null)]
public virtual grpc::AsyncUnaryCall<global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.NotificationStatusResponseDto> QueryNotificationStatusAsync(global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.NotificationStatusQueryDto request, grpc::CallOptions options)
{
return CallInvoker.AsyncUnaryCall(__Method_QueryNotificationStatus, null, options, request);
}
/// <summary>
/// Audit Log (#23) push telemetry. Reuses the SiteStreamService messages: the
/// batch a site drains from its SQLite hot path is byte-identical whichever
/// transport carries it.
/// </summary>
/// <param name="request">The request to send to the server.</param>
/// <param name="headers">The initial metadata to send with the call. This parameter is optional.</param>
/// <param name="deadline">An optional deadline for the call. The call will be cancelled if deadline is hit.</param>
/// <param name="cancellationToken">An optional token for canceling the call.</param>
/// <returns>The response received from the server.</returns>
[global::System.CodeDom.Compiler.GeneratedCode("grpc_csharp_plugin", null)]
public virtual global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.IngestAck IngestAuditEvents(global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.AuditEventBatch request, grpc::Metadata headers = null, global::System.DateTime? deadline = null, global::System.Threading.CancellationToken cancellationToken = default(global::System.Threading.CancellationToken))
{
return IngestAuditEvents(request, new grpc::CallOptions(headers, deadline, cancellationToken));
}
/// <summary>
/// Audit Log (#23) push telemetry. Reuses the SiteStreamService messages: the
/// batch a site drains from its SQLite hot path is byte-identical whichever
/// transport carries it.
/// </summary>
/// <param name="request">The request to send to the server.</param>
/// <param name="options">The options for the call.</param>
/// <returns>The response received from the server.</returns>
[global::System.CodeDom.Compiler.GeneratedCode("grpc_csharp_plugin", null)]
public virtual global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.IngestAck IngestAuditEvents(global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.AuditEventBatch request, grpc::CallOptions options)
{
return CallInvoker.BlockingUnaryCall(__Method_IngestAuditEvents, null, options, request);
}
/// <summary>
/// Audit Log (#23) push telemetry. Reuses the SiteStreamService messages: the
/// batch a site drains from its SQLite hot path is byte-identical whichever
/// transport carries it.
/// </summary>
/// <param name="request">The request to send to the server.</param>
/// <param name="headers">The initial metadata to send with the call. This parameter is optional.</param>
/// <param name="deadline">An optional deadline for the call. The call will be cancelled if deadline is hit.</param>
/// <param name="cancellationToken">An optional token for canceling the call.</param>
/// <returns>The call object.</returns>
[global::System.CodeDom.Compiler.GeneratedCode("grpc_csharp_plugin", null)]
public virtual grpc::AsyncUnaryCall<global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.IngestAck> IngestAuditEventsAsync(global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.AuditEventBatch request, grpc::Metadata headers = null, global::System.DateTime? deadline = null, global::System.Threading.CancellationToken cancellationToken = default(global::System.Threading.CancellationToken))
{
return IngestAuditEventsAsync(request, new grpc::CallOptions(headers, deadline, cancellationToken));
}
/// <summary>
/// Audit Log (#23) push telemetry. Reuses the SiteStreamService messages: the
/// batch a site drains from its SQLite hot path is byte-identical whichever
/// transport carries it.
/// </summary>
/// <param name="request">The request to send to the server.</param>
/// <param name="options">The options for the call.</param>
/// <returns>The call object.</returns>
[global::System.CodeDom.Compiler.GeneratedCode("grpc_csharp_plugin", null)]
public virtual grpc::AsyncUnaryCall<global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.IngestAck> IngestAuditEventsAsync(global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.AuditEventBatch request, grpc::CallOptions options)
{
return CallInvoker.AsyncUnaryCall(__Method_IngestAuditEvents, null, options, request);
}
/// <summary>
/// Audit Log (#23) M3 combined cached-call telemetry (audit row + SiteCalls
/// operational upsert, written in one central transaction).
/// </summary>
/// <param name="request">The request to send to the server.</param>
/// <param name="headers">The initial metadata to send with the call. This parameter is optional.</param>
/// <param name="deadline">An optional deadline for the call. The call will be cancelled if deadline is hit.</param>
/// <param name="cancellationToken">An optional token for canceling the call.</param>
/// <returns>The response received from the server.</returns>
[global::System.CodeDom.Compiler.GeneratedCode("grpc_csharp_plugin", null)]
public virtual global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.IngestAck IngestCachedTelemetry(global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.CachedTelemetryBatch request, grpc::Metadata headers = null, global::System.DateTime? deadline = null, global::System.Threading.CancellationToken cancellationToken = default(global::System.Threading.CancellationToken))
{
return IngestCachedTelemetry(request, new grpc::CallOptions(headers, deadline, cancellationToken));
}
/// <summary>
/// Audit Log (#23) M3 combined cached-call telemetry (audit row + SiteCalls
/// operational upsert, written in one central transaction).
/// </summary>
/// <param name="request">The request to send to the server.</param>
/// <param name="options">The options for the call.</param>
/// <returns>The response received from the server.</returns>
[global::System.CodeDom.Compiler.GeneratedCode("grpc_csharp_plugin", null)]
public virtual global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.IngestAck IngestCachedTelemetry(global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.CachedTelemetryBatch request, grpc::CallOptions options)
{
return CallInvoker.BlockingUnaryCall(__Method_IngestCachedTelemetry, null, options, request);
}
/// <summary>
/// Audit Log (#23) M3 combined cached-call telemetry (audit row + SiteCalls
/// operational upsert, written in one central transaction).
/// </summary>
/// <param name="request">The request to send to the server.</param>
/// <param name="headers">The initial metadata to send with the call. This parameter is optional.</param>
/// <param name="deadline">An optional deadline for the call. The call will be cancelled if deadline is hit.</param>
/// <param name="cancellationToken">An optional token for canceling the call.</param>
/// <returns>The call object.</returns>
[global::System.CodeDom.Compiler.GeneratedCode("grpc_csharp_plugin", null)]
public virtual grpc::AsyncUnaryCall<global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.IngestAck> IngestCachedTelemetryAsync(global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.CachedTelemetryBatch request, grpc::Metadata headers = null, global::System.DateTime? deadline = null, global::System.Threading.CancellationToken cancellationToken = default(global::System.Threading.CancellationToken))
{
return IngestCachedTelemetryAsync(request, new grpc::CallOptions(headers, deadline, cancellationToken));
}
/// <summary>
/// Audit Log (#23) M3 combined cached-call telemetry (audit row + SiteCalls
/// operational upsert, written in one central transaction).
/// </summary>
/// <param name="request">The request to send to the server.</param>
/// <param name="options">The options for the call.</param>
/// <returns>The call object.</returns>
[global::System.CodeDom.Compiler.GeneratedCode("grpc_csharp_plugin", null)]
public virtual grpc::AsyncUnaryCall<global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.IngestAck> IngestCachedTelemetryAsync(global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.CachedTelemetryBatch request, grpc::CallOptions options)
{
return CallInvoker.AsyncUnaryCall(__Method_IngestCachedTelemetry, null, options, request);
}
/// <summary>
/// Node-startup self-heal: the node's local deployed inventory in, fetch
/// tokens for whatever it is missing or stale out.
/// </summary>
/// <param name="request">The request to send to the server.</param>
/// <param name="headers">The initial metadata to send with the call. This parameter is optional.</param>
/// <param name="deadline">An optional deadline for the call. The call will be cancelled if deadline is hit.</param>
/// <param name="cancellationToken">An optional token for canceling the call.</param>
/// <returns>The response received from the server.</returns>
[global::System.CodeDom.Compiler.GeneratedCode("grpc_csharp_plugin", null)]
public virtual global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.ReconcileSiteResponseDto ReconcileSite(global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.ReconcileSiteRequestDto request, grpc::Metadata headers = null, global::System.DateTime? deadline = null, global::System.Threading.CancellationToken cancellationToken = default(global::System.Threading.CancellationToken))
{
return ReconcileSite(request, new grpc::CallOptions(headers, deadline, cancellationToken));
}
/// <summary>
/// Node-startup self-heal: the node's local deployed inventory in, fetch
/// tokens for whatever it is missing or stale out.
/// </summary>
/// <param name="request">The request to send to the server.</param>
/// <param name="options">The options for the call.</param>
/// <returns>The response received from the server.</returns>
[global::System.CodeDom.Compiler.GeneratedCode("grpc_csharp_plugin", null)]
public virtual global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.ReconcileSiteResponseDto ReconcileSite(global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.ReconcileSiteRequestDto request, grpc::CallOptions options)
{
return CallInvoker.BlockingUnaryCall(__Method_ReconcileSite, null, options, request);
}
/// <summary>
/// Node-startup self-heal: the node's local deployed inventory in, fetch
/// tokens for whatever it is missing or stale out.
/// </summary>
/// <param name="request">The request to send to the server.</param>
/// <param name="headers">The initial metadata to send with the call. This parameter is optional.</param>
/// <param name="deadline">An optional deadline for the call. The call will be cancelled if deadline is hit.</param>
/// <param name="cancellationToken">An optional token for canceling the call.</param>
/// <returns>The call object.</returns>
[global::System.CodeDom.Compiler.GeneratedCode("grpc_csharp_plugin", null)]
public virtual grpc::AsyncUnaryCall<global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.ReconcileSiteResponseDto> ReconcileSiteAsync(global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.ReconcileSiteRequestDto request, grpc::Metadata headers = null, global::System.DateTime? deadline = null, global::System.Threading.CancellationToken cancellationToken = default(global::System.Threading.CancellationToken))
{
return ReconcileSiteAsync(request, new grpc::CallOptions(headers, deadline, cancellationToken));
}
/// <summary>
/// Node-startup self-heal: the node's local deployed inventory in, fetch
/// tokens for whatever it is missing or stale out.
/// </summary>
/// <param name="request">The request to send to the server.</param>
/// <param name="options">The options for the call.</param>
/// <returns>The call object.</returns>
[global::System.CodeDom.Compiler.GeneratedCode("grpc_csharp_plugin", null)]
public virtual grpc::AsyncUnaryCall<global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.ReconcileSiteResponseDto> ReconcileSiteAsync(global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.ReconcileSiteRequestDto request, grpc::CallOptions options)
{
return CallInvoker.AsyncUnaryCall(__Method_ReconcileSite, null, options, request);
}
/// <summary>
/// Periodic site health report (30 s cadence). The ack makes delivery
/// observable end-to-end so the sender can restore its per-interval counters
/// when a report is lost.
/// </summary>
/// <param name="request">The request to send to the server.</param>
/// <param name="headers">The initial metadata to send with the call. This parameter is optional.</param>
/// <param name="deadline">An optional deadline for the call. The call will be cancelled if deadline is hit.</param>
/// <param name="cancellationToken">An optional token for canceling the call.</param>
/// <returns>The response received from the server.</returns>
[global::System.CodeDom.Compiler.GeneratedCode("grpc_csharp_plugin", null)]
public virtual global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.SiteHealthReportAckDto ReportSiteHealth(global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.SiteHealthReportDto request, grpc::Metadata headers = null, global::System.DateTime? deadline = null, global::System.Threading.CancellationToken cancellationToken = default(global::System.Threading.CancellationToken))
{
return ReportSiteHealth(request, new grpc::CallOptions(headers, deadline, cancellationToken));
}
/// <summary>
/// Periodic site health report (30 s cadence). The ack makes delivery
/// observable end-to-end so the sender can restore its per-interval counters
/// when a report is lost.
/// </summary>
/// <param name="request">The request to send to the server.</param>
/// <param name="options">The options for the call.</param>
/// <returns>The response received from the server.</returns>
[global::System.CodeDom.Compiler.GeneratedCode("grpc_csharp_plugin", null)]
public virtual global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.SiteHealthReportAckDto ReportSiteHealth(global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.SiteHealthReportDto request, grpc::CallOptions options)
{
return CallInvoker.BlockingUnaryCall(__Method_ReportSiteHealth, null, options, request);
}
/// <summary>
/// Periodic site health report (30 s cadence). The ack makes delivery
/// observable end-to-end so the sender can restore its per-interval counters
/// when a report is lost.
/// </summary>
/// <param name="request">The request to send to the server.</param>
/// <param name="headers">The initial metadata to send with the call. This parameter is optional.</param>
/// <param name="deadline">An optional deadline for the call. The call will be cancelled if deadline is hit.</param>
/// <param name="cancellationToken">An optional token for canceling the call.</param>
/// <returns>The call object.</returns>
[global::System.CodeDom.Compiler.GeneratedCode("grpc_csharp_plugin", null)]
public virtual grpc::AsyncUnaryCall<global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.SiteHealthReportAckDto> ReportSiteHealthAsync(global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.SiteHealthReportDto request, grpc::Metadata headers = null, global::System.DateTime? deadline = null, global::System.Threading.CancellationToken cancellationToken = default(global::System.Threading.CancellationToken))
{
return ReportSiteHealthAsync(request, new grpc::CallOptions(headers, deadline, cancellationToken));
}
/// <summary>
/// Periodic site health report (30 s cadence). The ack makes delivery
/// observable end-to-end so the sender can restore its per-interval counters
/// when a report is lost.
/// </summary>
/// <param name="request">The request to send to the server.</param>
/// <param name="options">The options for the call.</param>
/// <returns>The call object.</returns>
[global::System.CodeDom.Compiler.GeneratedCode("grpc_csharp_plugin", null)]
public virtual grpc::AsyncUnaryCall<global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.SiteHealthReportAckDto> ReportSiteHealthAsync(global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.SiteHealthReportDto request, grpc::CallOptions options)
{
return CallInvoker.AsyncUnaryCall(__Method_ReportSiteHealth, null, options, request);
}
/// <summary>
/// Application heartbeat. Returns Empty because the message is
/// fire-and-forget: nothing on the site consumes a reply, and a failure here
/// must never surface as a fault on the heartbeat timer path.
/// </summary>
/// <param name="request">The request to send to the server.</param>
/// <param name="headers">The initial metadata to send with the call. This parameter is optional.</param>
/// <param name="deadline">An optional deadline for the call. The call will be cancelled if deadline is hit.</param>
/// <param name="cancellationToken">An optional token for canceling the call.</param>
/// <returns>The response received from the server.</returns>
[global::System.CodeDom.Compiler.GeneratedCode("grpc_csharp_plugin", null)]
public virtual global::Google.Protobuf.WellKnownTypes.Empty Heartbeat(global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.HeartbeatDto request, grpc::Metadata headers = null, global::System.DateTime? deadline = null, global::System.Threading.CancellationToken cancellationToken = default(global::System.Threading.CancellationToken))
{
return Heartbeat(request, new grpc::CallOptions(headers, deadline, cancellationToken));
}
/// <summary>
/// Application heartbeat. Returns Empty because the message is
/// fire-and-forget: nothing on the site consumes a reply, and a failure here
/// must never surface as a fault on the heartbeat timer path.
/// </summary>
/// <param name="request">The request to send to the server.</param>
/// <param name="options">The options for the call.</param>
/// <returns>The response received from the server.</returns>
[global::System.CodeDom.Compiler.GeneratedCode("grpc_csharp_plugin", null)]
public virtual global::Google.Protobuf.WellKnownTypes.Empty Heartbeat(global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.HeartbeatDto request, grpc::CallOptions options)
{
return CallInvoker.BlockingUnaryCall(__Method_Heartbeat, null, options, request);
}
/// <summary>
/// Application heartbeat. Returns Empty because the message is
/// fire-and-forget: nothing on the site consumes a reply, and a failure here
/// must never surface as a fault on the heartbeat timer path.
/// </summary>
/// <param name="request">The request to send to the server.</param>
/// <param name="headers">The initial metadata to send with the call. This parameter is optional.</param>
/// <param name="deadline">An optional deadline for the call. The call will be cancelled if deadline is hit.</param>
/// <param name="cancellationToken">An optional token for canceling the call.</param>
/// <returns>The call object.</returns>
[global::System.CodeDom.Compiler.GeneratedCode("grpc_csharp_plugin", null)]
public virtual grpc::AsyncUnaryCall<global::Google.Protobuf.WellKnownTypes.Empty> HeartbeatAsync(global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.HeartbeatDto request, grpc::Metadata headers = null, global::System.DateTime? deadline = null, global::System.Threading.CancellationToken cancellationToken = default(global::System.Threading.CancellationToken))
{
return HeartbeatAsync(request, new grpc::CallOptions(headers, deadline, cancellationToken));
}
/// <summary>
/// Application heartbeat. Returns Empty because the message is
/// fire-and-forget: nothing on the site consumes a reply, and a failure here
/// must never surface as a fault on the heartbeat timer path.
/// </summary>
/// <param name="request">The request to send to the server.</param>
/// <param name="options">The options for the call.</param>
/// <returns>The call object.</returns>
[global::System.CodeDom.Compiler.GeneratedCode("grpc_csharp_plugin", null)]
public virtual grpc::AsyncUnaryCall<global::Google.Protobuf.WellKnownTypes.Empty> HeartbeatAsync(global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.HeartbeatDto request, grpc::CallOptions options)
{
return CallInvoker.AsyncUnaryCall(__Method_Heartbeat, null, options, request);
}
/// <summary>Creates a new instance of client from given <c>ClientBaseConfiguration</c>.</summary>
[global::System.CodeDom.Compiler.GeneratedCode("grpc_csharp_plugin", null)]
protected override CentralControlServiceClient NewInstance(ClientBaseConfiguration configuration)
{
return new CentralControlServiceClient(configuration);
}
}
/// <summary>Creates service definition that can be registered with a server</summary>
/// <param name="serviceImpl">An object implementing the server-side handling logic.</param>
[global::System.CodeDom.Compiler.GeneratedCode("grpc_csharp_plugin", null)]
public static grpc::ServerServiceDefinition BindService(CentralControlServiceBase serviceImpl)
{
return grpc::ServerServiceDefinition.CreateBuilder()
.AddMethod(__Method_SubmitNotification, serviceImpl.SubmitNotification)
.AddMethod(__Method_QueryNotificationStatus, serviceImpl.QueryNotificationStatus)
.AddMethod(__Method_IngestAuditEvents, serviceImpl.IngestAuditEvents)
.AddMethod(__Method_IngestCachedTelemetry, serviceImpl.IngestCachedTelemetry)
.AddMethod(__Method_ReconcileSite, serviceImpl.ReconcileSite)
.AddMethod(__Method_ReportSiteHealth, serviceImpl.ReportSiteHealth)
.AddMethod(__Method_Heartbeat, serviceImpl.Heartbeat).Build();
}
/// <summary>Register service method with a service binder with or without implementation. Useful when customizing the service binding logic.
/// Note: this method is part of an experimental API that can change or be removed without any prior notice.</summary>
/// <param name="serviceBinder">Service methods will be bound by calling <c>AddMethod</c> on this object.</param>
/// <param name="serviceImpl">An object implementing the server-side handling logic.</param>
[global::System.CodeDom.Compiler.GeneratedCode("grpc_csharp_plugin", null)]
public static void BindService(grpc::ServiceBinderBase serviceBinder, CentralControlServiceBase serviceImpl)
{
serviceBinder.AddMethod(__Method_SubmitNotification, serviceImpl == null ? null : new grpc::UnaryServerMethod<global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.NotificationSubmitDto, global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.NotificationSubmitAckDto>(serviceImpl.SubmitNotification));
serviceBinder.AddMethod(__Method_QueryNotificationStatus, serviceImpl == null ? null : new grpc::UnaryServerMethod<global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.NotificationStatusQueryDto, global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.NotificationStatusResponseDto>(serviceImpl.QueryNotificationStatus));
serviceBinder.AddMethod(__Method_IngestAuditEvents, serviceImpl == null ? null : new grpc::UnaryServerMethod<global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.AuditEventBatch, global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.IngestAck>(serviceImpl.IngestAuditEvents));
serviceBinder.AddMethod(__Method_IngestCachedTelemetry, serviceImpl == null ? null : new grpc::UnaryServerMethod<global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.CachedTelemetryBatch, global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.IngestAck>(serviceImpl.IngestCachedTelemetry));
serviceBinder.AddMethod(__Method_ReconcileSite, serviceImpl == null ? null : new grpc::UnaryServerMethod<global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.ReconcileSiteRequestDto, global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.ReconcileSiteResponseDto>(serviceImpl.ReconcileSite));
serviceBinder.AddMethod(__Method_ReportSiteHealth, serviceImpl == null ? null : new grpc::UnaryServerMethod<global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.SiteHealthReportDto, global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.SiteHealthReportAckDto>(serviceImpl.ReportSiteHealth));
serviceBinder.AddMethod(__Method_Heartbeat, serviceImpl == null ? null : new grpc::UnaryServerMethod<global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.HeartbeatDto, global::Google.Protobuf.WellKnownTypes.Empty>(serviceImpl.Heartbeat));
}
}
}
#endregion
@@ -1,11 +1,67 @@
namespace ZB.MOM.WW.ScadaBridge.Communication;
/// Which transport carries the seven site→central control messages. Selected per node by
/// <c>ScadaBridge:Communication:CentralTransport</c>; the migration ships with
/// <see cref="Akka"/> as the default so nothing flips until a node opts in.
/// </summary>
public enum CentralTransportMode
{
/// <summary>Akka <c>ClusterClient</c> — the transport in production today, and the default.</summary>
Akka = 0,
/// <summary>gRPC dial of the central <c>CentralControlService</c> (Phase 1A migration target).</summary>
Grpc = 1,
}
/// <summary>
/// Selects the transport the central→site command plane rides on. The Akka
/// per-site <c>ClusterClient</c> path is the default until the gRPC cutover
/// (ClusterClient→gRPC migration, Phase 1B); flipping to <see cref="Grpc"/> is the
/// rollback-by-flag switch.
/// </summary>
public enum SiteTransportKind
{
/// <summary>Route <c>SiteEnvelope</c>s through the per-site Akka <c>ClusterClient</c> (today's default).</summary>
Akka,
/// <summary>Route <c>SiteEnvelope</c>s over the site <c>SiteCommandService</c> gRPC plane.</summary>
Grpc
}
/// <summary>
/// Configuration options for central-site communication, including per-pattern
/// timeouts and transport heartbeat settings.
/// </summary>
public class CommunicationOptions
{
/// <summary>
/// Which transport carries the site→central control messages. Default <see cref="CentralTransportMode.Akka"/>
/// (ClusterClient) — coexistence rule: a node flips to gRPC only by setting this to <c>Grpc</c>,
/// and rollback is flipping it back. Selecting <c>Grpc</c> requires <see cref="CentralGrpcEndpoints"/>.
/// </summary>
public CentralTransportMode CentralTransport { get; set; } = CentralTransportMode.Akka;
/// <summary>
/// Central control-plane gRPC endpoints (preferred first), e.g.
/// <c>["http://scadabridge-central-a:8083", "http://scadabridge-central-b:8083"]</c>. Dialled by
/// <see cref="Grpc.CentralChannelProvider"/> with sticky failover/failback. Required when
/// <see cref="CentralTransport"/> is <see cref="CentralTransportMode.Grpc"/>, ignored otherwise.
/// </summary>
/// <remarks>
/// Sites reach central by container/host name, NOT via Traefik (which is HTTP/1 only; gRPC is
/// h2c on the central node's dedicated <c>CentralGrpcPort</c>).
/// </remarks>
public List<string> CentralGrpcEndpoints { get; set; } = new();
/// <summary>
/// Which transport the central→site command plane uses. Default <see cref="SiteTransportKind.Akka"/>
/// (the per-site ClusterClient path) — flipping to <see cref="SiteTransportKind.Grpc"/> moves
/// every <c>SiteEnvelope</c> onto the site <c>SiteCommandService</c> gRPC plane. Selected inside
/// <c>CentralCommunicationActor</c>; <c>CommunicationService</c> and <c>SiteCallAuditActor</c>
/// are unchanged either way. Rollback at any point = flip this back to <c>Akka</c>.
/// </summary>
public SiteTransportKind SiteTransport { get; set; } = SiteTransportKind.Akka;
/// <summary>Timeout for deployment commands (typically longest due to apply logic).</summary>
public TimeSpan DeploymentTimeout { get; set; } = TimeSpan.FromMinutes(2);
@@ -66,6 +66,19 @@ public sealed class CommunicationOptionsValidator : OptionsValidatorBase<Communi
builder.RequireThat(options.GrpcMaxConcurrentStreams > 0,
$"ScadaBridge:Communication:GrpcMaxConcurrentStreams must be positive (was {options.GrpcMaxConcurrentStreams}).");
// The gRPC site→central transport needs at least one central endpoint to dial. Only
// enforced when that transport is selected — the default Akka path ignores the list, so a
// node on ClusterClient must not be forced to declare gRPC endpoints it never uses.
if (options.CentralTransport == CentralTransportMode.Grpc)
{
builder.RequireThat(
options.CentralGrpcEndpoints.Count > 0
&& options.CentralGrpcEndpoints.All(e => !string.IsNullOrWhiteSpace(e)),
"ScadaBridge:Communication:CentralGrpcEndpoints must list at least one non-empty "
+ "central gRPC endpoint when CentralTransport is Grpc "
+ $"(was {options.CentralGrpcEndpoints.Count} entr{(options.CentralGrpcEndpoints.Count == 1 ? "y" : "ies")}).");
}
// ── Aggregated live alarm cache (plan #10, Task 6) ───────────────────────
// Linger drives a Timer dueTime; TimeSpan.Zero is valid (stop the aggregator
// immediately when the last viewer leaves), only a negative value is invalid.
@@ -0,0 +1,274 @@
using Google.Protobuf.WellKnownTypes;
using Grpc.Core;
using Grpc.Net.Client;
using Microsoft.Extensions.Logging;
namespace ZB.MOM.WW.ScadaBridge.Communication.Grpc;
/// <summary>
/// A pair (or more) of gRPC channels to the central cluster's control-plane nodes, with the
/// sticky-failover + background-failback policy of the design (§3.5). The site holds one channel
/// per central endpoint, prefers the first, and stays on it until a call proves it unreachable —
/// only then flipping to the next, and only <see cref="StatusCode.Unavailable"/> / connect
/// failures count (a <see cref="StatusCode.DeadlineExceeded"/> never flips or retries, because the
/// call may have run).
/// </summary>
/// <remarks>
/// <para>
/// <b>Sticky.</b> All calls go to the current channel; a healthy preferred endpoint never
/// ping-pongs. <see cref="ReportUnavailable"/> flips to the next endpoint (round-robin) when the
/// caller sees the current one refuse a connection.
/// </para>
/// <para>
/// <b>Failback.</b> While off the preferred endpoint a background probe (a cheap <c>Heartbeat</c>
/// ping — always answered, even by a not-yet-ready node) re-checks the preferred one. On the first
/// success new calls return to it; in-flight calls finish where they are. The probe is
/// event-driven: it arms on a flip and stops the moment we are back on the preferred endpoint, so
/// a steady healthy pair spends no cycles. Its cadence backs off exponentially — 1 s, doubling,
/// capped at 60 s — while the preferred endpoint stays down, so a genuinely dead node is not
/// probed hard.
/// </para>
/// <para>
/// <b>Auth.</b> Every channel carries the site's own preshared key and its
/// <c>x-scadabridge-site</c> identity via <see cref="ControlPlaneCredentials"/> — the same
/// insecure-h2c call-credentials shape the streaming client uses. The <paramref name="handlerFactory"/>
/// seam lets a test point a channel at an in-process <c>TestServer</c>; production uses a
/// keepalive-configured <see cref="SocketsHttpHandler"/>.
/// </para>
/// </remarks>
public sealed class CentralChannelProvider : IDisposable
{
private static readonly TimeSpan DefaultBackoffBase = TimeSpan.FromSeconds(1);
private static readonly TimeSpan DefaultBackoffCap = TimeSpan.FromSeconds(60);
private static readonly TimeSpan DefaultProbeDeadline = TimeSpan.FromSeconds(5);
private readonly IReadOnlyList<string> _endpoints;
private readonly GrpcChannel[] _channels;
private readonly CentralControlService.CentralControlServiceClient[] _clients;
private readonly ILogger _logger;
private readonly string _siteId;
private readonly TimeSpan _backoffBase;
private readonly TimeSpan _backoffCap;
private readonly TimeSpan _probeDeadline;
private readonly Timer? _failbackTimer;
private readonly object _gate = new();
private volatile int _current; // preferred == 0
private int _consecutiveProbeFailures;
private bool _disposed;
/// <summary>Creates the provider and opens one channel per endpoint.</summary>
/// <param name="endpoints">Central control-plane endpoints, preferred first (index 0). Must be non-empty.</param>
/// <param name="pskProvider">Resolves this site's preshared key (site-side: a single-key provider).</param>
/// <param name="siteId">This site's identity, sent as the <c>x-scadabridge-site</c> header.</param>
/// <param name="options">Communication options supplying gRPC keepalive settings.</param>
/// <param name="logger">Logger for flip/failback diagnostics.</param>
/// <param name="handlerFactory">Test seam: per-endpoint <see cref="HttpMessageHandler"/>; null uses a production socket handler.</param>
/// <param name="probeDeadline">Deadline for a failback probe. Null uses 5 s.</param>
/// <param name="backoffBase">Initial failback-probe backoff. Null uses 1 s.</param>
/// <param name="backoffCap">Maximum failback-probe backoff. Null uses 60 s.</param>
public CentralChannelProvider(
IReadOnlyList<string> endpoints,
ISitePskProvider pskProvider,
string siteId,
CommunicationOptions options,
ILogger logger,
Func<string, HttpMessageHandler>? handlerFactory = null,
TimeSpan? probeDeadline = null,
TimeSpan? backoffBase = null,
TimeSpan? backoffCap = null)
{
ArgumentNullException.ThrowIfNull(endpoints);
ArgumentNullException.ThrowIfNull(pskProvider);
ArgumentException.ThrowIfNullOrWhiteSpace(siteId);
ArgumentNullException.ThrowIfNull(options);
ArgumentNullException.ThrowIfNull(logger);
if (endpoints.Count == 0)
{
throw new ArgumentException("At least one central gRPC endpoint is required.", nameof(endpoints));
}
_endpoints = endpoints;
_logger = logger;
_siteId = siteId;
_backoffBase = backoffBase ?? DefaultBackoffBase;
_backoffCap = backoffCap ?? DefaultBackoffCap;
_probeDeadline = probeDeadline ?? DefaultProbeDeadline;
_channels = new GrpcChannel[endpoints.Count];
_clients = new CentralControlService.CentralControlServiceClient[endpoints.Count];
for (var i = 0; i < endpoints.Count; i++)
{
var channelOptions = new GrpcChannelOptions
{
HttpHandler = handlerFactory?.Invoke(endpoints[i]) ?? new SocketsHttpHandler
{
KeepAlivePingDelay = options.GrpcKeepAlivePingDelay,
KeepAlivePingTimeout = options.GrpcKeepAlivePingTimeout,
KeepAlivePingPolicy = HttpKeepAlivePingPolicy.Always,
EnableMultipleHttp2Connections = true,
},
}.WithSiteCredentials(pskProvider, siteId);
_channels[i] = GrpcChannel.ForAddress(endpoints[i], channelOptions);
_clients[i] = new CentralControlService.CentralControlServiceClient(_channels[i]);
}
// Only a multi-endpoint pair can ever fail over, so a lone endpoint needs no probe.
if (endpoints.Count > 1)
{
_failbackTimer = new Timer(_ => _ = FailbackTickAsync(), null, Timeout.Infinite, Timeout.Infinite);
}
}
/// <summary>The number of endpoints in the pair.</summary>
public int EndpointCount => _endpoints.Count;
/// <summary>The index of the endpoint calls are currently routed to (preferred == 0).</summary>
public int CurrentIndex => _current;
/// <summary>The endpoint address calls are currently routed to.</summary>
public string CurrentEndpoint => _endpoints[_current];
/// <summary>
/// The endpoint index and client calls should use right now. Captured together so a caller can
/// tell <see cref="ReportUnavailable"/> exactly which endpoint failed even if a concurrent flip
/// has already moved <see cref="CurrentIndex"/>.
/// </summary>
/// <returns>The current endpoint index and its client.</returns>
public (int Index, CentralControlService.CentralControlServiceClient Client) Current()
{
var idx = _current;
return (idx, _clients[idx]);
}
/// <summary>
/// Reports that the endpoint at <paramref name="failedIndex"/> refused a connection (an
/// <see cref="StatusCode.Unavailable"/> / connect failure). If it is still the current endpoint
/// and another exists, flips to the next one and — when now off the preferred endpoint — arms
/// the failback probe. Idempotent under a concurrent flip: a stale index is ignored.
/// </summary>
/// <param name="failedIndex">The endpoint index the caller's failed call used.</param>
public void ReportUnavailable(int failedIndex)
{
if (_endpoints.Count < 2)
{
return; // nothing to fail over to
}
lock (_gate)
{
if (_disposed || failedIndex != _current)
{
return; // a concurrent flip already moved us; do not double-flip
}
var next = (failedIndex + 1) % _endpoints.Count;
_current = next;
_logger.LogWarning(
"Central control-plane endpoint {Failed} is unavailable; site {SiteId} failed over to {Next}.",
_endpoints[failedIndex], _siteId, _endpoints[next]);
if (_current != 0)
{
_consecutiveProbeFailures = 0;
ArmFailback(_backoffBase);
}
}
}
private void ArmFailback(TimeSpan due)
{
if (_disposed)
{
return;
}
_failbackTimer?.Change(due, Timeout.InfiniteTimeSpan);
}
private async Task FailbackTickAsync()
{
int currentAtTick = _current;
if (_disposed || currentAtTick == 0)
{
return; // already back on the preferred endpoint (or shutting down)
}
var preferred = _clients[0];
try
{
await preferred.HeartbeatAsync(
new HeartbeatDto
{
SiteId = _siteId,
NodeHostname = "failback-probe",
IsActive = false,
Timestamp = Timestamp.FromDateTimeOffset(DateTimeOffset.UtcNow),
},
deadline: DateTime.UtcNow.Add(_probeDeadline)).ConfigureAwait(false);
// The preferred endpoint answered — return new calls to it.
lock (_gate)
{
if (_disposed)
{
return;
}
_current = 0;
_consecutiveProbeFailures = 0;
}
_logger.LogInformation(
"Central control-plane preferred endpoint {Preferred} is reachable again; site {SiteId} failed back.",
_endpoints[0], _siteId);
}
catch (Exception ex)
{
lock (_gate)
{
if (_disposed || _current == 0)
{
return;
}
_consecutiveProbeFailures++;
var backoff = NextBackoff(_consecutiveProbeFailures);
_logger.LogDebug(ex,
"Failback probe of preferred central endpoint {Preferred} failed; re-probing in {Backoff}.",
_endpoints[0], backoff);
ArmFailback(backoff);
}
}
}
private TimeSpan NextBackoff(int failures)
{
// 1 s, doubling, capped at 60 s. Guard the shift against overflow for a long outage.
var exponent = Math.Min(failures - 1, 20);
var scaled = _backoffBase.Ticks * (1L << exponent);
var cap = _backoffCap.Ticks;
return TimeSpan.FromTicks(scaled >= cap || scaled < 0 ? cap : scaled);
}
/// <inheritdoc />
public void Dispose()
{
lock (_gate)
{
if (_disposed)
{
return;
}
_disposed = true;
}
_failbackTimer?.Dispose();
foreach (var channel in _channels)
{
channel.Dispose();
}
}
}
@@ -0,0 +1,764 @@
using ZB.MOM.WW.ScadaBridge.Commons.Messages.Audit;
using ZB.MOM.WW.ScadaBridge.Commons.Messages.Deployment;
using ZB.MOM.WW.ScadaBridge.Commons.Messages.Health;
using ZB.MOM.WW.ScadaBridge.Commons.Messages.Notification;
using ZB.MOM.WW.ScadaBridge.Commons.Types;
using ZB.MOM.WW.ScadaBridge.Commons.Types.Enums;
using Timestamp = Google.Protobuf.WellKnownTypes.Timestamp;
namespace ZB.MOM.WW.ScadaBridge.Communication.Grpc;
/// <summary>
/// Canonical bridge between the seven in-process messages the site sends to central
/// and the wire format of <c>CentralControlService</c> (<c>Protos/central_control.proto</c>).
/// </summary>
/// <remarks>
/// <para>
/// The seven pairs mirror, one for one, the seven messages
/// <c>SiteCommunicationActor</c> forwards to <c>/user/central-communication</c> over
/// Akka <c>ClusterClient</c> today. Both transports carry the SAME message types
/// end-to-end — central's handlers are untouched by the migration — so this mapper is
/// the only place the two representations meet, and a field that does not survive a
/// round-trip here is a field the gRPC transport silently drops.
/// </para>
/// <para>
/// <b>Ingest is deliberately absent from the message list.</b> <c>IngestAuditEvents</c>
/// and <c>IngestCachedTelemetry</c> reuse the <c>AuditEventBatch</c> /
/// <c>CachedTelemetryBatch</c> / <c>IngestAck</c> messages already defined for the
/// site-hosted <c>SiteStreamService</c>, so the per-row work is delegated to the
/// existing <see cref="AuditEventDtoMapper"/> and <see cref="SiteCallDtoMapper"/>; only
/// the batch/ack envelopes are assembled here.
/// </para>
///
/// <para><b>Conventions, applied uniformly across every method below.</b></para>
/// <list type="bullet">
/// <item>
/// <b>Nullable strings ↔ empty strings.</b> A proto3 scalar string cannot be absent,
/// so a null .NET string is written as <see cref="string.Empty"/> and an empty wire
/// string is read back as <see langword="null"/>. This is the convention already in
/// force on <see cref="AuditEventDtoMapper"/>, and it is why no field on this wire
/// may distinguish "null" from "deliberately empty".
/// </item>
/// <item>
/// <b>Nullable <see cref="Guid"/> ↔ string.</b> Execution ids travel as their "D"
/// string form; the empty string means <see langword="null"/>. A malformed non-empty
/// value throws out of <c>FromDto</c> rather than degrading to null — a corrupt
/// correlation id must not be laundered into "no correlation".
/// </item>
/// <item>
/// <b>Nullable numbers and booleans ↔ protobuf wrapper types.</b>
/// <c>Int32Value</c>/<c>Int64Value</c>/<c>DoubleValue</c>/<c>BoolValue</c> preserve
/// true null. Several health gauges (<c>LocalDbOplogBacklog</c>,
/// <c>LocalDbReplicationConnected</c>) are documented as "null means unknown, and
/// that is NOT the same as zero/false"; collapsing them to a bare scalar would
/// report a broken replication pair as healthy.
/// </item>
/// <item>
/// <b>Nullable collections ↔ wrapper messages.</b> proto3 cannot express presence on
/// a <c>repeated</c> or <c>map</c> field, so the three nullable
/// <see cref="SiteHealthReport"/> collections travel inside single-field wrapper
/// messages (<c>ConnectionEndpointMapDto</c>, <c>TagQualityMapDto</c>,
/// <c>NodeStatusListDto</c>). An absent wrapper is null; a present-but-empty wrapper
/// is an empty collection.
/// </item>
/// <item>
/// <b><see cref="DateTimeOffset"/> normalizes to a UTC instant.</b> A protobuf
/// <c>Timestamp</c> is an instant, not an offset-qualified local time, so the offset
/// component is dropped and the value round-trips with <c>Offset == TimeSpan.Zero</c>.
/// Every producer in this system stamps UTC (the repo-wide invariant; e.g.
/// <c>Notify.Send</c> uses <c>DateTimeOffset.UtcNow</c>), so this is lossless in
/// practice and the instant is preserved regardless.
/// </item>
/// </list>
/// </remarks>
public static class CentralControlDtoMapper
{
// -----------------------------------------------------------------------
// Notification Outbox (#21)
// -----------------------------------------------------------------------
/// <summary>Projects a <see cref="NotificationSubmit"/> onto its wire DTO.</summary>
/// <param name="msg">The notification submission to project.</param>
/// <returns>The wire-format DTO; null strings and null execution ids collapse to empty strings.</returns>
public static NotificationSubmitDto ToDto(NotificationSubmit msg)
{
ArgumentNullException.ThrowIfNull(msg);
return new NotificationSubmitDto
{
NotificationId = msg.NotificationId,
ListName = msg.ListName,
Subject = msg.Subject,
Body = msg.Body,
SourceSiteId = msg.SourceSiteId,
SourceInstanceId = msg.SourceInstanceId ?? string.Empty,
SourceScript = msg.SourceScript ?? string.Empty,
SiteEnqueuedAt = Timestamp.FromDateTimeOffset(msg.SiteEnqueuedAt),
OriginExecutionId = GuidToWire(msg.OriginExecutionId),
OriginParentExecutionId = GuidToWire(msg.OriginParentExecutionId),
SourceNode = msg.SourceNode ?? string.Empty,
};
}
/// <summary>Reconstructs a <see cref="NotificationSubmit"/> from its wire DTO.</summary>
/// <param name="dto">The wire-format DTO to reconstruct.</param>
/// <returns>The in-process message; empty strings rehydrate as null.</returns>
public static NotificationSubmit FromDto(NotificationSubmitDto dto)
{
ArgumentNullException.ThrowIfNull(dto);
return new NotificationSubmit(
NotificationId: dto.NotificationId,
ListName: dto.ListName,
Subject: dto.Subject,
Body: dto.Body,
SourceSiteId: dto.SourceSiteId,
SourceInstanceId: NullIfEmpty(dto.SourceInstanceId),
SourceScript: NullIfEmpty(dto.SourceScript),
SiteEnqueuedAt: dto.SiteEnqueuedAt.ToDateTimeOffset(),
OriginExecutionId: GuidFromWire(dto.OriginExecutionId),
OriginParentExecutionId: GuidFromWire(dto.OriginParentExecutionId),
SourceNode: NullIfEmpty(dto.SourceNode));
}
/// <summary>Projects a <see cref="NotificationSubmitAck"/> onto its wire DTO.</summary>
/// <param name="msg">The ack to project.</param>
/// <returns>The wire-format DTO; a null error collapses to an empty string.</returns>
public static NotificationSubmitAckDto ToDto(NotificationSubmitAck msg)
{
ArgumentNullException.ThrowIfNull(msg);
return new NotificationSubmitAckDto
{
NotificationId = msg.NotificationId,
Accepted = msg.Accepted,
Error = msg.Error ?? string.Empty,
};
}
/// <summary>Reconstructs a <see cref="NotificationSubmitAck"/> from its wire DTO.</summary>
/// <param name="dto">The wire-format DTO to reconstruct.</param>
/// <returns>The in-process ack; an empty error rehydrates as null.</returns>
public static NotificationSubmitAck FromDto(NotificationSubmitAckDto dto)
{
ArgumentNullException.ThrowIfNull(dto);
return new NotificationSubmitAck(
NotificationId: dto.NotificationId,
Accepted: dto.Accepted,
Error: NullIfEmpty(dto.Error));
}
/// <summary>Projects a <see cref="NotificationStatusQuery"/> onto its wire DTO.</summary>
/// <param name="msg">The status query to project.</param>
/// <returns>The wire-format DTO.</returns>
public static NotificationStatusQueryDto ToDto(NotificationStatusQuery msg)
{
ArgumentNullException.ThrowIfNull(msg);
return new NotificationStatusQueryDto
{
CorrelationId = msg.CorrelationId,
NotificationId = msg.NotificationId,
};
}
/// <summary>Reconstructs a <see cref="NotificationStatusQuery"/> from its wire DTO.</summary>
/// <param name="dto">The wire-format DTO to reconstruct.</param>
/// <returns>The in-process query.</returns>
public static NotificationStatusQuery FromDto(NotificationStatusQueryDto dto)
{
ArgumentNullException.ThrowIfNull(dto);
return new NotificationStatusQuery(
CorrelationId: dto.CorrelationId,
NotificationId: dto.NotificationId);
}
/// <summary>Projects a <see cref="NotificationStatusResponse"/> onto its wire DTO.</summary>
/// <param name="msg">The status response to project.</param>
/// <returns>The wire-format DTO; a null delivery timestamp leaves the field unset.</returns>
public static NotificationStatusResponseDto ToDto(NotificationStatusResponse msg)
{
ArgumentNullException.ThrowIfNull(msg);
var dto = new NotificationStatusResponseDto
{
CorrelationId = msg.CorrelationId,
Found = msg.Found,
Status = msg.Status,
RetryCount = msg.RetryCount,
LastError = msg.LastError ?? string.Empty,
};
if (msg.DeliveredAt.HasValue)
{
dto.DeliveredAt = Timestamp.FromDateTimeOffset(msg.DeliveredAt.Value);
}
return dto;
}
/// <summary>Reconstructs a <see cref="NotificationStatusResponse"/> from its wire DTO.</summary>
/// <param name="dto">The wire-format DTO to reconstruct.</param>
/// <returns>The in-process response; an unset delivery timestamp rehydrates as null.</returns>
public static NotificationStatusResponse FromDto(NotificationStatusResponseDto dto)
{
ArgumentNullException.ThrowIfNull(dto);
return new NotificationStatusResponse(
CorrelationId: dto.CorrelationId,
Found: dto.Found,
Status: dto.Status,
RetryCount: dto.RetryCount,
LastError: NullIfEmpty(dto.LastError),
DeliveredAt: dto.DeliveredAt?.ToDateTimeOffset());
}
// -----------------------------------------------------------------------
// Audit Log (#23) ingest — envelopes only; rows go through the existing mappers
// -----------------------------------------------------------------------
/// <summary>
/// Projects an <see cref="IngestAuditEventsCommand"/> onto the shared
/// <see cref="AuditEventBatch"/> wire message.
/// </summary>
/// <remarks>
/// The per-row projection is <see cref="AuditEventDtoMapper.ToDto"/>, which is lossy
/// by design: <c>ForwardState</c> is site-local storage state and <c>IngestedAtUtc</c>
/// is stamped centrally at ingest, so neither travels.
/// </remarks>
/// <param name="cmd">The ingest command to project.</param>
/// <returns>A batch carrying one DTO per audit event, in order.</returns>
public static AuditEventBatch ToDto(IngestAuditEventsCommand cmd)
{
ArgumentNullException.ThrowIfNull(cmd);
var batch = new AuditEventBatch();
foreach (var evt in cmd.Events)
{
batch.Events.Add(AuditEventDtoMapper.ToDto(evt));
}
return batch;
}
/// <summary>
/// Reconstructs an <see cref="IngestAuditEventsCommand"/> from the shared
/// <see cref="AuditEventBatch"/> wire message — the shape central's
/// <c>CentralCommunicationActor</c> already handles.
/// </summary>
/// <param name="batch">The wire batch to reconstruct.</param>
/// <returns>The in-process ingest command.</returns>
public static IngestAuditEventsCommand FromDto(AuditEventBatch batch)
{
ArgumentNullException.ThrowIfNull(batch);
var events = new List<ZB.MOM.WW.Audit.AuditEvent>(batch.Events.Count);
foreach (var dto in batch.Events)
{
events.Add(AuditEventDtoMapper.FromDto(dto));
}
return new IngestAuditEventsCommand(events);
}
/// <summary>
/// Projects an <see cref="IngestCachedTelemetryCommand"/> onto the shared
/// <see cref="CachedTelemetryBatch"/> wire message.
/// </summary>
/// <param name="cmd">The cached-telemetry ingest command to project.</param>
/// <returns>A batch carrying one packet (audit row + operational row) per entry, in order.</returns>
public static CachedTelemetryBatch ToDto(IngestCachedTelemetryCommand cmd)
{
ArgumentNullException.ThrowIfNull(cmd);
var batch = new CachedTelemetryBatch();
foreach (var entry in cmd.Entries)
{
batch.Packets.Add(new CachedTelemetryPacket
{
AuditEvent = AuditEventDtoMapper.ToDto(entry.Audit),
Operational = SiteCallDtoMapper.ToDto(entry.SiteCall),
});
}
return batch;
}
/// <summary>
/// Reconstructs an <see cref="IngestCachedTelemetryCommand"/> from the shared
/// <see cref="CachedTelemetryBatch"/> wire message.
/// </summary>
/// <param name="batch">The wire batch to reconstruct.</param>
/// <returns>The in-process dual-write ingest command.</returns>
public static IngestCachedTelemetryCommand FromDto(CachedTelemetryBatch batch)
{
ArgumentNullException.ThrowIfNull(batch);
var entries = new List<CachedTelemetryEntry>(batch.Packets.Count);
foreach (var packet in batch.Packets)
{
entries.Add(new CachedTelemetryEntry(
AuditEventDtoMapper.FromDto(packet.AuditEvent),
SiteCallDtoMapper.FromDto(packet.Operational)));
}
return new IngestCachedTelemetryCommand(entries);
}
/// <summary>
/// Projects the accepted-id list of an ingest reply onto the shared
/// <see cref="IngestAck"/> wire message. Shared by both ingest RPCs — the two
/// central reply types differ only in which handler produced them.
/// </summary>
/// <param name="acceptedEventIds">Ids central considers durably persisted.</param>
/// <returns>The wire ack carrying the ids in "D" string form, in order.</returns>
public static IngestAck ToIngestAck(IReadOnlyList<Guid> acceptedEventIds)
{
ArgumentNullException.ThrowIfNull(acceptedEventIds);
var ack = new IngestAck();
foreach (var id in acceptedEventIds)
{
ack.AcceptedEventIds.Add(id.ToString());
}
return ack;
}
/// <summary>
/// Reads the accepted-id list back out of an <see cref="IngestAck"/>.
/// </summary>
/// <param name="ack">The wire ack to read.</param>
/// <returns>The accepted event ids, in wire order.</returns>
public static IReadOnlyList<Guid> FromIngestAck(IngestAck ack)
{
ArgumentNullException.ThrowIfNull(ack);
var ids = new List<Guid>(ack.AcceptedEventIds.Count);
foreach (var id in ack.AcceptedEventIds)
{
ids.Add(Guid.Parse(id));
}
return ids;
}
// -----------------------------------------------------------------------
// Startup reconciliation
// -----------------------------------------------------------------------
/// <summary>Projects a <see cref="ReconcileSiteRequest"/> onto its wire DTO.</summary>
/// <param name="msg">The reconcile request to project.</param>
/// <returns>The wire-format DTO carrying the node's local name→revision-hash inventory.</returns>
public static ReconcileSiteRequestDto ToDto(ReconcileSiteRequest msg)
{
ArgumentNullException.ThrowIfNull(msg);
var dto = new ReconcileSiteRequestDto
{
SiteIdentifier = msg.SiteIdentifier,
NodeId = msg.NodeId,
};
foreach (var (name, hash) in msg.LocalNameToRevisionHash)
{
dto.LocalNameToRevisionHash[name] = hash;
}
return dto;
}
/// <summary>Reconstructs a <see cref="ReconcileSiteRequest"/> from its wire DTO.</summary>
/// <param name="dto">The wire-format DTO to reconstruct.</param>
/// <returns>The in-process reconcile request.</returns>
public static ReconcileSiteRequest FromDto(ReconcileSiteRequestDto dto)
{
ArgumentNullException.ThrowIfNull(dto);
return new ReconcileSiteRequest(
SiteIdentifier: dto.SiteIdentifier,
NodeId: dto.NodeId,
LocalNameToRevisionHash: new Dictionary<string, string>(dto.LocalNameToRevisionHash));
}
/// <summary>Projects a <see cref="ReconcileSiteResponse"/> onto its wire DTO.</summary>
/// <param name="msg">The reconcile response to project.</param>
/// <returns>The wire-format DTO carrying the gap items, orphan names and fetch base URL.</returns>
public static ReconcileSiteResponseDto ToDto(ReconcileSiteResponse msg)
{
ArgumentNullException.ThrowIfNull(msg);
var dto = new ReconcileSiteResponseDto
{
CentralFetchBaseUrl = msg.CentralFetchBaseUrl,
};
foreach (var item in msg.Gap)
{
dto.Gap.Add(new ReconcileGapItemDto
{
InstanceUniqueName = item.InstanceUniqueName,
DeploymentId = item.DeploymentId,
RevisionHash = item.RevisionHash,
IsEnabled = item.IsEnabled,
FetchToken = item.FetchToken,
});
}
dto.OrphanNames.AddRange(msg.OrphanNames);
return dto;
}
/// <summary>Reconstructs a <see cref="ReconcileSiteResponse"/> from its wire DTO.</summary>
/// <param name="dto">The wire-format DTO to reconstruct.</param>
/// <returns>The in-process reconcile response.</returns>
public static ReconcileSiteResponse FromDto(ReconcileSiteResponseDto dto)
{
ArgumentNullException.ThrowIfNull(dto);
var gap = new List<ReconcileGapItem>(dto.Gap.Count);
foreach (var item in dto.Gap)
{
gap.Add(new ReconcileGapItem(
InstanceUniqueName: item.InstanceUniqueName,
DeploymentId: item.DeploymentId,
RevisionHash: item.RevisionHash,
IsEnabled: item.IsEnabled,
FetchToken: item.FetchToken));
}
return new ReconcileSiteResponse(
Gap: gap,
OrphanNames: dto.OrphanNames.ToList(),
CentralFetchBaseUrl: dto.CentralFetchBaseUrl);
}
// -----------------------------------------------------------------------
// Health Monitoring (#11)
// -----------------------------------------------------------------------
/// <summary>Projects a <see cref="SiteHealthReport"/> onto its wire DTO.</summary>
/// <remarks>
/// The three nullable collections travel inside wrapper messages so a null stays
/// distinguishable from an empty collection; the nullable gauges travel in protobuf
/// wrapper types for the same reason.
/// </remarks>
/// <param name="msg">The health report to project.</param>
/// <returns>The wire-format DTO.</returns>
public static SiteHealthReportDto ToDto(SiteHealthReport msg)
{
ArgumentNullException.ThrowIfNull(msg);
var dto = new SiteHealthReportDto
{
SiteId = msg.SiteId,
SequenceNumber = msg.SequenceNumber,
ReportTimestamp = Timestamp.FromDateTimeOffset(msg.ReportTimestamp),
ScriptErrorCount = msg.ScriptErrorCount,
AlarmEvaluationErrorCount = msg.AlarmEvaluationErrorCount,
DeadLetterCount = msg.DeadLetterCount,
DeployedInstanceCount = msg.DeployedInstanceCount,
EnabledInstanceCount = msg.EnabledInstanceCount,
DisabledInstanceCount = msg.DisabledInstanceCount,
NodeRole = msg.NodeRole,
NodeHostname = msg.NodeHostname,
ParkedMessageCount = msg.ParkedMessageCount,
SiteAuditWriteFailures = msg.SiteAuditWriteFailures,
AuditRedactionFailure = msg.AuditRedactionFailure,
SiteEventLogWriteFailures = msg.SiteEventLogWriteFailures,
OldestParkedMessageAgeSeconds = msg.OldestParkedMessageAgeSeconds,
ScriptQueueDepth = msg.ScriptQueueDepth,
ScriptBusyThreads = msg.ScriptBusyThreads,
ScriptOldestBusyAgeSeconds = msg.ScriptOldestBusyAgeSeconds,
LocalDbReplicationConnected = msg.LocalDbReplicationConnected,
LocalDbOplogBacklog = msg.LocalDbOplogBacklog,
};
foreach (var (name, health) in msg.DataConnectionStatuses)
{
dto.DataConnectionStatuses[name] = ToDto(health);
}
foreach (var (name, resolution) in msg.TagResolutionCounts)
{
dto.TagResolutionCounts[name] = new TagResolutionStatusDto
{
TotalSubscribed = resolution.TotalSubscribed,
SuccessfullyResolved = resolution.SuccessfullyResolved,
};
}
foreach (var (name, depth) in msg.StoreAndForwardBufferDepths)
{
dto.StoreAndForwardBufferDepths[name] = depth;
}
if (msg.DataConnectionEndpoints is { } endpoints)
{
var wrapper = new ConnectionEndpointMapDto();
foreach (var (name, endpoint) in endpoints)
{
wrapper.Entries[name] = endpoint;
}
dto.DataConnectionEndpoints = wrapper;
}
if (msg.DataConnectionTagQuality is { } tagQuality)
{
var wrapper = new TagQualityMapDto();
foreach (var (name, counts) in tagQuality)
{
wrapper.Entries[name] = new TagQualityCountsDto
{
Good = counts.Good,
Bad = counts.Bad,
Uncertain = counts.Uncertain,
};
}
dto.DataConnectionTagQuality = wrapper;
}
if (msg.ClusterNodes is { } clusterNodes)
{
var wrapper = new NodeStatusListDto();
foreach (var node in clusterNodes)
{
wrapper.Nodes.Add(new NodeStatusDto
{
Hostname = node.Hostname,
IsOnline = node.IsOnline,
Role = node.Role,
});
}
dto.ClusterNodes = wrapper;
}
if (msg.SiteAuditBacklog is { } backlog)
{
var snapshot = new SiteAuditBacklogSnapshotDto
{
PendingCount = backlog.PendingCount,
OnDiskBytes = backlog.OnDiskBytes,
};
if (backlog.OldestPendingUtc.HasValue)
{
snapshot.OldestPendingUtc = Timestamp.FromDateTime(EnsureUtc(backlog.OldestPendingUtc.Value));
}
dto.SiteAuditBacklog = snapshot;
}
return dto;
}
/// <summary>Reconstructs a <see cref="SiteHealthReport"/> from its wire DTO.</summary>
/// <param name="dto">The wire-format DTO to reconstruct.</param>
/// <returns>The in-process health report; absent wrappers rehydrate as null, not as empty.</returns>
public static SiteHealthReport FromDto(SiteHealthReportDto dto)
{
ArgumentNullException.ThrowIfNull(dto);
var statuses = new Dictionary<string, ConnectionHealth>(dto.DataConnectionStatuses.Count);
foreach (var (name, health) in dto.DataConnectionStatuses)
{
statuses[name] = FromDto(health);
}
var resolution = new Dictionary<string, TagResolutionStatus>(dto.TagResolutionCounts.Count);
foreach (var (name, counts) in dto.TagResolutionCounts)
{
resolution[name] = new TagResolutionStatus(counts.TotalSubscribed, counts.SuccessfullyResolved);
}
var bufferDepths = new Dictionary<string, int>(dto.StoreAndForwardBufferDepths);
Dictionary<string, string>? endpoints = null;
if (dto.DataConnectionEndpoints is { } endpointWrapper)
{
endpoints = new Dictionary<string, string>(endpointWrapper.Entries);
}
Dictionary<string, TagQualityCounts>? tagQuality = null;
if (dto.DataConnectionTagQuality is { } tagQualityWrapper)
{
tagQuality = new Dictionary<string, TagQualityCounts>(tagQualityWrapper.Entries.Count);
foreach (var (name, counts) in tagQualityWrapper.Entries)
{
tagQuality[name] = new TagQualityCounts(counts.Good, counts.Bad, counts.Uncertain);
}
}
List<NodeStatus>? clusterNodes = null;
if (dto.ClusterNodes is { } nodeWrapper)
{
clusterNodes = new List<NodeStatus>(nodeWrapper.Nodes.Count);
foreach (var node in nodeWrapper.Nodes)
{
clusterNodes.Add(new NodeStatus(node.Hostname, node.IsOnline, node.Role));
}
}
SiteAuditBacklogSnapshot? backlog = null;
if (dto.SiteAuditBacklog is { } snapshot)
{
backlog = new SiteAuditBacklogSnapshot(
PendingCount: snapshot.PendingCount,
OldestPendingUtc: snapshot.OldestPendingUtc is null
? null
: DateTime.SpecifyKind(snapshot.OldestPendingUtc.ToDateTime(), DateTimeKind.Utc),
OnDiskBytes: snapshot.OnDiskBytes);
}
return new SiteHealthReport(
SiteId: dto.SiteId,
SequenceNumber: dto.SequenceNumber,
ReportTimestamp: dto.ReportTimestamp.ToDateTimeOffset(),
DataConnectionStatuses: statuses,
TagResolutionCounts: resolution,
ScriptErrorCount: dto.ScriptErrorCount,
AlarmEvaluationErrorCount: dto.AlarmEvaluationErrorCount,
StoreAndForwardBufferDepths: bufferDepths,
DeadLetterCount: dto.DeadLetterCount,
DeployedInstanceCount: dto.DeployedInstanceCount,
EnabledInstanceCount: dto.EnabledInstanceCount,
DisabledInstanceCount: dto.DisabledInstanceCount,
NodeRole: dto.NodeRole,
NodeHostname: dto.NodeHostname,
DataConnectionEndpoints: endpoints,
DataConnectionTagQuality: tagQuality,
ParkedMessageCount: dto.ParkedMessageCount,
ClusterNodes: clusterNodes,
SiteAuditWriteFailures: dto.SiteAuditWriteFailures,
AuditRedactionFailure: dto.AuditRedactionFailure,
SiteAuditBacklog: backlog,
SiteEventLogWriteFailures: dto.SiteEventLogWriteFailures,
OldestParkedMessageAgeSeconds: dto.OldestParkedMessageAgeSeconds)
{
// Init-only members: SiteHealthReport surfaces the scheduler and LocalDb
// gauges as init properties rather than positional parameters, so they
// cannot be passed to the constructor above.
ScriptQueueDepth = dto.ScriptQueueDepth,
ScriptBusyThreads = dto.ScriptBusyThreads,
ScriptOldestBusyAgeSeconds = dto.ScriptOldestBusyAgeSeconds,
LocalDbReplicationConnected = dto.LocalDbReplicationConnected,
LocalDbOplogBacklog = dto.LocalDbOplogBacklog,
};
}
/// <summary>Projects a <see cref="SiteHealthReportAck"/> onto its wire DTO.</summary>
/// <param name="msg">The ack to project.</param>
/// <returns>The wire-format DTO; a null error collapses to an empty string.</returns>
public static SiteHealthReportAckDto ToDto(SiteHealthReportAck msg)
{
ArgumentNullException.ThrowIfNull(msg);
return new SiteHealthReportAckDto
{
SiteId = msg.SiteId,
SequenceNumber = msg.SequenceNumber,
Accepted = msg.Accepted,
Error = msg.Error ?? string.Empty,
};
}
/// <summary>Reconstructs a <see cref="SiteHealthReportAck"/> from its wire DTO.</summary>
/// <param name="dto">The wire-format DTO to reconstruct.</param>
/// <returns>The in-process ack; an empty error rehydrates as null.</returns>
public static SiteHealthReportAck FromDto(SiteHealthReportAckDto dto)
{
ArgumentNullException.ThrowIfNull(dto);
return new SiteHealthReportAck(
SiteId: dto.SiteId,
SequenceNumber: dto.SequenceNumber,
Accepted: dto.Accepted,
Error: NullIfEmpty(dto.Error));
}
/// <summary>Projects a <see cref="HeartbeatMessage"/> onto its wire DTO.</summary>
/// <param name="msg">The heartbeat to project.</param>
/// <returns>The wire-format DTO.</returns>
public static HeartbeatDto ToDto(HeartbeatMessage msg)
{
ArgumentNullException.ThrowIfNull(msg);
return new HeartbeatDto
{
SiteId = msg.SiteId,
NodeHostname = msg.NodeHostname,
IsActive = msg.IsActive,
Timestamp = Timestamp.FromDateTimeOffset(msg.Timestamp),
};
}
/// <summary>Reconstructs a <see cref="HeartbeatMessage"/> from its wire DTO.</summary>
/// <param name="dto">The wire-format DTO to reconstruct.</param>
/// <returns>The in-process heartbeat.</returns>
public static HeartbeatMessage FromDto(HeartbeatDto dto)
{
ArgumentNullException.ThrowIfNull(dto);
return new HeartbeatMessage(
SiteId: dto.SiteId,
NodeHostname: dto.NodeHostname,
IsActive: dto.IsActive,
Timestamp: dto.Timestamp.ToDateTimeOffset());
}
/// <summary>Projects a <see cref="ConnectionHealth"/> onto its wire enum.</summary>
/// <param name="health">The connection health to project.</param>
/// <returns>The wire enum value. Never <c>ConnectionHealthUnspecified</c>.</returns>
/// <exception cref="ArgumentOutOfRangeException">
/// A <see cref="ConnectionHealth"/> value was added without extending this mapping.
/// Throwing beats a silent default: an unmapped state would otherwise be reported as
/// whichever value happened to be first.
/// </exception>
public static ConnectionHealthEnum ToDto(ConnectionHealth health) => health switch
{
ConnectionHealth.Connected => ConnectionHealthEnum.ConnectionHealthConnected,
ConnectionHealth.Disconnected => ConnectionHealthEnum.ConnectionHealthDisconnected,
ConnectionHealth.Connecting => ConnectionHealthEnum.ConnectionHealthConnecting,
ConnectionHealth.Error => ConnectionHealthEnum.ConnectionHealthError,
_ => throw new ArgumentOutOfRangeException(nameof(health), health, "Unmapped ConnectionHealth value"),
};
/// <summary>Reconstructs a <see cref="ConnectionHealth"/> from its wire enum.</summary>
/// <remarks>
/// An unspecified or unknown wire value decodes to <see cref="ConnectionHealth.Error"/>,
/// never to <see cref="ConnectionHealth.Connected"/>. A newer site sending a value this
/// build has never heard of must not have it rendered as "healthy" on the central
/// health page — the safe direction for an unknown connection state is "not working".
/// </remarks>
/// <param name="health">The wire enum value to reconstruct.</param>
/// <returns>The in-process connection health.</returns>
public static ConnectionHealth FromDto(ConnectionHealthEnum health) => health switch
{
ConnectionHealthEnum.ConnectionHealthConnected => ConnectionHealth.Connected,
ConnectionHealthEnum.ConnectionHealthDisconnected => ConnectionHealth.Disconnected,
ConnectionHealthEnum.ConnectionHealthConnecting => ConnectionHealth.Connecting,
_ => ConnectionHealth.Error,
};
private static string GuidToWire(Guid? value) =>
value?.ToString() ?? string.Empty;
private static Guid? GuidFromWire(string? value) =>
string.IsNullOrEmpty(value) ? null : Guid.Parse(value);
private static string? NullIfEmpty(string? value) =>
string.IsNullOrEmpty(value) ? null : value;
// All ScadaBridge timestamps are UTC by invariant; Timestamp.FromDateTime requires
// UTC kind. Specify (never convert) so a value read back from SQLite with Kind=Utc
// passes through and a defensively-unspecified one is treated as the UTC it already
// is. Mirrors AuditEventDtoMapper/SiteCallDtoMapper.EnsureUtc.
private static DateTime EnsureUtc(DateTime value) =>
value.Kind == DateTimeKind.Utc ? value : DateTime.SpecifyKind(value, DateTimeKind.Utc);
}
@@ -0,0 +1,299 @@
using Akka.Actor;
using Google.Protobuf.WellKnownTypes;
using Grpc.Core;
using Microsoft.Extensions.Logging;
using Microsoft.Extensions.Options;
using ZB.MOM.WW.ScadaBridge.Commons.Messages.Audit;
using ZB.MOM.WW.ScadaBridge.Commons.Messages.Deployment;
using ZB.MOM.WW.ScadaBridge.Commons.Messages.Health;
using ZB.MOM.WW.ScadaBridge.Commons.Messages.Notification;
using GrpcStatus = Grpc.Core.Status;
namespace ZB.MOM.WW.ScadaBridge.Communication.Grpc;
/// <summary>
/// Central-hosted gRPC face of the seven site→central control messages
/// (<c>Protos/central_control.proto</c>). Decodes each request onto the SAME in-process
/// message type the Akka <c>ClusterClient</c> path already carries, <c>Ask</c>s
/// <see cref="Actors.CentralCommunicationActor"/>, and encodes the reply back.
/// </summary>
/// <remarks>
/// <para>
/// <b>Direction is inverted from <see cref="SiteStreamGrpcServer"/>.</b> That server runs on a
/// site and central dials in; this one runs on CENTRAL and the site dials in. The two listen on
/// the same port number (8083) on their respective nodes, which is symmetry, not a collision —
/// a node is either central or a site, never both.
/// </para>
/// <para>
/// <b>Zero handler logic lives here.</b> Every RPC lands on a receive
/// <c>CentralCommunicationActor</c> already implements for the ClusterClient path, so the two
/// transports cannot drift in behaviour: the actor is the single implementation, and this class
/// is a codec plus an <c>Ask</c>. That is also why the service takes the actor through
/// <see cref="SetReady"/> rather than resolving anything from DI — the actor is created by the
/// host's Akka bootstrap, not by the container.
/// </para>
/// <para>
/// <b>Fault semantics deliberately differ from <see cref="SiteStreamGrpcServer"/>'s ingest
/// RPCs.</b> That server answers a failed audit ingest with an EMPTY <c>IngestAck</c>; this one
/// fails the call with a non-OK status. Both leave the site's rows <c>Pending</c> for the next
/// drain, so the outcome is the same — but this service replaces the ClusterClient path, whose
/// documented behaviour is to propagate the fault (<c>CentralCommunicationActor</c>'s
/// <c>HandleIngestAuditEvents</c> pipes a <c>Status.Failure</c> back), and preserving that keeps
/// a lost batch visible as a failure rather than as a successful call that acked nothing.
/// </para>
/// <para>
/// <b>Status mapping, and why it is not uniform.</b> A site transport may safely re-send a call
/// to the peer central node only when the call provably never ran. So:
/// </para>
/// <list type="bullet">
/// <item><see cref="StatusCode.Unavailable"/> — this node is not ready; nothing was dispatched,
/// so a cross-node retry is safe and correct.</item>
/// <item><see cref="StatusCode.DeadlineExceeded"/> — the <c>Ask</c> timed out. The message WAS
/// delivered and may have been processed; retrying it on the other node would duplicate work.</item>
/// <item><see cref="StatusCode.Internal"/> — the handler faulted (a piped
/// <see cref="Akka.Actor.Status.Failure"/>, e.g. a database error inside reconcile). Same
/// reasoning: it ran, so do not re-send it elsewhere.</item>
/// </list>
/// </remarks>
public sealed class CentralControlGrpcService : CentralControlService.CentralControlServiceBase
{
private readonly ILogger<CentralControlGrpcService> _logger;
private readonly CommunicationOptions _options;
// Null until the host's Akka bootstrap hands the actor over. Doubles as the readiness
// flag: a call arriving before then cannot be served and is refused with Unavailable.
// Volatile because SetReady runs on the startup thread while calls are served on
// Kestrel's thread pool.
private volatile IActorRef? _central;
/// <summary>
/// Creates the service. <b>This must remain the only public constructor</b> — see
/// <see cref="SetReady"/> for how the actor arrives, and the Host's
/// <c>CentralControlAuthInterceptor</c> for the interceptor-side version of the same rule.
/// </summary>
/// <param name="logger">Logger for readiness and fault diagnostics.</param>
/// <param name="options">Communication options supplying the per-RPC <c>Ask</c> timeouts.</param>
public CentralControlGrpcService(
ILogger<CentralControlGrpcService> logger,
IOptions<CommunicationOptions> options)
{
ArgumentNullException.ThrowIfNull(logger);
ArgumentNullException.ThrowIfNull(options);
_logger = logger;
_options = options.Value;
}
/// <summary>
/// Hands the <c>CentralCommunicationActor</c> to the service and opens the gate. Mirrors
/// <see cref="SiteStreamGrpcServer.SetReady"/>: the gRPC service is a DI singleton created
/// before the actor system exists, so the actor arrives post-construction.
/// </summary>
/// <remarks>
/// The contract is deliberately narrow, exactly as on the site side: it asserts that the
/// actor exists and can receive, NOT that every downstream singleton proxy
/// (<c>notification-outbox</c>, <c>audit-log-ingest</c>) has registered itself yet. Those
/// register moments later in the same startup path, and the actor already answers a call
/// that beats them with the same "not available, retry" reply it gives on the ClusterClient
/// path — so gating readiness on them would add nothing but a longer window in which sites
/// see <see cref="StatusCode.Unavailable"/>.
/// </remarks>
/// <param name="centralCommunicationActor">The central communication actor.</param>
public void SetReady(IActorRef centralCommunicationActor)
{
ArgumentNullException.ThrowIfNull(centralCommunicationActor);
_central = centralCommunicationActor;
}
/// <summary>Exposed for wiring assertions in tests.</summary>
internal bool IsReady => _central is not null;
/// <inheritdoc />
public override async Task<NotificationSubmitAckDto> SubmitNotification(
NotificationSubmitDto request, ServerCallContext context)
{
var central = RequireReady(context);
var ack = await AskAsync<NotificationSubmitAck>(
central,
CentralControlDtoMapper.FromDto(request),
_options.NotificationForwardTimeout,
context).ConfigureAwait(false);
return CentralControlDtoMapper.ToDto(ack);
}
/// <inheritdoc />
public override async Task<NotificationStatusResponseDto> QueryNotificationStatus(
NotificationStatusQueryDto request, ServerCallContext context)
{
var central = RequireReady(context);
var response = await AskAsync<NotificationStatusResponse>(
central,
CentralControlDtoMapper.FromDto(request),
_options.NotificationForwardTimeout,
context).ConfigureAwait(false);
return CentralControlDtoMapper.ToDto(response);
}
/// <inheritdoc />
public override async Task<IngestAck> IngestAuditEvents(
AuditEventBatch request, ServerCallContext context)
{
// An empty batch is a no-op the actor need never see; answering it here also means a
// site that drains an empty queue does not fail against a not-yet-ready central.
if (request.Events.Count == 0)
{
return new IngestAck();
}
var central = RequireReady(context);
var reply = await AskAsync<IngestAuditEventsReply>(
central,
CentralControlDtoMapper.FromDto(request),
SiteStreamGrpcServer.AuditIngestAskTimeout,
context).ConfigureAwait(false);
return CentralControlDtoMapper.ToIngestAck(reply.AcceptedEventIds);
}
/// <inheritdoc />
public override async Task<IngestAck> IngestCachedTelemetry(
CachedTelemetryBatch request, ServerCallContext context)
{
if (request.Packets.Count == 0)
{
return new IngestAck();
}
var central = RequireReady(context);
var reply = await AskAsync<IngestCachedTelemetryReply>(
central,
CentralControlDtoMapper.FromDto(request),
SiteStreamGrpcServer.AuditIngestAskTimeout,
context).ConfigureAwait(false);
return CentralControlDtoMapper.ToIngestAck(reply.AcceptedEventIds);
}
/// <inheritdoc />
public override async Task<ReconcileSiteResponseDto> ReconcileSite(
ReconcileSiteRequestDto request, ServerCallContext context)
{
var central = RequireReady(context);
var response = await AskAsync<ReconcileSiteResponse>(
central,
CentralControlDtoMapper.FromDto(request),
_options.QueryTimeout,
context).ConfigureAwait(false);
return CentralControlDtoMapper.ToDto(response);
}
/// <inheritdoc />
public override async Task<SiteHealthReportAckDto> ReportSiteHealth(
SiteHealthReportDto request, ServerCallContext context)
{
var central = RequireReady(context);
var ack = await AskAsync<SiteHealthReportAck>(
central,
CentralControlDtoMapper.FromDto(request),
_options.HealthReportTimeout,
context).ConfigureAwait(false);
return CentralControlDtoMapper.ToDto(ack);
}
/// <summary>
/// Application heartbeat — <b>always answers OK</b>, even when this node is not ready.
/// </summary>
/// <remarks>
/// The heartbeat is fire-and-forget on both sides: nothing on the site consumes the reply,
/// and the site's heartbeat timer must never take a fault (a failing heartbeat that raised
/// an error would be a self-inflicted outage on a purely informational signal). So this is
/// the one RPC that does not go through <see cref="RequireReady"/>: a heartbeat arriving
/// before the actor exists is logged and dropped, exactly as the actor itself drops one
/// that arrives before <c>ICentralHealthAggregator</c> is resolvable. Liveness is still
/// detected — the aggregator's offline timeout fires when the heartbeats stop landing.
/// </remarks>
/// <param name="request">The heartbeat.</param>
/// <param name="context">The gRPC call context.</param>
/// <returns>An empty reply, always.</returns>
public override Task<Empty> Heartbeat(HeartbeatDto request, ServerCallContext context)
{
var central = _central;
if (central is null)
{
_logger.LogDebug(
"Dropped a heartbeat from site {SiteId}: the central communication actor is not "
+ "ready yet. Heartbeats are fire-and-forget, so the call still succeeds.",
request.SiteId);
return Task.FromResult(new Empty());
}
// Tell, never Ask: the actor's HandleHeartbeat sends no reply.
central.Tell(CentralControlDtoMapper.FromDto(request), ActorRefs.NoSender);
return Task.FromResult(new Empty());
}
/// <summary>
/// Returns the central communication actor, or throws <see cref="StatusCode.Unavailable"/>
/// when the host has not finished bringing the actor system up.
/// </summary>
private IActorRef RequireReady(ServerCallContext context)
{
var central = _central;
if (central is not null)
{
return central;
}
_logger.LogWarning(
"Refused a control-plane call to {Method}: the central communication actor is not "
+ "ready yet. Nothing was dispatched, so the caller may retry (including against "
+ "the peer central node).",
context.Method);
throw new RpcException(new GrpcStatus(
StatusCode.Unavailable,
"Central control plane is not ready: the actor system is still starting."));
}
/// <summary>
/// Asks the central actor and maps a fault onto the status code that tells the caller
/// whether a cross-node retry is safe. See the class remarks for the mapping rationale.
/// </summary>
private async Task<TReply> AskAsync<TReply>(
IActorRef central, object message, TimeSpan timeout, ServerCallContext context)
{
try
{
return await central.Ask<TReply>(message, timeout, context.CancellationToken)
.ConfigureAwait(false);
}
catch (AskTimeoutException ex)
{
_logger.LogWarning(ex,
"Control-plane call {Method} timed out after {Timeout} waiting for the central "
+ "communication actor.",
context.Method, timeout);
throw new RpcException(new GrpcStatus(
StatusCode.DeadlineExceeded,
$"Central did not answer within {timeout}."));
}
catch (OperationCanceledException)
{
// The client gave up or its deadline expired; there is no one left to answer.
throw new RpcException(new GrpcStatus(
StatusCode.Cancelled, "The call was cancelled."));
}
catch (Exception ex)
{
_logger.LogError(ex,
"Control-plane call {Method} faulted inside the central communication actor.",
context.Method);
throw new RpcException(new GrpcStatus(
StatusCode.Internal, "Central failed to process the call."));
}
}
}
@@ -0,0 +1,258 @@
using Akka.Actor;
using Grpc.Core;
using Microsoft.Extensions.Logging;
using ZB.MOM.WW.ScadaBridge.Commons.Messages.Audit;
using ZB.MOM.WW.ScadaBridge.Commons.Messages.Deployment;
using ZB.MOM.WW.ScadaBridge.Commons.Messages.Health;
using ZB.MOM.WW.ScadaBridge.Commons.Messages.Notification;
using ZB.MOM.WW.ScadaBridge.Communication.Actors;
using AkkaStatus = Akka.Actor.Status;
namespace ZB.MOM.WW.ScadaBridge.Communication.Grpc;
/// <summary>
/// The <see cref="ICentralTransport"/> that carries the seven site→central sends over gRPC — the
/// migration target for the Akka <c>ClusterClient</c> path. Each method encodes the message with
/// <see cref="CentralControlDtoMapper"/>, dials <c>CentralControlService</c> through the sticky
/// <see cref="CentralChannelProvider"/>, and delivers the decoded reply (or a transient-failure
/// signal) to the waiting Ask.
/// </summary>
/// <remarks>
/// <para>
/// The actor handlers are synchronous; each method here kicks off the RPC as a detached task and
/// <c>Tell</c>s the result to <paramref name="replyTo"/> when it completes — <c>IActorRef.Tell</c>
/// is thread-safe, so the reply lands at the Ask exactly as central's ClusterClient reply did. On
/// any non-OK status the reply is <see cref="Status.Failure"/>, which the S&amp;F / audit / health
/// layers already treat as transient.
/// </para>
/// <para>
/// <b>Cross-node retry only on provably-unsent failures.</b> An <see cref="StatusCode.Unavailable"/>
/// (connection refused / node not ready) flips the channel pair and retries once on the peer.
/// A <see cref="StatusCode.DeadlineExceeded"/> is NEVER retried across nodes — a deploy / write /
/// failover may already have executed, and duplicating it is worse than surfacing a transient
/// failure the layer above tolerates.
/// </para>
/// <para>
/// <b>Per-call deadlines mirror today's Ask timeouts.</b> Notification submit/status →
/// <c>NotificationForwardTimeout</c> (30 s, the value the S&amp;F forwarder and the central service
/// both use); health → <c>HealthReportTimeout</c> (10 s); reconcile → <c>QueryTimeout</c> (30 s);
/// both ingest RPCs → <see cref="SiteStreamGrpcServer.AuditIngestAskTimeout"/> (the one shared 30 s
/// constant). The heartbeat, fire-and-forget with no server-side Ask, is merely bounded by
/// <c>HealthReportTimeout</c> and its failures are swallowed.
/// </para>
/// </remarks>
public sealed class GrpcCentralTransport : ICentralTransport
{
private readonly CentralChannelProvider _channels;
private readonly CommunicationOptions _options;
private readonly ILogger<GrpcCentralTransport> _logger;
/// <summary>Creates the transport over a channel pair.</summary>
/// <param name="channels">The sticky central channel pair.</param>
/// <param name="options">Communication options supplying the per-call deadlines.</param>
/// <param name="logger">Logger for failover/fault diagnostics.</param>
public GrpcCentralTransport(
CentralChannelProvider channels,
CommunicationOptions options,
ILogger<GrpcCentralTransport> logger)
{
ArgumentNullException.ThrowIfNull(channels);
ArgumentNullException.ThrowIfNull(options);
ArgumentNullException.ThrowIfNull(logger);
_channels = channels;
_options = options;
_logger = logger;
}
/// <inheritdoc />
public void SubmitNotification(NotificationSubmit message, IActorRef replyTo)
{
var dto = CentralControlDtoMapper.ToDto(message);
Dispatch(replyTo, _options.NotificationForwardTimeout,
(c, o) => c.SubmitNotificationAsync(dto, o),
ack => CentralControlDtoMapper.FromDto(ack));
}
/// <inheritdoc />
public void QueryNotificationStatus(NotificationStatusQuery message, IActorRef replyTo)
{
var dto = CentralControlDtoMapper.ToDto(message);
Dispatch(replyTo, _options.NotificationForwardTimeout,
(c, o) => c.QueryNotificationStatusAsync(dto, o),
response => CentralControlDtoMapper.FromDto(response));
}
/// <inheritdoc />
public void IngestAuditEvents(IngestAuditEventsCommand message, IActorRef replyTo)
{
var batch = CentralControlDtoMapper.ToDto(message);
Dispatch(replyTo, SiteStreamGrpcServer.AuditIngestAskTimeout,
(c, o) => c.IngestAuditEventsAsync(batch, o),
ack => new IngestAuditEventsReply(CentralControlDtoMapper.FromIngestAck(ack)));
}
/// <inheritdoc />
public void IngestCachedTelemetry(IngestCachedTelemetryCommand message, IActorRef replyTo)
{
var batch = CentralControlDtoMapper.ToDto(message);
Dispatch(replyTo, SiteStreamGrpcServer.AuditIngestAskTimeout,
(c, o) => c.IngestCachedTelemetryAsync(batch, o),
ack => new IngestCachedTelemetryReply(CentralControlDtoMapper.FromIngestAck(ack)));
}
/// <inheritdoc />
public void ReconcileSite(ReconcileSiteRequest message, IActorRef replyTo)
{
var dto = CentralControlDtoMapper.ToDto(message);
Dispatch(replyTo, _options.QueryTimeout,
(c, o) => c.ReconcileSiteAsync(dto, o),
response => CentralControlDtoMapper.FromDto(response));
}
/// <inheritdoc />
public void ReportSiteHealth(SiteHealthReport message, IActorRef replyTo)
{
var dto = CentralControlDtoMapper.ToDto(message);
Dispatch(replyTo, _options.HealthReportTimeout,
(c, o) => c.ReportSiteHealthAsync(dto, o),
ack => CentralControlDtoMapper.FromDto(ack));
}
/// <inheritdoc />
public void SendHeartbeat(HeartbeatMessage message, IActorRef self)
{
_ = SendHeartbeatAsync(message);
}
private async Task SendHeartbeatAsync(HeartbeatMessage message)
{
var (index, client) = _channels.Current();
var dto = CentralControlDtoMapper.ToDto(message);
try
{
var options = new CallOptions(deadline: DateTime.UtcNow.Add(_options.HealthReportTimeout));
using var call = client.HeartbeatAsync(dto, options);
await call.ResponseAsync.ConfigureAwait(false);
}
catch (RpcException ex) when (IsConnectFailure(ex))
{
// Nudge the pair so the next call tries the peer, but never fault: a heartbeat
// failure must not surface on the site's heartbeat timer path.
_channels.ReportUnavailable(index);
_logger.LogDebug(ex, "Heartbeat to central endpoint {Endpoint} was unavailable.", CurrentEndpointSafe());
}
catch (Exception ex)
{
_logger.LogDebug(ex, "Heartbeat to central failed (swallowed — heartbeats are fire-and-forget).");
}
}
/// <summary>
/// Runs a unary RPC on the current channel, delivers the decoded reply to
/// <paramref name="replyTo"/>, and applies the sticky-failover / no-retry-on-deadline policy.
/// </summary>
private void Dispatch<TWire>(
IActorRef replyTo,
TimeSpan timeout,
Func<CentralControlService.CentralControlServiceClient, CallOptions, AsyncUnaryCall<TWire>> call,
Func<TWire, object> decode)
{
_ = DispatchAsync(replyTo, timeout, call, decode);
}
private async Task DispatchAsync<TWire>(
IActorRef replyTo,
TimeSpan timeout,
Func<CentralControlService.CentralControlServiceClient, CallOptions, AsyncUnaryCall<TWire>> call,
Func<TWire, object> decode)
{
var (index, client) = _channels.Current();
try
{
var reply = await InvokeAsync(client, timeout, call).ConfigureAwait(false);
replyTo.Tell(decode(reply));
}
catch (RpcException ex) when (IsConnectFailure(ex))
{
// Provably unsent: the connection was refused / the node was not ready. Fail over
// to the peer and retry ONCE. This is the only status we retry across nodes.
_channels.ReportUnavailable(index);
var (retryIndex, retryClient) = _channels.Current();
if (retryIndex != index)
{
try
{
var reply = await InvokeAsync(retryClient, timeout, call).ConfigureAwait(false);
replyTo.Tell(decode(reply));
return;
}
catch (Exception retryEx)
{
_logger.LogWarning(retryEx,
"Central control-plane call failed on both endpoints; surfacing as transient.");
replyTo.Tell(new AkkaStatus.Failure(retryEx));
return;
}
}
replyTo.Tell(new AkkaStatus.Failure(ex));
}
catch (Exception ex)
{
// DeadlineExceeded / Internal / PermissionDenied / a PSK-resolution throw — do NOT
// retry across nodes (the call may have run). Surface as the transient failure the
// layer above already tolerates.
replyTo.Tell(new AkkaStatus.Failure(ex));
}
}
private static async Task<TWire> InvokeAsync<TWire>(
CentralControlService.CentralControlServiceClient client,
TimeSpan timeout,
Func<CentralControlService.CentralControlServiceClient, CallOptions, AsyncUnaryCall<TWire>> call)
{
var options = new CallOptions(deadline: DateTime.UtcNow.Add(timeout));
using var asyncCall = call(client, options);
return await asyncCall.ResponseAsync.ConfigureAwait(false);
}
/// <summary>
/// A failure that provably never reached a server — the only class safe to retry on the peer.
/// <see cref="StatusCode.DeadlineExceeded"/> is deliberately excluded (the call may have run).
/// </summary>
/// <remarks>
/// Two shapes qualify: a server-signalled <see cref="StatusCode.Unavailable"/> (e.g. a node
/// that returns Unavailable while it is still starting), and a client-side failure to even
/// start the call — Grpc.Net surfaces a refused/failed connection as
/// <see cref="StatusCode.Internal"/> "Error starting gRPC call" with the transport exception
/// attached, and there the request never left the client. Anything else — including a deadline,
/// a permission denial, or a generic server-side Internal after the call reached the server —
/// is NOT retried across nodes.
/// </remarks>
private static bool IsConnectFailure(RpcException ex)
{
if (ex.StatusCode == StatusCode.Unavailable)
{
return true;
}
// "Error starting gRPC call" is Grpc.Net's marker for a call that could not be sent — a
// refused/failed connection carrying an HttpRequestException. Provably unsent.
return ex.StatusCode == StatusCode.Internal
&& (ex.Status.DebugException is HttpRequestException
|| ex.Status.Detail.StartsWith("Error starting gRPC call", StringComparison.Ordinal));
}
private string CurrentEndpointSafe()
{
try
{
return _channels.CurrentEndpoint;
}
catch
{
return "(unknown)";
}
}
}
@@ -0,0 +1,244 @@
using Akka.Actor;
using Grpc.Net.Client;
using Microsoft.Extensions.Logging;
using ZB.MOM.WW.ScadaBridge.Commons.Messages.Artifacts;
using ZB.MOM.WW.ScadaBridge.Commons.Messages.DataConnection;
using ZB.MOM.WW.ScadaBridge.Commons.Messages.DebugView;
using ZB.MOM.WW.ScadaBridge.Commons.Messages.Deployment;
using ZB.MOM.WW.ScadaBridge.Commons.Messages.InboundApi;
using ZB.MOM.WW.ScadaBridge.Commons.Messages.Lifecycle;
using ZB.MOM.WW.ScadaBridge.Commons.Messages.Management;
using ZB.MOM.WW.ScadaBridge.Commons.Messages.RemoteQuery;
using ZB.MOM.WW.ScadaBridge.Communication.Actors;
namespace ZB.MOM.WW.ScadaBridge.Communication.Grpc;
/// <summary>
/// The gRPC central→site command transport: encodes each <see cref="SiteEnvelope"/> command with
/// <see cref="SiteCommandDtoMapper"/>, dials the site's <c>SiteCommandService</c> through the sticky
/// <see cref="SitePairChannelProvider"/> (PSK + <c>x-scadabridge-site</c> already on the channel),
/// and routes the decoded reply back to the waiting <c>Ask</c> (or the debug-bridge actor).
/// </summary>
/// <remarks>
/// <para>
/// <b>Per-call deadlines match today's Ask timeouts exactly</b> — see <see cref="ResolveDeadline"/>.
/// Behaviour is otherwise unchanged from the Akka path: an unknown/unconfigured site is warned and
/// dropped (the caller's Ask times out), and a transport fault surfaces to the caller as a
/// <see cref="Status.Failure"/>, which the S&amp;F/audit layers already treat as transient.
/// </para>
/// <para>
/// <b>Cross-node retry is the channel provider's job</b> and happens only on
/// <c>Unavailable</c> — never on <c>DeadlineExceeded</c> (a write/deploy/failover may have run).
/// </para>
/// </remarks>
public sealed class GrpcSiteTransport : ISiteCommandTransport
{
private readonly SitePairChannelProvider _channels;
private readonly CommunicationOptions _options;
private readonly ILogger<GrpcSiteTransport> _logger;
// Sites this transport has pushed into the channel provider — used to diff removals per refresh,
// the gRPC analogue of the Akka transport's _siteClients key set. Touched only on the actor thread.
private readonly HashSet<string> _knownSites = new(StringComparer.Ordinal);
/// <summary>Creates the gRPC transport and ensures the provider's failback loop is running.</summary>
/// <param name="channels">The shared per-site channel-pair provider.</param>
/// <param name="options">Communication options supplying the per-command deadlines.</param>
/// <param name="logger">Logger for drop/fault diagnostics.</param>
public GrpcSiteTransport(
SitePairChannelProvider channels,
CommunicationOptions options,
ILogger<GrpcSiteTransport> logger)
{
_channels = channels ?? throw new ArgumentNullException(nameof(channels));
_options = options ?? throw new ArgumentNullException(nameof(options));
_logger = logger ?? throw new ArgumentNullException(nameof(logger));
_channels.EnsureFailbackLoop();
}
/// <inheritdoc />
public void Send(SiteEnvelope envelope, IActorRef replyTo)
{
ArgumentNullException.ThrowIfNull(envelope);
// UnsubscribeDebugView is the one fire-and-forget command: the site never acks it over Akka
// (the debug bridge Tells and expects nothing), so we run the RPC but do not deliver its
// synthetic ack — mirroring today's no-reply behaviour and avoiding a dead-lettered ack.
var fireAndForget = envelope.Message is UnsubscribeDebugViewRequest;
_ = RunAsync(envelope, replyTo, fireAndForget);
}
private async Task RunAsync(SiteEnvelope envelope, IActorRef replyTo, bool fireAndForget)
{
try
{
var reply = await SendCoreAsync(envelope, CancellationToken.None).ConfigureAwait(false);
if (!fireAndForget && !replyTo.IsNobody())
{
replyTo.Tell(reply, ActorRefs.NoSender);
}
}
catch (SiteChannelUnavailableException)
{
// Parity with the Akka "no ClusterClient for site" path: warn and drop, so the caller's
// Ask times out. Central never buffers.
_logger.LogWarning(
"No gRPC channel for site {SiteId}; dropping {Message} (caller's Ask will time out)",
envelope.SiteId, envelope.Message.GetType().Name);
}
catch (Exception ex)
{
if (!fireAndForget && !replyTo.IsNobody())
{
// A timeout or non-OK status faults the caller's Ask exactly as an Akka Ask timeout
// did — the S&F/audit layers already treat that as transient.
replyTo.Tell(new Status.Failure(ex), ActorRefs.NoSender);
}
else
{
_logger.LogWarning(ex,
"Fire-and-forget {Message} to site {SiteId} faulted",
envelope.Message.GetType().Name, envelope.SiteId);
}
}
}
private Task<object> SendCoreAsync(SiteEnvelope envelope, CancellationToken ct)
{
var message = envelope.Message;
// Compute the absolute deadline ONCE so a cross-node failover retry shares the overall
// budget rather than restarting it.
var deadline = DateTime.UtcNow + ResolveDeadline(message);
var group = SiteCommandDtoMapper.GroupOf(message);
return _channels.ExecuteAsync(
envelope.SiteId,
(channel, callCt) => InvokeAsync(channel, group, message, deadline, callCt),
ct);
}
private static async Task<object> InvokeAsync(
GrpcChannel channel, SiteCommandGroup group, object message, DateTime deadline, CancellationToken ct)
{
var client = new SiteCommandService.SiteCommandServiceClient(channel);
switch (group)
{
case SiteCommandGroup.Lifecycle:
{
var reply = await client.ExecuteLifecycleAsync(
SiteCommandDtoMapper.ToLifecycleRequest(message),
deadline: deadline, cancellationToken: ct);
return SiteCommandDtoMapper.FromLifecycleReply(reply);
}
case SiteCommandGroup.OpcUa:
{
var reply = await client.ExecuteOpcUaAsync(
SiteCommandDtoMapper.ToOpcUaRequest(message),
deadline: deadline, cancellationToken: ct);
return SiteCommandDtoMapper.FromOpcUaReply(reply);
}
case SiteCommandGroup.Query:
{
var reply = await client.ExecuteQueryAsync(
SiteCommandDtoMapper.ToQueryRequest(message),
deadline: deadline, cancellationToken: ct);
return SiteCommandDtoMapper.FromQueryReply(reply);
}
case SiteCommandGroup.Parked:
{
var reply = await client.ExecuteParkedAsync(
SiteCommandDtoMapper.ToParkedRequest(message),
deadline: deadline, cancellationToken: ct);
return SiteCommandDtoMapper.FromParkedReply(reply);
}
case SiteCommandGroup.Route:
{
var reply = await client.ExecuteRouteAsync(
SiteCommandDtoMapper.ToRouteRequest(message),
deadline: deadline, cancellationToken: ct);
return SiteCommandDtoMapper.FromRouteReply(reply);
}
case SiteCommandGroup.Failover:
{
var ack = await client.TriggerFailoverAsync(
SiteCommandDtoMapper.ToProto((TriggerSiteFailover)message),
deadline: deadline, cancellationToken: ct);
return SiteCommandDtoMapper.FromProto(ack);
}
default:
throw new ArgumentOutOfRangeException(nameof(group), group, "Unknown site command group.");
}
}
/// <summary>
/// The per-command deadline, set EQUAL to the Ask timeout <c>CommunicationService</c> uses for
/// that command today, so flipping the transport changes nothing about how long a call waits.
/// Note the group is NOT a uniform deadline class: within Lifecycle, <c>DeploymentStateQuery</c>
/// uses <c>QueryTimeout</c> (not <c>LifecycleTimeout</c>), and <c>TriggerSiteFailover</c> uses
/// <c>QueryTimeout</c> (not <c>LifecycleTimeout</c>) — matching the real Ask sites, not the
/// plan's per-group table. The two parked relays (<c>RetryParkedOperation</c>/
/// <c>DiscardParkedOperation</c>) map to <c>QueryTimeout</c> (30s), preserving the
/// <c>SiteCallAuditActor</c> inner <c>RelayTimeout</c> (10s) &lt; 30s ordering. WaitForAttribute
/// keeps its dynamic <c>request.Timeout + IntegrationTimeout</c> budget.
/// </summary>
/// <param name="message">The command being sent.</param>
/// <returns>The deadline duration for that command.</returns>
internal TimeSpan ResolveDeadline(object message) => message switch
{
RefreshDeploymentCommand => _options.DeploymentTimeout,
EnableInstanceCommand or DisableInstanceCommand or DeleteInstanceCommand => _options.LifecycleTimeout,
DeploymentStateQueryRequest => _options.QueryTimeout,
DeployArtifactsCommand => _options.ArtifactDeploymentTimeout,
BrowseNodeCommand or SearchAddressSpaceCommand or ReadTagValuesCommand or VerifyEndpointCommand
or TrustServerCertCommand or ListServerCertsCommand or RemoveServerCertCommand or WriteTagRequest
=> _options.QueryTimeout,
EventLogQueryRequest or DebugSnapshotRequest => _options.QueryTimeout,
SubscribeDebugViewRequest or UnsubscribeDebugViewRequest => _options.DebugViewTimeout,
ParkedMessageQueryRequest or ParkedMessageRetryRequest or ParkedMessageDiscardRequest
or RetryParkedOperation or DiscardParkedOperation
=> _options.QueryTimeout,
RouteToCallRequest or RouteToGetAttributesRequest or RouteToSetAttributesRequest
=> _options.IntegrationTimeout,
RouteToWaitForAttributeRequest r => r.Timeout + _options.IntegrationTimeout,
TriggerSiteFailover => _options.QueryTimeout,
_ => throw new ArgumentException(
$"'{message.GetType().Name}' is not a migrated site command.", nameof(message))
};
/// <inheritdoc />
public void ReconcileSites(SiteAddressCacheLoaded cache)
{
ArgumentNullException.ThrowIfNull(cache);
var desired = cache.GrpcContacts;
// Drop sites that lost their gRPC endpoints (or were deleted).
foreach (var removed in _knownSites.Where(s => !desired.ContainsKey(s)).ToList())
{
_channels.RemoveSite(removed);
_knownSites.Remove(removed);
}
// Create/refresh the rest.
foreach (var (siteId, endpoints) in desired)
{
_channels.UpdateSite(siteId, endpoints.NodeA, endpoints.NodeB);
_knownSites.Add(siteId);
}
}
}
@@ -0,0 +1,219 @@
using System.Collections;
using System.Globalization;
using System.Text.Json;
using Google.Protobuf;
namespace ZB.MOM.WW.ScadaBridge.Communication.Grpc;
/// <summary>
/// Codec for the loosely-typed <c>object?</c> members that survive on the site
/// command plane — script parameters and return values, attribute values, and
/// OPC UA tag read/write values — mapping them to and from the type-tagged
/// <see cref="LooseValue"/> proto carrier.
/// </summary>
/// <remarks>
/// <para>
/// <b>Why a tagged union rather than a string or JSON blob.</b> These values
/// reach an operator's screen (Test Bindings, Debug View) and a device write
/// (<c>WriteTag</c>), so collapsing them to text would change behaviour: today
/// the Akka JSON serializer runs with <c>TypeNameHandling</c> on and preserves
/// the boxed CLR type end-to-end. The tags below cover every CLR type these
/// fields actually carry, so those values keep their runtime type across the
/// wire exactly as they do over Akka remoting.
/// </para>
/// <para>
/// <b>The one documented lossy path.</b> Anything outside the tagged set —
/// an exotic numeric (<see cref="byte"/>, <see cref="uint"/>, …), an enum, a
/// POCO — falls back to <see cref="LooseValue.JsonValue"/> and decodes as a
/// <see cref="JsonElement"/> rather than its original CLR type. The value
/// itself is preserved; its CLR identity is not. Collections and string-keyed
/// dictionaries do NOT take that path: they encode recursively (see
/// <see cref="LooseValueList"/>/<see cref="LooseValueMap"/>) and decode to
/// <c>List&lt;object?&gt;</c> / <c>Dictionary&lt;string, object?&gt;</c>, so
/// their ELEMENTS keep their types while the container type widens.
/// </para>
/// <para>
/// <b>Why dates ride as strings.</b> <c>google.protobuf.Timestamp</c>
/// normalises everything to UTC, which silently discards
/// <see cref="DateTime.Kind"/> and <see cref="DateTimeOffset.Offset"/>. For a
/// timestamped tag value that is data loss, not normalisation, so
/// <see cref="DateTime"/>/<see cref="DateTimeOffset"/>/<see cref="TimeSpan"/>
/// use invariant round-trip formats instead. (Message FIELDS that are declared
/// <c>DateTimeOffset</c> in the DTO are a different case and do use
/// <c>Timestamp</c> — see <see cref="SiteCommandDtoMapper"/>.)
/// </para>
/// </remarks>
public static class LooseValueCodec
{
private static readonly JsonSerializerOptions JsonOpts = new() { WriteIndented = false };
/// <summary>
/// Encodes a boxed value for a NULLABLE message field: <c>null</c> returns
/// <c>null</c> so the field is simply left unset on the wire.
/// </summary>
/// <param name="value">The boxed value to encode, or <c>null</c>.</param>
/// <returns>The encoded carrier, or <c>null</c> when <paramref name="value"/> is <c>null</c>.</returns>
public static LooseValue? ToProtoOrNull(object? value) =>
value is null ? null : ToProto(value);
/// <summary>
/// Encodes a boxed value, representing <c>null</c> explicitly as
/// <see cref="LooseNull"/>. Used inside maps and lists, where an absent
/// entry means "no such key/element" rather than "a null value".
/// </summary>
/// <param name="value">The boxed value to encode, or <c>null</c>.</param>
/// <returns>A populated <see cref="LooseValue"/>; never <c>null</c>.</returns>
public static LooseValue ToProto(object? value) => value switch
{
null => new LooseValue { NullValue = new LooseNull() },
bool b => new LooseValue { BoolValue = b },
int i => new LooseValue { Int32Value = i },
long l => new LooseValue { Int64Value = l },
double d => new LooseValue { DoubleValue = d },
float f => new LooseValue { FloatValue = f },
string s => new LooseValue { StringValue = s },
decimal m => new LooseValue { DecimalValue = m.ToString(CultureInfo.InvariantCulture) },
DateTime dt => new LooseValue { DateTimeValue = dt.ToString("O", CultureInfo.InvariantCulture) },
DateTimeOffset dto => new LooseValue { DateTimeOffsetValue = dto.ToString("O", CultureInfo.InvariantCulture) },
Guid g => new LooseValue { GuidValue = g.ToString("D") },
TimeSpan ts => new LooseValue { TimeSpanValue = ts.ToString("c", CultureInfo.InvariantCulture) },
byte[] bytes => new LooseValue { BytesValue = ByteString.CopyFrom(bytes) },
IDictionary dict => new LooseValue { MapValue = ToMap(dict) },
IEnumerable seq => new LooseValue { ListValue = ToList(seq) },
// Escape hatch. Preserves the value, not the CLR type — see the remarks.
_ => new LooseValue { JsonValue = JsonSerializer.Serialize(value, value.GetType(), JsonOpts) }
};
/// <summary>
/// Decodes a carrier back to a boxed value. An unset field
/// (<c>null</c>) and an explicit <see cref="LooseNull"/> both decode to
/// <c>null</c>, so the two encoders above are interchangeable on read.
/// </summary>
/// <param name="value">The carrier to decode, or <c>null</c> when the field was unset.</param>
/// <returns>The decoded boxed value; <c>null</c> for an unset or explicitly-null carrier.</returns>
public static object? FromProto(LooseValue? value)
{
if (value is null)
{
return null;
}
return value.KindCase switch
{
LooseValue.KindOneofCase.None => null,
LooseValue.KindOneofCase.NullValue => null,
LooseValue.KindOneofCase.BoolValue => value.BoolValue,
LooseValue.KindOneofCase.Int32Value => value.Int32Value,
LooseValue.KindOneofCase.Int64Value => value.Int64Value,
LooseValue.KindOneofCase.DoubleValue => value.DoubleValue,
LooseValue.KindOneofCase.FloatValue => value.FloatValue,
LooseValue.KindOneofCase.StringValue => value.StringValue,
LooseValue.KindOneofCase.DecimalValue =>
decimal.Parse(value.DecimalValue, CultureInfo.InvariantCulture),
LooseValue.KindOneofCase.DateTimeValue =>
DateTime.Parse(value.DateTimeValue, CultureInfo.InvariantCulture, DateTimeStyles.RoundtripKind),
LooseValue.KindOneofCase.DateTimeOffsetValue =>
DateTimeOffset.Parse(value.DateTimeOffsetValue, CultureInfo.InvariantCulture, DateTimeStyles.RoundtripKind),
LooseValue.KindOneofCase.GuidValue => Guid.Parse(value.GuidValue),
LooseValue.KindOneofCase.TimeSpanValue =>
TimeSpan.ParseExact(value.TimeSpanValue, "c", CultureInfo.InvariantCulture),
LooseValue.KindOneofCase.BytesValue => value.BytesValue.ToByteArray(),
LooseValue.KindOneofCase.ListValue => FromList(value.ListValue),
LooseValue.KindOneofCase.MapValue => FromMap(value.MapValue),
LooseValue.KindOneofCase.JsonValue => JsonSerializer.Deserialize<JsonElement>(value.JsonValue),
_ => null
};
}
/// <summary>
/// Encodes a string-keyed dictionary onto the wire. Null values are carried
/// explicitly (<see cref="LooseNull"/>), so a key present with a null value
/// stays distinct from an absent key.
/// </summary>
/// <param name="values">The dictionary to encode.</param>
/// <returns>A populated <see cref="LooseValueMap"/>.</returns>
public static LooseValueMap ToProtoMap(IReadOnlyDictionary<string, object?> values)
{
ArgumentNullException.ThrowIfNull(values);
var map = new LooseValueMap();
foreach (var (key, value) in values)
{
map.Entries[key] = ToProto(value);
}
return map;
}
/// <summary>
/// Encodes a NULLABLE string-keyed dictionary: <c>null</c> returns
/// <c>null</c> so the field is left unset and the null/empty distinction
/// survives (proto3 <c>map</c> alone cannot express it).
/// </summary>
/// <param name="values">The dictionary to encode, or <c>null</c>.</param>
/// <returns>The encoded map, or <c>null</c> when <paramref name="values"/> is <c>null</c>.</returns>
public static LooseValueMap? ToProtoMapOrNull(IReadOnlyDictionary<string, object?>? values) =>
values is null ? null : ToProtoMap(values);
/// <summary>Decodes a wire map back to a dictionary; an unset field decodes to <c>null</c>.</summary>
/// <param name="map">The wire map, or <c>null</c> when the field was unset.</param>
/// <returns>The decoded dictionary, or <c>null</c>.</returns>
public static IReadOnlyDictionary<string, object?>? FromProtoMapOrNull(LooseValueMap? map) =>
map is null ? null : FromMap(map);
/// <summary>Decodes a wire map back to a dictionary; an unset field decodes to an EMPTY dictionary.</summary>
/// <remarks>For DTO members that are declared non-nullable, so "absent" can only mean "empty".</remarks>
/// <param name="map">The wire map, or <c>null</c> when the field was unset.</param>
/// <returns>The decoded dictionary; empty when <paramref name="map"/> is <c>null</c>.</returns>
public static IReadOnlyDictionary<string, object?> FromProtoMap(LooseValueMap? map) =>
map is null ? new Dictionary<string, object?>() : FromMap(map);
private static LooseValueList ToList(IEnumerable source)
{
var list = new LooseValueList();
foreach (var element in source)
{
list.Items.Add(ToProto(element));
}
return list;
}
private static LooseValueMap ToMap(IDictionary source)
{
var map = new LooseValueMap();
foreach (DictionaryEntry entry in source)
{
// Non-string keys are outside the wire contract; the invariant-culture
// rendering keeps the entry readable rather than dropping it silently.
var key = entry.Key as string
?? Convert.ToString(entry.Key, CultureInfo.InvariantCulture)
?? string.Empty;
map.Entries[key] = ToProto(entry.Value);
}
return map;
}
private static List<object?> FromList(LooseValueList list)
{
var result = new List<object?>(list.Items.Count);
foreach (var item in list.Items)
{
result.Add(FromProto(item));
}
return result;
}
private static Dictionary<string, object?> FromMap(LooseValueMap map)
{
var result = new Dictionary<string, object?>(map.Entries.Count);
foreach (var (key, value) in map.Entries)
{
result[key] = FromProto(value);
}
return result;
}
}
@@ -21,15 +21,17 @@ namespace ZB.MOM.WW.ScadaBridge.Communication.Grpc;
/// Mirrors the sibling <see cref="AuditEventDtoMapper"/>.
/// </para>
/// <para>
/// Two directions are provided. <see cref="FromDto"/> rehydrates the central
/// Three directions are provided. <see cref="FromDto"/> rehydrates the central
/// <see cref="SiteCall"/> entity central writes into the <c>SiteCalls</c> table.
/// <see cref="ToDto"/> projects a site-local <see cref="SiteCallOperational"/>
/// onto the wire — used by the Site Call Audit <c>PullSiteCalls</c>
/// reconciliation handler (the central→site self-heal pull). The
/// <see cref="SiteCall"/> entity itself is never mapped back onto the wire:
/// sites emit operational state from <see cref="SiteCallOperational"/>, never
/// from the central <see cref="SiteCall"/>, so a <c>SiteCall</c>→DTO method
/// would be dead code.
/// <see cref="ToDto(SiteCallOperational)"/> projects a site-local
/// <see cref="SiteCallOperational"/> onto the wire — used by the Site Call Audit
/// <c>PullSiteCalls</c> reconciliation handler (the central→site self-heal pull).
/// <see cref="ToDto(SiteCall)"/> projects the entity form back onto the wire; it
/// exists for the gRPC central control plane, whose site-side transport receives
/// an already-decoded <c>IngestCachedTelemetryCommand</c> (which carries
/// <see cref="SiteCall"/>, not <see cref="SiteCallOperational"/>) and must
/// re-encode it. It was previously documented here as necessarily dead code —
/// true only while ClusterClient was the sole path from that command to central.
/// </para>
/// <para>
/// String nullability convention: proto3 scalar strings cannot be absent, so the
@@ -120,6 +122,51 @@ public static class SiteCallDtoMapper
return dto;
}
/// <summary>
/// Projects a <see cref="SiteCall"/> entity onto its wire-format DTO — the
/// inverse of <see cref="FromDto"/>, so the pair round-trips exactly.
/// </summary>
/// <remarks>
/// <see cref="SiteCall.IngestedAtUtc"/> is deliberately NOT written: it is
/// central-set inside the dual-write transaction and the value carried on the
/// wire is informational only (<see cref="FromDto"/> stamps a placeholder that
/// the ingest actor overwrites). Every other field survives; null
/// <c>SourceNode</c>/<c>LastError</c> collapse to empty strings while the
/// nullable <c>HttpStatus</c>/<c>TerminalAtUtc</c> stay unset on the wire.
/// </remarks>
/// <param name="siteCall">The central operational-state entity to project.</param>
/// <returns>A populated <see cref="SiteCallOperationalDto"/> ready for transmission.</returns>
public static SiteCallOperationalDto ToDto(SiteCall siteCall)
{
ArgumentNullException.ThrowIfNull(siteCall);
var dto = new SiteCallOperationalDto
{
TrackedOperationId = siteCall.TrackedOperationId.ToString(),
Channel = siteCall.Channel,
Target = siteCall.Target,
SourceSite = siteCall.SourceSite,
SourceNode = siteCall.SourceNode ?? string.Empty,
Status = siteCall.Status,
RetryCount = siteCall.RetryCount,
LastError = siteCall.LastError ?? string.Empty,
CreatedAtUtc = Timestamp.FromDateTime(EnsureUtc(siteCall.CreatedAtUtc)),
UpdatedAtUtc = Timestamp.FromDateTime(EnsureUtc(siteCall.UpdatedAtUtc)),
};
if (siteCall.HttpStatus.HasValue)
{
dto.HttpStatus = siteCall.HttpStatus.Value;
}
if (siteCall.TerminalAtUtc.HasValue)
{
dto.TerminalAtUtc = Timestamp.FromDateTime(EnsureUtc(siteCall.TerminalAtUtc.Value));
}
return dto;
}
// All ScadaBridge timestamps are UTC by invariant; Timestamp.FromDateTime
// requires UTC kind. Specify (never convert) so a row read back from SQLite
// with Kind=Utc passes through and a defensively-unspecified value is
File diff suppressed because it is too large Load Diff
@@ -0,0 +1,33 @@
namespace ZB.MOM.WW.ScadaBridge.Communication.Grpc;
/// <summary>
/// The six domain groups the 28 migrated central→site commands are partitioned
/// into — one group per <c>SiteCommandService</c> RPC.
/// </summary>
/// <remarks>
/// The partition is not cosmetic: every command inside a group shares a
/// DEADLINE class today (the <c>CommunicationOptions</c> timeout the central
/// <c>Ask</c> uses), so one RPC per group keeps the deadline decision in one
/// place on the client and one dispatch switch on the server, while the
/// <c>oneof</c> envelope keeps each command individually typed.
/// </remarks>
public enum SiteCommandGroup
{
/// <summary>Deployment refresh, instance enable/disable/delete, deployment-state query, artifact deployment.</summary>
Lifecycle,
/// <summary>Interactive OPC UA / MxGateway design-time commands: browse, search, read, verify, cert trust, write tag.</summary>
OpcUa,
/// <summary>Read-only remote queries: site event log and debug view snapshot/subscribe/unsubscribe.</summary>
Query,
/// <summary>Parked store-and-forward message actions and parked cached-operation retry/discard relays.</summary>
Parked,
/// <summary>Inbound-API <c>Route.To()</c> relays: call, get/set attributes, wait for attribute.</summary>
Route,
/// <summary>Operator-initiated manual site-pair failover.</summary>
Failover
}
@@ -0,0 +1,227 @@
using Akka.Actor;
using Grpc.Core;
using Microsoft.Extensions.Logging;
using ZB.MOM.WW.ScadaBridge.Commons.Messages.RemoteQuery;
using ZB.MOM.WW.ScadaBridge.Communication.Actors;
using GrpcStatus = Grpc.Core.Status;
namespace ZB.MOM.WW.ScadaBridge.Communication.Grpc;
/// <summary>
/// gRPC front door for the central→site command plane: decodes a proto command (via
/// <see cref="SiteCommandDtoMapper"/>), routes it through the ONE
/// <see cref="SiteCommandDispatcher"/> the Akka <c>SiteCommunicationActor</c> also uses, and
/// encodes the reply. One routing truth, two transports.
/// </summary>
/// <remarks>
/// <para>
/// Mapped in the site branch next to <c>SiteStreamGrpcServer</c>, behind the same
/// <c>ControlPlaneAuthInterceptor</c> PSK gate and the same readiness convention: calls are
/// rejected with <see cref="StatusCode.Unavailable"/> until <see cref="SetReady"/> is called
/// once the site actor system is up (mirrors <c>SiteStreamGrpcServer.SetReady</c>).
/// </para>
/// <para>
/// <b>Coexistence — server-side only.</b> This makes the site ALSO listen on gRPC for commands;
/// nothing central flips to gRPC here (that is T1B.3). Central still dials sites via ClusterClient.
/// </para>
/// <para>
/// <b>Fire-and-forget.</b> The Akka path never acks <c>UnsubscribeDebugView</c>, but a unary RPC
/// must answer, so the dispatcher marks it <see cref="SiteCommandDispatcher.RouteDisposition.TellFireAndForget"/>
/// and this service Tells the target then returns the synthetic ack.
/// </para>
/// <para>
/// <b>Ack-before-Leave.</b> <c>TriggerFailover</c> resolves the standby WITHOUT leaving
/// (<see cref="SiteCommandDispatcher.PrepareFailover"/>), returns the ack, and only THEN schedules
/// the real <c>Cluster.Leave</c> — so a caller reaching the very node that is about to leave still
/// receives its ack instead of a broken stream.
/// </para>
/// </remarks>
public sealed class SiteCommandGrpcService : SiteCommandService.SiteCommandServiceBase
{
// A local Ask has no client deadline of its own; fall back to a generous ceiling when the
// caller set none (a deadline-less client is a test or an internal caller). When the caller
// DID set a gRPC deadline, honour the remaining time instead.
private static readonly TimeSpan DefaultAskTimeout = TimeSpan.FromMinutes(2);
private readonly ILogger<SiteCommandGrpcService> _logger;
private readonly Action<Action> _leaveScheduler;
// Set once by SetReady after the site actor system and dispatcher exist. Read on Kestrel
// threads, written on the host bring-up thread — volatile.
private volatile SiteCommandDispatcher? _dispatcher;
private volatile bool _ready;
/// <summary>DI constructor.</summary>
/// <param name="logger">Logger for denial/dispatch diagnostics.</param>
public SiteCommandGrpcService(ILogger<SiteCommandGrpcService> logger)
: this(logger, DeferLeaveUntilAfterReply)
{
}
/// <summary>
/// Test constructor letting a test observe when the deferred leave runs. Internal so DI sees a
/// single public constructor.
/// </summary>
/// <param name="logger">Logger.</param>
/// <param name="leaveScheduler">Runs the deferred <c>Cluster.Leave</c> after the ack is returned.</param>
internal SiteCommandGrpcService(ILogger<SiteCommandGrpcService> logger, Action<Action> leaveScheduler)
{
ArgumentNullException.ThrowIfNull(logger);
ArgumentNullException.ThrowIfNull(leaveScheduler);
_logger = logger;
_leaveScheduler = leaveScheduler;
}
/// <summary>
/// Marks the service ready and injects the shared routing table. Called once the site actor
/// system, the Deployment Manager singleton, and the local handlers are up — the same point
/// <c>SiteStreamGrpcServer.SetReady</c> is called.
/// </summary>
/// <param name="dispatcher">The shared dispatcher (the same instance the actor routes through).</param>
public void SetReady(SiteCommandDispatcher dispatcher)
{
ArgumentNullException.ThrowIfNull(dispatcher);
_dispatcher = dispatcher;
_ready = true;
}
/// <summary>Whether the service is accepting commands. Exposed for tests.</summary>
internal bool IsReady => _ready;
/// <inheritdoc />
public override async Task<LifecycleReply> ExecuteLifecycle(LifecycleRequest request, ServerCallContext context)
{
var command = SiteCommandDtoMapper.FromLifecycleRequest(request);
var reply = await DispatchAsync(command, context).ConfigureAwait(false);
return SiteCommandDtoMapper.ToLifecycleReply(reply);
}
/// <inheritdoc />
public override async Task<OpcUaReply> ExecuteOpcUa(OpcUaRequest request, ServerCallContext context)
{
var command = SiteCommandDtoMapper.FromOpcUaRequest(request);
var reply = await DispatchAsync(command, context).ConfigureAwait(false);
return SiteCommandDtoMapper.ToOpcUaReply(reply);
}
/// <inheritdoc />
public override async Task<QueryReply> ExecuteQuery(QueryRequest request, ServerCallContext context)
{
var command = SiteCommandDtoMapper.FromQueryRequest(request);
var reply = await DispatchAsync(command, context).ConfigureAwait(false);
return SiteCommandDtoMapper.ToQueryReply(reply);
}
/// <inheritdoc />
public override async Task<ParkedReply> ExecuteParked(ParkedRequest request, ServerCallContext context)
{
var command = SiteCommandDtoMapper.FromParkedRequest(request);
var reply = await DispatchAsync(command, context).ConfigureAwait(false);
return SiteCommandDtoMapper.ToParkedReply(reply);
}
/// <inheritdoc />
public override async Task<RouteReply> ExecuteRoute(RouteRequest request, ServerCallContext context)
{
var command = SiteCommandDtoMapper.FromRouteRequest(request);
var reply = await DispatchAsync(command, context).ConfigureAwait(false);
return SiteCommandDtoMapper.ToRouteReply(reply);
}
/// <inheritdoc />
public override Task<SiteFailoverAckDto> TriggerFailover(TriggerSiteFailoverDto request, ServerCallContext context)
{
var dispatcher = EnsureReady();
var msg = SiteCommandDtoMapper.FromProto(request);
// Resolve the standby and build the ack WITHOUT leaving. The real leave is deferred so the
// ack is on the wire first (ack-before-Leave).
var outcome = dispatcher.PrepareFailover(msg);
var dto = SiteCommandDtoMapper.ToProto(outcome.Ack);
if (outcome.CommitLeave is not null)
{
_leaveScheduler(outcome.CommitLeave);
}
return Task.FromResult(dto);
}
/// <summary>
/// Routes a decoded command through the shared dispatcher and returns the reply record.
/// Forward → local <c>Ask</c>; fire-and-forget → local <c>Tell</c> + synthetic ack; no target →
/// the dispatcher's synthetic "handler not available" reply.
/// </summary>
private async Task<object> DispatchAsync(object command, ServerCallContext context)
{
var dispatcher = EnsureReady();
var route = dispatcher.ResolveRoute(command);
switch (route.Disposition)
{
case SiteCommandDispatcher.RouteDisposition.ImmediateReply:
return route.Reply!;
case SiteCommandDispatcher.RouteDisposition.TellFireAndForget:
route.Target!.Tell(command);
return route.Reply!;
default:
try
{
return await route.Target!.Ask<object>(
command, AskTimeout(context), context.CancellationToken).ConfigureAwait(false);
}
catch (RpcException)
{
throw;
}
catch (OperationCanceledException)
{
// Client cancelled or the deadline elapsed.
throw new RpcException(new GrpcStatus(
StatusCode.DeadlineExceeded, "Site did not answer within the deadline."));
}
catch (Exception ex)
{
_logger.LogWarning(ex,
"Local dispatch of {Command} faulted", command.GetType().Name);
throw new RpcException(new GrpcStatus(StatusCode.Internal, ex.Message));
}
}
}
private SiteCommandDispatcher EnsureReady()
{
var dispatcher = _dispatcher;
if (!_ready || dispatcher is null)
{
throw new RpcException(new GrpcStatus(
StatusCode.Unavailable, "Site command plane not ready."));
}
return dispatcher;
}
private static TimeSpan AskTimeout(ServerCallContext context)
{
var deadline = context.Deadline;
if (deadline == DateTime.MaxValue)
{
return DefaultAskTimeout;
}
var remaining = deadline - DateTime.UtcNow;
return remaining > TimeSpan.Zero ? remaining : TimeSpan.FromMilliseconds(1);
}
// Default deferred-leave: return control (so the ack serialises) before the graceful Leave
// runs. A yield hands the current continuation back before the leave begins; the leave itself
// is slow-async (member marked Leaving, CoordinatedShutdown over seconds), so the ack is long
// gone by the time Kestrel actually stops.
private static void DeferLeaveUntilAfterReply(Action commitLeave)
=> _ = Task.Run(async () =>
{
await Task.Yield();
commitLeave();
});
}
@@ -0,0 +1,436 @@
using System.Collections.Concurrent;
using Grpc.Core;
using Grpc.Net.Client;
using Microsoft.Extensions.Logging;
using Microsoft.Extensions.Options;
namespace ZB.MOM.WW.ScadaBridge.Communication.Grpc;
/// <summary>
/// Raised when a site has no usable gRPC channel — an unknown site, or a site with neither
/// <c>GrpcNodeAAddress</c> nor <c>GrpcNodeBAddress</c> configured. The gRPC transport treats this
/// the way the Akka path treats "no ClusterClient for site": warn and drop, so the caller's Ask
/// times out (central never buffers).
/// </summary>
public sealed class SiteChannelUnavailableException(string siteId)
: Exception($"No gRPC channel is configured for site '{siteId}'.")
{
/// <summary>The site with no configured channel.</summary>
public string SiteId { get; } = siteId;
}
/// <summary>
/// Shared, per-site gRPC channel PAIR provider with sticky failover + failback (design §3.5).
/// Central holds one <see cref="GrpcChannel"/> per site node (NodeA/NodeB) rather than one channel
/// with re-dial logic; NodeA is the preferred node (config order). Every call goes to the current
/// sticky channel; on connect failure / <see cref="StatusCode.Unavailable"/> / readiness rejection
/// it flips to the other node and stays there. A background probe returns to the preferred node
/// once it answers again. Reconnect probing backs off 1 s → doubling → 60 s cap.
/// </summary>
/// <remarks>
/// <para>
/// <b>Retry safety.</b> <see cref="ExecuteAsync{T}"/> retries on the other node ONLY when the first
/// attempt failed with <see cref="StatusCode.Unavailable"/> (provably never reached a server —
/// connect refused or readiness-rejected before response headers). A
/// <see cref="StatusCode.DeadlineExceeded"/>, or any other status, is rethrown WITHOUT a cross-node
/// retry: a <c>WriteTag</c>/<c>DeployArtifacts</c>/<c>TriggerSiteFailover</c> may already have
/// executed, and the layers above tolerate the one-shot ambiguity exactly as ClusterClient's Ask
/// did — the migration must not turn one write into two.
/// </para>
/// <para>
/// <b>Addresses.</b> Fed by <see cref="UpdateSite"/>/<see cref="RemoveSite"/> from the single DB
/// refresh loop (the <c>GrpcNodeAAddress</c>/<c>GrpcNodeBAddress</c> columns). PSK credentials and
/// the <c>x-scadabridge-site</c> header ride every channel via
/// <see cref="ControlPlaneCredentials.WithSiteCredentials"/>; removing a site invalidates its
/// cached key.
/// </para>
/// </remarks>
public sealed class SitePairChannelProvider : IDisposable
{
private static readonly TimeSpan InitialBackoff = TimeSpan.FromSeconds(1);
private static readonly TimeSpan MaxBackoff = TimeSpan.FromSeconds(60);
private static readonly TimeSpan FailbackSweepInterval = TimeSpan.FromSeconds(1);
private readonly ISitePskProvider _pskProvider;
private readonly CommunicationOptions _options;
private readonly ILogger<SitePairChannelProvider> _logger;
private readonly Func<string, HttpMessageHandler>? _handlerFactory;
private readonly Func<GrpcChannel, CancellationToken, Task<bool>> _reachabilityProbe;
private readonly ConcurrentDictionary<string, SitePair> _sites = new(StringComparer.Ordinal);
private readonly CancellationTokenSource _shutdown = new();
private Task? _failbackLoop;
private readonly object _loopGate = new();
private bool _disposed;
/// <summary>Creates the provider.</summary>
/// <param name="pskProvider">Resolves each site's preshared key for channel credentials.</param>
/// <param name="options">Communication options (keepalive settings applied to production channels).</param>
/// <param name="logger">Logger for failover/failback diagnostics.</param>
/// <param name="handlerFactory">
/// Test seam mapping an endpoint to the <see cref="HttpMessageHandler"/> its channel should use
/// (e.g. an in-process <c>TestServer</c> handler). Null in production, where each channel builds
/// a keepalive-configured <see cref="SocketsHttpHandler"/>.
/// </param>
/// <param name="reachabilityProbe">
/// Test seam deciding whether a preferred node is reachable during a failback sweep. Null in
/// production, where <see cref="GrpcChannel.ConnectAsync"/> is used.
/// </param>
public SitePairChannelProvider(
ISitePskProvider pskProvider,
IOptions<CommunicationOptions> options,
ILogger<SitePairChannelProvider> logger,
Func<string, HttpMessageHandler>? handlerFactory = null,
Func<GrpcChannel, CancellationToken, Task<bool>>? reachabilityProbe = null)
{
ArgumentNullException.ThrowIfNull(pskProvider);
ArgumentNullException.ThrowIfNull(options);
ArgumentNullException.ThrowIfNull(logger);
_pskProvider = pskProvider;
_options = options.Value;
_logger = logger;
_handlerFactory = handlerFactory;
_reachabilityProbe = reachabilityProbe ?? DefaultReachabilityProbe;
}
/// <summary>
/// (Re)builds a site's channel pair from its gRPC node endpoints. A no-op when neither endpoint
/// changed. Disposes and rebuilds a channel whose endpoint changed, and resets stickiness to the
/// preferred node when the pair is (re)created. Called on the DB refresh tick.
/// </summary>
/// <param name="siteId">Site identifier.</param>
/// <param name="nodeAEndpoint">NodeA gRPC base address, or null when unconfigured.</param>
/// <param name="nodeBEndpoint">NodeB gRPC base address, or null when unconfigured.</param>
public void UpdateSite(string siteId, string? nodeAEndpoint, string? nodeBEndpoint)
{
ArgumentException.ThrowIfNullOrWhiteSpace(siteId);
var pair = _sites.GetOrAdd(siteId, id => new SitePair(id));
lock (pair.Gate)
{
var changedA = !string.Equals(pair.NodeAEndpoint, nodeAEndpoint, StringComparison.Ordinal);
var changedB = !string.Equals(pair.NodeBEndpoint, nodeBEndpoint, StringComparison.Ordinal);
if (!changedA && !changedB)
{
return;
}
if (changedA)
{
pair.ChannelA?.Dispose();
pair.ChannelA = string.IsNullOrWhiteSpace(nodeAEndpoint) ? null : BuildChannel(nodeAEndpoint, siteId);
pair.NodeAEndpoint = nodeAEndpoint;
}
if (changedB)
{
pair.ChannelB?.Dispose();
pair.ChannelB = string.IsNullOrWhiteSpace(nodeBEndpoint) ? null : BuildChannel(nodeBEndpoint, siteId);
pair.NodeBEndpoint = nodeBEndpoint;
}
// Addresses changed — return to the preferred node and reset failback backoff.
pair.CurrentIsA = pair.ChannelA is not null;
pair.ResetFailback();
_logger.LogInformation(
"gRPC channel pair for site {SiteId} refreshed (nodeA={HasA}, nodeB={HasB})",
siteId, pair.ChannelA is not null, pair.ChannelB is not null);
}
}
/// <summary>Disposes a removed site's channels and invalidates its cached preshared key.</summary>
/// <param name="siteId">Site identifier to remove.</param>
public void RemoveSite(string siteId)
{
if (string.IsNullOrWhiteSpace(siteId))
{
return;
}
if (_sites.TryRemove(siteId, out var pair))
{
lock (pair.Gate)
{
pair.ChannelA?.Dispose();
pair.ChannelB?.Dispose();
pair.ChannelA = null;
pair.ChannelB = null;
}
_pskProvider.Invalidate(siteId);
_logger.LogInformation("gRPC channel pair for site {SiteId} removed", siteId);
}
}
/// <summary>
/// Runs <paramref name="call"/> against the site's current sticky channel, failing over to the
/// other node once on <see cref="StatusCode.Unavailable"/> (never on
/// <see cref="StatusCode.DeadlineExceeded"/>). Ensures the background failback loop is running.
/// </summary>
/// <typeparam name="T">The RPC reply type.</typeparam>
/// <param name="siteId">Target site.</param>
/// <param name="call">The RPC to run against a given channel.</param>
/// <param name="ct">Cancellation token.</param>
/// <returns>The RPC reply.</returns>
/// <exception cref="SiteChannelUnavailableException">The site has no configured channel.</exception>
public async Task<T> ExecuteAsync<T>(
string siteId, Func<GrpcChannel, CancellationToken, Task<T>> call, CancellationToken ct)
{
ArgumentException.ThrowIfNullOrWhiteSpace(siteId);
ArgumentNullException.ThrowIfNull(call);
EnsureFailbackLoop();
if (!_sites.TryGetValue(siteId, out var pair))
{
throw new SiteChannelUnavailableException(siteId);
}
GrpcChannel primary;
GrpcChannel? secondary;
bool primaryIsA;
lock (pair.Gate)
{
primaryIsA = pair.CurrentIsA;
primary = (primaryIsA ? pair.ChannelA : pair.ChannelB)
?? pair.ChannelA ?? pair.ChannelB
?? throw new SiteChannelUnavailableException(siteId);
// Recompute which node `primary` actually is (the current side may be null).
primaryIsA = ReferenceEquals(primary, pair.ChannelA);
secondary = primaryIsA ? pair.ChannelB : pair.ChannelA;
}
try
{
return await call(primary, ct).ConfigureAwait(false);
}
catch (RpcException ex) when (ex.StatusCode == StatusCode.Unavailable && secondary is not null)
{
// Provably never reached a server → safe to try the other node, even for a write.
_logger.LogWarning(
"Site {SiteId} node {FromNode} unavailable; failing over to node {ToNode}",
siteId, primaryIsA ? "A" : "B", primaryIsA ? "B" : "A");
FlipTo(pair, toIsA: !primaryIsA);
return await call(secondary, ct).ConfigureAwait(false);
}
}
/// <summary>
/// Probes a site's preferred node (NodeA) and, if reachable, returns stickiness to it. The
/// background loop calls this per due site; tests call it to drive failback deterministically.
/// </summary>
/// <param name="siteId">Site identifier.</param>
/// <param name="ct">Cancellation token.</param>
/// <returns>True when the site is now (or already) pointed at its preferred node.</returns>
internal async Task<bool> TryFailbackAsync(string siteId, CancellationToken ct)
{
if (!_sites.TryGetValue(siteId, out var pair))
{
return false;
}
GrpcChannel? preferred;
lock (pair.Gate)
{
if (pair.CurrentIsA || pair.ChannelA is null)
{
return true; // already on preferred (or no preferred channel to fail back to)
}
preferred = pair.ChannelA;
}
var reachable = await _reachabilityProbe(preferred, ct).ConfigureAwait(false);
lock (pair.Gate)
{
if (reachable)
{
pair.CurrentIsA = true;
pair.ResetFailback();
_logger.LogInformation("Site {SiteId} preferred node A reachable again; failing back", siteId);
return true;
}
pair.BumpFailbackBackoff();
return false;
}
}
/// <summary>Test/diagnostic accessor: true when the site currently targets its preferred node (A).</summary>
/// <param name="siteId">Site identifier.</param>
/// <returns>True when on NodeA (or when the site is unknown).</returns>
internal bool IsOnPreferredNode(string siteId)
=> !_sites.TryGetValue(siteId, out var pair) || pair.CurrentIsA;
/// <summary>Starts the background failback loop if not already running. Idempotent.</summary>
public void EnsureFailbackLoop()
{
if (_failbackLoop is not null || _disposed)
{
return;
}
lock (_loopGate)
{
if (_failbackLoop is null && !_disposed)
{
_failbackLoop = Task.Run(() => FailbackLoopAsync(_shutdown.Token));
}
}
}
private async Task FailbackLoopAsync(CancellationToken ct)
{
using var timer = new PeriodicTimer(FailbackSweepInterval);
try
{
while (await timer.WaitForNextTickAsync(ct).ConfigureAwait(false))
{
var now = DateTime.UtcNow;
foreach (var kvp in _sites)
{
var pair = kvp.Value;
bool due;
lock (pair.Gate)
{
due = !pair.CurrentIsA && pair.ChannelA is not null && now >= pair.NextProbeUtc;
}
if (due)
{
try
{
await TryFailbackAsync(kvp.Key, ct).ConfigureAwait(false);
}
catch (OperationCanceledException)
{
throw;
}
catch (Exception ex)
{
_logger.LogDebug(ex, "Failback probe for site {SiteId} faulted", kvp.Key);
}
}
}
}
}
catch (OperationCanceledException)
{
// Shutting down.
}
}
private void FlipTo(SitePair pair, bool toIsA)
{
lock (pair.Gate)
{
pair.CurrentIsA = toIsA;
if (!toIsA)
{
// Failed over off the preferred node → schedule the first failback probe.
pair.ScheduleFailback(InitialBackoff);
}
else
{
pair.ResetFailback();
}
}
}
private GrpcChannel BuildChannel(string endpoint, string siteId)
{
var channelOptions = new GrpcChannelOptions
{
HttpHandler = _handlerFactory is not null
? _handlerFactory(endpoint)
: new SocketsHttpHandler
{
KeepAlivePingDelay = _options.GrpcKeepAlivePingDelay,
KeepAlivePingTimeout = _options.GrpcKeepAlivePingTimeout,
KeepAlivePingPolicy = HttpKeepAlivePingPolicy.Always,
EnableMultipleHttp2Connections = true
}
}.WithSiteCredentials(_pskProvider, siteId);
return GrpcChannel.ForAddress(endpoint, channelOptions);
}
private static async Task<bool> DefaultReachabilityProbe(GrpcChannel channel, CancellationToken ct)
{
try
{
using var timeout = CancellationTokenSource.CreateLinkedTokenSource(ct);
timeout.CancelAfter(TimeSpan.FromSeconds(5));
await channel.ConnectAsync(timeout.Token).ConfigureAwait(false);
return true;
}
catch
{
return false;
}
}
/// <inheritdoc />
public void Dispose()
{
if (_disposed)
{
return;
}
_disposed = true;
_shutdown.Cancel();
try
{
_failbackLoop?.Wait(TimeSpan.FromSeconds(2));
}
catch
{
// Best-effort loop drain on shutdown.
}
_shutdown.Dispose();
foreach (var pair in _sites.Values)
{
lock (pair.Gate)
{
pair.ChannelA?.Dispose();
pair.ChannelB?.Dispose();
}
}
_sites.Clear();
}
/// <summary>Per-site mutable channel state, guarded by <see cref="Gate"/>.</summary>
private sealed class SitePair(string siteId)
{
public string SiteId { get; } = siteId;
public readonly object Gate = new();
public string? NodeAEndpoint;
public string? NodeBEndpoint;
public GrpcChannel? ChannelA;
public GrpcChannel? ChannelB;
/// <summary>Sticky current node; NodeA is preferred. True = pointing at NodeA.</summary>
public bool CurrentIsA = true;
/// <summary>Next time a failback probe is due (while failed over). MaxValue = not scheduled.</summary>
public DateTime NextProbeUtc = DateTime.MaxValue;
private TimeSpan _backoff = InitialBackoff;
public void ScheduleFailback(TimeSpan delay)
{
_backoff = delay;
NextProbeUtc = DateTime.UtcNow + _backoff;
}
public void BumpFailbackBackoff()
{
var next = TimeSpan.FromTicks(Math.Min(_backoff.Ticks * 2, MaxBackoff.Ticks));
_backoff = next;
NextProbeUtc = DateTime.UtcNow + _backoff;
}
public void ResetFailback()
{
_backoff = InitialBackoff;
NextProbeUtc = DateTime.MaxValue;
}
}
}
@@ -0,0 +1,43 @@
namespace ZB.MOM.WW.ScadaBridge.Communication.Grpc;
/// <summary>
/// Site-side <see cref="ISitePskProvider"/> over a single fixed key — the one key a site node
/// presents on every control-plane call it makes to central (<c>CommunicationOptions.GrpcPsk</c>).
/// Central's provider resolves a key <em>per site</em>; a site has exactly one, so it ignores the
/// requested <c>siteId</c> and returns its own key.
/// </summary>
/// <remarks>
/// <b>Fail-closed.</b> An empty key throws, matching the contract on <see cref="ISitePskProvider"/>
/// and the interceptor's own posture: a node shipped without a key must not degrade to an
/// unauthenticated dial.
/// </remarks>
public sealed class StaticSitePskProvider : ISitePskProvider
{
private readonly string _key;
/// <summary>Creates the provider bound to a site's own preshared key.</summary>
/// <param name="key">The site's <c>GrpcPsk</c>. Empty is permitted at construction but throws on use.</param>
public StaticSitePskProvider(string key)
{
_key = key ?? string.Empty;
}
/// <inheritdoc />
public ValueTask<string> GetAsync(string siteId, CancellationToken ct)
{
if (string.IsNullOrEmpty(_key))
{
throw new InvalidOperationException(
"No gRPC preshared key is configured for this site (ScadaBridge:Communication:GrpcPsk). "
+ "The control plane is fail-closed: an unauthenticated dial does not happen.");
}
return new ValueTask<string>(_key);
}
/// <inheritdoc />
public void Invalidate(string siteId)
{
// A single static key never changes for the process lifetime; nothing to drop.
}
}
@@ -0,0 +1,247 @@
syntax = "proto3";
option csharp_namespace = "ZB.MOM.WW.ScadaBridge.Communication.Grpc";
package scadabridge.centralcontrol.v1;
import "google/protobuf/empty.proto";
import "google/protobuf/timestamp.proto";
import "google/protobuf/wrappers.proto";
// The two ingest RPCs deliberately REUSE the batch/ack messages already defined
// for the site-hosted SiteStreamService rather than redeclaring them. The site
// telemetry actor builds AuditEventBatch / CachedTelemetryBatch today and hands
// them to ISiteStreamAuditClient; duplicating the shapes here would fork one
// wire contract into two that must be kept in lockstep by hand.
import "Protos/sitestream.proto";
// Central-hosted control plane (Phase 1A of the ClusterClientgRPC migration).
//
// Direction: SITE is the client, CENTRAL is the server the inverse of
// SiteStreamService, where central dials the site. That asymmetry is deliberate
// and mirrors the direction the Akka ClusterClient traffic flows today: these
// seven calls are exactly the seven messages SiteCommunicationActor sends to
// /user/central-communication.
//
// Every call is gated by ControlPlaneAuthInterceptor: `authorization: Bearer <psk>`
// plus the `x-scadabridge-site` metadata header naming which site's preshared key
// central must verify against.
service CentralControlService {
// Store-and-forward handoff of one notification for central delivery. The
// ack is idempotent on notification_id a duplicate submit after a lost ack
// must not produce a second delivery.
rpc SubmitNotification(NotificationSubmitDto) returns (NotificationSubmitAckDto);
// Notify.Status(id) round-trip for a notification that has already left the
// site buffer. `found = false` sends the caller back to the site-local buffer
// to decide Forwarding vs Unknown.
rpc QueryNotificationStatus(NotificationStatusQueryDto) returns (NotificationStatusResponseDto);
// Audit Log (#23) push telemetry. Reuses the SiteStreamService messages: the
// batch a site drains from its SQLite hot path is byte-identical whichever
// transport carries it.
rpc IngestAuditEvents(sitestream.AuditEventBatch) returns (sitestream.IngestAck);
// Audit Log (#23) M3 combined cached-call telemetry (audit row + SiteCalls
// operational upsert, written in one central transaction).
rpc IngestCachedTelemetry(sitestream.CachedTelemetryBatch) returns (sitestream.IngestAck);
// Node-startup self-heal: the node's local deployed inventory in, fetch
// tokens for whatever it is missing or stale out.
rpc ReconcileSite(ReconcileSiteRequestDto) returns (ReconcileSiteResponseDto);
// Periodic site health report (30 s cadence). The ack makes delivery
// observable end-to-end so the sender can restore its per-interval counters
// when a report is lost.
rpc ReportSiteHealth(SiteHealthReportDto) returns (SiteHealthReportAckDto);
// Application heartbeat. Returns Empty because the message is
// fire-and-forget: nothing on the site consumes a reply, and a failure here
// must never surface as a fault on the heartbeat timer path.
rpc Heartbeat(HeartbeatDto) returns (google.protobuf.Empty);
}
// ---------------------------------------------------------------------------
// Notification Outbox (#21)
// ---------------------------------------------------------------------------
// Site -> Central: submit a buffered notification for central delivery.
// Mirrors Commons NotificationSubmit.
message NotificationSubmitDto {
string notification_id = 1; // GUID string, the idempotency key
string list_name = 2;
string subject = 3;
string body = 4;
string source_site_id = 5;
string source_instance_id = 6; // empty string represents null
string source_script = 7; // empty string represents null
google.protobuf.Timestamp site_enqueued_at = 8;
string origin_execution_id = 9; // GUID string; empty represents null
string origin_parent_execution_id = 10; // GUID string; empty represents null
string source_node = 11; // empty string represents null
}
// Central -> Site: ack sent after the Notifications row is persisted.
message NotificationSubmitAckDto {
string notification_id = 1;
bool accepted = 2;
string error = 3; // empty string represents null
}
// Site -> Central: Notify.Status(id) lookup against the central outbox.
message NotificationStatusQueryDto {
string correlation_id = 1;
string notification_id = 2;
}
// Central -> Site: current central delivery state for a queried notification.
message NotificationStatusResponseDto {
string correlation_id = 1;
bool found = 2;
string status = 3;
int32 retry_count = 4;
string last_error = 5; // empty string represents null
google.protobuf.Timestamp delivered_at = 6; // absent when null
}
// ---------------------------------------------------------------------------
// Startup reconciliation (Deployment Manager)
// ---------------------------------------------------------------------------
// Site -> Central: the node's local deployed inventory at startup.
message ReconcileSiteRequestDto {
string site_identifier = 1;
string node_id = 2;
// Instance unique name -> revision hash of the config the node currently holds.
map<string, string> local_name_to_revision_hash = 3;
}
// Central -> Site: the gap the node must (re)fetch, plus orphans to log.
message ReconcileSiteResponseDto {
repeated ReconcileGapItemDto gap = 1;
repeated string orphan_names = 2;
string central_fetch_base_url = 3;
}
// One instance the node must (re)fetch, with a freshly-minted short-TTL token.
message ReconcileGapItemDto {
string instance_unique_name = 1;
string deployment_id = 2;
string revision_hash = 3;
bool is_enabled = 4;
string fetch_token = 5;
}
// ---------------------------------------------------------------------------
// Health Monitoring (#11)
// ---------------------------------------------------------------------------
// Wire form of the Commons ConnectionHealth enum.
//
// CONNECTION_HEALTH_UNSPECIFIED exists only to keep the proto3 zero value from
// meaning something. Mapping Connected onto 0 would make an absent/garbled
// value decode as "healthy", which is precisely the wrong direction to fail;
// the mapper decodes UNSPECIFIED as Error instead and never emits it.
enum ConnectionHealthEnum {
CONNECTION_HEALTH_UNSPECIFIED = 0;
CONNECTION_HEALTH_CONNECTED = 1;
CONNECTION_HEALTH_DISCONNECTED = 2;
CONNECTION_HEALTH_CONNECTING = 3;
CONNECTION_HEALTH_ERROR = 4;
}
message TagResolutionStatusDto {
int32 total_subscribed = 1;
int32 successfully_resolved = 2;
}
message TagQualityCountsDto {
int32 good = 1;
int32 bad = 2;
int32 uncertain = 3;
}
message NodeStatusDto {
string hostname = 1;
bool is_online = 2;
string role = 3;
}
// Point-in-time snapshot of the site-local SQLite audit queue.
message SiteAuditBacklogSnapshotDto {
int32 pending_count = 1;
google.protobuf.Timestamp oldest_pending_utc = 2; // absent when the queue is empty
int64 on_disk_bytes = 3;
}
// The three collection wrappers below exist so null and empty stay
// distinguishable. proto3 cannot express presence on a `repeated` or `map`
// field an unset one and an empty one are the same bytes but the
// corresponding SiteHealthReport members are genuinely nullable
// (SiteHealthCollector emits `ClusterNodes: _clusterNodes?.ToList()`), and the
// central health surface reads null as "this producer doesn't report the
// signal" rather than "the signal is empty". Wrapping in a message restores
// message presence and makes the distinction survive the round-trip.
message ConnectionEndpointMapDto {
map<string, string> entries = 1;
}
message TagQualityMapDto {
map<string, TagQualityCountsDto> entries = 1;
}
message NodeStatusListDto {
repeated NodeStatusDto nodes = 1;
}
// Site -> Central: periodic site health report. Mirrors Commons SiteHealthReport.
// Additive-only evolution: field numbers are never reused.
message SiteHealthReportDto {
string site_id = 1;
int64 sequence_number = 2;
google.protobuf.Timestamp report_timestamp = 3;
map<string, ConnectionHealthEnum> data_connection_statuses = 4;
map<string, TagResolutionStatusDto> tag_resolution_counts = 5;
int32 script_error_count = 6;
int32 alarm_evaluation_error_count = 7;
map<string, int32> store_and_forward_buffer_depths = 8;
int32 dead_letter_count = 9;
int32 deployed_instance_count = 10;
int32 enabled_instance_count = 11;
int32 disabled_instance_count = 12;
string node_role = 13;
string node_hostname = 14;
ConnectionEndpointMapDto data_connection_endpoints = 15; // absent when null
TagQualityMapDto data_connection_tag_quality = 16; // absent when null
int32 parked_message_count = 17;
NodeStatusListDto cluster_nodes = 18; // absent when null
int32 site_audit_write_failures = 19;
int32 audit_redaction_failure = 20;
SiteAuditBacklogSnapshotDto site_audit_backlog = 21; // absent when no data yet
int64 site_event_log_write_failures = 22;
google.protobuf.DoubleValue oldest_parked_message_age_seconds = 23; // absent when nothing parked
int32 script_queue_depth = 24;
int32 script_busy_threads = 25;
google.protobuf.DoubleValue script_oldest_busy_age_seconds = 26; // absent when the pool is idle
// Nullable on purpose: absent means "replication not wired on this node",
// which is NOT the same as false ("wired but currently disconnected").
google.protobuf.BoolValue local_db_replication_connected = 27;
// Absent means UNKNOWN, never zero a failed backlog read rendered as 0
// would report a broken replication pair as perfectly healthy.
google.protobuf.Int64Value local_db_oplog_backlog = 28;
}
// Central -> Site: health report ack, so a lost report is observable.
message SiteHealthReportAckDto {
string site_id = 1;
int64 sequence_number = 2;
bool accepted = 3;
string error = 4; // empty string represents null
}
// Site -> Central: application heartbeat (fire-and-forget; reply is Empty).
message HeartbeatDto {
string site_id = 1;
string node_hostname = 2;
bool is_active = 3;
google.protobuf.Timestamp timestamp = 4;
}
@@ -0,0 +1,857 @@
syntax = "proto3";
option csharp_namespace = "ZB.MOM.WW.ScadaBridge.Communication.Grpc";
package scadabridge.sitecommand.v1;
import "google/protobuf/timestamp.proto";
import "google/protobuf/duration.proto";
import "google/protobuf/wrappers.proto";
//
// Site command plane (central site) the gRPC replacement for the per-site
// ClusterClient command/control channel handled by SiteCommunicationActor.
//
// The 28 migrated commands are grouped into six domain RPCs rather than 28
// individual RPCs, because the grouping is what carries the DEADLINE policy:
// every command inside one group shares a timeout class today (see
// CommunicationOptions), so one RPC per group keeps the deadline choice in one
// place on the client and one dispatch switch on the server. A `oneof` request
// / reply envelope preserves per-command typing inside the group.
//
// IntegrationCallRequest (the 29th command) is deliberately NOT here it is
// dead code at both ends; see
// docs/known-issues/2026-07-22-integration-call-routing-is-dead-code.md.
//
// EVOLUTION RULE (same as sitestream.proto): field numbers are never reused and
// changes are additive only. New commands take the next free oneof tag; an
// older peer simply sees an unset oneof and answers with a clean error.
//
// The generated C# is CHECKED IN under Communication/SiteCommandGrpc/ because
// protoc segfaults inside the linux_arm64 Docker build image. Regenerate with
// docker/regen-proto.sh (or by hand per the recipe in the .csproj) never by
// leaving an active <Protobuf> item in the project file.
//
service SiteCommandService {
// Deployment + instance lifecycle (6 commands).
rpc ExecuteLifecycle(LifecycleRequest) returns (LifecycleReply);
// Interactive OPC UA / MxGateway design-time commands (8 commands).
rpc ExecuteOpcUa(OpcUaRequest) returns (OpcUaReply);
// Remote read-only queries: event log + debug view (4 commands).
rpc ExecuteQuery(QueryRequest) returns (QueryReply);
// Parked store-and-forward message + cached-operation actions (5 commands).
rpc ExecuteParked(ParkedRequest) returns (ParkedReply);
// Inbound-API Route.To() relays (4 commands).
rpc ExecuteRoute(RouteRequest) returns (RouteReply);
// Operator-initiated manual site-pair failover (1 command).
rpc TriggerFailover(TriggerSiteFailoverDto) returns (SiteFailoverAckDto);
}
//
// Shared value carrier
//
// Explicit null marker. Needed because a map<string, LooseValue> entry always
// has a value message present, so "the dictionary holds a null for this key"
// cannot be expressed by absence the way a nullable message FIELD can.
message LooseNull {}
message LooseValueList { repeated LooseValue items = 1; }
message LooseValueMap { map<string, LooseValue> entries = 1; }
// Type-tagged carrier for the `object?` members the command plane still uses
// (script parameters and return values, attribute values, tag read/write
// values). The tags cover every CLR type these fields actually carry, so those
// values round-trip with their runtime type intact; anything else falls back to
// `json_value` (see the mapper's documented lossiness note).
//
// Date/time and decimal ride as invariant round-trip STRINGS rather than
// google.protobuf.Timestamp: Timestamp normalises everything to UTC and would
// silently drop DateTime.Kind / DateTimeOffset.Offset, which for an operator's
// tag value is data loss, not normalisation.
message LooseValue {
oneof kind {
LooseNull null_value = 1;
bool bool_value = 2;
int32 int32_value = 3;
int64 int64_value = 4;
double double_value = 5;
float float_value = 6;
string string_value = 7;
string decimal_value = 8; // invariant-culture round-trip
string date_time_value = 9; // DateTime, "O" (preserves Kind)
string date_time_offset_value = 10; // DateTimeOffset, "O" (preserves Offset)
string guid_value = 11; // "D"
string time_span_value = 12; // "c" (constant/round-trip)
bytes bytes_value = 13;
LooseValueList list_value = 14;
LooseValueMap map_value = 15;
string json_value = 16; // fallback; decodes to JsonElement
}
}
//
// Enums. Every enum reserves 0 for _UNSPECIFIED so a value that is absent on
// the wire is never mistaken for the first CLR member. The mapper translates
// explicitly in both directions (never by ordinal), so reordering a C# enum
// cannot silently re-map the wire.
//
enum DeploymentStatusDto {
DEPLOYMENT_STATUS_DTO_UNSPECIFIED = 0;
DEPLOYMENT_STATUS_DTO_PENDING = 1;
DEPLOYMENT_STATUS_DTO_IN_PROGRESS = 2;
DEPLOYMENT_STATUS_DTO_SUCCESS = 3;
DEPLOYMENT_STATUS_DTO_FAILED = 4;
}
enum BrowseNodeClassDto {
BROWSE_NODE_CLASS_DTO_UNSPECIFIED = 0;
BROWSE_NODE_CLASS_DTO_OBJECT = 1;
BROWSE_NODE_CLASS_DTO_VARIABLE = 2;
BROWSE_NODE_CLASS_DTO_METHOD = 3;
BROWSE_NODE_CLASS_DTO_OTHER = 4;
}
enum BrowseFailureKindDto {
BROWSE_FAILURE_KIND_DTO_UNSPECIFIED = 0;
BROWSE_FAILURE_KIND_DTO_CONNECTION_NOT_FOUND = 1;
BROWSE_FAILURE_KIND_DTO_CONNECTION_NOT_CONNECTED = 2;
BROWSE_FAILURE_KIND_DTO_NOT_BROWSABLE = 3;
BROWSE_FAILURE_KIND_DTO_TIMEOUT = 4;
BROWSE_FAILURE_KIND_DTO_SERVER_ERROR = 5;
}
enum ReadTagValuesFailureKindDto {
READ_TAG_VALUES_FAILURE_KIND_DTO_UNSPECIFIED = 0;
READ_TAG_VALUES_FAILURE_KIND_DTO_CONNECTION_NOT_FOUND = 1;
READ_TAG_VALUES_FAILURE_KIND_DTO_CONNECTION_NOT_CONNECTED = 2;
READ_TAG_VALUES_FAILURE_KIND_DTO_TIMEOUT = 3;
READ_TAG_VALUES_FAILURE_KIND_DTO_SERVER_ERROR = 4;
}
enum VerifyFailureKindDto {
VERIFY_FAILURE_KIND_DTO_UNSPECIFIED = 0;
VERIFY_FAILURE_KIND_DTO_UNREACHABLE = 1;
VERIFY_FAILURE_KIND_DTO_AUTH_FAILED = 2;
VERIFY_FAILURE_KIND_DTO_UNTRUSTED_CERTIFICATE = 3;
VERIFY_FAILURE_KIND_DTO_TIMEOUT = 4;
VERIFY_FAILURE_KIND_DTO_SERVER_ERROR = 5;
}
enum StoreAndForwardCategoryDto {
STORE_AND_FORWARD_CATEGORY_DTO_UNSPECIFIED = 0;
STORE_AND_FORWARD_CATEGORY_DTO_EXTERNAL_SYSTEM = 1;
STORE_AND_FORWARD_CATEGORY_DTO_NOTIFICATION = 2;
STORE_AND_FORWARD_CATEGORY_DTO_CACHED_DB_WRITE = 3;
}
enum AlarmStateDto {
ALARM_STATE_DTO_UNSPECIFIED = 0;
ALARM_STATE_DTO_ACTIVE = 1;
ALARM_STATE_DTO_NORMAL = 2;
}
enum AlarmLevelDto {
ALARM_LEVEL_DTO_UNSPECIFIED = 0;
ALARM_LEVEL_DTO_NONE = 1;
ALARM_LEVEL_DTO_LOW = 2;
ALARM_LEVEL_DTO_LOW_LOW = 3;
ALARM_LEVEL_DTO_HIGH = 4;
ALARM_LEVEL_DTO_HIGH_HIGH = 5;
}
enum AlarmKindDto {
ALARM_KIND_DTO_UNSPECIFIED = 0;
ALARM_KIND_DTO_COMPUTED = 1;
ALARM_KIND_DTO_NATIVE_OPC_UA = 2;
ALARM_KIND_DTO_NATIVE_MX_ACCESS = 3;
}
enum AlarmShelveStateDto {
ALARM_SHELVE_STATE_DTO_UNSPECIFIED = 0;
ALARM_SHELVE_STATE_DTO_UNSHELVED = 1;
ALARM_SHELVE_STATE_DTO_ONE_SHOT_SHELVED = 2;
ALARM_SHELVE_STATE_DTO_TIMED_SHELVED = 3;
ALARM_SHELVE_STATE_DTO_PERMANENT_SHELVED = 4;
}
//
// RPC 1 ExecuteLifecycle
//
message RefreshDeploymentCommandDto {
string deployment_id = 1;
string instance_unique_name = 2;
string revision_hash = 3;
string deployed_by = 4;
google.protobuf.Timestamp timestamp = 5;
string central_fetch_base_url = 6;
string fetch_token = 7;
}
message EnableInstanceCommandDto {
string command_id = 1;
string instance_unique_name = 2;
google.protobuf.Timestamp timestamp = 3;
}
message DisableInstanceCommandDto {
string command_id = 1;
string instance_unique_name = 2;
google.protobuf.Timestamp timestamp = 3;
}
message DeleteInstanceCommandDto {
string command_id = 1;
string instance_unique_name = 2;
google.protobuf.Timestamp timestamp = 3;
}
message DeploymentStateQueryRequestDto {
string correlation_id = 1;
string instance_unique_name = 2;
google.protobuf.Timestamp timestamp = 3;
}
message SharedScriptArtifactDto {
string name = 1;
string code = 2;
string parameter_definitions = 3; // empty string represents null
string return_definition = 4; // empty string represents null
}
message ExternalSystemArtifactDto {
string name = 1;
string endpoint_url = 2;
string auth_type = 3;
string auth_configuration = 4; // empty string represents null
string method_definitions_json = 5; // empty string represents null
int32 timeout_seconds = 6;
}
message DatabaseConnectionArtifactDto {
string name = 1;
string connection_string = 2;
int32 max_retries = 3;
google.protobuf.Duration retry_delay = 4;
}
message NotificationListArtifactDto {
string name = 1;
repeated string recipient_emails = 2;
}
message DataConnectionArtifactDto {
string name = 1;
string protocol = 2;
string primary_configuration_json = 3; // empty string represents null
string backup_configuration_json = 4; // empty string represents null
int32 failover_retry_count = 5;
}
message SmtpConfigurationArtifactDto {
string name = 1;
string server = 2;
int32 port = 3;
string auth_mode = 4;
string from_address = 5;
string username = 6; // empty string represents null
string password = 7; // empty string represents null
string oauth_config = 8; // empty string represents null
}
// Each artifact collection on DeployArtifactsCommand is a NULLABLE list, and
// proto3 `repeated` cannot distinguish null from empty. Wrapping each in its
// own message makes the null/empty distinction a message-presence question,
// which proto3 does model the same silent-data-loss class the transport
// round-trip guard caught in PLAN-05 T8.
message SharedScriptArtifactListDto { repeated SharedScriptArtifactDto items = 1; }
message ExternalSystemArtifactListDto { repeated ExternalSystemArtifactDto items = 1; }
message DatabaseConnectionArtifactListDto { repeated DatabaseConnectionArtifactDto items = 1; }
message NotificationListArtifactListDto { repeated NotificationListArtifactDto items = 1; }
message DataConnectionArtifactListDto { repeated DataConnectionArtifactDto items = 1; }
message SmtpConfigurationArtifactListDto { repeated SmtpConfigurationArtifactDto items = 1; }
message DeployArtifactsCommandDto {
string deployment_id = 1;
SharedScriptArtifactListDto shared_scripts = 2; // absent => null
ExternalSystemArtifactListDto external_systems = 3; // absent => null
DatabaseConnectionArtifactListDto database_connections = 4; // absent => null
NotificationListArtifactListDto notification_lists = 5; // absent => null
DataConnectionArtifactListDto data_connections = 6; // absent => null
SmtpConfigurationArtifactListDto smtp_configurations = 7; // absent => null
google.protobuf.Timestamp timestamp = 8;
}
message DeploymentStatusResponseDto {
string deployment_id = 1;
string instance_unique_name = 2;
DeploymentStatusDto status = 3;
string error_message = 4; // empty string represents null
google.protobuf.Timestamp timestamp = 5;
}
message InstanceLifecycleResponseDto {
string command_id = 1;
string instance_unique_name = 2;
bool success = 3;
string error_message = 4; // empty string represents null
google.protobuf.Timestamp timestamp = 5;
}
message DeploymentStateQueryResponseDto {
string correlation_id = 1;
string instance_unique_name = 2;
bool is_deployed = 3;
string applied_deployment_id = 4; // empty string represents null
string applied_revision_hash = 5; // empty string represents null
google.protobuf.Timestamp timestamp = 6;
}
message ArtifactDeploymentResponseDto {
string deployment_id = 1;
string site_id = 2;
bool success = 3;
string error_message = 4; // empty string represents null
google.protobuf.Timestamp timestamp = 5;
}
message LifecycleRequest {
oneof command {
RefreshDeploymentCommandDto refresh_deployment = 1;
EnableInstanceCommandDto enable_instance = 2;
DisableInstanceCommandDto disable_instance = 3;
DeleteInstanceCommandDto delete_instance = 4;
DeploymentStateQueryRequestDto deployment_state_query = 5;
DeployArtifactsCommandDto deploy_artifacts = 6;
}
}
message LifecycleReply {
oneof reply {
DeploymentStatusResponseDto deployment_status = 1;
InstanceLifecycleResponseDto instance_lifecycle = 2;
DeploymentStateQueryResponseDto deployment_state_query = 3;
ArtifactDeploymentResponseDto artifact_deployment = 4;
}
}
//
// RPC 2 ExecuteOpcUa
//
message BrowseNodeCommandDto {
string connection_name = 1;
string parent_node_id = 2; // empty string represents null (browse root)
string continuation_token = 3; // empty string represents null (first page)
string site_identifier = 4; // empty string represents null
}
message BrowseNodeDto {
string node_id = 1;
string display_name = 2;
BrowseNodeClassDto node_class = 3;
bool has_children = 4;
string data_type = 5; // empty string represents null
google.protobuf.Int32Value value_rank = 6; // absent => null
google.protobuf.BoolValue writable = 7; // absent => null
}
message BrowseFailureDto {
BrowseFailureKindDto kind = 1;
string message = 2;
}
message BrowseNodeResultDto {
repeated BrowseNodeDto children = 1;
bool truncated = 2;
BrowseFailureDto failure = 3; // absent => null (success)
string continuation_token = 4; // empty string represents null (final page)
}
message SearchAddressSpaceCommandDto {
string connection_name = 1;
string query = 2;
int32 max_depth = 3;
int32 max_results = 4;
string site_identifier = 5; // empty string represents null
}
message AddressSpaceMatchDto {
BrowseNodeDto node = 1;
string path = 2;
}
message SearchAddressSpaceResultDto {
repeated AddressSpaceMatchDto matches = 1;
bool cap_reached = 2;
BrowseFailureDto failure = 3; // absent => null (success)
}
message ReadTagValuesCommandDto {
string connection_name = 1;
repeated string tag_paths = 2;
}
message TagReadOutcomeDto {
string tag_path = 1;
bool success = 2;
LooseValue value = 3; // absent => null
string quality = 4;
google.protobuf.Timestamp timestamp = 5;
string error_message = 6; // empty string represents null
}
message ReadTagValuesFailureDto {
ReadTagValuesFailureKindDto kind = 1;
string message = 2;
}
message ReadTagValuesResultDto {
repeated TagReadOutcomeDto outcomes = 1;
ReadTagValuesFailureDto failure = 2; // absent => null (success)
}
message VerifyEndpointCommandDto {
string connection_name = 1;
string protocol = 2;
string config_json = 3;
string site_identifier = 4; // empty string represents null
}
message ServerCertInfoDto {
string thumbprint = 1;
string subject = 2;
string issuer = 3;
google.protobuf.Timestamp not_before_utc = 4;
google.protobuf.Timestamp not_after_utc = 5;
string der_base64 = 6;
}
// failure_kind is a NULLABLE enum on the CLR side, and proto3 enums have no
// presence, so it rides inside a one-field message exactly like Int32Value.
message VerifyFailureKindValue { VerifyFailureKindDto value = 1; }
message VerifyEndpointResultDto {
bool success = 1;
VerifyFailureKindValue failure_kind = 2; // absent => null (success)
string error = 3; // empty string represents null
ServerCertInfoDto cert = 4; // absent => null
}
message TrustServerCertCommandDto {
string connection_name = 1;
string der_base64 = 2;
string thumbprint = 3;
string site_identifier = 4; // empty string represents null
}
message ListServerCertsCommandDto {
string site_identifier = 1; // empty string represents null
}
message RemoveServerCertCommandDto {
string thumbprint = 1;
string site_identifier = 2; // empty string represents null
}
message TrustedCertInfoDto {
string thumbprint = 1;
string subject = 2;
string issuer = 3;
google.protobuf.Timestamp not_before_utc = 4;
google.protobuf.Timestamp not_after_utc = 5;
bool rejected = 6;
}
// CertTrustResult.Certs is a nullable list (null for trust/remove, populated
// for list) same null-vs-empty problem as the artifact collections.
message TrustedCertInfoListDto { repeated TrustedCertInfoDto items = 1; }
message CertTrustResultDto {
bool success = 1;
string error = 2; // empty string represents null
TrustedCertInfoListDto certs = 3; // absent => null
}
message WriteTagRequestDto {
string correlation_id = 1;
string connection_name = 2;
string tag_path = 3;
LooseValue value = 4; // absent => null
google.protobuf.Timestamp timestamp = 5;
}
message WriteTagResponseDto {
string correlation_id = 1;
bool success = 2;
string error_message = 3; // empty string represents null
google.protobuf.Timestamp timestamp = 4;
}
message OpcUaRequest {
oneof command {
BrowseNodeCommandDto browse_node = 1;
SearchAddressSpaceCommandDto search_address_space = 2;
ReadTagValuesCommandDto read_tag_values = 3;
VerifyEndpointCommandDto verify_endpoint = 4;
TrustServerCertCommandDto trust_server_cert = 5;
ListServerCertsCommandDto list_server_certs = 6;
RemoveServerCertCommandDto remove_server_cert = 7;
WriteTagRequestDto write_tag = 8;
}
}
message OpcUaReply {
oneof reply {
BrowseNodeResultDto browse_node = 1;
SearchAddressSpaceResultDto search_address_space = 2;
ReadTagValuesResultDto read_tag_values = 3;
VerifyEndpointResultDto verify_endpoint = 4;
CertTrustResultDto cert_trust = 5;
WriteTagResponseDto write_tag = 6;
}
}
//
// RPC 3 ExecuteQuery
//
message EventLogQueryRequestDto {
string correlation_id = 1;
string site_id = 2;
google.protobuf.Timestamp from = 3; // absent => null
google.protobuf.Timestamp to = 4; // absent => null
string event_type = 5; // empty string represents null
string severity = 6; // empty string represents null
string instance_id = 7; // empty string represents null
string keyword_filter = 8; // empty string represents null
string continuation_token = 9; // empty string represents null
int32 page_size = 10;
google.protobuf.Timestamp timestamp = 11;
}
message EventLogEntryDto {
string id = 1;
google.protobuf.Timestamp timestamp = 2;
string event_type = 3;
string severity = 4;
string instance_id = 5; // empty string represents null
string source = 6;
string message = 7;
string details = 8; // empty string represents null
}
message EventLogQueryResponseDto {
string correlation_id = 1;
string site_id = 2;
repeated EventLogEntryDto entries = 3;
string continuation_token = 4; // empty string represents null
bool has_more = 5;
bool success = 6;
string error_message = 7; // empty string represents null
google.protobuf.Timestamp timestamp = 8;
}
message DebugSnapshotRequestDto {
string instance_unique_name = 1;
string correlation_id = 2;
}
message SubscribeDebugViewRequestDto {
string instance_unique_name = 1;
string correlation_id = 2;
}
message UnsubscribeDebugViewRequestDto {
string instance_unique_name = 1;
string correlation_id = 2;
}
// UnsubscribeDebugView is a Tell today (CommunicationService.UnsubscribeDebugView
// fires and forgets). A unary RPC must still answer something, so the site sends
// this empty ack; the central transport ignores it to keep the caller-visible
// fire-and-forget semantics identical.
message UnsubscribeDebugViewAckDto {}
message AlarmConditionStateDto {
bool active = 1;
bool acknowledged = 2;
google.protobuf.BoolValue confirmed = 3; // absent => null (not confirmable)
AlarmShelveStateDto shelve = 4;
bool suppressed = 5;
int32 severity = 6;
}
message DebugAttributeValueDto {
string instance_unique_name = 1;
string attribute_path = 2;
string attribute_name = 3;
LooseValue value = 4; // absent => null
string quality = 5;
google.protobuf.Timestamp timestamp = 6;
}
// Full-fidelity projection of Commons AlarmStateChanged. Deliberately NOT the
// sitestream AlarmStateUpdate: that one flattens the value to a display string
// and the shelve state to free text, which is right for a live stream but would
// lose data on a snapshot the central UI renders as authoritative state.
message DebugAlarmStateDto {
string instance_unique_name = 1;
string alarm_name = 2;
AlarmStateDto state = 3;
int32 priority = 4;
google.protobuf.Timestamp timestamp = 5;
AlarmLevelDto level = 6;
string message = 7;
AlarmKindDto kind = 8;
// Absent means "not explicitly set" the CLR record then derives the
// computed default from state + priority. See the mapper's note on why the
// encoder omits a condition that already equals that derived default.
AlarmConditionStateDto condition = 9;
string source_reference = 10;
string alarm_type_name = 11;
string category = 12;
string operator_user = 13;
string operator_comment = 14;
google.protobuf.Timestamp original_raise_time = 15; // absent => null
string current_value = 16;
string limit_value = 17;
string native_source_canonical_name = 18;
bool is_configured_placeholder = 19;
}
message DebugViewSnapshotDto {
string instance_unique_name = 1;
repeated DebugAttributeValueDto attribute_values = 2;
repeated DebugAlarmStateDto alarm_states = 3;
google.protobuf.Timestamp snapshot_timestamp = 4;
bool instance_not_found = 5;
}
message QueryRequest {
oneof command {
EventLogQueryRequestDto event_log_query = 1;
DebugSnapshotRequestDto debug_snapshot = 2;
SubscribeDebugViewRequestDto subscribe_debug_view = 3;
UnsubscribeDebugViewRequestDto unsubscribe_debug_view = 4;
}
}
message QueryReply {
oneof reply {
EventLogQueryResponseDto event_log_query = 1;
DebugViewSnapshotDto debug_view_snapshot = 2;
UnsubscribeDebugViewAckDto unsubscribe_debug_view = 3;
}
}
//
// RPC 4 ExecuteParked
//
message ParkedMessageQueryRequestDto {
string correlation_id = 1;
string site_id = 2;
int32 page_number = 3;
int32 page_size = 4;
google.protobuf.Timestamp timestamp = 5;
}
message ParkedMessageEntryDto {
string message_id = 1;
string target_system = 2;
string method_name = 3;
string error_message = 4;
int32 attempt_count = 5;
google.protobuf.Timestamp original_timestamp = 6;
google.protobuf.Timestamp last_attempt_timestamp = 7;
int32 max_attempts = 8;
StoreAndForwardCategoryDto category = 9;
string origin_instance = 10; // empty string represents null
}
message ParkedMessageQueryResponseDto {
string correlation_id = 1;
string site_id = 2;
repeated ParkedMessageEntryDto messages = 3;
int32 total_count = 4;
int32 page_number = 5;
int32 page_size = 6;
bool success = 7;
string error_message = 8; // empty string represents null
google.protobuf.Timestamp timestamp = 9;
}
message ParkedMessageRetryRequestDto {
string correlation_id = 1;
string site_id = 2;
string message_id = 3;
google.protobuf.Timestamp timestamp = 4;
}
message ParkedMessageRetryResponseDto {
string correlation_id = 1;
bool success = 2;
string error_message = 3; // empty string represents null
}
message ParkedMessageDiscardRequestDto {
string correlation_id = 1;
string site_id = 2;
string message_id = 3;
google.protobuf.Timestamp timestamp = 4;
}
message ParkedMessageDiscardResponseDto {
string correlation_id = 1;
bool success = 2;
string error_message = 3; // empty string represents null
}
message RetryParkedOperationDto {
string correlation_id = 1;
string tracked_operation_id = 2; // TrackedOperationId GUID, "D" format
}
message DiscardParkedOperationDto {
string correlation_id = 1;
string tracked_operation_id = 2; // TrackedOperationId GUID, "D" format
}
message ParkedOperationActionAckDto {
string correlation_id = 1;
bool applied = 2;
string error_message = 3; // empty string represents null
}
message ParkedRequest {
oneof command {
ParkedMessageQueryRequestDto parked_message_query = 1;
ParkedMessageRetryRequestDto parked_message_retry = 2;
ParkedMessageDiscardRequestDto parked_message_discard = 3;
RetryParkedOperationDto retry_parked_operation = 4;
DiscardParkedOperationDto discard_parked_operation = 5;
}
}
message ParkedReply {
oneof reply {
ParkedMessageQueryResponseDto parked_message_query = 1;
ParkedMessageRetryResponseDto parked_message_retry = 2;
ParkedMessageDiscardResponseDto parked_message_discard = 3;
ParkedOperationActionAckDto parked_operation_action = 4;
}
}
//
// RPC 5 ExecuteRoute
//
message RouteToCallRequestDto {
string correlation_id = 1;
string instance_unique_name = 2;
string script_name = 3;
LooseValueMap parameters = 4; // absent => null (distinct from empty)
google.protobuf.Timestamp timestamp = 5;
string parent_execution_id = 6; // empty string represents null
}
message RouteToCallResponseDto {
string correlation_id = 1;
bool success = 2;
LooseValue return_value = 3; // absent => null
string error_message = 4; // empty string represents null
google.protobuf.Timestamp timestamp = 5;
}
message RouteToGetAttributesRequestDto {
string correlation_id = 1;
string instance_unique_name = 2;
repeated string attribute_names = 3;
google.protobuf.Timestamp timestamp = 4;
string parent_execution_id = 5; // empty string represents null
}
message RouteToGetAttributesResponseDto {
string correlation_id = 1;
LooseValueMap values = 2; // non-nullable on the CLR side; absent => empty
bool success = 3;
string error_message = 4; // empty string represents null
google.protobuf.Timestamp timestamp = 5;
}
message RouteToSetAttributesRequestDto {
string correlation_id = 1;
string instance_unique_name = 2;
map<string, string> attribute_values = 3;
google.protobuf.Timestamp timestamp = 4;
string parent_execution_id = 5; // empty string represents null
}
message RouteToSetAttributesResponseDto {
string correlation_id = 1;
bool success = 2;
string error_message = 3; // empty string represents null
google.protobuf.Timestamp timestamp = 4;
}
message RouteToWaitForAttributeRequestDto {
string correlation_id = 1;
string instance_unique_name = 2;
string attribute_name = 3;
// NOT the empty-string-means-null convention: "wait for this attribute to
// become the empty string" is a legitimate target that must stay distinct
// from "no target supplied", so this one nullable string rides a wrapper.
google.protobuf.StringValue target_value_encoded = 4;
google.protobuf.Duration timeout = 5;
google.protobuf.Timestamp timestamp = 6;
string parent_execution_id = 7; // empty string represents null
bool require_good_quality = 8;
}
message RouteToWaitForAttributeResponseDto {
string correlation_id = 1;
bool matched = 2;
LooseValue value = 3; // absent => null
string quality = 4; // empty string represents null
bool timed_out = 5;
bool success = 6;
string error_message = 7; // empty string represents null
google.protobuf.Timestamp timestamp = 8;
}
message RouteRequest {
oneof command {
RouteToCallRequestDto route_to_call = 1;
RouteToGetAttributesRequestDto route_to_get_attributes = 2;
RouteToSetAttributesRequestDto route_to_set_attributes = 3;
RouteToWaitForAttributeRequestDto route_to_wait_for_attribute = 4;
}
}
message RouteReply {
oneof reply {
RouteToCallResponseDto route_to_call = 1;
RouteToGetAttributesResponseDto route_to_get_attributes = 2;
RouteToSetAttributesResponseDto route_to_set_attributes = 3;
RouteToWaitForAttributeResponseDto route_to_wait_for_attribute = 4;
}
}
//
// RPC 6 TriggerFailover
//
message TriggerSiteFailoverDto {
string correlation_id = 1;
string site_id = 2;
}
message SiteFailoverAckDto {
string correlation_id = 1;
bool accepted = 2;
string target_address = 3; // empty string represents null
string error_message = 4; // empty string represents null
}
File diff suppressed because it is too large Load Diff
@@ -0,0 +1,559 @@
// <auto-generated>
// Generated by the protocol buffer compiler. DO NOT EDIT!
// source: Protos/site_command.proto
// </auto-generated>
#pragma warning disable 0414, 1591, 8981, 0612
#region Designer generated code
using grpc = global::Grpc.Core;
namespace ZB.MOM.WW.ScadaBridge.Communication.Grpc {
public static partial class SiteCommandService
{
static readonly string __ServiceName = "scadabridge.sitecommand.v1.SiteCommandService";
[global::System.CodeDom.Compiler.GeneratedCode("grpc_csharp_plugin", null)]
static void __Helper_SerializeMessage(global::Google.Protobuf.IMessage message, grpc::SerializationContext context)
{
#if !GRPC_DISABLE_PROTOBUF_BUFFER_SERIALIZATION
if (message is global::Google.Protobuf.IBufferMessage)
{
context.SetPayloadLength(message.CalculateSize());
global::Google.Protobuf.MessageExtensions.WriteTo(message, context.GetBufferWriter());
context.Complete();
return;
}
#endif
context.Complete(global::Google.Protobuf.MessageExtensions.ToByteArray(message));
}
[global::System.CodeDom.Compiler.GeneratedCode("grpc_csharp_plugin", null)]
static class __Helper_MessageCache<T>
{
public static readonly bool IsBufferMessage = global::System.Reflection.IntrospectionExtensions.GetTypeInfo(typeof(global::Google.Protobuf.IBufferMessage)).IsAssignableFrom(typeof(T));
}
[global::System.CodeDom.Compiler.GeneratedCode("grpc_csharp_plugin", null)]
static T __Helper_DeserializeMessage<T>(grpc::DeserializationContext context, global::Google.Protobuf.MessageParser<T> parser) where T : global::Google.Protobuf.IMessage<T>
{
#if !GRPC_DISABLE_PROTOBUF_BUFFER_SERIALIZATION
if (__Helper_MessageCache<T>.IsBufferMessage)
{
return parser.ParseFrom(context.PayloadAsReadOnlySequence());
}
#endif
return parser.ParseFrom(context.PayloadAsNewBuffer());
}
[global::System.CodeDom.Compiler.GeneratedCode("grpc_csharp_plugin", null)]
static readonly grpc::Marshaller<global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.LifecycleRequest> __Marshaller_scadabridge_sitecommand_v1_LifecycleRequest = grpc::Marshallers.Create(__Helper_SerializeMessage, context => __Helper_DeserializeMessage(context, global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.LifecycleRequest.Parser));
[global::System.CodeDom.Compiler.GeneratedCode("grpc_csharp_plugin", null)]
static readonly grpc::Marshaller<global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.LifecycleReply> __Marshaller_scadabridge_sitecommand_v1_LifecycleReply = grpc::Marshallers.Create(__Helper_SerializeMessage, context => __Helper_DeserializeMessage(context, global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.LifecycleReply.Parser));
[global::System.CodeDom.Compiler.GeneratedCode("grpc_csharp_plugin", null)]
static readonly grpc::Marshaller<global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.OpcUaRequest> __Marshaller_scadabridge_sitecommand_v1_OpcUaRequest = grpc::Marshallers.Create(__Helper_SerializeMessage, context => __Helper_DeserializeMessage(context, global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.OpcUaRequest.Parser));
[global::System.CodeDom.Compiler.GeneratedCode("grpc_csharp_plugin", null)]
static readonly grpc::Marshaller<global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.OpcUaReply> __Marshaller_scadabridge_sitecommand_v1_OpcUaReply = grpc::Marshallers.Create(__Helper_SerializeMessage, context => __Helper_DeserializeMessage(context, global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.OpcUaReply.Parser));
[global::System.CodeDom.Compiler.GeneratedCode("grpc_csharp_plugin", null)]
static readonly grpc::Marshaller<global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.QueryRequest> __Marshaller_scadabridge_sitecommand_v1_QueryRequest = grpc::Marshallers.Create(__Helper_SerializeMessage, context => __Helper_DeserializeMessage(context, global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.QueryRequest.Parser));
[global::System.CodeDom.Compiler.GeneratedCode("grpc_csharp_plugin", null)]
static readonly grpc::Marshaller<global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.QueryReply> __Marshaller_scadabridge_sitecommand_v1_QueryReply = grpc::Marshallers.Create(__Helper_SerializeMessage, context => __Helper_DeserializeMessage(context, global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.QueryReply.Parser));
[global::System.CodeDom.Compiler.GeneratedCode("grpc_csharp_plugin", null)]
static readonly grpc::Marshaller<global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.ParkedRequest> __Marshaller_scadabridge_sitecommand_v1_ParkedRequest = grpc::Marshallers.Create(__Helper_SerializeMessage, context => __Helper_DeserializeMessage(context, global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.ParkedRequest.Parser));
[global::System.CodeDom.Compiler.GeneratedCode("grpc_csharp_plugin", null)]
static readonly grpc::Marshaller<global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.ParkedReply> __Marshaller_scadabridge_sitecommand_v1_ParkedReply = grpc::Marshallers.Create(__Helper_SerializeMessage, context => __Helper_DeserializeMessage(context, global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.ParkedReply.Parser));
[global::System.CodeDom.Compiler.GeneratedCode("grpc_csharp_plugin", null)]
static readonly grpc::Marshaller<global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.RouteRequest> __Marshaller_scadabridge_sitecommand_v1_RouteRequest = grpc::Marshallers.Create(__Helper_SerializeMessage, context => __Helper_DeserializeMessage(context, global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.RouteRequest.Parser));
[global::System.CodeDom.Compiler.GeneratedCode("grpc_csharp_plugin", null)]
static readonly grpc::Marshaller<global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.RouteReply> __Marshaller_scadabridge_sitecommand_v1_RouteReply = grpc::Marshallers.Create(__Helper_SerializeMessage, context => __Helper_DeserializeMessage(context, global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.RouteReply.Parser));
[global::System.CodeDom.Compiler.GeneratedCode("grpc_csharp_plugin", null)]
static readonly grpc::Marshaller<global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.TriggerSiteFailoverDto> __Marshaller_scadabridge_sitecommand_v1_TriggerSiteFailoverDto = grpc::Marshallers.Create(__Helper_SerializeMessage, context => __Helper_DeserializeMessage(context, global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.TriggerSiteFailoverDto.Parser));
[global::System.CodeDom.Compiler.GeneratedCode("grpc_csharp_plugin", null)]
static readonly grpc::Marshaller<global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.SiteFailoverAckDto> __Marshaller_scadabridge_sitecommand_v1_SiteFailoverAckDto = grpc::Marshallers.Create(__Helper_SerializeMessage, context => __Helper_DeserializeMessage(context, global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.SiteFailoverAckDto.Parser));
[global::System.CodeDom.Compiler.GeneratedCode("grpc_csharp_plugin", null)]
static readonly grpc::Method<global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.LifecycleRequest, global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.LifecycleReply> __Method_ExecuteLifecycle = new grpc::Method<global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.LifecycleRequest, global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.LifecycleReply>(
grpc::MethodType.Unary,
__ServiceName,
"ExecuteLifecycle",
__Marshaller_scadabridge_sitecommand_v1_LifecycleRequest,
__Marshaller_scadabridge_sitecommand_v1_LifecycleReply);
[global::System.CodeDom.Compiler.GeneratedCode("grpc_csharp_plugin", null)]
static readonly grpc::Method<global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.OpcUaRequest, global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.OpcUaReply> __Method_ExecuteOpcUa = new grpc::Method<global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.OpcUaRequest, global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.OpcUaReply>(
grpc::MethodType.Unary,
__ServiceName,
"ExecuteOpcUa",
__Marshaller_scadabridge_sitecommand_v1_OpcUaRequest,
__Marshaller_scadabridge_sitecommand_v1_OpcUaReply);
[global::System.CodeDom.Compiler.GeneratedCode("grpc_csharp_plugin", null)]
static readonly grpc::Method<global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.QueryRequest, global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.QueryReply> __Method_ExecuteQuery = new grpc::Method<global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.QueryRequest, global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.QueryReply>(
grpc::MethodType.Unary,
__ServiceName,
"ExecuteQuery",
__Marshaller_scadabridge_sitecommand_v1_QueryRequest,
__Marshaller_scadabridge_sitecommand_v1_QueryReply);
[global::System.CodeDom.Compiler.GeneratedCode("grpc_csharp_plugin", null)]
static readonly grpc::Method<global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.ParkedRequest, global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.ParkedReply> __Method_ExecuteParked = new grpc::Method<global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.ParkedRequest, global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.ParkedReply>(
grpc::MethodType.Unary,
__ServiceName,
"ExecuteParked",
__Marshaller_scadabridge_sitecommand_v1_ParkedRequest,
__Marshaller_scadabridge_sitecommand_v1_ParkedReply);
[global::System.CodeDom.Compiler.GeneratedCode("grpc_csharp_plugin", null)]
static readonly grpc::Method<global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.RouteRequest, global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.RouteReply> __Method_ExecuteRoute = new grpc::Method<global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.RouteRequest, global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.RouteReply>(
grpc::MethodType.Unary,
__ServiceName,
"ExecuteRoute",
__Marshaller_scadabridge_sitecommand_v1_RouteRequest,
__Marshaller_scadabridge_sitecommand_v1_RouteReply);
[global::System.CodeDom.Compiler.GeneratedCode("grpc_csharp_plugin", null)]
static readonly grpc::Method<global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.TriggerSiteFailoverDto, global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.SiteFailoverAckDto> __Method_TriggerFailover = new grpc::Method<global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.TriggerSiteFailoverDto, global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.SiteFailoverAckDto>(
grpc::MethodType.Unary,
__ServiceName,
"TriggerFailover",
__Marshaller_scadabridge_sitecommand_v1_TriggerSiteFailoverDto,
__Marshaller_scadabridge_sitecommand_v1_SiteFailoverAckDto);
/// <summary>Service descriptor</summary>
public static global::Google.Protobuf.Reflection.ServiceDescriptor Descriptor
{
get { return global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.SiteCommandReflection.Descriptor.Services[0]; }
}
/// <summary>Base class for server-side implementations of SiteCommandService</summary>
[grpc::BindServiceMethod(typeof(SiteCommandService), "BindService")]
public abstract partial class SiteCommandServiceBase
{
/// <summary>
/// Deployment + instance lifecycle (6 commands).
/// </summary>
/// <param name="request">The request received from the client.</param>
/// <param name="context">The context of the server-side call handler being invoked.</param>
/// <returns>The response to send back to the client (wrapped by a task).</returns>
[global::System.CodeDom.Compiler.GeneratedCode("grpc_csharp_plugin", null)]
public virtual global::System.Threading.Tasks.Task<global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.LifecycleReply> ExecuteLifecycle(global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.LifecycleRequest request, grpc::ServerCallContext context)
{
throw new grpc::RpcException(new grpc::Status(grpc::StatusCode.Unimplemented, ""));
}
/// <summary>
/// Interactive OPC UA / MxGateway design-time commands (8 commands).
/// </summary>
/// <param name="request">The request received from the client.</param>
/// <param name="context">The context of the server-side call handler being invoked.</param>
/// <returns>The response to send back to the client (wrapped by a task).</returns>
[global::System.CodeDom.Compiler.GeneratedCode("grpc_csharp_plugin", null)]
public virtual global::System.Threading.Tasks.Task<global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.OpcUaReply> ExecuteOpcUa(global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.OpcUaRequest request, grpc::ServerCallContext context)
{
throw new grpc::RpcException(new grpc::Status(grpc::StatusCode.Unimplemented, ""));
}
/// <summary>
/// Remote read-only queries: event log + debug view (4 commands).
/// </summary>
/// <param name="request">The request received from the client.</param>
/// <param name="context">The context of the server-side call handler being invoked.</param>
/// <returns>The response to send back to the client (wrapped by a task).</returns>
[global::System.CodeDom.Compiler.GeneratedCode("grpc_csharp_plugin", null)]
public virtual global::System.Threading.Tasks.Task<global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.QueryReply> ExecuteQuery(global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.QueryRequest request, grpc::ServerCallContext context)
{
throw new grpc::RpcException(new grpc::Status(grpc::StatusCode.Unimplemented, ""));
}
/// <summary>
/// Parked store-and-forward message + cached-operation actions (5 commands).
/// </summary>
/// <param name="request">The request received from the client.</param>
/// <param name="context">The context of the server-side call handler being invoked.</param>
/// <returns>The response to send back to the client (wrapped by a task).</returns>
[global::System.CodeDom.Compiler.GeneratedCode("grpc_csharp_plugin", null)]
public virtual global::System.Threading.Tasks.Task<global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.ParkedReply> ExecuteParked(global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.ParkedRequest request, grpc::ServerCallContext context)
{
throw new grpc::RpcException(new grpc::Status(grpc::StatusCode.Unimplemented, ""));
}
/// <summary>
/// Inbound-API Route.To() relays (4 commands).
/// </summary>
/// <param name="request">The request received from the client.</param>
/// <param name="context">The context of the server-side call handler being invoked.</param>
/// <returns>The response to send back to the client (wrapped by a task).</returns>
[global::System.CodeDom.Compiler.GeneratedCode("grpc_csharp_plugin", null)]
public virtual global::System.Threading.Tasks.Task<global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.RouteReply> ExecuteRoute(global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.RouteRequest request, grpc::ServerCallContext context)
{
throw new grpc::RpcException(new grpc::Status(grpc::StatusCode.Unimplemented, ""));
}
/// <summary>
/// Operator-initiated manual site-pair failover (1 command).
/// </summary>
/// <param name="request">The request received from the client.</param>
/// <param name="context">The context of the server-side call handler being invoked.</param>
/// <returns>The response to send back to the client (wrapped by a task).</returns>
[global::System.CodeDom.Compiler.GeneratedCode("grpc_csharp_plugin", null)]
public virtual global::System.Threading.Tasks.Task<global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.SiteFailoverAckDto> TriggerFailover(global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.TriggerSiteFailoverDto request, grpc::ServerCallContext context)
{
throw new grpc::RpcException(new grpc::Status(grpc::StatusCode.Unimplemented, ""));
}
}
/// <summary>Client for SiteCommandService</summary>
public partial class SiteCommandServiceClient : grpc::ClientBase<SiteCommandServiceClient>
{
/// <summary>Creates a new client for SiteCommandService</summary>
/// <param name="channel">The channel to use to make remote calls.</param>
[global::System.CodeDom.Compiler.GeneratedCode("grpc_csharp_plugin", null)]
public SiteCommandServiceClient(grpc::ChannelBase channel) : base(channel)
{
}
/// <summary>Creates a new client for SiteCommandService that uses a custom <c>CallInvoker</c>.</summary>
/// <param name="callInvoker">The callInvoker to use to make remote calls.</param>
[global::System.CodeDom.Compiler.GeneratedCode("grpc_csharp_plugin", null)]
public SiteCommandServiceClient(grpc::CallInvoker callInvoker) : base(callInvoker)
{
}
/// <summary>Protected parameterless constructor to allow creation of test doubles.</summary>
[global::System.CodeDom.Compiler.GeneratedCode("grpc_csharp_plugin", null)]
protected SiteCommandServiceClient() : base()
{
}
/// <summary>Protected constructor to allow creation of configured clients.</summary>
/// <param name="configuration">The client configuration.</param>
[global::System.CodeDom.Compiler.GeneratedCode("grpc_csharp_plugin", null)]
protected SiteCommandServiceClient(ClientBaseConfiguration configuration) : base(configuration)
{
}
/// <summary>
/// Deployment + instance lifecycle (6 commands).
/// </summary>
/// <param name="request">The request to send to the server.</param>
/// <param name="headers">The initial metadata to send with the call. This parameter is optional.</param>
/// <param name="deadline">An optional deadline for the call. The call will be cancelled if deadline is hit.</param>
/// <param name="cancellationToken">An optional token for canceling the call.</param>
/// <returns>The response received from the server.</returns>
[global::System.CodeDom.Compiler.GeneratedCode("grpc_csharp_plugin", null)]
public virtual global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.LifecycleReply ExecuteLifecycle(global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.LifecycleRequest request, grpc::Metadata headers = null, global::System.DateTime? deadline = null, global::System.Threading.CancellationToken cancellationToken = default(global::System.Threading.CancellationToken))
{
return ExecuteLifecycle(request, new grpc::CallOptions(headers, deadline, cancellationToken));
}
/// <summary>
/// Deployment + instance lifecycle (6 commands).
/// </summary>
/// <param name="request">The request to send to the server.</param>
/// <param name="options">The options for the call.</param>
/// <returns>The response received from the server.</returns>
[global::System.CodeDom.Compiler.GeneratedCode("grpc_csharp_plugin", null)]
public virtual global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.LifecycleReply ExecuteLifecycle(global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.LifecycleRequest request, grpc::CallOptions options)
{
return CallInvoker.BlockingUnaryCall(__Method_ExecuteLifecycle, null, options, request);
}
/// <summary>
/// Deployment + instance lifecycle (6 commands).
/// </summary>
/// <param name="request">The request to send to the server.</param>
/// <param name="headers">The initial metadata to send with the call. This parameter is optional.</param>
/// <param name="deadline">An optional deadline for the call. The call will be cancelled if deadline is hit.</param>
/// <param name="cancellationToken">An optional token for canceling the call.</param>
/// <returns>The call object.</returns>
[global::System.CodeDom.Compiler.GeneratedCode("grpc_csharp_plugin", null)]
public virtual grpc::AsyncUnaryCall<global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.LifecycleReply> ExecuteLifecycleAsync(global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.LifecycleRequest request, grpc::Metadata headers = null, global::System.DateTime? deadline = null, global::System.Threading.CancellationToken cancellationToken = default(global::System.Threading.CancellationToken))
{
return ExecuteLifecycleAsync(request, new grpc::CallOptions(headers, deadline, cancellationToken));
}
/// <summary>
/// Deployment + instance lifecycle (6 commands).
/// </summary>
/// <param name="request">The request to send to the server.</param>
/// <param name="options">The options for the call.</param>
/// <returns>The call object.</returns>
[global::System.CodeDom.Compiler.GeneratedCode("grpc_csharp_plugin", null)]
public virtual grpc::AsyncUnaryCall<global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.LifecycleReply> ExecuteLifecycleAsync(global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.LifecycleRequest request, grpc::CallOptions options)
{
return CallInvoker.AsyncUnaryCall(__Method_ExecuteLifecycle, null, options, request);
}
/// <summary>
/// Interactive OPC UA / MxGateway design-time commands (8 commands).
/// </summary>
/// <param name="request">The request to send to the server.</param>
/// <param name="headers">The initial metadata to send with the call. This parameter is optional.</param>
/// <param name="deadline">An optional deadline for the call. The call will be cancelled if deadline is hit.</param>
/// <param name="cancellationToken">An optional token for canceling the call.</param>
/// <returns>The response received from the server.</returns>
[global::System.CodeDom.Compiler.GeneratedCode("grpc_csharp_plugin", null)]
public virtual global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.OpcUaReply ExecuteOpcUa(global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.OpcUaRequest request, grpc::Metadata headers = null, global::System.DateTime? deadline = null, global::System.Threading.CancellationToken cancellationToken = default(global::System.Threading.CancellationToken))
{
return ExecuteOpcUa(request, new grpc::CallOptions(headers, deadline, cancellationToken));
}
/// <summary>
/// Interactive OPC UA / MxGateway design-time commands (8 commands).
/// </summary>
/// <param name="request">The request to send to the server.</param>
/// <param name="options">The options for the call.</param>
/// <returns>The response received from the server.</returns>
[global::System.CodeDom.Compiler.GeneratedCode("grpc_csharp_plugin", null)]
public virtual global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.OpcUaReply ExecuteOpcUa(global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.OpcUaRequest request, grpc::CallOptions options)
{
return CallInvoker.BlockingUnaryCall(__Method_ExecuteOpcUa, null, options, request);
}
/// <summary>
/// Interactive OPC UA / MxGateway design-time commands (8 commands).
/// </summary>
/// <param name="request">The request to send to the server.</param>
/// <param name="headers">The initial metadata to send with the call. This parameter is optional.</param>
/// <param name="deadline">An optional deadline for the call. The call will be cancelled if deadline is hit.</param>
/// <param name="cancellationToken">An optional token for canceling the call.</param>
/// <returns>The call object.</returns>
[global::System.CodeDom.Compiler.GeneratedCode("grpc_csharp_plugin", null)]
public virtual grpc::AsyncUnaryCall<global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.OpcUaReply> ExecuteOpcUaAsync(global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.OpcUaRequest request, grpc::Metadata headers = null, global::System.DateTime? deadline = null, global::System.Threading.CancellationToken cancellationToken = default(global::System.Threading.CancellationToken))
{
return ExecuteOpcUaAsync(request, new grpc::CallOptions(headers, deadline, cancellationToken));
}
/// <summary>
/// Interactive OPC UA / MxGateway design-time commands (8 commands).
/// </summary>
/// <param name="request">The request to send to the server.</param>
/// <param name="options">The options for the call.</param>
/// <returns>The call object.</returns>
[global::System.CodeDom.Compiler.GeneratedCode("grpc_csharp_plugin", null)]
public virtual grpc::AsyncUnaryCall<global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.OpcUaReply> ExecuteOpcUaAsync(global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.OpcUaRequest request, grpc::CallOptions options)
{
return CallInvoker.AsyncUnaryCall(__Method_ExecuteOpcUa, null, options, request);
}
/// <summary>
/// Remote read-only queries: event log + debug view (4 commands).
/// </summary>
/// <param name="request">The request to send to the server.</param>
/// <param name="headers">The initial metadata to send with the call. This parameter is optional.</param>
/// <param name="deadline">An optional deadline for the call. The call will be cancelled if deadline is hit.</param>
/// <param name="cancellationToken">An optional token for canceling the call.</param>
/// <returns>The response received from the server.</returns>
[global::System.CodeDom.Compiler.GeneratedCode("grpc_csharp_plugin", null)]
public virtual global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.QueryReply ExecuteQuery(global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.QueryRequest request, grpc::Metadata headers = null, global::System.DateTime? deadline = null, global::System.Threading.CancellationToken cancellationToken = default(global::System.Threading.CancellationToken))
{
return ExecuteQuery(request, new grpc::CallOptions(headers, deadline, cancellationToken));
}
/// <summary>
/// Remote read-only queries: event log + debug view (4 commands).
/// </summary>
/// <param name="request">The request to send to the server.</param>
/// <param name="options">The options for the call.</param>
/// <returns>The response received from the server.</returns>
[global::System.CodeDom.Compiler.GeneratedCode("grpc_csharp_plugin", null)]
public virtual global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.QueryReply ExecuteQuery(global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.QueryRequest request, grpc::CallOptions options)
{
return CallInvoker.BlockingUnaryCall(__Method_ExecuteQuery, null, options, request);
}
/// <summary>
/// Remote read-only queries: event log + debug view (4 commands).
/// </summary>
/// <param name="request">The request to send to the server.</param>
/// <param name="headers">The initial metadata to send with the call. This parameter is optional.</param>
/// <param name="deadline">An optional deadline for the call. The call will be cancelled if deadline is hit.</param>
/// <param name="cancellationToken">An optional token for canceling the call.</param>
/// <returns>The call object.</returns>
[global::System.CodeDom.Compiler.GeneratedCode("grpc_csharp_plugin", null)]
public virtual grpc::AsyncUnaryCall<global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.QueryReply> ExecuteQueryAsync(global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.QueryRequest request, grpc::Metadata headers = null, global::System.DateTime? deadline = null, global::System.Threading.CancellationToken cancellationToken = default(global::System.Threading.CancellationToken))
{
return ExecuteQueryAsync(request, new grpc::CallOptions(headers, deadline, cancellationToken));
}
/// <summary>
/// Remote read-only queries: event log + debug view (4 commands).
/// </summary>
/// <param name="request">The request to send to the server.</param>
/// <param name="options">The options for the call.</param>
/// <returns>The call object.</returns>
[global::System.CodeDom.Compiler.GeneratedCode("grpc_csharp_plugin", null)]
public virtual grpc::AsyncUnaryCall<global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.QueryReply> ExecuteQueryAsync(global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.QueryRequest request, grpc::CallOptions options)
{
return CallInvoker.AsyncUnaryCall(__Method_ExecuteQuery, null, options, request);
}
/// <summary>
/// Parked store-and-forward message + cached-operation actions (5 commands).
/// </summary>
/// <param name="request">The request to send to the server.</param>
/// <param name="headers">The initial metadata to send with the call. This parameter is optional.</param>
/// <param name="deadline">An optional deadline for the call. The call will be cancelled if deadline is hit.</param>
/// <param name="cancellationToken">An optional token for canceling the call.</param>
/// <returns>The response received from the server.</returns>
[global::System.CodeDom.Compiler.GeneratedCode("grpc_csharp_plugin", null)]
public virtual global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.ParkedReply ExecuteParked(global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.ParkedRequest request, grpc::Metadata headers = null, global::System.DateTime? deadline = null, global::System.Threading.CancellationToken cancellationToken = default(global::System.Threading.CancellationToken))
{
return ExecuteParked(request, new grpc::CallOptions(headers, deadline, cancellationToken));
}
/// <summary>
/// Parked store-and-forward message + cached-operation actions (5 commands).
/// </summary>
/// <param name="request">The request to send to the server.</param>
/// <param name="options">The options for the call.</param>
/// <returns>The response received from the server.</returns>
[global::System.CodeDom.Compiler.GeneratedCode("grpc_csharp_plugin", null)]
public virtual global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.ParkedReply ExecuteParked(global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.ParkedRequest request, grpc::CallOptions options)
{
return CallInvoker.BlockingUnaryCall(__Method_ExecuteParked, null, options, request);
}
/// <summary>
/// Parked store-and-forward message + cached-operation actions (5 commands).
/// </summary>
/// <param name="request">The request to send to the server.</param>
/// <param name="headers">The initial metadata to send with the call. This parameter is optional.</param>
/// <param name="deadline">An optional deadline for the call. The call will be cancelled if deadline is hit.</param>
/// <param name="cancellationToken">An optional token for canceling the call.</param>
/// <returns>The call object.</returns>
[global::System.CodeDom.Compiler.GeneratedCode("grpc_csharp_plugin", null)]
public virtual grpc::AsyncUnaryCall<global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.ParkedReply> ExecuteParkedAsync(global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.ParkedRequest request, grpc::Metadata headers = null, global::System.DateTime? deadline = null, global::System.Threading.CancellationToken cancellationToken = default(global::System.Threading.CancellationToken))
{
return ExecuteParkedAsync(request, new grpc::CallOptions(headers, deadline, cancellationToken));
}
/// <summary>
/// Parked store-and-forward message + cached-operation actions (5 commands).
/// </summary>
/// <param name="request">The request to send to the server.</param>
/// <param name="options">The options for the call.</param>
/// <returns>The call object.</returns>
[global::System.CodeDom.Compiler.GeneratedCode("grpc_csharp_plugin", null)]
public virtual grpc::AsyncUnaryCall<global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.ParkedReply> ExecuteParkedAsync(global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.ParkedRequest request, grpc::CallOptions options)
{
return CallInvoker.AsyncUnaryCall(__Method_ExecuteParked, null, options, request);
}
/// <summary>
/// Inbound-API Route.To() relays (4 commands).
/// </summary>
/// <param name="request">The request to send to the server.</param>
/// <param name="headers">The initial metadata to send with the call. This parameter is optional.</param>
/// <param name="deadline">An optional deadline for the call. The call will be cancelled if deadline is hit.</param>
/// <param name="cancellationToken">An optional token for canceling the call.</param>
/// <returns>The response received from the server.</returns>
[global::System.CodeDom.Compiler.GeneratedCode("grpc_csharp_plugin", null)]
public virtual global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.RouteReply ExecuteRoute(global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.RouteRequest request, grpc::Metadata headers = null, global::System.DateTime? deadline = null, global::System.Threading.CancellationToken cancellationToken = default(global::System.Threading.CancellationToken))
{
return ExecuteRoute(request, new grpc::CallOptions(headers, deadline, cancellationToken));
}
/// <summary>
/// Inbound-API Route.To() relays (4 commands).
/// </summary>
/// <param name="request">The request to send to the server.</param>
/// <param name="options">The options for the call.</param>
/// <returns>The response received from the server.</returns>
[global::System.CodeDom.Compiler.GeneratedCode("grpc_csharp_plugin", null)]
public virtual global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.RouteReply ExecuteRoute(global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.RouteRequest request, grpc::CallOptions options)
{
return CallInvoker.BlockingUnaryCall(__Method_ExecuteRoute, null, options, request);
}
/// <summary>
/// Inbound-API Route.To() relays (4 commands).
/// </summary>
/// <param name="request">The request to send to the server.</param>
/// <param name="headers">The initial metadata to send with the call. This parameter is optional.</param>
/// <param name="deadline">An optional deadline for the call. The call will be cancelled if deadline is hit.</param>
/// <param name="cancellationToken">An optional token for canceling the call.</param>
/// <returns>The call object.</returns>
[global::System.CodeDom.Compiler.GeneratedCode("grpc_csharp_plugin", null)]
public virtual grpc::AsyncUnaryCall<global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.RouteReply> ExecuteRouteAsync(global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.RouteRequest request, grpc::Metadata headers = null, global::System.DateTime? deadline = null, global::System.Threading.CancellationToken cancellationToken = default(global::System.Threading.CancellationToken))
{
return ExecuteRouteAsync(request, new grpc::CallOptions(headers, deadline, cancellationToken));
}
/// <summary>
/// Inbound-API Route.To() relays (4 commands).
/// </summary>
/// <param name="request">The request to send to the server.</param>
/// <param name="options">The options for the call.</param>
/// <returns>The call object.</returns>
[global::System.CodeDom.Compiler.GeneratedCode("grpc_csharp_plugin", null)]
public virtual grpc::AsyncUnaryCall<global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.RouteReply> ExecuteRouteAsync(global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.RouteRequest request, grpc::CallOptions options)
{
return CallInvoker.AsyncUnaryCall(__Method_ExecuteRoute, null, options, request);
}
/// <summary>
/// Operator-initiated manual site-pair failover (1 command).
/// </summary>
/// <param name="request">The request to send to the server.</param>
/// <param name="headers">The initial metadata to send with the call. This parameter is optional.</param>
/// <param name="deadline">An optional deadline for the call. The call will be cancelled if deadline is hit.</param>
/// <param name="cancellationToken">An optional token for canceling the call.</param>
/// <returns>The response received from the server.</returns>
[global::System.CodeDom.Compiler.GeneratedCode("grpc_csharp_plugin", null)]
public virtual global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.SiteFailoverAckDto TriggerFailover(global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.TriggerSiteFailoverDto request, grpc::Metadata headers = null, global::System.DateTime? deadline = null, global::System.Threading.CancellationToken cancellationToken = default(global::System.Threading.CancellationToken))
{
return TriggerFailover(request, new grpc::CallOptions(headers, deadline, cancellationToken));
}
/// <summary>
/// Operator-initiated manual site-pair failover (1 command).
/// </summary>
/// <param name="request">The request to send to the server.</param>
/// <param name="options">The options for the call.</param>
/// <returns>The response received from the server.</returns>
[global::System.CodeDom.Compiler.GeneratedCode("grpc_csharp_plugin", null)]
public virtual global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.SiteFailoverAckDto TriggerFailover(global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.TriggerSiteFailoverDto request, grpc::CallOptions options)
{
return CallInvoker.BlockingUnaryCall(__Method_TriggerFailover, null, options, request);
}
/// <summary>
/// Operator-initiated manual site-pair failover (1 command).
/// </summary>
/// <param name="request">The request to send to the server.</param>
/// <param name="headers">The initial metadata to send with the call. This parameter is optional.</param>
/// <param name="deadline">An optional deadline for the call. The call will be cancelled if deadline is hit.</param>
/// <param name="cancellationToken">An optional token for canceling the call.</param>
/// <returns>The call object.</returns>
[global::System.CodeDom.Compiler.GeneratedCode("grpc_csharp_plugin", null)]
public virtual grpc::AsyncUnaryCall<global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.SiteFailoverAckDto> TriggerFailoverAsync(global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.TriggerSiteFailoverDto request, grpc::Metadata headers = null, global::System.DateTime? deadline = null, global::System.Threading.CancellationToken cancellationToken = default(global::System.Threading.CancellationToken))
{
return TriggerFailoverAsync(request, new grpc::CallOptions(headers, deadline, cancellationToken));
}
/// <summary>
/// Operator-initiated manual site-pair failover (1 command).
/// </summary>
/// <param name="request">The request to send to the server.</param>
/// <param name="options">The options for the call.</param>
/// <returns>The call object.</returns>
[global::System.CodeDom.Compiler.GeneratedCode("grpc_csharp_plugin", null)]
public virtual grpc::AsyncUnaryCall<global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.SiteFailoverAckDto> TriggerFailoverAsync(global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.TriggerSiteFailoverDto request, grpc::CallOptions options)
{
return CallInvoker.AsyncUnaryCall(__Method_TriggerFailover, null, options, request);
}
/// <summary>Creates a new instance of client from given <c>ClientBaseConfiguration</c>.</summary>
[global::System.CodeDom.Compiler.GeneratedCode("grpc_csharp_plugin", null)]
protected override SiteCommandServiceClient NewInstance(ClientBaseConfiguration configuration)
{
return new SiteCommandServiceClient(configuration);
}
}
/// <summary>Creates service definition that can be registered with a server</summary>
/// <param name="serviceImpl">An object implementing the server-side handling logic.</param>
[global::System.CodeDom.Compiler.GeneratedCode("grpc_csharp_plugin", null)]
public static grpc::ServerServiceDefinition BindService(SiteCommandServiceBase serviceImpl)
{
return grpc::ServerServiceDefinition.CreateBuilder()
.AddMethod(__Method_ExecuteLifecycle, serviceImpl.ExecuteLifecycle)
.AddMethod(__Method_ExecuteOpcUa, serviceImpl.ExecuteOpcUa)
.AddMethod(__Method_ExecuteQuery, serviceImpl.ExecuteQuery)
.AddMethod(__Method_ExecuteParked, serviceImpl.ExecuteParked)
.AddMethod(__Method_ExecuteRoute, serviceImpl.ExecuteRoute)
.AddMethod(__Method_TriggerFailover, serviceImpl.TriggerFailover).Build();
}
/// <summary>Register service method with a service binder with or without implementation. Useful when customizing the service binding logic.
/// Note: this method is part of an experimental API that can change or be removed without any prior notice.</summary>
/// <param name="serviceBinder">Service methods will be bound by calling <c>AddMethod</c> on this object.</param>
/// <param name="serviceImpl">An object implementing the server-side handling logic.</param>
[global::System.CodeDom.Compiler.GeneratedCode("grpc_csharp_plugin", null)]
public static void BindService(grpc::ServiceBinderBase serviceBinder, SiteCommandServiceBase serviceImpl)
{
serviceBinder.AddMethod(__Method_ExecuteLifecycle, serviceImpl == null ? null : new grpc::UnaryServerMethod<global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.LifecycleRequest, global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.LifecycleReply>(serviceImpl.ExecuteLifecycle));
serviceBinder.AddMethod(__Method_ExecuteOpcUa, serviceImpl == null ? null : new grpc::UnaryServerMethod<global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.OpcUaRequest, global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.OpcUaReply>(serviceImpl.ExecuteOpcUa));
serviceBinder.AddMethod(__Method_ExecuteQuery, serviceImpl == null ? null : new grpc::UnaryServerMethod<global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.QueryRequest, global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.QueryReply>(serviceImpl.ExecuteQuery));
serviceBinder.AddMethod(__Method_ExecuteParked, serviceImpl == null ? null : new grpc::UnaryServerMethod<global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.ParkedRequest, global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.ParkedReply>(serviceImpl.ExecuteParked));
serviceBinder.AddMethod(__Method_ExecuteRoute, serviceImpl == null ? null : new grpc::UnaryServerMethod<global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.RouteRequest, global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.RouteReply>(serviceImpl.ExecuteRoute));
serviceBinder.AddMethod(__Method_TriggerFailover, serviceImpl == null ? null : new grpc::UnaryServerMethod<global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.TriggerSiteFailoverDto, global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.SiteFailoverAckDto>(serviceImpl.TriggerFailover));
}
}
}
#endregion
@@ -10,6 +10,9 @@
<ItemGroup>
<InternalsVisibleTo Include="ZB.MOM.WW.ScadaBridge.Communication.Tests" />
<InternalsVisibleTo Include="ZB.MOM.WW.ScadaBridge.IntegrationTests" />
<!-- GrpcSiteTransport failover/failback TestServer coverage (T1B.3) lives in Host.Tests,
which owns the ASP.NET TestHost stack; it drives the provider's internal failback seams. -->
<InternalsVisibleTo Include="ZB.MOM.WW.ScadaBridge.Host.Tests" />
</ItemGroup>
<ItemGroup>
@@ -32,20 +35,31 @@
<ProjectReference Include="../ZB.MOM.WW.ScadaBridge.HealthMonitoring/ZB.MOM.WW.ScadaBridge.HealthMonitoring.csproj" />
</ItemGroup>
<!-- gRPC proto generation. The compiled C# is checked in under
SiteStreamGrpc/ (Sitestream.cs + SitestreamGrpc.cs) because protoc
segfaults inside our linux_arm64 Docker build image. To regenerate
after schema changes:
1. Temporarily uncomment the Protobuf ItemGroup below.
2. Delete SiteStreamGrpc/*.cs.
<!-- gRPC proto generation. The compiled C# is checked in under SiteStreamGrpc/
(Sitestream.cs + SitestreamGrpc.cs) for Protos/sitestream.proto,
CentralControlGrpc/ (CentralControl.cs + CentralControlGrpc.cs) for
Protos/central_control.proto, and SiteCommandGrpc/ (SiteCommand.cs +
SiteCommandGrpc.cs) for Protos/site_command.proto — because protoc segfaults
inside our linux_arm64 Docker build image. To regenerate after schema changes
run `docker/regen-proto.sh [sitestream|centralcontrol|sitecommand|all]`, which
does all of the following and always leaves this file as it found it:
1. Temporarily uncomment the Protobuf ItemGroup below (just the line for
the proto you changed — the other files' checked-in C# is already
compiled, so enabling several at once duplicates types).
2. Delete the matching checked-in *.cs.
3. `dotnet build` (on macOS) — Grpc.Tools writes fresh files to obj/.
4. Copy obj/Debug/net10.0/Protos/*.cs into SiteStreamGrpc/.
4. Copy obj/Debug/net10.0/Protos/*.cs into the matching folder.
5. Re-comment the ItemGroup.
Eventually we should switch the Docker build image to one with a
working protoc on arm64. -->
central_control.proto imports sitestream.proto, so protoc resolves it from the
project-relative path without sitestream.proto needing its own Protobuf item.
An ACTIVE Protobuf item must never be committed — it breaks the Docker image
build. Eventually we should switch the Docker build image to one with a working
protoc on arm64. -->
<!--
<ItemGroup>
<Protobuf Include="Protos\sitestream.proto" GrpcServices="Both" />
<Protobuf Include="Protos\central_control.proto" GrpcServices="Both" />
<Protobuf Include="Protos\site_command.proto" GrpcServices="Both" />
</ItemGroup>
-->
@@ -8,6 +8,7 @@ using Microsoft.Extensions.Options;
using ZB.MOM.WW.ScadaBridge.ClusterInfrastructure;
using ZB.MOM.WW.ScadaBridge.Communication;
using ZB.MOM.WW.ScadaBridge.Communication.Actors;
using ZB.MOM.WW.ScadaBridge.Communication.Grpc;
using ZB.MOM.WW.ScadaBridge.Host.Actors;
using ZB.MOM.WW.ScadaBridge.SiteRuntime;
using ZB.MOM.WW.ScadaBridge.SiteRuntime.Actors;
@@ -436,6 +437,19 @@ akka {{
ClusterClientReceptionist.Get(_actorSystem).RegisterService(centralCommActor);
_logger.LogInformation("CentralCommunicationActor registered with ClusterClientReceptionist");
// Hand the same actor to the central-hosted gRPC control plane (T1A.2) and open its
// readiness gate — the gRPC face Asks this exact actor, so both transports resolve to
// one handler implementation. Mirrors SiteStreamGrpcServer.SetReady on the site side:
// the service is a DI singleton created before the actor system exists, so the actor
// arrives here post-construction. Null on a host that did not register the service
// (e.g. an in-process test harness), so the wiring is a guarded no-op there.
var centralControlGrpc = _serviceProvider
.GetService<ZB.MOM.WW.ScadaBridge.Communication.Grpc.CentralControlGrpcService>();
centralControlGrpc?.SetReady(centralCommActor);
_logger.LogInformation(
"CentralControlGrpcService readiness set (service bound: {Bound})",
centralControlGrpc is not null);
// Wire up the CommunicationService with the actor reference
var commService = _serviceProvider.GetService<CommunicationService>();
commService?.SetCommunicationActor(centralCommActor);
@@ -819,13 +833,59 @@ akka {{
_logger, role: siteRole);
var dmProxy = dm.Proxy;
// Create SiteCommunicationActor for receiving messages from central
// Select the site→central transport behind the coexistence flag (default Akka
// ClusterClient). When gRPC is chosen the site dials CentralControlService directly with
// a sticky-failover channel pair, presenting its own preshared key; the ClusterClient
// below is then not created at all.
ICentralTransport? centralTransport = null;
if (_communicationOptions.CentralTransport == CentralTransportMode.Grpc)
{
var loggerFactory = _serviceProvider.GetRequiredService<ILoggerFactory>();
var channelProvider = new CentralChannelProvider(
_communicationOptions.CentralGrpcEndpoints,
new StaticSitePskProvider(_communicationOptions.GrpcPsk),
_nodeOptions.SiteId!,
_communicationOptions,
loggerFactory.CreateLogger<CentralChannelProvider>());
_trackedDisposables.Add(channelProvider);
centralTransport = new GrpcCentralTransport(
channelProvider,
_communicationOptions,
loggerFactory.CreateLogger<GrpcCentralTransport>());
_logger.LogInformation(
"Site→central transport: gRPC to {Count} central endpoint(s) for site {SiteId}",
_communicationOptions.CentralGrpcEndpoints.Count, _nodeOptions.SiteId);
}
else
{
_logger.LogInformation(
"Site→central transport: Akka ClusterClient (default) for site {SiteId}",
_nodeOptions.SiteId);
}
// The ONE routing table for central→site commands, shared by the Akka
// SiteCommunicationActor (below) and the gRPC SiteCommandGrpcService (SetReady at the end
// of this method). Its failover seam resolves (dryRun) or performs the graceful Leave via
// the shared ClusterFailoverCoordinator so the central/site paths cannot drift.
var siteCommandDispatcher = new SiteCommandDispatcher(
_nodeOptions.SiteId!,
dmProxy,
(role, dryRun) => ZB.MOM.WW.ScadaBridge.Communication.ClusterState.ClusterFailoverCoordinator
.FailOverOldest(_actorSystem!, role, dryRun)?.ToString());
// Create SiteCommunicationActor for receiving messages from central. It routes commands
// through the shared dispatcher; RegisterLocalHandler (below) registers the handlers INTO
// that dispatcher, so the gRPC command service sees the same registrations. The site→central
// transport is selected above (default Akka); both are passed in.
var siteCommActor = _actorSystem.ActorOf(
Props.Create(() => new SiteCommunicationActor(
_nodeOptions.SiteId!,
_communicationOptions,
dmProxy,
activeNodeCheck)),
activeNodeCheck,
null,
centralTransport,
siteCommandDispatcher)),
"site-communication");
// Register local handlers with SiteCommunicationActor
@@ -944,8 +1004,12 @@ akka {{
"Site actors registered. DeploymentManager singleton scoped to role={SiteRole}, SiteCommunicationActor created.",
siteRole);
// Create ClusterClient to central if contact points are configured
if (_communicationOptions.CentralContactPoints.Count > 0)
// Create ClusterClient to central if contact points are configured — but only on the Akka
// transport. On the gRPC transport the SiteCommunicationActor already holds a
// GrpcCentralTransport and never receives RegisterCentralClient, so a ClusterClient here
// would be dead weight (and keep an unwanted cross-cluster Akka association alive).
if (_communicationOptions.CentralTransport == CentralTransportMode.Akka
&& _communicationOptions.CentralContactPoints.Count > 0)
{
var contacts = _communicationOptions.CentralContactPoints
.Select(cp => ActorPath.Parse($"{cp}/system/receptionist"))
@@ -1091,5 +1155,12 @@ akka {{
grpcServer?.SetOperationTrackingStore(siteTrackingStore);
}
grpcServer?.SetReady(_actorSystem!);
// Site command plane: hand the shared dispatcher to the gRPC command service and flip it
// ready at the same point as the streaming server — the actor graph exists and the local
// handlers have been registered into the dispatcher, so it can route commands now.
var commandService = _serviceProvider
.GetService<ZB.MOM.WW.ScadaBridge.Communication.Grpc.SiteCommandGrpcService>();
commandService?.SetReady(siteCommandDispatcher);
}
}
@@ -0,0 +1,245 @@
using System.Security.Cryptography;
using System.Text;
using Grpc.Core;
using Grpc.Core.Interceptors;
using Microsoft.Extensions.Logging;
using ZB.MOM.WW.ScadaBridge.Communication.Grpc;
namespace ZB.MOM.WW.ScadaBridge.Host;
/// <summary>
/// Gates the central-hosted gRPC control plane (<c>CentralControlService</c>) with each site's
/// preshared key. The sibling of <see cref="ControlPlaneAuthInterceptor"/>, but with the
/// verification model inverted: a site checks one bearer token against its own single key, whereas
/// central must check the presented token against the key belonging to the SITE that sent it.
/// </summary>
/// <remarks>
/// <para>
/// <b>Why a separate class rather than a second constructor on
/// <see cref="ControlPlaneAuthInterceptor"/>.</b> <c>Grpc.AspNetCore</c> registers a
/// type-registered interceptor through <c>InterceptorRegistration.GetFactory()</c>, which throws
/// <c>"Multiple constructors accepting all given argument types have been found"</c> the moment a
/// second public constructor is applicable. That throw lands inside the pipeline on every call and
/// surfaces as <c>Unknown / "Exception was thrown by handler"</c> — the node boots healthy and
/// every gated call dies looking like a handler bug, with correct/wrong/no key all producing the
/// identical error. It shipped once with a fully green suite and was caught only on the rig.
/// Central's model genuinely differs (per-site key by header, not the site's one own-key), so it
/// gets its own class with its own single public constructor rather than a variant ctor on the
/// site interceptor. Both classes are pinned by a reflection test asserting exactly one public
/// constructor.
/// </para>
/// <para>
/// <b>Fail-closed on every branch.</b> A gated call is refused with
/// <see cref="StatusCode.PermissionDenied"/> when: the <c>x-scadabridge-site</c> header is
/// missing or blank; no key can be resolved for that site (<see cref="ISitePskProvider"/> throws);
/// or the presented bearer token does not match. There is no pass-through — an unresolvable or
/// absent identity never degrades to "let it in". Non-gated services (should any share the
/// listener) return immediately, matching the site interceptor's shape.
/// </para>
/// <para>
/// The comparison is <see cref="CryptographicOperations.FixedTimeEquals"/> over UTF-8, the same
/// constant-time compare the site interceptor and LocalDb sync use.
/// </para>
/// </remarks>
public sealed class CentralControlAuthInterceptor : Interceptor
{
/// <summary>
/// Service prefixes gated by default — the one central-hosted control-plane service. Taken
/// from the generated <c>package scadabridge.centralcontrol.v1; service CentralControlService</c>.
/// </summary>
public static readonly IReadOnlyList<string> DefaultGatedPrefixes =
new[] { $"/{CentralControlService.Descriptor.FullName}/" };
private readonly IReadOnlyList<string> _gatedPrefixes;
private readonly ISitePskProvider _pskProvider;
private readonly ILogger<CentralControlAuthInterceptor> _logger;
/// <summary>
/// Creates the interceptor gating <see cref="DefaultGatedPrefixes"/>.
/// </summary>
/// <remarks>
/// <b>This must remain the ONLY public constructor</b> — see the class remarks for why a
/// second one silently disables the gate. Pinned by
/// <c>CentralControlAuthInterceptorTests.TheInterceptorHasExactlyOnePublicConstructor</c>.
/// </remarks>
/// <param name="pskProvider">Resolves each site's preshared key.</param>
/// <param name="logger">Logger for denial diagnostics.</param>
public CentralControlAuthInterceptor(
ISitePskProvider pskProvider,
ILogger<CentralControlAuthInterceptor> logger)
: this(pskProvider, logger, DefaultGatedPrefixes)
{
}
/// <summary>
/// Creates the interceptor gating an explicit prefix set. <b>Internal</b> — a public second
/// constructor would reintroduce the ambiguous-constructor defect described on the class.
/// </summary>
/// <param name="pskProvider">Resolves each site's preshared key.</param>
/// <param name="logger">Logger for denial diagnostics.</param>
/// <param name="gatedPrefixes">Method-path prefixes to gate.</param>
internal CentralControlAuthInterceptor(
ISitePskProvider pskProvider,
ILogger<CentralControlAuthInterceptor> logger,
IReadOnlyList<string> gatedPrefixes)
{
ArgumentNullException.ThrowIfNull(pskProvider);
ArgumentNullException.ThrowIfNull(logger);
ArgumentNullException.ThrowIfNull(gatedPrefixes);
_pskProvider = pskProvider;
_logger = logger;
_gatedPrefixes = gatedPrefixes;
}
/// <inheritdoc />
public override async Task<TResponse> UnaryServerHandler<TRequest, TResponse>(
TRequest request,
ServerCallContext context,
UnaryServerMethod<TRequest, TResponse> continuation)
{
await AuthorizeAsync(context).ConfigureAwait(false);
return await continuation(request, context).ConfigureAwait(false);
}
/// <inheritdoc />
public override async Task DuplexStreamingServerHandler<TRequest, TResponse>(
IAsyncStreamReader<TRequest> requestStream,
IServerStreamWriter<TResponse> responseStream,
ServerCallContext context,
DuplexStreamingServerMethod<TRequest, TResponse> continuation)
{
await AuthorizeAsync(context).ConfigureAwait(false);
await continuation(requestStream, responseStream, context).ConfigureAwait(false);
}
/// <inheritdoc />
public override async Task<TResponse> ClientStreamingServerHandler<TRequest, TResponse>(
IAsyncStreamReader<TRequest> requestStream,
ServerCallContext context,
ClientStreamingServerMethod<TRequest, TResponse> continuation)
{
await AuthorizeAsync(context).ConfigureAwait(false);
return await continuation(requestStream, context).ConfigureAwait(false);
}
/// <inheritdoc />
public override async Task ServerStreamingServerHandler<TRequest, TResponse>(
TRequest request,
IServerStreamWriter<TResponse> responseStream,
ServerCallContext context,
ServerStreamingServerMethod<TRequest, TResponse> continuation)
{
await AuthorizeAsync(context).ConfigureAwait(false);
await continuation(request, responseStream, context).ConfigureAwait(false);
}
/// <summary>
/// Throws <see cref="RpcException"/> with <see cref="StatusCode.PermissionDenied"/> unless a
/// gated call carries a valid <c>x-scadabridge-site</c> header AND a bearer token matching
/// that site's resolved key. Non-gated calls return immediately.
/// </summary>
private async Task AuthorizeAsync(ServerCallContext context)
{
if (!IsGated(context.Method))
{
return;
}
var siteId = ExtractSiteId(context.RequestHeaders);
if (string.IsNullOrWhiteSpace(siteId))
{
_logger.LogWarning(
"Rejected a central control-plane call to {Method}: the required "
+ "'{Header}' metadata header is missing or blank, so there is no per-site key to "
+ "verify against.",
context.Method, ControlPlaneCredentials.SiteHeader);
throw Denied("missing site identity header");
}
string expected;
try
{
expected = await _pskProvider.GetAsync(siteId, context.CancellationToken)
.ConfigureAwait(false);
}
catch (OperationCanceledException)
{
throw;
}
catch (Exception ex)
{
// Fail-closed: an unresolvable key is a denial, never a pass-through. The provider
// has already logged the specific cause (missing secret / missing config entry).
_logger.LogWarning(ex,
"Rejected a central control-plane call to {Method}: no preshared key could be "
+ "resolved for site {SiteId}.",
context.Method, siteId);
throw Denied("no key configured for the presented site");
}
var presented = ExtractBearerToken(context.RequestHeaders);
if (presented is null || !FixedTimeEquals(presented, expected))
{
_logger.LogWarning(
"Rejected a central control-plane call to {Method} from site {SiteId}: {Reason}.",
context.Method, siteId,
presented is null ? "no bearer token presented" : "bearer token did not match");
throw Denied("control plane authentication failed");
}
}
private static RpcException Denied(string reason)
=> new(new Status(StatusCode.PermissionDenied, $"Control plane authentication failed: {reason}."));
private bool IsGated(string method)
{
foreach (var prefix in _gatedPrefixes)
{
if (method.StartsWith(prefix, StringComparison.Ordinal))
{
return true;
}
}
return false;
}
private static string? ExtractSiteId(Metadata headers)
{
foreach (var entry in headers)
{
if (string.Equals(entry.Key, ControlPlaneCredentials.SiteHeader,
StringComparison.OrdinalIgnoreCase))
{
return entry.Value;
}
}
return null;
}
private static string? ExtractBearerToken(Metadata headers)
{
// gRPC lowercases header keys on the wire; compare case-insensitively so a hand-built
// Metadata in a test behaves the same as a real request.
foreach (var entry in headers)
{
if (!string.Equals(entry.Key, ControlPlaneCredentials.AuthorizationHeader,
StringComparison.OrdinalIgnoreCase))
{
continue;
}
var value = entry.Value;
if (value is not null && value.StartsWith("Bearer ", StringComparison.OrdinalIgnoreCase))
{
return value["Bearer ".Length..];
}
}
return null;
}
private static bool FixedTimeEquals(string presented, string expected)
=> CryptographicOperations.FixedTimeEquals(
Encoding.UTF8.GetBytes(presented), Encoding.UTF8.GetBytes(expected));
}
@@ -51,12 +51,18 @@ namespace ZB.MOM.WW.ScadaBridge.Host;
public sealed class ControlPlaneAuthInterceptor : Interceptor
{
/// <summary>
/// Service prefixes gated by default. Read from the generated <c>sitestream.proto</c>
/// package/service names — <c>package sitestream; service SiteStreamService</c>.
/// Later phases append their own services here.
/// Service prefixes gated by default. Read from the generated service descriptors —
/// <c>package sitestream; service SiteStreamService</c> (real-time data + audit pull) and
/// <c>package scadabridge.sitecommand.v1; service SiteCommandService</c> (the T1B command
/// plane). Later phases append their own services here rather than adding a second
/// interceptor or constructor (see the public constructor's remarks).
/// </summary>
public static readonly IReadOnlyList<string> DefaultGatedPrefixes =
new[] { "/sitestream.SiteStreamService/" };
new[]
{
$"/{SiteStreamService.Descriptor.FullName}/",
$"/{SiteCommandService.Descriptor.FullName}/",
};
private readonly IReadOnlyList<string> _gatedPrefixes;
private readonly IOptions<CommunicationOptions> _options;
@@ -21,6 +21,15 @@ public class NodeOptions
/// <summary>Gets or sets the gRPC port for the site stream server.</summary>
public int GrpcPort { get; set; } = 8083;
/// <summary>
/// HTTP/2 (h2c) port the CENTRAL node listens on for the site→central
/// <c>CentralControlService</c> gRPC control plane. Default 8083 — deliberately symmetric
/// with the site <see cref="GrpcPort"/>, since a node is either central or a site and the
/// two never share a process. This listener is distinct from central's <c>:5000</c> HTTP/1
/// surface (Central UI, Management/Inbound API), which stays exactly as-is: gRPC does NOT go
/// through Traefik (HTTP/1 only). Ignored on site nodes.
/// </summary>
public int CentralGrpcPort { get; set; } = 8083;
/// <summary>
/// HTTP/1.1 port serving the Prometheus /metrics scrape endpoint on site nodes.
/// Defaults to 8084 — deliberately distinct from <see cref="RemotingPort"/> (8082)
/// and <see cref="GrpcPort"/> (8083) so the Kestrel metrics listener never contends
@@ -23,6 +23,7 @@ public sealed class NodeOptionsValidator : OptionsValidatorBase<NodeOptions>
RequirePort(builder, options.RemotingPort, nameof(NodeOptions.RemotingPort));
RequirePort(builder, options.GrpcPort, nameof(NodeOptions.GrpcPort));
RequirePort(builder, options.MetricsPort, nameof(NodeOptions.MetricsPort));
RequirePort(builder, options.CentralGrpcPort, nameof(NodeOptions.CentralGrpcPort));
}
// 0 stays valid (dynamic-port request); reject only out-of-TCP-range values.
+135 -1
View File
@@ -97,6 +97,49 @@ try
// Windows Service support (no-op when not running as a Windows Service)
builder.Host.UseWindowsService();
// Explicit Kestrel h2c listener for the central-hosted gRPC control plane
// (CentralControlService): HTTP/2-only, on its own port (default 8083, symmetric
// with the site GrpcPort — a node is either central or a site, never both). gRPC
// does NOT go through Traefik (HTTP/1 only); sites reach this port by container name.
//
// WARNING — the trap the rig caught (T1A.2 shipped it, fixed here): calling
// options.Listen*/ListenAnyIP puts Kestrel into EXPLICIT-ENDPOINTS mode, which
// SUPPRESSES the URLs from ASPNETCORE_URLS/--urls entirely — it is NOT additive.
// Central's ENTIRE HTTP/1 surface (Central UI, Management + Inbound API, and the
// /health/* endpoints Traefik + IActiveNodeGate depend on) lives on that URL
// (http://+:5000 on the rig, a different port in production). If we bind only the
// gRPC port, :5000 vanishes and central is reachable over gRPC while the UI, the
// management API and health checks are all dead — with no startup error. Unit tests
// use TestServer and never bind real Kestrel, so only a live node exposes this.
// The site branch has the same ConfigureKestrel shape but no ASPNETCORE_URLS surface
// to lose (it binds every port it needs — gRPC + metrics — explicitly). Central must
// therefore RE-BIND its HTTP port(s) here alongside the gRPC port.
var centralGrpcPort = configuration.GetValue<int>("ScadaBridge:Node:CentralGrpcPort", 8083);
// "urls" is WebHostDefaults.ServerUrlsKey — the host setting ASPNETCORE_URLS/--urls
// populate. Read it as a literal so this needs no extra Hosting using.
var httpUrls = configuration["ASPNETCORE_URLS"]
?? builder.WebHost.GetSetting("urls")
?? "http://+:5000";
var httpPorts = ParseHttpBindPorts(httpUrls);
builder.WebHost.ConfigureKestrel(options =>
{
// The HTTP/1.1 (+ HTTP/2) surface from the configured URLs, re-declared so it
// survives the explicit-endpoints switch above.
foreach (var httpPort in httpPorts)
{
options.ListenAnyIP(httpPort, listenOptions =>
{
listenOptions.Protocols = Microsoft.AspNetCore.Server.Kestrel.Core.HttpProtocols.Http1AndHttp2;
});
}
// The gRPC control plane, HTTP/2 h2c only, on its own port.
options.ListenAnyIP(centralGrpcPort, listenOptions =>
{
listenOptions.Protocols = Microsoft.AspNetCore.Server.Kestrel.Core.HttpProtocols.Http2;
});
});
// Shared components
builder.Services.AddClusterInfrastructure();
builder.Services.AddCommunication();
@@ -108,6 +151,30 @@ try
// node uses for its single key). Registered before the clients that consume it.
builder.Services.AddSingleton<
ZB.MOM.WW.ScadaBridge.Communication.Grpc.ISitePskProvider, SitePskProvider>();
// Central-hosted gRPC control plane (T1A.2). CentralControlAuthInterceptor is the
// per-site sibling of the site's ControlPlaneAuthInterceptor: it verifies the Bearer
// token against the key for the site named in the required x-scadabridge-site header,
// resolved through the ISitePskProvider registered just above. Registered BY TYPE on
// AddGrpc exactly as the site branch registers its interceptors — NOT as a DI singleton,
// which would let DI hand the instance back and bypass Grpc.AspNetCore's own activation
// path (the shape that once hid a two-public-constructor defect until the rig caught it).
// CentralControlGrpcService decodes each request onto the same in-process message the
// ClusterClient path carries and Asks CentralCommunicationActor — zero handler logic here.
builder.Services.AddGrpc(options =>
{
options.Interceptors.Add<CentralControlAuthInterceptor>();
});
builder.Services.AddSingleton<
ZB.MOM.WW.ScadaBridge.Communication.Grpc.CentralControlGrpcService>();
// Shared per-site gRPC channel-pair provider (sticky failover/failback) backing the
// GrpcSiteTransport when ScadaBridge:Communication:SiteTransport=Grpc. Central-only, and a
// no-op until CentralCommunicationActor builds the gRPC transport (its failback loop starts
// on first use), so registering it unconditionally is harmless under the default Akka path.
builder.Services.AddSingleton<
ZB.MOM.WW.ScadaBridge.Communication.Grpc.SitePairChannelProvider>();
builder.Services.AddHealthMonitoring();
builder.Services.AddCentralHealthAggregation();
builder.Services.AddExternalSystemGateway();
@@ -462,6 +529,14 @@ try
// Requires endpoint routing (app.UseRouting() above).
app.MapZbMetrics();
// Central-hosted gRPC control plane (T1A.2) — the site→central CentralControlService.
// Runs on the dedicated h2c listener configured above (default :8083), gated by
// CentralControlAuthInterceptor and readiness-gated by the service itself (Unavailable
// until AkkaHostedService hands the CentralCommunicationActor over via SetReady). It
// shares the app's endpoint routing with the HTTP/1 surface; Kestrel steers each
// connection to the right pipeline by listener/protocol.
app.MapGrpcService<ZB.MOM.WW.ScadaBridge.Communication.Grpc.CentralControlGrpcService>();
app.MapStaticAssets();
app.MapCentralUI<ZB.MOM.WW.ScadaBridge.Host.Components.App>();
app.MapInboundAPI();
@@ -543,6 +618,10 @@ try
options.Interceptors.Add<ControlPlaneAuthInterceptor>();
});
builder.Services.AddSingleton<ZB.MOM.WW.ScadaBridge.Communication.Grpc.SiteStreamGrpcServer>();
// Site command plane (central→site) — the gRPC peer of SiteStreamGrpcServer. Both share
// the h2c listener and the ControlPlaneAuthInterceptor PSK gate; both are readiness-gated
// (SetReady, below). Central still dials over ClusterClient until T1B.3.
builder.Services.AddSingleton<ZB.MOM.WW.ScadaBridge.Communication.Grpc.SiteCommandGrpcService>();
// Existing site service registrations (this is also where LocalDb and its
// replication engine are registered — see SiteServiceRegistration)
@@ -564,6 +643,9 @@ try
// Map gRPC service — resolves the singleton SiteStreamGrpcServer from DI
app.MapGrpcService<ZB.MOM.WW.ScadaBridge.Communication.Grpc.SiteStreamGrpcServer>();
// Map the site command plane (central→site) alongside it, on the same gated listener.
app.MapGrpcService<ZB.MOM.WW.ScadaBridge.Communication.Grpc.SiteCommandGrpcService>();
// The passive half of LocalDb replication: the peer node dials THIS endpoint.
// It shares the Kestrel h2c listener the site gRPC server already uses, so no
// listener or port changes are needed. Mapping it is harmless with no peer
@@ -603,4 +685,56 @@ finally
/// <summary>
/// Exposes the auto-generated Program class for test infrastructure (e.g. WebApplicationFactory).
/// </summary>
public partial class Program { }
public partial class Program
{
/// <summary>
/// Extracts the distinct TCP ports from an ASP.NET Core server-URLs string (the
/// <c>ASPNETCORE_URLS</c> / <c>--urls</c> value, e.g. <c>"http://+:5000"</c> or a
/// semicolon-separated list). Used to re-declare central's HTTP surface after the gRPC
/// listener switches Kestrel into explicit-endpoints mode — see the call site's warning.
/// </summary>
/// <remarks>
/// Bind hosts (<c>+</c>, <c>*</c>, <c>0.0.0.0</c>, <c>[::]</c>, a hostname) are irrelevant
/// here because the caller re-binds via <c>ListenAnyIP</c>; only the port matters. A URL
/// with no explicit port falls back to the scheme default (80/443). Unparseable entries
/// are skipped rather than throwing — a bad URL should not take the node down at boot.
/// </remarks>
/// <param name="serverUrls">The server-URLs string; may be null/empty.</param>
/// <returns>The distinct ports, in first-seen order.</returns>
internal static IReadOnlyList<int> ParseHttpBindPorts(string? serverUrls)
{
var ports = new List<int>();
if (string.IsNullOrWhiteSpace(serverUrls))
{
return ports;
}
foreach (var raw in serverUrls.Split(';', StringSplitOptions.RemoveEmptyEntries | StringSplitOptions.TrimEntries))
{
int port;
// Uri can't parse the wildcard hosts Kestrel accepts (+, *), so normalize them
// to a placeholder host before parsing; the host is discarded anyway.
var normalized = raw.Replace("://+", "://placeholder").Replace("://*", "://placeholder");
if (Uri.TryCreate(normalized, UriKind.Absolute, out var uri))
{
port = uri.Port; // Uri fills the scheme default (80/443) when none is given.
}
else
{
// Last-ditch: pull the port after the final ':' (handles odd inputs Uri rejects).
var colon = raw.LastIndexOf(':');
if (colon < 0 || !int.TryParse(raw.AsSpan(colon + 1), out port))
{
continue;
}
}
if (port is > 0 and <= 65535 && !ports.Contains(port))
{
ports.Add(port);
}
}
return ports;
}
}
@@ -0,0 +1,67 @@
using Akka.Actor;
using Akka.Cluster.Tools.Client;
using Akka.TestKit.Xunit2;
using ZB.MOM.WW.ScadaBridge.Commons.Messages.Notification;
using ZB.MOM.WW.ScadaBridge.Communication.Actors;
namespace ZB.MOM.WW.ScadaBridge.Communication.Tests.Actors;
/// <summary>
/// T1A.3: the regression-prone bit of the Akka transport — that a forwarded
/// <see cref="ClusterClient.Send"/> carries the caller's sender, so central's reply routes
/// straight back to the waiting Ask rather than to the site communication actor — plus the
/// no-ClusterClient-yet fallback that keeps the S&amp;F layer treating the send as transient.
/// </summary>
public class AkkaCentralTransportTests : TestKit
{
[Fact]
public void SubmitNotification_ForwardsToClusterClient_WithReplyToAsSender()
{
var transport = new AkkaCentralTransport();
var clusterClient = CreateTestProbe();
var replyTo = CreateTestProbe();
transport.SetCentralClient(clusterClient.Ref);
var submit = new NotificationSubmit(
"notif-1", "Operators", "S", "B", "site1", null, null, DateTimeOffset.UtcNow);
transport.SubmitNotification(submit, replyTo.Ref);
// The ClusterClient receives a Send addressed to the central actor...
var send = clusterClient.ExpectMsg<ClusterClient.Send>();
Assert.Equal("/user/central-communication", send.Path);
Assert.IsType<NotificationSubmit>(send.Message);
// ...and replying to it lands at replyTo, proving the sender was forwarded (not the
// transport / actor). This is the routing the waiting Ask relies on.
clusterClient.Reply(new NotificationSubmitAck("notif-1", Accepted: true, Error: null));
replyTo.ExpectMsg<NotificationSubmitAck>(ack => ack.NotificationId == "notif-1" && ack.Accepted);
}
[Fact]
public void SubmitNotification_WithNoClusterClient_RepliesNonAcceptedToReplyTo()
{
var transport = new AkkaCentralTransport();
var replyTo = CreateTestProbe();
transport.SubmitNotification(
new NotificationSubmit("notif-2", "Operators", "S", "B", "site1", null, null, DateTimeOffset.UtcNow),
replyTo.Ref);
replyTo.ExpectMsg<NotificationSubmitAck>(ack => ack.NotificationId == "notif-2" && !ack.Accepted);
}
[Fact]
public void IngestAuditEvents_WithNoClusterClient_FaultsTheReplyTo()
{
// The audit drain treats a faulted Ask as transient and keeps rows Pending — so the
// no-client path must be a Status.Failure, not a silent drop.
var transport = new AkkaCentralTransport();
var replyTo = CreateTestProbe();
transport.IngestAuditEvents(
new Commons.Messages.Audit.IngestAuditEventsCommand(new List<ZB.MOM.WW.Audit.AuditEvent>()),
replyTo.Ref);
replyTo.ExpectMsg<Status.Failure>();
}
}
@@ -0,0 +1,166 @@
using Akka.Actor;
using Akka.TestKit.Xunit2;
using NSubstitute;
using ZB.MOM.WW.Audit;
using ZB.MOM.WW.ScadaBridge.Commons.Messages.Audit;
using ZB.MOM.WW.ScadaBridge.Commons.Messages.Deployment;
using ZB.MOM.WW.ScadaBridge.Commons.Messages.Health;
using ZB.MOM.WW.ScadaBridge.Commons.Messages.Notification;
using ZB.MOM.WW.ScadaBridge.Commons.Types.Enums;
using ZB.MOM.WW.ScadaBridge.Communication.Actors;
namespace ZB.MOM.WW.ScadaBridge.Communication.Tests.Actors;
/// <summary>
/// T1A.3: the site communication actor delegates each of the seven site→central sends to the
/// injected <see cref="ICentralTransport"/>, preserving the current <c>Sender</c> as the reply
/// target — and a transport failure surfaces to that sender exactly as the S&amp;F / audit / health
/// layers already expect, while a heartbeat transport fault never faults the actor.
/// </summary>
public class SiteCommunicationActorTransportTests : TestKit
{
private readonly CommunicationOptions _options = new();
private (IActorRef actor, ICentralTransport transport) NewActor()
{
var transport = Substitute.For<ICentralTransport>();
var dmProbe = CreateTestProbe();
var actor = Sys.ActorOf(Props.Create(() =>
new SiteCommunicationActor("site1", _options, dmProbe.Ref, () => false, null, transport)));
return (actor, transport);
}
[Fact]
public void NotificationSubmit_DelegatesToTransport_WithSenderAsReplyTo()
{
var (actor, transport) = NewActor();
var submit = new NotificationSubmit(
"notif-1", "Operators", "Subj", "Body", "site1", "inst1", "alarmScript", DateTimeOffset.UtcNow);
actor.Tell(submit, TestActor);
AwaitAssert(() => transport.Received(1).SubmitNotification(
Arg.Is<NotificationSubmit>(m => m.NotificationId == "notif-1"), Arg.Is(TestActor)));
}
[Fact]
public void NotificationStatusQuery_DelegatesToTransport_WithSenderAsReplyTo()
{
var (actor, transport) = NewActor();
actor.Tell(new NotificationStatusQuery("corr-1", "notif-1"), TestActor);
AwaitAssert(() => transport.Received(1).QueryNotificationStatus(
Arg.Is<NotificationStatusQuery>(m => m.NotificationId == "notif-1"), Arg.Is(TestActor)));
}
[Fact]
public void IngestAuditEvents_DelegatesToTransport_WithSenderAsReplyTo()
{
var (actor, transport) = NewActor();
actor.Tell(new IngestAuditEventsCommand(new List<AuditEvent>()), TestActor);
AwaitAssert(() => transport.Received(1).IngestAuditEvents(Arg.Any<IngestAuditEventsCommand>(), Arg.Is(TestActor)));
}
[Fact]
public void IngestCachedTelemetry_DelegatesToTransport_WithSenderAsReplyTo()
{
var (actor, transport) = NewActor();
actor.Tell(new IngestCachedTelemetryCommand(new List<CachedTelemetryEntry>()), TestActor);
AwaitAssert(() => transport.Received(1).IngestCachedTelemetry(Arg.Any<IngestCachedTelemetryCommand>(), Arg.Is(TestActor)));
}
[Fact]
public void ReconcileSite_DelegatesToTransport_WithSenderAsReplyTo()
{
var (actor, transport) = NewActor();
actor.Tell(
new ReconcileSiteRequest("site1", "node-a", new Dictionary<string, string>()), TestActor);
AwaitAssert(() => transport.Received(1).ReconcileSite(
Arg.Is<ReconcileSiteRequest>(m => m.NodeId == "node-a"), Arg.Is(TestActor)));
}
[Fact]
public void ReportSiteHealth_DelegatesToTransport_WithSenderAsReplyTo()
{
var (actor, transport) = NewActor();
var report = MinimalHealthReport(sequence: 7);
actor.Tell(report, TestActor);
AwaitAssert(() => transport.Received(1).ReportSiteHealth(
Arg.Is<SiteHealthReport>(m => m.SequenceNumber == 7), Arg.Is(TestActor)));
}
[Fact]
public void TransportFailureReply_RoutesBackToTheWaitingSender()
{
// The seam preserves the reply-routing the S&F layer depends on: when the transport
// answers the captured replyTo with a Status.Failure (its transient-failure signal on a
// non-OK status), that failure reaches the original sender — here the test actor — so the
// waiting Ask faults, exactly as it did on the ClusterClient path.
var transport = Substitute.For<ICentralTransport>();
transport
.When(t => t.SubmitNotification(Arg.Any<NotificationSubmit>(), Arg.Any<IActorRef>()))
.Do(ci => ci.Arg<IActorRef>().Tell(
new Status.Failure(new InvalidOperationException("central unavailable"))));
var dmProbe = CreateTestProbe();
var actor = Sys.ActorOf(Props.Create(() =>
new SiteCommunicationActor("site1", _options, dmProbe.Ref, () => false, null, transport)));
actor.Tell(new NotificationSubmit(
"notif-x", "Operators", "S", "B", "site1", null, null, DateTimeOffset.UtcNow), TestActor);
var failure = ExpectMsg<Status.Failure>();
Assert.IsType<InvalidOperationException>(failure.Cause);
}
[Fact]
public void HeartbeatTransportThrow_DoesNotFaultTheActor()
{
// A transport whose SendHeartbeat throws must not fault the actor — heartbeats are
// fire-and-forget and their failure is swallowed. Prove the actor still serves messages
// after a heartbeat that threw.
var transport = Substitute.For<ICentralTransport>();
transport
.When(t => t.SendHeartbeat(Arg.Any<HeartbeatMessage>(), Arg.Any<IActorRef>()))
.Do(_ => throw new InvalidOperationException("boom"));
var dmProbe = CreateTestProbe();
var actor = Sys.ActorOf(Props.Create(() =>
new SiteCommunicationActor(
"site1",
new CommunicationOptions { ApplicationHeartbeatInterval = TimeSpan.FromMilliseconds(50) },
dmProbe.Ref, () => true, null, transport)));
// Let the heartbeat timer fire a few times (each throws inside the transport).
Thread.Sleep(250);
// The actor is still alive and delegating: a subsequent send is handled normally.
actor.Tell(new NotificationSubmit(
"after-heartbeat", "Operators", "S", "B", "site1", null, null, DateTimeOffset.UtcNow), TestActor);
AwaitAssert(() => transport.Received(1).SubmitNotification(
Arg.Is<NotificationSubmit>(m => m.NotificationId == "after-heartbeat"), Arg.Is(TestActor)));
}
private static SiteHealthReport MinimalHealthReport(long sequence) => new(
SiteId: "site1",
SequenceNumber: sequence,
ReportTimestamp: DateTimeOffset.UtcNow,
DataConnectionStatuses: new Dictionary<string, ConnectionHealth>(),
TagResolutionCounts: new Dictionary<string, TagResolutionStatus>(),
ScriptErrorCount: 0,
AlarmEvaluationErrorCount: 0,
StoreAndForwardBufferDepths: new Dictionary<string, int>(),
DeadLetterCount: 0,
DeployedInstanceCount: 0,
EnabledInstanceCount: 0,
DisabledInstanceCount: 0);
}
@@ -39,7 +39,8 @@ public class CentralCommunicationActorClientLifecycleTests : TestKit
private static SiteAddressCacheLoaded Load(string siteId, params string[] addrs) =>
new(new Dictionary<string, IReadOnlyList<string>>
{ [siteId] = addrs.ToList().AsReadOnly() },
new[] { siteId });
new[] { siteId },
new Dictionary<string, SiteGrpcEndpoints>());
[Fact]
public void PeriodicRefresh_PrunesDeletedSites_FromHealthAggregator()
@@ -0,0 +1,136 @@
using Akka.Actor;
using Akka.TestKit.Xunit2;
using Microsoft.Extensions.DependencyInjection;
using NSubstitute;
using ZB.MOM.WW.ScadaBridge.Commons.Entities.Sites;
using ZB.MOM.WW.ScadaBridge.Commons.Interfaces.Repositories;
using ZB.MOM.WW.ScadaBridge.Commons.Messages.Deployment;
using ZB.MOM.WW.ScadaBridge.Commons.Messages.Lifecycle;
using ZB.MOM.WW.ScadaBridge.Commons.Types.Enums;
using ZB.MOM.WW.ScadaBridge.Communication.Actors;
namespace ZB.MOM.WW.ScadaBridge.Communication.Tests;
/// <summary>
/// T1B.3 seam tests: <see cref="CentralCommunicationActor"/> routing a <see cref="SiteEnvelope"/>
/// through an injected <see cref="ISiteCommandTransport"/> (transport-agnostic), proving the
/// envelope reaches the transport, the reply routes back to the waiting Ask, and the DB refresh
/// tick reconciles per-site transport resources.
/// </summary>
public class CentralCommunicationActorTransportTests : TestKit
{
public CentralCommunicationActorTransportTests() : base(@"akka.loglevel = WARNING") { }
private static Site GrpcSite(string id, string? grpcA = "http://a:8083", string? grpcB = "http://b:8083") =>
new("Test " + id, id)
{
NodeAAddress = $"akka.tcp://scadabridge@{id}-a:8081",
GrpcNodeAAddress = grpcA,
GrpcNodeBAddress = grpcB
};
private (IActorRef actor, ISiteCommandTransport transport, ISiteRepository repo) CreateActor(
IEnumerable<Site>? sites = null)
{
var repo = Substitute.For<ISiteRepository>();
repo.GetAllSitesAsync(Arg.Any<CancellationToken>()).Returns(sites?.ToList() ?? new List<Site>());
var services = new ServiceCollection();
services.AddScoped(_ => repo);
var sp = services.BuildServiceProvider();
var transport = Substitute.For<ISiteCommandTransport>();
var actor = Sys.ActorOf(Props.Create(() => new CentralCommunicationActor(sp, transport, (TimeSpan?)null)));
return (actor, transport, repo);
}
[Fact]
public void SiteEnvelope_IsRoutedToTheTransport_WithTheSenderAsReplyTo()
{
var (actor, transport, _) = CreateActor();
var probe = CreateTestProbe();
var cmd = new EnableInstanceCommand("cmd-1", "Site1.Pump1", DateTimeOffset.UtcNow);
actor.Tell(new SiteEnvelope("site-a", cmd), probe.Ref);
AwaitAssert(() => transport.Received(1).Send(
Arg.Is<SiteEnvelope>(e => e.SiteId == "site-a" && ReferenceEquals(e.Message, cmd)),
probe.Ref));
}
[Fact]
public async Task AskReply_FromTransport_RoutesBackToTheWaitingAsk()
{
var (actor, transport, _) = CreateActor();
// The transport substitute stands in for the site: when handed the envelope it delivers a
// reply to the captured replyTo, exactly as a real transport pipes the site's reply back.
var reply = new DeploymentStatusResponse(
"dep-1", "Site1.Pump1", DeploymentStatus.Success, null, DateTimeOffset.UtcNow);
transport
.When(t => t.Send(Arg.Any<SiteEnvelope>(), Arg.Any<IActorRef>()))
.Do(ci => ci.Arg<IActorRef>().Tell(reply));
var cmd = new RefreshDeploymentCommand(
"dep-1", "Site1.Pump1", "hash", "multi-role", DateTimeOffset.UtcNow, "http://c:5000", "tok");
var result = await actor.Ask<DeploymentStatusResponse>(
new SiteEnvelope("site-a", cmd), TimeSpan.FromSeconds(3));
Assert.Equal("dep-1", result.DeploymentId);
Assert.Equal(DeploymentStatus.Success, result.Status);
}
[Fact]
public void DbRefresh_ReconcilesTheTransport_WithTheLoadedGrpcEndpoints()
{
var (_, transport, _) = CreateActor(new[] { GrpcSite("site-a") });
// PreStart fires the refresh at Zero; the loaded cache is handed to the transport.
AwaitAssert(() => transport.Received().ReconcileSites(
Arg.Is<SiteAddressCacheLoaded>(c =>
c.KnownSiteIds.Contains("site-a")
&& c.GrpcContacts.ContainsKey("site-a")
&& c.GrpcContacts["site-a"].NodeA == "http://a:8083"
&& c.GrpcContacts["site-a"].NodeB == "http://b:8083")),
TimeSpan.FromSeconds(3));
}
[Fact]
public void SiteAddedAndRemoved_AcrossRefreshes_FlowsThroughToTheTransport()
{
var (actor, transport, repo) = CreateActor(new[] { GrpcSite("site-a") });
AwaitAssert(() => transport.Received().ReconcileSites(
Arg.Is<SiteAddressCacheLoaded>(c => c.GrpcContacts.ContainsKey("site-a"))),
TimeSpan.FromSeconds(3));
// site-a removed, site-b added.
repo.GetAllSitesAsync(Arg.Any<CancellationToken>()).Returns(new List<Site> { GrpcSite("site-b") });
actor.Tell(new RefreshSiteAddresses());
AwaitAssert(() => transport.Received().ReconcileSites(
Arg.Is<SiteAddressCacheLoaded>(c =>
c.GrpcContacts.ContainsKey("site-b")
&& !c.GrpcContacts.ContainsKey("site-a")
&& c.KnownSiteIds.Contains("site-b")
&& !c.KnownSiteIds.Contains("site-a"))),
TimeSpan.FromSeconds(3));
}
[Fact]
public void SiteWithoutGrpcAddresses_IsAbsentFromGrpcContacts_ButStillKnown()
{
// A site with only Akka addresses (no gRPC columns) is a known site but carries no gRPC
// endpoint — the gRPC transport must not try to dial it, the Akka one still can.
var akkaOnly = new Site("Akka only", "site-x")
{
NodeAAddress = "akka.tcp://scadabridge@site-x-a:8081"
};
var (_, transport, _) = CreateActor(new[] { akkaOnly });
AwaitAssert(() => transport.Received().ReconcileSites(
Arg.Is<SiteAddressCacheLoaded>(c =>
c.KnownSiteIds.Contains("site-x") && !c.GrpcContacts.ContainsKey("site-x"))),
TimeSpan.FromSeconds(3));
}
}
@@ -104,4 +104,57 @@ public class CommunicationOptionsValidatorTests
Assert.True(result.Failed);
Assert.Contains("LiveAlarmCachePublishCoalesce", result.FailureMessage);
}
// ── T1A.3: gRPC central transport endpoints (required only when selected) ────
[Fact]
public void GrpcTransport_WithNoEndpoints_IsRejected()
{
var result = Validate(new CommunicationOptions
{
CentralTransport = CentralTransportMode.Grpc,
CentralGrpcEndpoints = new List<string>(),
});
Assert.True(result.Failed);
Assert.Contains("CentralGrpcEndpoints", result.FailureMessage);
}
[Fact]
public void GrpcTransport_WithBlankEndpoint_IsRejected()
{
var result = Validate(new CommunicationOptions
{
CentralTransport = CentralTransportMode.Grpc,
CentralGrpcEndpoints = new List<string> { " " },
});
Assert.True(result.Failed);
Assert.Contains("CentralGrpcEndpoints", result.FailureMessage);
}
[Fact]
public void GrpcTransport_WithEndpoints_IsValid()
{
var result = Validate(new CommunicationOptions
{
CentralTransport = CentralTransportMode.Grpc,
CentralGrpcEndpoints = new List<string>
{
"http://scadabridge-central-a:8083",
"http://scadabridge-central-b:8083",
},
});
Assert.True(result.Succeeded, result.FailureMessage);
}
[Fact]
public void AkkaTransport_IgnoresMissingGrpcEndpoints()
{
// The default transport must not be forced to declare gRPC endpoints it never dials.
var result = Validate(new CommunicationOptions
{
CentralTransport = CentralTransportMode.Akka,
CentralGrpcEndpoints = new List<string>(),
});
Assert.True(result.Succeeded, result.FailureMessage);
}
}
@@ -0,0 +1,643 @@
using Google.Protobuf;
using ZB.MOM.WW.ScadaBridge.Commons.Entities.Audit;
using ZB.MOM.WW.ScadaBridge.Commons.Messages.Audit;
using ZB.MOM.WW.ScadaBridge.Commons.Messages.Deployment;
using ZB.MOM.WW.ScadaBridge.Commons.Messages.Health;
using ZB.MOM.WW.ScadaBridge.Commons.Messages.Notification;
using ZB.MOM.WW.ScadaBridge.Commons.Types;
using ZB.MOM.WW.ScadaBridge.Commons.Types.Audit;
using ZB.MOM.WW.ScadaBridge.Commons.Types.Enums;
using ZB.MOM.WW.ScadaBridge.Communication.Grpc;
namespace ZB.MOM.WW.ScadaBridge.Communication.Tests.Grpc;
/// <summary>
/// Round-trip golden tests for <see cref="CentralControlDtoMapper"/> — the DTO bridge
/// for the seven <c>CentralControlService</c> RPCs (Phase 1A of the ClusterClient→gRPC
/// migration).
/// </summary>
/// <remarks>
/// <para>
/// Every message gets a pair: a FULLY-POPULATED case that proves no field is dropped,
/// and a MINIMAL case that proves every nullable/optional/empty-collection member comes
/// back as null-or-empty rather than as a zero-valued stand-in. The minimal cases are
/// the ones that matter: a per-field unit test happily passes while a whole optional
/// branch is silently never written, which is exactly the class of bug the round-trip
/// guard in PLAN-05 T8 caught five times over.
/// </para>
/// <para>
/// When a round-trip here fails, the mapper is wrong — not the test.
/// </para>
/// </remarks>
public class CentralControlDtoMapperTests
{
private static readonly DateTimeOffset SampleInstant =
new(2026, 7, 22, 9, 30, 15, 250, TimeSpan.Zero);
// -----------------------------------------------------------------------
// NotificationSubmit / Ack
// -----------------------------------------------------------------------
[Fact]
public void NotificationSubmit_RoundTrip_FullyPopulated_PreservesEveryField()
{
var original = new NotificationSubmit(
NotificationId: Guid.NewGuid().ToString(),
ListName: "plant-ops",
Subject: "Tank 4 overfill",
Body: "Level exceeded 95% for 5 minutes.",
SourceSiteId: "site-a",
SourceInstanceId: "Tank04",
SourceScript: "OnLevelHigh",
SiteEnqueuedAt: SampleInstant,
OriginExecutionId: Guid.NewGuid(),
OriginParentExecutionId: Guid.NewGuid(),
SourceNode: "node-b");
var roundTripped = CentralControlDtoMapper.FromDto(Wire(CentralControlDtoMapper.ToDto(original)));
// Every member is a scalar, so record value-equality is a true deep compare.
Assert.Equal(original, roundTripped);
}
[Fact]
public void NotificationSubmit_RoundTrip_AllOptionalsNull_StayNull()
{
var original = new NotificationSubmit(
NotificationId: Guid.NewGuid().ToString(),
ListName: "plant-ops",
Subject: "s",
Body: "b",
SourceSiteId: "site-a",
SourceInstanceId: null,
SourceScript: null,
SiteEnqueuedAt: SampleInstant,
OriginExecutionId: null,
OriginParentExecutionId: null,
SourceNode: null);
var roundTripped = CentralControlDtoMapper.FromDto(Wire(CentralControlDtoMapper.ToDto(original)));
Assert.Equal(original, roundTripped);
Assert.Null(roundTripped.SourceInstanceId);
Assert.Null(roundTripped.SourceScript);
Assert.Null(roundTripped.OriginExecutionId);
Assert.Null(roundTripped.OriginParentExecutionId);
Assert.Null(roundTripped.SourceNode);
}
[Fact]
public void NotificationSubmit_NonUtcOffset_NormalizesToTheSameInstant()
{
// Documented contract: a protobuf Timestamp is an instant, so the offset
// component of a DateTimeOffset does not survive. Every producer stamps UTC
// (Notify.Send uses DateTimeOffset.UtcNow), so the instant is what matters.
var melbourne = new DateTimeOffset(2026, 7, 22, 19, 30, 15, TimeSpan.FromHours(10));
var original = new NotificationSubmit(
"id", "list", "s", "b", "site-a", null, null, melbourne);
var roundTripped = CentralControlDtoMapper.FromDto(Wire(CentralControlDtoMapper.ToDto(original)));
Assert.Equal(melbourne.UtcDateTime, roundTripped.SiteEnqueuedAt.UtcDateTime);
Assert.Equal(TimeSpan.Zero, roundTripped.SiteEnqueuedAt.Offset);
}
[Fact]
public void NotificationSubmitAck_RoundTrip_Accepted_And_Rejected()
{
var accepted = new NotificationSubmitAck("n1", Accepted: true, Error: null);
var rejected = new NotificationSubmitAck("n2", Accepted: false, Error: "list not found");
Assert.Equal(accepted, CentralControlDtoMapper.FromDto(Wire(CentralControlDtoMapper.ToDto(accepted))));
Assert.Equal(rejected, CentralControlDtoMapper.FromDto(Wire(CentralControlDtoMapper.ToDto(rejected))));
Assert.Null(CentralControlDtoMapper.FromDto(Wire(CentralControlDtoMapper.ToDto(accepted))).Error);
}
// -----------------------------------------------------------------------
// NotificationStatusQuery / Response
// -----------------------------------------------------------------------
[Fact]
public void NotificationStatusQuery_RoundTrip_PreservesEveryField()
{
var original = new NotificationStatusQuery("corr-1", Guid.NewGuid().ToString());
Assert.Equal(original, CentralControlDtoMapper.FromDto(Wire(CentralControlDtoMapper.ToDto(original))));
}
[Fact]
public void NotificationStatusResponse_RoundTrip_FullyPopulated_PreservesEveryField()
{
var original = new NotificationStatusResponse(
CorrelationId: "corr-1",
Found: true,
Status: "Delivered",
RetryCount: 3,
LastError: "smtp 421",
DeliveredAt: SampleInstant);
Assert.Equal(original, CentralControlDtoMapper.FromDto(Wire(CentralControlDtoMapper.ToDto(original))));
}
[Fact]
public void NotificationStatusResponse_RoundTrip_NotFound_LeavesOptionalsNull()
{
var original = new NotificationStatusResponse(
CorrelationId: "corr-1",
Found: false,
Status: "Unknown",
RetryCount: 0,
LastError: null,
DeliveredAt: null);
var roundTripped = CentralControlDtoMapper.FromDto(Wire(CentralControlDtoMapper.ToDto(original)));
Assert.Equal(original, roundTripped);
Assert.Null(roundTripped.LastError);
Assert.Null(roundTripped.DeliveredAt);
}
// -----------------------------------------------------------------------
// Audit ingest envelopes (rows delegate to AuditEventDtoMapper / SiteCallDtoMapper)
// -----------------------------------------------------------------------
[Fact]
public void IngestAuditEventsCommand_RoundTrip_PreservesOrderAndRows()
{
var first = NewAuditEvent(sourceScript: "OnDemand");
var second = NewAuditEvent(sourceScript: "OnTrigger");
var original = new IngestAuditEventsCommand([first, second]);
var roundTripped = CentralControlDtoMapper.FromDto(Wire(CentralControlDtoMapper.ToDto(original)));
Assert.Equal(2, roundTripped.Events.Count);
Assert.Equal(first.AsRow().EventId, roundTripped.Events[0].AsRow().EventId);
Assert.Equal("OnDemand", roundTripped.Events[0].AsRow().SourceScript);
Assert.Equal(second.AsRow().EventId, roundTripped.Events[1].AsRow().EventId);
Assert.Equal("OnTrigger", roundTripped.Events[1].AsRow().SourceScript);
}
[Fact]
public void IngestAuditEventsCommand_RoundTrip_EmptyBatch_StaysEmpty()
{
var roundTripped = CentralControlDtoMapper.FromDto(
Wire(CentralControlDtoMapper.ToDto(new IngestAuditEventsCommand([]))));
Assert.Empty(roundTripped.Events);
}
[Fact]
public void IngestCachedTelemetryCommand_RoundTrip_PreservesBothHalvesOfEachPacket()
{
var audit = NewAuditEvent(sourceScript: "OnDemand");
var siteCall = NewSiteCall();
var original = new IngestCachedTelemetryCommand([new CachedTelemetryEntry(audit, siteCall)]);
var roundTripped = CentralControlDtoMapper.FromDto(Wire(CentralControlDtoMapper.ToDto(original)));
var entry = Assert.Single(roundTripped.Entries);
Assert.Equal(audit.AsRow().EventId, entry.Audit.AsRow().EventId);
// IngestedAtUtc is central-set inside the dual-write transaction and is
// deliberately off the wire, so it is the one member excluded from the compare.
Assert.Equal(siteCall with { IngestedAtUtc = default }, entry.SiteCall with { IngestedAtUtc = default });
}
[Fact]
public void IngestCachedTelemetryCommand_RoundTrip_NullableSiteCallFields_StayNull()
{
var siteCall = NewSiteCall() with
{
SourceNode = null,
LastError = null,
HttpStatus = null,
TerminalAtUtc = null,
};
var original = new IngestCachedTelemetryCommand(
[new CachedTelemetryEntry(NewAuditEvent(), siteCall)]);
var roundTripped = CentralControlDtoMapper.FromDto(Wire(CentralControlDtoMapper.ToDto(original)));
var entry = Assert.Single(roundTripped.Entries);
Assert.Null(entry.SiteCall.SourceNode);
Assert.Null(entry.SiteCall.LastError);
Assert.Null(entry.SiteCall.HttpStatus);
Assert.Null(entry.SiteCall.TerminalAtUtc);
}
[Fact]
public void IngestCachedTelemetryCommand_RoundTrip_EmptyBatch_StaysEmpty()
{
var roundTripped = CentralControlDtoMapper.FromDto(
Wire(CentralControlDtoMapper.ToDto(new IngestCachedTelemetryCommand([]))));
Assert.Empty(roundTripped.Entries);
}
[Fact]
public void IngestAck_RoundTrip_PreservesIdsAndOrder()
{
var ids = new[] { Guid.NewGuid(), Guid.NewGuid(), Guid.NewGuid() };
var roundTripped = CentralControlDtoMapper.FromIngestAck(
Wire(CentralControlDtoMapper.ToIngestAck(ids)));
Assert.Equal(ids, roundTripped);
}
[Fact]
public void IngestAck_RoundTrip_NothingAccepted_StaysEmpty()
{
// An empty ack is meaningful — "zero rows persisted" leaves the site's rows
// Pending — so it must not be indistinguishable from a dropped field.
Assert.Empty(CentralControlDtoMapper.FromIngestAck(
Wire(CentralControlDtoMapper.ToIngestAck([]))));
}
// -----------------------------------------------------------------------
// Startup reconciliation
// -----------------------------------------------------------------------
[Fact]
public void ReconcileSiteRequest_RoundTrip_PreservesInventoryMap()
{
var original = new ReconcileSiteRequest(
SiteIdentifier: "site-a",
NodeId: "node-a",
LocalNameToRevisionHash: new Dictionary<string, string>
{
["Plant.Tank04"] = "hash-1",
["Plant.Pump01"] = "hash-2",
});
var roundTripped = CentralControlDtoMapper.FromDto(Wire(CentralControlDtoMapper.ToDto(original)));
Assert.Equal(original.SiteIdentifier, roundTripped.SiteIdentifier);
Assert.Equal(original.NodeId, roundTripped.NodeId);
Assert.Equal(
original.LocalNameToRevisionHash.OrderBy(kv => kv.Key),
roundTripped.LocalNameToRevisionHash.OrderBy(kv => kv.Key));
}
[Fact]
public void ReconcileSiteRequest_RoundTrip_EmptyInventory_StaysEmpty()
{
// A node with nothing deployed is the normal first-boot case, and central
// must see an empty inventory rather than a missing one.
var roundTripped = CentralControlDtoMapper.FromDto(Wire(CentralControlDtoMapper.ToDto(
new ReconcileSiteRequest("site-a", "node-a", new Dictionary<string, string>()))));
Assert.Empty(roundTripped.LocalNameToRevisionHash);
}
[Fact]
public void ReconcileSiteResponse_RoundTrip_PreservesGapOrphansAndUrl()
{
var original = new ReconcileSiteResponse(
Gap:
[
new ReconcileGapItem("Plant.Tank04", "dep-1", "hash-1", IsEnabled: true, "token-1"),
new ReconcileGapItem("Plant.Pump01", "dep-2", "hash-2", IsEnabled: false, "token-2"),
],
OrphanNames: ["Plant.Retired01", "Plant.Retired02"],
CentralFetchBaseUrl: "http://scadabridge-central-a:5000");
var roundTripped = CentralControlDtoMapper.FromDto(Wire(CentralControlDtoMapper.ToDto(original)));
Assert.Equal(original.Gap, roundTripped.Gap);
Assert.Equal(original.OrphanNames, roundTripped.OrphanNames);
Assert.Equal(original.CentralFetchBaseUrl, roundTripped.CentralFetchBaseUrl);
}
[Fact]
public void ReconcileSiteResponse_RoundTrip_NoGap_StaysEmpty()
{
var roundTripped = CentralControlDtoMapper.FromDto(Wire(CentralControlDtoMapper.ToDto(
new ReconcileSiteResponse([], [], "http://central:5000"))));
Assert.Empty(roundTripped.Gap);
Assert.Empty(roundTripped.OrphanNames);
}
// -----------------------------------------------------------------------
// Site health
// -----------------------------------------------------------------------
[Fact]
public void SiteHealthReport_RoundTrip_FullyPopulated_PreservesEveryField()
{
var original = FullyPopulatedHealthReport();
var roundTripped = CentralControlDtoMapper.FromDto(Wire(CentralControlDtoMapper.ToDto(original)));
AssertHealthReportsEqual(original, roundTripped);
}
[Fact]
public void SiteHealthReport_RoundTrip_MinimalReport_LeavesEveryOptionalNullOrEmpty()
{
// The shape a producer emits before any reporter has run: no endpoints map, no
// tag-quality map, no cluster-node list, no audit backlog, no nullable gauges.
var original = new SiteHealthReport(
SiteId: "site-a",
SequenceNumber: 1,
ReportTimestamp: SampleInstant,
DataConnectionStatuses: new Dictionary<string, ConnectionHealth>(),
TagResolutionCounts: new Dictionary<string, TagResolutionStatus>(),
ScriptErrorCount: 0,
AlarmEvaluationErrorCount: 0,
StoreAndForwardBufferDepths: new Dictionary<string, int>(),
DeadLetterCount: 0,
DeployedInstanceCount: 0,
EnabledInstanceCount: 0,
DisabledInstanceCount: 0);
var roundTripped = CentralControlDtoMapper.FromDto(Wire(CentralControlDtoMapper.ToDto(original)));
AssertHealthReportsEqual(original, roundTripped);
Assert.Empty(roundTripped.DataConnectionStatuses);
Assert.Empty(roundTripped.TagResolutionCounts);
Assert.Empty(roundTripped.StoreAndForwardBufferDepths);
Assert.Null(roundTripped.DataConnectionEndpoints);
Assert.Null(roundTripped.DataConnectionTagQuality);
Assert.Null(roundTripped.ClusterNodes);
Assert.Null(roundTripped.SiteAuditBacklog);
Assert.Null(roundTripped.OldestParkedMessageAgeSeconds);
Assert.Null(roundTripped.ScriptOldestBusyAgeSeconds);
Assert.Null(roundTripped.LocalDbReplicationConnected);
Assert.Null(roundTripped.LocalDbOplogBacklog);
}
[Fact]
public void SiteHealthReport_RoundTrip_EmptyNullableCollections_StayEmptyNotNull()
{
// The distinction the three wrapper messages exist for. Null means "this node
// does not report the signal"; empty means "it reports it, and there is
// nothing in it". proto3 cannot express presence on repeated/map fields, so
// collapsing one into the other here would be invisible until an operator
// misread the health page.
var original = FullyPopulatedHealthReport() with
{
DataConnectionEndpoints = new Dictionary<string, string>(),
DataConnectionTagQuality = new Dictionary<string, TagQualityCounts>(),
ClusterNodes = [],
};
var roundTripped = CentralControlDtoMapper.FromDto(Wire(CentralControlDtoMapper.ToDto(original)));
Assert.NotNull(roundTripped.DataConnectionEndpoints);
Assert.Empty(roundTripped.DataConnectionEndpoints);
Assert.NotNull(roundTripped.DataConnectionTagQuality);
Assert.Empty(roundTripped.DataConnectionTagQuality);
Assert.NotNull(roundTripped.ClusterNodes);
Assert.Empty(roundTripped.ClusterNodes);
}
[Fact]
public void SiteHealthReport_RoundTrip_FalseAndZeroGauges_StayFalseAndZero()
{
// The mirror of the null case: LocalDbReplicationConnected=false ("configured
// but disconnected") and LocalDbOplogBacklog=0 ("healthy, nothing queued") are
// real values that must not decay into null on the wire.
var original = FullyPopulatedHealthReport() with
{
LocalDbReplicationConnected = false,
LocalDbOplogBacklog = 0,
OldestParkedMessageAgeSeconds = 0d,
ScriptOldestBusyAgeSeconds = 0d,
};
var roundTripped = CentralControlDtoMapper.FromDto(Wire(CentralControlDtoMapper.ToDto(original)));
Assert.False(roundTripped.LocalDbReplicationConnected);
Assert.Equal(0L, roundTripped.LocalDbOplogBacklog);
Assert.Equal(0d, roundTripped.OldestParkedMessageAgeSeconds);
Assert.Equal(0d, roundTripped.ScriptOldestBusyAgeSeconds);
}
[Fact]
public void SiteHealthReport_RoundTrip_BacklogWithEmptyQueue_KeepsNullOldestPending()
{
var original = FullyPopulatedHealthReport() with
{
SiteAuditBacklog = new SiteAuditBacklogSnapshot(0, null, 4096),
};
var roundTripped = CentralControlDtoMapper.FromDto(Wire(CentralControlDtoMapper.ToDto(original)));
Assert.Equal(new SiteAuditBacklogSnapshot(0, null, 4096), roundTripped.SiteAuditBacklog);
}
[Fact]
public void SiteHealthReportAck_RoundTrip_Accepted_And_Rejected()
{
var accepted = new SiteHealthReportAck("site-a", 42, Accepted: true);
var rejected = new SiteHealthReportAck("site-a", 43, Accepted: false, Error: "aggregator down");
Assert.Equal(accepted, CentralControlDtoMapper.FromDto(Wire(CentralControlDtoMapper.ToDto(accepted))));
Assert.Equal(rejected, CentralControlDtoMapper.FromDto(Wire(CentralControlDtoMapper.ToDto(rejected))));
Assert.Null(CentralControlDtoMapper.FromDto(Wire(CentralControlDtoMapper.ToDto(accepted))).Error);
}
// -----------------------------------------------------------------------
// Heartbeat + ConnectionHealth enum
// -----------------------------------------------------------------------
[Theory]
[InlineData(true)]
[InlineData(false)]
public void HeartbeatMessage_RoundTrip_PreservesEveryField(bool isActive)
{
var original = new HeartbeatMessage("site-a", "scadabridge-site-a-node-b", isActive, SampleInstant);
Assert.Equal(original, CentralControlDtoMapper.FromDto(Wire(CentralControlDtoMapper.ToDto(original))));
}
[Theory]
[InlineData(ConnectionHealth.Connected)]
[InlineData(ConnectionHealth.Disconnected)]
[InlineData(ConnectionHealth.Connecting)]
[InlineData(ConnectionHealth.Error)]
public void ConnectionHealth_RoundTrip_EveryValueSurvives(ConnectionHealth health)
{
Assert.Equal(health, CentralControlDtoMapper.FromDto(CentralControlDtoMapper.ToDto(health)));
}
[Fact]
public void ConnectionHealth_Unspecified_DecodesToErrorNotConnected()
{
// Fail-safe direction: a wire value this build does not recognise must never
// render as "healthy" on the central health page.
Assert.Equal(
ConnectionHealth.Error,
CentralControlDtoMapper.FromDto(ConnectionHealthEnum.ConnectionHealthUnspecified));
}
[Fact]
public void ConnectionHealth_EveryEnumMemberIsMapped()
{
// Guards the ArgumentOutOfRangeException path: adding a ConnectionHealth value
// without extending the mapper should fail here, not silently on a rig.
foreach (var health in Enum.GetValues<ConnectionHealth>())
{
var wire = CentralControlDtoMapper.ToDto(health);
Assert.NotEqual(ConnectionHealthEnum.ConnectionHealthUnspecified, wire);
}
}
// -----------------------------------------------------------------------
// Fixtures
// -----------------------------------------------------------------------
private static SiteHealthReport FullyPopulatedHealthReport() =>
new(
SiteId: "site-a",
SequenceNumber: 987654321L,
ReportTimestamp: SampleInstant,
DataConnectionStatuses: new Dictionary<string, ConnectionHealth>
{
["opc-main"] = ConnectionHealth.Connected,
["mx-gateway"] = ConnectionHealth.Error,
["opc-backup"] = ConnectionHealth.Connecting,
["legacy"] = ConnectionHealth.Disconnected,
},
TagResolutionCounts: new Dictionary<string, TagResolutionStatus>
{
["opc-main"] = new(TotalSubscribed: 120, SuccessfullyResolved: 118),
},
ScriptErrorCount: 4,
AlarmEvaluationErrorCount: 2,
StoreAndForwardBufferDepths: new Dictionary<string, int>
{
["erp"] = 17,
["mes"] = 0,
},
DeadLetterCount: 5,
DeployedInstanceCount: 30,
EnabledInstanceCount: 28,
DisabledInstanceCount: 2,
NodeRole: "Active",
NodeHostname: "scadabridge-site-a-node-a",
DataConnectionEndpoints: new Dictionary<string, string>
{
["opc-main"] = "opc.tcp://opcua:4840",
},
DataConnectionTagQuality: new Dictionary<string, TagQualityCounts>
{
["opc-main"] = new(Good: 100, Bad: 3, Uncertain: 15),
},
ParkedMessageCount: 6,
ClusterNodes:
[
new NodeStatus("scadabridge-site-a-node-a", IsOnline: true, "Active"),
new NodeStatus("scadabridge-site-a-node-b", IsOnline: false, "Standby"),
],
SiteAuditWriteFailures: 7,
AuditRedactionFailure: 8,
SiteAuditBacklog: new SiteAuditBacklogSnapshot(
PendingCount: 42,
OldestPendingUtc: new DateTime(2026, 7, 22, 8, 0, 0, DateTimeKind.Utc),
OnDiskBytes: 1_234_567L),
SiteEventLogWriteFailures: 9L,
OldestParkedMessageAgeSeconds: 3600.5d)
{
ScriptQueueDepth = 11,
ScriptBusyThreads = 3,
ScriptOldestBusyAgeSeconds = 12.25d,
LocalDbReplicationConnected = true,
LocalDbOplogBacklog = 250L,
};
/// <summary>
/// Compares two reports member by member. <see cref="SiteHealthReport"/> is a record,
/// but its dictionary/list members compare by reference under record equality, so
/// <c>Assert.Equal(a, b)</c> would pass trivially for the scalars and never look
/// inside the collections — the exact blind spot these goldens exist to close.
/// </summary>
private static void AssertHealthReportsEqual(SiteHealthReport expected, SiteHealthReport actual)
{
Assert.Equal(expected.SiteId, actual.SiteId);
Assert.Equal(expected.SequenceNumber, actual.SequenceNumber);
Assert.Equal(expected.ReportTimestamp, actual.ReportTimestamp);
Assert.Equal(
expected.DataConnectionStatuses.OrderBy(kv => kv.Key),
actual.DataConnectionStatuses.OrderBy(kv => kv.Key));
Assert.Equal(
expected.TagResolutionCounts.OrderBy(kv => kv.Key),
actual.TagResolutionCounts.OrderBy(kv => kv.Key));
Assert.Equal(expected.ScriptErrorCount, actual.ScriptErrorCount);
Assert.Equal(expected.AlarmEvaluationErrorCount, actual.AlarmEvaluationErrorCount);
Assert.Equal(
expected.StoreAndForwardBufferDepths.OrderBy(kv => kv.Key),
actual.StoreAndForwardBufferDepths.OrderBy(kv => kv.Key));
Assert.Equal(expected.DeadLetterCount, actual.DeadLetterCount);
Assert.Equal(expected.DeployedInstanceCount, actual.DeployedInstanceCount);
Assert.Equal(expected.EnabledInstanceCount, actual.EnabledInstanceCount);
Assert.Equal(expected.DisabledInstanceCount, actual.DisabledInstanceCount);
Assert.Equal(expected.NodeRole, actual.NodeRole);
Assert.Equal(expected.NodeHostname, actual.NodeHostname);
Assert.Equal(
expected.DataConnectionEndpoints?.OrderBy(kv => kv.Key),
actual.DataConnectionEndpoints?.OrderBy(kv => kv.Key));
Assert.Equal(
expected.DataConnectionTagQuality?.OrderBy(kv => kv.Key),
actual.DataConnectionTagQuality?.OrderBy(kv => kv.Key));
Assert.Equal(expected.ParkedMessageCount, actual.ParkedMessageCount);
Assert.Equal(expected.ClusterNodes, actual.ClusterNodes);
Assert.Equal(expected.SiteAuditWriteFailures, actual.SiteAuditWriteFailures);
Assert.Equal(expected.AuditRedactionFailure, actual.AuditRedactionFailure);
Assert.Equal(expected.SiteAuditBacklog, actual.SiteAuditBacklog);
Assert.Equal(expected.SiteEventLogWriteFailures, actual.SiteEventLogWriteFailures);
Assert.Equal(expected.OldestParkedMessageAgeSeconds, actual.OldestParkedMessageAgeSeconds);
Assert.Equal(expected.ScriptQueueDepth, actual.ScriptQueueDepth);
Assert.Equal(expected.ScriptBusyThreads, actual.ScriptBusyThreads);
Assert.Equal(expected.ScriptOldestBusyAgeSeconds, actual.ScriptOldestBusyAgeSeconds);
Assert.Equal(expected.LocalDbReplicationConnected, actual.LocalDbReplicationConnected);
Assert.Equal(expected.LocalDbOplogBacklog, actual.LocalDbOplogBacklog);
}
/// <summary>
/// Serializes a wire message and parses it back, so every round-trip below crosses a
/// real protobuf encode/decode rather than only exercising the mapper's object graph.
/// This is what proves the three collection wrapper messages keep their presence
/// when empty — an empty message encodes as a tag with zero-length payload, and a
/// mapper that used a bare <c>repeated</c>/<c>map</c> field instead would decode an
/// empty collection back as null with no test-visible difference at the object level.
/// </summary>
/// <typeparam name="T">Generated protobuf message type.</typeparam>
/// <param name="message">The message to send through an encode/decode cycle.</param>
/// <returns>An independent instance parsed from <paramref name="message"/>'s bytes.</returns>
private static T Wire<T>(T message) where T : IMessage<T>, new() =>
new MessageParser<T>(() => new T()).ParseFrom(message.ToByteArray());
private static ZB.MOM.WW.Audit.AuditEvent NewAuditEvent(string? sourceScript = null) =>
ScadaBridgeAuditEventFactory.Create(
channel: AuditChannel.ApiOutbound,
kind: AuditKind.ApiCallCached,
status: AuditStatus.Forwarded,
eventId: Guid.NewGuid(),
occurredAtUtc: new DateTime(2026, 7, 22, 9, 0, 0, DateTimeKind.Utc),
target: "ERP.GetOrder",
sourceSiteId: "site-a",
sourceNode: "node-a",
sourceScript: sourceScript);
private static SiteCall NewSiteCall() => new()
{
TrackedOperationId = TrackedOperationId.New(),
Channel = "ApiOutbound",
Target = "ERP.GetOrder",
SourceSite = "site-a",
SourceNode = "node-a",
Status = "Delivered",
RetryCount = 2,
LastError = "transient 503",
HttpStatus = 200,
CreatedAtUtc = new DateTime(2026, 7, 22, 8, 0, 0, DateTimeKind.Utc),
UpdatedAtUtc = new DateTime(2026, 7, 22, 8, 5, 0, DateTimeKind.Utc),
TerminalAtUtc = new DateTime(2026, 7, 22, 8, 10, 0, DateTimeKind.Utc),
IngestedAtUtc = new DateTime(2026, 7, 22, 8, 10, 1, DateTimeKind.Utc),
};
}
@@ -0,0 +1,181 @@
using Akka.Actor;
using Akka.TestKit;
using Akka.TestKit.Xunit2;
using Grpc.Core;
using Microsoft.Extensions.Logging.Abstractions;
using Microsoft.Extensions.Options;
using NSubstitute;
using ZB.MOM.WW.ScadaBridge.Commons.Messages.Audit;
using ZB.MOM.WW.ScadaBridge.Commons.Messages.Notification;
using ZB.MOM.WW.ScadaBridge.Communication;
using ZB.MOM.WW.ScadaBridge.Communication.Grpc;
namespace ZB.MOM.WW.ScadaBridge.Communication.Tests.Grpc;
/// <summary>
/// Behaviour of <see cref="CentralControlGrpcService"/> that does not need a real gRPC pipeline:
/// the readiness gate, the fire-and-forget heartbeat contract, the ingest-timeout status
/// mapping, and the shared audit-ingest timeout constant. The auth interceptor + real transport
/// are proven separately in <c>CentralControlEndToEndTests</c> (Host.Tests).
/// </summary>
public class CentralControlGrpcServiceTests : TestKit
{
private static ServerCallContext NewContext(CancellationToken ct = default)
{
var context = Substitute.For<ServerCallContext>();
context.CancellationToken.Returns(ct);
return context;
}
private CentralControlGrpcService CreateService(CommunicationOptions? options = null)
=> new(
NullLogger<CentralControlGrpcService>.Instance,
Options.Create(options ?? new CommunicationOptions()));
[Fact]
public async Task BeforeSetReady_AUnaryCall_IsUnavailable()
{
// Nothing was dispatched, so Unavailable is the right status — it tells a site transport
// the call never ran and a cross-node retry is safe.
var service = CreateService();
var ex = await Assert.ThrowsAsync<RpcException>(
() => service.SubmitNotification(new NotificationSubmitDto(), NewContext()));
Assert.Equal(StatusCode.Unavailable, ex.StatusCode);
}
[Fact]
public async Task BeforeSetReady_ANonEmptyIngestBatch_IsUnavailable()
{
var service = CreateService();
var batch = new AuditEventBatch();
batch.Events.Add(NewAuditDto());
var ex = await Assert.ThrowsAsync<RpcException>(
() => service.IngestAuditEvents(batch, NewContext()));
Assert.Equal(StatusCode.Unavailable, ex.StatusCode);
}
[Fact]
public async Task AfterSetReady_AUnaryCall_ReachesTheActor()
{
var stub = Sys.ActorOf(Props.Create(() => new StubActor()));
var service = CreateService();
service.SetReady(stub);
var ack = await service.SubmitNotification(NewNotificationDto(), NewContext());
Assert.True(ack.Accepted);
}
[Fact]
public async Task Heartbeat_BeforeSetReady_SucceedsAndIsDropped()
{
// Fire-and-forget end-to-end: a heartbeat must never fault the site's timer, so even
// with no actor wired the call returns OK rather than Unavailable.
var service = CreateService();
var reply = await service.Heartbeat(NewHeartbeatDto(), NewContext());
Assert.NotNull(reply);
}
[Fact]
public async Task Heartbeat_AfterSetReady_TellsTheActor_AndNeverAsks()
{
// A black-hole actor that never replies would hang an Ask forever; the heartbeat still
// returns immediately, proving it is a Tell, not an Ask.
var blackHole = Sys.ActorOf(Props.Create(() => new NeverRepliesActor()));
var service = CreateService();
service.SetReady(blackHole);
var reply = await service.Heartbeat(NewHeartbeatDto(), NewContext());
Assert.NotNull(reply);
}
[Fact]
public async Task WhenTheActorNeverReplies_TheCall_IsDeadlineExceeded_NotUnavailable()
{
// The message WAS delivered (Unavailable would wrongly invite a duplicate retry on the
// peer node); a timeout is DeadlineExceeded, which callers never cross-node-retry.
var blackHole = Sys.ActorOf(Props.Create(() => new NeverRepliesActor()));
var service = CreateService(new CommunicationOptions
{
NotificationForwardTimeout = TimeSpan.FromMilliseconds(200),
});
service.SetReady(blackHole);
var ex = await Assert.ThrowsAsync<RpcException>(
() => service.SubmitNotification(NewNotificationDto(), NewContext()));
Assert.Equal(StatusCode.DeadlineExceeded, ex.StatusCode);
}
[Fact]
public async Task EmptyIngestBatch_ShortCircuits_EvenBeforeSetReady()
{
// An empty batch is a no-op the actor need never see; it must not depend on readiness.
var service = CreateService();
var ack = await service.IngestAuditEvents(new AuditEventBatch(), NewContext());
Assert.Empty(ack.AcceptedEventIds);
}
[Fact]
public void TheAuditIngestTimeout_IsTheOneSharedConstant()
{
// The plan calls SiteStreamGrpcServer.AuditIngestAskTimeout "one source of truth" shared
// between the two audit-ingest transports; the central service must not re-declare 30s.
Assert.Equal(TimeSpan.FromSeconds(30), SiteStreamGrpcServer.AuditIngestAskTimeout);
}
// The mapper reads SiteEnqueuedAt unconditionally, so a DTO that reaches mapping must carry
// a timestamp. Only DTOs that get past the readiness/auth gate map, so the negative tests
// above can pass a bare DTO.
private static NotificationSubmitDto NewNotificationDto() => new()
{
NotificationId = Guid.NewGuid().ToString(),
ListName = "ops",
SiteEnqueuedAt = Google.Protobuf.WellKnownTypes.Timestamp.FromDateTime(
DateTime.SpecifyKind(new DateTime(2026, 5, 20, 10, 0, 0), DateTimeKind.Utc)),
};
private static HeartbeatDto NewHeartbeatDto() => new()
{
SiteId = "site-a",
NodeHostname = "node-a",
IsActive = true,
Timestamp = Google.Protobuf.WellKnownTypes.Timestamp.FromDateTime(
DateTime.SpecifyKind(new DateTime(2026, 5, 20, 10, 0, 0), DateTimeKind.Utc)),
};
private static AuditEventDto NewAuditDto() => new()
{
EventId = Guid.NewGuid().ToString(),
OccurredAtUtc = Google.Protobuf.WellKnownTypes.Timestamp.FromDateTime(
DateTime.SpecifyKind(new DateTime(2026, 5, 20, 10, 0, 0), DateTimeKind.Utc)),
Channel = "ApiOutbound",
Kind = "ApiCall",
Status = "Delivered",
SourceSiteId = "site-a",
};
private sealed class StubActor : ReceiveActor
{
public StubActor()
{
Receive<NotificationSubmit>(msg =>
Sender.Tell(new NotificationSubmitAck(msg.NotificationId, Accepted: true, Error: null)));
}
}
/// <summary>Swallows every message and never replies, so an Ask against it times out.</summary>
private sealed class NeverRepliesActor : ReceiveActor
{
public NeverRepliesActor() => ReceiveAny(_ => { });
}
}
@@ -0,0 +1,107 @@
using Microsoft.Extensions.Logging.Abstractions;
using Microsoft.Extensions.Options;
using ZB.MOM.WW.ScadaBridge.Commons.Messages.Artifacts;
using ZB.MOM.WW.ScadaBridge.Commons.Messages.DataConnection;
using ZB.MOM.WW.ScadaBridge.Commons.Messages.DebugView;
using ZB.MOM.WW.ScadaBridge.Commons.Messages.Deployment;
using ZB.MOM.WW.ScadaBridge.Commons.Messages.InboundApi;
using ZB.MOM.WW.ScadaBridge.Commons.Messages.Lifecycle;
using ZB.MOM.WW.ScadaBridge.Commons.Messages.Management;
using ZB.MOM.WW.ScadaBridge.Commons.Messages.RemoteQuery;
using ZB.MOM.WW.ScadaBridge.Commons.Types;
using ZB.MOM.WW.ScadaBridge.Communication.Grpc;
namespace ZB.MOM.WW.ScadaBridge.Communication.Tests;
/// <summary>
/// Pins <see cref="GrpcSiteTransport.ResolveDeadline"/> to the EXACT Ask timeout
/// <see cref="CommunicationService"/> uses for each command today, so flipping the transport cannot
/// change how long any call waits. Every timeout is given a distinct value so a wrong mapping is
/// caught, not masked by two defaults that happen to be equal (QueryTimeout == IntegrationTimeout in
/// production).
/// </summary>
public class GrpcSiteTransportDeadlineTests
{
private static readonly CommunicationOptions Opts = new()
{
DeploymentTimeout = TimeSpan.FromSeconds(120),
LifecycleTimeout = TimeSpan.FromSeconds(30),
ArtifactDeploymentTimeout = TimeSpan.FromSeconds(60),
QueryTimeout = TimeSpan.FromSeconds(31),
IntegrationTimeout = TimeSpan.FromSeconds(32),
DebugViewTimeout = TimeSpan.FromSeconds(10)
};
private static readonly GrpcSiteTransport Transport = new(
new SitePairChannelProvider(
new NoKeyProvider(), Options.Create(Opts), NullLogger<SitePairChannelProvider>.Instance),
Opts,
NullLogger<GrpcSiteTransport>.Instance);
private static readonly DateTimeOffset T = DateTimeOffset.UtcNow;
public static IEnumerable<object[]> Cases()
{
// Lifecycle group — note it is NOT one deadline class.
yield return [new RefreshDeploymentCommand("d", "i", "h", "by", T, "u", "t"), Opts.DeploymentTimeout];
yield return [new EnableInstanceCommand("c", "i", T), Opts.LifecycleTimeout];
yield return [new DisableInstanceCommand("c", "i", T), Opts.LifecycleTimeout];
yield return [new DeleteInstanceCommand("c", "i", T), Opts.LifecycleTimeout];
// DeploymentStateQuery uses QueryTimeout in CommunicationService, NOT LifecycleTimeout.
yield return [new DeploymentStateQueryRequest("c", "i", T), Opts.QueryTimeout];
yield return [new DeployArtifactsCommand("d", null, null, null, null, null, null, T), Opts.ArtifactDeploymentTimeout];
// OPC UA group — all QueryTimeout.
yield return [new BrowseNodeCommand("conn", null, null, null), Opts.QueryTimeout];
yield return [new SearchAddressSpaceCommand("conn", "q", 1, 1, null), Opts.QueryTimeout];
yield return [new ReadTagValuesCommand("conn", []), Opts.QueryTimeout];
yield return [new VerifyEndpointCommand("conn", "OpcUa", "{}", null), Opts.QueryTimeout];
yield return [new TrustServerCertCommand("conn", "ZGVy", "AA", null), Opts.QueryTimeout];
yield return [new ListServerCertsCommand(null), Opts.QueryTimeout];
yield return [new RemoveServerCertCommand("AA", null), Opts.QueryTimeout];
yield return [new WriteTagRequest("c", "conn", "tag", 1, T), Opts.QueryTimeout];
// Query group.
yield return [new EventLogQueryRequest("c", "s", null, null, null, null, null, null, null, 10, T), Opts.QueryTimeout];
yield return [new DebugSnapshotRequest("i", "c"), Opts.QueryTimeout];
yield return [new SubscribeDebugViewRequest("i", "c"), Opts.DebugViewTimeout];
yield return [new UnsubscribeDebugViewRequest("i", "c"), Opts.DebugViewTimeout];
// Parked group — the two relays keep QueryTimeout (30s) so SiteCallAudit's inner
// RelayTimeout (10s) still expires first.
yield return [new ParkedMessageQueryRequest("c", "s", 1, 10, T), Opts.QueryTimeout];
yield return [new ParkedMessageRetryRequest("c", "s", "m", T), Opts.QueryTimeout];
yield return [new ParkedMessageDiscardRequest("c", "s", "m", T), Opts.QueryTimeout];
yield return [new RetryParkedOperation("c", new TrackedOperationId(Guid.NewGuid())), Opts.QueryTimeout];
yield return [new DiscardParkedOperation("c", new TrackedOperationId(Guid.NewGuid())), Opts.QueryTimeout];
// Route group.
yield return [new RouteToCallRequest("c", "i", "m", null, T), Opts.IntegrationTimeout];
yield return [new RouteToGetAttributesRequest("c", "i", [], T), Opts.IntegrationTimeout];
yield return [new RouteToSetAttributesRequest("c", "i", new Dictionary<string, string>(), T), Opts.IntegrationTimeout];
// Failover — QueryTimeout in CommunicationService, NOT LifecycleTimeout.
yield return [new TriggerSiteFailover("c", "s"), Opts.QueryTimeout];
}
[Theory]
[MemberData(nameof(Cases))]
public void ResolveDeadline_MatchesTodaysAskTimeout(object command, TimeSpan expected)
{
Assert.Equal(expected, Transport.ResolveDeadline(command));
}
[Fact]
public void WaitForAttribute_UsesItsDynamicTimeoutPlusIntegrationSlack()
{
// CommunicationService.RouteToWaitForAttributeAsync uses request.Timeout + IntegrationTimeout.
var wait = new RouteToWaitForAttributeRequest("c", "i", "attr", "10", TimeSpan.FromSeconds(45), T);
Assert.Equal(TimeSpan.FromSeconds(45) + Opts.IntegrationTimeout, Transport.ResolveDeadline(wait));
}
private sealed class NoKeyProvider : ISitePskProvider
{
public ValueTask<string> GetAsync(string siteId, CancellationToken ct) => new("k");
public void Invalidate(string siteId) { }
}
}
@@ -0,0 +1,153 @@
using System.Text.Json;
using ZB.MOM.WW.ScadaBridge.Communication.Grpc;
namespace ZB.MOM.WW.ScadaBridge.Communication.Tests;
/// <summary>
/// Goldens for <see cref="LooseValueCodec"/> — the type-tagged carrier for the
/// <c>object?</c> members of the site command plane (script parameters and
/// return values, attribute values, tag reads/writes).
/// </summary>
/// <remarks>
/// The contract these tests pin down is that a boxed value keeps its RUNTIME
/// CLR TYPE across the wire, not merely its printed form. Today's Akka JSON
/// serializer preserves it, and an operator-facing tag value that silently
/// turns from <c>int</c> into <c>long</c> (or from <c>DateTime</c> into a
/// UTC-normalised copy) is a behaviour change, not a refactor.
/// </remarks>
public class LooseValueCodecTests
{
public static IEnumerable<object[]> TaggedScalars() =>
[
[true],
[false],
[42],
[-1],
[long.MaxValue],
[3.5d],
[1.5f],
["a string"],
[string.Empty],
[12.3456789m],
[new DateTime(2026, 7, 22, 1, 2, 3, DateTimeKind.Utc)],
[new DateTimeOffset(2026, 7, 22, 1, 2, 3, TimeSpan.FromHours(-5))],
[Guid.Parse("11111111-2222-3333-4444-555555555555")],
[TimeSpan.FromMinutes(90)],
[new byte[] { 1, 2, 3 }]
];
[Theory]
[MemberData(nameof(TaggedScalars))]
public void TaggedValues_KeepTheirClrType(object value)
{
var restored = LooseValueCodec.FromProto(LooseValueCodec.ToProto(value));
Assert.NotNull(restored);
Assert.Equal(value.GetType(), restored.GetType());
StructuralEquality.AssertDeepEqual(value, restored, value.GetType().Name);
}
/// <summary>
/// A local <see cref="DateTime"/> must come back local, and an offset
/// <see cref="DateTimeOffset"/> must come back with the same offset — the
/// reason these ride as round-trip strings instead of proto timestamps.
/// </summary>
[Fact]
public void DateValues_KeepKindAndOffset()
{
var local = new DateTime(2026, 7, 22, 1, 2, 3, DateTimeKind.Local);
var offset = new DateTimeOffset(2026, 7, 22, 1, 2, 3, TimeSpan.FromHours(5.5));
var restoredLocal = Assert.IsType<DateTime>(LooseValueCodec.FromProto(LooseValueCodec.ToProto(local)));
var restoredOffset =
Assert.IsType<DateTimeOffset>(LooseValueCodec.FromProto(LooseValueCodec.ToProto(offset)));
Assert.Equal(DateTimeKind.Local, restoredLocal.Kind);
Assert.Equal(local, restoredLocal);
Assert.Equal(TimeSpan.FromHours(5.5), restoredOffset.Offset);
}
/// <summary>An unset carrier and an explicit null marker both decode to null.</summary>
[Fact]
public void NullIsCarriedTwoWays()
{
Assert.Null(LooseValueCodec.FromProto(null));
Assert.Null(LooseValueCodec.FromProto(LooseValueCodec.ToProto(null)));
Assert.Null(LooseValueCodec.ToProtoOrNull(null));
}
/// <summary>
/// A dictionary entry whose value is null stays distinct from an absent key —
/// the reason <c>LooseNull</c> exists at all.
/// </summary>
[Fact]
public void NullMapValue_SurvivesAsAPresentKey()
{
var original = new Dictionary<string, object?> { ["present"] = null };
var restored = LooseValueCodec.FromProtoMap(LooseValueCodec.ToProtoMap(original));
Assert.True(restored.ContainsKey("present"));
Assert.Null(restored["present"]);
}
/// <summary>A null dictionary stays null; an empty one stays empty.</summary>
[Fact]
public void NullMap_StaysDistinctFromEmptyMap()
{
Assert.Null(LooseValueCodec.ToProtoMapOrNull(null));
Assert.Null(LooseValueCodec.FromProtoMapOrNull(null));
var empty = LooseValueCodec.FromProtoMapOrNull(
LooseValueCodec.ToProtoMapOrNull(new Dictionary<string, object?>()));
Assert.NotNull(empty);
Assert.Empty(empty);
}
/// <summary>Nested lists and maps recurse, so element values keep their types too.</summary>
[Fact]
public void NestedCollections_KeepElementTypes()
{
object original = new List<object?>
{
1,
"two",
null,
new Dictionary<string, object?> { ["deep"] = 3.5d }
};
var restored = LooseValueCodec.FromProto(LooseValueCodec.ToProto(original));
StructuralEquality.AssertDeepEqual(original, restored, "list");
}
/// <summary>
/// The documented widening: a typed collection keeps its ELEMENTS' types but
/// the container comes back as <c>List&lt;object?&gt;</c>. Recorded here so
/// the behaviour is a decision rather than a surprise.
/// </summary>
[Fact]
public void TypedList_WidensToObjectList()
{
var restored = LooseValueCodec.FromProto(LooseValueCodec.ToProto(new List<int> { 1, 2, 3 }));
var list = Assert.IsType<List<object?>>(restored);
Assert.Equal([1, 2, 3], list.Cast<int>());
}
/// <summary>
/// The documented lossy escape hatch: a value outside the tagged set keeps its
/// DATA but not its CLR identity — it decodes as a <see cref="JsonElement"/>.
/// Nothing on the command plane carries such a value today; the fallback
/// exists so an unexpected one cannot fault a command.
/// </summary>
[Fact]
public void UntaggedValue_FallsBackToJsonAndLosesItsClrType()
{
var restored = LooseValueCodec.FromProto(LooseValueCodec.ToProto((byte)7));
var element = Assert.IsType<JsonElement>(restored);
Assert.Equal("7", element.GetRawText());
}
}
@@ -0,0 +1,360 @@
using Akka.Actor;
using Akka.TestKit.Xunit2;
using ZB.MOM.WW.ScadaBridge.Commons.Messages.Artifacts;
using ZB.MOM.WW.ScadaBridge.Commons.Messages.Integration;
using ZB.MOM.WW.ScadaBridge.Commons.Messages.RemoteQuery;
using ZB.MOM.WW.ScadaBridge.Commons.Types;
using ZB.MOM.WW.ScadaBridge.Communication.Actors;
using ZB.MOM.WW.ScadaBridge.Communication.Grpc;
namespace ZB.MOM.WW.ScadaBridge.Communication.Tests;
/// <summary>
/// The single routing truth for the 28 migrated central→site commands. Proves each command
/// resolves to EXACTLY the target the actor used inline before the extraction — including the
/// node-local parked handler (never the singleton proxy) and the local failover path — so the
/// Akka actor and the gRPC service can share one table without drift.
/// </summary>
public class SiteCommandDispatcherTests : TestKit
{
private const string SiteId = "site1";
private SiteCommandDispatcher Build(IActorRef dmProxy, Func<string, bool, string?>? failover = null)
=> new(SiteId, dmProxy, failover ?? ((_, _) => null));
// ── The 19 commands that forward to the Deployment Manager singleton proxy ──
/// <summary>The proxy-routed commands (lifecycle, OPC UA, debug snapshot/subscribe, route).</summary>
public static IEnumerable<object[]> ProxyCommandTypes() => new[]
{
typeof(Commons.Messages.Deployment.RefreshDeploymentCommand),
typeof(Commons.Messages.Lifecycle.EnableInstanceCommand),
typeof(Commons.Messages.Lifecycle.DisableInstanceCommand),
typeof(Commons.Messages.Lifecycle.DeleteInstanceCommand),
typeof(Commons.Messages.Deployment.DeploymentStateQueryRequest),
typeof(Commons.Messages.Management.BrowseNodeCommand),
typeof(Commons.Messages.Management.SearchAddressSpaceCommand),
typeof(Commons.Messages.Management.ReadTagValuesCommand),
typeof(Commons.Messages.Management.VerifyEndpointCommand),
typeof(Commons.Messages.Management.TrustServerCertCommand),
typeof(Commons.Messages.Management.ListServerCertsCommand),
typeof(Commons.Messages.Management.RemoveServerCertCommand),
typeof(Commons.Messages.DataConnection.WriteTagRequest),
typeof(Commons.Messages.DebugView.DebugSnapshotRequest),
typeof(Commons.Messages.DebugView.SubscribeDebugViewRequest),
typeof(Commons.Messages.InboundApi.RouteToCallRequest),
typeof(Commons.Messages.InboundApi.RouteToGetAttributesRequest),
typeof(Commons.Messages.InboundApi.RouteToSetAttributesRequest),
typeof(Commons.Messages.InboundApi.RouteToWaitForAttributeRequest),
}.Select(t => new object[] { t });
[Theory]
[MemberData(nameof(ProxyCommandTypes))]
public void ProxyCommands_ForwardToDeploymentManager(Type commandType)
{
var dm = CreateTestProbe();
var dispatcher = Build(dm.Ref);
var command = SiteCommandSamples.All[commandType][0];
var route = dispatcher.ResolveRoute(command);
Assert.Equal(SiteCommandDispatcher.RouteDisposition.Forward, route.Disposition);
Assert.Same(dm.Ref, route.Target);
}
[Fact]
public void UnsubscribeDebugView_IsFireAndForget_ToDeploymentManager_WithSyntheticAck()
{
// Fire-and-forget: the Deployment Manager never acks an unsubscribe. The actor Forwards it;
// the gRPC transport Tells it and returns this synthetic ack so a unary RPC still answers.
var dm = CreateTestProbe();
var dispatcher = Build(dm.Ref);
var command = SiteCommandSamples.All[typeof(Commons.Messages.DebugView.UnsubscribeDebugViewRequest)][0];
var route = dispatcher.ResolveRoute(command);
Assert.Equal(SiteCommandDispatcher.RouteDisposition.TellFireAndForget, route.Disposition);
Assert.Same(dm.Ref, route.Target);
Assert.Same(UnsubscribeDebugViewAck.Instance, route.Reply);
}
// ── Artifact handler (null-guarded) ──
[Fact]
public void DeployArtifacts_WithHandler_ForwardsToArtifactHandler_NotTheProxy()
{
var dm = CreateTestProbe();
var artifact = CreateTestProbe();
var dispatcher = Build(dm.Ref);
dispatcher.RegisterArtifactHandler(artifact.Ref);
var route = dispatcher.ResolveRoute(SiteCommandSamples.All[typeof(DeployArtifactsCommand)][0]);
Assert.Equal(SiteCommandDispatcher.RouteDisposition.Forward, route.Disposition);
Assert.Same(artifact.Ref, route.Target);
Assert.NotSame(dm.Ref, route.Target);
}
[Fact]
public void DeployArtifacts_WithoutHandler_RepliesHandlerNotAvailable()
{
var dm = CreateTestProbe();
var dispatcher = Build(dm.Ref);
var command = (DeployArtifactsCommand)SiteCommandSamples.All[typeof(DeployArtifactsCommand)][0];
var route = dispatcher.ResolveRoute(command);
Assert.Equal(SiteCommandDispatcher.RouteDisposition.ImmediateReply, route.Disposition);
var reply = Assert.IsType<ArtifactDeploymentResponse>(route.Reply);
Assert.False(reply.Success);
Assert.Equal("Artifact handler not available", reply.ErrorMessage);
Assert.Equal(command.DeploymentId, reply.DeploymentId);
Assert.Equal(SiteId, reply.SiteId);
}
// ── Event-log handler (null-guarded) ──
[Fact]
public void EventLogQuery_WithHandler_ForwardsToEventLogHandler()
{
var dm = CreateTestProbe();
var eventLog = CreateTestProbe();
var dispatcher = Build(dm.Ref);
dispatcher.RegisterEventLogHandler(eventLog.Ref);
var route = dispatcher.ResolveRoute(SiteCommandSamples.All[typeof(EventLogQueryRequest)][0]);
Assert.Equal(SiteCommandDispatcher.RouteDisposition.Forward, route.Disposition);
Assert.Same(eventLog.Ref, route.Target);
}
[Fact]
public void EventLogQuery_WithoutHandler_RepliesHandlerNotAvailable()
{
var dm = CreateTestProbe();
var dispatcher = Build(dm.Ref);
var command = (EventLogQueryRequest)SiteCommandSamples.All[typeof(EventLogQueryRequest)][0];
var route = dispatcher.ResolveRoute(command);
Assert.Equal(SiteCommandDispatcher.RouteDisposition.ImmediateReply, route.Disposition);
var reply = Assert.IsType<EventLogQueryResponse>(route.Reply);
Assert.False(reply.Success);
Assert.Equal(command.CorrelationId, reply.CorrelationId);
}
// ── Parked handler (null-guarded) — stays NODE-LOCAL on purpose (replicated store) ──
/// <summary>The five parked commands, each of which routes to the per-node parked handler.</summary>
public static IEnumerable<object[]> ParkedCommandTypes() => new[]
{
typeof(ParkedMessageQueryRequest),
typeof(ParkedMessageRetryRequest),
typeof(ParkedMessageDiscardRequest),
typeof(RetryParkedOperation),
typeof(DiscardParkedOperation),
}.Select(t => new object[] { t });
[Theory]
[MemberData(nameof(ParkedCommandTypes))]
public void ParkedCommands_WithHandler_RouteToNodeLocalParkedHandler_NeverTheProxy(Type commandType)
{
// Node-locality proof: a parked retry/discard must run on the node holding the replicated
// store row, so it goes to the per-node parked handler — NEVER the singleton proxy. This is
// the constraint the extraction must not "fix".
var dm = CreateTestProbe();
var parked = CreateTestProbe();
var dispatcher = Build(dm.Ref);
dispatcher.RegisterParkedMessageHandler(parked.Ref);
var route = dispatcher.ResolveRoute(SiteCommandSamples.All[commandType][0]);
Assert.Equal(SiteCommandDispatcher.RouteDisposition.Forward, route.Disposition);
Assert.Same(parked.Ref, route.Target);
Assert.NotSame(dm.Ref, route.Target);
}
[Fact]
public void ParkedMessageQuery_WithoutHandler_RepliesHandlerNotAvailable()
{
var dispatcher = Build(CreateTestProbe().Ref);
var command = (ParkedMessageQueryRequest)SiteCommandSamples.All[typeof(ParkedMessageQueryRequest)][0];
var route = dispatcher.ResolveRoute(command);
var reply = Assert.IsType<ParkedMessageQueryResponse>(route.Reply);
Assert.False(reply.Success);
Assert.Equal(command.CorrelationId, reply.CorrelationId);
Assert.Equal(command.PageNumber, reply.PageNumber);
}
[Fact]
public void ParkedMessageRetry_WithoutHandler_RepliesHandlerNotAvailable()
{
var dispatcher = Build(CreateTestProbe().Ref);
var command = (ParkedMessageRetryRequest)SiteCommandSamples.All[typeof(ParkedMessageRetryRequest)][0];
var reply = Assert.IsType<ParkedMessageRetryResponse>(dispatcher.ResolveRoute(command).Reply);
Assert.False(reply.Success);
Assert.Equal(command.CorrelationId, reply.CorrelationId);
}
[Fact]
public void ParkedMessageDiscard_WithoutHandler_RepliesHandlerNotAvailable()
{
var dispatcher = Build(CreateTestProbe().Ref);
var command = (ParkedMessageDiscardRequest)SiteCommandSamples.All[typeof(ParkedMessageDiscardRequest)][0];
var reply = Assert.IsType<ParkedMessageDiscardResponse>(dispatcher.ResolveRoute(command).Reply);
Assert.False(reply.Success);
Assert.Equal(command.CorrelationId, reply.CorrelationId);
}
[Fact]
public void RetryParkedOperation_WithoutHandler_RepliesNotAppliedAck()
{
var dispatcher = Build(CreateTestProbe().Ref);
var command = new RetryParkedOperation("corr-x", TrackedOperationId.New());
var reply = Assert.IsType<ParkedOperationActionAck>(dispatcher.ResolveRoute(command).Reply);
Assert.False(reply.Applied);
Assert.Equal("corr-x", reply.CorrelationId);
Assert.NotNull(reply.ErrorMessage);
}
[Fact]
public void DiscardParkedOperation_WithoutHandler_RepliesNotAppliedAck()
{
var dispatcher = Build(CreateTestProbe().Ref);
var command = new DiscardParkedOperation("corr-y", TrackedOperationId.New());
var reply = Assert.IsType<ParkedOperationActionAck>(dispatcher.ResolveRoute(command).Reply);
Assert.False(reply.Applied);
Assert.Equal("corr-y", reply.CorrelationId);
}
// ── Commands that must NOT enter ResolveRoute ──
[Fact]
public void ResolveRoute_RejectsFailover_ItGoesThroughPrepareFailover()
{
var dispatcher = Build(CreateTestProbe().Ref);
Assert.Throws<ArgumentException>(() =>
dispatcher.ResolveRoute(new TriggerSiteFailover("c", SiteId)));
}
[Fact]
public void ResolveRoute_RejectsTheExcludedIntegrationCommand()
{
// IntegrationCallRequest is the 29th command, dead at both ends and deliberately excluded
// (28 of 29 migrate). It never enters the dispatcher.
var dispatcher = Build(CreateTestProbe().Ref);
var command = new IntegrationCallRequest(
"c", SiteId, "inst", "es", "m", new Dictionary<string, object?>(), DateTimeOffset.UtcNow);
Assert.Throws<ArgumentException>(() => dispatcher.ResolveRoute(command));
}
// ── Failover (local path) ──
[Fact]
public void HandleFailover_ResolvesAndLeaves_ThenAcksWithTheTarget()
{
string? roleAsked = null;
var leaveIssued = false;
Func<string, bool, string?> resolve = (role, dryRun) =>
{
roleAsked = role;
leaveIssued = !dryRun;
return "akka.tcp://scadabridge@site1-a:8082";
};
var dispatcher = Build(CreateTestProbe().Ref, resolve);
var ack = dispatcher.HandleFailover(new TriggerSiteFailover("corr-1", SiteId));
Assert.True(ack.Accepted);
Assert.Equal("corr-1", ack.CorrelationId);
Assert.Equal("akka.tcp://scadabridge@site1-a:8082", ack.TargetAddress);
Assert.Null(ack.ErrorMessage);
// Site singletons are scoped to the site-specific role.
Assert.Equal("site-site1", roleAsked);
// The actor path leaves in one step (dryRun:false).
Assert.True(leaveIssued);
}
[Fact]
public void HandleFailover_RefusesWhenThereIsNoPeer()
{
var dispatcher = Build(CreateTestProbe().Ref, (_, _) => null);
var ack = dispatcher.HandleFailover(new TriggerSiteFailover("corr-2", SiteId));
Assert.False(ack.Accepted);
Assert.Null(ack.TargetAddress);
Assert.NotNull(ack.ErrorMessage);
}
[Fact]
public void HandleFailover_RefusesACommandAddressedToAnotherSite_WithoutTouchingTheResolver()
{
var invoked = false;
var dispatcher = Build(CreateTestProbe().Ref, (_, _) => { invoked = true; return "addr"; });
var ack = dispatcher.HandleFailover(new TriggerSiteFailover("corr-3", "site2"));
Assert.False(ack.Accepted);
Assert.Contains("site2", ack.ErrorMessage);
Assert.False(invoked);
}
[Fact]
public void HandleFailover_FaultInTheLeave_IsReportedNotThrown()
{
var dispatcher = Build(CreateTestProbe().Ref,
(_, _) => throw new InvalidOperationException("cluster unavailable"));
var ack = dispatcher.HandleFailover(new TriggerSiteFailover("corr-4", SiteId));
Assert.False(ack.Accepted);
Assert.Contains("cluster unavailable", ack.ErrorMessage);
}
[Fact]
public void PrepareFailover_BuildsTheAckFromADryRun_WithoutLeaving_UntilCommitLeaveRuns()
{
// The gRPC ack-before-Leave proof at the routing-truth level: PrepareFailover resolves the
// standby with a DRY-RUN only (so the ack is built without leaving), and hands back a
// deferred CommitLeave. Only invoking CommitLeave — what the gRPC service does AFTER the
// ack is on the wire — performs the real leave.
var events = new List<string>();
Func<string, bool, string?> resolve = (_, dryRun) =>
{
events.Add(dryRun ? "resolve" : "leave");
return "akka.tcp://scadabridge@site1-a:8082";
};
var dispatcher = Build(CreateTestProbe().Ref, resolve);
var outcome = dispatcher.PrepareFailover(new TriggerSiteFailover("corr-5", SiteId));
Assert.True(outcome.Ack.Accepted);
Assert.Equal("akka.tcp://scadabridge@site1-a:8082", outcome.Ack.TargetAddress);
Assert.NotNull(outcome.CommitLeave);
// Building the ack did NOT leave.
Assert.Equal(new[] { "resolve" }, events);
outcome.CommitLeave!();
// The leave runs strictly after — the caller sends the ack first.
Assert.Equal(new[] { "resolve", "leave" }, events);
}
[Fact]
public void PrepareFailover_WhenRefused_HasNoDeferredLeave()
{
var dispatcher = Build(CreateTestProbe().Ref, (_, _) => null);
var outcome = dispatcher.PrepareFailover(new TriggerSiteFailover("corr-6", SiteId));
Assert.False(outcome.Ack.Accepted);
Assert.Null(outcome.CommitLeave);
}
}
@@ -0,0 +1,569 @@
using System.Reflection;
using Google.Protobuf;
using Google.Protobuf.Reflection;
using ZB.MOM.WW.ScadaBridge.Commons.Interfaces.Protocol;
using ZB.MOM.WW.ScadaBridge.Commons.Messages.DebugView;
using ZB.MOM.WW.ScadaBridge.Commons.Messages.InboundApi;
using ZB.MOM.WW.ScadaBridge.Commons.Messages.Management;
using ZB.MOM.WW.ScadaBridge.Commons.Messages.RemoteQuery;
using ZB.MOM.WW.ScadaBridge.Commons.Messages.Streaming;
using ZB.MOM.WW.ScadaBridge.Commons.Types.Alarms;
using ZB.MOM.WW.ScadaBridge.Commons.Types.Enums;
using ZB.MOM.WW.ScadaBridge.Communication.Grpc;
namespace ZB.MOM.WW.ScadaBridge.Communication.Tests;
/// <summary>
/// Round-trip goldens for <see cref="SiteCommandDtoMapper"/> — the 28 migrated
/// central→site commands, all 22 reply shapes, and every nested type they
/// carry, each proven to survive record → proto → record unchanged.
/// </summary>
/// <remarks>
/// <para>
/// The suite is REFLECTION-DRIVEN on purpose. Both the per-type theory and the
/// coverage guards enumerate the mapper's real surface (its <c>ToProto</c>
/// overloads) and the real proto contract (the generated <c>oneof</c>
/// descriptors), so adding a command without a golden fails the build rather
/// than quietly shipping an untested field. A hand-maintained list across 50
/// types would drift on the first change.
/// </para>
/// <para>
/// Where a round trip is not bit-exact, the mapper is the thing that gets fixed
/// — never the fixture. The two deliberate normalisations
/// (empty-string-means-null, and the implicit alarm <c>Condition</c>) are
/// asserted explicitly below rather than papered over.
/// </para>
/// </remarks>
public class SiteCommandDtoMapperGoldenTests
{
private static readonly IReadOnlyDictionary<string, Type> TypesByName =
SiteCommandSamples.All.Keys.ToDictionary(t => t.FullName!, t => t);
/// <summary>One theory case per (mapped type, golden sample).</summary>
/// <returns>Type name and sample index pairs covering every golden fixture.</returns>
public static IEnumerable<object[]> AllSamples() =>
SiteCommandSamples.All
.OrderBy(kv => kv.Key.FullName, StringComparer.Ordinal)
.SelectMany(kv => Enumerable.Range(0, kv.Value.Length)
.Select(i => new object[] { kv.Key.FullName!, i }));
[Theory]
[MemberData(nameof(AllSamples))]
public void RoundTrips_Unchanged(string typeName, int sampleIndex)
{
var type = TypesByName[typeName];
var original = SiteCommandSamples.All[type][sampleIndex];
var restored = RoundTrip(original, type);
StructuralEquality.AssertDeepEqual(original, restored, type.Name);
}
// ─────────────────────────────────────────────────────────────────────
// Coverage guards
// ─────────────────────────────────────────────────────────────────────
/// <summary>
/// Every non-enum type the mapper can project must have goldens. This is the
/// guard that makes the suite self-maintaining.
/// </summary>
[Fact]
public void EveryMappedType_HasGoldenSamples()
{
var missing = MappedRecordTypes()
.Where(t => !SiteCommandSamples.All.ContainsKey(t))
.Select(t => t.Name)
.OrderBy(n => n, StringComparer.Ordinal)
.ToList();
Assert.True(
missing.Count == 0,
"SiteCommandDtoMapper projects types with no golden sample: " + string.Join(", ", missing));
}
/// <summary>Every golden sample type must have at least two samples (populated + minimal).</summary>
[Fact]
public void EveryGoldenType_HasAtLeastOneSample()
{
var empty = SiteCommandSamples.All
.Where(kv => kv.Value.Length == 0)
.Select(kv => kv.Key.Name)
.ToList();
Assert.True(empty.Count == 0, "Golden types with no samples: " + string.Join(", ", empty));
}
/// <summary>
/// The contract carries exactly 28 commands — 29 on
/// <c>SiteCommunicationActor</c>'s receive table minus the dead
/// <c>IntegrationCallRequest</c>. If the site gains a command, this count
/// moves deliberately, not silently.
/// </summary>
[Fact]
public void CommandInventory_Is28_AcrossSixGroups()
{
var commands = CommandSampleTypes().ToList();
Assert.Equal(28, commands.Count);
Assert.Equal(
[6, 8, 4, 5, 4, 1],
new[]
{
SiteCommandGroup.Lifecycle, SiteCommandGroup.OpcUa, SiteCommandGroup.Query,
SiteCommandGroup.Parked, SiteCommandGroup.Route, SiteCommandGroup.Failover
}.Select(g => commands.Count(t => SiteCommandDtoMapper.GroupOf(Sample(t)) == g)).ToArray());
}
/// <summary>The contract carries 22 reply shapes across the same six groups.</summary>
[Fact]
public void ReplyInventory_Is22_AcrossSixGroups()
{
var replies = ReplyTypes().ToList();
Assert.Equal(22, replies.Count);
Assert.Equal(
[4, 6, 3, 4, 4, 1],
new[]
{
SiteCommandGroup.Lifecycle, SiteCommandGroup.OpcUa, SiteCommandGroup.Query,
SiteCommandGroup.Parked, SiteCommandGroup.Route, SiteCommandGroup.Failover
}.Select(g => replies.Count(t => SiteCommandDtoMapper.GroupOfReply(SampleReply(t)) == g)).ToArray());
}
/// <summary>
/// Packing every command golden must exercise EVERY <c>oneof</c> case
/// declared in the four multi-command request envelopes. A new proto case
/// with no producing command fails here.
/// </summary>
[Theory]
[MemberData(nameof(RequestEnvelopes))]
public void EveryRequestOneofCase_IsProducedByACommand(string envelopeName)
{
var produced = CommandSampleTypes()
.Select(t => PackCommand(Sample(t)))
.OfType<IMessage>()
.Where(m => m.Descriptor.Name == envelopeName)
.Select(OneofCaseName)
.ToHashSet(StringComparer.Ordinal);
var declared = DescriptorFor(envelopeName).Oneofs[0].Fields
.Select(f => f.PropertyName)
.ToHashSet(StringComparer.Ordinal);
Assert.Equal(declared.OrderBy(n => n, StringComparer.Ordinal), produced.OrderBy(n => n, StringComparer.Ordinal));
}
/// <summary>The reply-envelope mirror of <see cref="EveryRequestOneofCase_IsProducedByACommand"/>.</summary>
[Theory]
[MemberData(nameof(ReplyEnvelopes))]
public void EveryReplyOneofCase_IsProducedByAReply(string envelopeName)
{
var produced = ReplyTypes()
.Select(t => PackReply(SampleReply(t)))
.OfType<IMessage>()
.Where(m => m.Descriptor.Name == envelopeName)
.Select(OneofCaseName)
.ToHashSet(StringComparer.Ordinal);
var declared = DescriptorFor(envelopeName).Oneofs[0].Fields
.Select(f => f.PropertyName)
.ToHashSet(StringComparer.Ordinal);
Assert.Equal(declared.OrderBy(n => n, StringComparer.Ordinal), produced.OrderBy(n => n, StringComparer.Ordinal));
}
/// <summary>Names of the four multi-command request envelopes.</summary>
/// <returns>Envelope message names.</returns>
public static IEnumerable<object[]> RequestEnvelopes() =>
[["LifecycleRequest"], ["OpcUaRequest"], ["QueryRequest"], ["ParkedRequest"], ["RouteRequest"]];
/// <summary>Names of the four multi-reply reply envelopes.</summary>
/// <returns>Envelope message names.</returns>
public static IEnumerable<object[]> ReplyEnvelopes() =>
[["LifecycleReply"], ["OpcUaReply"], ["QueryReply"], ["ParkedReply"], ["RouteReply"]];
/// <summary>Every command golden survives the full envelope pack/unpack, not just its own message.</summary>
[Fact]
public void EveryCommandGolden_SurvivesItsEnvelope()
{
foreach (var (type, samples) in SiteCommandSamples.All)
{
if (!IsCommand(type))
{
continue;
}
foreach (var sample in samples)
{
var restored = UnpackCommand(PackCommand(sample));
StructuralEquality.AssertDeepEqual(sample, restored, $"{type.Name}(envelope)");
}
}
}
/// <summary>Every reply golden survives the full envelope pack/unpack.</summary>
[Fact]
public void EveryReplyGolden_SurvivesItsEnvelope()
{
foreach (var (type, samples) in SiteCommandSamples.All)
{
if (!IsReply(type))
{
continue;
}
foreach (var sample in samples)
{
var restored = UnpackReply(PackReply(sample));
StructuralEquality.AssertDeepEqual(sample, restored, $"{type.Name}(envelope)");
}
}
}
/// <summary>The fire-and-forget unsubscribe ack is a real envelope case, not a special path.</summary>
[Fact]
public void UnsubscribeDebugViewAck_RoundTripsThroughQueryReply()
{
var envelope = SiteCommandDtoMapper.ToQueryReply(UnsubscribeDebugViewAck.Instance);
Assert.Equal(QueryReply.ReplyOneofCase.UnsubscribeDebugView, envelope.ReplyCase);
Assert.Same(UnsubscribeDebugViewAck.Instance, SiteCommandDtoMapper.FromQueryReply(envelope));
}
// ─────────────────────────────────────────────────────────────────────
// Enum exhaustiveness
// ─────────────────────────────────────────────────────────────────────
[Theory]
[InlineData(DeploymentStatus.Pending)]
[InlineData(DeploymentStatus.InProgress)]
[InlineData(DeploymentStatus.Success)]
[InlineData(DeploymentStatus.Failed)]
public void DeploymentStatus_RoundTrips(DeploymentStatus value) =>
Assert.Equal(value, SiteCommandDtoMapper.FromProto(SiteCommandDtoMapper.ToProto(value)));
[Theory]
[InlineData(BrowseNodeClass.Object)]
[InlineData(BrowseNodeClass.Variable)]
[InlineData(BrowseNodeClass.Method)]
[InlineData(BrowseNodeClass.Other)]
public void BrowseNodeClass_RoundTrips(BrowseNodeClass value) =>
Assert.Equal(value, SiteCommandDtoMapper.FromProto(SiteCommandDtoMapper.ToProto(value)));
[Theory]
[InlineData(BrowseFailureKind.ConnectionNotFound)]
[InlineData(BrowseFailureKind.ConnectionNotConnected)]
[InlineData(BrowseFailureKind.NotBrowsable)]
[InlineData(BrowseFailureKind.Timeout)]
[InlineData(BrowseFailureKind.ServerError)]
public void BrowseFailureKind_RoundTrips(BrowseFailureKind value) =>
Assert.Equal(value, SiteCommandDtoMapper.FromProto(SiteCommandDtoMapper.ToProto(value)));
[Theory]
[InlineData(ReadTagValuesFailureKind.ConnectionNotFound)]
[InlineData(ReadTagValuesFailureKind.ConnectionNotConnected)]
[InlineData(ReadTagValuesFailureKind.Timeout)]
[InlineData(ReadTagValuesFailureKind.ServerError)]
public void ReadTagValuesFailureKind_RoundTrips(ReadTagValuesFailureKind value) =>
Assert.Equal(value, SiteCommandDtoMapper.FromProto(SiteCommandDtoMapper.ToProto(value)));
[Theory]
[InlineData(VerifyFailureKind.Unreachable)]
[InlineData(VerifyFailureKind.AuthFailed)]
[InlineData(VerifyFailureKind.UntrustedCertificate)]
[InlineData(VerifyFailureKind.Timeout)]
[InlineData(VerifyFailureKind.ServerError)]
public void VerifyFailureKind_RoundTrips(VerifyFailureKind value) =>
Assert.Equal(value, SiteCommandDtoMapper.FromProto(SiteCommandDtoMapper.ToProto(value)));
[Theory]
[InlineData(StoreAndForwardCategory.ExternalSystem)]
[InlineData(StoreAndForwardCategory.Notification)]
[InlineData(StoreAndForwardCategory.CachedDbWrite)]
public void StoreAndForwardCategory_RoundTrips(StoreAndForwardCategory value) =>
Assert.Equal(value, SiteCommandDtoMapper.FromProto(SiteCommandDtoMapper.ToProto(value)));
[Theory]
[InlineData(AlarmState.Active)]
[InlineData(AlarmState.Normal)]
public void AlarmState_RoundTrips(AlarmState value) =>
Assert.Equal(value, SiteCommandDtoMapper.FromProto(SiteCommandDtoMapper.ToProto(value)));
[Theory]
[InlineData(AlarmLevel.None)]
[InlineData(AlarmLevel.Low)]
[InlineData(AlarmLevel.LowLow)]
[InlineData(AlarmLevel.High)]
[InlineData(AlarmLevel.HighHigh)]
public void AlarmLevel_RoundTrips(AlarmLevel value) =>
Assert.Equal(value, SiteCommandDtoMapper.FromProto(SiteCommandDtoMapper.ToProto(value)));
[Theory]
[InlineData(AlarmKind.Computed)]
[InlineData(AlarmKind.NativeOpcUa)]
[InlineData(AlarmKind.NativeMxAccess)]
public void AlarmKind_RoundTrips(AlarmKind value) =>
Assert.Equal(value, SiteCommandDtoMapper.FromProto(SiteCommandDtoMapper.ToProto(value)));
[Theory]
[InlineData(AlarmShelveState.Unshelved)]
[InlineData(AlarmShelveState.OneShotShelved)]
[InlineData(AlarmShelveState.TimedShelved)]
[InlineData(AlarmShelveState.PermanentShelved)]
public void AlarmShelveState_RoundTrips(AlarmShelveState value) =>
Assert.Equal(value, SiteCommandDtoMapper.FromProto(SiteCommandDtoMapper.ToProto(value)));
/// <summary>
/// An unspecified wire enum — what a peer on an older contract sends — must
/// decode to the documented safe default instead of faulting the command.
/// </summary>
[Fact]
public void UnspecifiedWireEnums_DecodeToSafeDefaults()
{
Assert.Equal(DeploymentStatus.Failed, SiteCommandDtoMapper.FromProto(DeploymentStatusDto.Unspecified));
Assert.Equal(BrowseNodeClass.Other, SiteCommandDtoMapper.FromProto(BrowseNodeClassDto.Unspecified));
Assert.Equal(BrowseFailureKind.ServerError, SiteCommandDtoMapper.FromProto(BrowseFailureKindDto.Unspecified));
Assert.Equal(
ReadTagValuesFailureKind.ServerError,
SiteCommandDtoMapper.FromProto(ReadTagValuesFailureKindDto.Unspecified));
Assert.Equal(VerifyFailureKind.ServerError, SiteCommandDtoMapper.FromProto(VerifyFailureKindDto.Unspecified));
Assert.Equal(
StoreAndForwardCategory.ExternalSystem,
SiteCommandDtoMapper.FromProto(StoreAndForwardCategoryDto.Unspecified));
Assert.Equal(AlarmState.Normal, SiteCommandDtoMapper.FromProto(AlarmStateDto.Unspecified));
Assert.Equal(AlarmLevel.None, SiteCommandDtoMapper.FromProto(AlarmLevelDto.Unspecified));
Assert.Equal(AlarmKind.Computed, SiteCommandDtoMapper.FromProto(AlarmKindDto.Unspecified));
Assert.Equal(AlarmShelveState.Unshelved, SiteCommandDtoMapper.FromProto(AlarmShelveStateDto.Unspecified));
}
// ─────────────────────────────────────────────────────────────────────
// The two deliberate normalisations, asserted rather than hidden
// ─────────────────────────────────────────────────────────────────────
/// <summary>
/// Nullable strings ride as plain proto3 strings, so an empty one comes back
/// as null. Documented in the mapper; asserted here so it stays a choice.
/// </summary>
[Fact]
public void EmptyNullableString_NormalisesToNull()
{
var original = new SiteFailoverAck("corr", Accepted: false, TargetAddress: "", ErrorMessage: "");
var restored = SiteCommandDtoMapper.FromProto(SiteCommandDtoMapper.ToProto(original));
Assert.Null(restored.TargetAddress);
Assert.Null(restored.ErrorMessage);
}
/// <summary>
/// …but NOT for a wait target, where the empty string is a real value. This
/// is why that one field carries a <c>StringValue</c> wrapper.
/// </summary>
[Fact]
public void EmptyWaitTarget_StaysDistinctFromNull()
{
var empty = SiteCommandDtoMapper.FromProto(SiteCommandDtoMapper.ToProto(
new RouteToWaitForAttributeRequest("c", "i", "a", "", TimeSpan.FromSeconds(1), DateTimeOffset.UtcNow)));
var missing = SiteCommandDtoMapper.FromProto(SiteCommandDtoMapper.ToProto(
new RouteToWaitForAttributeRequest("c", "i", "a", null, TimeSpan.FromSeconds(1), DateTimeOffset.UtcNow)));
Assert.Equal(string.Empty, empty.TargetValueEncoded);
Assert.Null(missing.TargetValueEncoded);
}
/// <summary>
/// A computed alarm leaves <see cref="AlarmStateChanged.Condition"/> implicit
/// (the record derives it from State + Priority). The encoder omits a
/// condition that already equals that derived value, so the record comes back
/// byte-for-byte equal — including under record equality, which compares the
/// nullable backing field, not the property.
/// </summary>
[Fact]
public void ComputedAlarm_KeepsItsImplicitCondition()
{
var original = new AlarmStateChanged("inst", "alarm", AlarmState.Active, 500, DateTimeOffset.UtcNow);
var dto = SiteCommandDtoMapper.ToProto(original);
var restored = SiteCommandDtoMapper.FromProto(dto);
Assert.Null(dto.Condition);
Assert.Equal(original, restored);
}
/// <summary>
/// The flip side of the normalisation: a condition set EXPLICITLY to the value
/// the record would have derived comes back implicit. The
/// <see cref="AlarmStateChanged.Condition"/> value is identical — only the
/// record's private "was it set?" bit differs, which no consumer can observe.
/// </summary>
[Fact]
public void ExplicitConditionEqualToTheDerivedDefault_NormalisesToImplicit()
{
var original = new AlarmStateChanged("inst", "alarm", AlarmState.Normal, 100, DateTimeOffset.UtcNow)
{
Condition = AlarmConditionStateFactory.ForComputed(AlarmState.Normal, 100)
};
var restored = SiteCommandDtoMapper.FromProto(SiteCommandDtoMapper.ToProto(original));
Assert.Equal(original.Condition, restored.Condition);
}
/// <summary>A native alarm's explicit, non-derived condition is carried verbatim.</summary>
[Fact]
public void NativeAlarm_KeepsItsExplicitCondition()
{
var condition = new AlarmConditionState(true, false, true, AlarmShelveState.PermanentShelved, true, 999);
var original = new AlarmStateChanged("inst", "alarm", AlarmState.Active, 500, DateTimeOffset.UtcNow)
{
Kind = AlarmKind.NativeMxAccess,
Condition = condition
};
var dto = SiteCommandDtoMapper.ToProto(original);
var restored = SiteCommandDtoMapper.FromProto(dto);
Assert.NotNull(dto.Condition);
Assert.Equal(condition, restored.Condition);
}
// ─────────────────────────────────────────────────────────────────────
// Group classification + rejection
// ─────────────────────────────────────────────────────────────────────
/// <summary><c>IntegrationCallRequest</c> is excluded by design and must be rejected, not silently dropped.</summary>
[Fact]
public void IntegrationCallRequest_IsRejected()
{
var dead = new Commons.Messages.Integration.IntegrationCallRequest(
"corr", "site-a", "Instance", "System", "Method",
new Dictionary<string, object?>(), DateTimeOffset.UtcNow);
Assert.Throws<ArgumentException>(() => SiteCommandDtoMapper.GroupOf(dead));
}
/// <summary>Packing a command into the wrong group's envelope is a hard error, not a silent no-op.</summary>
[Fact]
public void PackingIntoTheWrongGroup_Throws() =>
Assert.Throws<ArgumentException>(() =>
SiteCommandDtoMapper.ToLifecycleRequest(new DebugSnapshotRequest("inst", "corr")));
/// <summary>An envelope with no oneof set (a newer peer's unknown case) surfaces as a clear failure.</summary>
[Fact]
public void UnsetOneof_ThrowsNotSupported() =>
Assert.Throws<NotSupportedException>(() => SiteCommandDtoMapper.FromLifecycleRequest(new LifecycleRequest()));
// ─────────────────────────────────────────────────────────────────────
// Reflection plumbing
// ─────────────────────────────────────────────────────────────────────
private static object RoundTrip(object original, Type type)
{
var toProto = typeof(SiteCommandDtoMapper)
.GetMethods(BindingFlags.Public | BindingFlags.Static)
.Single(m => m.Name == "ToProto" && m.GetParameters() is [{ } p] && p.ParameterType == type);
var wire = toProto.Invoke(null, [original])!;
var fromProto = typeof(SiteCommandDtoMapper)
.GetMethods(BindingFlags.Public | BindingFlags.Static)
.Single(m => m.Name == "FromProto"
&& m.GetParameters() is [{ } p]
&& p.ParameterType == wire.GetType());
return fromProto.Invoke(null, [wire])!;
}
private static IEnumerable<Type> MappedRecordTypes() =>
typeof(SiteCommandDtoMapper)
.GetMethods(BindingFlags.Public | BindingFlags.Static)
.Where(m => m.Name == "ToProto" && m.GetParameters().Length == 1)
.Select(m => m.GetParameters()[0].ParameterType)
.Where(t => !t.IsEnum)
.Distinct();
private static IEnumerable<Type> CommandSampleTypes() =>
SiteCommandSamples.All.Keys.Where(IsCommand);
/// <summary>
/// Reply shapes = the mapper's own classification, plus the synthetic
/// unsubscribe ack, which has no <c>ToProto</c> overload of its own.
/// </summary>
private static IEnumerable<Type> ReplyTypes() =>
SiteCommandSamples.All.Keys.Where(IsReply).Append(typeof(UnsubscribeDebugViewAck));
private static bool IsCommand(Type type) => Classifies(type, SiteCommandDtoMapper.GroupOf);
private static bool IsReply(Type type) => Classifies(type, SiteCommandDtoMapper.GroupOfReply);
private static bool Classifies(Type type, Func<object, SiteCommandGroup> classify)
{
try
{
classify(SiteCommandSamples.All[type][0]);
return true;
}
catch (ArgumentException)
{
return false;
}
}
private static object Sample(Type type) => SiteCommandSamples.All[type][0];
private static object SampleReply(Type type) =>
type == typeof(UnsubscribeDebugViewAck) ? UnsubscribeDebugViewAck.Instance : Sample(type);
private static object PackCommand(object command) => SiteCommandDtoMapper.GroupOf(command) switch
{
SiteCommandGroup.Lifecycle => SiteCommandDtoMapper.ToLifecycleRequest(command),
SiteCommandGroup.OpcUa => SiteCommandDtoMapper.ToOpcUaRequest(command),
SiteCommandGroup.Query => SiteCommandDtoMapper.ToQueryRequest(command),
SiteCommandGroup.Parked => SiteCommandDtoMapper.ToParkedRequest(command),
SiteCommandGroup.Route => SiteCommandDtoMapper.ToRouteRequest(command),
_ => SiteCommandDtoMapper.ToProto((TriggerSiteFailover)command)
};
private static object UnpackCommand(object envelope) => envelope switch
{
LifecycleRequest r => SiteCommandDtoMapper.FromLifecycleRequest(r),
OpcUaRequest r => SiteCommandDtoMapper.FromOpcUaRequest(r),
QueryRequest r => SiteCommandDtoMapper.FromQueryRequest(r),
ParkedRequest r => SiteCommandDtoMapper.FromParkedRequest(r),
RouteRequest r => SiteCommandDtoMapper.FromRouteRequest(r),
TriggerSiteFailoverDto d => SiteCommandDtoMapper.FromProto(d),
_ => throw new InvalidOperationException($"Unknown request envelope {envelope.GetType().Name}.")
};
private static object PackReply(object reply) => SiteCommandDtoMapper.GroupOfReply(reply) switch
{
SiteCommandGroup.Lifecycle => SiteCommandDtoMapper.ToLifecycleReply(reply),
SiteCommandGroup.OpcUa => SiteCommandDtoMapper.ToOpcUaReply(reply),
SiteCommandGroup.Query => SiteCommandDtoMapper.ToQueryReply(reply),
SiteCommandGroup.Parked => SiteCommandDtoMapper.ToParkedReply(reply),
SiteCommandGroup.Route => SiteCommandDtoMapper.ToRouteReply(reply),
_ => SiteCommandDtoMapper.ToProto((SiteFailoverAck)reply)
};
private static object UnpackReply(object envelope) => envelope switch
{
LifecycleReply r => SiteCommandDtoMapper.FromLifecycleReply(r),
OpcUaReply r => SiteCommandDtoMapper.FromOpcUaReply(r),
QueryReply r => SiteCommandDtoMapper.FromQueryReply(r),
ParkedReply r => SiteCommandDtoMapper.FromParkedReply(r),
RouteReply r => SiteCommandDtoMapper.FromRouteReply(r),
SiteFailoverAckDto d => SiteCommandDtoMapper.FromProto(d),
_ => throw new InvalidOperationException($"Unknown reply envelope {envelope.GetType().Name}.")
};
private static MessageDescriptor DescriptorFor(string name) =>
SiteCommandReflection.Descriptor.MessageTypes.Single(m => m.Name == name);
private static string OneofCaseName(IMessage envelope)
{
var oneof = envelope.Descriptor.Oneofs[0];
var field = oneof.Accessor.GetCaseFieldDescriptor(envelope);
Assert.NotNull(field);
return field.PropertyName;
}
}
@@ -0,0 +1,304 @@
using ZB.MOM.WW.ScadaBridge.Commons.Interfaces.Protocol;
using ZB.MOM.WW.ScadaBridge.Commons.Messages.Artifacts;
using ZB.MOM.WW.ScadaBridge.Commons.Messages.DataConnection;
using ZB.MOM.WW.ScadaBridge.Commons.Messages.DebugView;
using ZB.MOM.WW.ScadaBridge.Commons.Messages.Deployment;
using ZB.MOM.WW.ScadaBridge.Commons.Messages.InboundApi;
using ZB.MOM.WW.ScadaBridge.Commons.Messages.Lifecycle;
using ZB.MOM.WW.ScadaBridge.Commons.Messages.Management;
using ZB.MOM.WW.ScadaBridge.Commons.Messages.RemoteQuery;
using ZB.MOM.WW.ScadaBridge.Commons.Messages.Streaming;
using ZB.MOM.WW.ScadaBridge.Commons.Types;
using ZB.MOM.WW.ScadaBridge.Commons.Types.Alarms;
using ZB.MOM.WW.ScadaBridge.Commons.Types.Enums;
namespace ZB.MOM.WW.ScadaBridge.Communication.Tests;
/// <summary>
/// Golden fixtures for the site-command round-trip suite: for every command,
/// reply and nested type the mapper handles, at least one MAXIMALLY populated
/// sample and one MINIMAL sample (every nullable null, every optional
/// collection empty or absent).
/// </summary>
/// <remarks>
/// <para>
/// The two-sample rule is the point. A single "typical" sample proves only that
/// the happy path survives; the pairs are what catch a dropped optional field or
/// a null/empty collapse — the class of silent data loss the transport
/// round-trip guard exposed in PLAN-05 T8, which per-field unit tests had all
/// missed.
/// </para>
/// <para>
/// <see cref="SiteCommandDtoMapperGoldenTests"/> drives this table by
/// reflection and FAILS when the mapper gains a type that has no sample here, so
/// the coverage cannot silently drift as commands are added.
/// </para>
/// </remarks>
internal static class SiteCommandSamples
{
private static readonly DateTimeOffset T1 = new(2026, 7, 22, 13, 45, 12, 345, TimeSpan.Zero);
private static readonly DateTimeOffset T2 = new(2026, 1, 2, 3, 4, 5, TimeSpan.Zero);
private static readonly DateTime D1 = new(2026, 3, 4, 5, 6, 7, DateTimeKind.Utc);
private static readonly DateTime D2 = new(2027, 3, 4, 5, 6, 7, DateTimeKind.Utc);
private static readonly Guid G1 = Guid.Parse("11111111-2222-3333-4444-555555555555");
/// <summary>
/// Every CLR type the mapper round-trips, mapped to its golden samples.
/// Enums are excluded — they get their own exhaustive test.
/// </summary>
public static IReadOnlyDictionary<Type, object[]> All { get; } = Build();
private static Dictionary<Type, object[]> Build()
{
var samples = new Dictionary<Type, object[]>();
void Add<T>(params T[] values) where T : notnull =>
samples[typeof(T)] = values.Cast<object>().ToArray();
// ── Lifecycle commands ──
Add(new RefreshDeploymentCommand(
"dep-1", "Site1.Pump1", "hash-abc", "multi-role", T1, "http://central:5000", "tok-1"));
Add(new EnableInstanceCommand("cmd-1", "Site1.Pump1", T1));
Add(new DisableInstanceCommand("cmd-2", "Site1.Pump1", T2));
Add(new DeleteInstanceCommand("cmd-3", "Site1.Pump1", T1));
Add(new DeploymentStateQueryRequest("corr-1", "Site1.Pump1", T1));
Add(
// Full: every artifact collection populated.
new DeployArtifactsCommand(
"dep-2",
[SharedScript(), new SharedScriptArtifact("s2", "return 2;", null, null)],
[ExternalSystem()],
[new DatabaseConnectionArtifact("db", "Server=x;", 3, TimeSpan.FromSeconds(7.5))],
[new NotificationListArtifact("ops", ["a@x.com", "b@x.com"])],
[DataConnection()],
[Smtp()],
T1),
// Minimal: every collection NULL — must not come back as empty lists.
new DeployArtifactsCommand("dep-3", null, null, null, null, null, null, T2),
// Boundary: every collection present but EMPTY — must not come back null.
new DeployArtifactsCommand("dep-4", [], [], [], [], [], [], T1));
Add(SharedScript(), new SharedScriptArtifact("bare", "", null, null));
Add(ExternalSystem(), new ExternalSystemArtifact("bare", "http://x", "None", null, null, 0));
Add(new DatabaseConnectionArtifact("db", "Server=x;", 3, TimeSpan.FromMilliseconds(1500)),
new DatabaseConnectionArtifact("db2", "", 0, TimeSpan.Zero));
Add(new NotificationListArtifact("ops", ["a@x.com"]), new NotificationListArtifact("empty", []));
Add(DataConnection(), new DataConnectionArtifact("bare", "OpcUa", null, null, 0));
Add(Smtp(), new SmtpConfigurationArtifact("bare", "smtp", 25, "None", "f@x.com", null, null, null));
// ── Lifecycle replies ──
Add(new DeploymentStatusResponse("dep-1", "Site1.Pump1", DeploymentStatus.Success, null, T1),
new DeploymentStatusResponse("dep-1", "Site1.Pump1", DeploymentStatus.Failed, "boom", T2));
Add(new InstanceLifecycleResponse("cmd-1", "Site1.Pump1", true, null, T1),
new InstanceLifecycleResponse("cmd-1", "Site1.Pump1", false, "nope", T2));
Add(new DeploymentStateQueryResponse("corr-1", "Site1.Pump1", true, "dep-1", "hash-abc", T1),
new DeploymentStateQueryResponse("corr-1", "Site1.Pump1", false, null, null, T2));
Add(new ArtifactDeploymentResponse("dep-2", "site-a", true, null, T1),
new ArtifactDeploymentResponse("dep-2", "site-a", false, "handler missing", T2));
// ── OPC UA commands ──
Add(new BrowseNodeCommand("conn", "ns=2;s=Root", "cursor-1", "site-a"),
new BrowseNodeCommand("conn", null, null, null));
Add(new SearchAddressSpaceCommand("conn", "pump", 4, 200, "site-a"),
new SearchAddressSpaceCommand("conn", "", 0, 0, null));
Add(new ReadTagValuesCommand("conn", ["a", "b"]), new ReadTagValuesCommand("conn", []));
Add(new VerifyEndpointCommand("conn", "OpcUa", "{\"url\":\"opc.tcp://x\"}", "site-a"),
new VerifyEndpointCommand("conn", "OpcUa", "{}", null));
Add(new TrustServerCertCommand("conn", "ZGVy", "AABB", "site-a"),
new TrustServerCertCommand("conn", "ZGVy", "AABB", null));
Add(new ListServerCertsCommand("site-a"), new ListServerCertsCommand(null));
Add(new RemoveServerCertCommand("AABB", "site-a"), new RemoveServerCertCommand("AABB", null));
Add(new WriteTagRequest("corr-2", "conn", "ns=2;s=Speed", 42.5d, T1),
new WriteTagRequest("corr-2", "conn", "ns=2;s=Speed", null, T2),
new WriteTagRequest("corr-2", "conn", "ns=2;s=Flag", true, T1),
new WriteTagRequest("corr-2", "conn", "ns=2;s=Name", "manual", T1));
// ── OPC UA replies ──
Add(new BrowseNodeResult([Node(), NodeBare()], true, null, "cursor-2"),
new BrowseNodeResult([], false, new BrowseFailure(BrowseFailureKind.Timeout, "timed out"), null));
Add(Node(), NodeBare());
Add(new BrowseFailure(BrowseFailureKind.ConnectionNotConnected, "not connected"),
new BrowseFailure(BrowseFailureKind.ServerError, ""));
Add(new SearchAddressSpaceResult([new AddressSpaceMatch(Node(), "/Root/Pump1")], true, null),
new SearchAddressSpaceResult([], false, new BrowseFailure(BrowseFailureKind.NotBrowsable, "no")));
Add(new AddressSpaceMatch(Node(), "/Root/Pump1"), new AddressSpaceMatch(NodeBare(), ""));
Add(new ReadTagValuesResult([Outcome(), OutcomeFailed()], null),
new ReadTagValuesResult([], new ReadTagValuesFailure(ReadTagValuesFailureKind.ConnectionNotFound, "gone")));
Add(Outcome(), OutcomeFailed());
Add(new ReadTagValuesFailure(ReadTagValuesFailureKind.Timeout, "slow"),
new ReadTagValuesFailure(ReadTagValuesFailureKind.ServerError, ""));
Add(new VerifyEndpointResult(true, null, null, null),
new VerifyEndpointResult(false, VerifyFailureKind.UntrustedCertificate, "untrusted", Cert()),
new VerifyEndpointResult(false, VerifyFailureKind.Unreachable, "refused", null));
Add(Cert());
Add(new CertTrustResult(true, null, [Trusted(), TrustedRejected()]),
new CertTrustResult(true, null, null),
new CertTrustResult(false, "partial failure", []));
Add(Trusted(), TrustedRejected());
Add(new WriteTagResponse("corr-2", true, null, T1),
new WriteTagResponse("corr-2", false, "denied", T2));
// ── Query commands ──
Add(new EventLogQueryRequest(
"corr-3", "site-a", T1, T2, "Lifecycle", "Warning", "Site1.Pump1", "restart", "cursor-3", 50, T1),
new EventLogQueryRequest("corr-3", "site-a", null, null, null, null, null, null, null, 25, T2));
Add(new DebugSnapshotRequest("Site1.Pump1", "corr-4"));
Add(new SubscribeDebugViewRequest("Site1.Pump1", "corr-5"));
Add(new UnsubscribeDebugViewRequest("Site1.Pump1", "corr-6"));
// ── Query replies ──
Add(new EventLogQueryResponse("corr-3", "site-a", [Entry(), EntryBare()], "cursor-4", true, true, null, T1),
new EventLogQueryResponse("corr-3", "site-a", [], null, false, false, "handler missing", T2));
Add(Entry(), EntryBare());
Add(new DebugViewSnapshot("Site1.Pump1", [AttrValue(), AttrValueNull()], [ComputedAlarm(), NativeAlarm()], T1),
new DebugViewSnapshot("Site1.Pump1", [], [], T2, InstanceNotFound: true));
Add(AttrValue(), AttrValueNull(), AttrValueList());
Add(ComputedAlarm(), NativeAlarm());
Add(new AlarmConditionState(true, false, true, AlarmShelveState.TimedShelved, true, 900),
new AlarmConditionState(false, true, null, AlarmShelveState.Unshelved, false, 0));
// ── Parked commands ──
Add(new ParkedMessageQueryRequest("corr-7", "site-a", 2, 25, T1));
Add(new ParkedMessageRetryRequest("corr-8", "site-a", "msg-1", T1));
Add(new ParkedMessageDiscardRequest("corr-9", "site-a", "msg-1", T2));
Add(new RetryParkedOperation("corr-10", new TrackedOperationId(G1)),
new RetryParkedOperation("corr-10", default));
Add(new DiscardParkedOperation("corr-11", new TrackedOperationId(G1)),
new DiscardParkedOperation("corr-11", default));
// ── Parked replies ──
Add(new ParkedMessageQueryResponse("corr-7", "site-a", [Parked(), ParkedBare()], 2, 1, 25, true, null, T1),
new ParkedMessageQueryResponse("corr-7", "site-a", [], 0, 1, 25, false, "handler missing", T2));
Add(Parked(), ParkedBare());
Add(new ParkedMessageRetryResponse("corr-8", true),
new ParkedMessageRetryResponse("corr-8", false, "not parked"));
Add(new ParkedMessageDiscardResponse("corr-9", true),
new ParkedMessageDiscardResponse("corr-9", false, "not parked"));
Add(new ParkedOperationActionAck("corr-10", true),
new ParkedOperationActionAck("corr-10", false, "handler missing"));
// ── Route commands ──
Add(new RouteToCallRequest("corr-12", "Site1.Pump1", "Start", Parameters(), T1, G1),
new RouteToCallRequest("corr-12", "Site1.Pump1", "Start", null, T2),
new RouteToCallRequest("corr-12", "Site1.Pump1", "Start", new Dictionary<string, object?>(), T1));
Add(new RouteToGetAttributesRequest("corr-13", "Site1.Pump1", ["Speed", "Flow"], T1, G1),
new RouteToGetAttributesRequest("corr-13", "Site1.Pump1", [], T2));
Add(new RouteToSetAttributesRequest(
"corr-14", "Site1.Pump1", new Dictionary<string, string> { ["Speed"] = "10" }, T1, G1),
new RouteToSetAttributesRequest("corr-14", "Site1.Pump1", new Dictionary<string, string>(), T2));
Add(new RouteToWaitForAttributeRequest(
"corr-15", "Site1.Pump1", "Speed", "10", TimeSpan.FromSeconds(30), T1, G1, true),
new RouteToWaitForAttributeRequest(
"corr-15", "Site1.Pump1", "Speed", null, TimeSpan.Zero, T2),
// "" is a legitimate wait target and must NOT collapse to null.
new RouteToWaitForAttributeRequest(
"corr-15", "Site1.Pump1", "Speed", "", TimeSpan.FromMinutes(1), T1));
// ── Route replies ──
Add(new RouteToCallResponse("corr-12", true, 17L, null, T1),
new RouteToCallResponse("corr-12", false, null, "script faulted", T2));
Add(new RouteToGetAttributesResponse("corr-13", Parameters(), true, null, T1),
new RouteToGetAttributesResponse("corr-13", new Dictionary<string, object?>(), false, "no instance", T2));
Add(new RouteToSetAttributesResponse("corr-14", true, null, T1),
new RouteToSetAttributesResponse("corr-14", false, "locked", T2));
Add(new RouteToWaitForAttributeResponse("corr-15", true, 10, "Good", false, true, null, T1),
new RouteToWaitForAttributeResponse("corr-15", false, null, null, true, true, null, T2));
// ── Failover ──
Add(new TriggerSiteFailover("corr-16", "site-a"));
Add(new SiteFailoverAck("corr-16", true, "akka.tcp://scadabridge@node-b:8081", null),
new SiteFailoverAck("corr-16", false, null, "no standby available"));
return samples;
}
private static SharedScriptArtifact SharedScript() =>
new("Calc", "return 1;", "{\"p\":\"int\"}", "{\"r\":\"int\"}");
private static ExternalSystemArtifact ExternalSystem() =>
new("Mes", "https://mes/api", "ApiKey", "{\"key\":\"x\"}", "[{\"name\":\"Post\"}]", 45);
private static DataConnectionArtifact DataConnection() =>
new("Plc1", "OpcUa", "{\"url\":\"opc.tcp://a\"}", "{\"url\":\"opc.tcp://b\"}", 5);
private static SmtpConfigurationArtifact Smtp() =>
new("Default", "smtp.host", 587, "Basic", "from@x.com", "user", "secret", "{\"tenant\":\"t\"}");
private static BrowseNode Node() => new("ns=2;s=Pump1.Speed", "Speed", BrowseNodeClass.Variable, false, "Double", -1, true);
private static BrowseNode NodeBare() => new("ns=2;s=Root", "Root", BrowseNodeClass.Object, true);
private static TagReadOutcome Outcome() => new("ns=2;s=Speed", true, 12.5d, "Good", T1, null);
private static TagReadOutcome OutcomeFailed() => new("ns=2;s=Bad", false, null, "Bad", T2, "no such node");
private static ServerCertInfo Cert() => new("AABB", "CN=server", "CN=issuer", D1, D2, "ZGVy");
private static TrustedCertInfo Trusted() => new("AABB", "CN=server", "CN=issuer", D1, D2, false);
private static TrustedCertInfo TrustedRejected() => new("CCDD", "CN=other", "CN=issuer", D1, D2, true);
private static EventLogEntry Entry() =>
new(G1.ToString("D"), T1, "Lifecycle", "Warning", "Site1.Pump1", "InstanceActor", "restarted", "{\"n\":1}");
private static EventLogEntry EntryBare() =>
new(Guid.Empty.ToString("D"), T2, "System", "Info", null, "Host", "started", null);
private static ParkedMessageEntry Parked() =>
new("msg-1", "Mes", "Post", "500 from server", 7, T1, T2, 10, StoreAndForwardCategory.CachedDbWrite, "Site1.Pump1");
private static ParkedMessageEntry ParkedBare() =>
new("msg-2", "Mes", "Post", "", 0, T2, T2);
private static AttributeValueChanged AttrValue() =>
new("Site1.Pump1", "Pump1.Speed", "Speed", 12.5d, "Good", T1);
private static AttributeValueChanged AttrValueNull() =>
new("Site1.Pump1", "Pump1.Speed", "Speed", null, "Bad", T2);
private static AttributeValueChanged AttrValueList() =>
new("Site1.Pump1", "Pump1.Trend", "Trend", new List<object?> { 1, 2.5d, "x", null }, "Good", T1);
/// <summary>A computed alarm: <c>Condition</c> left implicit, which is the common case.</summary>
private static AlarmStateChanged ComputedAlarm() =>
new("Site1.Pump1", "HighSpeed", AlarmState.Active, 700, T1)
{
Level = AlarmLevel.HighHigh,
Message = "Speed critically high"
};
/// <summary>A mirrored native alarm: every native enrichment field populated, <c>Condition</c> explicit.</summary>
private static AlarmStateChanged NativeAlarm() =>
new("Site1.Pump1", "Tank01.Level.HiHi", AlarmState.Active, 850, T2)
{
Level = AlarmLevel.None,
Message = "Level high",
Kind = AlarmKind.NativeOpcUa,
Condition = new AlarmConditionState(true, false, false, AlarmShelveState.OneShotShelved, true, 850),
SourceReference = "Tank01.Level.HiHi",
AlarmTypeName = "AnalogLimitAlarm.HiHi",
Category = "Process",
OperatorUser = "multi-role",
OperatorComment = "ack'd at panel",
OriginalRaiseTime = T1,
CurrentValue = "91.2",
LimitValue = "90.0",
NativeSourceCanonicalName = "Tank01.TankAlarms",
IsConfiguredPlaceholder = true
};
private static Dictionary<string, object?> Parameters() => new()
{
["count"] = 3,
["ratio"] = 1.25d,
["name"] = "batch-1",
["enabled"] = true,
["missing"] = null,
["when"] = T1,
["id"] = G1,
["items"] = new List<object?> { 1L, "two", null },
["nested"] = new Dictionary<string, object?> { ["inner"] = 5f }
};
}
@@ -0,0 +1,157 @@
using System.Collections;
using System.Reflection;
using System.Text.Json;
namespace ZB.MOM.WW.ScadaBridge.Communication.Tests;
/// <summary>
/// Structural deep-equality assertion for the site-command round-trip goldens.
/// </summary>
/// <remarks>
/// <para>
/// Record equality is NOT usable here. A C# record's generated <c>Equals</c>
/// compares members with <c>EqualityComparer&lt;T&gt;.Default</c>, which for
/// <c>IReadOnlyList&lt;T&gt;</c> / <c>IReadOnlyDictionary&lt;,&gt;</c> members
/// degrades to reference equality — so <c>Assert.Equal(dto, roundTripped)</c>
/// would fail on every collection-bearing message even when the mapper is
/// perfect, and (worse) would pass vacuously nowhere useful. This walker
/// compares by shape instead: dictionaries by key, sequences element-wise, and
/// everything else property-by-property down to leaf values.
/// </para>
/// <para>
/// The failure message carries the full property path, so a dropped field
/// reports as e.g. <c>DeployArtifactsCommand.ExternalSystems[0].TimeoutSeconds</c>
/// rather than an opaque "objects differ".
/// </para>
/// </remarks>
internal static class StructuralEquality
{
/// <summary>Asserts two object graphs are structurally identical.</summary>
/// <param name="expected">The original value.</param>
/// <param name="actual">The value that came back through the mapper.</param>
/// <param name="rootName">Name used as the root of the reported property path.</param>
public static void AssertDeepEqual(object? expected, object? actual, string rootName)
=> Compare(expected, actual, rootName);
private static void Compare(object? expected, object? actual, string path)
{
if (expected is null && actual is null)
{
return;
}
if (expected is null || actual is null)
{
Assert.Fail($"{path}: expected {Describe(expected)} but got {Describe(actual)}.");
return;
}
// JsonElement is the documented lossy escape hatch of LooseValueCodec; it
// has no useful Equals, so compare the canonical JSON text.
if (expected is JsonElement expectedJson && actual is JsonElement actualJson)
{
Assert.Equal(expectedJson.GetRawText(), actualJson.GetRawText());
return;
}
// Collections are compared by CONTENT, not by concrete container type.
// Members are declared IReadOnlyList<T>/IReadOnlyDictionary<,>, so the
// backing type (a collection-expression array here, a List<T> out of the
// mapper) is not part of the contract. Element and value types below are
// still compared strictly.
if (expected is not string && actual is not string)
{
if (expected is IDictionary expectedMap && actual is IDictionary actualMap)
{
CompareDictionaries(expectedMap, actualMap, path);
return;
}
if (expected is IEnumerable expectedSeq && actual is IEnumerable actualSeq)
{
CompareSequences(expectedSeq, actualSeq, path);
return;
}
}
var expectedType = expected.GetType();
var actualType = actual.GetType();
if (expectedType != actualType)
{
Assert.Fail(
$"{path}: type changed across the round trip — expected {expectedType.Name}, got {actualType.Name}.");
return;
}
if (IsLeaf(expectedType))
{
Assert.True(
Equals(expected, actual),
$"{path}: expected '{expected}' but got '{actual}'.");
return;
}
CompareProperties(expected, actual, expectedType, path);
}
private static void CompareDictionaries(IDictionary expected, IDictionary actual, string path)
{
Assert.True(
expected.Count == actual.Count,
$"{path}: entry count changed — expected {expected.Count}, got {actual.Count}.");
foreach (DictionaryEntry entry in expected)
{
Assert.True(
actual.Contains(entry.Key),
$"{path}: key '{entry.Key}' is missing after the round trip.");
Compare(entry.Value, actual[entry.Key], $"{path}['{entry.Key}']");
}
}
private static void CompareSequences(IEnumerable expected, IEnumerable actual, string path)
{
var expectedItems = expected.Cast<object?>().ToList();
var actualItems = actual.Cast<object?>().ToList();
Assert.True(
expectedItems.Count == actualItems.Count,
$"{path}: element count changed — expected {expectedItems.Count}, got {actualItems.Count}.");
for (var i = 0; i < expectedItems.Count; i++)
{
Compare(expectedItems[i], actualItems[i], $"{path}[{i}]");
}
}
private static void CompareProperties(object expected, object actual, Type type, string path)
{
var properties = type
.GetProperties(BindingFlags.Public | BindingFlags.Instance)
.Where(p => p.CanRead && p.GetIndexParameters().Length == 0)
.ToList();
Assert.True(properties.Count > 0, $"{path}: {type.Name} exposes no readable properties to compare.");
foreach (var property in properties)
{
Compare(property.GetValue(expected), property.GetValue(actual), $"{path}.{property.Name}");
}
}
private static bool IsLeaf(Type type) =>
type.IsPrimitive
|| type.IsEnum
|| type == typeof(string)
|| type == typeof(decimal)
|| type == typeof(DateTime)
|| type == typeof(DateTimeOffset)
|| type == typeof(TimeSpan)
|| type == typeof(Guid)
// Value types with no collection members (e.g. TrackedOperationId) have a
// correct structural Equals of their own.
|| (type.IsValueType && !type.IsGenericType);
private static string Describe(object? value) => value is null ? "<null>" : $"{value.GetType().Name}('{value}')";
}
@@ -0,0 +1,197 @@
using Grpc.Core;
using Microsoft.Extensions.Logging.Abstractions;
using ZB.MOM.WW.ScadaBridge.Communication.Grpc;
namespace ZB.MOM.WW.ScadaBridge.Host.Tests;
/// <summary>
/// Central's inbound gate for the site→central gRPC control plane (T1A.2). The mirror of
/// <see cref="ControlPlaneAuthInterceptorTests"/>, but central's verification model is inverted:
/// it looks up the key for the site named in the <c>x-scadabridge-site</c> header and checks the
/// bearer token against THAT, rather than against one own-key.
/// </summary>
/// <remarks>
/// Central genuinely needs a distinct verification model, so it is a separate class with its own
/// single public constructor — not a variant ctor on <see cref="ControlPlaneAuthInterceptor"/>,
/// whose two-public-constructor form once silently disabled the gate. The one-public-ctor
/// invariant is pinned below exactly as the sibling pins it.
/// </remarks>
public class CentralControlAuthInterceptorTests
{
private const string ControlMethod =
"/scadabridge.centralcontrol.v1.CentralControlService/SubmitNotification";
private const string SiteStreamMethod = "/sitestream.SiteStreamService/SubscribeInstance";
private const string SiteA = "site-a";
private const string SiteAKey = "site-a-preshared-key";
private const string SiteB = "site-b";
private const string SiteBKey = "site-b-preshared-key";
/// <summary>An <see cref="ISitePskProvider"/> backed by a fixed site→key map; throws for unknown sites.</summary>
private sealed class MapPskProvider(IReadOnlyDictionary<string, string> keys) : ISitePskProvider
{
public ValueTask<string> GetAsync(string siteId, CancellationToken ct)
=> keys.TryGetValue(siteId, out var key)
? new ValueTask<string>(key)
: throw new InvalidOperationException($"no key for '{siteId}'");
public void Invalidate(string siteId) { }
}
private static CentralControlAuthInterceptor CreateInterceptor()
=> new(
new MapPskProvider(new Dictionary<string, string> { [SiteA] = SiteAKey, [SiteB] = SiteBKey }),
NullLogger<CentralControlAuthInterceptor>.Instance);
private static ServerCallContext CreateContext(
string method, string? siteHeader, string? authorizationHeader)
{
var headers = new Metadata();
if (siteHeader is not null)
headers.Add(ControlPlaneCredentials.SiteHeader, siteHeader);
if (authorizationHeader is not null)
headers.Add(ControlPlaneCredentials.AuthorizationHeader, authorizationHeader);
return new FakeServerCallContext(method, headers);
}
private sealed class FakeServerCallContext(string method, Metadata requestHeaders)
: ServerCallContext
{
protected override string MethodCore => method;
protected override string HostCore => "localhost";
protected override string PeerCore => "ipv4:127.0.0.1:12345";
protected override DateTime DeadlineCore => DateTime.UtcNow.AddMinutes(1);
protected override Metadata RequestHeadersCore => requestHeaders;
protected override CancellationToken CancellationTokenCore => CancellationToken.None;
protected override Metadata ResponseTrailersCore { get; } = [];
protected override Status StatusCore { get; set; }
protected override WriteOptions? WriteOptionsCore { get; set; }
protected override AuthContext AuthContextCore { get; } =
new(null, new Dictionary<string, List<AuthProperty>>());
protected override ContextPropagationToken CreatePropagationTokenCore(
ContextPropagationOptions? options)
=> throw new NotSupportedException();
protected override Task WriteResponseHeadersAsyncCore(Metadata responseHeaders)
=> Task.CompletedTask;
}
private static Task<string> Invoke(
CentralControlAuthInterceptor interceptor, ServerCallContext context)
=> interceptor.UnaryServerHandler<string, string>(
"request", context, (_, _) => Task.FromResult("ok"));
[Fact]
public async Task NonGatedMethod_PassesThrough()
{
// Central's interceptor gates only CentralControlService; anything else on the listener
// (there is nothing today) is not its concern.
var interceptor = CreateInterceptor();
var context = CreateContext(SiteStreamMethod, siteHeader: null, authorizationHeader: null);
Assert.Equal("ok", await Invoke(interceptor, context));
}
[Fact]
public async Task CorrectSiteAndKey_IsAccepted()
{
var interceptor = CreateInterceptor();
var context = CreateContext(ControlMethod, SiteA, $"Bearer {SiteAKey}");
Assert.Equal("ok", await Invoke(interceptor, context));
}
[Fact]
public async Task MissingSiteHeader_IsDenied()
{
// Fail-closed: no header means no per-site key to verify against, so there is nothing to
// pass through TO. A present bearer token does not rescue it.
var interceptor = CreateInterceptor();
var context = CreateContext(ControlMethod, siteHeader: null, $"Bearer {SiteAKey}");
var ex = await Assert.ThrowsAsync<RpcException>(() => Invoke(interceptor, context));
Assert.Equal(StatusCode.PermissionDenied, ex.StatusCode);
}
[Fact]
public async Task BlankSiteHeader_IsDenied()
{
var interceptor = CreateInterceptor();
var context = CreateContext(ControlMethod, siteHeader: " ", $"Bearer {SiteAKey}");
var ex = await Assert.ThrowsAsync<RpcException>(() => Invoke(interceptor, context));
Assert.Equal(StatusCode.PermissionDenied, ex.StatusCode);
}
[Fact]
public async Task UnknownSite_IsDenied_NotPassedThrough()
{
// The provider throws for an unknown site; the interceptor must turn that into a denial,
// never swallow it and let the call proceed unauthenticated.
var interceptor = CreateInterceptor();
var context = CreateContext(ControlMethod, "site-nonexistent", "Bearer whatever");
var ex = await Assert.ThrowsAsync<RpcException>(() => Invoke(interceptor, context));
Assert.Equal(StatusCode.PermissionDenied, ex.StatusCode);
}
[Fact]
public async Task NoBearerToken_IsDenied()
{
var interceptor = CreateInterceptor();
var context = CreateContext(ControlMethod, SiteA, authorizationHeader: null);
var ex = await Assert.ThrowsAsync<RpcException>(() => Invoke(interceptor, context));
Assert.Equal(StatusCode.PermissionDenied, ex.StatusCode);
}
[Fact]
public async Task WrongKeyForTheSite_IsDenied()
{
// site-a presents site-b's key. Both keys are valid keys; the point is the token must
// match the key for THIS site, not just be a key central knows.
var interceptor = CreateInterceptor();
var context = CreateContext(ControlMethod, SiteA, $"Bearer {SiteBKey}");
var ex = await Assert.ThrowsAsync<RpcException>(() => Invoke(interceptor, context));
Assert.Equal(StatusCode.PermissionDenied, ex.StatusCode);
}
[Fact]
public async Task EachSiteIsVerifiedAgainstItsOwnKey()
{
// site-b with site-b's key is accepted by the same interceptor instance that rejected
// site-a-with-site-b's-key above — proving the per-site lookup, not a single shared key.
var interceptor = CreateInterceptor();
var context = CreateContext(ControlMethod, SiteB, $"Bearer {SiteBKey}");
Assert.Equal("ok", await Invoke(interceptor, context));
}
[Fact]
public void TheInterceptorHasExactlyOnePublicConstructor()
{
// Grpc.AspNetCore registers this interceptor BY TYPE; InterceptorRegistration.GetFactory()
// throws "Multiple constructors accepting all given argument types have been found" the
// moment a second public constructor is applicable, and the throw lands inside the
// pipeline on every call — a gate that authorizes nothing while looking like a handler
// bug. The explicit-prefix constructor is internal to keep it from recurring; this pins
// that. (Same invariant as ControlPlaneAuthInterceptorTests.)
var publicCtors = typeof(CentralControlAuthInterceptor).GetConstructors();
Assert.Single(publicCtors);
}
[Fact]
public void DefaultGatedPrefixes_MatchTheRealCentralControlServicePath()
{
// A typo here disables the whole gate silently: every call would pass through. Pin it
// against the generated service descriptor, not the proto text.
var method = CentralControlService.Descriptor.FullName;
Assert.Contains(
CentralControlAuthInterceptor.DefaultGatedPrefixes,
p => p == $"/{method}/");
}
}
@@ -0,0 +1,256 @@
using Akka.Actor;
using Google.Protobuf.WellKnownTypes;
using Grpc.Core;
using Grpc.Net.Client;
using Microsoft.AspNetCore.Builder;
using Microsoft.AspNetCore.Hosting;
using Microsoft.AspNetCore.TestHost;
using Microsoft.Extensions.DependencyInjection;
using Microsoft.Extensions.Hosting;
using Microsoft.Extensions.Logging.Abstractions;
using Microsoft.Extensions.Options;
using ZB.MOM.WW.ScadaBridge.Commons.Messages.Audit;
using ZB.MOM.WW.ScadaBridge.Commons.Messages.Notification;
using ZB.MOM.WW.ScadaBridge.Communication;
using ZB.MOM.WW.ScadaBridge.Communication.Grpc;
namespace ZB.MOM.WW.ScadaBridge.Host.Tests;
/// <summary>
/// End-to-end proof of the central-hosted gRPC control plane (T1A.2): the real
/// <see cref="CentralControlAuthInterceptor"/> AND the real
/// <see cref="CentralControlGrpcService"/> over a real gRPC stack, the service Asking a real
/// (stub) <c>CentralCommunicationActor</c> stand-in.
/// </summary>
/// <remarks>
/// <para>
/// The interceptor is registered <b>BY TYPE on <c>AddGrpc</c></b>, exactly as <c>Program.cs</c>
/// does, and is deliberately NOT placed in DI as a singleton — pre-registering it lets DI hand
/// the instance back and bypasses <c>Grpc.AspNetCore</c>'s own activation path, which is how the
/// two-public-constructor defect escaped a green suite once before. This harness copies the
/// shape of <see cref="ControlPlaneAuthEndToEndTests"/> for the same reason.
/// </para>
/// <para>
/// Runs in-process over <see cref="TestServer"/>: no ports, no containers. A tiny Akka actor
/// stands in for <c>CentralCommunicationActor</c> so the test never touches MSSQL or a cluster;
/// the method paths, message types and mapper are the real generated/production ones.
/// </para>
/// </remarks>
public class CentralControlEndToEndTests : IAsyncLifetime
{
private const string SiteA = "site-a";
private const string SiteAKey = "site-a-preshared-key";
private const string SiteB = "site-b";
private const string SiteBKey = "site-b-preshared-key";
// The deterministic id the stub actor "accepts" for every ingest batch, so the ingest
// bridge can be asserted without extracting ids out of a decoded AuditEvent.
private static readonly Guid AcceptedId = Guid.Parse("11111111-1111-1111-1111-111111111111");
private IHost _host = null!;
private TestServer _server = null!;
private ActorSystem _actorSystem = null!;
private CentralControlGrpcService _service = null!;
/// <inheritdoc />
public async Task InitializeAsync()
{
_actorSystem = ActorSystem.Create("centralcontrol-e2e-test");
var stub = _actorSystem.ActorOf(Props.Create(() => new StubCentralActor(AcceptedId)), "central-stub");
_service = new CentralControlGrpcService(
NullLogger<CentralControlGrpcService>.Instance,
Options.Create(new CommunicationOptions()));
_service.SetReady(stub);
var pskProvider = new MapPskProvider(new Dictionary<string, string>
{
[SiteA] = SiteAKey,
[SiteB] = SiteBKey,
});
_host = await new HostBuilder()
.ConfigureWebHost(web => web
.UseTestServer()
.ConfigureServices(services =>
{
// BY TYPE, and NOT also in DI — see the class remarks.
services.AddGrpc(o => o.Interceptors.Add<CentralControlAuthInterceptor>());
services.AddSingleton<ISitePskProvider>(pskProvider);
services.AddSingleton(_service);
})
.Configure(app =>
{
app.UseRouting();
app.UseEndpoints(e => e.MapGrpcService<CentralControlGrpcService>());
}))
.StartAsync();
_server = _host.GetTestServer();
}
/// <inheritdoc />
public async Task DisposeAsync()
{
await _host.StopAsync();
_host.Dispose();
await _actorSystem.Terminate();
}
/// <summary>Builds a channel credentialed for <paramref name="siteId"/> with <paramref name="key"/>.</summary>
private GrpcChannel Channel(string? key, string siteId)
{
var options = new GrpcChannelOptions { HttpHandler = _server.CreateHandler() };
if (key is not null)
{
options.WithSiteCredentials(new FixedPskProvider(key), siteId);
}
return GrpcChannel.ForAddress(_server.BaseAddress, options);
}
private CentralControlService.CentralControlServiceClient Client(string? key, string siteId)
=> new(Channel(key, siteId));
// ---- Auth positives / negatives, all through the real pipeline ----
[Fact]
public async Task CorrectSiteAndKey_ReachesTheService_OnAUnaryCall()
{
var client = Client(SiteAKey, SiteA);
var ack = await client.SubmitNotificationAsync(NewNotificationDto());
Assert.True(ack.Accepted);
}
[Fact]
public async Task NoCredentialsAtAll_IsRejected_WithPermissionDenied()
{
var client = Client(key: null, SiteA);
var ex = await Assert.ThrowsAsync<RpcException>(
async () => await client.SubmitNotificationAsync(new NotificationSubmitDto()));
Assert.Equal(StatusCode.PermissionDenied, ex.StatusCode);
}
[Fact]
public async Task WrongKeyForTheSite_IsRejected_WithPermissionDenied()
{
// site-a presents site-b's (valid, but wrong-for-this-site) key.
var client = Client(SiteBKey, SiteA);
var ex = await Assert.ThrowsAsync<RpcException>(
async () => await client.SubmitNotificationAsync(new NotificationSubmitDto()));
Assert.Equal(StatusCode.PermissionDenied, ex.StatusCode);
}
[Fact]
public async Task UnknownSite_IsRejected_WithPermissionDenied()
{
var client = Client("any-key", "site-nonexistent");
var ex = await Assert.ThrowsAsync<RpcException>(
async () => await client.SubmitNotificationAsync(new NotificationSubmitDto()));
Assert.Equal(StatusCode.PermissionDenied, ex.StatusCode);
}
[Fact]
public async Task TheAcceptAndRejectPathsAreDistinguishable()
{
// A gate whose accept and reject paths produce the same observable result is not a gate.
// Correct key → the service answers (Accepted:true); wrong key → PermissionDenied,
// never reaching the service. These are two different outcomes, which is the whole point.
var ok = await Client(SiteAKey, SiteA).SubmitNotificationAsync(NewNotificationDto());
Assert.True(ok.Accepted);
var ex = await Assert.ThrowsAsync<RpcException>(
async () => await Client("wrong", SiteA).SubmitNotificationAsync(new NotificationSubmitDto()));
Assert.Equal(StatusCode.PermissionDenied, ex.StatusCode);
}
// ---- One RPC per shape: unary (above) + the ingest bridge ----
[Fact]
public async Task IngestAuditEvents_DecodesTheBatch_AsksTheActor_EncodesTheAck()
{
var client = Client(SiteAKey, SiteA);
var batch = new AuditEventBatch();
batch.Events.Add(NewAuditDto());
batch.Events.Add(NewAuditDto());
var ack = await client.IngestAuditEventsAsync(batch);
// The stub actor accepts one deterministic id per non-empty batch; its presence proves
// the full DTO→command→Ask→reply→ack bridge ran, gated call and all.
Assert.Contains(AcceptedId.ToString(), ack.AcceptedEventIds);
}
[Fact]
public async Task IngestAuditEvents_EmptyBatch_ShortCircuits_WithoutAskingTheActor()
{
// Even the empty-batch fast path is behind the gate — it still needs a valid key.
var client = Client(SiteAKey, SiteA);
var ack = await client.IngestAuditEventsAsync(new AuditEventBatch());
Assert.Empty(ack.AcceptedEventIds);
}
// The mapper reads SiteEnqueuedAt unconditionally; only DTOs that clear the gate reach it,
// so the accept-path tests carry a timestamp while the negative tests can pass a bare DTO.
private static NotificationSubmitDto NewNotificationDto() => new()
{
NotificationId = Guid.NewGuid().ToString(),
ListName = "ops",
SiteEnqueuedAt = Timestamp.FromDateTime(
DateTime.SpecifyKind(new DateTime(2026, 5, 20, 10, 0, 0), DateTimeKind.Utc)),
};
private static AuditEventDto NewAuditDto() => new()
{
EventId = Guid.NewGuid().ToString(),
OccurredAtUtc = Timestamp.FromDateTime(
DateTime.SpecifyKind(new DateTime(2026, 5, 20, 10, 0, 0), DateTimeKind.Utc)),
Channel = "ApiOutbound",
Kind = "ApiCall",
Status = "Delivered",
SourceSiteId = SiteA,
};
private sealed class FixedPskProvider(string key) : ISitePskProvider
{
public ValueTask<string> GetAsync(string siteId, CancellationToken ct) => new(key);
public void Invalidate(string siteId) { }
}
private sealed class MapPskProvider(IReadOnlyDictionary<string, string> keys) : ISitePskProvider
{
public ValueTask<string> GetAsync(string siteId, CancellationToken ct)
=> keys.TryGetValue(siteId, out var key)
? new ValueTask<string>(key)
: throw new InvalidOperationException($"no key for '{siteId}'");
public void Invalidate(string siteId) { }
}
/// <summary>
/// Minimal stand-in for <c>CentralCommunicationActor</c>: answers the two RPC shapes this
/// test exercises. Replies straight to the Ask's temp sender, exactly as the real actor's
/// Forward/PipeTo paths do.
/// </summary>
private sealed class StubCentralActor : ReceiveActor
{
public StubCentralActor(Guid acceptedId)
{
Receive<NotificationSubmit>(msg =>
Sender.Tell(new NotificationSubmitAck(msg.NotificationId, Accepted: true, Error: null)));
Receive<IngestAuditEventsCommand>(_ =>
Sender.Tell(new IngestAuditEventsReply(new[] { acceptedId })));
}
}
}
@@ -0,0 +1,56 @@
using Xunit;
namespace ZB.MOM.WW.ScadaBridge.Host.Tests;
/// <summary>
/// Pins <see cref="Program.ParseHttpBindPorts"/>, which re-declares central's HTTP surface
/// after the gRPC listener switches Kestrel into explicit-endpoints mode.
/// </summary>
/// <remarks>
/// This exists because the regression it guards is invisible to every other test: a central
/// node that binds only the gRPC port boots clean, joins the cluster and serves gRPC, while
/// the Central UI, Management/Inbound API and <c>/health/*</c> are silently dead. TestServer
/// never binds real Kestrel, so the parser is the one seam that can be asserted in-process.
/// The rig caught the original defect; this keeps it caught.
/// </remarks>
public class CentralHttpBindPortsTests
{
[Theory]
[InlineData("http://+:5000", new[] { 5000 })]
[InlineData("http://0.0.0.0:5000", new[] { 5000 })]
[InlineData("http://[::]:5000", new[] { 5000 })]
[InlineData("http://localhost:5000", new[] { 5000 })]
[InlineData("http://*:8085", new[] { 8085 })]
[InlineData("http://+:5000;http://+:5001", new[] { 5000, 5001 })]
[InlineData("http://+:5000 ; http://+:5001", new[] { 5000, 5001 })]
[InlineData("http://+:5000;http://+:5000", new[] { 5000 })] // de-duped
public void ParsesPortsFromServerUrls(string urls, int[] expected)
{
Assert.Equal(expected, Program.ParseHttpBindPorts(urls));
}
[Theory]
[InlineData("https://+:443", 443)] // scheme default when no explicit port
[InlineData("http://+:80", 80)]
public void HandlesSchemeDefaultAndExplicitPort(string url, int expected)
{
Assert.Equal(new[] { expected }, Program.ParseHttpBindPorts(url));
}
[Theory]
[InlineData(null)]
[InlineData("")]
[InlineData(" ")]
public void EmptyOrNull_YieldsNoPorts(string? urls)
{
Assert.Empty(Program.ParseHttpBindPorts(urls));
}
[Fact]
public void UnparseableEntry_IsSkipped_NotThrown()
{
// A malformed URL must not take the node down at boot; the good one still binds.
var ports = Program.ParseHttpBindPorts("not-a-url;http://+:5000");
Assert.Equal(new[] { 5000 }, ports);
}
}
@@ -0,0 +1,373 @@
using System.Collections.Concurrent;
using Akka.Actor;
using Microsoft.AspNetCore.Builder;
using Microsoft.AspNetCore.Hosting;
using Microsoft.AspNetCore.TestHost;
using Microsoft.Extensions.DependencyInjection;
using Microsoft.Extensions.Hosting;
using Microsoft.Extensions.Logging.Abstractions;
using Microsoft.Extensions.Options;
using ZB.MOM.WW.ScadaBridge.Commons.Messages.Notification;
using ZB.MOM.WW.ScadaBridge.Communication;
using ZB.MOM.WW.ScadaBridge.Communication.Grpc;
namespace ZB.MOM.WW.ScadaBridge.Host.Tests;
/// <summary>
/// T1A.3: <see cref="GrpcCentralTransport"/> + <see cref="CentralChannelProvider"/> over a real
/// gRPC stack (two in-process <see cref="TestServer"/> central nodes, the real
/// <see cref="CentralControlGrpcService"/> and <see cref="CentralControlAuthInterceptor"/>). Proves
/// the sticky failover/failback policy, the PSK + site-header attachment, the per-call deadline,
/// and — the hard rule — no cross-node retry on <c>DeadlineExceeded</c>.
/// </summary>
/// <remarks>
/// A "down" node is modelled by a <see cref="ToggleHandler"/> that throws before reaching the
/// TestServer, so BOTH the unary call and the failback <c>Heartbeat</c> probe see it as
/// <c>Unavailable</c> — the honest shape of a refused connection, and the only class the transport
/// fails over on. Readiness is always set, so a node that is "up" answers everything.
/// </remarks>
public class GrpcCentralTransportTests : IAsyncLifetime
{
private const string SiteA = "site-a";
private const string SiteAKey = "site-a-preshared-key";
private const string EndpointA = "http://central-a/";
private const string EndpointB = "http://central-b/";
private ActorSystem _system = null!;
private CentralNode _nodeA = null!;
private CentralNode _nodeB = null!;
/// <inheritdoc />
public async Task InitializeAsync()
{
_system = ActorSystem.Create("grpc-central-transport-test");
_nodeA = await CentralNode.StartAsync(_system, "A", SiteA, SiteAKey, repliesToSubmit: true);
_nodeB = await CentralNode.StartAsync(_system, "B", SiteA, SiteAKey, repliesToSubmit: true);
}
/// <inheritdoc />
public async Task DisposeAsync()
{
await _nodeA.DisposeAsync();
await _nodeB.DisposeAsync();
await _system.Terminate();
}
private CentralChannelProvider NewProvider(string? pskKey = SiteAKey) => new(
new[] { EndpointA, EndpointB },
new FixedPskProvider(pskKey),
SiteA,
new CommunicationOptions(),
NullLogger<CentralChannelProvider>.Instance,
handlerFactory: HandlerFor,
probeDeadline: TimeSpan.FromSeconds(2),
backoffBase: TimeSpan.FromMilliseconds(50),
backoffCap: TimeSpan.FromMilliseconds(200));
private HttpMessageHandler HandlerFor(string endpoint) => endpoint == EndpointA
? new ToggleHandler(_nodeA.Server.CreateHandler(), () => _nodeA.IsUp)
: new ToggleHandler(_nodeB.Server.CreateHandler(), () => _nodeB.IsUp);
private GrpcCentralTransport NewTransport(CentralChannelProvider provider, CommunicationOptions? options = null)
=> new(provider, options ?? new CommunicationOptions(), NullLogger<GrpcCentralTransport>.Instance);
[Fact]
public async Task HappyPath_ReachesThePreferredNode_AndRoutesTheAckBack()
{
using var provider = NewProvider();
var transport = NewTransport(provider);
var inbox = new Capture(_system);
transport.SubmitNotification(NewSubmit("n1"), inbox.Ref);
var ack = Assert.IsType<NotificationSubmitAck>(inbox.Receive(TimeSpan.FromSeconds(5)));
Assert.True(ack.Accepted);
Assert.Equal("n1", ack.NotificationId);
Assert.Equal(0, provider.CurrentIndex); // stayed on preferred
Assert.Equal(1, _nodeA.SubmitCount);
Assert.Equal(0, _nodeB.SubmitCount);
}
[Fact]
public async Task Sticky_StaysOnThePreferredNode_WhileHealthy()
{
using var provider = NewProvider();
var transport = NewTransport(provider);
for (var i = 0; i < 4; i++)
{
var inbox = new Capture(_system);
transport.SubmitNotification(NewSubmit($"n{i}"), inbox.Ref);
Assert.IsType<NotificationSubmitAck>(inbox.Receive(TimeSpan.FromSeconds(5)));
}
Assert.Equal(0, provider.CurrentIndex);
Assert.Equal(4, _nodeA.SubmitCount);
Assert.Equal(0, _nodeB.SubmitCount);
}
[Fact]
public async Task Failover_FlipsToThePeer_WhenThePreferredIsUnavailable()
{
using var provider = NewProvider();
var transport = NewTransport(provider);
_nodeA.IsUp = false; // preferred refuses connections
var inbox = new Capture(_system);
transport.SubmitNotification(NewSubmit("n1"), inbox.Ref);
var ack = Assert.IsType<NotificationSubmitAck>(inbox.Receive(TimeSpan.FromSeconds(5)));
Assert.True(ack.Accepted);
Assert.Equal(1, provider.CurrentIndex); // flipped to the peer
Assert.Equal(0, _nodeA.SubmitCount);
Assert.Equal(1, _nodeB.SubmitCount);
}
[Fact]
public async Task Failback_ReturnsToThePreferred_OnceItIsReachableAgain()
{
using var provider = NewProvider();
var transport = NewTransport(provider);
// Take the preferred down and drive one call so we flip to the peer + arm the failback probe.
_nodeA.IsUp = false;
var inbox = new Capture(_system);
transport.SubmitNotification(NewSubmit("n1"), inbox.Ref);
Assert.IsType<NotificationSubmitAck>(inbox.Receive(TimeSpan.FromSeconds(5)));
Assert.Equal(1, provider.CurrentIndex);
// Bring the preferred back; the background probe should fail us back within a few backoffs.
_nodeA.IsUp = true;
await WaitUntil(() => provider.CurrentIndex == 0, TimeSpan.FromSeconds(5));
Assert.Equal(0, provider.CurrentIndex);
// New calls resume on the preferred node.
var inbox2 = new Capture(_system);
transport.SubmitNotification(NewSubmit("n2"), inbox2.Ref);
Assert.IsType<NotificationSubmitAck>(inbox2.Receive(TimeSpan.FromSeconds(5)));
Assert.True(_nodeA.SubmitCount >= 1);
}
[Fact]
public async Task PskAndSiteHeader_AreAttached_SoTheGatedCallReachesTheService()
{
// The service is gated by CentralControlAuthInterceptor; a call that reaches it (and gets
// Accepted) proves both the bearer PSK and the x-scadabridge-site header were attached.
using var provider = NewProvider(pskKey: SiteAKey);
var transport = NewTransport(provider);
var inbox = new Capture(_system);
transport.SubmitNotification(NewSubmit("n1"), inbox.Ref);
var ack = Assert.IsType<NotificationSubmitAck>(inbox.Receive(TimeSpan.FromSeconds(5)));
Assert.True(ack.Accepted);
}
[Fact]
public async Task WrongPsk_IsRejected_AndNotRetriedOnThePeer()
{
// PermissionDenied is not a connect failure — the transport surfaces it as a transient
// Status.Failure without flipping to the peer.
using var provider = NewProvider(pskKey: "the-wrong-key");
var transport = NewTransport(provider);
var inbox = new Capture(_system);
transport.SubmitNotification(NewSubmit("n1"), inbox.Ref);
Assert.IsType<Status.Failure>(inbox.Receive(TimeSpan.FromSeconds(5)));
Assert.Equal(0, provider.CurrentIndex); // no flip
Assert.Equal(0, _nodeB.SubmitCount); // peer never tried
}
[Fact]
public async Task DeadlineExceeded_IsNotRetriedOnThePeer()
{
// THE hard rule. Node A is UP but never replies, so the call deadlines. The transport must
// surface Status.Failure and must NOT try node B (the call may already have executed).
_nodeA.SetBlackHole();
var shortDeadline = new CommunicationOptions { NotificationForwardTimeout = TimeSpan.FromMilliseconds(300) };
using var provider = NewProvider();
var transport = NewTransport(provider, shortDeadline);
var inbox = new Capture(_system);
transport.SubmitNotification(NewSubmit("n1"), inbox.Ref);
// A per-call deadline is applied (the call returns fast instead of hanging on the black hole).
Assert.IsType<Status.Failure>(inbox.Receive(TimeSpan.FromSeconds(5)));
Assert.Equal(0, provider.CurrentIndex); // no failover on a deadline
Assert.Equal(0, _nodeB.SubmitCount); // peer never tried
}
private static NotificationSubmit NewSubmit(string id) => new(
NotificationId: id,
ListName: "ops",
Subject: "s",
Body: "b",
SourceSiteId: SiteA,
SourceInstanceId: null,
SourceScript: null,
SiteEnqueuedAt: DateTimeOffset.UtcNow);
private static async Task WaitUntil(Func<bool> condition, TimeSpan timeout)
{
var deadline = DateTime.UtcNow + timeout;
while (DateTime.UtcNow < deadline)
{
if (condition())
{
return;
}
await Task.Delay(25);
}
}
/// <summary>
/// A raw message sink used as the transport's <c>replyTo</c>. Unlike Akka's <c>Inbox</c>, which
/// rethrows a <see cref="Status.Failure"/>'s cause on receive, this captures every message
/// verbatim so a test can assert on the <see cref="Status.Failure"/> itself.
/// </summary>
private sealed class Capture
{
private readonly BlockingCollection<object> _messages = new();
public Capture(ActorSystem system)
{
Ref = system.ActorOf(Props.Create(() => new CaptureActor(_messages)));
}
public IActorRef Ref { get; }
public object Receive(TimeSpan timeout)
=> _messages.TryTake(out var message, timeout)
? message
: throw new TimeoutException("No message captured within the timeout.");
private sealed class CaptureActor : ReceiveActor
{
public CaptureActor(BlockingCollection<object> messages) => ReceiveAny(messages.Add);
}
}
/// <summary>A gRPC channel handler that throws (a refused connection) while its node is "down".</summary>
private sealed class ToggleHandler : DelegatingHandler
{
private readonly Func<bool> _isUp;
public ToggleHandler(HttpMessageHandler inner, Func<bool> isUp)
{
InnerHandler = inner;
_isUp = isUp;
}
protected override async Task<HttpResponseMessage> SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
{
if (!_isUp())
{
throw new HttpRequestException("simulated central node down");
}
return await base.SendAsync(request, cancellationToken).ConfigureAwait(false);
}
}
private sealed class FixedPskProvider(string? key) : ISitePskProvider
{
public ValueTask<string> GetAsync(string siteId, CancellationToken ct)
=> key is null ? throw new InvalidOperationException("no key") : new ValueTask<string>(key);
public void Invalidate(string siteId) { }
}
/// <summary>One in-process central node: TestServer + real service/interceptor + a stub actor.</summary>
private sealed class CentralNode : IAsyncDisposable
{
private IHost _host = null!;
private IActorRef _stub = null!;
private readonly StubCounters _counters = new();
public TestServer Server { get; private set; } = null!;
public volatile bool IsUp = true;
public int SubmitCount => _counters.Submits;
public static async Task<CentralNode> StartAsync(
ActorSystem system, string label, string site, string key, bool repliesToSubmit)
{
var node = new CentralNode();
node._stub = system.ActorOf(
Props.Create(() => new StubCentralActor(node._counters, repliesToSubmit)), $"stub-{label}");
var service = new CentralControlGrpcService(
NullLogger<CentralControlGrpcService>.Instance,
Options.Create(new CommunicationOptions()));
service.SetReady(node._stub);
var psk = new MapPskProvider(new Dictionary<string, string> { [site] = key });
node._host = await new HostBuilder()
.ConfigureWebHost(web => web
.UseTestServer()
.ConfigureServices(services =>
{
services.AddGrpc(o => o.Interceptors.Add<CentralControlAuthInterceptor>());
services.AddSingleton<ISitePskProvider>(psk);
services.AddSingleton(service);
})
.Configure(app =>
{
app.UseRouting();
app.UseEndpoints(e => e.MapGrpcService<CentralControlGrpcService>());
}))
.StartAsync();
node.Server = node._host.GetTestServer();
return node;
}
/// <summary>Switches the node's actor to a black hole that counts but never replies.</summary>
public void SetBlackHole() => _counters.BlackHole = true;
public async ValueTask DisposeAsync()
{
await _host.StopAsync();
_host.Dispose();
}
private sealed class StubCounters
{
private int _submits;
public int Submits => Volatile.Read(ref _submits);
public void IncrementSubmits() => Interlocked.Increment(ref _submits);
public volatile bool BlackHole;
}
private sealed class StubCentralActor : ReceiveActor
{
public StubCentralActor(StubCounters counters, bool repliesToSubmit)
{
Receive<NotificationSubmit>(msg =>
{
counters.IncrementSubmits();
if (repliesToSubmit && !counters.BlackHole)
{
Sender.Tell(new NotificationSubmitAck(msg.NotificationId, Accepted: true, Error: null));
}
});
// Heartbeat lands here as a Tell (the failback probe); ignore it, no reply expected.
ReceiveAny(_ => { });
}
}
private sealed class MapPskProvider(IReadOnlyDictionary<string, string> keys) : ISitePskProvider
{
public ValueTask<string> GetAsync(string siteId, CancellationToken ct)
=> keys.TryGetValue(siteId, out var key)
? new ValueTask<string>(key)
: throw new InvalidOperationException($"no key for '{siteId}'");
public void Invalidate(string siteId) { }
}
}
}
@@ -0,0 +1,218 @@
using Grpc.Core;
using Grpc.Net.Client;
using Microsoft.AspNetCore.Builder;
using Microsoft.AspNetCore.Hosting;
using Microsoft.AspNetCore.TestHost;
using Microsoft.Extensions.DependencyInjection;
using Microsoft.Extensions.Hosting;
using Microsoft.Extensions.Logging.Abstractions;
using Microsoft.Extensions.Options;
using ZB.MOM.WW.ScadaBridge.Commons.Messages.Lifecycle;
using ZB.MOM.WW.ScadaBridge.Communication;
using ZB.MOM.WW.ScadaBridge.Communication.Grpc;
namespace ZB.MOM.WW.ScadaBridge.Host.Tests;
/// <summary>
/// T1B.3: <see cref="SitePairChannelProvider"/> failover/failback + credential/deadline proof over
/// two in-process gRPC <see cref="TestServer"/>s (NodeA preferred, NodeB standby). Exercises the
/// real client stack — PSK call credentials, per-call deadline, sticky failover, no-retry on
/// <see cref="StatusCode.DeadlineExceeded"/>, and failback to the preferred node.
/// </summary>
public sealed class GrpcSiteTransportFailoverTests : IAsyncLifetime
{
private const string SiteId = "site-1";
private const string SiteKey = "the-site-1-key";
private const string EndpointA = "http://node-a";
private const string EndpointB = "http://node-b";
private IHost _hostA = null!;
private IHost _hostB = null!;
private StubSiteCommandService _stubA = null!;
private StubSiteCommandService _stubB = null!;
private SitePairChannelProvider _provider = null!;
private volatile bool _preferredReachable;
/// <inheritdoc />
public async Task InitializeAsync()
{
(_hostA, _stubA) = await StartServerAsync();
(_hostB, _stubB) = await StartServerAsync();
var handlerA = _hostA.GetTestServer().CreateHandler();
var handlerB = _hostB.GetTestServer().CreateHandler();
_provider = new SitePairChannelProvider(
new FixedPskProvider(SiteKey),
Options.Create(new CommunicationOptions()),
NullLogger<SitePairChannelProvider>.Instance,
handlerFactory: endpoint => endpoint == EndpointA ? handlerA : handlerB,
reachabilityProbe: (_, _) => Task.FromResult(_preferredReachable));
_provider.UpdateSite(SiteId, EndpointA, EndpointB);
}
/// <inheritdoc />
public async Task DisposeAsync()
{
_provider.Dispose();
await _hostA.StopAsync();
await _hostB.StopAsync();
_hostA.Dispose();
_hostB.Dispose();
}
private static async Task<(IHost, StubSiteCommandService)> StartServerAsync()
{
var stub = new StubSiteCommandService();
var host = await new HostBuilder()
.ConfigureWebHost(web => web
.UseTestServer()
.ConfigureServices(services =>
{
services.AddGrpc();
services.AddSingleton(stub);
})
.Configure(app =>
{
app.UseRouting();
app.UseEndpoints(e => e.MapGrpcService<StubSiteCommandService>());
}))
.StartAsync();
return (host, stub);
}
private static LifecycleRequest EnableReq() =>
SiteCommandDtoMapper.ToLifecycleRequest(
new EnableInstanceCommand("cmd-1", "Site1.Pump1", DateTimeOffset.UtcNow));
private Task<LifecycleReply> CallAsync(TimeSpan? deadline = null) =>
_provider.ExecuteAsync(
SiteId,
(channel, ct) =>
{
var client = new SiteCommandService.SiteCommandServiceClient(channel);
return client.ExecuteLifecycleAsync(
EnableReq(),
deadline: DateTime.UtcNow + (deadline ?? TimeSpan.FromSeconds(10)),
cancellationToken: ct).ResponseAsync;
},
CancellationToken.None);
[Fact]
public async Task HealthyCall_HitsPreferredNodeA_WithPskAndSiteHeaderAndDeadline()
{
var reply = await CallAsync();
Assert.Equal(LifecycleReply.ReplyOneofCase.InstanceLifecycle, reply.ReplyCase);
Assert.Equal(1, _stubA.LifecycleCalls);
Assert.Equal(0, _stubB.LifecycleCalls);
Assert.Equal($"Bearer {SiteKey}", _stubA.LastAuthHeader);
Assert.Equal(SiteId, _stubA.LastSiteHeader);
Assert.True(_stubA.DeadlineWasSet, "the client must set a per-call deadline");
Assert.True(_provider.IsOnPreferredNode(SiteId));
}
[Fact]
public async Task Unavailable_OnNodeA_FailsOverToNodeB_AndStaysSticky()
{
_stubA.ThrowUnavailable = true;
var reply = await CallAsync();
// Failed over: A was tried (and threw), B answered.
Assert.Equal(LifecycleReply.ReplyOneofCase.InstanceLifecycle, reply.ReplyCase);
Assert.Equal(1, _stubA.LifecycleCalls);
Assert.Equal(1, _stubB.LifecycleCalls);
Assert.False(_provider.IsOnPreferredNode(SiteId));
// Sticky: the next call goes straight to B without re-touching A.
await CallAsync();
Assert.Equal(1, _stubA.LifecycleCalls);
Assert.Equal(2, _stubB.LifecycleCalls);
}
[Fact]
public async Task Failback_ReturnsToPreferredNodeA_OncePreferredIsReachableAgain()
{
_stubA.ThrowUnavailable = true;
await CallAsync(); // flips to B
Assert.False(_provider.IsOnPreferredNode(SiteId));
// NodeA recovers; the failback probe reports it reachable.
_stubA.ThrowUnavailable = false;
_preferredReachable = true;
var back = await _provider.TryFailbackAsync(SiteId, CancellationToken.None);
Assert.True(back);
Assert.True(_provider.IsOnPreferredNode(SiteId));
var beforeA = _stubA.LifecycleCalls;
await CallAsync();
Assert.Equal(beforeA + 1, _stubA.LifecycleCalls); // next call back on A
}
[Fact]
public async Task DeadlineExceeded_OnNodeA_IsNotRetriedOnNodeB()
{
// A DeadlineExceeded is ambiguous (a WriteTag/Deploy/Failover may already have executed), so
// — unlike Unavailable — it must NOT fail over to B. Modelled by A returning the status
// directly, isolating the retry-decision from TestServer's own timeout mechanics.
_stubA.StatusToThrow = StatusCode.DeadlineExceeded;
var ex = await Assert.ThrowsAsync<RpcException>(() => CallAsync());
Assert.Equal(StatusCode.DeadlineExceeded, ex.StatusCode);
Assert.Equal(1, _stubA.LifecycleCalls);
Assert.Equal(0, _stubB.LifecycleCalls); // B was never tried
Assert.True(_provider.IsOnPreferredNode(SiteId), "a deadline must not flip stickiness");
}
[Fact]
public async Task UnknownSite_Throws_SiteChannelUnavailable()
{
await Assert.ThrowsAsync<SiteChannelUnavailableException>(
() => _provider.ExecuteAsync<LifecycleReply>(
"not-configured", (_, _) => Task.FromResult(new LifecycleReply()), CancellationToken.None));
}
private sealed class FixedPskProvider(string key) : ISitePskProvider
{
public ValueTask<string> GetAsync(string siteId, CancellationToken ct) => new(key);
public void Invalidate(string siteId) { }
}
/// <summary>Stub SiteCommandService recording what the client sent and letting a test steer faults.</summary>
private sealed class StubSiteCommandService : SiteCommandService.SiteCommandServiceBase
{
private int _lifecycleCalls;
public int LifecycleCalls => Volatile.Read(ref _lifecycleCalls);
public string? LastAuthHeader { get; private set; }
public string? LastSiteHeader { get; private set; }
public bool DeadlineWasSet { get; private set; }
public volatile bool ThrowUnavailable;
public StatusCode? StatusToThrow;
public override Task<LifecycleReply> ExecuteLifecycle(LifecycleRequest request, ServerCallContext context)
{
Interlocked.Increment(ref _lifecycleCalls);
LastAuthHeader = context.RequestHeaders
.FirstOrDefault(h => h.Key == ControlPlaneCredentials.AuthorizationHeader)?.Value;
LastSiteHeader = context.RequestHeaders
.FirstOrDefault(h => h.Key == ControlPlaneCredentials.SiteHeader)?.Value;
DeadlineWasSet = context.Deadline != DateTime.MaxValue;
if (ThrowUnavailable)
{
throw new RpcException(new Status(StatusCode.Unavailable, "node down"));
}
if (StatusToThrow is { } status)
{
throw new RpcException(new Status(status, "modelled fault"));
}
return Task.FromResult(SiteCommandDtoMapper.ToLifecycleReply(
new InstanceLifecycleResponse("cmd-1", "Site1.Pump1", true, null, DateTimeOffset.UtcNow)));
}
}
}
@@ -0,0 +1,285 @@
using Akka.Actor;
using Grpc.Core;
using Grpc.Net.Client;
using Microsoft.AspNetCore.Builder;
using Microsoft.AspNetCore.Hosting;
using Microsoft.AspNetCore.TestHost;
using Microsoft.Extensions.DependencyInjection;
using Microsoft.Extensions.Hosting;
using Microsoft.Extensions.Logging.Abstractions;
using Microsoft.Extensions.Options;
using ZB.MOM.WW.ScadaBridge.Commons.Messages.DebugView;
using ZB.MOM.WW.ScadaBridge.Commons.Messages.InboundApi;
using ZB.MOM.WW.ScadaBridge.Commons.Messages.Lifecycle;
using ZB.MOM.WW.ScadaBridge.Commons.Messages.Management;
using ZB.MOM.WW.ScadaBridge.Commons.Messages.RemoteQuery;
using ZB.MOM.WW.ScadaBridge.Communication;
using ZB.MOM.WW.ScadaBridge.Communication.Actors;
using ZB.MOM.WW.ScadaBridge.Communication.Grpc;
namespace ZB.MOM.WW.ScadaBridge.Host.Tests;
/// <summary>
/// The site command plane's gRPC front door (T1B.2): the same PSK gate and readiness convention
/// as <c>SiteStreamGrpcServer</c>, and a decode→dispatch→encode round trip through the ONE
/// <see cref="SiteCommandDispatcher"/> for one command in each of the six oneof groups. Runs
/// in-process over <see cref="TestServer"/> — no ports, no containers — with the interceptor
/// registered BY TYPE on <c>AddGrpc</c> (never pre-registered in DI), the shape that keeps
/// <c>Grpc.AspNetCore</c>'s own activation in the picture. See
/// <see cref="ControlPlaneAuthEndToEndTests"/> for why that matters.
/// </summary>
public sealed class SiteCommandGrpcServiceTests : IDisposable
{
private const string SiteKey = "the-site-a-command-key";
private const string SiteId = "site-a";
private readonly ActorSystem _system = ActorSystem.Create("sitecmd-tests");
/// <inheritdoc />
public void Dispose() => _system.Dispose();
/// <summary>A dispatcher whose Deployment Manager proxy is a canned-reply responder.</summary>
private SiteCommandDispatcher DispatcherWithResponder(out IActorRef responder)
{
responder = _system.ActorOf(Props.Create(() => new Responder()));
var dispatcher = new SiteCommandDispatcher(SiteId, responder, (_, _) => null);
// The parked handler is node-local; point it at the responder too so the Parked group can
// be exercised end-to-end (the null-guard path is covered by the dispatcher unit tests).
dispatcher.RegisterParkedMessageHandler(responder);
return dispatcher;
}
private SiteCommandGrpcService ReadyService()
{
var service = new SiteCommandGrpcService(NullLogger<SiteCommandGrpcService>.Instance);
service.SetReady(DispatcherWithResponder(out _));
return service;
}
private static async Task<IHost> StartHost(SiteCommandGrpcService service)
=> await new HostBuilder()
.ConfigureWebHost(web => web
.UseTestServer()
.ConfigureServices(services =>
{
// BY TYPE on AddGrpc — never AddSingleton the interceptor (see the sibling test).
services.AddGrpc(o => o.Interceptors.Add<ControlPlaneAuthInterceptor>());
services.AddSingleton(Options.Create(new CommunicationOptions { GrpcPsk = SiteKey }));
services.AddSingleton(service);
})
.Configure(app =>
{
app.UseRouting();
app.UseEndpoints(e => e.MapGrpcService<SiteCommandGrpcService>());
}))
.StartAsync();
private static SiteCommandService.SiteCommandServiceClient Client(IHost host, string? key)
{
var server = host.GetTestServer();
var options = new GrpcChannelOptions { HttpHandler = server.CreateHandler() };
if (key is not null)
{
options.WithSiteCredentials(new FixedPskProvider(key), SiteId);
}
var channel = GrpcChannel.ForAddress(server.BaseAddress, options);
return new SiteCommandService.SiteCommandServiceClient(channel);
}
private sealed class FixedPskProvider(string key) : ISitePskProvider
{
public ValueTask<string> GetAsync(string siteId, CancellationToken ct) => new(key);
public void Invalidate(string siteId) { }
}
// ── Auth gating (delegated to ControlPlaneAuthInterceptor, proven wired here) ──
[Fact]
public async Task NoCredentials_IsRejected_WithPermissionDenied()
{
using var host = await StartHost(ReadyService());
var client = Client(host, key: null);
var ex = await Assert.ThrowsAsync<RpcException>(() => client.ExecuteQueryAsync(
new QueryRequest { UnsubscribeDebugView = new UnsubscribeDebugViewRequestDto() }).ResponseAsync);
Assert.Equal(StatusCode.PermissionDenied, ex.StatusCode);
}
[Fact]
public async Task WrongKey_IsRejected_WithPermissionDenied()
{
using var host = await StartHost(ReadyService());
var client = Client(host, "some-other-key");
var ex = await Assert.ThrowsAsync<RpcException>(() => client.ExecuteQueryAsync(
new QueryRequest { UnsubscribeDebugView = new UnsubscribeDebugViewRequestDto() }).ResponseAsync);
Assert.Equal(StatusCode.PermissionDenied, ex.StatusCode);
}
// ── Readiness ──
[Fact]
public async Task BeforeSetReady_IsRejected_WithUnavailable_EvenWithACorrectKey()
{
// Auth passes (correct key); the readiness gate then rejects until the site actor graph is up.
var unready = new SiteCommandGrpcService(NullLogger<SiteCommandGrpcService>.Instance);
using var host = await StartHost(unready);
var client = Client(host, SiteKey);
var ex = await Assert.ThrowsAsync<RpcException>(() => client.ExecuteLifecycleAsync(
new LifecycleRequest { EnableInstance = new EnableInstanceCommandDto { CommandId = "c" } }).ResponseAsync);
Assert.Equal(StatusCode.Unavailable, ex.StatusCode);
}
// ── decode → dispatch → encode, one command per oneof group ──
[Fact]
public async Task ExecuteLifecycle_RoundTripsThroughTheDispatcher()
{
using var host = await StartHost(ReadyService());
var client = Client(host, SiteKey);
var command = new EnableInstanceCommand("cmd-1", "Site1.Pump1", DateTimeOffset.UtcNow);
var reply = await client.ExecuteLifecycleAsync(
new LifecycleRequest { EnableInstance = SiteCommandDtoMapper.ToProto(command) });
var decoded = Assert.IsType<InstanceLifecycleResponse>(SiteCommandDtoMapper.FromLifecycleReply(reply));
Assert.True(decoded.Success);
Assert.Equal("cmd-1", decoded.CommandId);
}
[Fact]
public async Task ExecuteOpcUa_RoundTripsThroughTheDispatcher()
{
using var host = await StartHost(ReadyService());
var client = Client(host, SiteKey);
var command = new BrowseNodeCommand("conn", null, null, null);
var reply = await client.ExecuteOpcUaAsync(
new OpcUaRequest { BrowseNode = SiteCommandDtoMapper.ToProto(command) });
Assert.IsType<BrowseNodeResult>(SiteCommandDtoMapper.FromOpcUaReply(reply));
}
[Fact]
public async Task ExecuteQuery_DebugSnapshot_RoundTripsThroughTheDispatcher()
{
using var host = await StartHost(ReadyService());
var client = Client(host, SiteKey);
var command = new DebugSnapshotRequest("Site1.Pump1", "corr-4");
var reply = await client.ExecuteQueryAsync(
new QueryRequest { DebugSnapshot = SiteCommandDtoMapper.ToProto(command) });
var decoded = Assert.IsType<DebugViewSnapshot>(SiteCommandDtoMapper.FromQueryReply(reply));
Assert.Equal("Site1.Pump1", decoded.InstanceUniqueName);
}
[Fact]
public async Task ExecuteQuery_UnsubscribeDebugView_ReturnsTheFireAndForgetAck()
{
// Fire-and-forget: the service Tells the target and returns the synthetic ack so a unary
// RPC still answers, keeping the caller-visible fire-and-forget semantics.
using var host = await StartHost(ReadyService());
var client = Client(host, SiteKey);
var reply = await client.ExecuteQueryAsync(new QueryRequest
{
UnsubscribeDebugView = SiteCommandDtoMapper.ToProto(
new UnsubscribeDebugViewRequest("Site1.Pump1", "corr-6")),
});
Assert.Equal(QueryReply.ReplyOneofCase.UnsubscribeDebugView, reply.ReplyCase);
}
[Fact]
public async Task ExecuteParked_RoundTripsThroughTheNodeLocalHandler()
{
using var host = await StartHost(ReadyService());
var client = Client(host, SiteKey);
var command = new ParkedMessageQueryRequest("corr-7", SiteId, 2, 25, DateTimeOffset.UtcNow);
var reply = await client.ExecuteParkedAsync(
new ParkedRequest { ParkedMessageQuery = SiteCommandDtoMapper.ToProto(command) });
var decoded = Assert.IsType<ParkedMessageQueryResponse>(SiteCommandDtoMapper.FromParkedReply(reply));
Assert.True(decoded.Success);
Assert.Equal("corr-7", decoded.CorrelationId);
}
[Fact]
public async Task ExecuteRoute_RoundTripsThroughTheDispatcher()
{
using var host = await StartHost(ReadyService());
var client = Client(host, SiteKey);
var command = new RouteToCallRequest(
"corr-12", "Site1.Pump1", "Start", new Dictionary<string, object?>(), DateTimeOffset.UtcNow, null);
var reply = await client.ExecuteRouteAsync(
new RouteRequest { RouteToCall = SiteCommandDtoMapper.ToProto(command) });
var decoded = Assert.IsType<RouteToCallResponse>(SiteCommandDtoMapper.FromRouteReply(reply));
Assert.True(decoded.Success);
Assert.Equal("corr-12", decoded.CorrelationId);
}
// ── Failover: the reply completes before the leave is initiated ──
[Fact]
public async Task TriggerFailover_ReturnsTheAck_BeforeTheLeaveIsInitiated()
{
// The resolver records the order of its calls: PrepareFailover does a DRY-RUN resolve to
// build the ack (recorded synchronously, before the RPC returns), and the real leave runs
// only on the deferred CommitLeave — so the recorded order is always resolve-then-leave,
// i.e. the ack is on the wire before the node begins leaving.
var events = new List<string>();
var leaveHappened = new ManualResetEventSlim(false);
Func<string, bool, string?> resolve = (_, dryRun) =>
{
lock (events) { events.Add(dryRun ? "resolve" : "leave"); }
if (!dryRun) leaveHappened.Set();
return "akka.tcp://scadabridge@site-a-node-a:8082";
};
var service = new SiteCommandGrpcService(NullLogger<SiteCommandGrpcService>.Instance);
var dispatcher = new SiteCommandDispatcher(SiteId, _system.ActorOf(Props.Create(() => new Responder())), resolve);
service.SetReady(dispatcher);
using var host = await StartHost(service);
var client = Client(host, SiteKey);
var reply = await client.TriggerFailoverAsync(
new TriggerSiteFailoverDto { CorrelationId = "corr-16", SiteId = SiteId });
Assert.True(reply.Accepted);
Assert.Equal("akka.tcp://scadabridge@site-a-node-a:8082", reply.TargetAddress);
Assert.True(leaveHappened.Wait(TimeSpan.FromSeconds(5)), "the deferred leave never ran");
lock (events)
{
Assert.Equal(new[] { "resolve", "leave" }, events);
}
}
/// <summary>Deployment Manager stand-in: replies each command with a canned reply of the right group.</summary>
private sealed class Responder : ReceiveActor
{
public Responder() => ReceiveAny(msg => Sender.Tell(ReplyFor(msg)));
private static object ReplyFor(object m) => m switch
{
EnableInstanceCommand e =>
new InstanceLifecycleResponse(e.CommandId, e.InstanceUniqueName, true, null, DateTimeOffset.UtcNow),
BrowseNodeCommand => new BrowseNodeResult([], false, null, null),
DebugSnapshotRequest d => new DebugViewSnapshot(d.InstanceUniqueName, [], [], DateTimeOffset.UtcNow),
RouteToCallRequest r =>
new RouteToCallResponse(r.CorrelationId, true, null, null, DateTimeOffset.UtcNow),
ParkedMessageQueryRequest p => new ParkedMessageQueryResponse(
p.CorrelationId, p.SiteId, [], 0, p.PageNumber, p.PageSize, true, null, DateTimeOffset.UtcNow),
_ => new Akka.Actor.Status.Failure(new InvalidOperationException($"no canned reply for {m.GetType().Name}")),
};
}
}