10 Commits

Author SHA1 Message Date
Joseph Doherty f7c7811940 docs(grpc): Phase 1A live gate — PASS (3 RPCs + restart-reconcile + coexistence); records the Kestrel-drop defect 2026-07-22 19:55:09 -04:00
Joseph Doherty 0e162cb250 fix(grpc): central Kestrel gRPC listener was dropping the :5000 HTTP surface (T1A.2 regression)
Caught on the Phase 1A rig proof: central-a logged only "Now listening on:
http://[::]:8083" and nothing else. Central UI, the Management + Inbound API, and
the /health/* endpoints Traefik + IActiveNodeGate depend on were all gone, with no
startup error — the node booted, joined the cluster and served gRPC fine.

Cause: calling options.ListenAnyIP in ConfigureKestrel puts Kestrel into
explicit-endpoints mode, which SUPPRESSES the URLs from ASPNETCORE_URLS/--urls
entirely — it is not additive, contrary to the comment T1A.2 shipped. Central's whole
HTTP/1 surface lives on that URL (http://+:5000 on the rig, a different port in prod),
so binding only the gRPC port silently deleted it. The site branch has the same shape
but no ASPNETCORE_URLS surface to lose — it binds every port it needs explicitly.

Fix: parse the port(s) from the configured URLs and re-declare them (Http1AndHttp2)
alongside the gRPC port (Http2) in the one ConfigureKestrel call. New
Program.ParseHttpBindPorts + CentralHttpBindPortsTests (14 cases: wildcard/ipv6/
hostname hosts, multi-URL, de-dupe, scheme-default, null/blank, unparseable-skipped).

Why no test caught it originally: unit/E2E tests use TestServer, which never binds
real Kestrel. Only a live node exposes the missing listener — which is exactly what
the rig proof is for.
2026-07-22 19:50:32 -04:00
Joseph Doherty 33b15f10a4 feat(grpc): site-side ICentralTransport seam + gRPC transport (T1A.3)
Introduce ICentralTransport as the site->central choke point inside
SiteCommunicationActor. The seven site->central sends (notification submit/
status, audit + cached-telemetry ingest, reconcile, health, heartbeat) now
delegate to an injected transport instead of owning ClusterClient.Send inline.

- AkkaCentralTransport: verbatim extraction of today's ClusterClient.Send path,
  including the exact sender-forwarding that routes central's reply straight
  back to the waiting Ask. Default when no transport is injected -> behaviour
  unchanged, existing SiteCommunicationActorTests pass as-is.
- GrpcCentralTransport + CentralChannelProvider: dial CentralControlService with
  sticky failover + background failback (1s-doubling-cap-60s), PSK +
  x-scadabridge-site via ControlPlaneCredentials, per-call deadlines mirroring
  today's Ask timeouts. Cross-node retry ONLY on provably-unsent
  connect failures; never on DeadlineExceeded. Heartbeat stays fire-and-forget.
- StaticSitePskProvider: site's single own-key provider (fail-closed).
- CommunicationOptions: CentralTransport flag (default Akka) + CentralGrpcEndpoints
  (validator: required when transport=Grpc). Host selects the impl; the
  ClusterClient is created only on the Akka path.

Tests: actor-with-fake-transport (7 delegations + fault routing + heartbeat
no-fault), AkkaCentralTransport sender-forwarding, GrpcCentralTransport over
in-process TestServer (failover flip, sticky, failback, PSK+header, deadline,
no-retry-on-deadline), validator. Communication.Tests 371 green, Host.Tests 391
green; the three above-seam suites pass unmodified.
2026-07-22 19:29:14 -04:00
Joseph Doherty 780bb9c369 feat(grpc): host CentralControlService on the central node (T1A.2)
Central now ALSO listens for the seven site→central control messages over
gRPC, alongside the existing ClusterClient path. Nothing flips to gRPC yet —
sites keep CentralTransport=Akka (T1A.3's job); central simply starts also
accepting.

- CentralControlGrpcService (Communication.Grpc): decodes each RPC onto the
  SAME in-process message the ClusterClient path carries, Asks the existing
  CentralCommunicationActor (zero handler-logic changes), encodes the reply via
  the T1A.1 mapper. Readiness-gated like SiteStreamGrpcServer.SetReady —
  Unavailable until AkkaHostedService hands the actor over. Heartbeat stays
  fire-and-forget (Tell, always-OK, never gated on readiness). Ingest reuses the
  shared SiteStreamGrpcServer.AuditIngestAskTimeout constant. Fault→status
  mapping is retry-aware: Unavailable (never dispatched, safe to cross-node
  retry) vs DeadlineExceeded/Internal (it ran, do not re-send elsewhere).

- CentralControlAuthInterceptor (Host): a SEPARATE interceptor class, not a
  variant constructor on ControlPlaneAuthInterceptor. Central's model is per-site
  (verify the Bearer token against the key for the site in the required
  x-scadabridge-site header, via ISitePskProvider) where a site verifies its one
  own-key — a genuinely different model. Fail-closed on every branch: missing or
  blank header, unresolvable key, and mismatched token all → PermissionDenied,
  never pass-through. One public constructor only (the explicit-prefix ctor is
  internal), pinned by a reflection test — a second public ctor makes
  Grpc.AspNetCore's GetFactory() throw per-call and silently disables the gate.

- Explicit Kestrel h2c listener on new option ScadaBridge:Node:CentralGrpcPort
  (default 8083, symmetric with sites), mirroring the Site branch. Additive to
  central's :5000 HTTP/1 surface, which is untouched — gRPC does NOT go through
  Traefik (HTTP/1 only). Registered by type on AddGrpc; service mapped with
  MapGrpcService. Port range-validated by NodeOptionsValidator.

- Rig: publish the central gRPC port 9013:8083 / 9014:8083 on both central nodes
  so a later task can exercise it.

Tests: CentralControlEndToEndTests (Host.Tests, TestServer + real interceptor +
real service over a stub actor) proves auth positives/negatives are
distinguishable and covers unary + the ingest bridge shapes; the interceptor is
registered BY TYPE, never in DI. CentralControlAuthInterceptorTests pins the
per-site gate + one-public-ctor invariant. CentralControlGrpcServiceTests
(Communication.Tests, TestKit) covers the readiness gate, fire-and-forget
heartbeat, and the DeadlineExceeded-vs-Unavailable status mapping. No active
<Protobuf> item. Communication.Tests (356) + Host.Tests (384) green.
2026-07-22 18:53:59 -04:00
Joseph Doherty d7455577a8 feat(grpc): central_control.proto + DTO mapper for the 7 site→central control RPCs (T1A.1)
Phase 1A of the ClusterClient→gRPC migration needs a wire contract for the
seven messages SiteCommunicationActor forwards to /user/central-communication.
This lands the contract and its mapper only — hosting (T1A.2) and the site-side
transport seam (T1A.3) follow.

`Protos/central_control.proto` (package scadabridge.centralcontrol.v1, service
CentralControlService) declares SubmitNotification, QueryNotificationStatus,
IngestAuditEvents, IngestCachedTelemetry, ReconcileSite, ReportSiteHealth and
Heartbeat. Note the direction is the inverse of SiteStreamService: here the site
dials and central serves.

Decisions worth recording:

- The two ingest RPCs IMPORT sitestream.proto and reuse AuditEventBatch /
  CachedTelemetryBatch / IngestAck rather than redeclaring them. The site
  telemetry actor already builds those messages, so a second copy would fork one
  wire contract into two kept in lockstep by hand. ForwardState / IngestedAtUtc
  stay off-wire exactly as they are today.
- Heartbeat replies google.protobuf.Empty — it is fire-and-forget and must never
  surface a fault onto the heartbeat timer path.
- The three NULLABLE SiteHealthReport collections travel inside single-field
  wrapper messages (ConnectionEndpointMapDto / TagQualityMapDto /
  NodeStatusListDto). proto3 cannot express presence on repeated/map fields, but
  null and empty genuinely differ here — SiteHealthCollector emits
  `ClusterNodes: _clusterNodes?.ToList()` and the central health surface reads
  null as "not reported", not as "reported empty". Same reasoning drives the
  BoolValue/Int64Value/DoubleValue wrappers on LocalDbReplicationConnected,
  LocalDbOplogBacklog and the two age gauges, whose docs are explicit that null
  is not zero/false.
- ConnectionHealthEnum reserves 0 for UNSPECIFIED instead of mapping Connected
  onto it, and the decoder resolves anything unknown to ConnectionHealth.Error.
  An unrecognised connection state must not render as "healthy".
- Guid? execution ids travel as "D" strings with empty meaning null; a malformed
  non-empty value throws rather than being laundered into "no correlation".
- DateTimeOffset normalizes to a UTC instant (a protobuf Timestamp has no
  offset). Lossless in practice — every producer stamps UTC — and documented +
  asserted rather than left implicit.

SiteCallDtoMapper gains a ToDto(SiteCall) overload. Its doc comment previously
asserted such a method "would be dead code"; that held only while ClusterClient
was the sole path from IngestCachedTelemetryCommand (which carries SiteCall, not
SiteCallOperational) to central. Comment corrected alongside.

Golden tests round-trip every message through a real protobuf encode/decode —
DTO → proto → bytes → proto → DTO — with a fully-populated case and a
null/empty/minimal case for every optional member. Verified to have teeth by
mutation: dropping a scalar, collapsing an empty nullable collection to absent,
and nulling a gauge each fail a test.

Codegen stays CHECKED IN under CentralControlGrpc/ (protoc segfaults in the
linux_arm64 Docker image); no active <Protobuf> item is committed.
docker/regen-proto.sh is generalized to `regen-proto.sh [sitestream|
centralcontrol|all]` — it now injects the ItemGroup rather than unwrapping a
comment, so it no longer depends on there being exactly one Protobuf line, and
it restores the csproj verbatim on every exit path.
2026-07-22 18:28:27 -04:00
Joseph Doherty aa7c5cd138 docs(plans): tick Phase 0 DoD — PR #25 merged to main @ 3fa95555 2026-07-22 18:12:01 -04:00
Joseph Doherty 3fa955556d docs(grpc): record the Playwright result and root-cause both failures
Phase 0's gate doc now carries the full suite picture, not just the rig checks.

Playwright: 170 pass / 2 fail / 1 skip of 173. Both failures were run down to
root cause and both are pre-existing on main, unrelated to this branch (which
touches no EF, CentralUI, Transport or ManagementService file):

- TransportImportTests is a REAL production bug: BundleImporter.cs:1298 opens a
  user-initiated transaction while the central context has EnableRetryOnFailure,
  so SqlServerRetryingExecutionStrategy refuses the split query inside it and
  bundle import fails against real MS SQL. The unit/integration suite cannot see
  it -- the in-memory EF provider has no retrying strategy and BeginTransaction
  is a no-op there.

- SmsNotificationE2ETests is a stale fixture: SID 'ACtest123' (2026-06-19) vs the
  ^AC[0-9a-fA-F]{32}$ guard added 2026-07-10 (40088a21). Failing since then, which
  has also silenced everything after the toast assertion -- including the
  secret-non-leak check on the Auth Token.

Also records that the earlier 44-failure run is void: a concurrent deploy.sh was
recreating the cluster underneath it.

Neither is fixed here; both are out of scope for a PSK-auth branch.
2026-07-22 18:09:07 -04:00
Joseph Doherty 6ef8c7d70a docs(grpc): Phase 0 live gate PASS — record results, the inert-gate defect, and the trap for phases 1A/1B
The gate's first run failed on a defect the green suite could not see: two public
constructors on ControlPlaneAuthInterceptor made Grpc.AspNetCore's activation
throw per call, so correct key, wrong key and no key all produced identical
errors. Recorded in full because the symptom (Unknown / "Exception was thrown by
handler") points at the handler, not at auth, and because phases 1A/1B both add
services to this same interceptor — they must extend DefaultGatedPrefixes rather
than add a second public constructor.

Also records what the gate does NOT cover: live streaming under load, key
rotation on a running pair, and docker-env2 (keyed but neither redeployed nor
gated).
2026-07-22 18:01:11 -04:00
Joseph Doherty 228ff8b428 fix(grpc): one public constructor on ControlPlaneAuthInterceptor — two made the gate inert
Caught by the Phase 0 live gate, not by the suite.

Grpc.AspNetCore registers the interceptor BY TYPE, and
InterceptorRegistration.GetFactory() throws "Multiple constructors accepting all
given argument types have been found" when more than one public constructor is
applicable. The interceptor had two: the DI one and a prefix-set overload added
for later phases.

The failure mode is nasty. The throw happens inside the interceptor pipeline on
every call, so nothing fails at startup — the site node boots, joins, reports
healthy. Every gated call then dies with Unknown / "Exception was thrown by
handler", which reads as a handler bug rather than an auth bug. And it fails
OPEN in the sense that matters least and closed in the sense that matters most:
no call is ever authorized, but no call is ever correctly REFUSED either, so the
rig showed identical errors for a correct key, a wrong key and no key at all.
Live evidence, site-a: three PullAuditEvents calls, three identical
InvalidOperationExceptions in the node log.

Fix: the prefix-set constructor is internal (Host.Tests already has
InternalsVisibleTo). Later phases extend DefaultGatedPrefixes rather than adding
a second public registration shape.

Why the tests missed it, and what changed: ControlPlaneAuthEndToEndTests
registered the interceptor with AddSingleton alongside AddGrpc, so DI handed
back the instance and Grpc.AspNetCore's activation path — the thing that throws
— never ran. The harness now registers exactly as Program.cs does, by type and
not in DI. Plus a direct reflection assertion that the type has exactly one
public constructor, since that is the real invariant and it is cheap to pin.
2026-07-22 17:56:51 -04:00
Joseph Doherty 2ee84af1c0 feat(grpc): PSK-authenticate the site gRPC control plane; drop the vestigial management receptionist registration
Phase 0 of the ClusterClient→gRPC migration
(docs/plans/2026-07-22-clusterclient-to-grpc-plan.md). Standalone hardening: it
closes a gap that exists today and is a precondition for moving command/control
onto gRPC in later phases.

T0.1 — delete the ManagementActor ClusterClientReceptionist registration.
It was built for an out-of-cluster CLI that was never written: the shipped CLI
speaks HTTP Basic to /management, which asks the actor in-process through
ManagementActorHolder. Nothing in the repo ever sent to /user/management. The
actor still runs there; only the cross-boundary advertisement is gone. Six
documents claimed the CLI used ClusterClient — including the CLI's own README
"Architecture Notes" — and are corrected here rather than left to rot.

T0.2 — record, do not port, the dead integration-routing path.
IntegrationCallRequest is unwired at BOTH ends: RouteIntegrationCallAsync has
zero callers anywhere, and RegisterLocalHandler(Integration, …) appears only in
a test, so production always answers "Integration handler not available". It is
excluded from the gRPC contract (28 of 29 commands migrate) rather than
enshrined on an additive-only wire format, and deleting it during a
transport migration would mix a behavioural change into a change whose whole
value is that behaviour is identical. See
docs/known-issues/2026-07-22-integration-call-routing-is-dead-code.md.

T0.3 — preshared-key authentication on SiteStreamService.
The service shipped with no auth at all: plaintext h2c, no interceptor, so
anything that could reach a site node's :8083 could open a live data stream or
read audit rows back via PullAuditEvents/PullSiteCalls. ControlPlaneAuthInterceptor
now gates /sitestream.SiteStreamService/ — modeled on LocalDbSyncAuthInterceptor
(constant-time compare, fail-closed, PermissionDenied) but gating a SET of
service prefixes so phases 1A/1B add services rather than interceptors. LocalDb
sync keeps its own separate key: it authenticates the pair partner, not central,
and collapsing the two would make a site's central-facing key also admit writes
into its database.

Keys are per site (SB-GRPC-PSK-<siteId>), never fleet-wide, so a compromised
site yields only its own. Central attaches them through ControlPlaneCredentials,
which binds CallCredentials to the channel — covering unary and streaming
uniformly, and letting the key resolve asynchronously, which a client
interceptor could not do without blocking. All three central→site channel
creation sites go through it (SiteStreamGrpcClient and both audit pull invokers);
the pull invokers' channel caches are re-keyed by (site, endpoint) because
credentials are per-site and bound to the channel.

Two decisions beyond the plan:

  * StartupValidator now requires GrpcPsk on Site nodes. The plan specified only
    the runtime gate, but fail-closed with no boot check produces a node that
    joins, answers heartbeats and reports healthy while refusing every stream,
    audit pull and telemetry ingest — silent and total. Same reasoning as the
    existing inbound API-key pepper rule.

  * Added Communication:SitePsks as a central-side key map. The plan assumed
    central would read the store, seeded via a dev KEK; the docker rig
    deliberately boots with no master key, so store-only resolution would leave
    it unable to dial its own sites. The store stays primary — it is the only
    source that can serve a site added at runtime — with the map covering
    key-less hosts and one-off pins. Neither source falling back to
    "unauthenticated" is the invariant.

T0.4 — dev keys on both rigs and tests.
34 tests. The seven that matter most exercise a real in-process gRPC stack over
TestServer: the unit tests on either side of the wire would both stay green if
the halves disagreed, and gRPC refuses call credentials on a plaintext channel
by default — the UnsafeUseInsecureChannelCallCredentials opt-in is only provable
by making a real call. They confirm correct key passes on unary AND streaming,
wrong key and no-credentials both get PermissionDenied, and an unresolvable key
fails the call with nothing reaching the service.

OPERATIONAL: a site node upgraded to this build without a key will not boot.
That includes the gitignored deploy/wonder-app-vd03/ overlay.
2026-07-22 17:51:09 -04:00
73 changed files with 13791 additions and 296 deletions
+1 -1
View File
@@ -138,7 +138,7 @@ Other peers in the `scadaproj` family (see `scadaproj/CLAUDE.md` for details): `
- **All clusters share ONE ActorSystem name**, `"scadabridge"` — hardcoded at `AkkaHostedService.cs:191`. Central and each site are separate clusters *only* by seed-node partitioning. This is required, not incidental: Akka.Remote address matching means a ClusterClient could not reach a differently-named system. - **All clusters share ONE ActorSystem name**, `"scadabridge"` — hardcoded at `AkkaHostedService.cs:191`. Central and each site are separate clusters *only* by seed-node partitioning. This is required, not incidental: Akka.Remote address matching means a ClusterClient could not reach a differently-named system.
- **`ActiveNodeEvaluator.SelfIsOldestUp` is THE single definition of "active node"** (`Communication/ClusterState/ActiveNodeEvaluator.cs`) — the **oldest Up member** in a role scope, and explicitly **never `cluster.State.Leader`**: leadership (lowest address) is an Akka-internal concept that diverges from singleton placement permanently once the original first node restarts and rejoins, and both sides claim it during a partition. The equivalence *oldest-Up == where `ClusterSingletonManager` places singletons* **is** the design. `ClusterActivityEvaluator.SelfIsOldest`, the S&F delivery gate, `/health/active` and the heartbeat `IsActive` stamp all delegate here. - **`ActiveNodeEvaluator.SelfIsOldestUp` is THE single definition of "active node"** (`Communication/ClusterState/ActiveNodeEvaluator.cs`) — the **oldest Up member** in a role scope, and explicitly **never `cluster.State.Leader`**: leadership (lowest address) is an Akka-internal concept that diverges from singleton placement permanently once the original first node restarts and rejoins, and both sides claim it during a partition. The equivalence *oldest-Up == where `ClusterSingletonManager` places singletons* **is** the design. `ClusterActivityEvaluator.SelfIsOldest`, the S&F delivery gate, `/health/active` and the heartbeat `IsActive` stamp all delegate here.
- Site nodes carry **two Akka roles**: the base `Site` plus a site-specific `site-{SiteId}` (`BuildRoles`, `AkkaHostedService.cs:386`). Singletons scope to the **site-specific** role. - Site nodes carry **two Akka roles**: the base `Site` plus a site-specific `site-{SiteId}` (`BuildRoles`, `AkkaHostedService.cs:386`). Singletons scope to the **site-specific** role.
- **Neither inter-cluster transport is authenticated or encrypted.** Akka remoting sets no `enable-ssl`, no secure cookie, no `trusted-selection-paths`; the gRPC listener is **h2c** and `LocalDbSyncAuthInterceptor` gates *only* `/localdb_sync.v1.LocalDbSync/`, so the whole `SiteStreamService` surface — including the `PullAuditEvents`/`PullSiteCalls` RPCs that return audit rows — is callable by anyone who can reach :8083. The boundary assumes a trusted network. (The HTTP fetch token and the LocalDb sync interceptor are the only authenticated pieces.) - **The gRPC boundary is authenticated (PSK) as of 2026-07-22; Akka remoting still is not, and nothing is encrypted.** Akka remoting sets no `enable-ssl`, no secure cookie, no `trusted-selection-paths` — so the ClusterClient command/control path remains open to anyone who can reach the remoting port, and the boundary still assumes a trusted network. The gRPC listener stays **h2c**, but `SiteStreamService` is no longer open: `ControlPlaneAuthInterceptor` (`Host/ControlPlaneAuthInterceptor.cs`) gates `/sitestream.SiteStreamService/` — including the `PullAuditEvents`/`PullSiteCalls` RPCs that return audit rows — against a **per-site preshared key**, fail-closed, constant-time compared, alongside the separate `LocalDbSyncAuthInterceptor` on `/localdb_sync.v1.LocalDbSync/` with its own separate key. **Site side:** `ScadaBridge:Communication:GrpcPsk`, in production `${secret:SB-GRPC-PSK-<siteId>}`, and **`StartupValidator` refuses to boot a site node without it** (an unset key would leave the node healthy-looking but serving nothing). **Central side:** `SitePskProvider` resolves `SB-GRPC-PSK-{siteId}` from the secrets store at channel-build time (sites are added at runtime, so no boot-time expansion is possible), with `ScadaBridge:Communication:SitePsks:{siteId}` as an override for hosts running without a master key — the docker rig uses the latter. One key per site, never fleet-wide. A bearer token over h2c is readable and replayable on-path; TLS is the follow-on hardening and needs no change to this design. Introduced by Phase 0 of the ClusterClient→gRPC migration (`docs/plans/2026-07-22-clusterclient-to-grpc-plan.md`).
- **Akka frame size is the default 128 KB with `log-frame-size-exceeding` off**, and no custom serializer is configured — so payload carried over ClusterClient is JSON-escaped a second time by the default Newtonsoft serializer, roughly doubling it. Over the limit the transport drops **that one message** without tearing down the association (heartbeats keep flowing, the site still reports healthy) and the central Ask simply times out. See `docs/known-issues/2026-06-26-deploy-config-exceeds-akka-frame-size.md`; `DeployArtifactsCommand` still carries payload and remains exposed. - **Akka frame size is the default 128 KB with `log-frame-size-exceeding` off**, and no custom serializer is configured — so payload carried over ClusterClient is JSON-escaped a second time by the default Newtonsoft serializer, roughly doubling it. Over the limit the transport drops **that one message** without tearing down the association (heartbeats keep flowing, the site still reports healthy) and the central Ask simply times out. See `docs/known-issues/2026-06-26-deploy-config-exceeds-akka-frame-size.md`; `DeployArtifactsCommand` still carries payload and remains exposed.
- gRPC streaming channel — **note the direction is inverted from the data flow**: data moves site→central, but each **site node hosts the gRPC server** (`SiteStreamGrpcServer`, Kestrel h2c, port 8083, mapped **only in the Site branch** of `Program.cs`) and **central is the client**, dialling in. There is **no gRPC server on central at all**, which is why the two `Ingest*` unary RPCs — documented as a "central-side ingest surface" — are dead in practice (acknowledged in `AkkaHostedService.cs:510-519`); sites reach central over ClusterClient instead. Central creates per-site `SiteStreamGrpcClient` via `SiteStreamGrpcClientFactory`, keyed **`(siteId, endpoint)`** — the key was widened from site-only to fix an arch-review High where one session's NodeA→NodeB flip disposed a channel another session was still using. Site entity has GrpcNodeAAddress/GrpcNodeBAddress fields. Proto: `sitestream.proto`, **6 RPCs** (2 server-streaming: `SubscribeInstance`, `SubscribeSite` (site-wide, alarm-only); 4 unary: `IngestAuditEvents`, `IngestCachedTelemetry`, `PullAuditEvents`, `PullSiteCalls`), `SiteStreamEvent` (oneof: AttributeValueUpdate, AlarmStateUpdate). Field numbers are never reused; evolution is additive only (`AlarmStateUpdate` grew 7→23 fields for the native-alarm mirror). Generated C# is **vendored** under `Communication/SiteStreamGrpc/` with the `<Protobuf>` include commented out — regeneration is a manual toggle-build-copy-untoggle. DebugStreamEvent message removed (no longer flows through ClusterClient). - gRPC streaming channel — **note the direction is inverted from the data flow**: data moves site→central, but each **site node hosts the gRPC server** (`SiteStreamGrpcServer`, Kestrel h2c, port 8083, mapped **only in the Site branch** of `Program.cs`) and **central is the client**, dialling in. There is **no gRPC server on central at all**, which is why the two `Ingest*` unary RPCs — documented as a "central-side ingest surface" — are dead in practice (acknowledged in `AkkaHostedService.cs:510-519`); sites reach central over ClusterClient instead. Central creates per-site `SiteStreamGrpcClient` via `SiteStreamGrpcClientFactory`, keyed **`(siteId, endpoint)`** — the key was widened from site-only to fix an arch-review High where one session's NodeA→NodeB flip disposed a channel another session was still using. Site entity has GrpcNodeAAddress/GrpcNodeBAddress fields. Proto: `sitestream.proto`, **6 RPCs** (2 server-streaming: `SubscribeInstance`, `SubscribeSite` (site-wide, alarm-only); 4 unary: `IngestAuditEvents`, `IngestCachedTelemetry`, `PullAuditEvents`, `PullSiteCalls`), `SiteStreamEvent` (oneof: AttributeValueUpdate, AlarmStateUpdate). Field numbers are never reused; evolution is additive only (`AlarmStateUpdate` grew 7→23 fields for the native-alarm mirror). Generated C# is **vendored** under `Communication/SiteStreamGrpc/` with the `<Protobuf>` include commented out — regeneration is a manual toggle-build-copy-untoggle. DebugStreamEvent message removed (no longer flows through ClusterClient).
- Native alarms: a read-only mirror of native alarms from OPC UA Alarms & Conditions servers and the MxAccess Gateway, unified onto an A&C-style condition model (`AlarmConditionState`: orthogonal Active/Acked/Confirmed/Shelved/Suppressed + 01000 severity) plus an `AlarmKind` discriminator (Computed/NativeOpcUa/NativeMxAccess). New DCL capability seam `IAlarmSubscribableConnection` (implemented by the OPC UA and MxGateway adapters); the `DataConnectionActor` opens ONE alarm feed per connection and routes transitions to instances by source-object reference. A `NativeAlarmActor` (peer to the computed `AlarmActor` under `InstanceActor`) mirrors one source binding: snapshot atomic-swap on (re)subscribe, retention (drops once inactive+acked), per-source cap, and site SQLite persistence (`native_alarm_state`, survives failover, cleared on redeploy/undeploy — mirrors static overrides). State streams to central over the additively-enriched gRPC `AlarmStateUpdate` (the existing computed `AlarmStateChanged` was enriched additively) and seeds via the DebugView snapshot. Authoring: `TemplateNativeAlarmSource` / `InstanceNativeAlarmSourceOverride` entities flatten to `ResolvedNativeAlarmSource` (inherit/compose/override); management commands + ManagementActor handlers + CLI (`template/instance native-alarm-source`) + Central UI (template editor tab + instance override panel) + enriched DebugView alarm table. Read-only — no ack-back; no central tables. - Native alarms: a read-only mirror of native alarms from OPC UA Alarms & Conditions servers and the MxAccess Gateway, unified onto an A&C-style condition model (`AlarmConditionState`: orthogonal Active/Acked/Confirmed/Shelved/Suppressed + 01000 severity) plus an `AlarmKind` discriminator (Computed/NativeOpcUa/NativeMxAccess). New DCL capability seam `IAlarmSubscribableConnection` (implemented by the OPC UA and MxGateway adapters); the `DataConnectionActor` opens ONE alarm feed per connection and routes transitions to instances by source-object reference. A `NativeAlarmActor` (peer to the computed `AlarmActor` under `InstanceActor`) mirrors one source binding: snapshot atomic-swap on (re)subscribe, retention (drops once inactive+acked), per-source cap, and site SQLite persistence (`native_alarm_state`, survives failover, cleared on redeploy/undeploy — mirrors static overrides). State streams to central over the additively-enriched gRPC `AlarmStateUpdate` (the existing computed `AlarmStateChanged` was enriched additively) and seeds via the DebugView snapshot. Authoring: `TemplateNativeAlarmSource` / `InstanceNativeAlarmSourceOverride` entities flatten to `ResolvedNativeAlarmSource` (inherit/compose/override); management commands + ManagementActor handlers + CLI (`template/instance native-alarm-source`) + Central UI (template editor tab + instance override panel) + enriched DebugView alarm table. Read-only — no ack-back; no central tables.
+8
View File
@@ -16,6 +16,10 @@ services:
# pepper per the "different per environment" guidance; real deployments inject a # pepper per the "different per environment" guidance; real deployments inject a
# true secret out-of-band, never from source control. Both Central nodes share it. # true secret out-of-band, never from source control. Both Central nodes share it.
ScadaBridge__InboundApi__ApiKeyPepper: "dev-only-insecure-pepper-env2-cluster-0001" ScadaBridge__InboundApi__ApiKeyPepper: "dev-only-insecure-pepper-env2-cluster-0001"
# DEV-ONLY gRPC control-plane preshared key for site-x — NOT a real secret.
# Must match ScadaBridge:Communication:GrpcPsk in site-x-node-*/appsettings.Site.json.
# Production seeds SB-GRPC-PSK-<siteId> into the secret store instead.
ScadaBridge__Communication__SitePsks__site-x: "dev-grpc-psk-docker-env2-site-x"
ports: ports:
- "9101:5000" # Web UI + Inbound API - "9101:5000" # Web UI + Inbound API
- "9111:8081" # Akka remoting - "9111:8081" # Akka remoting
@@ -43,6 +47,10 @@ services:
# pepper per the "different per environment" guidance; real deployments inject a # pepper per the "different per environment" guidance; real deployments inject a
# true secret out-of-band, never from source control. Both Central nodes share it. # true secret out-of-band, never from source control. Both Central nodes share it.
ScadaBridge__InboundApi__ApiKeyPepper: "dev-only-insecure-pepper-env2-cluster-0001" ScadaBridge__InboundApi__ApiKeyPepper: "dev-only-insecure-pepper-env2-cluster-0001"
# DEV-ONLY gRPC control-plane preshared key for site-x — NOT a real secret.
# Must match ScadaBridge:Communication:GrpcPsk in site-x-node-*/appsettings.Site.json.
# Production seeds SB-GRPC-PSK-<siteId> into the secret store instead.
ScadaBridge__Communication__SitePsks__site-x: "dev-grpc-psk-docker-env2-site-x"
ports: ports:
- "9102:5000" # Web UI + Inbound API - "9102:5000" # Web UI + Inbound API
- "9112:8081" # Akka remoting - "9112:8081" # Akka remoting
@@ -40,6 +40,13 @@
"SqliteDbPath": "/app/data/store-and-forward.db" "SqliteDbPath": "/app/data/store-and-forward.db"
}, },
"Communication": { "Communication": {
// DEV-ONLY control-plane preshared key — NOT a real secret. Must be
// IDENTICAL on both nodes of the pair and match the central-side entry in
// ScadaBridge__Communication__SitePsks__<siteId> (docker-compose.yml).
// Production supplies this as ${secret:SB-GRPC-PSK-<siteId>}. Without it the
// node fails StartupValidator: the gate is fail-closed, so an unset key would
// refuse every SiteStream call while the node still looked healthy.
"GrpcPsk": "dev-grpc-psk-docker-env2-site-x",
"CentralContactPoints": [ "CentralContactPoints": [
"akka.tcp://scadabridge@scadabridge-env2-central-a:8081", "akka.tcp://scadabridge@scadabridge-env2-central-a:8081",
"akka.tcp://scadabridge@scadabridge-env2-central-b:8081" "akka.tcp://scadabridge@scadabridge-env2-central-b:8081"
@@ -40,6 +40,13 @@
"SqliteDbPath": "/app/data/store-and-forward.db" "SqliteDbPath": "/app/data/store-and-forward.db"
}, },
"Communication": { "Communication": {
// DEV-ONLY control-plane preshared key — NOT a real secret. Must be
// IDENTICAL on both nodes of the pair and match the central-side entry in
// ScadaBridge__Communication__SitePsks__<siteId> (docker-compose.yml).
// Production supplies this as ${secret:SB-GRPC-PSK-<siteId>}. Without it the
// node fails StartupValidator: the gate is fail-closed, so an unset key would
// refuse every SiteStream call while the node still looked healthy.
"GrpcPsk": "dev-grpc-psk-docker-env2-site-x",
"CentralContactPoints": [ "CentralContactPoints": [
"akka.tcp://scadabridge@scadabridge-env2-central-a:8081", "akka.tcp://scadabridge@scadabridge-env2-central-a:8081",
"akka.tcp://scadabridge@scadabridge-env2-central-b:8081" "akka.tcp://scadabridge@scadabridge-env2-central-b:8081"
+25
View File
@@ -120,6 +120,31 @@ docker/
└── logs/ └── logs/
``` ```
## gRPC control-plane keys (dev)
The site gRPC service (`SiteStreamService` on 8083 — live subscriptions, audit pull,
cached-telemetry ingest) is gated by a preshared key, and the gate is **fail-closed**: a site node
with no key refuses every call, and `StartupValidator` refuses to boot it at all. So the rig
carries dev keys, one per site:
| Where | Setting | Value |
|---|---|---|
| `site-{a,b,c}-node-*/appsettings.Site.json` | `ScadaBridge:Communication:GrpcPsk` | `dev-grpc-psk-docker-site-{a,b,c}` |
| `docker-compose.yml`, both central nodes | `ScadaBridge__Communication__SitePsks__site-{a,b,c}` | same value |
Both nodes of a pair carry the same key; each site's key is different from the others'. The
central half lives in compose env rather than the mounted `appsettings.Central.json`, which by
convention holds no plaintext credentials. Production uses `${secret:SB-GRPC-PSK-<siteId>}` on
the site and the matching secret in central's store — see
[`docs/deployment/topology-guide.md`](../docs/deployment/topology-guide.md).
**These are not real secrets and are committed deliberately**, exactly like the LocalDb sync key
(`dev-site-a-localdb-sync-key`) beside them. The two are separate keys on purpose: the LocalDb one
authenticates the *pair partner* for database replication, not central.
If you add a site to the rig, add its key in both places or its streams will fail with
`PermissionDenied`.
## Commands ## Commands
### Initial Setup ### Initial Setup
+20
View File
@@ -27,9 +27,19 @@ services:
ScadaBridge__Database__MachineDataDb: "Server=scadabridge-mssql,1433;Database=ScadaBridgeMachineData;User Id=scadabridge_app;Password=ScadaBridge_Dev1#;TrustServerCertificate=true" ScadaBridge__Database__MachineDataDb: "Server=scadabridge-mssql,1433;Database=ScadaBridgeMachineData;User Id=scadabridge_app;Password=ScadaBridge_Dev1#;TrustServerCertificate=true"
ScadaBridge__Security__Ldap__ServiceAccountPassword: "serviceaccount123" ScadaBridge__Security__Ldap__ServiceAccountPassword: "serviceaccount123"
ScadaBridge__Security__JwtSigningKey: "scadabridge-dev-jwt-signing-key-must-be-at-least-32-characters-long" ScadaBridge__Security__JwtSigningKey: "scadabridge-dev-jwt-signing-key-must-be-at-least-32-characters-long"
# DEV-ONLY gRPC control-plane preshared keys, one per site — NOT real secrets.
# Central verifies/presents these; each site node carries the same value as
# ScadaBridge:Communication:GrpcPsk in its mounted appsettings.Site.json. Kept as
# env overrides (not in the mounted central appsettings) so that file stays free of
# plaintext credentials. Production instead seeds SB-GRPC-PSK-<siteId> into the
# secret store, which is also the only source that can serve a site added at runtime.
ScadaBridge__Communication__SitePsks__site-a: "dev-grpc-psk-docker-site-a"
ScadaBridge__Communication__SitePsks__site-b: "dev-grpc-psk-docker-site-b"
ScadaBridge__Communication__SitePsks__site-c: "dev-grpc-psk-docker-site-c"
ports: ports:
- "9001:5000" # Web UI + Inbound API - "9001:5000" # Web UI + Inbound API
- "9011:8081" # Akka remoting (host access for CLI/debugging) - "9011:8081" # Akka remoting (host access for CLI/debugging)
- "9013:8083" # gRPC control plane (CentralControlService, T1A.2)
volumes: volumes:
- ./central-node-a/appsettings.Central.json:/app/appsettings.Central.json:ro - ./central-node-a/appsettings.Central.json:/app/appsettings.Central.json:ro
- ./central-node-a/logs:/app/logs - ./central-node-a/logs:/app/logs
@@ -65,9 +75,19 @@ services:
ScadaBridge__Database__MachineDataDb: "Server=scadabridge-mssql,1433;Database=ScadaBridgeMachineData;User Id=scadabridge_app;Password=ScadaBridge_Dev1#;TrustServerCertificate=true" ScadaBridge__Database__MachineDataDb: "Server=scadabridge-mssql,1433;Database=ScadaBridgeMachineData;User Id=scadabridge_app;Password=ScadaBridge_Dev1#;TrustServerCertificate=true"
ScadaBridge__Security__Ldap__ServiceAccountPassword: "serviceaccount123" ScadaBridge__Security__Ldap__ServiceAccountPassword: "serviceaccount123"
ScadaBridge__Security__JwtSigningKey: "scadabridge-dev-jwt-signing-key-must-be-at-least-32-characters-long" ScadaBridge__Security__JwtSigningKey: "scadabridge-dev-jwt-signing-key-must-be-at-least-32-characters-long"
# DEV-ONLY gRPC control-plane preshared keys, one per site — NOT real secrets.
# Central verifies/presents these; each site node carries the same value as
# ScadaBridge:Communication:GrpcPsk in its mounted appsettings.Site.json. Kept as
# env overrides (not in the mounted central appsettings) so that file stays free of
# plaintext credentials. Production instead seeds SB-GRPC-PSK-<siteId> into the
# secret store, which is also the only source that can serve a site added at runtime.
ScadaBridge__Communication__SitePsks__site-a: "dev-grpc-psk-docker-site-a"
ScadaBridge__Communication__SitePsks__site-b: "dev-grpc-psk-docker-site-b"
ScadaBridge__Communication__SitePsks__site-c: "dev-grpc-psk-docker-site-c"
ports: ports:
- "9002:5000" # Web UI + Inbound API - "9002:5000" # Web UI + Inbound API
- "9012:8081" # Akka remoting - "9012:8081" # Akka remoting
- "9014:8083" # gRPC control plane (CentralControlService, T1A.2)
volumes: volumes:
- ./central-node-b/appsettings.Central.json:/app/appsettings.Central.json:ro - ./central-node-b/appsettings.Central.json:/app/appsettings.Central.json:ro
- ./central-node-b/logs:/app/logs - ./central-node-b/logs:/app/logs
+75 -49
View File
@@ -1,92 +1,118 @@
#!/usr/bin/env bash #!/usr/bin/env bash
# #
# Regenerates the gRPC C# files from sitestream.proto. # Regenerates the gRPC C# files from the Communication project's .proto files.
# #
# Background: protoc (linux/arm64) segfaults inside our Docker build container # Background: protoc (linux/arm64) segfaults inside our Docker build container
# (Grpc.Tools 2.71.0). As a workaround the generated Sitestream.cs + # (Grpc.Tools 2.71.0). As a workaround the generated C# is checked into
# SitestreamGrpc.cs are checked into src/ZB.MOM.WW.ScadaBridge.Communication/SiteStreamGrpc/ # src/ZB.MOM.WW.ScadaBridge.Communication/SiteStreamGrpc/ for sitestream.proto
# and the Protobuf ItemGroup in the .csproj is commented out — Docker just # and CentralControlGrpc/ for central_control.proto — and the Protobuf ItemGroup
# compiles the checked-in C# files. # in the .csproj is commented out, so Docker just compiles the checked-in files.
# #
# Run this script ON YOUR DEV MACHINE whenever Protos/sitestream.proto changes: # Run this script ON YOUR DEV MACHINE whenever a .proto changes:
# #
# 1. Temporarily uncomments the Protobuf ItemGroup so Grpc.Tools runs. # docker/regen-proto.sh [sitestream|centralcontrol|all] (default: all)
# 2. dotnet build (regen writes fresh files to obj/). #
# 3. Copies the regenerated files back into SiteStreamGrpc/. # 1. Injects a Protobuf ItemGroup for the selected proto(s) so Grpc.Tools runs.
# 4. Re-comments the Protobuf ItemGroup so Docker builds stay safe. # 2. Deletes the stale checked-in C# so a failed regen is obvious.
# 3. dotnet build (regen writes fresh files to obj/).
# 4. Copies the regenerated files back into the source tree.
# 5. Restores the original csproj so no active Protobuf item is left behind.
#
# Only the SELECTED protos get a Protobuf item. Enabling one whose generated C#
# is still checked in would define every generated type twice, which is why the
# per-proto selection exists. central_control.proto imports sitestream.proto,
# but protoc resolves that from the project-relative path — the import needs no
# Protobuf item of its own.
# #
# Once we move to a Dockerfile base image that ships a working linux/arm64 # Once we move to a Dockerfile base image that ships a working linux/arm64
# protoc, this script can be retired and Docker can regen the proto on every # protoc, this script can be retired and Docker can regen the protos on every
# build like every other normal .NET project. # build like every other normal .NET project.
set -euo pipefail set -euo pipefail
TARGET="${1:-all}"
case "$TARGET" in
sitestream|centralcontrol|all) ;;
*) echo "usage: $0 [sitestream|centralcontrol|all]" >&2; exit 2 ;;
esac
SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)" SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)"
REPO_ROOT="$(cd "$SCRIPT_DIR/.." && pwd)" REPO_ROOT="$(cd "$SCRIPT_DIR/.." && pwd)"
COMM_DIR="$REPO_ROOT/src/ZB.MOM.WW.ScadaBridge.Communication" COMM_DIR="$REPO_ROOT/src/ZB.MOM.WW.ScadaBridge.Communication"
CSPROJ="$COMM_DIR/ZB.MOM.WW.ScadaBridge.Communication.csproj" CSPROJ="$COMM_DIR/ZB.MOM.WW.ScadaBridge.Communication.csproj"
GEN_DIR="$COMM_DIR/SiteStreamGrpc" GEN="$COMM_DIR/obj/Debug/net10.0/Protos"
echo "=== Regenerating gRPC files from sitestream.proto ===" echo "=== Regenerating gRPC files ($TARGET) ==="
if [[ ! -f "$CSPROJ" ]]; then if [[ ! -f "$CSPROJ" ]]; then
echo "ERROR: csproj not found at $CSPROJ" >&2 echo "ERROR: csproj not found at $CSPROJ" >&2
exit 1 exit 1
fi fi
# Backup so we can always restore the comment state on failure. # Backup so we can always restore the comment state on failure. Leaving the
# csproj with an active Protobuf item is the one outcome that breaks Docker, so
# every exit path restores this copy.
BACKUP="$(mktemp)" BACKUP="$(mktemp)"
cp "$CSPROJ" "$BACKUP" cp "$CSPROJ" "$BACKUP"
trap 'cp "$BACKUP" "$CSPROJ"; rm -f "$BACKUP"; echo "Restored csproj from backup."' ERR trap 'cp "$BACKUP" "$CSPROJ"; rm -f "$BACKUP"; echo "Restored csproj from backup."' ERR
# 1. Uncomment the Protobuf ItemGroup (strip the surrounding <!-- ... --> wrapper). # 1. Inject an ItemGroup holding just the selected protos, immediately before
python3 - <<PY # the closing </Project>. The documented commented-out block is left alone.
import re, pathlib python3 - "$CSPROJ" "$TARGET" <<'PY'
p = pathlib.Path("$CSPROJ") import pathlib, sys
src = p.read_text()
# Find the commented Protobuf block and unwrap it. csproj, target = pathlib.Path(sys.argv[1]), sys.argv[2]
new = re.sub(
r"<!--\s*\n(\s*<ItemGroup>\s*\n\s*<Protobuf [^>]*/>\s*\n\s*</ItemGroup>)\s*\n\s*-->", protos = []
r"\1", if target in ("sitestream", "all"):
src, protos.append("sitestream.proto")
count=1, if target in ("centralcontrol", "all"):
) protos.append("central_control.proto")
if new == src:
raise SystemExit("Couldn't find commented Protobuf ItemGroup to enable.") items = "\n".join(
p.write_text(new) f' <Protobuf Include="Protos\\{p}" GrpcServices="Both" />' for p in protos)
block = f" <ItemGroup>\n{items}\n </ItemGroup>\n\n</Project>"
src = csproj.read_text()
if "</Project>" not in src:
raise SystemExit("Couldn't find </Project> to inject the Protobuf ItemGroup before.")
csproj.write_text(src.replace("</Project>", block, 1))
PY PY
# 2. Delete the stale files so any failure to regen is obvious. # 2. Delete the stale files so any failure to regen is obvious.
rm -f "$GEN_DIR/Sitestream.cs" "$GEN_DIR/SitestreamGrpc.cs" if [[ "$TARGET" == "sitestream" || "$TARGET" == "all" ]]; then
rm -f "$COMM_DIR/SiteStreamGrpc/Sitestream.cs" "$COMM_DIR/SiteStreamGrpc/SitestreamGrpc.cs"
fi
if [[ "$TARGET" == "centralcontrol" || "$TARGET" == "all" ]]; then
rm -f "$COMM_DIR/CentralControlGrpc/CentralControl.cs" \
"$COMM_DIR/CentralControlGrpc/CentralControlGrpc.cs"
fi
# 3. Regenerate by building. # 3. Regenerate by building.
echo "Building Communication project (regen)..." echo "Building Communication project (regen)..."
dotnet build "$CSPROJ" --nologo -v minimal | tail -5 dotnet build "$CSPROJ" --nologo -v minimal | tail -5
# 4. Copy generated files back into the source tree. # 4. Copy generated files back into the source tree.
mkdir -p "$GEN_DIR" if [[ "$TARGET" == "sitestream" || "$TARGET" == "all" ]]; then
cp "$COMM_DIR/obj/Debug/net10.0/Protos/Sitestream.cs" "$GEN_DIR/Sitestream.cs" mkdir -p "$COMM_DIR/SiteStreamGrpc"
cp "$COMM_DIR/obj/Debug/net10.0/Protos/SitestreamGrpc.cs" "$GEN_DIR/SitestreamGrpc.cs" cp "$GEN/Sitestream.cs" "$GEN/SitestreamGrpc.cs" "$COMM_DIR/SiteStreamGrpc/"
echo "Copied regenerated files to $GEN_DIR/" echo "Copied regenerated files to SiteStreamGrpc/"
fi
# 5. Re-comment the Protobuf ItemGroup so Docker builds keep working. if [[ "$TARGET" == "centralcontrol" || "$TARGET" == "all" ]]; then
python3 - <<PY mkdir -p "$COMM_DIR/CentralControlGrpc"
import re, pathlib cp "$GEN/CentralControl.cs" "$GEN/CentralControlGrpc.cs" "$COMM_DIR/CentralControlGrpc/"
p = pathlib.Path("$CSPROJ") echo "Copied regenerated files to CentralControlGrpc/"
src = p.read_text() fi
new = re.sub(
r"(\s*<ItemGroup>\s*\n\s*<Protobuf [^>]*/>\s*\n\s*</ItemGroup>)",
r"\n <!--\1\n -->",
src,
count=1,
)
p.write_text(new)
PY
# 5. Restore the backed-up csproj — i.e. drop the injected ItemGroup — so Docker
# builds keep working.
cp "$BACKUP" "$CSPROJ"
rm -f "$BACKUP" rm -f "$BACKUP"
trap - ERR trap - ERR
echo "" echo ""
echo "Done. Review and commit:" echo "Done. Review and commit:"
echo " git diff src/ZB.MOM.WW.ScadaBridge.Communication/Protos/sitestream.proto" echo " git diff src/ZB.MOM.WW.ScadaBridge.Communication/Protos/"
echo " git diff src/ZB.MOM.WW.ScadaBridge.Communication/SiteStreamGrpc/" echo " git diff src/ZB.MOM.WW.ScadaBridge.Communication/SiteStreamGrpc/"
echo " git diff src/ZB.MOM.WW.ScadaBridge.Communication/CentralControlGrpc/"
echo " git diff -- src/ZB.MOM.WW.ScadaBridge.Communication/*.csproj # must be EMPTY"
@@ -41,6 +41,13 @@
"SqliteDbPath": "/app/data/store-and-forward.db" "SqliteDbPath": "/app/data/store-and-forward.db"
}, },
"Communication": { "Communication": {
// DEV-ONLY control-plane preshared key — NOT a real secret. Must be
// IDENTICAL on both nodes of the pair and match the central-side entry in
// ScadaBridge__Communication__SitePsks__<siteId> (docker-compose.yml).
// Production supplies this as ${secret:SB-GRPC-PSK-<siteId>}. Without it the
// node fails StartupValidator: the gate is fail-closed, so an unset key would
// refuse every SiteStream call while the node still looked healthy.
"GrpcPsk": "dev-grpc-psk-docker-site-a",
"CentralContactPoints": [ "CentralContactPoints": [
"akka.tcp://scadabridge@scadabridge-central-a:8081", "akka.tcp://scadabridge@scadabridge-central-a:8081",
"akka.tcp://scadabridge@scadabridge-central-b:8081" "akka.tcp://scadabridge@scadabridge-central-b:8081"
@@ -41,6 +41,13 @@
"SqliteDbPath": "/app/data/store-and-forward.db" "SqliteDbPath": "/app/data/store-and-forward.db"
}, },
"Communication": { "Communication": {
// DEV-ONLY control-plane preshared key — NOT a real secret. Must be
// IDENTICAL on both nodes of the pair and match the central-side entry in
// ScadaBridge__Communication__SitePsks__<siteId> (docker-compose.yml).
// Production supplies this as ${secret:SB-GRPC-PSK-<siteId>}. Without it the
// node fails StartupValidator: the gate is fail-closed, so an unset key would
// refuse every SiteStream call while the node still looked healthy.
"GrpcPsk": "dev-grpc-psk-docker-site-a",
"CentralContactPoints": [ "CentralContactPoints": [
"akka.tcp://scadabridge@scadabridge-central-a:8081", "akka.tcp://scadabridge@scadabridge-central-a:8081",
"akka.tcp://scadabridge@scadabridge-central-b:8081" "akka.tcp://scadabridge@scadabridge-central-b:8081"
@@ -41,6 +41,13 @@
"SqliteDbPath": "/app/data/store-and-forward.db" "SqliteDbPath": "/app/data/store-and-forward.db"
}, },
"Communication": { "Communication": {
// DEV-ONLY control-plane preshared key — NOT a real secret. Must be
// IDENTICAL on both nodes of the pair and match the central-side entry in
// ScadaBridge__Communication__SitePsks__<siteId> (docker-compose.yml).
// Production supplies this as ${secret:SB-GRPC-PSK-<siteId>}. Without it the
// node fails StartupValidator: the gate is fail-closed, so an unset key would
// refuse every SiteStream call while the node still looked healthy.
"GrpcPsk": "dev-grpc-psk-docker-site-b",
"CentralContactPoints": [ "CentralContactPoints": [
"akka.tcp://scadabridge@scadabridge-central-a:8081", "akka.tcp://scadabridge@scadabridge-central-a:8081",
"akka.tcp://scadabridge@scadabridge-central-b:8081" "akka.tcp://scadabridge@scadabridge-central-b:8081"
@@ -41,6 +41,13 @@
"SqliteDbPath": "/app/data/store-and-forward.db" "SqliteDbPath": "/app/data/store-and-forward.db"
}, },
"Communication": { "Communication": {
// DEV-ONLY control-plane preshared key — NOT a real secret. Must be
// IDENTICAL on both nodes of the pair and match the central-side entry in
// ScadaBridge__Communication__SitePsks__<siteId> (docker-compose.yml).
// Production supplies this as ${secret:SB-GRPC-PSK-<siteId>}. Without it the
// node fails StartupValidator: the gate is fail-closed, so an unset key would
// refuse every SiteStream call while the node still looked healthy.
"GrpcPsk": "dev-grpc-psk-docker-site-b",
"CentralContactPoints": [ "CentralContactPoints": [
"akka.tcp://scadabridge@scadabridge-central-a:8081", "akka.tcp://scadabridge@scadabridge-central-a:8081",
"akka.tcp://scadabridge@scadabridge-central-b:8081" "akka.tcp://scadabridge@scadabridge-central-b:8081"
@@ -41,6 +41,13 @@
"SqliteDbPath": "/app/data/store-and-forward.db" "SqliteDbPath": "/app/data/store-and-forward.db"
}, },
"Communication": { "Communication": {
// DEV-ONLY control-plane preshared key — NOT a real secret. Must be
// IDENTICAL on both nodes of the pair and match the central-side entry in
// ScadaBridge__Communication__SitePsks__<siteId> (docker-compose.yml).
// Production supplies this as ${secret:SB-GRPC-PSK-<siteId>}. Without it the
// node fails StartupValidator: the gate is fail-closed, so an unset key would
// refuse every SiteStream call while the node still looked healthy.
"GrpcPsk": "dev-grpc-psk-docker-site-c",
"CentralContactPoints": [ "CentralContactPoints": [
"akka.tcp://scadabridge@scadabridge-central-a:8081", "akka.tcp://scadabridge@scadabridge-central-a:8081",
"akka.tcp://scadabridge@scadabridge-central-b:8081" "akka.tcp://scadabridge@scadabridge-central-b:8081"
@@ -41,6 +41,13 @@
"SqliteDbPath": "/app/data/store-and-forward.db" "SqliteDbPath": "/app/data/store-and-forward.db"
}, },
"Communication": { "Communication": {
// DEV-ONLY control-plane preshared key — NOT a real secret. Must be
// IDENTICAL on both nodes of the pair and match the central-side entry in
// ScadaBridge__Communication__SitePsks__<siteId> (docker-compose.yml).
// Production supplies this as ${secret:SB-GRPC-PSK-<siteId>}. Without it the
// node fails StartupValidator: the gate is fail-closed, so an unset key would
// refuse every SiteStream call while the node still looked healthy.
"GrpcPsk": "dev-grpc-psk-docker-site-c",
"CentralContactPoints": [ "CentralContactPoints": [
"akka.tcp://scadabridge@scadabridge-central-a:8081", "akka.tcp://scadabridge@scadabridge-central-a:8081",
"akka.tcp://scadabridge@scadabridge-central-b:8081" "akka.tcp://scadabridge@scadabridge-central-b:8081"
+8 -2
View File
@@ -30,7 +30,13 @@ The transports are independent. A gRPC stream interruption does not affect in-fl
**The gRPC dial direction is inverted from its data direction.** Values flow site → central, but each **site node hosts the gRPC server** and **central is the client**. `MapGrpcService<SiteStreamGrpcServer>()` appears exactly once in the tree, inside the Site branch of `Program.cs` (`Host/Program.cs:542`); there is **no gRPC server on a central node at all**. That is why the two `Ingest*` unary RPCs — nominally a central-side ingest surface — are dead in the shipped topology (acknowledged in `AkkaHostedService.cs:505-508`: "when the gRPC server is not registered (current central topology)"); sites push audit telemetry to central over `ClusterClient` instead, and central pulls with `PullAuditEvents` / `PullSiteCalls` by dialling the site. **The gRPC dial direction is inverted from its data direction.** Values flow site → central, but each **site node hosts the gRPC server** and **central is the client**. `MapGrpcService<SiteStreamGrpcServer>()` appears exactly once in the tree, inside the Site branch of `Program.cs` (`Host/Program.cs:542`); there is **no gRPC server on a central node at all**. That is why the two `Ingest*` unary RPCs — nominally a central-side ingest surface — are dead in the shipped topology (acknowledged in `AkkaHostedService.cs:505-508`: "when the gRPC server is not registered (current central topology)"); sites push audit telemetry to central over `ClusterClient` instead, and central pulls with `PullAuditEvents` / `PullSiteCalls` by dialling the site.
**None of the three transports carries a mutual-auth or transport-encryption story.** `BuildHocon` emits no Akka `enable-ssl`, no secure cookie and no `trusted-selection-paths`, so remoting and `ClusterClient` are plaintext and unauthenticated. The site gRPC listener is **h2c**`ListenAnyIP(grpcPort, o => o.Protocols = HttpProtocols.Http2)` with no `UseHttps` (`Host/Program.cs:501-506`) — and `SiteStreamGrpcClient` opens a bare `GrpcChannel.ForAddress(endpoint, …)` with no credentials. `LocalDbSyncAuthInterceptor` shares that listener but gates **only** methods under `/localdb_sync.v1.LocalDbSync/`, explicitly letting every `SiteStreamService` call through untouched, so the whole surface — including the `PullAuditEvents` / `PullSiteCalls` RPCs that return audit rows — is callable by anything that can reach the site's gRPC port. The one authenticated piece is the HTTP config fetch, and its per-deployment token is the *entire* security boundary (the endpoint is `AllowAnonymous`). The design assumes a trusted network between central and sites. **No transport carries transport encryption; two of the three now carry authentication.**
- **Akka remoting / `ClusterClient` — unauthenticated.** `BuildHocon` emits no `enable-ssl`, no secure cookie and no `trusted-selection-paths`, so the command/control path is plaintext and open to anything that can reach the remoting port.
- **gRPC — authenticated by preshared key since 2026-07-22.** The listener is still **h2c**`ListenAnyIP(grpcPort, o => o.Protocols = HttpProtocols.Http2)` with no `UseHttps` — but `ControlPlaneAuthInterceptor` gates every method under `/sitestream.SiteStreamService/`, including the `PullAuditEvents` / `PullSiteCalls` RPCs that return audit rows. It is fail-closed (no key ⇒ everything refused, and `StartupValidator` will not boot a site node in that state), compares with `CryptographicOperations.FixedTimeEquals`, and rejects with `PermissionDenied`. Central attaches the key via `ControlPlaneCredentials`, which binds `CallCredentials` to each channel so unary and streaming calls are covered uniformly — `SiteStreamGrpcClient`, `GrpcPullAuditEventsInvoker` and `GrpcPullSiteCallsInvoker` all build their channels through it. Keys are **per site** (`SB-GRPC-PSK-<siteId>`), so a compromised site yields only its own. `LocalDbSyncAuthInterceptor` shares the listener and keeps its own separate key on `/localdb_sync.v1.LocalDbSync/` — the two authenticate different peers (central vs. the pair partner) and are never shared.
- **HTTP config fetch — token-authenticated.** Its per-deployment token is the *entire* security boundary (the endpoint is `AllowAnonymous`).
A bearer PSK over plaintext h2c is readable and replayable by anyone on the path, so the design still assumes a trusted network between central and sites — but the bar is now "read the traffic" rather than "reach the port". TLS on these listeners is the follow-on hardening and would not change the key design. Operational detail: [`docs/deployment/topology-guide.md`](../deployment/topology-guide.md).
### Notify-and-fetch: the deployment-config HTTP path ### Notify-and-fetch: the deployment-config HTTP path
@@ -262,7 +268,7 @@ Three layers of dead-client detection protect the gRPC stream path:
- [Notification Outbox (#21)](./NotificationOutbox.md) — `CentralCommunicationActor` routes `NotificationSubmit` / `NotificationStatusQuery` messages from sites to the `NotificationOutboxActor` proxy. `CommunicationService` Asks the proxy directly for central-UI outbox management calls. - [Notification Outbox (#21)](./NotificationOutbox.md) — `CentralCommunicationActor` routes `NotificationSubmit` / `NotificationStatusQuery` messages from sites to the `NotificationOutboxActor` proxy. `CommunicationService` Asks the proxy directly for central-UI outbox management calls.
- [Site Call Audit (#22)](./SiteCallAudit.md) — `CommunicationService` Asks the `SiteCallAuditActor` proxy directly for query and relay operations. `SiteCallAuditActor` issues `RetryParkedOperation` / `DiscardParkedOperation` relay commands to sites via `SiteEnvelope`; `SiteCommunicationActor` dispatches them to `_parkedMessageHandler`. - [Site Call Audit (#22)](./SiteCallAudit.md) — `CommunicationService` Asks the `SiteCallAuditActor` proxy directly for query and relay operations. `SiteCallAuditActor` issues `RetryParkedOperation` / `DiscardParkedOperation` relay commands to sites via `SiteEnvelope`; `SiteCommunicationActor` dispatches them to `_parkedMessageHandler`.
- [Store-and-Forward Engine (#6)](./StoreAndForward.md) — the site S&F Engine drives `NotificationSubmit` forwarding and cached-call telemetry emission through `SiteCommunicationActor`. Parked-message queries and retry/discard relay commands flow back the other way. - [Store-and-Forward Engine (#6)](./StoreAndForward.md) — the site S&F Engine drives `NotificationSubmit` forwarding and cached-call telemetry emission through `SiteCommunicationActor`. Parked-message queries and retry/discard relay commands flow back the other way.
- [Management Service (#18)](./ManagementService.md) — `ManagementActor` is registered with `ClusterClientReceptionist` at `/user/management` on central; the CLI connects via its own separate `ClusterClient`. This is a distinct `ClusterClient` usage from the inter-cluster hub-and-spoke connections managed by this component. Management Service also hosts `DeploymentConfigEndpoints` — the `GET /api/internal/deployments/{id}/config` route that terminates the third (HTTP) transport, mapped in the central-role block alongside `/api/audit/*` and `/management`. - [Management Service (#18)](./ManagementService.md) — `ManagementActor` runs at `/user/management` on central and is reached **in-process** through `ManagementActorHolder`; the CLI connects over HTTP, not `ClusterClient`. (It was `ClusterClientReceptionist`-registered until 2026-07-22, for a CLI that was never built that way.) So this component's `ClusterClient` usage is exclusively the inter-cluster hub-and-spoke connections. Management Service also hosts `DeploymentConfigEndpoints` — the `GET /api/internal/deployments/{id}/config` route that terminates the third (HTTP) transport, mapped in the central-role block alongside `/api/audit/*` and `/management`.
- Design spec: [Component-Communication.md](../requirements/Component-Communication.md). - Design spec: [Component-Communication.md](../requirements/Component-Communication.md).
## Troubleshooting ## Troubleshooting
+7 -4
View File
@@ -56,17 +56,18 @@ Mutating handlers that call repositories directly invoke `AuditAsync` (backed by
### Actor lifecycle and registration ### Actor lifecycle and registration
`AkkaHostedService` (in the Host) creates the `ManagementActor` under the path `/user/management` and registers it with `ClusterClientReceptionist`: `AkkaHostedService` (in the Host) creates the `ManagementActor` under the path `/user/management` and publishes it to `ManagementActorHolder`, which is the only way anything reaches it:
```csharp ```csharp
var mgmtActor = _actorSystem!.ActorOf( var mgmtActor = _actorSystem!.ActorOf(
Props.Create(() => new ManagementActor(_serviceProvider, mgmtLogger)), Props.Create(() => new ManagementActor(_serviceProvider, mgmtLogger)),
"management"); "management");
ClusterClientReceptionist.Get(_actorSystem).RegisterService(mgmtActor);
var mgmtHolder = _serviceProvider.GetRequiredService<ManagementActorHolder>(); var mgmtHolder = _serviceProvider.GetRequiredService<ManagementActorHolder>();
mgmtHolder.ActorRef = mgmtActor; mgmtHolder.ActorRef = mgmtActor;
``` ```
A `ClusterClientReceptionist.Get(_actorSystem).RegisterService(mgmtActor)` call sat between those two statements until 2026-07-22. It was deleted because nothing ever sent to it: the CLI it was built for uses HTTP, not ClusterClient.
`ClusterClientReceptionist` advertises the actor to `ClusterClient` senders without requiring them to join the Akka cluster. The `ManagementActorHolder.ActorRef` property is then the bridge from the HTTP endpoint (which runs in ASP.NET Core middleware) into the Akka actor world. `ClusterClientReceptionist` advertises the actor to `ClusterClient` senders without requiring them to join the Akka cluster. The `ManagementActorHolder.ActorRef` property is then the bridge from the HTTP endpoint (which runs in ASP.NET Core middleware) into the Akka actor world.
The actor declares an explicit supervisor strategy — one-for-one with Resume and no retry limit — to match the coordinator-actor convention and remain correct if child actors are added later. The actor declares an explicit supervisor strategy — one-for-one with Resume and no retry limit — to match the coordinator-actor convention and remain correct if child actors are added later.
@@ -154,9 +155,11 @@ Content-Type: application/json
A successful response is HTTP 200 with the JSON result. An authorization failure is HTTP 403 with `{ "error": "...", "code": "UNAUTHORIZED" }`. A successful response is HTTP 200 with the JSON result. An authorization failure is HTTP 403 with `{ "error": "...", "code": "UNAUTHORIZED" }`.
### Sending a command via ClusterClient ### Sending a command in-process
The `ManagementActor` is also reachable from any `ClusterClient` that has a contact point into the central cluster. The actor is registered under `/system/receptionist` with the path `/user/management`. Callers construct and `Tell` a `ManagementEnvelope` and expect one of `ManagementSuccess`, `ManagementError`, or `ManagementUnauthorized` in reply. `ManagementEnvelope` is also the in-process contract: a caller holding `ManagementActorHolder.ActorRef` asks the actor directly and expects one of `ManagementSuccess`, `ManagementError`, or `ManagementUnauthorized` in reply. `ManagementEndpoints` is that caller.
There is **no** out-of-process actor path. The actor was advertised via `ClusterClientReceptionist` until 2026-07-22, so a `ClusterClient` with a contact point into the central cluster could `Tell` it a `ManagementEnvelope`; no caller ever did, and the registration is gone. The HTTP endpoints above are the only remote management surface.
## Command Groups ## Command Groups
+39 -3
View File
@@ -178,9 +178,45 @@ database from its peer rather than letting it rejoin.
### Central-Site Communication ### Central-Site Communication
- Sites connect to central via Akka.NET remoting. Three transports cross the boundary, not one:
- The `Communication:CentralSeedNode` setting in the site config points to one of the central nodes.
- If that central node is down, the site's communication actor will retry until it connects to the active central node. - **Akka ClusterClient** — command/control. Sites list every central node in
`ScadaBridge:Communication:CentralContactPoints`; contact rotation reaches whichever node
answers, so no "active central" needs to be identified. (There is no `Communication:CentralSeedNode`
setting — earlier revisions of this guide named one that never existed in the code.)
- **gRPC** — real-time data and audit pull. Note the direction is inverted from the data flow:
each **site node hosts the gRPC server** on `GrpcPort` (default 8083, h2c) and central dials in.
- **Plain HTTP** — the deploy config itself, fetched by the site with a per-deployment token.
#### gRPC control-plane preshared key (required)
Every site node must set `ScadaBridge:Communication:GrpcPsk`, and central must hold the same
value for that site. **`StartupValidator` refuses to boot a site node without it**, deliberately:
the gate is fail-closed, so an unset key would leave the node joined, healthy-looking and
answering heartbeats while refusing every gRPC call — no live subscriptions, no audit pull, no
cached-telemetry ingest.
| Side | Where the key lives |
|---|---|
| Site node (both nodes of the pair, identical) | `ScadaBridge:Communication:GrpcPsk`, in production `${secret:SB-GRPC-PSK-<siteId>}` |
| Central | secret `SB-GRPC-PSK-<siteId>` in its store — **or** `ScadaBridge:Communication:SitePsks:<siteId>` |
The store is the source that matters in production, because sites are added at runtime and their
keys cannot be enumerated in configuration at boot; `SitePsks` covers a host running without a
master key (the docker rig) and one-off pins.
One key **per site**, never one for the fleet: a compromised site must not yield another site's
key. And never share it with `LocalDb:Replication:ApiKey` — that authenticates the *pair partner*
for database replication, a different trust relationship on the same listener.
**Rotation:** set the new value on both sides, then restart the pair (pairs restart together
anyway — see above). **Upgrading to a build that has this gate requires seeding the key first**,
including in the on-host `deploy/` overlays.
The key is a bearer token over plaintext h2c, so it is readable and replayable by anyone on the
path. That is the accepted posture today — the same trusted-network assumption the boundary
already made, now with authentication rather than none. TLS on these listeners is follow-on
hardening and needs no change to the key design.
## Scaling Guidelines ## Scaling Guidelines
@@ -0,0 +1,59 @@
# Integration call routing (`IntegrationCallRequest`) is dead on both ends
**Date:** 2026-07-22 · **Status:** OPEN (decision needed: wire or delete) · **Severity:** Low (no
runtime impact — the path cannot be reached) · **Area:** CentralSite Communication
## What
"Pattern 4: Integration Routing" — `CommunicationService.RouteIntegrationCallAsync`
`SiteEnvelope(IntegrationCallRequest)``SiteCommunicationActor` → an integration handler — is
plumbed end to end but connected at neither end.
- **No producer.** `RouteIntegrationCallAsync` (`CommunicationService.cs`, "Pattern 4") has **zero
callers** in `src/` or `tests/`. It is the only one of `CommunicationService`'s command methods
with none.
- **No handler.** `SiteCommunicationActor` forwards to `_integrationHandler` when one is
registered, but `RegisterLocalHandler(LocalHandlerType.Integration, …)` appears **only** in
`SiteCommunicationActorTests.cs`. `AkkaHostedService` registers the other three handler types
(`Artifacts`, `EventLog`, `ParkedMessages`) and never this one.
So if anything ever did call it, the site would answer
`IntegrationCallResponse(Success: false, Error: "Integration handler not available")`
(`SiteCommunicationActor.cs`, Pattern 4) — and the two tests that exercise the path both register
the handler themselves first, which is why the suite has never noticed.
Do not confuse this with the **Inbound API**'s routed-site-script path, which is live, tested, and
uses different messages entirely. This is a separate, unused routing pattern that predates it.
## Why it is recorded rather than fixed
Found during the recon for the ClusterClient→gRPC transport migration
([`docs/plans/2026-07-22-clusterclient-to-grpc-plan.md`](../plans/2026-07-22-clusterclient-to-grpc-plan.md),
T0.2), which had to enumerate every command crossing the site↔central boundary. Of the **29**
command types, this is the one that is excluded: **28 migrate to the gRPC contract.**
Porting it would mean designing a proto contract, a `oneof` slot and round-trip mapper tests for a
verb no caller can invoke and no site can service — and enshrining it on a wire format whose
evolution rules are additive-only, so an unused RPC slot is permanent. Deleting it during a
transport migration would mix an unrelated behavioural change into a change whose whole value is
that behaviour is identical. Hence: excluded from the contract, behaviour untouched, decision
deferred to its own change.
## Decision needed
Either:
1. **Delete** — remove `RouteIntegrationCallAsync`, the `IntegrationCallRequest`/`Response`
messages, the `SiteCommunicationActor` receive block, `LocalHandlerType.Integration`, and the
three tests that cover them. This is the default if no consumer is planned.
2. **Wire** — register a real integration handler on site nodes and give the method a caller. This
only makes sense if there is a requirement it serves; none is recorded in
`docs/requirements/`.
Whichever is chosen, do it **before Phase 4** of the migration, since Phase 4 deletes the Akka
transport underneath this path. If it is still dead at that point, option 1 is forced.
## Filing
To be filed as a Gitea issue on `dohertj2/scadabridge` by the repo owner — this note is the
in-repo record of the finding and of the migration exclusion it justifies.
@@ -0,0 +1,198 @@
# ClusterClient → gRPC migration — live gate results
Rig: `docker/` (2 central + 3×2 site + traefik), rebuilt from the branch under test via
`bash docker/deploy.sh`. Recorded check-by-check in the family's live-gate format. Phase 5's
full eight-check gate is recorded further down as those phases land; this file starts with
Phase 0, whose DoD has its own smaller gate.
---
## Phase 0 — PSK auth + dead-code removal — **PASS** (2026-07-22)
Branch `feat/grpc-phase0-psk` @ `228ff8b4`. Image rebuilt, all 9 containers recreated.
### Baseline (pre-change build, same rig)
An unauthenticated call from the host to a site's audit-pull RPC was **accepted**:
```
$ grpcurl -plaintext -d '{"batch_size":1}' localhost:9023 sitestream.SiteStreamService/PullAuditEvents
{}
```
That is the gap Phase 0 closes, reproduced rather than assumed.
### Checks
| # | Check | Result |
|---|---|---|
| 1 | All 8 nodes boot with keys configured (new `StartupValidator` rule) | **PASS** — all recreated and reached ready |
| 2 | Unauthenticated `PullAuditEvents``PermissionDenied`, all 3 sites | **PASS** |
| 3 | Wrong key (site-b's key presented to site-a) ⇒ `PermissionDenied` | **PASS** — per-site scoping is real, not decorative |
| 4 | Correct key ⇒ success, all 3 sites | **PASS** |
| 5 | Central's own authenticated paths still work | **PASS** — 14 successful `PullAuditEvents` from central to site-a; **0** auth failures in either central's log |
| 6 | LocalDb sync unaffected by the new interceptor | **PASS****0** control-plane rejections and **0** sync auth failures on the passive peer; session connected after one boot-order retry |
| 7 | No interceptor-activation errors | **PASS** — 0 (see the defect below) |
Evidence for 24:
```
=== NO CREDENTIALS ===
:9023 -> ERROR: Code: PermissionDenied Message: Control plane authentication failed.
:9033 -> ERROR: Code: PermissionDenied Message: Control plane authentication failed.
:9043 -> ERROR: Code: PermissionDenied Message: Control plane authentication failed.
=== WRONG KEY (site-b's key against site-a) ===
:9023 -> ERROR: Code: PermissionDenied Message: Control plane authentication failed.
=== CORRECT KEY ===
site-a :9023 -> {}
site-b :9033 -> {}
site-c :9043 -> {}
```
Site-a rejected exactly **2** calls — the two deliberate probes above — and nothing else.
### Defect the gate caught that the test suite did not
**First run of this gate FAILED**, and is worth recording because the failure mode is
deceptive.
`Grpc.AspNetCore` activates a type-registered interceptor through
`InterceptorRegistration.GetFactory()`, which throws when more than one public constructor is
applicable. `ControlPlaneAuthInterceptor` shipped with two — the DI one and a prefix-set
overload intended for later phases.
The throw happens **inside the pipeline, per call**, so:
- nothing failed at startup; the node booted, joined its pair and reported healthy;
- every gated call died with `Unknown / "Exception was thrown by handler"`, which reads as a
handler bug rather than an auth bug;
- **correct key, wrong key and no key produced identical errors** — the tell. A gate that
cannot distinguish those is not authenticating anything.
Site-a's log at the time: three `PullAuditEvents` calls, three identical
`System.InvalidOperationException: Multiple constructors accepting all given argument types
have been found in type 'ControlPlaneAuthInterceptor'`.
The full suite was green when this shipped — **29 suites, 6,872 tests, 0 failures**. The
in-process end-to-end test missed it because it registered the interceptor with
`AddSingleton` alongside `AddGrpc`, so DI returned the instance and gRPC's activation path
never ran.
Fixed in `228ff8b4`: the prefix-set constructor is `internal`; the end-to-end harness now
registers exactly as `Program.cs` does (by type, not in DI); and a reflection assertion pins
"exactly one public constructor", since that is the actual invariant.
**Lesson for phases 1A/1B, which both add services to this interceptor:** extend
`DefaultGatedPrefixes`; do not add a second public constructor. And any in-process harness for
a DI-activated component must mirror the production registration shape or it proves less than
it appears to.
### Test suite alongside the gate
Non-Playwright: **29 suites, 6,872 tests, 0 failures**.
Playwright (against this rig): **170 passed, 2 failed, 1 skipped** of 173. Both failures were
run down to root cause and **both are pre-existing on `main`, unrelated to Phase 0** — this
branch touches no EF, CentralUI, Transport or ManagementService file (`git diff --stat
main...HEAD -- src/` is 15 files, all Communication/Host/AuditLog gRPC plumbing).
An earlier run of this suite reported 44 failures. That run is **void**: a `docker/deploy.sh`
was recreating the cluster underneath it, so the fast `LoginTests`/`NavigationTests` failures
were "app unreachable", not defects.
**1. `TransportImportTests.ImportSyntheticBundle_AppliesAndShowsAuditDrillIn` — a real
production bug, not a test defect.** Central's log during the failure:
```
[ERR] An exception occurred while iterating over the results of a query ...
System.InvalidOperationException: The configured execution strategy
'SqlServerRetryingExecutionStrategy' does not support user-initiated transactions.
at Microsoft.EntityFrameworkCore.Query.Internal.SplitQueryingEnumerable`1.AsyncEnumerator.MoveNextAsync()
```
`BundleImporter.cs:1298` opens a user-initiated transaction; the central context is configured
with `EnableRetryOnFailure` (`ConfigurationDatabase/ServiceCollectionExtensions.cs:33`). SQL
Server's retrying strategy refuses to run a split query inside a caller's transaction, so
**bundle import fails against real MS SQL**. The fix is the one the exception names: wrap the
transaction in `Database.CreateExecutionStrategy().ExecuteAsync(...)`.
Why the whole unit/integration suite is green on it: those tests use the in-memory EF provider,
which has no retrying execution strategy — and `BeginTransactionAsync` is a no-op there. The
comment directly above line 1298 documents that divergence without drawing the conclusion. Only
a rig-backed test can see this.
**2. `SmsNotificationE2ETests.SmsConfigPage_CreateOrRender_NeverLeaksAuthToken` — a stale test
fixture.** No server-side error at all: the page renders 200, and no `INSERT INTO
SmsConfigurations` is ever issued. The test's fixture SID is `ACtest123` (`d6ead8ae`,
2026-06-19). `SmsConfiguration.razor:231` rejects anything not matching `^AC[0-9a-fA-F]{32}$`,
added by `40088a21` (2026-07-10) to close an un-escaped URI-interpolation hole. `Save()` sets
`_formError` and returns — no toast, exactly as observed. The fixture was never updated.
This has been failing since 2026-07-10, and it matters more than a red line: everything after
the toast assertion — including **the secret-non-leak assertion that the Auth Token value never
reaches the page HTML** — has not executed since. Fix is a valid 32-hex SID in the fixture.
### Not covered by this gate
- Streaming subscriptions were exercised in-process (TestServer), not over the rig. The
interceptor is path-scoped, not method-scoped, so the rig's `PullAuditEvents` evidence covers
the same code path — but a live `SubscribeInstance` under load is untested here.
- Key **rotation** on a live pair.
- `docker-env2` was updated with its own key but not redeployed or gated.
---
## Phase 1A — central control plane (site→central over gRPC) — **PASS** (2026-07-22)
Branch `feat/grpc-central-control` @ `0e162cb2`. Rig rebuilt; **site-a flipped to
`CentralTransport=Grpc`** with `CentralGrpcEndpoints=[central-a:8083, central-b:8083]`,
**site-b/c left on Akka** (default) to prove coexistence. The site-a flag flip was a
DoD-test-only rig edit — reverted from the branch, never committed (the plan keeps the default
`Akka` until Phase 4).
### The defect this gate caught (T1A.2 shipped it; fixed in `0e162cb2`)
**First rebuild: central's entire HTTP surface was gone.** central-a logged only
`Now listening on: http://[::]:8083` — no `:5000`. Central UI, the Management + Inbound API,
and every `/health/*` endpoint (Traefik routing + `IActiveNodeGate` both depend on them) were
dead. The node booted, joined the cluster and served gRPC fine; **no startup error.**
Cause: `builder.WebHost.ConfigureKestrel(o => o.ListenAnyIP(8083, Http2))` puts Kestrel into
explicit-endpoints mode, which **suppresses the URLs from `ASPNETCORE_URLS`/`--urls`** — it is
not additive, contrary to the comment T1A.2 shipped. Central's whole HTTP/1 surface lives on
that URL (`http://+:5000` on the rig; a different port in production). The site branch has the
same `ConfigureKestrel` shape but nothing on `ASPNETCORE_URLS` to lose — it binds every port it
needs explicitly — which is why the pattern looked safe.
Every unit + E2E test uses `TestServer`, which never binds real Kestrel, so the whole suite
(6,872) stayed green. Only a live node exposes a missing listener. Fix: parse the port(s) from
the configured URLs and re-declare them (`Http1AndHttp2`) alongside the gRPC port (`Http2`) in
the one `ConfigureKestrel` call — `Program.ParseHttpBindPorts` + `CentralHttpBindPortsTests`
(14 cases). After the fix: `Now listening on: http://[::]:5000` **and** `:8083`; 9001 ready
`200`/active Healthy, 9002 standby, Traefik LB `200`, CLI over the LB works.
### Checks (post-fix rebuild)
| # | Check | Result |
|---|---|---|
| 1 | site-a rides authenticated gRPC to `CentralControlService` | **PASS** — Heartbeat, `ReportSiteHealth`, `ReconcileSite` all HTTP/2 → 200; **0** auth failures on either central |
| 2 | Heartbeat drives the active flag | **PASS** — 144+ heartbeats over gRPC; site-a `online=True` at central |
| 3 | Health page live | **PASS**`ReportSiteHealth` lands; central shows site-a `online=True`, sequence advancing, alongside Akka site-b/c |
| 4 | Reconcile works after site restart | **PASS** — restarted `site-a-a`; it logged `Site→central transport: gRPC to 2 central endpoint(s)`, then `Reconcile pass … complete: 0 fetched, 0 failed, 0 orphan(s)` |
| 5 | Coexistence | **PASS** — site-b/c log `Created ClusterClient to central`; both `online=True` — Akka and gRPC sites side by side |
| 6 | Central HTTP surface intact under the new gRPC listener | **PASS** — after the fix (see above) |
### Not independently exercised on this gate
- **Notification e2e (`SubmitNotification`/`QueryNotificationStatus`) and audit ingest
(`IngestAuditEvents`/`IngestCachedTelemetry`)** were NOT driven live: the rig has templates
but **no deployed instance**, so nothing emits site→central notifications or audit rows on its
own. These four RPCs traverse the identical `CentralControlGrpcService`
`CentralCommunicationActor` Ask path that checks 13 proved live under real auth, and their
payload mappers carry 32 round-trip goldens — but the payloads themselves were not put over
the wire here. **Phase 2's central-kill S&F soak is where they get their live workout;** flag
for a fuller 1A proof if a deployed-instance rig is set up before then.
- Cross-node failover/failback of the site→central channel under a central-node kill (unit-proven
via TestServer; not exercised on the rig at 1A).
@@ -302,11 +302,11 @@ Critical path ≈ 1B: **~46 weeks total**, matching the design estimate.
## Task checklist (tick as you go; IDs reference the sections above) ## Task checklist (tick as you go; IDs reference the sections above)
**Phase 0 — PSK + dead code** (branch `feat/grpc-phase0-psk`) **Phase 0 — PSK + dead code** (branch `feat/grpc-phase0-psk`)
- [ ] T0.1 Delete `/user/management` receptionist registration (`AkkaHostedService.cs:459`) + Component-Host.md update - [x] T0.1 Delete `/user/management` receptionist registration (`AkkaHostedService.cs:459`) + Component-Host.md update
- [ ] T0.2 File dead-`IntegrationCallRequest` issue; record exclusion (28 of 29 migrate) - [x] T0.2 File dead-`IntegrationCallRequest` issue; record exclusion (28 of 29 migrate)
- [ ] T0.3 `ControlPlaneAuthInterceptor` + `CommunicationOptions.GrpcPsk` + `SitePskProvider`; gate SiteStream; PSK attached on central's streaming + pull clients - [x] T0.3 `ControlPlaneAuthInterceptor` + `CommunicationOptions.GrpcPsk` + `SitePskProvider`; gate SiteStream; PSK attached on central's streaming + pull clients
- [ ] T0.4 Rig dev keys (all 3 sites + central store seeds) + interceptor/provider/wiring tests - [x] T0.4 Rig dev keys (all 3 sites + central store seeds) + interceptor/provider/wiring tests
- [ ] Phase 0 DoD: suite green; rig unauthenticated ⇒ `PermissionDenied`, authenticated paths work; PR merged - [x] Phase 0 DoD: suite green; rig unauthenticated ⇒ `PermissionDenied`, authenticated paths work; PR merged (#25, ff to `main` @ `3fa95555`; gate PASS in `2026-07-22-clusterclient-to-grpc-live-gate.md`)
**Phase 1A — central control plane** (worktree, `feat/grpc-central-control`) **Phase 1A — central control plane** (worktree, `feat/grpc-central-control`)
- [ ] T1A.1 `central_control.proto` (7 RPCs; checked-in codegen) + `CentralControlDtoMapper` + round-trip golden tests - [ ] T1A.1 `central_control.proto` (7 RPCs; checked-in codegen) + `CentralControlDtoMapper` + round-trip golden tests
@@ -0,0 +1,241 @@
{
"plan": "docs/plans/2026-07-22-clusterclient-to-grpc-plan.md",
"design": "~/Desktop/scadaproj/scadabridge_clusterclient_to_grpc.md",
"worktrees": {
"feat/grpc-phase0-psk": "/Users/dohertj2/Desktop/ScadaBridge-phase0"
},
"tasks": [
{
"id": "T0.1",
"phase": "0",
"subject": "Delete the vestigial /user/management receptionist registration",
"status": "completed",
"activeForm": "Deleting the /user/management receptionist registration",
"files": [
"src/ZB.MOM.WW.ScadaBridge.Host/Actors/AkkaHostedService.cs",
"docs/requirements/Component-Host.md",
"docs/requirements/Component-ManagementService.md",
"docs/requirements/Component-Communication.md",
"docs/components/ManagementService.md",
"docs/components/Communication.md",
"src/ZB.MOM.WW.ScadaBridge.CLI/README.md"
],
"notes": "Actor stays; only the ClusterClientReceptionist.RegisterService call goes. Verified: CLI is Akka-free, so 6 docs claiming the CLI reaches ManagementActor over ClusterClient are stale and must be corrected in the same change."
},
{
"id": "T0.2",
"phase": "0",
"subject": "Record the dead IntegrationCallRequest exclusion (28 of 29 commands migrate)",
"status": "completed",
"activeForm": "Recording the dead IntegrationCallRequest exclusion",
"files": [
"docs/known-issues/"
],
"notes": "Plan says 'file a Gitea issue' \u2014 outward-facing, handed to the user. In-repo half is a known-issues note + the exclusion record."
},
{
"id": "T0.3",
"phase": "0",
"subject": "ControlPlaneAuthInterceptor + CommunicationOptions.GrpcPsk + SitePskProvider; gate SiteStream; attach PSK on central's clients",
"status": "completed",
"activeForm": "Building the control-plane PSK auth",
"files": [
"src/ZB.MOM.WW.ScadaBridge.Host/ControlPlaneAuthInterceptor.cs",
"src/ZB.MOM.WW.ScadaBridge.Communication/CommunicationOptions.cs",
"src/ZB.MOM.WW.ScadaBridge.Communication/Grpc/ISitePskProvider.cs",
"src/ZB.MOM.WW.ScadaBridge.Host/SitePskProvider.cs",
"src/ZB.MOM.WW.ScadaBridge.Communication/Grpc/SiteStreamGrpcClient.cs",
"src/ZB.MOM.WW.ScadaBridge.Communication/Grpc/SiteStreamGrpcClientFactory.cs",
"src/ZB.MOM.WW.ScadaBridge.AuditLog/Central/GrpcPullAuditEventsClient.cs",
"src/ZB.MOM.WW.ScadaBridge.AuditLog/Central/GrpcPullSiteCallsClient.cs",
"src/ZB.MOM.WW.ScadaBridge.Host/Program.cs"
],
"notes": "Service prefix verified from sitestream.proto: /sitestream.SiteStreamService/. Grpc 2.76 -> CallCredentials.FromInterceptor + UnsafeUseInsecureChannelCallCredentials is the async-safe attach path on h2c."
},
{
"id": "T0.4",
"phase": "0",
"subject": "Rig dev keys (3 sites + central secret seeds) + interceptor/provider/wiring tests",
"status": "completed",
"activeForm": "Seeding rig dev keys and writing the auth tests",
"files": [
"docker/site-a-node-a/appsettings.Site.json",
"docker/site-a-node-b/appsettings.Site.json",
"docker/site-b-node-a/appsettings.Site.json",
"docker/site-b-node-b/appsettings.Site.json",
"docker/site-c-node-a/appsettings.Site.json",
"docker/site-c-node-b/appsettings.Site.json",
"tests/ZB.MOM.WW.ScadaBridge.Host.Tests/ControlPlaneAuthInterceptorTests.cs"
],
"notes": "Fail-closed from day one \u2014 every environment (incl. docker-env2 and the gitignored deploy/wonder-app-vd03 overlay) needs its key before upgrade. Ops item for the user."
},
{
"id": "P0.DoD",
"phase": "0",
"subject": "Phase 0 DoD: suite green; rig unauthenticated => PermissionDenied, authenticated paths work; PR merged",
"status": "completed",
"activeForm": "Verifying the Phase 0 DoD",
"blockedBy": [
"T0.1",
"T0.2",
"T0.3",
"T0.4"
],
"notes": "Live gate PASS 2026-07-22 (all 7 checks, docs/plans/2026-07-22-clusterclient-to-grpc-live-gate.md). Suite: 29 non-Playwright suites / 6872 tests / 0 failures. Playwright 170 pass / 2 fail / 1 skip of 173 - BOTH failures root-caused and PRE-EXISTING on main, unrelated to Phase 0 (branch touches no EF/CentralUI/Transport/ManagementService file): (1) TransportImportTests - REAL production bug, BundleImporter.cs:1298 user-initiated transaction + EnableRetryOnFailure => import broken on real MS SQL, hidden by the in-memory EF provider; (2) SmsNotificationE2ETests - stale fixture SID 'ACtest123' vs the ^AC[0-9a-fA-F]{32}$ guard added 2026-07-10 (40088a21); failing since then, which also silences its secret-non-leak assertion. The earlier 44-failure run is VOID (concurrent rig rebuild). REMAINING: PR + merge, HELD for the user per the plan's stop-at-DoD rule."
},
{
"id": "T1A.1",
"phase": "1A",
"subject": "central_control.proto (7 RPCs, checked-in codegen) + CentralControlDtoMapper + round-trip golden tests",
"status": "in_progress",
"activeForm": "Authoring central_control.proto and its mappers",
"blockedBy": [
"P0.DoD"
]
},
{
"id": "T1A.2",
"phase": "1A",
"subject": "Central hosting: AddGrpc + per-site-PSK interceptor, CentralGrpcPort h2c listener, CentralControlGrpcService, readiness gate",
"status": "pending",
"activeForm": "Hosting CentralControlService on central",
"blockedBy": [
"T1A.1"
]
},
{
"id": "T1A.3",
"phase": "1A",
"subject": "ICentralTransport (Akka extract + Grpc impl), CentralChannelProvider, CentralTransport flag, CentralGrpcEndpoints option",
"status": "pending",
"activeForm": "Building the site->central transport seam",
"blockedBy": [
"T1A.1"
]
},
{
"id": "T1A.4",
"phase": "1A",
"subject": "Tests: actor-with-fake-transport x7, TestServer transport tests, S&F/audit/health suites pass unmodified",
"status": "pending",
"activeForm": "Testing the central control plane",
"blockedBy": [
"T1A.2",
"T1A.3"
]
},
{
"id": "P1A.DoD",
"phase": "1A",
"subject": "1A DoD: rig site-a on Grpc proves all 5 site->central paths while site-b/c stay Akka; PR merged before 1B",
"status": "pending",
"activeForm": "Verifying the 1A DoD",
"blockedBy": [
"T1A.4"
]
},
{
"id": "T1B.1",
"phase": "1B",
"subject": "site_command.proto (6 oneof RPCs / 28 commands) + SiteCommandDtoMapper + round-trip golden tests",
"status": "in_progress",
"activeForm": "Authoring site_command.proto and its mappers",
"blockedBy": [
"P0.DoD"
]
},
{
"id": "T1B.2",
"phase": "1B",
"subject": "SiteCommandDispatcher refactor (actor + SiteCommandGrpcService share it; parked stays node-local)",
"status": "pending",
"activeForm": "Extracting the site command dispatcher",
"blockedBy": [
"T1B.1"
]
},
{
"id": "T1B.3",
"phase": "1B",
"subject": "ISiteCommandTransport in CentralCommunicationActor (Akka extract + Grpc impl), SitePairChannelProvider, SiteTransport flag",
"status": "pending",
"activeForm": "Building the central->site transport seam",
"blockedBy": [
"T1B.1"
]
},
{
"id": "T1B.4",
"phase": "1B",
"subject": "Tests: dispatcher routing x28, actor envelope/reply plumbing, TestServer service tests, existing suites green",
"status": "pending",
"activeForm": "Testing the site command plane",
"blockedBy": [
"T1B.2",
"T1B.3"
]
},
{
"id": "P1B.DoD",
"phase": "1B",
"subject": "1B DoD: rig central on Grpc for site-a proves full command matrix incl. standby parked retry; rebased on 1A; PR merged",
"status": "pending",
"activeForm": "Verifying the 1B DoD",
"blockedBy": [
"T1B.4",
"P1A.DoD"
]
},
{
"id": "P2",
"phase": "2",
"subject": "All sites CentralTransport=Grpc; central-kill S&F soak, failback observed, health sequences clean",
"status": "pending",
"activeForm": "Running the site->central cutover soak",
"blockedBy": [
"P1A.DoD"
]
},
{
"id": "P3",
"phase": "3",
"subject": "Central SiteTransport=Grpc all sites; full UI command matrix; site-kill mid-command clean; zero ClusterClient activity",
"status": "pending",
"activeForm": "Running the central->site cutover soak",
"blockedBy": [
"P1B.DoD"
]
},
{
"id": "P4.1",
"phase": "4",
"subject": "Flip both flag defaults to Grpc + soak; delete Akka transports, ClusterClient creation, DefaultSiteClientFactory, receptionist registrations, CentralContactPoints, then the flags",
"status": "pending",
"activeForm": "Deleting the ClusterClient transport",
"blockedBy": [
"P2",
"P3"
]
},
{
"id": "P4.2",
"phase": "4",
"subject": "Grep-gates pass + docs updated (grpc_streams.md, Component-Host.md, Component-StoreAndForward.md, known-issues cross-ref)",
"status": "pending",
"activeForm": "Running the deletion grep-gates and doc sweep",
"blockedBy": [
"P4.1"
]
},
{
"id": "P5",
"phase": "5",
"subject": "Live gate, 8 checks, recorded in docs/plans/2026-07-22-clusterclient-to-grpc-live-gate.md",
"status": "pending",
"activeForm": "Running the live gate",
"blockedBy": [
"P4.2"
]
}
]
}
+28 -1
View File
@@ -109,6 +109,31 @@ The streaming protocol is defined in `sitestream.proto` (`src/ZB.MOM.WW.ScadaBri
- The `oneof event` pattern is extensible — future event types (health metrics, connection state changes) are added as new fields without breaking existing consumers. - The `oneof event` pattern is extensible — future event types (health metrics, connection state changes) are added as new fields without breaking existing consumers.
- Proto field numbers are never reused; new RPCs and message fields are appended additively. Old clients ignore unknown `oneof` variants. - Proto field numbers are never reused; new RPCs and message fields are appended additively. Old clients ignore unknown `oneof` variants.
##### Authentication (preshared key, 2026-07-22)
Every RPC on this service — streaming and unary alike — requires a preshared key, presented as
`authorization: Bearer <key>` metadata and verified by `ControlPlaneAuthInterceptor` on the site
node. The gate is **fail-closed**: with no key configured, every call is refused with
`PermissionDenied`, and `StartupValidator` refuses to boot a site node in that state (an unset key
would otherwise leave the node joined and reporting healthy while serving nothing).
Keys are scoped **one per site** — secret `SB-GRPC-PSK-<siteId>`, so a compromised site never
yields another site's key. The site node reads its own key from
`ScadaBridge:Communication:GrpcPsk` (in production a `${secret:}` reference expanded before the
host is built); central resolves each site's key at channel-build time via `SitePskProvider`,
because sites are created at runtime and cannot be enumerated in configuration at boot.
`ControlPlaneCredentials` binds the key to the channel as `CallCredentials`, so no individual call
site can omit it.
This is distinct from `LocalDb:Replication:ApiKey`, which gates
`/localdb_sync.v1.LocalDbSync/` on the same listener via its own interceptor: that key
authenticates the *pair partner* for database replication, a different trust relationship, and the
two are never shared.
The transport remains h2c, so the key is readable and replayable by anyone on the path — the
boundary still assumes a trusted network, with the bar raised from "can reach the port" to "can
read the traffic". TLS is follow-on hardening and does not change this design.
#### Enriched AlarmStateUpdate (Native Alarm Mirror) #### Enriched AlarmStateUpdate (Native Alarm Mirror)
`AlarmStateUpdate` carries the read-only native alarm mirror (Computed, native OPC UA, and native MxAccess Gateway alarms) to central over the **existing gRPC real-time stream** — no new transport, no command/control round-trip. The message was extended **additively**: existing fields 17 are unchanged, and fields 823 carry the enriched native-alarm state. Old clients that only read fields 17 continue to work; new fields are populated only where the source provides them. `AlarmStateUpdate` carries the read-only native alarm mirror (Computed, native OPC UA, and native MxAccess Gateway alarms) to central over the **existing gRPC real-time stream** — no new transport, no command/control round-trip. The message was extended **additively**: existing fields 17 are unchanged, and fields 823 carry the enriched native-alarm state. Old clients that only read fields 17 continue to work; new fields are populated only where the source provides them.
@@ -306,7 +331,9 @@ Akka.NET guarantees message ordering between a specific sender/receiver actor pa
## ManagementActor and ClusterClient ## ManagementActor and ClusterClient
The ManagementActor is registered at the well-known path `/user/management` on central nodes and advertised via **ClusterClientReceptionist**. External tools (primarily the CLI) connect using Akka.NET ClusterClient, which contacts the receptionist to discover the ManagementActor. This is a separate ClusterClient usage from the inter-cluster ClusterClient connections used for central-site messaging — the CLI does not participate in cluster membership or affect the hub-and-spoke topology. The ManagementActor runs at the well-known path `/user/management` on central nodes. It is **not** advertised via ClusterClientReceptionist, and no ClusterClient reaches it.
That registration existed until 2026-07-22 for an out-of-cluster CLI that was never built. The shipped CLI speaks **HTTP Basic to the central `/management` endpoints**; those endpoints ask the ManagementActor **in-process** through `ManagementActorHolder` (`ManagementEndpoints.cs`). Nothing in the repo ever sent to `/user/management` across the boundary, so the registration was deleted — leaving exactly one receptionist registration per cluster role (`CentralCommunicationActor` on central, `SiteCommunicationActor` on sites), both of which serve inter-cluster central↔site messaging and are themselves scheduled for removal by the gRPC transport migration (`docs/plans/2026-07-22-clusterclient-to-grpc-plan.md`).
## Connection Failure Behavior ## Connection Failure Behavior
+5 -1
View File
@@ -125,7 +125,11 @@ The Host bootstraps the Akka.NET actor system from a **hand-assembled, injection
### REQ-HOST-6a: ClusterClientReceptionist (Central Only) ### REQ-HOST-6a: ClusterClientReceptionist (Central Only)
On central nodes, the Host must configure the Akka.NET **ClusterClientReceptionist** and register the ManagementActor with it. This allows external processes (e.g., the CLI) to discover and communicate with the ManagementActor via ClusterClient without joining the cluster as full members. The receptionist is started as part of the Akka.NET bootstrap (REQ-HOST-6) on central nodes only. On central nodes, the Host must configure the Akka.NET **ClusterClientReceptionist** and register the **CentralCommunicationActor** with it, so that site clusters' ClusterClients can reach the central command/control endpoint without joining the central cluster.
**The ManagementActor is NOT registered with the receptionist** (removed 2026-07-22). That registration was written for an out-of-cluster CLI that was never built: the shipped CLI speaks HTTP Basic to the central `/management` endpoints, which ask the ManagementActor **in-process** via `ManagementActorHolder` (`ManagementEndpoints.cs`). No sender to `/user/management` existed anywhere in the repo, so the registration only widened the cluster-client surface for nothing. The actor itself still runs at `/user/management`; only its cross-boundary advertisement is gone.
> **Migration note.** This receptionist registration — and the `CentralCommunicationActor` one that remains — are scheduled for deletion once the site↔central transport moves to gRPC. See `docs/plans/2026-07-22-clusterclient-to-grpc-plan.md`.
### REQ-HOST-7: ASP.NET Web Endpoints ### REQ-HOST-7: ASP.NET Web Endpoints
@@ -24,7 +24,7 @@ Central cluster only. The ManagementActor runs as a plain actor on **every** cen
### ManagementActor ### ManagementActor
The central actor that receives and processes all management commands. Registered at a well-known actor path (`/user/management`) and with ClusterClientReceptionist. The central actor that receives and processes all management commands. Created at the well-known actor path (`/user/management`) and handed to `ManagementActorHolder`, which is how every in-process caller (notably `ManagementEndpoints`) reaches it. It is **not** advertised via ClusterClientReceptionist — see the note under "ManagementActor and ClusterClient" in Component-Communication.
### ManagementEndpoints ### ManagementEndpoints
@@ -145,7 +145,7 @@ public sealed class GrpcPullAuditEventsClient : IPullAuditEventsClient
{ {
try try
{ {
var reply = await _invoker.InvokeAsync(endpoint, request, ct).ConfigureAwait(false); var reply = await _invoker.InvokeAsync(siteId, endpoint, request, ct).ConfigureAwait(false);
return (reply, false); return (reply, false);
} }
catch (RpcException ex) when (IsTolerable(ex.StatusCode)) catch (RpcException ex) when (IsTolerable(ex.StatusCode))
@@ -226,11 +226,17 @@ public sealed class GrpcPullAuditEventsClient : IPullAuditEventsClient
/// May throw <see cref="RpcException"/> / <see cref="HttpRequestException"/> /// May throw <see cref="RpcException"/> / <see cref="HttpRequestException"/>
/// on transport faults — the caller classifies and swallows tolerable ones. /// on transport faults — the caller classifies and swallows tolerable ones.
/// </summary> /// </summary>
/// <param name="siteId">
/// The site being pulled from. Selects which preshared key the call presents —
/// <c>PullAuditEvents</c> is gated by the site's <c>ControlPlaneAuthInterceptor</c>, and
/// keys are per-site, so the endpoint alone is not enough to authenticate.
/// </param>
/// <param name="endpoint">The site gRPC authority (e.g. <c>http://site-a:8083</c>).</param> /// <param name="endpoint">The site gRPC authority (e.g. <c>http://site-a:8083</c>).</param>
/// <param name="request">The wire-format pull request.</param> /// <param name="request">The wire-format pull request.</param>
/// <param name="ct">Cancellation token.</param> /// <param name="ct">Cancellation token.</param>
/// <returns>The wire-format pull response.</returns> /// <returns>The wire-format pull response.</returns>
Task<ProtoPullResponse> InvokeAsync(string endpoint, ProtoPullRequest request, CancellationToken ct); Task<ProtoPullResponse> InvokeAsync(
string siteId, string endpoint, ProtoPullRequest request, CancellationToken ct);
} }
} }
@@ -251,8 +257,9 @@ public sealed class GrpcPullAuditEventsClient : IPullAuditEventsClient
public sealed class GrpcPullAuditEventsInvoker public sealed class GrpcPullAuditEventsInvoker
: GrpcPullAuditEventsClient.IPullAuditEventsInvoker, IDisposable : GrpcPullAuditEventsClient.IPullAuditEventsInvoker, IDisposable
{ {
private readonly ConcurrentDictionary<string, GrpcChannel> _channels = new(StringComparer.Ordinal); private readonly ConcurrentDictionary<(string Site, string Endpoint), GrpcChannel> _channels = new();
private readonly CommunicationOptions _options; private readonly CommunicationOptions _options;
private readonly ISitePskProvider? _pskProvider;
/// <summary> /// <summary>
/// Creates the invoker using default <see cref="CommunicationOptions"/>. /// Creates the invoker using default <see cref="CommunicationOptions"/>.
@@ -268,15 +275,27 @@ public sealed class GrpcPullAuditEventsInvoker
/// </summary> /// </summary>
/// <param name="options">Communication options supplying gRPC keepalive timings.</param> /// <param name="options">Communication options supplying gRPC keepalive timings.</param>
public GrpcPullAuditEventsInvoker(CommunicationOptions options) public GrpcPullAuditEventsInvoker(CommunicationOptions options)
: this(options, pskProvider: null)
{
}
/// <summary>
/// Creates the invoker with per-site call credentials, the production shape: the site's
/// <c>ControlPlaneAuthInterceptor</c> refuses an unauthenticated <c>PullAuditEvents</c>.
/// </summary>
/// <param name="options">Communication options supplying gRPC keepalive timings.</param>
/// <param name="pskProvider">Resolves each site's preshared key; null dials unauthenticated.</param>
public GrpcPullAuditEventsInvoker(CommunicationOptions options, ISitePskProvider? pskProvider)
{ {
_options = options ?? throw new ArgumentNullException(nameof(options)); _options = options ?? throw new ArgumentNullException(nameof(options));
_pskProvider = pskProvider;
} }
/// <inheritdoc /> /// <inheritdoc />
public async Task<ProtoPullResponse> InvokeAsync( public async Task<ProtoPullResponse> InvokeAsync(
string endpoint, ProtoPullRequest request, CancellationToken ct) string siteId, string endpoint, ProtoPullRequest request, CancellationToken ct)
{ {
var channel = GetOrCreateChannel(endpoint); var channel = GetOrCreateChannel(siteId, endpoint);
var client = new SiteStreamService.SiteStreamServiceClient(channel); var client = new SiteStreamService.SiteStreamServiceClient(channel);
using var call = client.PullAuditEventsAsync(request, cancellationToken: ct); using var call = client.PullAuditEventsAsync(request, cancellationToken: ct);
return await call.ResponseAsync.ConfigureAwait(false); return await call.ResponseAsync.ConfigureAwait(false);
@@ -288,12 +307,13 @@ public sealed class GrpcPullAuditEventsInvoker
// pool) and the loser would leak. Create-then-GetOrAdd-then-dispose-if-lost // pool) and the loser would leak. Create-then-GetOrAdd-then-dispose-if-lost
// mirrors SiteStreamGrpcClientFactory: only the channel actually installed // mirrors SiteStreamGrpcClientFactory: only the channel actually installed
// survives; a channel that lost the race is disposed immediately. // survives; a channel that lost the race is disposed immediately.
private GrpcChannel GetOrCreateChannel(string endpoint) private GrpcChannel GetOrCreateChannel(string siteId, string endpoint)
{ {
if (!_channels.TryGetValue(endpoint, out var channel)) var key = (siteId, endpoint);
if (!_channels.TryGetValue(key, out var channel))
{ {
var created = CreateChannel(endpoint); var created = CreateChannel(siteId, endpoint);
channel = _channels.GetOrAdd(endpoint, created); channel = _channels.GetOrAdd(key, created);
if (!ReferenceEquals(channel, created)) if (!ReferenceEquals(channel, created))
{ {
created.Dispose(); created.Dispose();
@@ -302,7 +322,10 @@ public sealed class GrpcPullAuditEventsInvoker
return channel; return channel;
} }
private GrpcChannel CreateChannel(string endpoint) => // Keyed by (site, endpoint) rather than endpoint alone: the call credentials are bound to
// the channel, and they are per-site, so two sites sharing an endpoint string would
// otherwise share one channel carrying the first site's key.
private GrpcChannel CreateChannel(string siteId, string endpoint) =>
GrpcChannel.ForAddress(endpoint, new GrpcChannelOptions GrpcChannel.ForAddress(endpoint, new GrpcChannelOptions
{ {
HttpHandler = new SocketsHttpHandler HttpHandler = new SocketsHttpHandler
@@ -311,7 +334,7 @@ public sealed class GrpcPullAuditEventsInvoker
KeepAlivePingTimeout = _options.GrpcKeepAlivePingTimeout, KeepAlivePingTimeout = _options.GrpcKeepAlivePingTimeout,
KeepAlivePingPolicy = HttpKeepAlivePingPolicy.Always, KeepAlivePingPolicy = HttpKeepAlivePingPolicy.Always,
}, },
}); }.WithSiteCredentials(_pskProvider, siteId));
/// <summary>Disposes all cached channels.</summary> /// <summary>Disposes all cached channels.</summary>
public void Dispose() public void Dispose()
@@ -174,7 +174,7 @@ public sealed class GrpcPullSiteCallsClient : IPullSiteCallsClient
{ {
try try
{ {
var reply = await _invoker.InvokeAsync(endpoint, request, ct).ConfigureAwait(false); var reply = await _invoker.InvokeAsync(siteId, endpoint, request, ct).ConfigureAwait(false);
return (reply, false); return (reply, false);
} }
catch (RpcException ex) when (IsTolerable(ex.StatusCode)) catch (RpcException ex) when (IsTolerable(ex.StatusCode))
@@ -254,11 +254,17 @@ public sealed class GrpcPullSiteCallsClient : IPullSiteCallsClient
/// May throw <see cref="RpcException"/> / <see cref="HttpRequestException"/> /// May throw <see cref="RpcException"/> / <see cref="HttpRequestException"/>
/// on transport faults — the caller classifies and swallows tolerable ones. /// on transport faults — the caller classifies and swallows tolerable ones.
/// </summary> /// </summary>
/// <param name="siteId">
/// The site being pulled from. Selects which preshared key the call presents —
/// <c>PullSiteCalls</c> is gated by the site's <c>ControlPlaneAuthInterceptor</c>, and
/// keys are per-site, so the endpoint alone is not enough to authenticate.
/// </param>
/// <param name="endpoint">The site gRPC authority (e.g. <c>http://site-a:8083</c>).</param> /// <param name="endpoint">The site gRPC authority (e.g. <c>http://site-a:8083</c>).</param>
/// <param name="request">The wire-format pull request.</param> /// <param name="request">The wire-format pull request.</param>
/// <param name="ct">Cancellation token.</param> /// <param name="ct">Cancellation token.</param>
/// <returns>The wire-format pull response.</returns> /// <returns>The wire-format pull response.</returns>
Task<ProtoPullResponse> InvokeAsync(string endpoint, ProtoPullRequest request, CancellationToken ct); Task<ProtoPullResponse> InvokeAsync(
string siteId, string endpoint, ProtoPullRequest request, CancellationToken ct);
} }
} }
@@ -277,8 +283,9 @@ public sealed class GrpcPullSiteCallsClient : IPullSiteCallsClient
public sealed class GrpcPullSiteCallsInvoker public sealed class GrpcPullSiteCallsInvoker
: GrpcPullSiteCallsClient.IPullSiteCallsInvoker, IDisposable : GrpcPullSiteCallsClient.IPullSiteCallsInvoker, IDisposable
{ {
private readonly ConcurrentDictionary<string, GrpcChannel> _channels = new(StringComparer.Ordinal); private readonly ConcurrentDictionary<(string Site, string Endpoint), GrpcChannel> _channels = new();
private readonly CommunicationOptions _options; private readonly CommunicationOptions _options;
private readonly ISitePskProvider? _pskProvider;
/// <summary>Creates the invoker using default <see cref="CommunicationOptions"/>.</summary> /// <summary>Creates the invoker using default <see cref="CommunicationOptions"/>.</summary>
public GrpcPullSiteCallsInvoker() public GrpcPullSiteCallsInvoker()
@@ -292,15 +299,27 @@ public sealed class GrpcPullSiteCallsInvoker
/// </summary> /// </summary>
/// <param name="options">Communication options supplying gRPC keepalive timings.</param> /// <param name="options">Communication options supplying gRPC keepalive timings.</param>
public GrpcPullSiteCallsInvoker(CommunicationOptions options) public GrpcPullSiteCallsInvoker(CommunicationOptions options)
: this(options, pskProvider: null)
{
}
/// <summary>
/// Creates the invoker with per-site call credentials, the production shape: the site's
/// <c>ControlPlaneAuthInterceptor</c> refuses an unauthenticated <c>PullSiteCalls</c>.
/// </summary>
/// <param name="options">Communication options supplying gRPC keepalive timings.</param>
/// <param name="pskProvider">Resolves each site's preshared key; null dials unauthenticated.</param>
public GrpcPullSiteCallsInvoker(CommunicationOptions options, ISitePskProvider? pskProvider)
{ {
_options = options ?? throw new ArgumentNullException(nameof(options)); _options = options ?? throw new ArgumentNullException(nameof(options));
_pskProvider = pskProvider;
} }
/// <inheritdoc /> /// <inheritdoc />
public async Task<ProtoPullResponse> InvokeAsync( public async Task<ProtoPullResponse> InvokeAsync(
string endpoint, ProtoPullRequest request, CancellationToken ct) string siteId, string endpoint, ProtoPullRequest request, CancellationToken ct)
{ {
var channel = GetOrCreateChannel(endpoint); var channel = GetOrCreateChannel(siteId, endpoint);
var client = new SiteStreamService.SiteStreamServiceClient(channel); var client = new SiteStreamService.SiteStreamServiceClient(channel);
using var call = client.PullSiteCallsAsync(request, cancellationToken: ct); using var call = client.PullSiteCallsAsync(request, cancellationToken: ct);
return await call.ResponseAsync.ConfigureAwait(false); return await call.ResponseAsync.ConfigureAwait(false);
@@ -310,12 +329,13 @@ public sealed class GrpcPullSiteCallsInvoker
// concurrent first dials of the same endpoint can both build a GrpcChannel; // concurrent first dials of the same endpoint can both build a GrpcChannel;
// only the channel actually installed survives, the loser is disposed. // only the channel actually installed survives, the loser is disposed.
// Mirrors SiteStreamGrpcClientFactory / GrpcPullAuditEventsInvoker. // Mirrors SiteStreamGrpcClientFactory / GrpcPullAuditEventsInvoker.
private GrpcChannel GetOrCreateChannel(string endpoint) private GrpcChannel GetOrCreateChannel(string siteId, string endpoint)
{ {
if (!_channels.TryGetValue(endpoint, out var channel)) var key = (siteId, endpoint);
if (!_channels.TryGetValue(key, out var channel))
{ {
var created = CreateChannel(endpoint); var created = CreateChannel(siteId, endpoint);
channel = _channels.GetOrAdd(endpoint, created); channel = _channels.GetOrAdd(key, created);
if (!ReferenceEquals(channel, created)) if (!ReferenceEquals(channel, created))
{ {
created.Dispose(); created.Dispose();
@@ -324,7 +344,10 @@ public sealed class GrpcPullSiteCallsInvoker
return channel; return channel;
} }
private GrpcChannel CreateChannel(string endpoint) => // Keyed by (site, endpoint) rather than endpoint alone: the call credentials are bound to
// the channel, and they are per-site, so two sites sharing an endpoint string would
// otherwise share one channel carrying the first site's key.
private GrpcChannel CreateChannel(string siteId, string endpoint) =>
GrpcChannel.ForAddress(endpoint, new GrpcChannelOptions GrpcChannel.ForAddress(endpoint, new GrpcChannelOptions
{ {
HttpHandler = new SocketsHttpHandler HttpHandler = new SocketsHttpHandler
@@ -333,7 +356,7 @@ public sealed class GrpcPullSiteCallsInvoker
KeepAlivePingTimeout = _options.GrpcKeepAlivePingTimeout, KeepAlivePingTimeout = _options.GrpcKeepAlivePingTimeout,
KeepAlivePingPolicy = HttpKeepAlivePingPolicy.Always, KeepAlivePingPolicy = HttpKeepAlivePingPolicy.Always,
}, },
}); }.WithSiteCredentials(_pskProvider, siteId));
/// <summary>Disposes all cached channels.</summary> /// <summary>Disposes all cached channels.</summary>
public void Dispose() public void Dispose()
@@ -511,9 +511,14 @@ public static class ServiceCollectionExtensions
var options = sp var options = sp
.GetService<Microsoft.Extensions.Options.IOptions< .GetService<Microsoft.Extensions.Options.IOptions<
ZB.MOM.WW.ScadaBridge.Communication.CommunicationOptions>>(); ZB.MOM.WW.ScadaBridge.Communication.CommunicationOptions>>();
// The PSK provider is central-only and optional in DI, so GetService (not
// GetRequiredService): a host without one dials unauthenticated and the site
// refuses it, which is the fail-closed outcome we want rather than a
// resolution crash at composition time.
var psk = sp.GetService<ZB.MOM.WW.ScadaBridge.Communication.Grpc.ISitePskProvider>();
return options is null return options is null
? new GrpcPullAuditEventsInvoker() ? new GrpcPullAuditEventsInvoker(new ZB.MOM.WW.ScadaBridge.Communication.CommunicationOptions(), psk)
: new GrpcPullAuditEventsInvoker(options.Value); : new GrpcPullAuditEventsInvoker(options.Value, psk);
}); });
services.TryAddSingleton<GrpcPullAuditEventsClient.IPullAuditEventsInvoker>( services.TryAddSingleton<GrpcPullAuditEventsClient.IPullAuditEventsInvoker>(
sp => sp.GetRequiredService<GrpcPullAuditEventsInvoker>()); sp => sp.GetRequiredService<GrpcPullAuditEventsInvoker>());
@@ -536,9 +541,14 @@ public static class ServiceCollectionExtensions
var options = sp var options = sp
.GetService<Microsoft.Extensions.Options.IOptions< .GetService<Microsoft.Extensions.Options.IOptions<
ZB.MOM.WW.ScadaBridge.Communication.CommunicationOptions>>(); ZB.MOM.WW.ScadaBridge.Communication.CommunicationOptions>>();
// The PSK provider is central-only and optional in DI, so GetService (not
// GetRequiredService): a host without one dials unauthenticated and the site
// refuses it, which is the fail-closed outcome we want rather than a
// resolution crash at composition time.
var psk = sp.GetService<ZB.MOM.WW.ScadaBridge.Communication.Grpc.ISitePskProvider>();
return options is null return options is null
? new GrpcPullSiteCallsInvoker() ? new GrpcPullSiteCallsInvoker(new ZB.MOM.WW.ScadaBridge.Communication.CommunicationOptions(), psk)
: new GrpcPullSiteCallsInvoker(options.Value); : new GrpcPullSiteCallsInvoker(options.Value, psk);
}); });
services.TryAddSingleton<GrpcPullSiteCallsClient.IPullSiteCallsInvoker>( services.TryAddSingleton<GrpcPullSiteCallsClient.IPullSiteCallsInvoker>(
sp => sp.GetRequiredService<GrpcPullSiteCallsInvoker>()); sp => sp.GetRequiredService<GrpcPullSiteCallsInvoker>());
+4 -2
View File
@@ -1987,9 +1987,11 @@ scadabridge --url <url> cached-call discard --site-id <string> --tracked-operati
## Architecture Notes ## Architecture Notes
The CLI connects to the Central cluster using Akka.NET's `ClusterClient`. It does not join the cluster — it contacts the `ClusterClientReceptionist` on one of the configured Central nodes and sends commands to the `ManagementActor` at path `/user/management`. The CLI connects to the Central cluster over **HTTP** (`ManagementHttpClient`), posting to the `/management` endpoints at the configured `managementUrl` — normally the Traefik load balancer, which routes to the active central node. It carries HTTP **Basic** credentials from `--username`/`--password`. Central's endpoint handler asks the `ManagementActor` in-process via `ManagementActorHolder`.
The connection is established per-command invocation and torn down cleanly via `CoordinatedShutdown` when the command completes. There is no Akka dependency in the CLI at all: it does not join the cluster, does not use `ClusterClient`, and does not contact a `ClusterClientReceptionist`. (Earlier revisions of this document described a ClusterClient transport that was never built.)
An `HttpClient` is created per command invocation and disposed when the command completes.
Role enforcement is applied by the ManagementActor on the server side. The CLI authenticates against LDAP using `--username` / `--password`, resolves LDAP group memberships, then maps groups to ScadaBridge roles (Admin, Design, Deployment) via role mappings configured in the security settings. Operations require the appropriate role — for example, creating templates requires `Design`, deploying requires `Deployment`. In the test environment, use the `multi-role` user (password: `password`) which has all three roles. Role enforcement is applied by the ManagementActor on the server side. The CLI authenticates against LDAP using `--username` / `--password`, resolves LDAP group memberships, then maps groups to ScadaBridge roles (Admin, Design, Deployment) via role mappings configured in the security settings. Operations require the appropriate role — for example, creating templates requires `Design`, deploying requires `Deployment`. In the test environment, use the `multi-role` user (password: `password`) which has all three roles.
@@ -0,0 +1,185 @@
using Akka.Actor;
using Akka.Cluster.Tools.Client;
using Akka.Event;
using ZB.MOM.WW.ScadaBridge.Commons.Messages.Audit;
using ZB.MOM.WW.ScadaBridge.Commons.Messages.Deployment;
using ZB.MOM.WW.ScadaBridge.Commons.Messages.Health;
using ZB.MOM.WW.ScadaBridge.Commons.Messages.Notification;
namespace ZB.MOM.WW.ScadaBridge.Communication.Actors;
/// <summary>
/// The <see cref="ICentralTransport"/> that carries the seven site→central sends over Akka
/// <c>ClusterClient</c> — the transport in production today, and the default. Every method is a
/// verbatim lift of the corresponding <c>SiteCommunicationActor</c> send block: it forwards a
/// <see cref="ClusterClient.Send"/> to <c>/user/central-communication</c> with the
/// <paramref name="replyTo"/> as the send's sender, so central's reply routes straight back to the
/// waiting Ask rather than through the site communication actor.
/// </summary>
/// <remarks>
/// The <c>ClusterClient</c> reference arrives after construction via
/// <see cref="SetCentralClient"/> (the actor forwards the <c>RegisterCentralClient</c> message it
/// receives once the Host builds the client). Until then — and if central contact points are not
/// configured at all — the client is null and each method answers the same transient-failure reply
/// the old inline handlers did.
/// </remarks>
public sealed class AkkaCentralTransport : ICentralTransport
{
/// <summary>The receptionist-registered path of the central communication actor.</summary>
private const string CentralPath = "/user/central-communication";
private readonly ILoggingAdapter? _log;
private IActorRef? _centralClient;
/// <summary>Creates the transport with no logging adapter (behaviourally identical; warnings are dropped).</summary>
public AkkaCentralTransport()
{
}
/// <summary>Creates the transport bound to the site communication actor's logging adapter.</summary>
/// <param name="log">Logging adapter used for the "no ClusterClient registered" warnings.</param>
public AkkaCentralTransport(ILoggingAdapter log)
{
_log = log;
}
/// <summary>
/// Registers the central <c>ClusterClient</c> once the Host has built it. Called from the site
/// communication actor's <c>RegisterCentralClient</c> handler.
/// </summary>
/// <param name="centralClient">The ClusterClient reaching the central cluster.</param>
public void SetCentralClient(IActorRef centralClient)
{
_centralClient = centralClient;
_log?.Info("Registered central ClusterClient");
}
/// <inheritdoc />
public void SubmitNotification(NotificationSubmit message, IActorRef replyTo)
{
if (_centralClient == null)
{
// No ClusterClient registered yet (e.g. central contact points not
// configured, or registration not yet completed). A non-accepted ack
// makes the S&F forwarder treat this as transient and retry later.
_log?.Warning(
"Cannot forward NotificationSubmit {0} — no central ClusterClient registered",
message.NotificationId);
replyTo.Tell(new NotificationSubmitAck(
message.NotificationId, Accepted: false, Error: "Central ClusterClient not registered"));
return;
}
_log?.Debug("Forwarding NotificationSubmit {0} to central", message.NotificationId);
_centralClient.Tell(new ClusterClient.Send(CentralPath, message), replyTo);
}
/// <inheritdoc />
public void QueryNotificationStatus(NotificationStatusQuery message, IActorRef replyTo)
{
if (_centralClient == null)
{
// No ClusterClient registered yet. Reply Found: false so Notify.Status
// falls back to the site S&F buffer to decide Forwarding vs Unknown.
_log?.Warning(
"Cannot forward NotificationStatusQuery {0} — no central ClusterClient registered",
message.NotificationId);
replyTo.Tell(new NotificationStatusResponse(
message.CorrelationId, Found: false, Status: "Unknown",
RetryCount: 0, LastError: null, DeliveredAt: null));
return;
}
_log?.Debug("Forwarding NotificationStatusQuery {0} to central", message.NotificationId);
_centralClient.Tell(new ClusterClient.Send(CentralPath, message), replyTo);
}
/// <inheritdoc />
public void IngestAuditEvents(IngestAuditEventsCommand message, IActorRef replyTo)
{
if (_centralClient == null)
{
// No ClusterClient registered yet. Faulting the Ask makes the
// SiteAuditTelemetryActor drain loop treat this as transient and keep
// the rows Pending for the next tick.
_log?.Warning(
"Cannot forward IngestAuditEventsCommand ({0} events) — no central ClusterClient registered",
message.Events.Count);
replyTo.Tell(new Status.Failure(
new InvalidOperationException("Central ClusterClient not registered")));
return;
}
_log?.Debug("Forwarding IngestAuditEventsCommand ({0} events) to central", message.Events.Count);
_centralClient.Tell(new ClusterClient.Send(CentralPath, message), replyTo);
}
/// <inheritdoc />
public void IngestCachedTelemetry(IngestCachedTelemetryCommand message, IActorRef replyTo)
{
if (_centralClient == null)
{
_log?.Warning(
"Cannot forward IngestCachedTelemetryCommand ({0} entries) — no central ClusterClient registered",
message.Entries.Count);
replyTo.Tell(new Status.Failure(
new InvalidOperationException("Central ClusterClient not registered")));
return;
}
_log?.Debug("Forwarding IngestCachedTelemetryCommand ({0} entries) to central", message.Entries.Count);
_centralClient.Tell(new ClusterClient.Send(CentralPath, message), replyTo);
}
/// <inheritdoc />
public void ReconcileSite(ReconcileSiteRequest message, IActorRef replyTo)
{
if (_centralClient == null)
{
// No ClusterClient registered yet. Faulting the Ask makes the
// SiteReconciliationActor treat the pass as best-effort-failed; it
// logs a warning and retries reconcile on the next node startup.
_log?.Warning(
"Cannot forward ReconcileSiteRequest for site {0} node {1} — no central ClusterClient registered",
message.SiteIdentifier, message.NodeId);
replyTo.Tell(new Status.Failure(
new InvalidOperationException("Central ClusterClient not registered")));
return;
}
_log?.Debug(
"Forwarding ReconcileSiteRequest for site {0} node {1} ({2} local instance(s)) to central",
message.SiteIdentifier, message.NodeId, message.LocalNameToRevisionHash.Count);
_centralClient.Tell(new ClusterClient.Send(CentralPath, message), replyTo);
}
/// <inheritdoc />
public void ReportSiteHealth(SiteHealthReport message, IActorRef replyTo)
{
if (_centralClient == null)
{
// No ClusterClient registered yet. A non-accepted ack makes the
// sender's counter-restore path treat this tick as a loss.
_log?.Warning(
"Cannot forward SiteHealthReport #{0} — no central ClusterClient registered",
message.SequenceNumber);
replyTo.Tell(new SiteHealthReportAck(
message.SiteId, message.SequenceNumber, Accepted: false,
Error: "Central ClusterClient not registered"));
return;
}
_centralClient.Tell(new ClusterClient.Send(CentralPath, message), replyTo);
}
/// <inheritdoc />
public void SendHeartbeat(HeartbeatMessage message, IActorRef self)
{
if (_centralClient == null)
{
return;
}
_centralClient.Tell(new ClusterClient.Send(CentralPath, message), self);
}
}
@@ -0,0 +1,78 @@
using Akka.Actor;
using ZB.MOM.WW.ScadaBridge.Commons.Messages.Audit;
using ZB.MOM.WW.ScadaBridge.Commons.Messages.Deployment;
using ZB.MOM.WW.ScadaBridge.Commons.Messages.Health;
using ZB.MOM.WW.ScadaBridge.Commons.Messages.Notification;
namespace ZB.MOM.WW.ScadaBridge.Communication.Actors;
/// <summary>
/// The site→central transport seam: one method per the seven messages
/// <see cref="SiteCommunicationActor"/> sends to <c>/user/central-communication</c> today.
/// </summary>
/// <remarks>
/// <para>
/// The actor's receive handlers no longer own the wire plumbing — they capture the current
/// <c>Sender</c> and hand it to the transport as <paramref name="replyTo"/>. Two implementations
/// exist behind the <c>ScadaBridge:Communication:CentralTransport</c> flag: the default
/// <see cref="AkkaCentralTransport"/> (verbatim of the old <c>ClusterClient.Send</c> path,
/// including the exact sender-forwarding that routes central's reply straight back to the waiting
/// Ask) and <see cref="Grpc.GrpcCentralTransport"/> (a gRPC dial of <c>CentralControlService</c>).
/// </para>
/// <para>
/// <b>Reply/fault contract, identical on both transports.</b> Each Ask-returning method (all but
/// the heartbeat) guarantees exactly one reply eventually lands at <paramref name="replyTo"/>:
/// either the real reply type the caller Asks for, or a transient-failure signal. The Akka path
/// sends a not-accepted ack / <see cref="Status.Failure"/> when no ClusterClient is registered;
/// the gRPC path sends <see cref="Status.Failure"/> on any non-OK status (a timeout or an
/// <c>Unavailable</c> that could not be failed over). Both are what the S&amp;F / audit / health
/// layers above the seam already treat as transient — rows stay buffered, counters restore, the
/// pass re-runs.
/// </para>
/// <para>
/// <b>The heartbeat stays fire-and-forget end-to-end.</b> <see cref="SendHeartbeat"/> takes no
/// <c>replyTo</c> and never surfaces a fault: a transport failure is swallowed and logged, exactly
/// as the old <c>Tell</c> dropped it. A failing heartbeat must never fault the site's heartbeat
/// timer path.
/// </para>
/// </remarks>
public interface ICentralTransport
{
/// <summary>Forwards a buffered notification; central replies <see cref="NotificationSubmitAck"/> to <paramref name="replyTo"/>.</summary>
/// <param name="message">The notification submission.</param>
/// <param name="replyTo">The actor (the S&amp;F forwarder's Ask) the ack routes back to.</param>
void SubmitNotification(NotificationSubmit message, IActorRef replyTo);
/// <summary>Forwards a Notify.Status query; central replies <see cref="NotificationStatusResponse"/> to <paramref name="replyTo"/>.</summary>
/// <param name="message">The status query.</param>
/// <param name="replyTo">The actor (the Notify helper's Ask) the response routes back to.</param>
void QueryNotificationStatus(NotificationStatusQuery message, IActorRef replyTo);
/// <summary>Pushes a batch of audit events; central replies <see cref="IngestAuditEventsReply"/> to <paramref name="replyTo"/>.</summary>
/// <param name="message">The audit-event ingest command.</param>
/// <param name="replyTo">The actor (the telemetry drain's Ask) the reply routes back to.</param>
void IngestAuditEvents(IngestAuditEventsCommand message, IActorRef replyTo);
/// <summary>Pushes a batch of combined cached-call telemetry; central replies <see cref="IngestCachedTelemetryReply"/> to <paramref name="replyTo"/>.</summary>
/// <param name="message">The cached-telemetry ingest command.</param>
/// <param name="replyTo">The actor (the telemetry drain's Ask) the reply routes back to.</param>
void IngestCachedTelemetry(IngestCachedTelemetryCommand message, IActorRef replyTo);
/// <summary>Reports a node's startup inventory; central replies <see cref="ReconcileSiteResponse"/> to <paramref name="replyTo"/>.</summary>
/// <param name="message">The reconcile request.</param>
/// <param name="replyTo">The actor (the reconciliation Ask) the response routes back to.</param>
void ReconcileSite(ReconcileSiteRequest message, IActorRef replyTo);
/// <summary>Reports periodic site health; central replies <see cref="SiteHealthReportAck"/> to <paramref name="replyTo"/>.</summary>
/// <param name="message">The health report.</param>
/// <param name="replyTo">The actor (the health transport's Ask) the ack routes back to.</param>
void ReportSiteHealth(SiteHealthReport message, IActorRef replyTo);
/// <summary>
/// Sends an application heartbeat, fire-and-forget. Never replies and never faults; a failure
/// is swallowed and logged.
/// </summary>
/// <param name="message">The heartbeat.</param>
/// <param name="self">The site communication actor, used as the sender on the Akka path (ignored on gRPC).</param>
void SendHeartbeat(HeartbeatMessage message, IActorRef self);
}
@@ -1,6 +1,5 @@
using Akka.Actor; using Akka.Actor;
using Akka.Cluster; using Akka.Cluster;
using Akka.Cluster.Tools.Client;
using Akka.Event; using Akka.Event;
using ZB.MOM.WW.ScadaBridge.Commons.Messages.Artifacts; using ZB.MOM.WW.ScadaBridge.Commons.Messages.Artifacts;
using ZB.MOM.WW.ScadaBridge.Commons.Messages.Audit; using ZB.MOM.WW.ScadaBridge.Commons.Messages.Audit;
@@ -45,10 +44,14 @@ public class SiteCommunicationActor : ReceiveActor, IWithTimers
private readonly IActorRef _deploymentManagerProxy; private readonly IActorRef _deploymentManagerProxy;
/// <summary> /// <summary>
/// ClusterClient reference for sending messages to the central cluster. /// The site→central transport. Finalized in <see cref="PreStart"/> to the injected instance,
/// Set via RegisterCentralClient message. /// or a default <see cref="AkkaCentralTransport"/> (ClusterClient) when none is supplied — so
/// the seven site→central sends delegate here rather than owning the wire plumbing inline.
/// </summary> /// </summary>
private IActorRef? _centralClient; private ICentralTransport _transport;
/// <summary>The transport supplied by the Host (null selects the default Akka transport).</summary>
private readonly ICentralTransport? _injectedTransport;
/// <summary> /// <summary>
/// Local actor references for routing specific message patterns. /// Local actor references for routing specific message patterns.
@@ -73,24 +76,36 @@ public class SiteCommunicationActor : ReceiveActor, IWithTimers
/// pass a stub so they do not need to load Akka.Cluster into the <c>TestKit</c> /// pass a stub so they do not need to load Akka.Cluster into the <c>TestKit</c>
/// ActorSystem. /// ActorSystem.
/// </param> /// </param>
/// <param name="transport">
/// The site→central transport. <c>null</c> (the default, used by every existing test and by
/// the Host's Akka path) selects an <see cref="AkkaCentralTransport"/> over ClusterClient; the
/// Host injects a <see cref="Grpc.GrpcCentralTransport"/> when
/// <c>ScadaBridge:Communication:CentralTransport</c> is <c>Grpc</c>.
/// </param>
public SiteCommunicationActor( public SiteCommunicationActor(
string siteId, string siteId,
CommunicationOptions options, CommunicationOptions options,
IActorRef deploymentManagerProxy, IActorRef deploymentManagerProxy,
Func<bool>? isActiveCheck = null, Func<bool>? isActiveCheck = null,
Func<string, string?>? failOverRole = null) Func<string, string?>? failOverRole = null,
ICentralTransport? transport = null)
{ {
_siteId = siteId; _siteId = siteId;
_options = options; _options = options;
_deploymentManagerProxy = deploymentManagerProxy; _deploymentManagerProxy = deploymentManagerProxy;
_isActiveCheck = isActiveCheck ?? DefaultIsActiveCheck; _isActiveCheck = isActiveCheck ?? DefaultIsActiveCheck;
_failOverRole = failOverRole ?? DefaultFailOverRole; _failOverRole = failOverRole ?? DefaultFailOverRole;
_injectedTransport = transport;
// Finalized in PreStart (where _log is usable for the default transport); assigned here
// too so the field is definitely-assigned for the constructor's Receive closures.
_transport = transport!;
// Registration // Registration. Feeding the ClusterClient into the transport is a no-op unless the
// default/Akka transport is in use — the gRPC transport dials configured endpoints and
// never receives this message (the Host does not create a ClusterClient for it).
Receive<RegisterCentralClient>(msg => Receive<RegisterCentralClient>(msg =>
{ {
_centralClient = msg.Client; (_transport as AkkaCentralTransport)?.SetCentralClient(msg.Client);
_log.Info("Registered central ClusterClient");
}); });
Receive<RegisterLocalHandler>(HandleRegisterLocalHandler); Receive<RegisterLocalHandler>(HandleRegisterLocalHandler);
@@ -274,157 +289,33 @@ public class SiteCommunicationActor : ReceiveActor, IWithTimers
// from cluster state, not from who received the message. // from cluster state, not from who received the message.
Receive<TriggerSiteFailover>(HandleTriggerSiteFailover); Receive<TriggerSiteFailover>(HandleTriggerSiteFailover);
// Notification Outbox: forward a buffered notification submitted by the site // The seven site→central sends now delegate to the injected transport (ClusterClient by
// Store-and-Forward Engine to the central cluster. The original Sender (the // default, gRPC when configured). Each handler captures the current Sender as the reply
// S&F forwarder's Ask) is forwarded as the ClusterClient.Send sender so the // target so central's reply routes straight back to the waiting Ask, not through this
// NotificationSubmitAck routes straight back to the waiting Ask, not here. // actor — the exact sender-forwarding the ClusterClient path relied on. The per-message
Receive<NotificationSubmit>(msg => // "no transport / not-accepted" fallbacks live inside the transport now.
{
if (_centralClient == null)
{
// No ClusterClient registered yet (e.g. central contact points not
// configured, or registration not yet completed). A non-accepted ack
// makes the S&F forwarder treat this as transient and retry later.
_log.Warning(
"Cannot forward NotificationSubmit {0} — no central ClusterClient registered",
msg.NotificationId);
Sender.Tell(new NotificationSubmitAck(
msg.NotificationId, Accepted: false, Error: "Central ClusterClient not registered"));
return;
}
_log.Debug("Forwarding NotificationSubmit {0} to central", msg.NotificationId); // Notification Outbox: forward a buffered notification (S&F forwarder's Ask → ack back).
_centralClient.Tell( Receive<NotificationSubmit>(msg => _transport.SubmitNotification(msg, Sender));
new ClusterClient.Send("/user/central-communication", msg), Sender);
});
// Notification Outbox: forward a Notify.Status query to the central cluster. // Notification Outbox: forward a Notify.Status query (Notify helper's Ask → response back).
// The original Sender (the Notify helper's Ask) is forwarded as the Receive<NotificationStatusQuery>(msg => _transport.QueryNotificationStatus(msg, Sender));
// ClusterClient.Send sender so the NotificationStatusResponse routes straight
// back to the waiting Ask, not here.
Receive<NotificationStatusQuery>(msg =>
{
if (_centralClient == null)
{
// No ClusterClient registered yet. Reply Found: false so Notify.Status
// falls back to the site S&F buffer to decide Forwarding vs Unknown.
_log.Warning(
"Cannot forward NotificationStatusQuery {0} — no central ClusterClient registered",
msg.NotificationId);
Sender.Tell(new NotificationStatusResponse(
msg.CorrelationId, Found: false, Status: "Unknown",
RetryCount: 0, LastError: null, DeliveredAt: null));
return;
}
_log.Debug("Forwarding NotificationStatusQuery {0} to central", msg.NotificationId); // Audit Log: forward a batch of site-local audit events (telemetry drain's Ask → reply back).
_centralClient.Tell( Receive<IngestAuditEventsCommand>(msg => _transport.IngestAuditEvents(msg, Sender));
new ClusterClient.Send("/user/central-communication", msg), Sender);
});
// Audit Log: forward a batch of site-local audit events to the // Audit Log: forward a batch of combined cached-call telemetry (telemetry drain's Ask → reply back).
// central cluster. The site SiteAuditTelemetryActor drains its SQLite Receive<IngestCachedTelemetryCommand>(msg => _transport.IngestCachedTelemetry(msg, Sender));
// Pending queue through the ClusterClientSiteAuditClient, which Asks
// this actor; the original Sender (that Ask) is passed as the
// ClusterClient.Send sender so the IngestAuditEventsReply routes
// straight back to the waiting Ask, not here. Mirrors NotificationSubmit.
Receive<IngestAuditEventsCommand>(msg =>
{
if (_centralClient == null)
{
// No ClusterClient registered yet (e.g. central contact points
// not configured, or registration not yet completed). Faulting
// the Ask makes the SiteAuditTelemetryActor drain loop treat
// this as transient and keep the rows Pending for the next tick.
_log.Warning(
"Cannot forward IngestAuditEventsCommand ({0} events) — no central ClusterClient registered",
msg.Events.Count);
Sender.Tell(new Status.Failure(
new InvalidOperationException("Central ClusterClient not registered")));
return;
}
_log.Debug("Forwarding IngestAuditEventsCommand ({0} events) to central", msg.Events.Count); // Site startup reconciliation: forward the node's local-inventory request (reconcile Ask → response back).
_centralClient.Tell( Receive<ReconcileSiteRequest>(msg => _transport.ReconcileSite(msg, Sender));
new ClusterClient.Send("/user/central-communication", msg), Sender);
});
// Audit Log: forward a batch of combined cached-call telemetry // Internal: send heartbeat tick.
// packets to the central cluster. Same forward + reply-routing pattern
// as IngestAuditEventsCommand; central replies with an
// IngestCachedTelemetryReply.
Receive<IngestCachedTelemetryCommand>(msg =>
{
if (_centralClient == null)
{
_log.Warning(
"Cannot forward IngestCachedTelemetryCommand ({0} entries) — no central ClusterClient registered",
msg.Entries.Count);
Sender.Tell(new Status.Failure(
new InvalidOperationException("Central ClusterClient not registered")));
return;
}
_log.Debug("Forwarding IngestCachedTelemetryCommand ({0} entries) to central", msg.Entries.Count);
_centralClient.Tell(
new ClusterClient.Send("/user/central-communication", msg), Sender);
});
// Site startup reconciliation: forward the node's local-inventory
// ReconcileSiteRequest to the central cluster. The original Sender (the
// SiteReconciliationActor's Ask) is passed as the ClusterClient.Send sender so
// the ReconcileSiteResponse routes straight back to the waiting Ask, not here.
// Mirrors IngestAuditEventsCommand.
Receive<ReconcileSiteRequest>(msg =>
{
if (_centralClient == null)
{
// No ClusterClient registered yet (e.g. central contact points not
// configured, or registration not yet completed). Faulting the Ask makes
// the SiteReconciliationActor treat the pass as best-effort-failed; it
// logs a warning and retries reconcile on the next node startup.
_log.Warning(
"Cannot forward ReconcileSiteRequest for site {0} node {1} — no central ClusterClient registered",
msg.SiteIdentifier, msg.NodeId);
Sender.Tell(new Status.Failure(
new InvalidOperationException("Central ClusterClient not registered")));
return;
}
_log.Debug(
"Forwarding ReconcileSiteRequest for site {0} node {1} ({2} local instance(s)) to central",
msg.SiteIdentifier, msg.NodeId, msg.LocalNameToRevisionHash.Count);
_centralClient.Tell(
new ClusterClient.Send("/user/central-communication", msg), Sender);
});
// Internal: send heartbeat tick
Receive<SendHeartbeat>(_ => SendHeartbeatToCentral()); Receive<SendHeartbeat>(_ => SendHeartbeatToCentral());
// Internal: forward health report to central. The original Sender (the // Internal: forward the periodic health report (health transport's Ask → ack back), so a
// AkkaHealthReportTransport's Ask) is forwarded as the ClusterClient.Send // lost report is observable end-to-end and the sender can restore its per-interval counters.
// sender so the central SiteHealthReportAck routes straight back to the Receive<SiteHealthReport>(msg => _transport.ReportSiteHealth(msg, Sender));
// waiting Ask — making report delivery observable end-to-end (review 01
// [Medium]). Mirrors the NotificationSubmit ack pattern above.
Receive<SiteHealthReport>(msg =>
{
if (_centralClient == null)
{
// No ClusterClient registered yet. A non-accepted ack makes the
// sender's counter-restore path treat this tick as a loss.
_log.Warning(
"Cannot forward SiteHealthReport #{0} — no central ClusterClient registered",
msg.SequenceNumber);
Sender.Tell(new SiteHealthReportAck(
msg.SiteId, msg.SequenceNumber, Accepted: false,
Error: "Central ClusterClient not registered"));
return;
}
_centralClient.Tell(
new ClusterClient.Send("/user/central-communication", msg), Sender);
});
} }
/// <inheritdoc /> /// <inheritdoc />
@@ -443,6 +334,12 @@ public class SiteCommunicationActor : ReceiveActor, IWithTimers
/// <inheritdoc /> /// <inheritdoc />
protected override void PreStart() protected override void PreStart()
{ {
// Finalize the transport now that the actor context (and _log) exist. The default Akka
// transport is given this actor's logging adapter so the "no ClusterClient registered"
// warnings are preserved exactly. PreStart always runs before any message, so the Receive
// closures above see a non-null _transport.
_transport = _injectedTransport ?? new AkkaCentralTransport(_log);
_log.Info("SiteCommunicationActor started for site {0}", _siteId); _log.Info("SiteCommunicationActor started for site {0}", _siteId);
// Schedule periodic heartbeat to central. Uses the application heartbeat // Schedule periodic heartbeat to central. Uses the application heartbeat
@@ -478,9 +375,6 @@ public class SiteCommunicationActor : ReceiveActor, IWithTimers
private void SendHeartbeatToCentral() private void SendHeartbeatToCentral()
{ {
if (_centralClient == null)
return;
var hostname = Environment.MachineName; var hostname = Environment.MachineName;
// Stamp HeartbeatMessage.IsActive with this node's // Stamp HeartbeatMessage.IsActive with this node's
@@ -512,8 +406,17 @@ public class SiteCommunicationActor : ReceiveActor, IWithTimers
IsActive: isActive, IsActive: isActive,
DateTimeOffset.UtcNow); DateTimeOffset.UtcNow);
_centralClient.Tell( // Fire-and-forget on both transports: a failure here must never fault the heartbeat timer
new ClusterClient.Send("/user/central-communication", heartbeat), Self); // path. Both real transports swallow their own errors; this catch is a belt-and-braces
// guarantee that no transport (including a future one) can turn a heartbeat into a fault.
try
{
_transport.SendHeartbeat(heartbeat, Self);
}
catch (Exception ex)
{
_log.Debug(ex, "Heartbeat send for site {0} failed; swallowed (heartbeats are fire-and-forget)", _siteId);
}
} }
/// <summary> /// <summary>
File diff suppressed because it is too large Load Diff
@@ -0,0 +1,704 @@
// <auto-generated>
// Generated by the protocol buffer compiler. DO NOT EDIT!
// source: Protos/central_control.proto
// </auto-generated>
#pragma warning disable 0414, 1591, 8981, 0612
#region Designer generated code
using grpc = global::Grpc.Core;
namespace ZB.MOM.WW.ScadaBridge.Communication.Grpc {
/// <summary>
/// Central-hosted control plane (Phase 1A of the ClusterClient→gRPC migration).
///
/// Direction: SITE is the client, CENTRAL is the server — the inverse of
/// SiteStreamService, where central dials the site. That asymmetry is deliberate
/// and mirrors the direction the Akka ClusterClient traffic flows today: these
/// seven calls are exactly the seven messages SiteCommunicationActor sends to
/// /user/central-communication.
///
/// Every call is gated by ControlPlaneAuthInterceptor: `authorization: Bearer &lt;psk>`
/// plus the `x-scadabridge-site` metadata header naming which site's preshared key
/// central must verify against.
/// </summary>
public static partial class CentralControlService
{
static readonly string __ServiceName = "scadabridge.centralcontrol.v1.CentralControlService";
[global::System.CodeDom.Compiler.GeneratedCode("grpc_csharp_plugin", null)]
static void __Helper_SerializeMessage(global::Google.Protobuf.IMessage message, grpc::SerializationContext context)
{
#if !GRPC_DISABLE_PROTOBUF_BUFFER_SERIALIZATION
if (message is global::Google.Protobuf.IBufferMessage)
{
context.SetPayloadLength(message.CalculateSize());
global::Google.Protobuf.MessageExtensions.WriteTo(message, context.GetBufferWriter());
context.Complete();
return;
}
#endif
context.Complete(global::Google.Protobuf.MessageExtensions.ToByteArray(message));
}
[global::System.CodeDom.Compiler.GeneratedCode("grpc_csharp_plugin", null)]
static class __Helper_MessageCache<T>
{
public static readonly bool IsBufferMessage = global::System.Reflection.IntrospectionExtensions.GetTypeInfo(typeof(global::Google.Protobuf.IBufferMessage)).IsAssignableFrom(typeof(T));
}
[global::System.CodeDom.Compiler.GeneratedCode("grpc_csharp_plugin", null)]
static T __Helper_DeserializeMessage<T>(grpc::DeserializationContext context, global::Google.Protobuf.MessageParser<T> parser) where T : global::Google.Protobuf.IMessage<T>
{
#if !GRPC_DISABLE_PROTOBUF_BUFFER_SERIALIZATION
if (__Helper_MessageCache<T>.IsBufferMessage)
{
return parser.ParseFrom(context.PayloadAsReadOnlySequence());
}
#endif
return parser.ParseFrom(context.PayloadAsNewBuffer());
}
[global::System.CodeDom.Compiler.GeneratedCode("grpc_csharp_plugin", null)]
static readonly grpc::Marshaller<global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.NotificationSubmitDto> __Marshaller_scadabridge_centralcontrol_v1_NotificationSubmitDto = grpc::Marshallers.Create(__Helper_SerializeMessage, context => __Helper_DeserializeMessage(context, global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.NotificationSubmitDto.Parser));
[global::System.CodeDom.Compiler.GeneratedCode("grpc_csharp_plugin", null)]
static readonly grpc::Marshaller<global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.NotificationSubmitAckDto> __Marshaller_scadabridge_centralcontrol_v1_NotificationSubmitAckDto = grpc::Marshallers.Create(__Helper_SerializeMessage, context => __Helper_DeserializeMessage(context, global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.NotificationSubmitAckDto.Parser));
[global::System.CodeDom.Compiler.GeneratedCode("grpc_csharp_plugin", null)]
static readonly grpc::Marshaller<global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.NotificationStatusQueryDto> __Marshaller_scadabridge_centralcontrol_v1_NotificationStatusQueryDto = grpc::Marshallers.Create(__Helper_SerializeMessage, context => __Helper_DeserializeMessage(context, global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.NotificationStatusQueryDto.Parser));
[global::System.CodeDom.Compiler.GeneratedCode("grpc_csharp_plugin", null)]
static readonly grpc::Marshaller<global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.NotificationStatusResponseDto> __Marshaller_scadabridge_centralcontrol_v1_NotificationStatusResponseDto = grpc::Marshallers.Create(__Helper_SerializeMessage, context => __Helper_DeserializeMessage(context, global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.NotificationStatusResponseDto.Parser));
[global::System.CodeDom.Compiler.GeneratedCode("grpc_csharp_plugin", null)]
static readonly grpc::Marshaller<global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.AuditEventBatch> __Marshaller_sitestream_AuditEventBatch = grpc::Marshallers.Create(__Helper_SerializeMessage, context => __Helper_DeserializeMessage(context, global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.AuditEventBatch.Parser));
[global::System.CodeDom.Compiler.GeneratedCode("grpc_csharp_plugin", null)]
static readonly grpc::Marshaller<global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.IngestAck> __Marshaller_sitestream_IngestAck = grpc::Marshallers.Create(__Helper_SerializeMessage, context => __Helper_DeserializeMessage(context, global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.IngestAck.Parser));
[global::System.CodeDom.Compiler.GeneratedCode("grpc_csharp_plugin", null)]
static readonly grpc::Marshaller<global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.CachedTelemetryBatch> __Marshaller_sitestream_CachedTelemetryBatch = grpc::Marshallers.Create(__Helper_SerializeMessage, context => __Helper_DeserializeMessage(context, global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.CachedTelemetryBatch.Parser));
[global::System.CodeDom.Compiler.GeneratedCode("grpc_csharp_plugin", null)]
static readonly grpc::Marshaller<global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.ReconcileSiteRequestDto> __Marshaller_scadabridge_centralcontrol_v1_ReconcileSiteRequestDto = grpc::Marshallers.Create(__Helper_SerializeMessage, context => __Helper_DeserializeMessage(context, global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.ReconcileSiteRequestDto.Parser));
[global::System.CodeDom.Compiler.GeneratedCode("grpc_csharp_plugin", null)]
static readonly grpc::Marshaller<global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.ReconcileSiteResponseDto> __Marshaller_scadabridge_centralcontrol_v1_ReconcileSiteResponseDto = grpc::Marshallers.Create(__Helper_SerializeMessage, context => __Helper_DeserializeMessage(context, global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.ReconcileSiteResponseDto.Parser));
[global::System.CodeDom.Compiler.GeneratedCode("grpc_csharp_plugin", null)]
static readonly grpc::Marshaller<global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.SiteHealthReportDto> __Marshaller_scadabridge_centralcontrol_v1_SiteHealthReportDto = grpc::Marshallers.Create(__Helper_SerializeMessage, context => __Helper_DeserializeMessage(context, global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.SiteHealthReportDto.Parser));
[global::System.CodeDom.Compiler.GeneratedCode("grpc_csharp_plugin", null)]
static readonly grpc::Marshaller<global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.SiteHealthReportAckDto> __Marshaller_scadabridge_centralcontrol_v1_SiteHealthReportAckDto = grpc::Marshallers.Create(__Helper_SerializeMessage, context => __Helper_DeserializeMessage(context, global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.SiteHealthReportAckDto.Parser));
[global::System.CodeDom.Compiler.GeneratedCode("grpc_csharp_plugin", null)]
static readonly grpc::Marshaller<global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.HeartbeatDto> __Marshaller_scadabridge_centralcontrol_v1_HeartbeatDto = grpc::Marshallers.Create(__Helper_SerializeMessage, context => __Helper_DeserializeMessage(context, global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.HeartbeatDto.Parser));
[global::System.CodeDom.Compiler.GeneratedCode("grpc_csharp_plugin", null)]
static readonly grpc::Marshaller<global::Google.Protobuf.WellKnownTypes.Empty> __Marshaller_google_protobuf_Empty = grpc::Marshallers.Create(__Helper_SerializeMessage, context => __Helper_DeserializeMessage(context, global::Google.Protobuf.WellKnownTypes.Empty.Parser));
[global::System.CodeDom.Compiler.GeneratedCode("grpc_csharp_plugin", null)]
static readonly grpc::Method<global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.NotificationSubmitDto, global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.NotificationSubmitAckDto> __Method_SubmitNotification = new grpc::Method<global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.NotificationSubmitDto, global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.NotificationSubmitAckDto>(
grpc::MethodType.Unary,
__ServiceName,
"SubmitNotification",
__Marshaller_scadabridge_centralcontrol_v1_NotificationSubmitDto,
__Marshaller_scadabridge_centralcontrol_v1_NotificationSubmitAckDto);
[global::System.CodeDom.Compiler.GeneratedCode("grpc_csharp_plugin", null)]
static readonly grpc::Method<global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.NotificationStatusQueryDto, global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.NotificationStatusResponseDto> __Method_QueryNotificationStatus = new grpc::Method<global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.NotificationStatusQueryDto, global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.NotificationStatusResponseDto>(
grpc::MethodType.Unary,
__ServiceName,
"QueryNotificationStatus",
__Marshaller_scadabridge_centralcontrol_v1_NotificationStatusQueryDto,
__Marshaller_scadabridge_centralcontrol_v1_NotificationStatusResponseDto);
[global::System.CodeDom.Compiler.GeneratedCode("grpc_csharp_plugin", null)]
static readonly grpc::Method<global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.AuditEventBatch, global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.IngestAck> __Method_IngestAuditEvents = new grpc::Method<global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.AuditEventBatch, global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.IngestAck>(
grpc::MethodType.Unary,
__ServiceName,
"IngestAuditEvents",
__Marshaller_sitestream_AuditEventBatch,
__Marshaller_sitestream_IngestAck);
[global::System.CodeDom.Compiler.GeneratedCode("grpc_csharp_plugin", null)]
static readonly grpc::Method<global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.CachedTelemetryBatch, global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.IngestAck> __Method_IngestCachedTelemetry = new grpc::Method<global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.CachedTelemetryBatch, global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.IngestAck>(
grpc::MethodType.Unary,
__ServiceName,
"IngestCachedTelemetry",
__Marshaller_sitestream_CachedTelemetryBatch,
__Marshaller_sitestream_IngestAck);
[global::System.CodeDom.Compiler.GeneratedCode("grpc_csharp_plugin", null)]
static readonly grpc::Method<global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.ReconcileSiteRequestDto, global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.ReconcileSiteResponseDto> __Method_ReconcileSite = new grpc::Method<global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.ReconcileSiteRequestDto, global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.ReconcileSiteResponseDto>(
grpc::MethodType.Unary,
__ServiceName,
"ReconcileSite",
__Marshaller_scadabridge_centralcontrol_v1_ReconcileSiteRequestDto,
__Marshaller_scadabridge_centralcontrol_v1_ReconcileSiteResponseDto);
[global::System.CodeDom.Compiler.GeneratedCode("grpc_csharp_plugin", null)]
static readonly grpc::Method<global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.SiteHealthReportDto, global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.SiteHealthReportAckDto> __Method_ReportSiteHealth = new grpc::Method<global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.SiteHealthReportDto, global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.SiteHealthReportAckDto>(
grpc::MethodType.Unary,
__ServiceName,
"ReportSiteHealth",
__Marshaller_scadabridge_centralcontrol_v1_SiteHealthReportDto,
__Marshaller_scadabridge_centralcontrol_v1_SiteHealthReportAckDto);
[global::System.CodeDom.Compiler.GeneratedCode("grpc_csharp_plugin", null)]
static readonly grpc::Method<global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.HeartbeatDto, global::Google.Protobuf.WellKnownTypes.Empty> __Method_Heartbeat = new grpc::Method<global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.HeartbeatDto, global::Google.Protobuf.WellKnownTypes.Empty>(
grpc::MethodType.Unary,
__ServiceName,
"Heartbeat",
__Marshaller_scadabridge_centralcontrol_v1_HeartbeatDto,
__Marshaller_google_protobuf_Empty);
/// <summary>Service descriptor</summary>
public static global::Google.Protobuf.Reflection.ServiceDescriptor Descriptor
{
get { return global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.CentralControlReflection.Descriptor.Services[0]; }
}
/// <summary>Base class for server-side implementations of CentralControlService</summary>
[grpc::BindServiceMethod(typeof(CentralControlService), "BindService")]
public abstract partial class CentralControlServiceBase
{
/// <summary>
/// Store-and-forward handoff of one notification for central delivery. The
/// ack is idempotent on notification_id — a duplicate submit after a lost ack
/// must not produce a second delivery.
/// </summary>
/// <param name="request">The request received from the client.</param>
/// <param name="context">The context of the server-side call handler being invoked.</param>
/// <returns>The response to send back to the client (wrapped by a task).</returns>
[global::System.CodeDom.Compiler.GeneratedCode("grpc_csharp_plugin", null)]
public virtual global::System.Threading.Tasks.Task<global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.NotificationSubmitAckDto> SubmitNotification(global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.NotificationSubmitDto request, grpc::ServerCallContext context)
{
throw new grpc::RpcException(new grpc::Status(grpc::StatusCode.Unimplemented, ""));
}
/// <summary>
/// Notify.Status(id) round-trip for a notification that has already left the
/// site buffer. `found = false` sends the caller back to the site-local buffer
/// to decide Forwarding vs Unknown.
/// </summary>
/// <param name="request">The request received from the client.</param>
/// <param name="context">The context of the server-side call handler being invoked.</param>
/// <returns>The response to send back to the client (wrapped by a task).</returns>
[global::System.CodeDom.Compiler.GeneratedCode("grpc_csharp_plugin", null)]
public virtual global::System.Threading.Tasks.Task<global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.NotificationStatusResponseDto> QueryNotificationStatus(global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.NotificationStatusQueryDto request, grpc::ServerCallContext context)
{
throw new grpc::RpcException(new grpc::Status(grpc::StatusCode.Unimplemented, ""));
}
/// <summary>
/// Audit Log (#23) push telemetry. Reuses the SiteStreamService messages: the
/// batch a site drains from its SQLite hot path is byte-identical whichever
/// transport carries it.
/// </summary>
/// <param name="request">The request received from the client.</param>
/// <param name="context">The context of the server-side call handler being invoked.</param>
/// <returns>The response to send back to the client (wrapped by a task).</returns>
[global::System.CodeDom.Compiler.GeneratedCode("grpc_csharp_plugin", null)]
public virtual global::System.Threading.Tasks.Task<global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.IngestAck> IngestAuditEvents(global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.AuditEventBatch request, grpc::ServerCallContext context)
{
throw new grpc::RpcException(new grpc::Status(grpc::StatusCode.Unimplemented, ""));
}
/// <summary>
/// Audit Log (#23) M3 combined cached-call telemetry (audit row + SiteCalls
/// operational upsert, written in one central transaction).
/// </summary>
/// <param name="request">The request received from the client.</param>
/// <param name="context">The context of the server-side call handler being invoked.</param>
/// <returns>The response to send back to the client (wrapped by a task).</returns>
[global::System.CodeDom.Compiler.GeneratedCode("grpc_csharp_plugin", null)]
public virtual global::System.Threading.Tasks.Task<global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.IngestAck> IngestCachedTelemetry(global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.CachedTelemetryBatch request, grpc::ServerCallContext context)
{
throw new grpc::RpcException(new grpc::Status(grpc::StatusCode.Unimplemented, ""));
}
/// <summary>
/// Node-startup self-heal: the node's local deployed inventory in, fetch
/// tokens for whatever it is missing or stale out.
/// </summary>
/// <param name="request">The request received from the client.</param>
/// <param name="context">The context of the server-side call handler being invoked.</param>
/// <returns>The response to send back to the client (wrapped by a task).</returns>
[global::System.CodeDom.Compiler.GeneratedCode("grpc_csharp_plugin", null)]
public virtual global::System.Threading.Tasks.Task<global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.ReconcileSiteResponseDto> ReconcileSite(global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.ReconcileSiteRequestDto request, grpc::ServerCallContext context)
{
throw new grpc::RpcException(new grpc::Status(grpc::StatusCode.Unimplemented, ""));
}
/// <summary>
/// Periodic site health report (30 s cadence). The ack makes delivery
/// observable end-to-end so the sender can restore its per-interval counters
/// when a report is lost.
/// </summary>
/// <param name="request">The request received from the client.</param>
/// <param name="context">The context of the server-side call handler being invoked.</param>
/// <returns>The response to send back to the client (wrapped by a task).</returns>
[global::System.CodeDom.Compiler.GeneratedCode("grpc_csharp_plugin", null)]
public virtual global::System.Threading.Tasks.Task<global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.SiteHealthReportAckDto> ReportSiteHealth(global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.SiteHealthReportDto request, grpc::ServerCallContext context)
{
throw new grpc::RpcException(new grpc::Status(grpc::StatusCode.Unimplemented, ""));
}
/// <summary>
/// Application heartbeat. Returns Empty because the message is
/// fire-and-forget: nothing on the site consumes a reply, and a failure here
/// must never surface as a fault on the heartbeat timer path.
/// </summary>
/// <param name="request">The request received from the client.</param>
/// <param name="context">The context of the server-side call handler being invoked.</param>
/// <returns>The response to send back to the client (wrapped by a task).</returns>
[global::System.CodeDom.Compiler.GeneratedCode("grpc_csharp_plugin", null)]
public virtual global::System.Threading.Tasks.Task<global::Google.Protobuf.WellKnownTypes.Empty> Heartbeat(global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.HeartbeatDto request, grpc::ServerCallContext context)
{
throw new grpc::RpcException(new grpc::Status(grpc::StatusCode.Unimplemented, ""));
}
}
/// <summary>Client for CentralControlService</summary>
public partial class CentralControlServiceClient : grpc::ClientBase<CentralControlServiceClient>
{
/// <summary>Creates a new client for CentralControlService</summary>
/// <param name="channel">The channel to use to make remote calls.</param>
[global::System.CodeDom.Compiler.GeneratedCode("grpc_csharp_plugin", null)]
public CentralControlServiceClient(grpc::ChannelBase channel) : base(channel)
{
}
/// <summary>Creates a new client for CentralControlService that uses a custom <c>CallInvoker</c>.</summary>
/// <param name="callInvoker">The callInvoker to use to make remote calls.</param>
[global::System.CodeDom.Compiler.GeneratedCode("grpc_csharp_plugin", null)]
public CentralControlServiceClient(grpc::CallInvoker callInvoker) : base(callInvoker)
{
}
/// <summary>Protected parameterless constructor to allow creation of test doubles.</summary>
[global::System.CodeDom.Compiler.GeneratedCode("grpc_csharp_plugin", null)]
protected CentralControlServiceClient() : base()
{
}
/// <summary>Protected constructor to allow creation of configured clients.</summary>
/// <param name="configuration">The client configuration.</param>
[global::System.CodeDom.Compiler.GeneratedCode("grpc_csharp_plugin", null)]
protected CentralControlServiceClient(ClientBaseConfiguration configuration) : base(configuration)
{
}
/// <summary>
/// Store-and-forward handoff of one notification for central delivery. The
/// ack is idempotent on notification_id — a duplicate submit after a lost ack
/// must not produce a second delivery.
/// </summary>
/// <param name="request">The request to send to the server.</param>
/// <param name="headers">The initial metadata to send with the call. This parameter is optional.</param>
/// <param name="deadline">An optional deadline for the call. The call will be cancelled if deadline is hit.</param>
/// <param name="cancellationToken">An optional token for canceling the call.</param>
/// <returns>The response received from the server.</returns>
[global::System.CodeDom.Compiler.GeneratedCode("grpc_csharp_plugin", null)]
public virtual global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.NotificationSubmitAckDto SubmitNotification(global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.NotificationSubmitDto request, grpc::Metadata headers = null, global::System.DateTime? deadline = null, global::System.Threading.CancellationToken cancellationToken = default(global::System.Threading.CancellationToken))
{
return SubmitNotification(request, new grpc::CallOptions(headers, deadline, cancellationToken));
}
/// <summary>
/// Store-and-forward handoff of one notification for central delivery. The
/// ack is idempotent on notification_id — a duplicate submit after a lost ack
/// must not produce a second delivery.
/// </summary>
/// <param name="request">The request to send to the server.</param>
/// <param name="options">The options for the call.</param>
/// <returns>The response received from the server.</returns>
[global::System.CodeDom.Compiler.GeneratedCode("grpc_csharp_plugin", null)]
public virtual global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.NotificationSubmitAckDto SubmitNotification(global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.NotificationSubmitDto request, grpc::CallOptions options)
{
return CallInvoker.BlockingUnaryCall(__Method_SubmitNotification, null, options, request);
}
/// <summary>
/// Store-and-forward handoff of one notification for central delivery. The
/// ack is idempotent on notification_id — a duplicate submit after a lost ack
/// must not produce a second delivery.
/// </summary>
/// <param name="request">The request to send to the server.</param>
/// <param name="headers">The initial metadata to send with the call. This parameter is optional.</param>
/// <param name="deadline">An optional deadline for the call. The call will be cancelled if deadline is hit.</param>
/// <param name="cancellationToken">An optional token for canceling the call.</param>
/// <returns>The call object.</returns>
[global::System.CodeDom.Compiler.GeneratedCode("grpc_csharp_plugin", null)]
public virtual grpc::AsyncUnaryCall<global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.NotificationSubmitAckDto> SubmitNotificationAsync(global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.NotificationSubmitDto request, grpc::Metadata headers = null, global::System.DateTime? deadline = null, global::System.Threading.CancellationToken cancellationToken = default(global::System.Threading.CancellationToken))
{
return SubmitNotificationAsync(request, new grpc::CallOptions(headers, deadline, cancellationToken));
}
/// <summary>
/// Store-and-forward handoff of one notification for central delivery. The
/// ack is idempotent on notification_id — a duplicate submit after a lost ack
/// must not produce a second delivery.
/// </summary>
/// <param name="request">The request to send to the server.</param>
/// <param name="options">The options for the call.</param>
/// <returns>The call object.</returns>
[global::System.CodeDom.Compiler.GeneratedCode("grpc_csharp_plugin", null)]
public virtual grpc::AsyncUnaryCall<global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.NotificationSubmitAckDto> SubmitNotificationAsync(global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.NotificationSubmitDto request, grpc::CallOptions options)
{
return CallInvoker.AsyncUnaryCall(__Method_SubmitNotification, null, options, request);
}
/// <summary>
/// Notify.Status(id) round-trip for a notification that has already left the
/// site buffer. `found = false` sends the caller back to the site-local buffer
/// to decide Forwarding vs Unknown.
/// </summary>
/// <param name="request">The request to send to the server.</param>
/// <param name="headers">The initial metadata to send with the call. This parameter is optional.</param>
/// <param name="deadline">An optional deadline for the call. The call will be cancelled if deadline is hit.</param>
/// <param name="cancellationToken">An optional token for canceling the call.</param>
/// <returns>The response received from the server.</returns>
[global::System.CodeDom.Compiler.GeneratedCode("grpc_csharp_plugin", null)]
public virtual global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.NotificationStatusResponseDto QueryNotificationStatus(global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.NotificationStatusQueryDto request, grpc::Metadata headers = null, global::System.DateTime? deadline = null, global::System.Threading.CancellationToken cancellationToken = default(global::System.Threading.CancellationToken))
{
return QueryNotificationStatus(request, new grpc::CallOptions(headers, deadline, cancellationToken));
}
/// <summary>
/// Notify.Status(id) round-trip for a notification that has already left the
/// site buffer. `found = false` sends the caller back to the site-local buffer
/// to decide Forwarding vs Unknown.
/// </summary>
/// <param name="request">The request to send to the server.</param>
/// <param name="options">The options for the call.</param>
/// <returns>The response received from the server.</returns>
[global::System.CodeDom.Compiler.GeneratedCode("grpc_csharp_plugin", null)]
public virtual global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.NotificationStatusResponseDto QueryNotificationStatus(global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.NotificationStatusQueryDto request, grpc::CallOptions options)
{
return CallInvoker.BlockingUnaryCall(__Method_QueryNotificationStatus, null, options, request);
}
/// <summary>
/// Notify.Status(id) round-trip for a notification that has already left the
/// site buffer. `found = false` sends the caller back to the site-local buffer
/// to decide Forwarding vs Unknown.
/// </summary>
/// <param name="request">The request to send to the server.</param>
/// <param name="headers">The initial metadata to send with the call. This parameter is optional.</param>
/// <param name="deadline">An optional deadline for the call. The call will be cancelled if deadline is hit.</param>
/// <param name="cancellationToken">An optional token for canceling the call.</param>
/// <returns>The call object.</returns>
[global::System.CodeDom.Compiler.GeneratedCode("grpc_csharp_plugin", null)]
public virtual grpc::AsyncUnaryCall<global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.NotificationStatusResponseDto> QueryNotificationStatusAsync(global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.NotificationStatusQueryDto request, grpc::Metadata headers = null, global::System.DateTime? deadline = null, global::System.Threading.CancellationToken cancellationToken = default(global::System.Threading.CancellationToken))
{
return QueryNotificationStatusAsync(request, new grpc::CallOptions(headers, deadline, cancellationToken));
}
/// <summary>
/// Notify.Status(id) round-trip for a notification that has already left the
/// site buffer. `found = false` sends the caller back to the site-local buffer
/// to decide Forwarding vs Unknown.
/// </summary>
/// <param name="request">The request to send to the server.</param>
/// <param name="options">The options for the call.</param>
/// <returns>The call object.</returns>
[global::System.CodeDom.Compiler.GeneratedCode("grpc_csharp_plugin", null)]
public virtual grpc::AsyncUnaryCall<global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.NotificationStatusResponseDto> QueryNotificationStatusAsync(global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.NotificationStatusQueryDto request, grpc::CallOptions options)
{
return CallInvoker.AsyncUnaryCall(__Method_QueryNotificationStatus, null, options, request);
}
/// <summary>
/// Audit Log (#23) push telemetry. Reuses the SiteStreamService messages: the
/// batch a site drains from its SQLite hot path is byte-identical whichever
/// transport carries it.
/// </summary>
/// <param name="request">The request to send to the server.</param>
/// <param name="headers">The initial metadata to send with the call. This parameter is optional.</param>
/// <param name="deadline">An optional deadline for the call. The call will be cancelled if deadline is hit.</param>
/// <param name="cancellationToken">An optional token for canceling the call.</param>
/// <returns>The response received from the server.</returns>
[global::System.CodeDom.Compiler.GeneratedCode("grpc_csharp_plugin", null)]
public virtual global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.IngestAck IngestAuditEvents(global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.AuditEventBatch request, grpc::Metadata headers = null, global::System.DateTime? deadline = null, global::System.Threading.CancellationToken cancellationToken = default(global::System.Threading.CancellationToken))
{
return IngestAuditEvents(request, new grpc::CallOptions(headers, deadline, cancellationToken));
}
/// <summary>
/// Audit Log (#23) push telemetry. Reuses the SiteStreamService messages: the
/// batch a site drains from its SQLite hot path is byte-identical whichever
/// transport carries it.
/// </summary>
/// <param name="request">The request to send to the server.</param>
/// <param name="options">The options for the call.</param>
/// <returns>The response received from the server.</returns>
[global::System.CodeDom.Compiler.GeneratedCode("grpc_csharp_plugin", null)]
public virtual global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.IngestAck IngestAuditEvents(global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.AuditEventBatch request, grpc::CallOptions options)
{
return CallInvoker.BlockingUnaryCall(__Method_IngestAuditEvents, null, options, request);
}
/// <summary>
/// Audit Log (#23) push telemetry. Reuses the SiteStreamService messages: the
/// batch a site drains from its SQLite hot path is byte-identical whichever
/// transport carries it.
/// </summary>
/// <param name="request">The request to send to the server.</param>
/// <param name="headers">The initial metadata to send with the call. This parameter is optional.</param>
/// <param name="deadline">An optional deadline for the call. The call will be cancelled if deadline is hit.</param>
/// <param name="cancellationToken">An optional token for canceling the call.</param>
/// <returns>The call object.</returns>
[global::System.CodeDom.Compiler.GeneratedCode("grpc_csharp_plugin", null)]
public virtual grpc::AsyncUnaryCall<global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.IngestAck> IngestAuditEventsAsync(global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.AuditEventBatch request, grpc::Metadata headers = null, global::System.DateTime? deadline = null, global::System.Threading.CancellationToken cancellationToken = default(global::System.Threading.CancellationToken))
{
return IngestAuditEventsAsync(request, new grpc::CallOptions(headers, deadline, cancellationToken));
}
/// <summary>
/// Audit Log (#23) push telemetry. Reuses the SiteStreamService messages: the
/// batch a site drains from its SQLite hot path is byte-identical whichever
/// transport carries it.
/// </summary>
/// <param name="request">The request to send to the server.</param>
/// <param name="options">The options for the call.</param>
/// <returns>The call object.</returns>
[global::System.CodeDom.Compiler.GeneratedCode("grpc_csharp_plugin", null)]
public virtual grpc::AsyncUnaryCall<global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.IngestAck> IngestAuditEventsAsync(global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.AuditEventBatch request, grpc::CallOptions options)
{
return CallInvoker.AsyncUnaryCall(__Method_IngestAuditEvents, null, options, request);
}
/// <summary>
/// Audit Log (#23) M3 combined cached-call telemetry (audit row + SiteCalls
/// operational upsert, written in one central transaction).
/// </summary>
/// <param name="request">The request to send to the server.</param>
/// <param name="headers">The initial metadata to send with the call. This parameter is optional.</param>
/// <param name="deadline">An optional deadline for the call. The call will be cancelled if deadline is hit.</param>
/// <param name="cancellationToken">An optional token for canceling the call.</param>
/// <returns>The response received from the server.</returns>
[global::System.CodeDom.Compiler.GeneratedCode("grpc_csharp_plugin", null)]
public virtual global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.IngestAck IngestCachedTelemetry(global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.CachedTelemetryBatch request, grpc::Metadata headers = null, global::System.DateTime? deadline = null, global::System.Threading.CancellationToken cancellationToken = default(global::System.Threading.CancellationToken))
{
return IngestCachedTelemetry(request, new grpc::CallOptions(headers, deadline, cancellationToken));
}
/// <summary>
/// Audit Log (#23) M3 combined cached-call telemetry (audit row + SiteCalls
/// operational upsert, written in one central transaction).
/// </summary>
/// <param name="request">The request to send to the server.</param>
/// <param name="options">The options for the call.</param>
/// <returns>The response received from the server.</returns>
[global::System.CodeDom.Compiler.GeneratedCode("grpc_csharp_plugin", null)]
public virtual global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.IngestAck IngestCachedTelemetry(global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.CachedTelemetryBatch request, grpc::CallOptions options)
{
return CallInvoker.BlockingUnaryCall(__Method_IngestCachedTelemetry, null, options, request);
}
/// <summary>
/// Audit Log (#23) M3 combined cached-call telemetry (audit row + SiteCalls
/// operational upsert, written in one central transaction).
/// </summary>
/// <param name="request">The request to send to the server.</param>
/// <param name="headers">The initial metadata to send with the call. This parameter is optional.</param>
/// <param name="deadline">An optional deadline for the call. The call will be cancelled if deadline is hit.</param>
/// <param name="cancellationToken">An optional token for canceling the call.</param>
/// <returns>The call object.</returns>
[global::System.CodeDom.Compiler.GeneratedCode("grpc_csharp_plugin", null)]
public virtual grpc::AsyncUnaryCall<global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.IngestAck> IngestCachedTelemetryAsync(global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.CachedTelemetryBatch request, grpc::Metadata headers = null, global::System.DateTime? deadline = null, global::System.Threading.CancellationToken cancellationToken = default(global::System.Threading.CancellationToken))
{
return IngestCachedTelemetryAsync(request, new grpc::CallOptions(headers, deadline, cancellationToken));
}
/// <summary>
/// Audit Log (#23) M3 combined cached-call telemetry (audit row + SiteCalls
/// operational upsert, written in one central transaction).
/// </summary>
/// <param name="request">The request to send to the server.</param>
/// <param name="options">The options for the call.</param>
/// <returns>The call object.</returns>
[global::System.CodeDom.Compiler.GeneratedCode("grpc_csharp_plugin", null)]
public virtual grpc::AsyncUnaryCall<global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.IngestAck> IngestCachedTelemetryAsync(global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.CachedTelemetryBatch request, grpc::CallOptions options)
{
return CallInvoker.AsyncUnaryCall(__Method_IngestCachedTelemetry, null, options, request);
}
/// <summary>
/// Node-startup self-heal: the node's local deployed inventory in, fetch
/// tokens for whatever it is missing or stale out.
/// </summary>
/// <param name="request">The request to send to the server.</param>
/// <param name="headers">The initial metadata to send with the call. This parameter is optional.</param>
/// <param name="deadline">An optional deadline for the call. The call will be cancelled if deadline is hit.</param>
/// <param name="cancellationToken">An optional token for canceling the call.</param>
/// <returns>The response received from the server.</returns>
[global::System.CodeDom.Compiler.GeneratedCode("grpc_csharp_plugin", null)]
public virtual global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.ReconcileSiteResponseDto ReconcileSite(global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.ReconcileSiteRequestDto request, grpc::Metadata headers = null, global::System.DateTime? deadline = null, global::System.Threading.CancellationToken cancellationToken = default(global::System.Threading.CancellationToken))
{
return ReconcileSite(request, new grpc::CallOptions(headers, deadline, cancellationToken));
}
/// <summary>
/// Node-startup self-heal: the node's local deployed inventory in, fetch
/// tokens for whatever it is missing or stale out.
/// </summary>
/// <param name="request">The request to send to the server.</param>
/// <param name="options">The options for the call.</param>
/// <returns>The response received from the server.</returns>
[global::System.CodeDom.Compiler.GeneratedCode("grpc_csharp_plugin", null)]
public virtual global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.ReconcileSiteResponseDto ReconcileSite(global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.ReconcileSiteRequestDto request, grpc::CallOptions options)
{
return CallInvoker.BlockingUnaryCall(__Method_ReconcileSite, null, options, request);
}
/// <summary>
/// Node-startup self-heal: the node's local deployed inventory in, fetch
/// tokens for whatever it is missing or stale out.
/// </summary>
/// <param name="request">The request to send to the server.</param>
/// <param name="headers">The initial metadata to send with the call. This parameter is optional.</param>
/// <param name="deadline">An optional deadline for the call. The call will be cancelled if deadline is hit.</param>
/// <param name="cancellationToken">An optional token for canceling the call.</param>
/// <returns>The call object.</returns>
[global::System.CodeDom.Compiler.GeneratedCode("grpc_csharp_plugin", null)]
public virtual grpc::AsyncUnaryCall<global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.ReconcileSiteResponseDto> ReconcileSiteAsync(global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.ReconcileSiteRequestDto request, grpc::Metadata headers = null, global::System.DateTime? deadline = null, global::System.Threading.CancellationToken cancellationToken = default(global::System.Threading.CancellationToken))
{
return ReconcileSiteAsync(request, new grpc::CallOptions(headers, deadline, cancellationToken));
}
/// <summary>
/// Node-startup self-heal: the node's local deployed inventory in, fetch
/// tokens for whatever it is missing or stale out.
/// </summary>
/// <param name="request">The request to send to the server.</param>
/// <param name="options">The options for the call.</param>
/// <returns>The call object.</returns>
[global::System.CodeDom.Compiler.GeneratedCode("grpc_csharp_plugin", null)]
public virtual grpc::AsyncUnaryCall<global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.ReconcileSiteResponseDto> ReconcileSiteAsync(global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.ReconcileSiteRequestDto request, grpc::CallOptions options)
{
return CallInvoker.AsyncUnaryCall(__Method_ReconcileSite, null, options, request);
}
/// <summary>
/// Periodic site health report (30 s cadence). The ack makes delivery
/// observable end-to-end so the sender can restore its per-interval counters
/// when a report is lost.
/// </summary>
/// <param name="request">The request to send to the server.</param>
/// <param name="headers">The initial metadata to send with the call. This parameter is optional.</param>
/// <param name="deadline">An optional deadline for the call. The call will be cancelled if deadline is hit.</param>
/// <param name="cancellationToken">An optional token for canceling the call.</param>
/// <returns>The response received from the server.</returns>
[global::System.CodeDom.Compiler.GeneratedCode("grpc_csharp_plugin", null)]
public virtual global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.SiteHealthReportAckDto ReportSiteHealth(global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.SiteHealthReportDto request, grpc::Metadata headers = null, global::System.DateTime? deadline = null, global::System.Threading.CancellationToken cancellationToken = default(global::System.Threading.CancellationToken))
{
return ReportSiteHealth(request, new grpc::CallOptions(headers, deadline, cancellationToken));
}
/// <summary>
/// Periodic site health report (30 s cadence). The ack makes delivery
/// observable end-to-end so the sender can restore its per-interval counters
/// when a report is lost.
/// </summary>
/// <param name="request">The request to send to the server.</param>
/// <param name="options">The options for the call.</param>
/// <returns>The response received from the server.</returns>
[global::System.CodeDom.Compiler.GeneratedCode("grpc_csharp_plugin", null)]
public virtual global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.SiteHealthReportAckDto ReportSiteHealth(global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.SiteHealthReportDto request, grpc::CallOptions options)
{
return CallInvoker.BlockingUnaryCall(__Method_ReportSiteHealth, null, options, request);
}
/// <summary>
/// Periodic site health report (30 s cadence). The ack makes delivery
/// observable end-to-end so the sender can restore its per-interval counters
/// when a report is lost.
/// </summary>
/// <param name="request">The request to send to the server.</param>
/// <param name="headers">The initial metadata to send with the call. This parameter is optional.</param>
/// <param name="deadline">An optional deadline for the call. The call will be cancelled if deadline is hit.</param>
/// <param name="cancellationToken">An optional token for canceling the call.</param>
/// <returns>The call object.</returns>
[global::System.CodeDom.Compiler.GeneratedCode("grpc_csharp_plugin", null)]
public virtual grpc::AsyncUnaryCall<global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.SiteHealthReportAckDto> ReportSiteHealthAsync(global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.SiteHealthReportDto request, grpc::Metadata headers = null, global::System.DateTime? deadline = null, global::System.Threading.CancellationToken cancellationToken = default(global::System.Threading.CancellationToken))
{
return ReportSiteHealthAsync(request, new grpc::CallOptions(headers, deadline, cancellationToken));
}
/// <summary>
/// Periodic site health report (30 s cadence). The ack makes delivery
/// observable end-to-end so the sender can restore its per-interval counters
/// when a report is lost.
/// </summary>
/// <param name="request">The request to send to the server.</param>
/// <param name="options">The options for the call.</param>
/// <returns>The call object.</returns>
[global::System.CodeDom.Compiler.GeneratedCode("grpc_csharp_plugin", null)]
public virtual grpc::AsyncUnaryCall<global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.SiteHealthReportAckDto> ReportSiteHealthAsync(global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.SiteHealthReportDto request, grpc::CallOptions options)
{
return CallInvoker.AsyncUnaryCall(__Method_ReportSiteHealth, null, options, request);
}
/// <summary>
/// Application heartbeat. Returns Empty because the message is
/// fire-and-forget: nothing on the site consumes a reply, and a failure here
/// must never surface as a fault on the heartbeat timer path.
/// </summary>
/// <param name="request">The request to send to the server.</param>
/// <param name="headers">The initial metadata to send with the call. This parameter is optional.</param>
/// <param name="deadline">An optional deadline for the call. The call will be cancelled if deadline is hit.</param>
/// <param name="cancellationToken">An optional token for canceling the call.</param>
/// <returns>The response received from the server.</returns>
[global::System.CodeDom.Compiler.GeneratedCode("grpc_csharp_plugin", null)]
public virtual global::Google.Protobuf.WellKnownTypes.Empty Heartbeat(global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.HeartbeatDto request, grpc::Metadata headers = null, global::System.DateTime? deadline = null, global::System.Threading.CancellationToken cancellationToken = default(global::System.Threading.CancellationToken))
{
return Heartbeat(request, new grpc::CallOptions(headers, deadline, cancellationToken));
}
/// <summary>
/// Application heartbeat. Returns Empty because the message is
/// fire-and-forget: nothing on the site consumes a reply, and a failure here
/// must never surface as a fault on the heartbeat timer path.
/// </summary>
/// <param name="request">The request to send to the server.</param>
/// <param name="options">The options for the call.</param>
/// <returns>The response received from the server.</returns>
[global::System.CodeDom.Compiler.GeneratedCode("grpc_csharp_plugin", null)]
public virtual global::Google.Protobuf.WellKnownTypes.Empty Heartbeat(global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.HeartbeatDto request, grpc::CallOptions options)
{
return CallInvoker.BlockingUnaryCall(__Method_Heartbeat, null, options, request);
}
/// <summary>
/// Application heartbeat. Returns Empty because the message is
/// fire-and-forget: nothing on the site consumes a reply, and a failure here
/// must never surface as a fault on the heartbeat timer path.
/// </summary>
/// <param name="request">The request to send to the server.</param>
/// <param name="headers">The initial metadata to send with the call. This parameter is optional.</param>
/// <param name="deadline">An optional deadline for the call. The call will be cancelled if deadline is hit.</param>
/// <param name="cancellationToken">An optional token for canceling the call.</param>
/// <returns>The call object.</returns>
[global::System.CodeDom.Compiler.GeneratedCode("grpc_csharp_plugin", null)]
public virtual grpc::AsyncUnaryCall<global::Google.Protobuf.WellKnownTypes.Empty> HeartbeatAsync(global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.HeartbeatDto request, grpc::Metadata headers = null, global::System.DateTime? deadline = null, global::System.Threading.CancellationToken cancellationToken = default(global::System.Threading.CancellationToken))
{
return HeartbeatAsync(request, new grpc::CallOptions(headers, deadline, cancellationToken));
}
/// <summary>
/// Application heartbeat. Returns Empty because the message is
/// fire-and-forget: nothing on the site consumes a reply, and a failure here
/// must never surface as a fault on the heartbeat timer path.
/// </summary>
/// <param name="request">The request to send to the server.</param>
/// <param name="options">The options for the call.</param>
/// <returns>The call object.</returns>
[global::System.CodeDom.Compiler.GeneratedCode("grpc_csharp_plugin", null)]
public virtual grpc::AsyncUnaryCall<global::Google.Protobuf.WellKnownTypes.Empty> HeartbeatAsync(global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.HeartbeatDto request, grpc::CallOptions options)
{
return CallInvoker.AsyncUnaryCall(__Method_Heartbeat, null, options, request);
}
/// <summary>Creates a new instance of client from given <c>ClientBaseConfiguration</c>.</summary>
[global::System.CodeDom.Compiler.GeneratedCode("grpc_csharp_plugin", null)]
protected override CentralControlServiceClient NewInstance(ClientBaseConfiguration configuration)
{
return new CentralControlServiceClient(configuration);
}
}
/// <summary>Creates service definition that can be registered with a server</summary>
/// <param name="serviceImpl">An object implementing the server-side handling logic.</param>
[global::System.CodeDom.Compiler.GeneratedCode("grpc_csharp_plugin", null)]
public static grpc::ServerServiceDefinition BindService(CentralControlServiceBase serviceImpl)
{
return grpc::ServerServiceDefinition.CreateBuilder()
.AddMethod(__Method_SubmitNotification, serviceImpl.SubmitNotification)
.AddMethod(__Method_QueryNotificationStatus, serviceImpl.QueryNotificationStatus)
.AddMethod(__Method_IngestAuditEvents, serviceImpl.IngestAuditEvents)
.AddMethod(__Method_IngestCachedTelemetry, serviceImpl.IngestCachedTelemetry)
.AddMethod(__Method_ReconcileSite, serviceImpl.ReconcileSite)
.AddMethod(__Method_ReportSiteHealth, serviceImpl.ReportSiteHealth)
.AddMethod(__Method_Heartbeat, serviceImpl.Heartbeat).Build();
}
/// <summary>Register service method with a service binder with or without implementation. Useful when customizing the service binding logic.
/// Note: this method is part of an experimental API that can change or be removed without any prior notice.</summary>
/// <param name="serviceBinder">Service methods will be bound by calling <c>AddMethod</c> on this object.</param>
/// <param name="serviceImpl">An object implementing the server-side handling logic.</param>
[global::System.CodeDom.Compiler.GeneratedCode("grpc_csharp_plugin", null)]
public static void BindService(grpc::ServiceBinderBase serviceBinder, CentralControlServiceBase serviceImpl)
{
serviceBinder.AddMethod(__Method_SubmitNotification, serviceImpl == null ? null : new grpc::UnaryServerMethod<global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.NotificationSubmitDto, global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.NotificationSubmitAckDto>(serviceImpl.SubmitNotification));
serviceBinder.AddMethod(__Method_QueryNotificationStatus, serviceImpl == null ? null : new grpc::UnaryServerMethod<global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.NotificationStatusQueryDto, global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.NotificationStatusResponseDto>(serviceImpl.QueryNotificationStatus));
serviceBinder.AddMethod(__Method_IngestAuditEvents, serviceImpl == null ? null : new grpc::UnaryServerMethod<global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.AuditEventBatch, global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.IngestAck>(serviceImpl.IngestAuditEvents));
serviceBinder.AddMethod(__Method_IngestCachedTelemetry, serviceImpl == null ? null : new grpc::UnaryServerMethod<global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.CachedTelemetryBatch, global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.IngestAck>(serviceImpl.IngestCachedTelemetry));
serviceBinder.AddMethod(__Method_ReconcileSite, serviceImpl == null ? null : new grpc::UnaryServerMethod<global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.ReconcileSiteRequestDto, global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.ReconcileSiteResponseDto>(serviceImpl.ReconcileSite));
serviceBinder.AddMethod(__Method_ReportSiteHealth, serviceImpl == null ? null : new grpc::UnaryServerMethod<global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.SiteHealthReportDto, global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.SiteHealthReportAckDto>(serviceImpl.ReportSiteHealth));
serviceBinder.AddMethod(__Method_Heartbeat, serviceImpl == null ? null : new grpc::UnaryServerMethod<global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.HeartbeatDto, global::Google.Protobuf.WellKnownTypes.Empty>(serviceImpl.Heartbeat));
}
}
}
#endregion
@@ -1,11 +1,44 @@
namespace ZB.MOM.WW.ScadaBridge.Communication; namespace ZB.MOM.WW.ScadaBridge.Communication;
/// <summary>
/// Which transport carries the seven site→central control messages. Selected per node by
/// <c>ScadaBridge:Communication:CentralTransport</c>; the migration ships with
/// <see cref="Akka"/> as the default so nothing flips until a node opts in.
/// </summary>
public enum CentralTransportMode
{
/// <summary>Akka <c>ClusterClient</c> — the transport in production today, and the default.</summary>
Akka = 0,
/// <summary>gRPC dial of the central <c>CentralControlService</c> (Phase 1A migration target).</summary>
Grpc = 1,
}
/// <summary> /// <summary>
/// Configuration options for central-site communication, including per-pattern /// Configuration options for central-site communication, including per-pattern
/// timeouts and transport heartbeat settings. /// timeouts and transport heartbeat settings.
/// </summary> /// </summary>
public class CommunicationOptions public class CommunicationOptions
{ {
/// <summary>
/// Which transport carries the site→central control messages. Default <see cref="CentralTransportMode.Akka"/>
/// (ClusterClient) — coexistence rule: a node flips to gRPC only by setting this to <c>Grpc</c>,
/// and rollback is flipping it back. Selecting <c>Grpc</c> requires <see cref="CentralGrpcEndpoints"/>.
/// </summary>
public CentralTransportMode CentralTransport { get; set; } = CentralTransportMode.Akka;
/// <summary>
/// Central control-plane gRPC endpoints (preferred first), e.g.
/// <c>["http://scadabridge-central-a:8083", "http://scadabridge-central-b:8083"]</c>. Dialled by
/// <see cref="Grpc.CentralChannelProvider"/> with sticky failover/failback. Required when
/// <see cref="CentralTransport"/> is <see cref="CentralTransportMode.Grpc"/>, ignored otherwise.
/// </summary>
/// <remarks>
/// Sites reach central by container/host name, NOT via Traefik (which is HTTP/1 only; gRPC is
/// h2c on the central node's dedicated <c>CentralGrpcPort</c>).
/// </remarks>
public List<string> CentralGrpcEndpoints { get; set; } = new();
/// <summary>Timeout for deployment commands (typically longest due to apply logic).</summary> /// <summary>Timeout for deployment commands (typically longest due to apply logic).</summary>
public TimeSpan DeploymentTimeout { get; set; } = TimeSpan.FromMinutes(2); public TimeSpan DeploymentTimeout { get; set; } = TimeSpan.FromMinutes(2);
@@ -41,6 +74,54 @@ public class CommunicationOptions
/// </summary> /// </summary>
public List<string> CentralContactPoints { get; set; } = new(); public List<string> CentralContactPoints { get; set; } = new();
/// <summary>
/// Preshared key authenticating this node's gRPC control plane — the site↔central
/// boundary. On a site node this is the key its inbound gate
/// (<c>ControlPlaneAuthInterceptor</c>) expects on every <c>SiteStreamService</c> call, and
/// which central must present; central resolves the matching value per site from its own
/// secret store under the name <c>SB-GRPC-PSK-{siteId}</c>.
/// </summary>
/// <remarks>
/// <para>
/// In production this is supplied as <c>${secret:SB-GRPC-PSK-&lt;siteId&gt;}</c> and expanded
/// out of the secrets store before the host is built, so the plaintext never sits in
/// appsettings. Development rigs set a literal, mirroring the LocalDb replication key.
/// </para>
/// <para>
/// <b>Empty means closed, not open.</b> With no key set the interceptor rejects every gated
/// call. This is not optional configuration: a node that ships without a key serves no
/// streams and no audit pulls.
/// </para>
/// <para>
/// Distinct from <c>LocalDb:Replication:ApiKey</c>, which authenticates the pair partner for
/// database replication over the same listener. The two are never shared.
/// </para>
/// </remarks>
public string GrpcPsk { get; set; } = "";
/// <summary>
/// Central-side per-site gRPC preshared keys, keyed by site identifier — the mirror image
/// of <see cref="GrpcPsk"/>, which is the single key a site node expects on its own inbound
/// gate. An entry here takes precedence over the secret store for that site.
/// </summary>
/// <remarks>
/// <para>
/// <b>Why both a config map and a secret store.</b> The store is the primary source and the
/// only one that works for the real case: sites are added at runtime from the Central UI, so
/// their keys cannot be enumerated in configuration at boot, and <c>SitePskProvider</c>
/// resolves <c>SB-GRPC-PSK-{siteId}</c> on demand. This map covers the cases the store
/// cannot or should not: a development rig that runs with no master key and injects every
/// credential as an environment override, and an operator pinning one site's key without
/// touching the store. Values may themselves be <c>${secret:…}</c> references, since a map
/// declared in configuration IS enumerable at boot.
/// </para>
/// <para>
/// Absence is not a fallback to "unauthenticated" in either source — a site with no key in
/// the map and none in the store cannot be dialed at all.
/// </para>
/// </remarks>
public Dictionary<string, string> SitePsks { get; set; } = new();
/// <summary>gRPC keepalive ping interval for streaming connections.</summary> /// <summary>gRPC keepalive ping interval for streaming connections.</summary>
public TimeSpan GrpcKeepAlivePingDelay { get; set; } = TimeSpan.FromSeconds(15); public TimeSpan GrpcKeepAlivePingDelay { get; set; } = TimeSpan.FromSeconds(15);
@@ -66,6 +66,19 @@ public sealed class CommunicationOptionsValidator : OptionsValidatorBase<Communi
builder.RequireThat(options.GrpcMaxConcurrentStreams > 0, builder.RequireThat(options.GrpcMaxConcurrentStreams > 0,
$"ScadaBridge:Communication:GrpcMaxConcurrentStreams must be positive (was {options.GrpcMaxConcurrentStreams})."); $"ScadaBridge:Communication:GrpcMaxConcurrentStreams must be positive (was {options.GrpcMaxConcurrentStreams}).");
// The gRPC site→central transport needs at least one central endpoint to dial. Only
// enforced when that transport is selected — the default Akka path ignores the list, so a
// node on ClusterClient must not be forced to declare gRPC endpoints it never uses.
if (options.CentralTransport == CentralTransportMode.Grpc)
{
builder.RequireThat(
options.CentralGrpcEndpoints.Count > 0
&& options.CentralGrpcEndpoints.All(e => !string.IsNullOrWhiteSpace(e)),
"ScadaBridge:Communication:CentralGrpcEndpoints must list at least one non-empty "
+ "central gRPC endpoint when CentralTransport is Grpc "
+ $"(was {options.CentralGrpcEndpoints.Count} entr{(options.CentralGrpcEndpoints.Count == 1 ? "y" : "ies")}).");
}
// ── Aggregated live alarm cache (plan #10, Task 6) ─────────────────────── // ── Aggregated live alarm cache (plan #10, Task 6) ───────────────────────
// Linger drives a Timer dueTime; TimeSpan.Zero is valid (stop the aggregator // Linger drives a Timer dueTime; TimeSpan.Zero is valid (stop the aggregator
// immediately when the last viewer leaves), only a negative value is invalid. // immediately when the last viewer leaves), only a negative value is invalid.
@@ -0,0 +1,274 @@
using Google.Protobuf.WellKnownTypes;
using Grpc.Core;
using Grpc.Net.Client;
using Microsoft.Extensions.Logging;
namespace ZB.MOM.WW.ScadaBridge.Communication.Grpc;
/// <summary>
/// A pair (or more) of gRPC channels to the central cluster's control-plane nodes, with the
/// sticky-failover + background-failback policy of the design (§3.5). The site holds one channel
/// per central endpoint, prefers the first, and stays on it until a call proves it unreachable —
/// only then flipping to the next, and only <see cref="StatusCode.Unavailable"/> / connect
/// failures count (a <see cref="StatusCode.DeadlineExceeded"/> never flips or retries, because the
/// call may have run).
/// </summary>
/// <remarks>
/// <para>
/// <b>Sticky.</b> All calls go to the current channel; a healthy preferred endpoint never
/// ping-pongs. <see cref="ReportUnavailable"/> flips to the next endpoint (round-robin) when the
/// caller sees the current one refuse a connection.
/// </para>
/// <para>
/// <b>Failback.</b> While off the preferred endpoint a background probe (a cheap <c>Heartbeat</c>
/// ping — always answered, even by a not-yet-ready node) re-checks the preferred one. On the first
/// success new calls return to it; in-flight calls finish where they are. The probe is
/// event-driven: it arms on a flip and stops the moment we are back on the preferred endpoint, so
/// a steady healthy pair spends no cycles. Its cadence backs off exponentially — 1 s, doubling,
/// capped at 60 s — while the preferred endpoint stays down, so a genuinely dead node is not
/// probed hard.
/// </para>
/// <para>
/// <b>Auth.</b> Every channel carries the site's own preshared key and its
/// <c>x-scadabridge-site</c> identity via <see cref="ControlPlaneCredentials"/> — the same
/// insecure-h2c call-credentials shape the streaming client uses. The <paramref name="handlerFactory"/>
/// seam lets a test point a channel at an in-process <c>TestServer</c>; production uses a
/// keepalive-configured <see cref="SocketsHttpHandler"/>.
/// </para>
/// </remarks>
public sealed class CentralChannelProvider : IDisposable
{
private static readonly TimeSpan DefaultBackoffBase = TimeSpan.FromSeconds(1);
private static readonly TimeSpan DefaultBackoffCap = TimeSpan.FromSeconds(60);
private static readonly TimeSpan DefaultProbeDeadline = TimeSpan.FromSeconds(5);
private readonly IReadOnlyList<string> _endpoints;
private readonly GrpcChannel[] _channels;
private readonly CentralControlService.CentralControlServiceClient[] _clients;
private readonly ILogger _logger;
private readonly string _siteId;
private readonly TimeSpan _backoffBase;
private readonly TimeSpan _backoffCap;
private readonly TimeSpan _probeDeadline;
private readonly Timer? _failbackTimer;
private readonly object _gate = new();
private volatile int _current; // preferred == 0
private int _consecutiveProbeFailures;
private bool _disposed;
/// <summary>Creates the provider and opens one channel per endpoint.</summary>
/// <param name="endpoints">Central control-plane endpoints, preferred first (index 0). Must be non-empty.</param>
/// <param name="pskProvider">Resolves this site's preshared key (site-side: a single-key provider).</param>
/// <param name="siteId">This site's identity, sent as the <c>x-scadabridge-site</c> header.</param>
/// <param name="options">Communication options supplying gRPC keepalive settings.</param>
/// <param name="logger">Logger for flip/failback diagnostics.</param>
/// <param name="handlerFactory">Test seam: per-endpoint <see cref="HttpMessageHandler"/>; null uses a production socket handler.</param>
/// <param name="probeDeadline">Deadline for a failback probe. Null uses 5 s.</param>
/// <param name="backoffBase">Initial failback-probe backoff. Null uses 1 s.</param>
/// <param name="backoffCap">Maximum failback-probe backoff. Null uses 60 s.</param>
public CentralChannelProvider(
IReadOnlyList<string> endpoints,
ISitePskProvider pskProvider,
string siteId,
CommunicationOptions options,
ILogger logger,
Func<string, HttpMessageHandler>? handlerFactory = null,
TimeSpan? probeDeadline = null,
TimeSpan? backoffBase = null,
TimeSpan? backoffCap = null)
{
ArgumentNullException.ThrowIfNull(endpoints);
ArgumentNullException.ThrowIfNull(pskProvider);
ArgumentException.ThrowIfNullOrWhiteSpace(siteId);
ArgumentNullException.ThrowIfNull(options);
ArgumentNullException.ThrowIfNull(logger);
if (endpoints.Count == 0)
{
throw new ArgumentException("At least one central gRPC endpoint is required.", nameof(endpoints));
}
_endpoints = endpoints;
_logger = logger;
_siteId = siteId;
_backoffBase = backoffBase ?? DefaultBackoffBase;
_backoffCap = backoffCap ?? DefaultBackoffCap;
_probeDeadline = probeDeadline ?? DefaultProbeDeadline;
_channels = new GrpcChannel[endpoints.Count];
_clients = new CentralControlService.CentralControlServiceClient[endpoints.Count];
for (var i = 0; i < endpoints.Count; i++)
{
var channelOptions = new GrpcChannelOptions
{
HttpHandler = handlerFactory?.Invoke(endpoints[i]) ?? new SocketsHttpHandler
{
KeepAlivePingDelay = options.GrpcKeepAlivePingDelay,
KeepAlivePingTimeout = options.GrpcKeepAlivePingTimeout,
KeepAlivePingPolicy = HttpKeepAlivePingPolicy.Always,
EnableMultipleHttp2Connections = true,
},
}.WithSiteCredentials(pskProvider, siteId);
_channels[i] = GrpcChannel.ForAddress(endpoints[i], channelOptions);
_clients[i] = new CentralControlService.CentralControlServiceClient(_channels[i]);
}
// Only a multi-endpoint pair can ever fail over, so a lone endpoint needs no probe.
if (endpoints.Count > 1)
{
_failbackTimer = new Timer(_ => _ = FailbackTickAsync(), null, Timeout.Infinite, Timeout.Infinite);
}
}
/// <summary>The number of endpoints in the pair.</summary>
public int EndpointCount => _endpoints.Count;
/// <summary>The index of the endpoint calls are currently routed to (preferred == 0).</summary>
public int CurrentIndex => _current;
/// <summary>The endpoint address calls are currently routed to.</summary>
public string CurrentEndpoint => _endpoints[_current];
/// <summary>
/// The endpoint index and client calls should use right now. Captured together so a caller can
/// tell <see cref="ReportUnavailable"/> exactly which endpoint failed even if a concurrent flip
/// has already moved <see cref="CurrentIndex"/>.
/// </summary>
/// <returns>The current endpoint index and its client.</returns>
public (int Index, CentralControlService.CentralControlServiceClient Client) Current()
{
var idx = _current;
return (idx, _clients[idx]);
}
/// <summary>
/// Reports that the endpoint at <paramref name="failedIndex"/> refused a connection (an
/// <see cref="StatusCode.Unavailable"/> / connect failure). If it is still the current endpoint
/// and another exists, flips to the next one and — when now off the preferred endpoint — arms
/// the failback probe. Idempotent under a concurrent flip: a stale index is ignored.
/// </summary>
/// <param name="failedIndex">The endpoint index the caller's failed call used.</param>
public void ReportUnavailable(int failedIndex)
{
if (_endpoints.Count < 2)
{
return; // nothing to fail over to
}
lock (_gate)
{
if (_disposed || failedIndex != _current)
{
return; // a concurrent flip already moved us; do not double-flip
}
var next = (failedIndex + 1) % _endpoints.Count;
_current = next;
_logger.LogWarning(
"Central control-plane endpoint {Failed} is unavailable; site {SiteId} failed over to {Next}.",
_endpoints[failedIndex], _siteId, _endpoints[next]);
if (_current != 0)
{
_consecutiveProbeFailures = 0;
ArmFailback(_backoffBase);
}
}
}
private void ArmFailback(TimeSpan due)
{
if (_disposed)
{
return;
}
_failbackTimer?.Change(due, Timeout.InfiniteTimeSpan);
}
private async Task FailbackTickAsync()
{
int currentAtTick = _current;
if (_disposed || currentAtTick == 0)
{
return; // already back on the preferred endpoint (or shutting down)
}
var preferred = _clients[0];
try
{
await preferred.HeartbeatAsync(
new HeartbeatDto
{
SiteId = _siteId,
NodeHostname = "failback-probe",
IsActive = false,
Timestamp = Timestamp.FromDateTimeOffset(DateTimeOffset.UtcNow),
},
deadline: DateTime.UtcNow.Add(_probeDeadline)).ConfigureAwait(false);
// The preferred endpoint answered — return new calls to it.
lock (_gate)
{
if (_disposed)
{
return;
}
_current = 0;
_consecutiveProbeFailures = 0;
}
_logger.LogInformation(
"Central control-plane preferred endpoint {Preferred} is reachable again; site {SiteId} failed back.",
_endpoints[0], _siteId);
}
catch (Exception ex)
{
lock (_gate)
{
if (_disposed || _current == 0)
{
return;
}
_consecutiveProbeFailures++;
var backoff = NextBackoff(_consecutiveProbeFailures);
_logger.LogDebug(ex,
"Failback probe of preferred central endpoint {Preferred} failed; re-probing in {Backoff}.",
_endpoints[0], backoff);
ArmFailback(backoff);
}
}
}
private TimeSpan NextBackoff(int failures)
{
// 1 s, doubling, capped at 60 s. Guard the shift against overflow for a long outage.
var exponent = Math.Min(failures - 1, 20);
var scaled = _backoffBase.Ticks * (1L << exponent);
var cap = _backoffCap.Ticks;
return TimeSpan.FromTicks(scaled >= cap || scaled < 0 ? cap : scaled);
}
/// <inheritdoc />
public void Dispose()
{
lock (_gate)
{
if (_disposed)
{
return;
}
_disposed = true;
}
_failbackTimer?.Dispose();
foreach (var channel in _channels)
{
channel.Dispose();
}
}
}
@@ -0,0 +1,764 @@
using ZB.MOM.WW.ScadaBridge.Commons.Messages.Audit;
using ZB.MOM.WW.ScadaBridge.Commons.Messages.Deployment;
using ZB.MOM.WW.ScadaBridge.Commons.Messages.Health;
using ZB.MOM.WW.ScadaBridge.Commons.Messages.Notification;
using ZB.MOM.WW.ScadaBridge.Commons.Types;
using ZB.MOM.WW.ScadaBridge.Commons.Types.Enums;
using Timestamp = Google.Protobuf.WellKnownTypes.Timestamp;
namespace ZB.MOM.WW.ScadaBridge.Communication.Grpc;
/// <summary>
/// Canonical bridge between the seven in-process messages the site sends to central
/// and the wire format of <c>CentralControlService</c> (<c>Protos/central_control.proto</c>).
/// </summary>
/// <remarks>
/// <para>
/// The seven pairs mirror, one for one, the seven messages
/// <c>SiteCommunicationActor</c> forwards to <c>/user/central-communication</c> over
/// Akka <c>ClusterClient</c> today. Both transports carry the SAME message types
/// end-to-end — central's handlers are untouched by the migration — so this mapper is
/// the only place the two representations meet, and a field that does not survive a
/// round-trip here is a field the gRPC transport silently drops.
/// </para>
/// <para>
/// <b>Ingest is deliberately absent from the message list.</b> <c>IngestAuditEvents</c>
/// and <c>IngestCachedTelemetry</c> reuse the <c>AuditEventBatch</c> /
/// <c>CachedTelemetryBatch</c> / <c>IngestAck</c> messages already defined for the
/// site-hosted <c>SiteStreamService</c>, so the per-row work is delegated to the
/// existing <see cref="AuditEventDtoMapper"/> and <see cref="SiteCallDtoMapper"/>; only
/// the batch/ack envelopes are assembled here.
/// </para>
///
/// <para><b>Conventions, applied uniformly across every method below.</b></para>
/// <list type="bullet">
/// <item>
/// <b>Nullable strings ↔ empty strings.</b> A proto3 scalar string cannot be absent,
/// so a null .NET string is written as <see cref="string.Empty"/> and an empty wire
/// string is read back as <see langword="null"/>. This is the convention already in
/// force on <see cref="AuditEventDtoMapper"/>, and it is why no field on this wire
/// may distinguish "null" from "deliberately empty".
/// </item>
/// <item>
/// <b>Nullable <see cref="Guid"/> ↔ string.</b> Execution ids travel as their "D"
/// string form; the empty string means <see langword="null"/>. A malformed non-empty
/// value throws out of <c>FromDto</c> rather than degrading to null — a corrupt
/// correlation id must not be laundered into "no correlation".
/// </item>
/// <item>
/// <b>Nullable numbers and booleans ↔ protobuf wrapper types.</b>
/// <c>Int32Value</c>/<c>Int64Value</c>/<c>DoubleValue</c>/<c>BoolValue</c> preserve
/// true null. Several health gauges (<c>LocalDbOplogBacklog</c>,
/// <c>LocalDbReplicationConnected</c>) are documented as "null means unknown, and
/// that is NOT the same as zero/false"; collapsing them to a bare scalar would
/// report a broken replication pair as healthy.
/// </item>
/// <item>
/// <b>Nullable collections ↔ wrapper messages.</b> proto3 cannot express presence on
/// a <c>repeated</c> or <c>map</c> field, so the three nullable
/// <see cref="SiteHealthReport"/> collections travel inside single-field wrapper
/// messages (<c>ConnectionEndpointMapDto</c>, <c>TagQualityMapDto</c>,
/// <c>NodeStatusListDto</c>). An absent wrapper is null; a present-but-empty wrapper
/// is an empty collection.
/// </item>
/// <item>
/// <b><see cref="DateTimeOffset"/> normalizes to a UTC instant.</b> A protobuf
/// <c>Timestamp</c> is an instant, not an offset-qualified local time, so the offset
/// component is dropped and the value round-trips with <c>Offset == TimeSpan.Zero</c>.
/// Every producer in this system stamps UTC (the repo-wide invariant; e.g.
/// <c>Notify.Send</c> uses <c>DateTimeOffset.UtcNow</c>), so this is lossless in
/// practice and the instant is preserved regardless.
/// </item>
/// </list>
/// </remarks>
public static class CentralControlDtoMapper
{
// -----------------------------------------------------------------------
// Notification Outbox (#21)
// -----------------------------------------------------------------------
/// <summary>Projects a <see cref="NotificationSubmit"/> onto its wire DTO.</summary>
/// <param name="msg">The notification submission to project.</param>
/// <returns>The wire-format DTO; null strings and null execution ids collapse to empty strings.</returns>
public static NotificationSubmitDto ToDto(NotificationSubmit msg)
{
ArgumentNullException.ThrowIfNull(msg);
return new NotificationSubmitDto
{
NotificationId = msg.NotificationId,
ListName = msg.ListName,
Subject = msg.Subject,
Body = msg.Body,
SourceSiteId = msg.SourceSiteId,
SourceInstanceId = msg.SourceInstanceId ?? string.Empty,
SourceScript = msg.SourceScript ?? string.Empty,
SiteEnqueuedAt = Timestamp.FromDateTimeOffset(msg.SiteEnqueuedAt),
OriginExecutionId = GuidToWire(msg.OriginExecutionId),
OriginParentExecutionId = GuidToWire(msg.OriginParentExecutionId),
SourceNode = msg.SourceNode ?? string.Empty,
};
}
/// <summary>Reconstructs a <see cref="NotificationSubmit"/> from its wire DTO.</summary>
/// <param name="dto">The wire-format DTO to reconstruct.</param>
/// <returns>The in-process message; empty strings rehydrate as null.</returns>
public static NotificationSubmit FromDto(NotificationSubmitDto dto)
{
ArgumentNullException.ThrowIfNull(dto);
return new NotificationSubmit(
NotificationId: dto.NotificationId,
ListName: dto.ListName,
Subject: dto.Subject,
Body: dto.Body,
SourceSiteId: dto.SourceSiteId,
SourceInstanceId: NullIfEmpty(dto.SourceInstanceId),
SourceScript: NullIfEmpty(dto.SourceScript),
SiteEnqueuedAt: dto.SiteEnqueuedAt.ToDateTimeOffset(),
OriginExecutionId: GuidFromWire(dto.OriginExecutionId),
OriginParentExecutionId: GuidFromWire(dto.OriginParentExecutionId),
SourceNode: NullIfEmpty(dto.SourceNode));
}
/// <summary>Projects a <see cref="NotificationSubmitAck"/> onto its wire DTO.</summary>
/// <param name="msg">The ack to project.</param>
/// <returns>The wire-format DTO; a null error collapses to an empty string.</returns>
public static NotificationSubmitAckDto ToDto(NotificationSubmitAck msg)
{
ArgumentNullException.ThrowIfNull(msg);
return new NotificationSubmitAckDto
{
NotificationId = msg.NotificationId,
Accepted = msg.Accepted,
Error = msg.Error ?? string.Empty,
};
}
/// <summary>Reconstructs a <see cref="NotificationSubmitAck"/> from its wire DTO.</summary>
/// <param name="dto">The wire-format DTO to reconstruct.</param>
/// <returns>The in-process ack; an empty error rehydrates as null.</returns>
public static NotificationSubmitAck FromDto(NotificationSubmitAckDto dto)
{
ArgumentNullException.ThrowIfNull(dto);
return new NotificationSubmitAck(
NotificationId: dto.NotificationId,
Accepted: dto.Accepted,
Error: NullIfEmpty(dto.Error));
}
/// <summary>Projects a <see cref="NotificationStatusQuery"/> onto its wire DTO.</summary>
/// <param name="msg">The status query to project.</param>
/// <returns>The wire-format DTO.</returns>
public static NotificationStatusQueryDto ToDto(NotificationStatusQuery msg)
{
ArgumentNullException.ThrowIfNull(msg);
return new NotificationStatusQueryDto
{
CorrelationId = msg.CorrelationId,
NotificationId = msg.NotificationId,
};
}
/// <summary>Reconstructs a <see cref="NotificationStatusQuery"/> from its wire DTO.</summary>
/// <param name="dto">The wire-format DTO to reconstruct.</param>
/// <returns>The in-process query.</returns>
public static NotificationStatusQuery FromDto(NotificationStatusQueryDto dto)
{
ArgumentNullException.ThrowIfNull(dto);
return new NotificationStatusQuery(
CorrelationId: dto.CorrelationId,
NotificationId: dto.NotificationId);
}
/// <summary>Projects a <see cref="NotificationStatusResponse"/> onto its wire DTO.</summary>
/// <param name="msg">The status response to project.</param>
/// <returns>The wire-format DTO; a null delivery timestamp leaves the field unset.</returns>
public static NotificationStatusResponseDto ToDto(NotificationStatusResponse msg)
{
ArgumentNullException.ThrowIfNull(msg);
var dto = new NotificationStatusResponseDto
{
CorrelationId = msg.CorrelationId,
Found = msg.Found,
Status = msg.Status,
RetryCount = msg.RetryCount,
LastError = msg.LastError ?? string.Empty,
};
if (msg.DeliveredAt.HasValue)
{
dto.DeliveredAt = Timestamp.FromDateTimeOffset(msg.DeliveredAt.Value);
}
return dto;
}
/// <summary>Reconstructs a <see cref="NotificationStatusResponse"/> from its wire DTO.</summary>
/// <param name="dto">The wire-format DTO to reconstruct.</param>
/// <returns>The in-process response; an unset delivery timestamp rehydrates as null.</returns>
public static NotificationStatusResponse FromDto(NotificationStatusResponseDto dto)
{
ArgumentNullException.ThrowIfNull(dto);
return new NotificationStatusResponse(
CorrelationId: dto.CorrelationId,
Found: dto.Found,
Status: dto.Status,
RetryCount: dto.RetryCount,
LastError: NullIfEmpty(dto.LastError),
DeliveredAt: dto.DeliveredAt?.ToDateTimeOffset());
}
// -----------------------------------------------------------------------
// Audit Log (#23) ingest — envelopes only; rows go through the existing mappers
// -----------------------------------------------------------------------
/// <summary>
/// Projects an <see cref="IngestAuditEventsCommand"/> onto the shared
/// <see cref="AuditEventBatch"/> wire message.
/// </summary>
/// <remarks>
/// The per-row projection is <see cref="AuditEventDtoMapper.ToDto"/>, which is lossy
/// by design: <c>ForwardState</c> is site-local storage state and <c>IngestedAtUtc</c>
/// is stamped centrally at ingest, so neither travels.
/// </remarks>
/// <param name="cmd">The ingest command to project.</param>
/// <returns>A batch carrying one DTO per audit event, in order.</returns>
public static AuditEventBatch ToDto(IngestAuditEventsCommand cmd)
{
ArgumentNullException.ThrowIfNull(cmd);
var batch = new AuditEventBatch();
foreach (var evt in cmd.Events)
{
batch.Events.Add(AuditEventDtoMapper.ToDto(evt));
}
return batch;
}
/// <summary>
/// Reconstructs an <see cref="IngestAuditEventsCommand"/> from the shared
/// <see cref="AuditEventBatch"/> wire message — the shape central's
/// <c>CentralCommunicationActor</c> already handles.
/// </summary>
/// <param name="batch">The wire batch to reconstruct.</param>
/// <returns>The in-process ingest command.</returns>
public static IngestAuditEventsCommand FromDto(AuditEventBatch batch)
{
ArgumentNullException.ThrowIfNull(batch);
var events = new List<ZB.MOM.WW.Audit.AuditEvent>(batch.Events.Count);
foreach (var dto in batch.Events)
{
events.Add(AuditEventDtoMapper.FromDto(dto));
}
return new IngestAuditEventsCommand(events);
}
/// <summary>
/// Projects an <see cref="IngestCachedTelemetryCommand"/> onto the shared
/// <see cref="CachedTelemetryBatch"/> wire message.
/// </summary>
/// <param name="cmd">The cached-telemetry ingest command to project.</param>
/// <returns>A batch carrying one packet (audit row + operational row) per entry, in order.</returns>
public static CachedTelemetryBatch ToDto(IngestCachedTelemetryCommand cmd)
{
ArgumentNullException.ThrowIfNull(cmd);
var batch = new CachedTelemetryBatch();
foreach (var entry in cmd.Entries)
{
batch.Packets.Add(new CachedTelemetryPacket
{
AuditEvent = AuditEventDtoMapper.ToDto(entry.Audit),
Operational = SiteCallDtoMapper.ToDto(entry.SiteCall),
});
}
return batch;
}
/// <summary>
/// Reconstructs an <see cref="IngestCachedTelemetryCommand"/> from the shared
/// <see cref="CachedTelemetryBatch"/> wire message.
/// </summary>
/// <param name="batch">The wire batch to reconstruct.</param>
/// <returns>The in-process dual-write ingest command.</returns>
public static IngestCachedTelemetryCommand FromDto(CachedTelemetryBatch batch)
{
ArgumentNullException.ThrowIfNull(batch);
var entries = new List<CachedTelemetryEntry>(batch.Packets.Count);
foreach (var packet in batch.Packets)
{
entries.Add(new CachedTelemetryEntry(
AuditEventDtoMapper.FromDto(packet.AuditEvent),
SiteCallDtoMapper.FromDto(packet.Operational)));
}
return new IngestCachedTelemetryCommand(entries);
}
/// <summary>
/// Projects the accepted-id list of an ingest reply onto the shared
/// <see cref="IngestAck"/> wire message. Shared by both ingest RPCs — the two
/// central reply types differ only in which handler produced them.
/// </summary>
/// <param name="acceptedEventIds">Ids central considers durably persisted.</param>
/// <returns>The wire ack carrying the ids in "D" string form, in order.</returns>
public static IngestAck ToIngestAck(IReadOnlyList<Guid> acceptedEventIds)
{
ArgumentNullException.ThrowIfNull(acceptedEventIds);
var ack = new IngestAck();
foreach (var id in acceptedEventIds)
{
ack.AcceptedEventIds.Add(id.ToString());
}
return ack;
}
/// <summary>
/// Reads the accepted-id list back out of an <see cref="IngestAck"/>.
/// </summary>
/// <param name="ack">The wire ack to read.</param>
/// <returns>The accepted event ids, in wire order.</returns>
public static IReadOnlyList<Guid> FromIngestAck(IngestAck ack)
{
ArgumentNullException.ThrowIfNull(ack);
var ids = new List<Guid>(ack.AcceptedEventIds.Count);
foreach (var id in ack.AcceptedEventIds)
{
ids.Add(Guid.Parse(id));
}
return ids;
}
// -----------------------------------------------------------------------
// Startup reconciliation
// -----------------------------------------------------------------------
/// <summary>Projects a <see cref="ReconcileSiteRequest"/> onto its wire DTO.</summary>
/// <param name="msg">The reconcile request to project.</param>
/// <returns>The wire-format DTO carrying the node's local name→revision-hash inventory.</returns>
public static ReconcileSiteRequestDto ToDto(ReconcileSiteRequest msg)
{
ArgumentNullException.ThrowIfNull(msg);
var dto = new ReconcileSiteRequestDto
{
SiteIdentifier = msg.SiteIdentifier,
NodeId = msg.NodeId,
};
foreach (var (name, hash) in msg.LocalNameToRevisionHash)
{
dto.LocalNameToRevisionHash[name] = hash;
}
return dto;
}
/// <summary>Reconstructs a <see cref="ReconcileSiteRequest"/> from its wire DTO.</summary>
/// <param name="dto">The wire-format DTO to reconstruct.</param>
/// <returns>The in-process reconcile request.</returns>
public static ReconcileSiteRequest FromDto(ReconcileSiteRequestDto dto)
{
ArgumentNullException.ThrowIfNull(dto);
return new ReconcileSiteRequest(
SiteIdentifier: dto.SiteIdentifier,
NodeId: dto.NodeId,
LocalNameToRevisionHash: new Dictionary<string, string>(dto.LocalNameToRevisionHash));
}
/// <summary>Projects a <see cref="ReconcileSiteResponse"/> onto its wire DTO.</summary>
/// <param name="msg">The reconcile response to project.</param>
/// <returns>The wire-format DTO carrying the gap items, orphan names and fetch base URL.</returns>
public static ReconcileSiteResponseDto ToDto(ReconcileSiteResponse msg)
{
ArgumentNullException.ThrowIfNull(msg);
var dto = new ReconcileSiteResponseDto
{
CentralFetchBaseUrl = msg.CentralFetchBaseUrl,
};
foreach (var item in msg.Gap)
{
dto.Gap.Add(new ReconcileGapItemDto
{
InstanceUniqueName = item.InstanceUniqueName,
DeploymentId = item.DeploymentId,
RevisionHash = item.RevisionHash,
IsEnabled = item.IsEnabled,
FetchToken = item.FetchToken,
});
}
dto.OrphanNames.AddRange(msg.OrphanNames);
return dto;
}
/// <summary>Reconstructs a <see cref="ReconcileSiteResponse"/> from its wire DTO.</summary>
/// <param name="dto">The wire-format DTO to reconstruct.</param>
/// <returns>The in-process reconcile response.</returns>
public static ReconcileSiteResponse FromDto(ReconcileSiteResponseDto dto)
{
ArgumentNullException.ThrowIfNull(dto);
var gap = new List<ReconcileGapItem>(dto.Gap.Count);
foreach (var item in dto.Gap)
{
gap.Add(new ReconcileGapItem(
InstanceUniqueName: item.InstanceUniqueName,
DeploymentId: item.DeploymentId,
RevisionHash: item.RevisionHash,
IsEnabled: item.IsEnabled,
FetchToken: item.FetchToken));
}
return new ReconcileSiteResponse(
Gap: gap,
OrphanNames: dto.OrphanNames.ToList(),
CentralFetchBaseUrl: dto.CentralFetchBaseUrl);
}
// -----------------------------------------------------------------------
// Health Monitoring (#11)
// -----------------------------------------------------------------------
/// <summary>Projects a <see cref="SiteHealthReport"/> onto its wire DTO.</summary>
/// <remarks>
/// The three nullable collections travel inside wrapper messages so a null stays
/// distinguishable from an empty collection; the nullable gauges travel in protobuf
/// wrapper types for the same reason.
/// </remarks>
/// <param name="msg">The health report to project.</param>
/// <returns>The wire-format DTO.</returns>
public static SiteHealthReportDto ToDto(SiteHealthReport msg)
{
ArgumentNullException.ThrowIfNull(msg);
var dto = new SiteHealthReportDto
{
SiteId = msg.SiteId,
SequenceNumber = msg.SequenceNumber,
ReportTimestamp = Timestamp.FromDateTimeOffset(msg.ReportTimestamp),
ScriptErrorCount = msg.ScriptErrorCount,
AlarmEvaluationErrorCount = msg.AlarmEvaluationErrorCount,
DeadLetterCount = msg.DeadLetterCount,
DeployedInstanceCount = msg.DeployedInstanceCount,
EnabledInstanceCount = msg.EnabledInstanceCount,
DisabledInstanceCount = msg.DisabledInstanceCount,
NodeRole = msg.NodeRole,
NodeHostname = msg.NodeHostname,
ParkedMessageCount = msg.ParkedMessageCount,
SiteAuditWriteFailures = msg.SiteAuditWriteFailures,
AuditRedactionFailure = msg.AuditRedactionFailure,
SiteEventLogWriteFailures = msg.SiteEventLogWriteFailures,
OldestParkedMessageAgeSeconds = msg.OldestParkedMessageAgeSeconds,
ScriptQueueDepth = msg.ScriptQueueDepth,
ScriptBusyThreads = msg.ScriptBusyThreads,
ScriptOldestBusyAgeSeconds = msg.ScriptOldestBusyAgeSeconds,
LocalDbReplicationConnected = msg.LocalDbReplicationConnected,
LocalDbOplogBacklog = msg.LocalDbOplogBacklog,
};
foreach (var (name, health) in msg.DataConnectionStatuses)
{
dto.DataConnectionStatuses[name] = ToDto(health);
}
foreach (var (name, resolution) in msg.TagResolutionCounts)
{
dto.TagResolutionCounts[name] = new TagResolutionStatusDto
{
TotalSubscribed = resolution.TotalSubscribed,
SuccessfullyResolved = resolution.SuccessfullyResolved,
};
}
foreach (var (name, depth) in msg.StoreAndForwardBufferDepths)
{
dto.StoreAndForwardBufferDepths[name] = depth;
}
if (msg.DataConnectionEndpoints is { } endpoints)
{
var wrapper = new ConnectionEndpointMapDto();
foreach (var (name, endpoint) in endpoints)
{
wrapper.Entries[name] = endpoint;
}
dto.DataConnectionEndpoints = wrapper;
}
if (msg.DataConnectionTagQuality is { } tagQuality)
{
var wrapper = new TagQualityMapDto();
foreach (var (name, counts) in tagQuality)
{
wrapper.Entries[name] = new TagQualityCountsDto
{
Good = counts.Good,
Bad = counts.Bad,
Uncertain = counts.Uncertain,
};
}
dto.DataConnectionTagQuality = wrapper;
}
if (msg.ClusterNodes is { } clusterNodes)
{
var wrapper = new NodeStatusListDto();
foreach (var node in clusterNodes)
{
wrapper.Nodes.Add(new NodeStatusDto
{
Hostname = node.Hostname,
IsOnline = node.IsOnline,
Role = node.Role,
});
}
dto.ClusterNodes = wrapper;
}
if (msg.SiteAuditBacklog is { } backlog)
{
var snapshot = new SiteAuditBacklogSnapshotDto
{
PendingCount = backlog.PendingCount,
OnDiskBytes = backlog.OnDiskBytes,
};
if (backlog.OldestPendingUtc.HasValue)
{
snapshot.OldestPendingUtc = Timestamp.FromDateTime(EnsureUtc(backlog.OldestPendingUtc.Value));
}
dto.SiteAuditBacklog = snapshot;
}
return dto;
}
/// <summary>Reconstructs a <see cref="SiteHealthReport"/> from its wire DTO.</summary>
/// <param name="dto">The wire-format DTO to reconstruct.</param>
/// <returns>The in-process health report; absent wrappers rehydrate as null, not as empty.</returns>
public static SiteHealthReport FromDto(SiteHealthReportDto dto)
{
ArgumentNullException.ThrowIfNull(dto);
var statuses = new Dictionary<string, ConnectionHealth>(dto.DataConnectionStatuses.Count);
foreach (var (name, health) in dto.DataConnectionStatuses)
{
statuses[name] = FromDto(health);
}
var resolution = new Dictionary<string, TagResolutionStatus>(dto.TagResolutionCounts.Count);
foreach (var (name, counts) in dto.TagResolutionCounts)
{
resolution[name] = new TagResolutionStatus(counts.TotalSubscribed, counts.SuccessfullyResolved);
}
var bufferDepths = new Dictionary<string, int>(dto.StoreAndForwardBufferDepths);
Dictionary<string, string>? endpoints = null;
if (dto.DataConnectionEndpoints is { } endpointWrapper)
{
endpoints = new Dictionary<string, string>(endpointWrapper.Entries);
}
Dictionary<string, TagQualityCounts>? tagQuality = null;
if (dto.DataConnectionTagQuality is { } tagQualityWrapper)
{
tagQuality = new Dictionary<string, TagQualityCounts>(tagQualityWrapper.Entries.Count);
foreach (var (name, counts) in tagQualityWrapper.Entries)
{
tagQuality[name] = new TagQualityCounts(counts.Good, counts.Bad, counts.Uncertain);
}
}
List<NodeStatus>? clusterNodes = null;
if (dto.ClusterNodes is { } nodeWrapper)
{
clusterNodes = new List<NodeStatus>(nodeWrapper.Nodes.Count);
foreach (var node in nodeWrapper.Nodes)
{
clusterNodes.Add(new NodeStatus(node.Hostname, node.IsOnline, node.Role));
}
}
SiteAuditBacklogSnapshot? backlog = null;
if (dto.SiteAuditBacklog is { } snapshot)
{
backlog = new SiteAuditBacklogSnapshot(
PendingCount: snapshot.PendingCount,
OldestPendingUtc: snapshot.OldestPendingUtc is null
? null
: DateTime.SpecifyKind(snapshot.OldestPendingUtc.ToDateTime(), DateTimeKind.Utc),
OnDiskBytes: snapshot.OnDiskBytes);
}
return new SiteHealthReport(
SiteId: dto.SiteId,
SequenceNumber: dto.SequenceNumber,
ReportTimestamp: dto.ReportTimestamp.ToDateTimeOffset(),
DataConnectionStatuses: statuses,
TagResolutionCounts: resolution,
ScriptErrorCount: dto.ScriptErrorCount,
AlarmEvaluationErrorCount: dto.AlarmEvaluationErrorCount,
StoreAndForwardBufferDepths: bufferDepths,
DeadLetterCount: dto.DeadLetterCount,
DeployedInstanceCount: dto.DeployedInstanceCount,
EnabledInstanceCount: dto.EnabledInstanceCount,
DisabledInstanceCount: dto.DisabledInstanceCount,
NodeRole: dto.NodeRole,
NodeHostname: dto.NodeHostname,
DataConnectionEndpoints: endpoints,
DataConnectionTagQuality: tagQuality,
ParkedMessageCount: dto.ParkedMessageCount,
ClusterNodes: clusterNodes,
SiteAuditWriteFailures: dto.SiteAuditWriteFailures,
AuditRedactionFailure: dto.AuditRedactionFailure,
SiteAuditBacklog: backlog,
SiteEventLogWriteFailures: dto.SiteEventLogWriteFailures,
OldestParkedMessageAgeSeconds: dto.OldestParkedMessageAgeSeconds)
{
// Init-only members: SiteHealthReport surfaces the scheduler and LocalDb
// gauges as init properties rather than positional parameters, so they
// cannot be passed to the constructor above.
ScriptQueueDepth = dto.ScriptQueueDepth,
ScriptBusyThreads = dto.ScriptBusyThreads,
ScriptOldestBusyAgeSeconds = dto.ScriptOldestBusyAgeSeconds,
LocalDbReplicationConnected = dto.LocalDbReplicationConnected,
LocalDbOplogBacklog = dto.LocalDbOplogBacklog,
};
}
/// <summary>Projects a <see cref="SiteHealthReportAck"/> onto its wire DTO.</summary>
/// <param name="msg">The ack to project.</param>
/// <returns>The wire-format DTO; a null error collapses to an empty string.</returns>
public static SiteHealthReportAckDto ToDto(SiteHealthReportAck msg)
{
ArgumentNullException.ThrowIfNull(msg);
return new SiteHealthReportAckDto
{
SiteId = msg.SiteId,
SequenceNumber = msg.SequenceNumber,
Accepted = msg.Accepted,
Error = msg.Error ?? string.Empty,
};
}
/// <summary>Reconstructs a <see cref="SiteHealthReportAck"/> from its wire DTO.</summary>
/// <param name="dto">The wire-format DTO to reconstruct.</param>
/// <returns>The in-process ack; an empty error rehydrates as null.</returns>
public static SiteHealthReportAck FromDto(SiteHealthReportAckDto dto)
{
ArgumentNullException.ThrowIfNull(dto);
return new SiteHealthReportAck(
SiteId: dto.SiteId,
SequenceNumber: dto.SequenceNumber,
Accepted: dto.Accepted,
Error: NullIfEmpty(dto.Error));
}
/// <summary>Projects a <see cref="HeartbeatMessage"/> onto its wire DTO.</summary>
/// <param name="msg">The heartbeat to project.</param>
/// <returns>The wire-format DTO.</returns>
public static HeartbeatDto ToDto(HeartbeatMessage msg)
{
ArgumentNullException.ThrowIfNull(msg);
return new HeartbeatDto
{
SiteId = msg.SiteId,
NodeHostname = msg.NodeHostname,
IsActive = msg.IsActive,
Timestamp = Timestamp.FromDateTimeOffset(msg.Timestamp),
};
}
/// <summary>Reconstructs a <see cref="HeartbeatMessage"/> from its wire DTO.</summary>
/// <param name="dto">The wire-format DTO to reconstruct.</param>
/// <returns>The in-process heartbeat.</returns>
public static HeartbeatMessage FromDto(HeartbeatDto dto)
{
ArgumentNullException.ThrowIfNull(dto);
return new HeartbeatMessage(
SiteId: dto.SiteId,
NodeHostname: dto.NodeHostname,
IsActive: dto.IsActive,
Timestamp: dto.Timestamp.ToDateTimeOffset());
}
/// <summary>Projects a <see cref="ConnectionHealth"/> onto its wire enum.</summary>
/// <param name="health">The connection health to project.</param>
/// <returns>The wire enum value. Never <c>ConnectionHealthUnspecified</c>.</returns>
/// <exception cref="ArgumentOutOfRangeException">
/// A <see cref="ConnectionHealth"/> value was added without extending this mapping.
/// Throwing beats a silent default: an unmapped state would otherwise be reported as
/// whichever value happened to be first.
/// </exception>
public static ConnectionHealthEnum ToDto(ConnectionHealth health) => health switch
{
ConnectionHealth.Connected => ConnectionHealthEnum.ConnectionHealthConnected,
ConnectionHealth.Disconnected => ConnectionHealthEnum.ConnectionHealthDisconnected,
ConnectionHealth.Connecting => ConnectionHealthEnum.ConnectionHealthConnecting,
ConnectionHealth.Error => ConnectionHealthEnum.ConnectionHealthError,
_ => throw new ArgumentOutOfRangeException(nameof(health), health, "Unmapped ConnectionHealth value"),
};
/// <summary>Reconstructs a <see cref="ConnectionHealth"/> from its wire enum.</summary>
/// <remarks>
/// An unspecified or unknown wire value decodes to <see cref="ConnectionHealth.Error"/>,
/// never to <see cref="ConnectionHealth.Connected"/>. A newer site sending a value this
/// build has never heard of must not have it rendered as "healthy" on the central
/// health page — the safe direction for an unknown connection state is "not working".
/// </remarks>
/// <param name="health">The wire enum value to reconstruct.</param>
/// <returns>The in-process connection health.</returns>
public static ConnectionHealth FromDto(ConnectionHealthEnum health) => health switch
{
ConnectionHealthEnum.ConnectionHealthConnected => ConnectionHealth.Connected,
ConnectionHealthEnum.ConnectionHealthDisconnected => ConnectionHealth.Disconnected,
ConnectionHealthEnum.ConnectionHealthConnecting => ConnectionHealth.Connecting,
_ => ConnectionHealth.Error,
};
private static string GuidToWire(Guid? value) =>
value?.ToString() ?? string.Empty;
private static Guid? GuidFromWire(string? value) =>
string.IsNullOrEmpty(value) ? null : Guid.Parse(value);
private static string? NullIfEmpty(string? value) =>
string.IsNullOrEmpty(value) ? null : value;
// All ScadaBridge timestamps are UTC by invariant; Timestamp.FromDateTime requires
// UTC kind. Specify (never convert) so a value read back from SQLite with Kind=Utc
// passes through and a defensively-unspecified one is treated as the UTC it already
// is. Mirrors AuditEventDtoMapper/SiteCallDtoMapper.EnsureUtc.
private static DateTime EnsureUtc(DateTime value) =>
value.Kind == DateTimeKind.Utc ? value : DateTime.SpecifyKind(value, DateTimeKind.Utc);
}
@@ -0,0 +1,299 @@
using Akka.Actor;
using Google.Protobuf.WellKnownTypes;
using Grpc.Core;
using Microsoft.Extensions.Logging;
using Microsoft.Extensions.Options;
using ZB.MOM.WW.ScadaBridge.Commons.Messages.Audit;
using ZB.MOM.WW.ScadaBridge.Commons.Messages.Deployment;
using ZB.MOM.WW.ScadaBridge.Commons.Messages.Health;
using ZB.MOM.WW.ScadaBridge.Commons.Messages.Notification;
using GrpcStatus = Grpc.Core.Status;
namespace ZB.MOM.WW.ScadaBridge.Communication.Grpc;
/// <summary>
/// Central-hosted gRPC face of the seven site→central control messages
/// (<c>Protos/central_control.proto</c>). Decodes each request onto the SAME in-process
/// message type the Akka <c>ClusterClient</c> path already carries, <c>Ask</c>s
/// <see cref="Actors.CentralCommunicationActor"/>, and encodes the reply back.
/// </summary>
/// <remarks>
/// <para>
/// <b>Direction is inverted from <see cref="SiteStreamGrpcServer"/>.</b> That server runs on a
/// site and central dials in; this one runs on CENTRAL and the site dials in. The two listen on
/// the same port number (8083) on their respective nodes, which is symmetry, not a collision —
/// a node is either central or a site, never both.
/// </para>
/// <para>
/// <b>Zero handler logic lives here.</b> Every RPC lands on a receive
/// <c>CentralCommunicationActor</c> already implements for the ClusterClient path, so the two
/// transports cannot drift in behaviour: the actor is the single implementation, and this class
/// is a codec plus an <c>Ask</c>. That is also why the service takes the actor through
/// <see cref="SetReady"/> rather than resolving anything from DI — the actor is created by the
/// host's Akka bootstrap, not by the container.
/// </para>
/// <para>
/// <b>Fault semantics deliberately differ from <see cref="SiteStreamGrpcServer"/>'s ingest
/// RPCs.</b> That server answers a failed audit ingest with an EMPTY <c>IngestAck</c>; this one
/// fails the call with a non-OK status. Both leave the site's rows <c>Pending</c> for the next
/// drain, so the outcome is the same — but this service replaces the ClusterClient path, whose
/// documented behaviour is to propagate the fault (<c>CentralCommunicationActor</c>'s
/// <c>HandleIngestAuditEvents</c> pipes a <c>Status.Failure</c> back), and preserving that keeps
/// a lost batch visible as a failure rather than as a successful call that acked nothing.
/// </para>
/// <para>
/// <b>Status mapping, and why it is not uniform.</b> A site transport may safely re-send a call
/// to the peer central node only when the call provably never ran. So:
/// </para>
/// <list type="bullet">
/// <item><see cref="StatusCode.Unavailable"/> — this node is not ready; nothing was dispatched,
/// so a cross-node retry is safe and correct.</item>
/// <item><see cref="StatusCode.DeadlineExceeded"/> — the <c>Ask</c> timed out. The message WAS
/// delivered and may have been processed; retrying it on the other node would duplicate work.</item>
/// <item><see cref="StatusCode.Internal"/> — the handler faulted (a piped
/// <see cref="Akka.Actor.Status.Failure"/>, e.g. a database error inside reconcile). Same
/// reasoning: it ran, so do not re-send it elsewhere.</item>
/// </list>
/// </remarks>
public sealed class CentralControlGrpcService : CentralControlService.CentralControlServiceBase
{
private readonly ILogger<CentralControlGrpcService> _logger;
private readonly CommunicationOptions _options;
// Null until the host's Akka bootstrap hands the actor over. Doubles as the readiness
// flag: a call arriving before then cannot be served and is refused with Unavailable.
// Volatile because SetReady runs on the startup thread while calls are served on
// Kestrel's thread pool.
private volatile IActorRef? _central;
/// <summary>
/// Creates the service. <b>This must remain the only public constructor</b> — see
/// <see cref="SetReady"/> for how the actor arrives, and the Host's
/// <c>CentralControlAuthInterceptor</c> for the interceptor-side version of the same rule.
/// </summary>
/// <param name="logger">Logger for readiness and fault diagnostics.</param>
/// <param name="options">Communication options supplying the per-RPC <c>Ask</c> timeouts.</param>
public CentralControlGrpcService(
ILogger<CentralControlGrpcService> logger,
IOptions<CommunicationOptions> options)
{
ArgumentNullException.ThrowIfNull(logger);
ArgumentNullException.ThrowIfNull(options);
_logger = logger;
_options = options.Value;
}
/// <summary>
/// Hands the <c>CentralCommunicationActor</c> to the service and opens the gate. Mirrors
/// <see cref="SiteStreamGrpcServer.SetReady"/>: the gRPC service is a DI singleton created
/// before the actor system exists, so the actor arrives post-construction.
/// </summary>
/// <remarks>
/// The contract is deliberately narrow, exactly as on the site side: it asserts that the
/// actor exists and can receive, NOT that every downstream singleton proxy
/// (<c>notification-outbox</c>, <c>audit-log-ingest</c>) has registered itself yet. Those
/// register moments later in the same startup path, and the actor already answers a call
/// that beats them with the same "not available, retry" reply it gives on the ClusterClient
/// path — so gating readiness on them would add nothing but a longer window in which sites
/// see <see cref="StatusCode.Unavailable"/>.
/// </remarks>
/// <param name="centralCommunicationActor">The central communication actor.</param>
public void SetReady(IActorRef centralCommunicationActor)
{
ArgumentNullException.ThrowIfNull(centralCommunicationActor);
_central = centralCommunicationActor;
}
/// <summary>Exposed for wiring assertions in tests.</summary>
internal bool IsReady => _central is not null;
/// <inheritdoc />
public override async Task<NotificationSubmitAckDto> SubmitNotification(
NotificationSubmitDto request, ServerCallContext context)
{
var central = RequireReady(context);
var ack = await AskAsync<NotificationSubmitAck>(
central,
CentralControlDtoMapper.FromDto(request),
_options.NotificationForwardTimeout,
context).ConfigureAwait(false);
return CentralControlDtoMapper.ToDto(ack);
}
/// <inheritdoc />
public override async Task<NotificationStatusResponseDto> QueryNotificationStatus(
NotificationStatusQueryDto request, ServerCallContext context)
{
var central = RequireReady(context);
var response = await AskAsync<NotificationStatusResponse>(
central,
CentralControlDtoMapper.FromDto(request),
_options.NotificationForwardTimeout,
context).ConfigureAwait(false);
return CentralControlDtoMapper.ToDto(response);
}
/// <inheritdoc />
public override async Task<IngestAck> IngestAuditEvents(
AuditEventBatch request, ServerCallContext context)
{
// An empty batch is a no-op the actor need never see; answering it here also means a
// site that drains an empty queue does not fail against a not-yet-ready central.
if (request.Events.Count == 0)
{
return new IngestAck();
}
var central = RequireReady(context);
var reply = await AskAsync<IngestAuditEventsReply>(
central,
CentralControlDtoMapper.FromDto(request),
SiteStreamGrpcServer.AuditIngestAskTimeout,
context).ConfigureAwait(false);
return CentralControlDtoMapper.ToIngestAck(reply.AcceptedEventIds);
}
/// <inheritdoc />
public override async Task<IngestAck> IngestCachedTelemetry(
CachedTelemetryBatch request, ServerCallContext context)
{
if (request.Packets.Count == 0)
{
return new IngestAck();
}
var central = RequireReady(context);
var reply = await AskAsync<IngestCachedTelemetryReply>(
central,
CentralControlDtoMapper.FromDto(request),
SiteStreamGrpcServer.AuditIngestAskTimeout,
context).ConfigureAwait(false);
return CentralControlDtoMapper.ToIngestAck(reply.AcceptedEventIds);
}
/// <inheritdoc />
public override async Task<ReconcileSiteResponseDto> ReconcileSite(
ReconcileSiteRequestDto request, ServerCallContext context)
{
var central = RequireReady(context);
var response = await AskAsync<ReconcileSiteResponse>(
central,
CentralControlDtoMapper.FromDto(request),
_options.QueryTimeout,
context).ConfigureAwait(false);
return CentralControlDtoMapper.ToDto(response);
}
/// <inheritdoc />
public override async Task<SiteHealthReportAckDto> ReportSiteHealth(
SiteHealthReportDto request, ServerCallContext context)
{
var central = RequireReady(context);
var ack = await AskAsync<SiteHealthReportAck>(
central,
CentralControlDtoMapper.FromDto(request),
_options.HealthReportTimeout,
context).ConfigureAwait(false);
return CentralControlDtoMapper.ToDto(ack);
}
/// <summary>
/// Application heartbeat — <b>always answers OK</b>, even when this node is not ready.
/// </summary>
/// <remarks>
/// The heartbeat is fire-and-forget on both sides: nothing on the site consumes the reply,
/// and the site's heartbeat timer must never take a fault (a failing heartbeat that raised
/// an error would be a self-inflicted outage on a purely informational signal). So this is
/// the one RPC that does not go through <see cref="RequireReady"/>: a heartbeat arriving
/// before the actor exists is logged and dropped, exactly as the actor itself drops one
/// that arrives before <c>ICentralHealthAggregator</c> is resolvable. Liveness is still
/// detected — the aggregator's offline timeout fires when the heartbeats stop landing.
/// </remarks>
/// <param name="request">The heartbeat.</param>
/// <param name="context">The gRPC call context.</param>
/// <returns>An empty reply, always.</returns>
public override Task<Empty> Heartbeat(HeartbeatDto request, ServerCallContext context)
{
var central = _central;
if (central is null)
{
_logger.LogDebug(
"Dropped a heartbeat from site {SiteId}: the central communication actor is not "
+ "ready yet. Heartbeats are fire-and-forget, so the call still succeeds.",
request.SiteId);
return Task.FromResult(new Empty());
}
// Tell, never Ask: the actor's HandleHeartbeat sends no reply.
central.Tell(CentralControlDtoMapper.FromDto(request), ActorRefs.NoSender);
return Task.FromResult(new Empty());
}
/// <summary>
/// Returns the central communication actor, or throws <see cref="StatusCode.Unavailable"/>
/// when the host has not finished bringing the actor system up.
/// </summary>
private IActorRef RequireReady(ServerCallContext context)
{
var central = _central;
if (central is not null)
{
return central;
}
_logger.LogWarning(
"Refused a control-plane call to {Method}: the central communication actor is not "
+ "ready yet. Nothing was dispatched, so the caller may retry (including against "
+ "the peer central node).",
context.Method);
throw new RpcException(new GrpcStatus(
StatusCode.Unavailable,
"Central control plane is not ready: the actor system is still starting."));
}
/// <summary>
/// Asks the central actor and maps a fault onto the status code that tells the caller
/// whether a cross-node retry is safe. See the class remarks for the mapping rationale.
/// </summary>
private async Task<TReply> AskAsync<TReply>(
IActorRef central, object message, TimeSpan timeout, ServerCallContext context)
{
try
{
return await central.Ask<TReply>(message, timeout, context.CancellationToken)
.ConfigureAwait(false);
}
catch (AskTimeoutException ex)
{
_logger.LogWarning(ex,
"Control-plane call {Method} timed out after {Timeout} waiting for the central "
+ "communication actor.",
context.Method, timeout);
throw new RpcException(new GrpcStatus(
StatusCode.DeadlineExceeded,
$"Central did not answer within {timeout}."));
}
catch (OperationCanceledException)
{
// The client gave up or its deadline expired; there is no one left to answer.
throw new RpcException(new GrpcStatus(
StatusCode.Cancelled, "The call was cancelled."));
}
catch (Exception ex)
{
_logger.LogError(ex,
"Control-plane call {Method} faulted inside the central communication actor.",
context.Method);
throw new RpcException(new GrpcStatus(
StatusCode.Internal, "Central failed to process the call."));
}
}
}
@@ -0,0 +1,258 @@
using Akka.Actor;
using Grpc.Core;
using Microsoft.Extensions.Logging;
using ZB.MOM.WW.ScadaBridge.Commons.Messages.Audit;
using ZB.MOM.WW.ScadaBridge.Commons.Messages.Deployment;
using ZB.MOM.WW.ScadaBridge.Commons.Messages.Health;
using ZB.MOM.WW.ScadaBridge.Commons.Messages.Notification;
using ZB.MOM.WW.ScadaBridge.Communication.Actors;
using AkkaStatus = Akka.Actor.Status;
namespace ZB.MOM.WW.ScadaBridge.Communication.Grpc;
/// <summary>
/// The <see cref="ICentralTransport"/> that carries the seven site→central sends over gRPC — the
/// migration target for the Akka <c>ClusterClient</c> path. Each method encodes the message with
/// <see cref="CentralControlDtoMapper"/>, dials <c>CentralControlService</c> through the sticky
/// <see cref="CentralChannelProvider"/>, and delivers the decoded reply (or a transient-failure
/// signal) to the waiting Ask.
/// </summary>
/// <remarks>
/// <para>
/// The actor handlers are synchronous; each method here kicks off the RPC as a detached task and
/// <c>Tell</c>s the result to <paramref name="replyTo"/> when it completes — <c>IActorRef.Tell</c>
/// is thread-safe, so the reply lands at the Ask exactly as central's ClusterClient reply did. On
/// any non-OK status the reply is <see cref="Status.Failure"/>, which the S&amp;F / audit / health
/// layers already treat as transient.
/// </para>
/// <para>
/// <b>Cross-node retry only on provably-unsent failures.</b> An <see cref="StatusCode.Unavailable"/>
/// (connection refused / node not ready) flips the channel pair and retries once on the peer.
/// A <see cref="StatusCode.DeadlineExceeded"/> is NEVER retried across nodes — a deploy / write /
/// failover may already have executed, and duplicating it is worse than surfacing a transient
/// failure the layer above tolerates.
/// </para>
/// <para>
/// <b>Per-call deadlines mirror today's Ask timeouts.</b> Notification submit/status →
/// <c>NotificationForwardTimeout</c> (30 s, the value the S&amp;F forwarder and the central service
/// both use); health → <c>HealthReportTimeout</c> (10 s); reconcile → <c>QueryTimeout</c> (30 s);
/// both ingest RPCs → <see cref="SiteStreamGrpcServer.AuditIngestAskTimeout"/> (the one shared 30 s
/// constant). The heartbeat, fire-and-forget with no server-side Ask, is merely bounded by
/// <c>HealthReportTimeout</c> and its failures are swallowed.
/// </para>
/// </remarks>
public sealed class GrpcCentralTransport : ICentralTransport
{
private readonly CentralChannelProvider _channels;
private readonly CommunicationOptions _options;
private readonly ILogger<GrpcCentralTransport> _logger;
/// <summary>Creates the transport over a channel pair.</summary>
/// <param name="channels">The sticky central channel pair.</param>
/// <param name="options">Communication options supplying the per-call deadlines.</param>
/// <param name="logger">Logger for failover/fault diagnostics.</param>
public GrpcCentralTransport(
CentralChannelProvider channels,
CommunicationOptions options,
ILogger<GrpcCentralTransport> logger)
{
ArgumentNullException.ThrowIfNull(channels);
ArgumentNullException.ThrowIfNull(options);
ArgumentNullException.ThrowIfNull(logger);
_channels = channels;
_options = options;
_logger = logger;
}
/// <inheritdoc />
public void SubmitNotification(NotificationSubmit message, IActorRef replyTo)
{
var dto = CentralControlDtoMapper.ToDto(message);
Dispatch(replyTo, _options.NotificationForwardTimeout,
(c, o) => c.SubmitNotificationAsync(dto, o),
ack => CentralControlDtoMapper.FromDto(ack));
}
/// <inheritdoc />
public void QueryNotificationStatus(NotificationStatusQuery message, IActorRef replyTo)
{
var dto = CentralControlDtoMapper.ToDto(message);
Dispatch(replyTo, _options.NotificationForwardTimeout,
(c, o) => c.QueryNotificationStatusAsync(dto, o),
response => CentralControlDtoMapper.FromDto(response));
}
/// <inheritdoc />
public void IngestAuditEvents(IngestAuditEventsCommand message, IActorRef replyTo)
{
var batch = CentralControlDtoMapper.ToDto(message);
Dispatch(replyTo, SiteStreamGrpcServer.AuditIngestAskTimeout,
(c, o) => c.IngestAuditEventsAsync(batch, o),
ack => new IngestAuditEventsReply(CentralControlDtoMapper.FromIngestAck(ack)));
}
/// <inheritdoc />
public void IngestCachedTelemetry(IngestCachedTelemetryCommand message, IActorRef replyTo)
{
var batch = CentralControlDtoMapper.ToDto(message);
Dispatch(replyTo, SiteStreamGrpcServer.AuditIngestAskTimeout,
(c, o) => c.IngestCachedTelemetryAsync(batch, o),
ack => new IngestCachedTelemetryReply(CentralControlDtoMapper.FromIngestAck(ack)));
}
/// <inheritdoc />
public void ReconcileSite(ReconcileSiteRequest message, IActorRef replyTo)
{
var dto = CentralControlDtoMapper.ToDto(message);
Dispatch(replyTo, _options.QueryTimeout,
(c, o) => c.ReconcileSiteAsync(dto, o),
response => CentralControlDtoMapper.FromDto(response));
}
/// <inheritdoc />
public void ReportSiteHealth(SiteHealthReport message, IActorRef replyTo)
{
var dto = CentralControlDtoMapper.ToDto(message);
Dispatch(replyTo, _options.HealthReportTimeout,
(c, o) => c.ReportSiteHealthAsync(dto, o),
ack => CentralControlDtoMapper.FromDto(ack));
}
/// <inheritdoc />
public void SendHeartbeat(HeartbeatMessage message, IActorRef self)
{
_ = SendHeartbeatAsync(message);
}
private async Task SendHeartbeatAsync(HeartbeatMessage message)
{
var (index, client) = _channels.Current();
var dto = CentralControlDtoMapper.ToDto(message);
try
{
var options = new CallOptions(deadline: DateTime.UtcNow.Add(_options.HealthReportTimeout));
using var call = client.HeartbeatAsync(dto, options);
await call.ResponseAsync.ConfigureAwait(false);
}
catch (RpcException ex) when (IsConnectFailure(ex))
{
// Nudge the pair so the next call tries the peer, but never fault: a heartbeat
// failure must not surface on the site's heartbeat timer path.
_channels.ReportUnavailable(index);
_logger.LogDebug(ex, "Heartbeat to central endpoint {Endpoint} was unavailable.", CurrentEndpointSafe());
}
catch (Exception ex)
{
_logger.LogDebug(ex, "Heartbeat to central failed (swallowed — heartbeats are fire-and-forget).");
}
}
/// <summary>
/// Runs a unary RPC on the current channel, delivers the decoded reply to
/// <paramref name="replyTo"/>, and applies the sticky-failover / no-retry-on-deadline policy.
/// </summary>
private void Dispatch<TWire>(
IActorRef replyTo,
TimeSpan timeout,
Func<CentralControlService.CentralControlServiceClient, CallOptions, AsyncUnaryCall<TWire>> call,
Func<TWire, object> decode)
{
_ = DispatchAsync(replyTo, timeout, call, decode);
}
private async Task DispatchAsync<TWire>(
IActorRef replyTo,
TimeSpan timeout,
Func<CentralControlService.CentralControlServiceClient, CallOptions, AsyncUnaryCall<TWire>> call,
Func<TWire, object> decode)
{
var (index, client) = _channels.Current();
try
{
var reply = await InvokeAsync(client, timeout, call).ConfigureAwait(false);
replyTo.Tell(decode(reply));
}
catch (RpcException ex) when (IsConnectFailure(ex))
{
// Provably unsent: the connection was refused / the node was not ready. Fail over
// to the peer and retry ONCE. This is the only status we retry across nodes.
_channels.ReportUnavailable(index);
var (retryIndex, retryClient) = _channels.Current();
if (retryIndex != index)
{
try
{
var reply = await InvokeAsync(retryClient, timeout, call).ConfigureAwait(false);
replyTo.Tell(decode(reply));
return;
}
catch (Exception retryEx)
{
_logger.LogWarning(retryEx,
"Central control-plane call failed on both endpoints; surfacing as transient.");
replyTo.Tell(new AkkaStatus.Failure(retryEx));
return;
}
}
replyTo.Tell(new AkkaStatus.Failure(ex));
}
catch (Exception ex)
{
// DeadlineExceeded / Internal / PermissionDenied / a PSK-resolution throw — do NOT
// retry across nodes (the call may have run). Surface as the transient failure the
// layer above already tolerates.
replyTo.Tell(new AkkaStatus.Failure(ex));
}
}
private static async Task<TWire> InvokeAsync<TWire>(
CentralControlService.CentralControlServiceClient client,
TimeSpan timeout,
Func<CentralControlService.CentralControlServiceClient, CallOptions, AsyncUnaryCall<TWire>> call)
{
var options = new CallOptions(deadline: DateTime.UtcNow.Add(timeout));
using var asyncCall = call(client, options);
return await asyncCall.ResponseAsync.ConfigureAwait(false);
}
/// <summary>
/// A failure that provably never reached a server — the only class safe to retry on the peer.
/// <see cref="StatusCode.DeadlineExceeded"/> is deliberately excluded (the call may have run).
/// </summary>
/// <remarks>
/// Two shapes qualify: a server-signalled <see cref="StatusCode.Unavailable"/> (e.g. a node
/// that returns Unavailable while it is still starting), and a client-side failure to even
/// start the call — Grpc.Net surfaces a refused/failed connection as
/// <see cref="StatusCode.Internal"/> "Error starting gRPC call" with the transport exception
/// attached, and there the request never left the client. Anything else — including a deadline,
/// a permission denial, or a generic server-side Internal after the call reached the server —
/// is NOT retried across nodes.
/// </remarks>
private static bool IsConnectFailure(RpcException ex)
{
if (ex.StatusCode == StatusCode.Unavailable)
{
return true;
}
// "Error starting gRPC call" is Grpc.Net's marker for a call that could not be sent — a
// refused/failed connection carrying an HttpRequestException. Provably unsent.
return ex.StatusCode == StatusCode.Internal
&& (ex.Status.DebugException is HttpRequestException
|| ex.Status.Detail.StartsWith("Error starting gRPC call", StringComparison.Ordinal));
}
private string CurrentEndpointSafe()
{
try
{
return _channels.CurrentEndpoint;
}
catch
{
return "(unknown)";
}
}
}
@@ -0,0 +1,132 @@
using Grpc.Core;
using Grpc.Net.Client;
namespace ZB.MOM.WW.ScadaBridge.Communication.Grpc;
/// <summary>
/// Resolves the preshared key that authenticates gRPC control-plane traffic for one site
/// relationship. Central holds one key per site; each site holds only its own.
/// </summary>
/// <remarks>
/// <para>
/// <b>Why per-site rather than one fleet-wide key.</b> A compromised site yields only its own
/// key, never another site's. A single shared key would be simpler to seed and strictly worse
/// on blast radius, which is why it was rejected in the design.
/// </para>
/// <para>
/// <b>Fail-closed.</b> Implementations throw when the key cannot be resolved. They must never
/// fall back to "no key means no authentication" — that is the failure mode this whole
/// mechanism exists to remove, and it would silently disable auth on exactly the default
/// configuration. A dial that cannot be authenticated does not happen.
/// </para>
/// <para>
/// The interface lives in Communication (not Host) because both sides need it: central's
/// site-dialing clients live here and in AuditLog, while the implementation over
/// <c>ISecretResolver</c> lives in Host, which owns the secrets container.
/// </para>
/// </remarks>
public interface ISitePskProvider
{
/// <summary>
/// Resolves the preshared key for <paramref name="siteId"/>, caching the result.
/// </summary>
/// <param name="siteId">Site identifier, as used in the <c>Site.SiteIdentifier</c> column.</param>
/// <param name="ct">Cancellation token.</param>
/// <returns>The preshared key. Never null or empty.</returns>
/// <exception cref="InvalidOperationException">
/// The key is not configured, not resolvable, or empty — the fail-closed path.
/// </exception>
ValueTask<string> GetAsync(string siteId, CancellationToken ct);
/// <summary>
/// Drops any cached key for <paramref name="siteId"/>, so the next
/// <see cref="GetAsync"/> re-reads the store. Called when a site is removed, and after a
/// key rotation.
/// </summary>
/// <param name="siteId">Site identifier whose cached key should be discarded.</param>
void Invalidate(string siteId);
}
/// <summary>
/// Builds the call credentials that carry a site's preshared key (and the site's own identity)
/// on every gRPC call central makes to that site — and, from Phase 1A, on the calls a site makes
/// to central.
/// </summary>
/// <remarks>
/// <para>
/// <b>Why <see cref="CallCredentials.FromInterceptor(AsyncAuthInterceptor)"/> rather than a
/// client <c>Interceptor</c>.</b> The key is resolved asynchronously from the secrets store, and
/// this is the one extension point in gRPC that is async by design. A client interceptor would
/// have to block on the resolve inside a synchronous <c>AsyncServerStreamingCall</c> path.
/// Credentials also apply uniformly to unary and streaming calls, so no call site can forget one.
/// </para>
/// <para>
/// <b>Why <c>UnsafeUseInsecureChannelCallCredentials</c>.</b> gRPC refuses to attach call
/// credentials to a plaintext channel by default, precisely because a bearer token on h2c is
/// readable and replayable by anyone on the path. That is a real and accepted limitation here:
/// these listeners are h2c today and the boundary assumes a trusted network. The PSK raises the
/// bar from "anyone who can reach the port" to "anyone who can read the traffic"; TLS on these
/// listeners is the follow-on hardening and requires no change to this code.
/// </para>
/// </remarks>
public static class ControlPlaneCredentials
{
/// <summary>
/// Metadata header naming the site a call belongs to. Central needs it to pick which
/// per-site key to verify against; a site's own inbound gate ignores it (a site has exactly
/// one key). Required by central's interceptor from Phase 1A.
/// </summary>
public const string SiteHeader = "x-scadabridge-site";
/// <summary>The bearer metadata header. Lowercase — gRPC lowercases header keys on the wire.</summary>
public const string AuthorizationHeader = "authorization";
/// <summary>
/// Creates call credentials that attach <c>authorization: Bearer &lt;psk&gt;</c> and
/// <c>x-scadabridge-site: &lt;siteId&gt;</c> to every call.
/// </summary>
/// <param name="provider">Resolves the site's preshared key.</param>
/// <param name="siteId">The site this channel talks to (or, site-side, this site's own id).</param>
/// <returns>Call credentials for a channel bound to that site.</returns>
public static CallCredentials ForSite(ISitePskProvider provider, string siteId)
{
ArgumentNullException.ThrowIfNull(provider);
ArgumentException.ThrowIfNullOrWhiteSpace(siteId);
return CallCredentials.FromInterceptor(async (context, metadata) =>
{
// A throw here fails the call, which is the point: an unauthenticated dial must
// not happen. Callers classify the resulting fault the same way they classify any
// other — the pull clients degrade to an empty batch and log, the streaming
// subscribers retry.
var psk = await provider.GetAsync(siteId, context.CancellationToken).ConfigureAwait(false);
metadata.Add(AuthorizationHeader, $"Bearer {psk}");
metadata.Add(SiteHeader, siteId);
});
}
/// <summary>
/// Applies per-site call credentials to channel options, if a provider is available.
/// A null provider leaves the options untouched — the shape used by test-only and
/// default constructors that never dial a gated endpoint.
/// </summary>
/// <param name="options">Channel options being built.</param>
/// <param name="provider">Key provider, or null to leave the channel unauthenticated.</param>
/// <param name="siteId">The site this channel talks to.</param>
/// <returns>The same options instance, for chaining.</returns>
public static GrpcChannelOptions WithSiteCredentials(
this GrpcChannelOptions options, ISitePskProvider? provider, string? siteId)
{
ArgumentNullException.ThrowIfNull(options);
if (provider is null || string.IsNullOrWhiteSpace(siteId))
{
return options;
}
options.Credentials = ChannelCredentials.Create(
ChannelCredentials.Insecure, ForSite(provider, siteId));
options.UnsafeUseInsecureChannelCallCredentials = true;
return options;
}
}
@@ -21,15 +21,17 @@ namespace ZB.MOM.WW.ScadaBridge.Communication.Grpc;
/// Mirrors the sibling <see cref="AuditEventDtoMapper"/>. /// Mirrors the sibling <see cref="AuditEventDtoMapper"/>.
/// </para> /// </para>
/// <para> /// <para>
/// Two directions are provided. <see cref="FromDto"/> rehydrates the central /// Three directions are provided. <see cref="FromDto"/> rehydrates the central
/// <see cref="SiteCall"/> entity central writes into the <c>SiteCalls</c> table. /// <see cref="SiteCall"/> entity central writes into the <c>SiteCalls</c> table.
/// <see cref="ToDto"/> projects a site-local <see cref="SiteCallOperational"/> /// <see cref="ToDto(SiteCallOperational)"/> projects a site-local
/// onto the wire — used by the Site Call Audit <c>PullSiteCalls</c> /// <see cref="SiteCallOperational"/> onto the wire — used by the Site Call Audit
/// reconciliation handler (the central→site self-heal pull). The /// <c>PullSiteCalls</c> reconciliation handler (the central→site self-heal pull).
/// <see cref="SiteCall"/> entity itself is never mapped back onto the wire: /// <see cref="ToDto(SiteCall)"/> projects the entity form back onto the wire; it
/// sites emit operational state from <see cref="SiteCallOperational"/>, never /// exists for the gRPC central control plane, whose site-side transport receives
/// from the central <see cref="SiteCall"/>, so a <c>SiteCall</c>→DTO method /// an already-decoded <c>IngestCachedTelemetryCommand</c> (which carries
/// would be dead code. /// <see cref="SiteCall"/>, not <see cref="SiteCallOperational"/>) and must
/// re-encode it. It was previously documented here as necessarily dead code —
/// true only while ClusterClient was the sole path from that command to central.
/// </para> /// </para>
/// <para> /// <para>
/// String nullability convention: proto3 scalar strings cannot be absent, so the /// String nullability convention: proto3 scalar strings cannot be absent, so the
@@ -120,6 +122,51 @@ public static class SiteCallDtoMapper
return dto; return dto;
} }
/// <summary>
/// Projects a <see cref="SiteCall"/> entity onto its wire-format DTO — the
/// inverse of <see cref="FromDto"/>, so the pair round-trips exactly.
/// </summary>
/// <remarks>
/// <see cref="SiteCall.IngestedAtUtc"/> is deliberately NOT written: it is
/// central-set inside the dual-write transaction and the value carried on the
/// wire is informational only (<see cref="FromDto"/> stamps a placeholder that
/// the ingest actor overwrites). Every other field survives; null
/// <c>SourceNode</c>/<c>LastError</c> collapse to empty strings while the
/// nullable <c>HttpStatus</c>/<c>TerminalAtUtc</c> stay unset on the wire.
/// </remarks>
/// <param name="siteCall">The central operational-state entity to project.</param>
/// <returns>A populated <see cref="SiteCallOperationalDto"/> ready for transmission.</returns>
public static SiteCallOperationalDto ToDto(SiteCall siteCall)
{
ArgumentNullException.ThrowIfNull(siteCall);
var dto = new SiteCallOperationalDto
{
TrackedOperationId = siteCall.TrackedOperationId.ToString(),
Channel = siteCall.Channel,
Target = siteCall.Target,
SourceSite = siteCall.SourceSite,
SourceNode = siteCall.SourceNode ?? string.Empty,
Status = siteCall.Status,
RetryCount = siteCall.RetryCount,
LastError = siteCall.LastError ?? string.Empty,
CreatedAtUtc = Timestamp.FromDateTime(EnsureUtc(siteCall.CreatedAtUtc)),
UpdatedAtUtc = Timestamp.FromDateTime(EnsureUtc(siteCall.UpdatedAtUtc)),
};
if (siteCall.HttpStatus.HasValue)
{
dto.HttpStatus = siteCall.HttpStatus.Value;
}
if (siteCall.TerminalAtUtc.HasValue)
{
dto.TerminalAtUtc = Timestamp.FromDateTime(EnsureUtc(siteCall.TerminalAtUtc.Value));
}
return dto;
}
// All ScadaBridge timestamps are UTC by invariant; Timestamp.FromDateTime // All ScadaBridge timestamps are UTC by invariant; Timestamp.FromDateTime
// requires UTC kind. Specify (never convert) so a row read back from SQLite // requires UTC kind. Specify (never convert) so a row read back from SQLite
// with Kind=Utc passes through and a defensively-unspecified value is // with Kind=Utc passes through and a defensively-unspecified value is
@@ -60,6 +60,27 @@ public class SiteStreamGrpcClient : IAsyncDisposable, IDisposable
/// <param name="logger">Logger for diagnostics and errors.</param> /// <param name="logger">Logger for diagnostics and errors.</param>
/// <param name="options">Communication options including keepalive settings.</param> /// <param name="options">Communication options including keepalive settings.</param>
public SiteStreamGrpcClient(string endpoint, ILogger logger, CommunicationOptions options) public SiteStreamGrpcClient(string endpoint, ILogger logger, CommunicationOptions options)
: this(endpoint, logger, options, pskProvider: null, siteIdentifier: null)
{
}
/// <summary>
/// Creates a client that authenticates every call with the site's preshared key.
/// This is the production shape: <c>SiteStreamService</c> is gated by
/// <c>ControlPlaneAuthInterceptor</c> on the site node, so a client without credentials
/// gets <see cref="StatusCode.PermissionDenied"/> on every call.
/// </summary>
/// <param name="endpoint">The gRPC endpoint address for the site.</param>
/// <param name="logger">Logger for diagnostics and errors.</param>
/// <param name="options">Communication options including keepalive settings.</param>
/// <param name="pskProvider">Resolves the site's preshared key; null leaves the channel unauthenticated.</param>
/// <param name="siteIdentifier">Site this channel talks to; null leaves the channel unauthenticated.</param>
public SiteStreamGrpcClient(
string endpoint,
ILogger logger,
CommunicationOptions options,
ISitePskProvider? pskProvider,
string? siteIdentifier)
{ {
Endpoint = endpoint; Endpoint = endpoint;
KeepAlivePingDelay = options.GrpcKeepAlivePingDelay; KeepAlivePingDelay = options.GrpcKeepAlivePingDelay;
@@ -72,7 +93,7 @@ public class SiteStreamGrpcClient : IAsyncDisposable, IDisposable
KeepAlivePingTimeout = options.GrpcKeepAlivePingTimeout, KeepAlivePingTimeout = options.GrpcKeepAlivePingTimeout,
KeepAlivePingPolicy = HttpKeepAlivePingPolicy.Always KeepAlivePingPolicy = HttpKeepAlivePingPolicy.Always
} }
}); }.WithSiteCredentials(pskProvider, siteIdentifier));
_client = new SiteStreamService.SiteStreamServiceClient(_channel); _client = new SiteStreamService.SiteStreamServiceClient(_channel);
_logger = logger; _logger = logger;
} }
@@ -26,9 +26,11 @@ public class SiteStreamGrpcClientFactory : IAsyncDisposable, IDisposable
private readonly ConcurrentDictionary<(string Site, string Endpoint), SiteStreamGrpcClient> _clients = new(); private readonly ConcurrentDictionary<(string Site, string Endpoint), SiteStreamGrpcClient> _clients = new();
private readonly ILoggerFactory _loggerFactory; private readonly ILoggerFactory _loggerFactory;
private readonly CommunicationOptions _options; private readonly CommunicationOptions _options;
private readonly ISitePskProvider? _pskProvider;
/// <summary> /// <summary>
/// Test/default constructor — uses default <see cref="CommunicationOptions"/>. /// Test/default constructor — uses default <see cref="CommunicationOptions"/> and creates
/// unauthenticated channels.
/// </summary> /// </summary>
/// <param name="loggerFactory">Logger factory passed to created clients.</param> /// <param name="loggerFactory">Logger factory passed to created clients.</param>
public SiteStreamGrpcClientFactory(ILoggerFactory loggerFactory) public SiteStreamGrpcClientFactory(ILoggerFactory loggerFactory)
@@ -37,16 +39,36 @@ public class SiteStreamGrpcClientFactory : IAsyncDisposable, IDisposable
} }
/// <summary> /// <summary>
/// DI constructor — flows <see cref="CommunicationOptions"/> into every created /// Constructor without a key provider — creates unauthenticated channels, which a gated
/// <see cref="SiteStreamGrpcClient"/> so the configured gRPC keepalive settings /// site will refuse. Retained for tests and for hosts that never dial a site.
/// are applied rather than hard-coded defaults.
/// </summary> /// </summary>
/// <param name="loggerFactory">Logger factory passed to created clients.</param> /// <param name="loggerFactory">Logger factory passed to created clients.</param>
/// <param name="options">Communication options applied to each created client.</param> /// <param name="options">Communication options applied to each created client.</param>
public SiteStreamGrpcClientFactory(ILoggerFactory loggerFactory, IOptions<CommunicationOptions> options) public SiteStreamGrpcClientFactory(ILoggerFactory loggerFactory, IOptions<CommunicationOptions> options)
: this(loggerFactory, options, pskProvider: null)
{
}
/// <summary>
/// DI constructor — flows <see cref="CommunicationOptions"/> into every created
/// <see cref="SiteStreamGrpcClient"/> so the configured gRPC keepalive settings are applied
/// rather than hard-coded defaults, and attaches the per-site preshared key that the site's
/// <c>ControlPlaneAuthInterceptor</c> requires.
/// </summary>
/// <param name="loggerFactory">Logger factory passed to created clients.</param>
/// <param name="options">Communication options applied to each created client.</param>
/// <param name="pskProvider">
/// Resolves each site's preshared key. Optional in DI so a host that registers no provider
/// (a site node, which never dials another site) still resolves this factory.
/// </param>
public SiteStreamGrpcClientFactory(
ILoggerFactory loggerFactory,
IOptions<CommunicationOptions> options,
ISitePskProvider? pskProvider)
{ {
_loggerFactory = loggerFactory; _loggerFactory = loggerFactory;
_options = options.Value; _options = options.Value;
_pskProvider = pskProvider;
} }
/// <summary> /// <summary>
@@ -59,7 +81,7 @@ public class SiteStreamGrpcClientFactory : IAsyncDisposable, IDisposable
/// <param name="grpcEndpoint">gRPC endpoint (second half of the cache key) the client is bound to.</param> /// <param name="grpcEndpoint">gRPC endpoint (second half of the cache key) the client is bound to.</param>
/// <returns>The cached or newly-created client bound to <paramref name="grpcEndpoint"/>.</returns> /// <returns>The cached or newly-created client bound to <paramref name="grpcEndpoint"/>.</returns>
public virtual SiteStreamGrpcClient GetOrCreate(string siteIdentifier, string grpcEndpoint) => public virtual SiteStreamGrpcClient GetOrCreate(string siteIdentifier, string grpcEndpoint) =>
_clients.GetOrAdd((siteIdentifier, grpcEndpoint), _ => CreateClient(grpcEndpoint)); _clients.GetOrAdd((siteIdentifier, grpcEndpoint), key => CreateClient(key.Site, key.Endpoint));
/// <summary> /// <summary>
/// Returns the cached client for <c>(site, endpoint)</c>, or <c>null</c> — never creates. /// Returns the cached client for <c>(site, endpoint)</c>, or <c>null</c> — never creates.
@@ -77,12 +99,13 @@ public class SiteStreamGrpcClientFactory : IAsyncDisposable, IDisposable
/// can substitute a tracking client while still exercising the factory's real /// can substitute a tracking client while still exercising the factory's real
/// caching and disposal machinery. /// caching and disposal machinery.
/// </summary> /// </summary>
/// <param name="siteIdentifier">Site the new client talks to; selects which preshared key it presents.</param>
/// <param name="grpcEndpoint">gRPC endpoint the new client will connect to.</param> /// <param name="grpcEndpoint">gRPC endpoint the new client will connect to.</param>
/// <returns>A new <see cref="SiteStreamGrpcClient"/> connected to <paramref name="grpcEndpoint"/>.</returns> /// <returns>A new <see cref="SiteStreamGrpcClient"/> connected to <paramref name="grpcEndpoint"/>.</returns>
protected virtual SiteStreamGrpcClient CreateClient(string grpcEndpoint) protected virtual SiteStreamGrpcClient CreateClient(string siteIdentifier, string grpcEndpoint)
{ {
var logger = _loggerFactory.CreateLogger<SiteStreamGrpcClient>(); var logger = _loggerFactory.CreateLogger<SiteStreamGrpcClient>();
return new SiteStreamGrpcClient(grpcEndpoint, logger, _options); return new SiteStreamGrpcClient(grpcEndpoint, logger, _options, _pskProvider, siteIdentifier);
} }
/// <summary> /// <summary>
@@ -99,6 +122,10 @@ public class SiteStreamGrpcClientFactory : IAsyncDisposable, IDisposable
if (_clients.TryRemove(key, out var client)) if (_clients.TryRemove(key, out var client))
await client.DisposeAsync(); await client.DisposeAsync();
} }
// Drop the cached preshared key too, so a site removed and re-added under the same
// identifier (with a rotated key) is not dialed with the stale one.
_pskProvider?.Invalidate(siteIdentifier);
} }
/// <summary> /// <summary>
@@ -0,0 +1,43 @@
namespace ZB.MOM.WW.ScadaBridge.Communication.Grpc;
/// <summary>
/// Site-side <see cref="ISitePskProvider"/> over a single fixed key — the one key a site node
/// presents on every control-plane call it makes to central (<c>CommunicationOptions.GrpcPsk</c>).
/// Central's provider resolves a key <em>per site</em>; a site has exactly one, so it ignores the
/// requested <c>siteId</c> and returns its own key.
/// </summary>
/// <remarks>
/// <b>Fail-closed.</b> An empty key throws, matching the contract on <see cref="ISitePskProvider"/>
/// and the interceptor's own posture: a node shipped without a key must not degrade to an
/// unauthenticated dial.
/// </remarks>
public sealed class StaticSitePskProvider : ISitePskProvider
{
private readonly string _key;
/// <summary>Creates the provider bound to a site's own preshared key.</summary>
/// <param name="key">The site's <c>GrpcPsk</c>. Empty is permitted at construction but throws on use.</param>
public StaticSitePskProvider(string key)
{
_key = key ?? string.Empty;
}
/// <inheritdoc />
public ValueTask<string> GetAsync(string siteId, CancellationToken ct)
{
if (string.IsNullOrEmpty(_key))
{
throw new InvalidOperationException(
"No gRPC preshared key is configured for this site (ScadaBridge:Communication:GrpcPsk). "
+ "The control plane is fail-closed: an unauthenticated dial does not happen.");
}
return new ValueTask<string>(_key);
}
/// <inheritdoc />
public void Invalidate(string siteId)
{
// A single static key never changes for the process lifetime; nothing to drop.
}
}
@@ -0,0 +1,247 @@
syntax = "proto3";
option csharp_namespace = "ZB.MOM.WW.ScadaBridge.Communication.Grpc";
package scadabridge.centralcontrol.v1;
import "google/protobuf/empty.proto";
import "google/protobuf/timestamp.proto";
import "google/protobuf/wrappers.proto";
// The two ingest RPCs deliberately REUSE the batch/ack messages already defined
// for the site-hosted SiteStreamService rather than redeclaring them. The site
// telemetry actor builds AuditEventBatch / CachedTelemetryBatch today and hands
// them to ISiteStreamAuditClient; duplicating the shapes here would fork one
// wire contract into two that must be kept in lockstep by hand.
import "Protos/sitestream.proto";
// Central-hosted control plane (Phase 1A of the ClusterClientgRPC migration).
//
// Direction: SITE is the client, CENTRAL is the server the inverse of
// SiteStreamService, where central dials the site. That asymmetry is deliberate
// and mirrors the direction the Akka ClusterClient traffic flows today: these
// seven calls are exactly the seven messages SiteCommunicationActor sends to
// /user/central-communication.
//
// Every call is gated by ControlPlaneAuthInterceptor: `authorization: Bearer <psk>`
// plus the `x-scadabridge-site` metadata header naming which site's preshared key
// central must verify against.
service CentralControlService {
// Store-and-forward handoff of one notification for central delivery. The
// ack is idempotent on notification_id a duplicate submit after a lost ack
// must not produce a second delivery.
rpc SubmitNotification(NotificationSubmitDto) returns (NotificationSubmitAckDto);
// Notify.Status(id) round-trip for a notification that has already left the
// site buffer. `found = false` sends the caller back to the site-local buffer
// to decide Forwarding vs Unknown.
rpc QueryNotificationStatus(NotificationStatusQueryDto) returns (NotificationStatusResponseDto);
// Audit Log (#23) push telemetry. Reuses the SiteStreamService messages: the
// batch a site drains from its SQLite hot path is byte-identical whichever
// transport carries it.
rpc IngestAuditEvents(sitestream.AuditEventBatch) returns (sitestream.IngestAck);
// Audit Log (#23) M3 combined cached-call telemetry (audit row + SiteCalls
// operational upsert, written in one central transaction).
rpc IngestCachedTelemetry(sitestream.CachedTelemetryBatch) returns (sitestream.IngestAck);
// Node-startup self-heal: the node's local deployed inventory in, fetch
// tokens for whatever it is missing or stale out.
rpc ReconcileSite(ReconcileSiteRequestDto) returns (ReconcileSiteResponseDto);
// Periodic site health report (30 s cadence). The ack makes delivery
// observable end-to-end so the sender can restore its per-interval counters
// when a report is lost.
rpc ReportSiteHealth(SiteHealthReportDto) returns (SiteHealthReportAckDto);
// Application heartbeat. Returns Empty because the message is
// fire-and-forget: nothing on the site consumes a reply, and a failure here
// must never surface as a fault on the heartbeat timer path.
rpc Heartbeat(HeartbeatDto) returns (google.protobuf.Empty);
}
// ---------------------------------------------------------------------------
// Notification Outbox (#21)
// ---------------------------------------------------------------------------
// Site -> Central: submit a buffered notification for central delivery.
// Mirrors Commons NotificationSubmit.
message NotificationSubmitDto {
string notification_id = 1; // GUID string, the idempotency key
string list_name = 2;
string subject = 3;
string body = 4;
string source_site_id = 5;
string source_instance_id = 6; // empty string represents null
string source_script = 7; // empty string represents null
google.protobuf.Timestamp site_enqueued_at = 8;
string origin_execution_id = 9; // GUID string; empty represents null
string origin_parent_execution_id = 10; // GUID string; empty represents null
string source_node = 11; // empty string represents null
}
// Central -> Site: ack sent after the Notifications row is persisted.
message NotificationSubmitAckDto {
string notification_id = 1;
bool accepted = 2;
string error = 3; // empty string represents null
}
// Site -> Central: Notify.Status(id) lookup against the central outbox.
message NotificationStatusQueryDto {
string correlation_id = 1;
string notification_id = 2;
}
// Central -> Site: current central delivery state for a queried notification.
message NotificationStatusResponseDto {
string correlation_id = 1;
bool found = 2;
string status = 3;
int32 retry_count = 4;
string last_error = 5; // empty string represents null
google.protobuf.Timestamp delivered_at = 6; // absent when null
}
// ---------------------------------------------------------------------------
// Startup reconciliation (Deployment Manager)
// ---------------------------------------------------------------------------
// Site -> Central: the node's local deployed inventory at startup.
message ReconcileSiteRequestDto {
string site_identifier = 1;
string node_id = 2;
// Instance unique name -> revision hash of the config the node currently holds.
map<string, string> local_name_to_revision_hash = 3;
}
// Central -> Site: the gap the node must (re)fetch, plus orphans to log.
message ReconcileSiteResponseDto {
repeated ReconcileGapItemDto gap = 1;
repeated string orphan_names = 2;
string central_fetch_base_url = 3;
}
// One instance the node must (re)fetch, with a freshly-minted short-TTL token.
message ReconcileGapItemDto {
string instance_unique_name = 1;
string deployment_id = 2;
string revision_hash = 3;
bool is_enabled = 4;
string fetch_token = 5;
}
// ---------------------------------------------------------------------------
// Health Monitoring (#11)
// ---------------------------------------------------------------------------
// Wire form of the Commons ConnectionHealth enum.
//
// CONNECTION_HEALTH_UNSPECIFIED exists only to keep the proto3 zero value from
// meaning something. Mapping Connected onto 0 would make an absent/garbled
// value decode as "healthy", which is precisely the wrong direction to fail;
// the mapper decodes UNSPECIFIED as Error instead and never emits it.
enum ConnectionHealthEnum {
CONNECTION_HEALTH_UNSPECIFIED = 0;
CONNECTION_HEALTH_CONNECTED = 1;
CONNECTION_HEALTH_DISCONNECTED = 2;
CONNECTION_HEALTH_CONNECTING = 3;
CONNECTION_HEALTH_ERROR = 4;
}
message TagResolutionStatusDto {
int32 total_subscribed = 1;
int32 successfully_resolved = 2;
}
message TagQualityCountsDto {
int32 good = 1;
int32 bad = 2;
int32 uncertain = 3;
}
message NodeStatusDto {
string hostname = 1;
bool is_online = 2;
string role = 3;
}
// Point-in-time snapshot of the site-local SQLite audit queue.
message SiteAuditBacklogSnapshotDto {
int32 pending_count = 1;
google.protobuf.Timestamp oldest_pending_utc = 2; // absent when the queue is empty
int64 on_disk_bytes = 3;
}
// The three collection wrappers below exist so null and empty stay
// distinguishable. proto3 cannot express presence on a `repeated` or `map`
// field an unset one and an empty one are the same bytes but the
// corresponding SiteHealthReport members are genuinely nullable
// (SiteHealthCollector emits `ClusterNodes: _clusterNodes?.ToList()`), and the
// central health surface reads null as "this producer doesn't report the
// signal" rather than "the signal is empty". Wrapping in a message restores
// message presence and makes the distinction survive the round-trip.
message ConnectionEndpointMapDto {
map<string, string> entries = 1;
}
message TagQualityMapDto {
map<string, TagQualityCountsDto> entries = 1;
}
message NodeStatusListDto {
repeated NodeStatusDto nodes = 1;
}
// Site -> Central: periodic site health report. Mirrors Commons SiteHealthReport.
// Additive-only evolution: field numbers are never reused.
message SiteHealthReportDto {
string site_id = 1;
int64 sequence_number = 2;
google.protobuf.Timestamp report_timestamp = 3;
map<string, ConnectionHealthEnum> data_connection_statuses = 4;
map<string, TagResolutionStatusDto> tag_resolution_counts = 5;
int32 script_error_count = 6;
int32 alarm_evaluation_error_count = 7;
map<string, int32> store_and_forward_buffer_depths = 8;
int32 dead_letter_count = 9;
int32 deployed_instance_count = 10;
int32 enabled_instance_count = 11;
int32 disabled_instance_count = 12;
string node_role = 13;
string node_hostname = 14;
ConnectionEndpointMapDto data_connection_endpoints = 15; // absent when null
TagQualityMapDto data_connection_tag_quality = 16; // absent when null
int32 parked_message_count = 17;
NodeStatusListDto cluster_nodes = 18; // absent when null
int32 site_audit_write_failures = 19;
int32 audit_redaction_failure = 20;
SiteAuditBacklogSnapshotDto site_audit_backlog = 21; // absent when no data yet
int64 site_event_log_write_failures = 22;
google.protobuf.DoubleValue oldest_parked_message_age_seconds = 23; // absent when nothing parked
int32 script_queue_depth = 24;
int32 script_busy_threads = 25;
google.protobuf.DoubleValue script_oldest_busy_age_seconds = 26; // absent when the pool is idle
// Nullable on purpose: absent means "replication not wired on this node",
// which is NOT the same as false ("wired but currently disconnected").
google.protobuf.BoolValue local_db_replication_connected = 27;
// Absent means UNKNOWN, never zero a failed backlog read rendered as 0
// would report a broken replication pair as perfectly healthy.
google.protobuf.Int64Value local_db_oplog_backlog = 28;
}
// Central -> Site: health report ack, so a lost report is observable.
message SiteHealthReportAckDto {
string site_id = 1;
int64 sequence_number = 2;
bool accepted = 3;
string error = 4; // empty string represents null
}
// Site -> Central: application heartbeat (fire-and-forget; reply is Empty).
message HeartbeatDto {
string site_id = 1;
string node_hostname = 2;
bool is_active = 3;
google.protobuf.Timestamp timestamp = 4;
}
@@ -1,5 +1,6 @@
using Microsoft.Extensions.DependencyInjection; using Microsoft.Extensions.DependencyInjection;
using Microsoft.Extensions.DependencyInjection.Extensions; using Microsoft.Extensions.DependencyInjection.Extensions;
using Microsoft.Extensions.Logging;
using Microsoft.Extensions.Options; using Microsoft.Extensions.Options;
using ZB.MOM.WW.ScadaBridge.Communication.Grpc; using ZB.MOM.WW.ScadaBridge.Communication.Grpc;
@@ -23,7 +24,15 @@ public static class ServiceCollectionExtensions
ServiceDescriptor.Singleton<IValidateOptions<CommunicationOptions>, CommunicationOptionsValidator>()); ServiceDescriptor.Singleton<IValidateOptions<CommunicationOptions>, CommunicationOptionsValidator>());
services.AddSingleton<CommunicationService>(); services.AddSingleton<CommunicationService>();
services.AddSingleton<SiteStreamGrpcClientFactory>();
// Explicit factory rather than AddSingleton<T>(): the ISitePskProvider dependency is
// optional (central registers one, a site node does not), and constructor selection
// over a nullable interface parameter is exactly the case the container cannot decide
// for itself — GetService returns null cleanly where constructor injection would throw.
services.AddSingleton(sp => new SiteStreamGrpcClientFactory(
sp.GetRequiredService<ILoggerFactory>(),
sp.GetRequiredService<IOptions<CommunicationOptions>>(),
sp.GetService<ISitePskProvider>()));
services.AddSingleton<DebugStreamService>(); services.AddSingleton<DebugStreamService>();
// Aggregated live alarm cache (plan #10, Task 4): transient, in-memory, shared // Aggregated live alarm cache (plan #10, Task 4): transient, in-memory, shared
@@ -32,20 +32,30 @@
<ProjectReference Include="../ZB.MOM.WW.ScadaBridge.HealthMonitoring/ZB.MOM.WW.ScadaBridge.HealthMonitoring.csproj" /> <ProjectReference Include="../ZB.MOM.WW.ScadaBridge.HealthMonitoring/ZB.MOM.WW.ScadaBridge.HealthMonitoring.csproj" />
</ItemGroup> </ItemGroup>
<!-- gRPC proto generation. The compiled C# is checked in under <!-- gRPC proto generation. The compiled C# is checked in — SiteStreamGrpc/
SiteStreamGrpc/ (Sitestream.cs + SitestreamGrpc.cs) because protoc (Sitestream.cs + SitestreamGrpc.cs) for Protos/sitestream.proto, and
segfaults inside our linux_arm64 Docker build image. To regenerate CentralControlGrpc/ (CentralControl.cs + CentralControlGrpc.cs) for
after schema changes: Protos/central_control.proto — because protoc segfaults inside our
1. Temporarily uncomment the Protobuf ItemGroup below. linux_arm64 Docker build image. To regenerate after schema changes run
2. Delete SiteStreamGrpc/*.cs. `docker/regen-proto.sh [sitestream|centralcontrol|all]`, which does all
of the following and always leaves this file as it found it:
1. Temporarily uncomment the Protobuf ItemGroup below (just the line
for the proto you changed — the other file's checked-in C# is
already compiled, so enabling both at once duplicates types).
2. Delete the matching checked-in *.cs.
3. `dotnet build` (on macOS) — Grpc.Tools writes fresh files to obj/. 3. `dotnet build` (on macOS) — Grpc.Tools writes fresh files to obj/.
4. Copy obj/Debug/net10.0/Protos/*.cs into SiteStreamGrpc/. 4. Copy obj/Debug/net10.0/Protos/*.cs into the matching folder.
5. Re-comment the ItemGroup. 5. Re-comment the ItemGroup.
Eventually we should switch the Docker build image to one with a central_control.proto imports sitestream.proto, so protoc resolves it
working protoc on arm64. --> from the project-relative path without sitestream.proto needing its own
Protobuf item.
An ACTIVE Protobuf item must never be committed — it breaks the Docker
image build. Eventually we should switch the Docker build image to one
with a working protoc on arm64. -->
<!-- <!--
<ItemGroup> <ItemGroup>
<Protobuf Include="Protos\sitestream.proto" GrpcServices="Both" /> <Protobuf Include="Protos\sitestream.proto" GrpcServices="Both" />
<Protobuf Include="Protos\central_control.proto" GrpcServices="Both" />
</ItemGroup> </ItemGroup>
--> -->
@@ -8,6 +8,7 @@ using Microsoft.Extensions.Options;
using ZB.MOM.WW.ScadaBridge.ClusterInfrastructure; using ZB.MOM.WW.ScadaBridge.ClusterInfrastructure;
using ZB.MOM.WW.ScadaBridge.Communication; using ZB.MOM.WW.ScadaBridge.Communication;
using ZB.MOM.WW.ScadaBridge.Communication.Actors; using ZB.MOM.WW.ScadaBridge.Communication.Actors;
using ZB.MOM.WW.ScadaBridge.Communication.Grpc;
using ZB.MOM.WW.ScadaBridge.Host.Actors; using ZB.MOM.WW.ScadaBridge.Host.Actors;
using ZB.MOM.WW.ScadaBridge.SiteRuntime; using ZB.MOM.WW.ScadaBridge.SiteRuntime;
using ZB.MOM.WW.ScadaBridge.SiteRuntime.Actors; using ZB.MOM.WW.ScadaBridge.SiteRuntime.Actors;
@@ -436,6 +437,19 @@ akka {{
ClusterClientReceptionist.Get(_actorSystem).RegisterService(centralCommActor); ClusterClientReceptionist.Get(_actorSystem).RegisterService(centralCommActor);
_logger.LogInformation("CentralCommunicationActor registered with ClusterClientReceptionist"); _logger.LogInformation("CentralCommunicationActor registered with ClusterClientReceptionist");
// Hand the same actor to the central-hosted gRPC control plane (T1A.2) and open its
// readiness gate — the gRPC face Asks this exact actor, so both transports resolve to
// one handler implementation. Mirrors SiteStreamGrpcServer.SetReady on the site side:
// the service is a DI singleton created before the actor system exists, so the actor
// arrives here post-construction. Null on a host that did not register the service
// (e.g. an in-process test harness), so the wiring is a guarded no-op there.
var centralControlGrpc = _serviceProvider
.GetService<ZB.MOM.WW.ScadaBridge.Communication.Grpc.CentralControlGrpcService>();
centralControlGrpc?.SetReady(centralCommActor);
_logger.LogInformation(
"CentralControlGrpcService readiness set (service bound: {Bound})",
centralControlGrpc is not null);
// Wire up the CommunicationService with the actor reference // Wire up the CommunicationService with the actor reference
var commService = _serviceProvider.GetService<CommunicationService>(); var commService = _serviceProvider.GetService<CommunicationService>();
commService?.SetCommunicationActor(centralCommActor); commService?.SetCommunicationActor(centralCommActor);
@@ -450,16 +464,22 @@ akka {{
siteAlarmLiveCache?.SetActorSystem(_actorSystem!); siteAlarmLiveCache?.SetActorSystem(_actorSystem!);
// Management Service — accessible via ClusterClient // Management Service — reached IN-PROCESS only, via ManagementActorHolder.
//
// This actor used to be registered with the ClusterClientReceptionist as well, for
// an out-of-cluster CLI that was never built (REQ-HOST-6a). The shipped CLI speaks
// HTTP Basic to /management, which asks this actor through the holder below
// (ManagementEndpoints), so the registration had no sender anywhere in the repo —
// it only advertised a management surface across the cluster-client boundary for
// free. Removed 2026-07-22 (ClusterClient→gRPC migration, T0.1).
var mgmtLogger = _serviceProvider.GetRequiredService<ILoggerFactory>() var mgmtLogger = _serviceProvider.GetRequiredService<ILoggerFactory>()
.CreateLogger<ZB.MOM.WW.ScadaBridge.ManagementService.ManagementActor>(); .CreateLogger<ZB.MOM.WW.ScadaBridge.ManagementService.ManagementActor>();
var mgmtActor = _actorSystem!.ActorOf( var mgmtActor = _actorSystem!.ActorOf(
Props.Create(() => new ZB.MOM.WW.ScadaBridge.ManagementService.ManagementActor(_serviceProvider, mgmtLogger)), Props.Create(() => new ZB.MOM.WW.ScadaBridge.ManagementService.ManagementActor(_serviceProvider, mgmtLogger)),
"management"); "management");
ClusterClientReceptionist.Get(_actorSystem).RegisterService(mgmtActor);
var mgmtHolder = _serviceProvider.GetRequiredService<ZB.MOM.WW.ScadaBridge.ManagementService.ManagementActorHolder>(); var mgmtHolder = _serviceProvider.GetRequiredService<ZB.MOM.WW.ScadaBridge.ManagementService.ManagementActorHolder>();
mgmtHolder.ActorRef = mgmtActor; mgmtHolder.ActorRef = mgmtActor;
_logger.LogInformation("ManagementActor registered with ClusterClientReceptionist"); _logger.LogInformation("ManagementActor started at /user/management (in-process access via ManagementActorHolder)");
// Notification Outbox — cluster singleton so exactly one node owns ingest, // Notification Outbox — cluster singleton so exactly one node owns ingest,
// the dispatch sweep and the purge loop. Central actors run on the base // the dispatch sweep and the purge loop. Central actors run on the base
@@ -813,13 +833,45 @@ akka {{
_logger, role: siteRole); _logger, role: siteRole);
var dmProxy = dm.Proxy; var dmProxy = dm.Proxy;
// Select the site→central transport behind the coexistence flag (default Akka
// ClusterClient). When gRPC is chosen the site dials CentralControlService directly with
// a sticky-failover channel pair, presenting its own preshared key; the ClusterClient
// below is then not created at all.
ICentralTransport? centralTransport = null;
if (_communicationOptions.CentralTransport == CentralTransportMode.Grpc)
{
var loggerFactory = _serviceProvider.GetRequiredService<ILoggerFactory>();
var channelProvider = new CentralChannelProvider(
_communicationOptions.CentralGrpcEndpoints,
new StaticSitePskProvider(_communicationOptions.GrpcPsk),
_nodeOptions.SiteId!,
_communicationOptions,
loggerFactory.CreateLogger<CentralChannelProvider>());
_trackedDisposables.Add(channelProvider);
centralTransport = new GrpcCentralTransport(
channelProvider,
_communicationOptions,
loggerFactory.CreateLogger<GrpcCentralTransport>());
_logger.LogInformation(
"Site→central transport: gRPC to {Count} central endpoint(s) for site {SiteId}",
_communicationOptions.CentralGrpcEndpoints.Count, _nodeOptions.SiteId);
}
else
{
_logger.LogInformation(
"Site→central transport: Akka ClusterClient (default) for site {SiteId}",
_nodeOptions.SiteId);
}
// Create SiteCommunicationActor for receiving messages from central // Create SiteCommunicationActor for receiving messages from central
var siteCommActor = _actorSystem.ActorOf( var siteCommActor = _actorSystem.ActorOf(
Props.Create(() => new SiteCommunicationActor( Props.Create(() => new SiteCommunicationActor(
_nodeOptions.SiteId!, _nodeOptions.SiteId!,
_communicationOptions, _communicationOptions,
dmProxy, dmProxy,
activeNodeCheck)), activeNodeCheck,
null,
centralTransport)),
"site-communication"); "site-communication");
// Register local handlers with SiteCommunicationActor // Register local handlers with SiteCommunicationActor
@@ -938,8 +990,12 @@ akka {{
"Site actors registered. DeploymentManager singleton scoped to role={SiteRole}, SiteCommunicationActor created.", "Site actors registered. DeploymentManager singleton scoped to role={SiteRole}, SiteCommunicationActor created.",
siteRole); siteRole);
// Create ClusterClient to central if contact points are configured // Create ClusterClient to central if contact points are configured — but only on the Akka
if (_communicationOptions.CentralContactPoints.Count > 0) // transport. On the gRPC transport the SiteCommunicationActor already holds a
// GrpcCentralTransport and never receives RegisterCentralClient, so a ClusterClient here
// would be dead weight (and keep an unwanted cross-cluster Akka association alive).
if (_communicationOptions.CentralTransport == CentralTransportMode.Akka
&& _communicationOptions.CentralContactPoints.Count > 0)
{ {
var contacts = _communicationOptions.CentralContactPoints var contacts = _communicationOptions.CentralContactPoints
.Select(cp => ActorPath.Parse($"{cp}/system/receptionist")) .Select(cp => ActorPath.Parse($"{cp}/system/receptionist"))
@@ -0,0 +1,245 @@
using System.Security.Cryptography;
using System.Text;
using Grpc.Core;
using Grpc.Core.Interceptors;
using Microsoft.Extensions.Logging;
using ZB.MOM.WW.ScadaBridge.Communication.Grpc;
namespace ZB.MOM.WW.ScadaBridge.Host;
/// <summary>
/// Gates the central-hosted gRPC control plane (<c>CentralControlService</c>) with each site's
/// preshared key. The sibling of <see cref="ControlPlaneAuthInterceptor"/>, but with the
/// verification model inverted: a site checks one bearer token against its own single key, whereas
/// central must check the presented token against the key belonging to the SITE that sent it.
/// </summary>
/// <remarks>
/// <para>
/// <b>Why a separate class rather than a second constructor on
/// <see cref="ControlPlaneAuthInterceptor"/>.</b> <c>Grpc.AspNetCore</c> registers a
/// type-registered interceptor through <c>InterceptorRegistration.GetFactory()</c>, which throws
/// <c>"Multiple constructors accepting all given argument types have been found"</c> the moment a
/// second public constructor is applicable. That throw lands inside the pipeline on every call and
/// surfaces as <c>Unknown / "Exception was thrown by handler"</c> — the node boots healthy and
/// every gated call dies looking like a handler bug, with correct/wrong/no key all producing the
/// identical error. It shipped once with a fully green suite and was caught only on the rig.
/// Central's model genuinely differs (per-site key by header, not the site's one own-key), so it
/// gets its own class with its own single public constructor rather than a variant ctor on the
/// site interceptor. Both classes are pinned by a reflection test asserting exactly one public
/// constructor.
/// </para>
/// <para>
/// <b>Fail-closed on every branch.</b> A gated call is refused with
/// <see cref="StatusCode.PermissionDenied"/> when: the <c>x-scadabridge-site</c> header is
/// missing or blank; no key can be resolved for that site (<see cref="ISitePskProvider"/> throws);
/// or the presented bearer token does not match. There is no pass-through — an unresolvable or
/// absent identity never degrades to "let it in". Non-gated services (should any share the
/// listener) return immediately, matching the site interceptor's shape.
/// </para>
/// <para>
/// The comparison is <see cref="CryptographicOperations.FixedTimeEquals"/> over UTF-8, the same
/// constant-time compare the site interceptor and LocalDb sync use.
/// </para>
/// </remarks>
public sealed class CentralControlAuthInterceptor : Interceptor
{
/// <summary>
/// Service prefixes gated by default — the one central-hosted control-plane service. Taken
/// from the generated <c>package scadabridge.centralcontrol.v1; service CentralControlService</c>.
/// </summary>
public static readonly IReadOnlyList<string> DefaultGatedPrefixes =
new[] { $"/{CentralControlService.Descriptor.FullName}/" };
private readonly IReadOnlyList<string> _gatedPrefixes;
private readonly ISitePskProvider _pskProvider;
private readonly ILogger<CentralControlAuthInterceptor> _logger;
/// <summary>
/// Creates the interceptor gating <see cref="DefaultGatedPrefixes"/>.
/// </summary>
/// <remarks>
/// <b>This must remain the ONLY public constructor</b> — see the class remarks for why a
/// second one silently disables the gate. Pinned by
/// <c>CentralControlAuthInterceptorTests.TheInterceptorHasExactlyOnePublicConstructor</c>.
/// </remarks>
/// <param name="pskProvider">Resolves each site's preshared key.</param>
/// <param name="logger">Logger for denial diagnostics.</param>
public CentralControlAuthInterceptor(
ISitePskProvider pskProvider,
ILogger<CentralControlAuthInterceptor> logger)
: this(pskProvider, logger, DefaultGatedPrefixes)
{
}
/// <summary>
/// Creates the interceptor gating an explicit prefix set. <b>Internal</b> — a public second
/// constructor would reintroduce the ambiguous-constructor defect described on the class.
/// </summary>
/// <param name="pskProvider">Resolves each site's preshared key.</param>
/// <param name="logger">Logger for denial diagnostics.</param>
/// <param name="gatedPrefixes">Method-path prefixes to gate.</param>
internal CentralControlAuthInterceptor(
ISitePskProvider pskProvider,
ILogger<CentralControlAuthInterceptor> logger,
IReadOnlyList<string> gatedPrefixes)
{
ArgumentNullException.ThrowIfNull(pskProvider);
ArgumentNullException.ThrowIfNull(logger);
ArgumentNullException.ThrowIfNull(gatedPrefixes);
_pskProvider = pskProvider;
_logger = logger;
_gatedPrefixes = gatedPrefixes;
}
/// <inheritdoc />
public override async Task<TResponse> UnaryServerHandler<TRequest, TResponse>(
TRequest request,
ServerCallContext context,
UnaryServerMethod<TRequest, TResponse> continuation)
{
await AuthorizeAsync(context).ConfigureAwait(false);
return await continuation(request, context).ConfigureAwait(false);
}
/// <inheritdoc />
public override async Task DuplexStreamingServerHandler<TRequest, TResponse>(
IAsyncStreamReader<TRequest> requestStream,
IServerStreamWriter<TResponse> responseStream,
ServerCallContext context,
DuplexStreamingServerMethod<TRequest, TResponse> continuation)
{
await AuthorizeAsync(context).ConfigureAwait(false);
await continuation(requestStream, responseStream, context).ConfigureAwait(false);
}
/// <inheritdoc />
public override async Task<TResponse> ClientStreamingServerHandler<TRequest, TResponse>(
IAsyncStreamReader<TRequest> requestStream,
ServerCallContext context,
ClientStreamingServerMethod<TRequest, TResponse> continuation)
{
await AuthorizeAsync(context).ConfigureAwait(false);
return await continuation(requestStream, context).ConfigureAwait(false);
}
/// <inheritdoc />
public override async Task ServerStreamingServerHandler<TRequest, TResponse>(
TRequest request,
IServerStreamWriter<TResponse> responseStream,
ServerCallContext context,
ServerStreamingServerMethod<TRequest, TResponse> continuation)
{
await AuthorizeAsync(context).ConfigureAwait(false);
await continuation(request, responseStream, context).ConfigureAwait(false);
}
/// <summary>
/// Throws <see cref="RpcException"/> with <see cref="StatusCode.PermissionDenied"/> unless a
/// gated call carries a valid <c>x-scadabridge-site</c> header AND a bearer token matching
/// that site's resolved key. Non-gated calls return immediately.
/// </summary>
private async Task AuthorizeAsync(ServerCallContext context)
{
if (!IsGated(context.Method))
{
return;
}
var siteId = ExtractSiteId(context.RequestHeaders);
if (string.IsNullOrWhiteSpace(siteId))
{
_logger.LogWarning(
"Rejected a central control-plane call to {Method}: the required "
+ "'{Header}' metadata header is missing or blank, so there is no per-site key to "
+ "verify against.",
context.Method, ControlPlaneCredentials.SiteHeader);
throw Denied("missing site identity header");
}
string expected;
try
{
expected = await _pskProvider.GetAsync(siteId, context.CancellationToken)
.ConfigureAwait(false);
}
catch (OperationCanceledException)
{
throw;
}
catch (Exception ex)
{
// Fail-closed: an unresolvable key is a denial, never a pass-through. The provider
// has already logged the specific cause (missing secret / missing config entry).
_logger.LogWarning(ex,
"Rejected a central control-plane call to {Method}: no preshared key could be "
+ "resolved for site {SiteId}.",
context.Method, siteId);
throw Denied("no key configured for the presented site");
}
var presented = ExtractBearerToken(context.RequestHeaders);
if (presented is null || !FixedTimeEquals(presented, expected))
{
_logger.LogWarning(
"Rejected a central control-plane call to {Method} from site {SiteId}: {Reason}.",
context.Method, siteId,
presented is null ? "no bearer token presented" : "bearer token did not match");
throw Denied("control plane authentication failed");
}
}
private static RpcException Denied(string reason)
=> new(new Status(StatusCode.PermissionDenied, $"Control plane authentication failed: {reason}."));
private bool IsGated(string method)
{
foreach (var prefix in _gatedPrefixes)
{
if (method.StartsWith(prefix, StringComparison.Ordinal))
{
return true;
}
}
return false;
}
private static string? ExtractSiteId(Metadata headers)
{
foreach (var entry in headers)
{
if (string.Equals(entry.Key, ControlPlaneCredentials.SiteHeader,
StringComparison.OrdinalIgnoreCase))
{
return entry.Value;
}
}
return null;
}
private static string? ExtractBearerToken(Metadata headers)
{
// gRPC lowercases header keys on the wire; compare case-insensitively so a hand-built
// Metadata in a test behaves the same as a real request.
foreach (var entry in headers)
{
if (!string.Equals(entry.Key, ControlPlaneCredentials.AuthorizationHeader,
StringComparison.OrdinalIgnoreCase))
{
continue;
}
var value = entry.Value;
if (value is not null && value.StartsWith("Bearer ", StringComparison.OrdinalIgnoreCase))
{
return value["Bearer ".Length..];
}
}
return null;
}
private static bool FixedTimeEquals(string presented, string expected)
=> CryptographicOperations.FixedTimeEquals(
Encoding.UTF8.GetBytes(presented), Encoding.UTF8.GetBytes(expected));
}
@@ -0,0 +1,228 @@
using System.Security.Cryptography;
using System.Text;
using Grpc.Core;
using Grpc.Core.Interceptors;
using Microsoft.Extensions.Logging;
using Microsoft.Extensions.Options;
using ZB.MOM.WW.ScadaBridge.Communication;
using ZB.MOM.WW.ScadaBridge.Communication.Grpc;
namespace ZB.MOM.WW.ScadaBridge.Host;
/// <summary>
/// Gates the site↔central gRPC control plane with a preshared key.
/// </summary>
/// <remarks>
/// <para>
/// <b>The gap this closes.</b> <c>SiteStreamService</c> shipped with no authentication at all —
/// plaintext h2c, no interceptor. Anything that could reach a site node's gRPC port could open a
/// live data stream or call <c>PullAuditEvents</c>/<c>PullSiteCalls</c> and read audit rows back.
/// The only gated surface on that listener was LocalDb sync, and only for its own service. That
/// gap exists independently of the ClusterClient→gRPC migration; it becomes indefensible once
/// every command crosses this listener.
/// </para>
/// <para>
/// <b>Modeled on <see cref="LocalDbSyncAuthInterceptor"/>,</b> deliberately: same four server
/// handlers funnelling into one <c>Authorize</c>, same <c>authorization: Bearer</c> extraction,
/// same <see cref="CryptographicOperations.FixedTimeEquals"/> comparison, same fail-closed
/// posture, same <see cref="StatusCode.PermissionDenied"/> rejection. Two differences:
/// </para>
/// <list type="number">
/// <item>It gates a <b>set</b> of service prefixes rather than one, so later phases can add the
/// new command/control services without a second interceptor.</item>
/// <item>Its expected key comes from <see cref="CommunicationOptions.GrpcPsk"/> — the site's own
/// key, supplied in production as <c>${secret:SB-GRPC-PSK-&lt;siteId&gt;}</c> and expanded before
/// the host is built.</item>
/// </list>
/// <para>
/// <b>The two keys are separate on purpose.</b> LocalDb sync keeps its own
/// <c>LocalDb:Replication:ApiKey</c>, which authenticates a different peer (the pair partner, not
/// central) over a different trust relationship. Sharing one key would mean a site's central-facing
/// key also admits writes into its database.
/// </para>
/// <para>
/// <b>Fail-closed, and not optional.</b> With no <c>GrpcPsk</c> configured, every gated call is
/// rejected — including the ones that work today. That is a deliberate break: LocalDb replication
/// is an opt-in feature whose "off" state is "no peer", whereas streaming and audit pull are
/// core paths, so "no key" must not silently mean "no authentication". Every environment must
/// carry a key before upgrading to this build.
/// </para>
/// </remarks>
public sealed class ControlPlaneAuthInterceptor : Interceptor
{
/// <summary>
/// Service prefixes gated by default. Read from the generated <c>sitestream.proto</c>
/// package/service names — <c>package sitestream; service SiteStreamService</c>.
/// Later phases append their own services here.
/// </summary>
public static readonly IReadOnlyList<string> DefaultGatedPrefixes =
new[] { "/sitestream.SiteStreamService/" };
private readonly IReadOnlyList<string> _gatedPrefixes;
private readonly IOptions<CommunicationOptions> _options;
private readonly ILogger<ControlPlaneAuthInterceptor> _logger;
/// <summary>
/// Creates the interceptor gating <see cref="DefaultGatedPrefixes"/>.
/// </summary>
/// <remarks>
/// <b>This must remain the ONLY public constructor.</b> <c>AddGrpc</c> registers the
/// interceptor by type, and <c>Grpc.AspNetCore.Server.InterceptorRegistration.GetFactory()</c>
/// throws <c>"Multiple constructors accepting all given argument types have been found"</c>
/// when a second one is applicable. That throw happens per call, inside the pipeline, and
/// surfaces to the caller as <c>Unknown / "Exception was thrown by handler"</c> — so the gate
/// silently stops authorizing anything while still failing every call. A second public
/// constructor added here in a later phase reintroduces exactly that. Pinned by
/// <c>ControlPlaneAuthInterceptorTests.TheInterceptorHasExactlyOnePublicConstructor</c>.
/// </remarks>
/// <param name="options">Communication options; <c>GrpcPsk</c> is the expected bearer token.</param>
/// <param name="logger">Logger for denial diagnostics.</param>
public ControlPlaneAuthInterceptor(
IOptions<CommunicationOptions> options,
ILogger<ControlPlaneAuthInterceptor> logger)
: this(options, logger, DefaultGatedPrefixes)
{
}
/// <summary>
/// Creates the interceptor gating an explicit set of service prefixes. <b>Internal</b> —
/// see the public constructor's remarks for why this cannot be public. Phases that add a
/// service to the gate should extend <see cref="DefaultGatedPrefixes"/> rather than reach
/// for a second registration shape.
/// </summary>
/// <param name="options">Communication options; <c>GrpcPsk</c> is the expected bearer token.</param>
/// <param name="logger">Logger for denial diagnostics.</param>
/// <param name="gatedPrefixes">Method-path prefixes to gate, e.g. <c>/sitestream.SiteStreamService/</c>.</param>
internal ControlPlaneAuthInterceptor(
IOptions<CommunicationOptions> options,
ILogger<ControlPlaneAuthInterceptor> logger,
IReadOnlyList<string> gatedPrefixes)
{
ArgumentNullException.ThrowIfNull(options);
ArgumentNullException.ThrowIfNull(logger);
ArgumentNullException.ThrowIfNull(gatedPrefixes);
_options = options;
_logger = logger;
_gatedPrefixes = gatedPrefixes;
}
/// <inheritdoc />
public override Task<TResponse> UnaryServerHandler<TRequest, TResponse>(
TRequest request,
ServerCallContext context,
UnaryServerMethod<TRequest, TResponse> continuation)
{
Authorize(context);
return continuation(request, context);
}
/// <inheritdoc />
public override Task DuplexStreamingServerHandler<TRequest, TResponse>(
IAsyncStreamReader<TRequest> requestStream,
IServerStreamWriter<TResponse> responseStream,
ServerCallContext context,
DuplexStreamingServerMethod<TRequest, TResponse> continuation)
{
Authorize(context);
return continuation(requestStream, responseStream, context);
}
/// <inheritdoc />
public override Task<TResponse> ClientStreamingServerHandler<TRequest, TResponse>(
IAsyncStreamReader<TRequest> requestStream,
ServerCallContext context,
ClientStreamingServerMethod<TRequest, TResponse> continuation)
{
Authorize(context);
return continuation(requestStream, context);
}
/// <inheritdoc />
public override Task ServerStreamingServerHandler<TRequest, TResponse>(
TRequest request,
IServerStreamWriter<TResponse> responseStream,
ServerCallContext context,
ServerStreamingServerMethod<TRequest, TResponse> continuation)
{
Authorize(context);
return continuation(request, responseStream, context);
}
/// <summary>
/// Throws <see cref="RpcException"/> with <see cref="StatusCode.PermissionDenied"/> if this
/// is a gated call that does not carry the configured bearer token. Calls to services
/// outside <c>gatedPrefixes</c> — notably LocalDb sync, which has its own interceptor —
/// return immediately.
/// </summary>
private void Authorize(ServerCallContext context)
{
if (!IsGated(context.Method))
{
return;
}
var expected = _options.Value.GrpcPsk;
if (string.IsNullOrEmpty(expected))
{
_logger.LogWarning(
"Rejected a control-plane call to {Method}: no ScadaBridge:Communication:GrpcPsk is "
+ "configured, so the control plane is closed. Set the same key here (in production, "
+ "as ${{secret:SB-GRPC-PSK-<siteId>}}) and in central's secret store.",
context.Method);
throw new RpcException(new Status(
StatusCode.PermissionDenied,
"Control plane is not accepting calls: no preshared key is configured on this node."));
}
var presented = ExtractBearerToken(context.RequestHeaders);
if (presented is null || !FixedTimeEquals(presented, expected))
{
_logger.LogWarning(
"Rejected a control-plane call to {Method}: {Reason}.",
context.Method,
presented is null ? "no bearer token presented" : "bearer token did not match");
throw new RpcException(new Status(
StatusCode.PermissionDenied,
"Control plane authentication failed."));
}
}
private bool IsGated(string method)
{
foreach (var prefix in _gatedPrefixes)
{
if (method.StartsWith(prefix, StringComparison.Ordinal))
{
return true;
}
}
return false;
}
private static string? ExtractBearerToken(Metadata headers)
{
// gRPC lowercases header keys on the wire; compare case-insensitively anyway so a
// hand-built Metadata in a test behaves the same as a real request.
foreach (var entry in headers)
{
if (!string.Equals(entry.Key, ControlPlaneCredentials.AuthorizationHeader,
StringComparison.OrdinalIgnoreCase))
{
continue;
}
var value = entry.Value;
if (value is not null && value.StartsWith("Bearer ", StringComparison.OrdinalIgnoreCase))
{
return value["Bearer ".Length..];
}
}
return null;
}
private static bool FixedTimeEquals(string presented, string expected)
=> CryptographicOperations.FixedTimeEquals(
Encoding.UTF8.GetBytes(presented), Encoding.UTF8.GetBytes(expected));
}
@@ -21,6 +21,15 @@ public class NodeOptions
/// <summary>Gets or sets the gRPC port for the site stream server.</summary> /// <summary>Gets or sets the gRPC port for the site stream server.</summary>
public int GrpcPort { get; set; } = 8083; public int GrpcPort { get; set; } = 8083;
/// <summary> /// <summary>
/// HTTP/2 (h2c) port the CENTRAL node listens on for the site→central
/// <c>CentralControlService</c> gRPC control plane. Default 8083 — deliberately symmetric
/// with the site <see cref="GrpcPort"/>, since a node is either central or a site and the
/// two never share a process. This listener is distinct from central's <c>:5000</c> HTTP/1
/// surface (Central UI, Management/Inbound API), which stays exactly as-is: gRPC does NOT go
/// through Traefik (HTTP/1 only). Ignored on site nodes.
/// </summary>
public int CentralGrpcPort { get; set; } = 8083;
/// <summary>
/// HTTP/1.1 port serving the Prometheus /metrics scrape endpoint on site nodes. /// HTTP/1.1 port serving the Prometheus /metrics scrape endpoint on site nodes.
/// Defaults to 8084 — deliberately distinct from <see cref="RemotingPort"/> (8082) /// Defaults to 8084 — deliberately distinct from <see cref="RemotingPort"/> (8082)
/// and <see cref="GrpcPort"/> (8083) so the Kestrel metrics listener never contends /// and <see cref="GrpcPort"/> (8083) so the Kestrel metrics listener never contends
@@ -23,6 +23,7 @@ public sealed class NodeOptionsValidator : OptionsValidatorBase<NodeOptions>
RequirePort(builder, options.RemotingPort, nameof(NodeOptions.RemotingPort)); RequirePort(builder, options.RemotingPort, nameof(NodeOptions.RemotingPort));
RequirePort(builder, options.GrpcPort, nameof(NodeOptions.GrpcPort)); RequirePort(builder, options.GrpcPort, nameof(NodeOptions.GrpcPort));
RequirePort(builder, options.MetricsPort, nameof(NodeOptions.MetricsPort)); RequirePort(builder, options.MetricsPort, nameof(NodeOptions.MetricsPort));
RequirePort(builder, options.CentralGrpcPort, nameof(NodeOptions.CentralGrpcPort));
} }
// 0 stays valid (dynamic-port request); reject only out-of-TCP-range values. // 0 stays valid (dynamic-port request); reject only out-of-TCP-range values.
+142 -6
View File
@@ -97,9 +97,76 @@ try
// Windows Service support (no-op when not running as a Windows Service) // Windows Service support (no-op when not running as a Windows Service)
builder.Host.UseWindowsService(); builder.Host.UseWindowsService();
// Explicit Kestrel h2c listener for the central-hosted gRPC control plane
// (CentralControlService): HTTP/2-only, on its own port (default 8083, symmetric
// with the site GrpcPort — a node is either central or a site, never both). gRPC
// does NOT go through Traefik (HTTP/1 only); sites reach this port by container name.
//
// WARNING — the trap the rig caught (T1A.2 shipped it, fixed here): calling
// options.Listen*/ListenAnyIP puts Kestrel into EXPLICIT-ENDPOINTS mode, which
// SUPPRESSES the URLs from ASPNETCORE_URLS/--urls entirely — it is NOT additive.
// Central's ENTIRE HTTP/1 surface (Central UI, Management + Inbound API, and the
// /health/* endpoints Traefik + IActiveNodeGate depend on) lives on that URL
// (http://+:5000 on the rig, a different port in production). If we bind only the
// gRPC port, :5000 vanishes and central is reachable over gRPC while the UI, the
// management API and health checks are all dead — with no startup error. Unit tests
// use TestServer and never bind real Kestrel, so only a live node exposes this.
// The site branch has the same ConfigureKestrel shape but no ASPNETCORE_URLS surface
// to lose (it binds every port it needs — gRPC + metrics — explicitly). Central must
// therefore RE-BIND its HTTP port(s) here alongside the gRPC port.
var centralGrpcPort = configuration.GetValue<int>("ScadaBridge:Node:CentralGrpcPort", 8083);
// "urls" is WebHostDefaults.ServerUrlsKey — the host setting ASPNETCORE_URLS/--urls
// populate. Read it as a literal so this needs no extra Hosting using.
var httpUrls = configuration["ASPNETCORE_URLS"]
?? builder.WebHost.GetSetting("urls")
?? "http://+:5000";
var httpPorts = ParseHttpBindPorts(httpUrls);
builder.WebHost.ConfigureKestrel(options =>
{
// The HTTP/1.1 (+ HTTP/2) surface from the configured URLs, re-declared so it
// survives the explicit-endpoints switch above.
foreach (var httpPort in httpPorts)
{
options.ListenAnyIP(httpPort, listenOptions =>
{
listenOptions.Protocols = Microsoft.AspNetCore.Server.Kestrel.Core.HttpProtocols.Http1AndHttp2;
});
}
// The gRPC control plane, HTTP/2 h2c only, on its own port.
options.ListenAnyIP(centralGrpcPort, listenOptions =>
{
listenOptions.Protocols = Microsoft.AspNetCore.Server.Kestrel.Core.HttpProtocols.Http2;
});
});
// Shared components // Shared components
builder.Services.AddClusterInfrastructure(); builder.Services.AddClusterInfrastructure();
builder.Services.AddCommunication(); builder.Services.AddCommunication();
// Per-site gRPC preshared keys. Central-only: it is the side that dials sites, and
// the only side whose key set is dynamic (sites come from the configuration
// database, so there is no fixed list of ${secret:} references to expand at boot —
// hence a runtime resolver rather than the pre-host SecretReferenceExpander a site
// node uses for its single key). Registered before the clients that consume it.
builder.Services.AddSingleton<
ZB.MOM.WW.ScadaBridge.Communication.Grpc.ISitePskProvider, SitePskProvider>();
// Central-hosted gRPC control plane (T1A.2). CentralControlAuthInterceptor is the
// per-site sibling of the site's ControlPlaneAuthInterceptor: it verifies the Bearer
// token against the key for the site named in the required x-scadabridge-site header,
// resolved through the ISitePskProvider registered just above. Registered BY TYPE on
// AddGrpc exactly as the site branch registers its interceptors — NOT as a DI singleton,
// which would let DI hand the instance back and bypass Grpc.AspNetCore's own activation
// path (the shape that once hid a two-public-constructor defect until the rig caught it).
// CentralControlGrpcService decodes each request onto the same in-process message the
// ClusterClient path carries and Asks CentralCommunicationActor — zero handler logic here.
builder.Services.AddGrpc(options =>
{
options.Interceptors.Add<CentralControlAuthInterceptor>();
});
builder.Services.AddSingleton<
ZB.MOM.WW.ScadaBridge.Communication.Grpc.CentralControlGrpcService>();
builder.Services.AddHealthMonitoring(); builder.Services.AddHealthMonitoring();
builder.Services.AddCentralHealthAggregation(); builder.Services.AddCentralHealthAggregation();
builder.Services.AddExternalSystemGateway(); builder.Services.AddExternalSystemGateway();
@@ -454,6 +521,14 @@ try
// Requires endpoint routing (app.UseRouting() above). // Requires endpoint routing (app.UseRouting() above).
app.MapZbMetrics(); app.MapZbMetrics();
// Central-hosted gRPC control plane (T1A.2) — the site→central CentralControlService.
// Runs on the dedicated h2c listener configured above (default :8083), gated by
// CentralControlAuthInterceptor and readiness-gated by the service itself (Unavailable
// until AkkaHostedService hands the CentralCommunicationActor over via SetReady). It
// shares the app's endpoint routing with the HTTP/1 surface; Kestrel steers each
// connection to the right pipeline by listener/protocol.
app.MapGrpcService<ZB.MOM.WW.ScadaBridge.Communication.Grpc.CentralControlGrpcService>();
app.MapStaticAssets(); app.MapStaticAssets();
app.MapCentralUI<ZB.MOM.WW.ScadaBridge.Host.Components.App>(); app.MapCentralUI<ZB.MOM.WW.ScadaBridge.Host.Components.App>();
app.MapInboundAPI(); app.MapInboundAPI();
@@ -519,12 +594,21 @@ try
}); });
}); });
// gRPC server registration // gRPC server registration. Two interceptors, two disjoint service prefixes, two
// The interceptor gates ONLY /localdb_sync.v1.LocalDbSync/ — SiteStream calls on // separate keys — neither one's absence weakens the other:
// this same pipeline pass through untouched. It is fail-closed: with no //
// LocalDb:Replication:ApiKey configured, no sync stream is accepted at all. // LocalDbSyncAuthInterceptor gates /localdb_sync.v1.LocalDbSync/ (LocalDb:Replication:ApiKey)
// ControlPlaneAuthInterceptor gates /sitestream.SiteStreamService/ (ScadaBridge:Communication:GrpcPsk)
//
// Both are fail-closed: an unset key closes that surface rather than opening it. For
// LocalDb that means replication simply does not start; for the control plane it means
// this node serves no streams and no audit pulls until a key is configured, which is
// why every environment must carry one before running this build.
builder.Services.AddGrpc(options => builder.Services.AddGrpc(options =>
options.Interceptors.Add<LocalDbSyncAuthInterceptor>()); {
options.Interceptors.Add<LocalDbSyncAuthInterceptor>();
options.Interceptors.Add<ControlPlaneAuthInterceptor>();
});
builder.Services.AddSingleton<ZB.MOM.WW.ScadaBridge.Communication.Grpc.SiteStreamGrpcServer>(); builder.Services.AddSingleton<ZB.MOM.WW.ScadaBridge.Communication.Grpc.SiteStreamGrpcServer>();
// Existing site service registrations (this is also where LocalDb and its // Existing site service registrations (this is also where LocalDb and its
@@ -586,4 +670,56 @@ finally
/// <summary> /// <summary>
/// Exposes the auto-generated Program class for test infrastructure (e.g. WebApplicationFactory). /// Exposes the auto-generated Program class for test infrastructure (e.g. WebApplicationFactory).
/// </summary> /// </summary>
public partial class Program { } public partial class Program
{
/// <summary>
/// Extracts the distinct TCP ports from an ASP.NET Core server-URLs string (the
/// <c>ASPNETCORE_URLS</c> / <c>--urls</c> value, e.g. <c>"http://+:5000"</c> or a
/// semicolon-separated list). Used to re-declare central's HTTP surface after the gRPC
/// listener switches Kestrel into explicit-endpoints mode — see the call site's warning.
/// </summary>
/// <remarks>
/// Bind hosts (<c>+</c>, <c>*</c>, <c>0.0.0.0</c>, <c>[::]</c>, a hostname) are irrelevant
/// here because the caller re-binds via <c>ListenAnyIP</c>; only the port matters. A URL
/// with no explicit port falls back to the scheme default (80/443). Unparseable entries
/// are skipped rather than throwing — a bad URL should not take the node down at boot.
/// </remarks>
/// <param name="serverUrls">The server-URLs string; may be null/empty.</param>
/// <returns>The distinct ports, in first-seen order.</returns>
internal static IReadOnlyList<int> ParseHttpBindPorts(string? serverUrls)
{
var ports = new List<int>();
if (string.IsNullOrWhiteSpace(serverUrls))
{
return ports;
}
foreach (var raw in serverUrls.Split(';', StringSplitOptions.RemoveEmptyEntries | StringSplitOptions.TrimEntries))
{
int port;
// Uri can't parse the wildcard hosts Kestrel accepts (+, *), so normalize them
// to a placeholder host before parsing; the host is discarded anyway.
var normalized = raw.Replace("://+", "://placeholder").Replace("://*", "://placeholder");
if (Uri.TryCreate(normalized, UriKind.Absolute, out var uri))
{
port = uri.Port; // Uri fills the scheme default (80/443) when none is given.
}
else
{
// Last-ditch: pull the port after the final ':' (handles odd inputs Uri rejects).
var colon = raw.LastIndexOf(':');
if (colon < 0 || !int.TryParse(raw.AsSpan(colon + 1), out port))
{
continue;
}
}
if (port is > 0 and <= 65535 && !ports.Contains(port))
{
ports.Add(port);
}
}
return ports;
}
}
@@ -0,0 +1,149 @@
using System.Collections.Concurrent;
using Microsoft.Extensions.Logging;
using Microsoft.Extensions.Options;
using ZB.MOM.WW.ScadaBridge.Communication;
using ZB.MOM.WW.ScadaBridge.Communication.Grpc;
using ZB.MOM.WW.Secrets.Abstractions;
namespace ZB.MOM.WW.ScadaBridge.Host;
/// <summary>
/// Central's <see cref="ISitePskProvider"/>: resolves each site's gRPC preshared key at
/// channel-build time — from <c>ScadaBridge:Communication:SitePsks</c> if the site is listed
/// there, otherwise from the secrets store under the name <c>SB-GRPC-PSK-{siteId}</c>.
/// </summary>
/// <remarks>
/// <para>
/// <b>Why resolved at runtime rather than expanded into config at boot.</b> Site nodes get their
/// single key through the pre-host <c>SecretReferenceExpander</c> — a <c>${secret:…}</c> in
/// appsettings, resolved once before the host is built. Central cannot do that: its set of sites
/// comes from the configuration database and changes while the process runs, so there is no
/// fixed list of references to expand at boot. This mirrors the pattern the MxGateway data
/// connection already uses for its per-connection API keys.
/// </para>
/// <para>
/// <b>Caching.</b> Successful resolves are cached indefinitely; a site's key changes only on
/// rotation, and rotation restarts the pair. Failures are deliberately NOT cached, so a key
/// seeded after the first (failed) dial is picked up on the next attempt instead of requiring a
/// restart. <see cref="Invalidate"/> drops an entry when a site is removed or a key is rotated
/// in place.
/// </para>
/// <para>
/// <b>Fail-closed.</b> A key absent from BOTH sources throws, and the dial that needed it fails.
/// It never degrades to an unauthenticated call. The message names the exact config key and the
/// exact secret name, so the fix is one setting or one <c>secret</c> CLI seed.
/// </para>
/// </remarks>
public sealed class SitePskProvider : ISitePskProvider
{
/// <summary>Prefix of the per-site secret name; the site identifier is appended verbatim.</summary>
public const string SecretNamePrefix = "SB-GRPC-PSK-";
private readonly ConcurrentDictionary<string, string> _cache = new(StringComparer.Ordinal);
private readonly ISecretResolver _resolver;
private readonly IOptionsMonitor<CommunicationOptions> _options;
private readonly ILogger<SitePskProvider> _logger;
/// <summary>Creates the provider over the host's secret resolver.</summary>
/// <param name="resolver">Runtime secret resolver from the host container.</param>
/// <param name="options">
/// Communication options supplying the optional <c>SitePsks</c> map. Read through an
/// <see cref="IOptionsMonitor{TOptions}"/> so a reloaded configuration is honoured on the
/// next uncached resolve rather than pinned at construction.
/// </param>
/// <param name="logger">Logger for resolution diagnostics.</param>
public SitePskProvider(
ISecretResolver resolver,
IOptionsMonitor<CommunicationOptions> options,
ILogger<SitePskProvider> logger)
{
ArgumentNullException.ThrowIfNull(resolver);
ArgumentNullException.ThrowIfNull(options);
ArgumentNullException.ThrowIfNull(logger);
_resolver = resolver;
_options = options;
_logger = logger;
}
/// <summary>Builds the secret name for a site.</summary>
/// <param name="siteId">Site identifier.</param>
/// <returns>The secret name, e.g. <c>SB-GRPC-PSK-site-a</c>.</returns>
public static string SecretNameFor(string siteId) => SecretNamePrefix + siteId;
/// <inheritdoc />
public async ValueTask<string> GetAsync(string siteId, CancellationToken ct)
{
ArgumentException.ThrowIfNullOrWhiteSpace(siteId);
if (_cache.TryGetValue(siteId, out var cached))
{
return cached;
}
// Configured map first: it is the explicit, operator-stated answer for this site, and
// it is the only source available to a host running without a secrets master key.
var configured = _options.CurrentValue.SitePsks;
if (configured.TryGetValue(siteId, out var fromConfig) && !string.IsNullOrEmpty(fromConfig))
{
_cache[siteId] = fromConfig;
return fromConfig;
}
// Otherwise the store, which is the only source that can serve a site added at runtime.
var secretName = SecretNameFor(siteId);
var value = await ResolveFromStoreAsync(secretName, siteId, ct).ConfigureAwait(false);
if (string.IsNullOrEmpty(value))
{
_logger.LogError(
"gRPC control-plane key for site {SiteId} could not be resolved: it is absent from "
+ "ScadaBridge:Communication:SitePsks and secret '{SecretName}' is missing, empty or "
+ "tombstoned. Calls to this site are refused until one of the two is set (the site "
+ "node must carry the same value in ScadaBridge:Communication:GrpcPsk).",
siteId, secretName);
throw new InvalidOperationException(
$"gRPC preshared key for site '{siteId}' could not be resolved from "
+ $"ScadaBridge:Communication:SitePsks or secret '{secretName}'.");
}
// Two concurrent first dials may both resolve; the read is idempotent and the values
// identical, so the loser simply overwrites with the same string.
_cache[siteId] = value;
return value;
}
/// <summary>
/// Reads the key from the secrets store, treating a store fault as "not found" rather than
/// letting it propagate. A host with no master key configured (the development rig) throws
/// from the resolver; that must produce the same clear "key not configured" error as a
/// missing secret, not an opaque cryptographic one.
/// </summary>
private async Task<string?> ResolveFromStoreAsync(string secretName, string siteId, CancellationToken ct)
{
try
{
return await _resolver.GetAsync(new SecretName(secretName), ct).ConfigureAwait(false);
}
catch (OperationCanceledException)
{
throw;
}
catch (Exception ex)
{
_logger.LogDebug(ex,
"Secret store lookup for '{SecretName}' (site {SiteId}) faulted; treating as not found.",
secretName, siteId);
return null;
}
}
/// <inheritdoc />
public void Invalidate(string siteId)
{
if (!string.IsNullOrWhiteSpace(siteId))
{
_cache.TryRemove(siteId, out _);
}
}
}
@@ -126,6 +126,23 @@ public static class StartupValidator
p.Require("ScadaBridge:Node:MetricsPort", _ => metricsPort != port, "must differ from RemotingPort"); p.Require("ScadaBridge:Node:MetricsPort", _ => metricsPort != port, "must differ from RemotingPort");
p.Require("ScadaBridge:Node:MetricsPort", _ => metricsPort != grpcPort, "must differ from GrpcPort"); p.Require("ScadaBridge:Node:MetricsPort", _ => metricsPort != grpcPort, "must differ from GrpcPort");
// The gRPC control-plane preshared key. Same argument as the inbound
// API-key pepper above: without it the node boots and looks healthy, but
// ControlPlaneAuthInterceptor is fail-closed, so every SiteStream call —
// live subscriptions, audit pulls, cached-telemetry ingest — is refused
// with PermissionDenied. A silent, total loss of the site's central-facing
// surface is far worse than a loud boot failure, so require it here.
// Central holds the matching value per site in its secret store as
// SB-GRPC-PSK-{SiteId}; production supplies this one as
// ${secret:SB-GRPC-PSK-<siteId>}, expanded before the host is built.
p.Require("ScadaBridge:Communication:GrpcPsk",
value => !string.IsNullOrWhiteSpace(value),
"is required for Site nodes: it is the preshared key the gRPC control "
+ "plane authenticates with, and the interceptor is fail-closed, so an "
+ "unset key refuses every SiteStream call. Set the same value here (in "
+ "production as ${secret:SB-GRPC-PSK-<siteId>}) and under the secret "
+ "name SB-GRPC-PSK-<siteId> in central's secret store");
// ScadaBridge:Database:SiteDbPath was required here until LocalDb // ScadaBridge:Database:SiteDbPath was required here until LocalDb
// Phase 2. The site's tables now live in the consolidated LocalDb // Phase 2. The site's tables now live in the consolidated LocalDb
// database (LocalDb:Path, which SiteServiceRegistration requires), // database (LocalDb:Path, which SiteServiceRegistration requires),
@@ -42,6 +42,8 @@
"SqliteDbPath": "./data/store-and-forward.db" "SqliteDbPath": "./data/store-and-forward.db"
}, },
"Communication": { "Communication": {
"_grpcPsk": "REQUIRED on Site nodes (StartupValidator fails the boot without it). The preshared key the gRPC control plane authenticates with: ControlPlaneAuthInterceptor is fail-closed, so an unset key refuses every SiteStream call — live subscriptions, audit pulls, cached-telemetry ingest — while the node still reports healthy. Supply it as ${secret:SB-GRPC-PSK-<siteId>} so the plaintext never sits in this file, and seed the SAME value on central: either as the secret SB-GRPC-PSK-<siteId> in its store, or as ScadaBridge:Communication:SitePsks:<siteId>. BOTH nodes of the pair carry the same key. Distinct from LocalDb:Replication:ApiKey, which authenticates the pair partner, not central — never share the two.",
"GrpcPsk": "${secret:SB-GRPC-PSK-site-1}",
"_centralContactPoints": "Host-016: each entry MUST be a central node's remoting endpoint, NOT this site's own remoting port. The single dev-loopback default below points only at central-a (localhost:8081). In a multi-central deployment add the second central node here (e.g. 'akka.tcp://scadabridge@central-b-host:8081') so ClusterClient can fail over when central-a is down. The previous template listed localhost:8082 as the second contact — that is THIS site's own RemotingPort and is a permanent failure in the initial-contact rotation.", "_centralContactPoints": "Host-016: each entry MUST be a central node's remoting endpoint, NOT this site's own remoting port. The single dev-loopback default below points only at central-a (localhost:8081). In a multi-central deployment add the second central node here (e.g. 'akka.tcp://scadabridge@central-b-host:8081') so ClusterClient can fail over when central-a is down. The previous template listed localhost:8082 as the second contact — that is THIS site's own RemotingPort and is a permanent failure in the initial-contact rotation.",
"CentralContactPoints": [ "CentralContactPoints": [
"akka.tcp://scadabridge@localhost:8081" "akka.tcp://scadabridge@localhost:8081"
@@ -58,7 +58,7 @@ public class GrpcPullAuditEventsClientTests
public static FakeInvoker Throwing(Exception ex) => new(null, ex); public static FakeInvoker Throwing(Exception ex) => new(null, ex);
public Task<ProtoPullResponse> InvokeAsync( public Task<ProtoPullResponse> InvokeAsync(
string endpoint, ProtoPullRequest request, CancellationToken ct) string siteId, string endpoint, ProtoPullRequest request, CancellationToken ct)
{ {
CallCount++; CallCount++;
Endpoint = endpoint; Endpoint = endpoint;
@@ -84,7 +84,7 @@ public class GrpcPullAuditEventsClientTests
_byEndpoint = byEndpoint; _byEndpoint = byEndpoint;
public Task<ProtoPullResponse> InvokeAsync( public Task<ProtoPullResponse> InvokeAsync(
string endpoint, ProtoPullRequest request, CancellationToken ct) string siteId, string endpoint, ProtoPullRequest request, CancellationToken ct)
{ {
Dialed.Add(endpoint); Dialed.Add(endpoint);
return Task.FromResult(_byEndpoint[endpoint]()); return Task.FromResult(_byEndpoint[endpoint]());
@@ -50,7 +50,7 @@ public class GrpcPullSiteCallsClientTests
public static FakeInvoker Throwing(Exception ex) => new(null, ex); public static FakeInvoker Throwing(Exception ex) => new(null, ex);
public Task<ProtoPullResponse> InvokeAsync( public Task<ProtoPullResponse> InvokeAsync(
string endpoint, ProtoPullRequest request, CancellationToken ct) string siteId, string endpoint, ProtoPullRequest request, CancellationToken ct)
{ {
CallCount++; CallCount++;
Endpoint = endpoint; Endpoint = endpoint;
@@ -77,7 +77,7 @@ public class GrpcPullSiteCallsClientTests
_byEndpoint = byEndpoint; _byEndpoint = byEndpoint;
public Task<ProtoPullResponse> InvokeAsync( public Task<ProtoPullResponse> InvokeAsync(
string endpoint, ProtoPullRequest request, CancellationToken ct) string siteId, string endpoint, ProtoPullRequest request, CancellationToken ct)
{ {
Dialed.Add(endpoint); Dialed.Add(endpoint);
// The Func may throw (transport fault) — the client's try/catch handles it. // The Func may throw (transport fault) — the client's try/catch handles it.
@@ -0,0 +1,67 @@
using Akka.Actor;
using Akka.Cluster.Tools.Client;
using Akka.TestKit.Xunit2;
using ZB.MOM.WW.ScadaBridge.Commons.Messages.Notification;
using ZB.MOM.WW.ScadaBridge.Communication.Actors;
namespace ZB.MOM.WW.ScadaBridge.Communication.Tests.Actors;
/// <summary>
/// T1A.3: the regression-prone bit of the Akka transport — that a forwarded
/// <see cref="ClusterClient.Send"/> carries the caller's sender, so central's reply routes
/// straight back to the waiting Ask rather than to the site communication actor — plus the
/// no-ClusterClient-yet fallback that keeps the S&amp;F layer treating the send as transient.
/// </summary>
public class AkkaCentralTransportTests : TestKit
{
[Fact]
public void SubmitNotification_ForwardsToClusterClient_WithReplyToAsSender()
{
var transport = new AkkaCentralTransport();
var clusterClient = CreateTestProbe();
var replyTo = CreateTestProbe();
transport.SetCentralClient(clusterClient.Ref);
var submit = new NotificationSubmit(
"notif-1", "Operators", "S", "B", "site1", null, null, DateTimeOffset.UtcNow);
transport.SubmitNotification(submit, replyTo.Ref);
// The ClusterClient receives a Send addressed to the central actor...
var send = clusterClient.ExpectMsg<ClusterClient.Send>();
Assert.Equal("/user/central-communication", send.Path);
Assert.IsType<NotificationSubmit>(send.Message);
// ...and replying to it lands at replyTo, proving the sender was forwarded (not the
// transport / actor). This is the routing the waiting Ask relies on.
clusterClient.Reply(new NotificationSubmitAck("notif-1", Accepted: true, Error: null));
replyTo.ExpectMsg<NotificationSubmitAck>(ack => ack.NotificationId == "notif-1" && ack.Accepted);
}
[Fact]
public void SubmitNotification_WithNoClusterClient_RepliesNonAcceptedToReplyTo()
{
var transport = new AkkaCentralTransport();
var replyTo = CreateTestProbe();
transport.SubmitNotification(
new NotificationSubmit("notif-2", "Operators", "S", "B", "site1", null, null, DateTimeOffset.UtcNow),
replyTo.Ref);
replyTo.ExpectMsg<NotificationSubmitAck>(ack => ack.NotificationId == "notif-2" && !ack.Accepted);
}
[Fact]
public void IngestAuditEvents_WithNoClusterClient_FaultsTheReplyTo()
{
// The audit drain treats a faulted Ask as transient and keeps rows Pending — so the
// no-client path must be a Status.Failure, not a silent drop.
var transport = new AkkaCentralTransport();
var replyTo = CreateTestProbe();
transport.IngestAuditEvents(
new Commons.Messages.Audit.IngestAuditEventsCommand(new List<ZB.MOM.WW.Audit.AuditEvent>()),
replyTo.Ref);
replyTo.ExpectMsg<Status.Failure>();
}
}
@@ -0,0 +1,166 @@
using Akka.Actor;
using Akka.TestKit.Xunit2;
using NSubstitute;
using ZB.MOM.WW.Audit;
using ZB.MOM.WW.ScadaBridge.Commons.Messages.Audit;
using ZB.MOM.WW.ScadaBridge.Commons.Messages.Deployment;
using ZB.MOM.WW.ScadaBridge.Commons.Messages.Health;
using ZB.MOM.WW.ScadaBridge.Commons.Messages.Notification;
using ZB.MOM.WW.ScadaBridge.Commons.Types.Enums;
using ZB.MOM.WW.ScadaBridge.Communication.Actors;
namespace ZB.MOM.WW.ScadaBridge.Communication.Tests.Actors;
/// <summary>
/// T1A.3: the site communication actor delegates each of the seven site→central sends to the
/// injected <see cref="ICentralTransport"/>, preserving the current <c>Sender</c> as the reply
/// target — and a transport failure surfaces to that sender exactly as the S&amp;F / audit / health
/// layers already expect, while a heartbeat transport fault never faults the actor.
/// </summary>
public class SiteCommunicationActorTransportTests : TestKit
{
private readonly CommunicationOptions _options = new();
private (IActorRef actor, ICentralTransport transport) NewActor()
{
var transport = Substitute.For<ICentralTransport>();
var dmProbe = CreateTestProbe();
var actor = Sys.ActorOf(Props.Create(() =>
new SiteCommunicationActor("site1", _options, dmProbe.Ref, () => false, null, transport)));
return (actor, transport);
}
[Fact]
public void NotificationSubmit_DelegatesToTransport_WithSenderAsReplyTo()
{
var (actor, transport) = NewActor();
var submit = new NotificationSubmit(
"notif-1", "Operators", "Subj", "Body", "site1", "inst1", "alarmScript", DateTimeOffset.UtcNow);
actor.Tell(submit, TestActor);
AwaitAssert(() => transport.Received(1).SubmitNotification(
Arg.Is<NotificationSubmit>(m => m.NotificationId == "notif-1"), Arg.Is(TestActor)));
}
[Fact]
public void NotificationStatusQuery_DelegatesToTransport_WithSenderAsReplyTo()
{
var (actor, transport) = NewActor();
actor.Tell(new NotificationStatusQuery("corr-1", "notif-1"), TestActor);
AwaitAssert(() => transport.Received(1).QueryNotificationStatus(
Arg.Is<NotificationStatusQuery>(m => m.NotificationId == "notif-1"), Arg.Is(TestActor)));
}
[Fact]
public void IngestAuditEvents_DelegatesToTransport_WithSenderAsReplyTo()
{
var (actor, transport) = NewActor();
actor.Tell(new IngestAuditEventsCommand(new List<AuditEvent>()), TestActor);
AwaitAssert(() => transport.Received(1).IngestAuditEvents(Arg.Any<IngestAuditEventsCommand>(), Arg.Is(TestActor)));
}
[Fact]
public void IngestCachedTelemetry_DelegatesToTransport_WithSenderAsReplyTo()
{
var (actor, transport) = NewActor();
actor.Tell(new IngestCachedTelemetryCommand(new List<CachedTelemetryEntry>()), TestActor);
AwaitAssert(() => transport.Received(1).IngestCachedTelemetry(Arg.Any<IngestCachedTelemetryCommand>(), Arg.Is(TestActor)));
}
[Fact]
public void ReconcileSite_DelegatesToTransport_WithSenderAsReplyTo()
{
var (actor, transport) = NewActor();
actor.Tell(
new ReconcileSiteRequest("site1", "node-a", new Dictionary<string, string>()), TestActor);
AwaitAssert(() => transport.Received(1).ReconcileSite(
Arg.Is<ReconcileSiteRequest>(m => m.NodeId == "node-a"), Arg.Is(TestActor)));
}
[Fact]
public void ReportSiteHealth_DelegatesToTransport_WithSenderAsReplyTo()
{
var (actor, transport) = NewActor();
var report = MinimalHealthReport(sequence: 7);
actor.Tell(report, TestActor);
AwaitAssert(() => transport.Received(1).ReportSiteHealth(
Arg.Is<SiteHealthReport>(m => m.SequenceNumber == 7), Arg.Is(TestActor)));
}
[Fact]
public void TransportFailureReply_RoutesBackToTheWaitingSender()
{
// The seam preserves the reply-routing the S&F layer depends on: when the transport
// answers the captured replyTo with a Status.Failure (its transient-failure signal on a
// non-OK status), that failure reaches the original sender — here the test actor — so the
// waiting Ask faults, exactly as it did on the ClusterClient path.
var transport = Substitute.For<ICentralTransport>();
transport
.When(t => t.SubmitNotification(Arg.Any<NotificationSubmit>(), Arg.Any<IActorRef>()))
.Do(ci => ci.Arg<IActorRef>().Tell(
new Status.Failure(new InvalidOperationException("central unavailable"))));
var dmProbe = CreateTestProbe();
var actor = Sys.ActorOf(Props.Create(() =>
new SiteCommunicationActor("site1", _options, dmProbe.Ref, () => false, null, transport)));
actor.Tell(new NotificationSubmit(
"notif-x", "Operators", "S", "B", "site1", null, null, DateTimeOffset.UtcNow), TestActor);
var failure = ExpectMsg<Status.Failure>();
Assert.IsType<InvalidOperationException>(failure.Cause);
}
[Fact]
public void HeartbeatTransportThrow_DoesNotFaultTheActor()
{
// A transport whose SendHeartbeat throws must not fault the actor — heartbeats are
// fire-and-forget and their failure is swallowed. Prove the actor still serves messages
// after a heartbeat that threw.
var transport = Substitute.For<ICentralTransport>();
transport
.When(t => t.SendHeartbeat(Arg.Any<HeartbeatMessage>(), Arg.Any<IActorRef>()))
.Do(_ => throw new InvalidOperationException("boom"));
var dmProbe = CreateTestProbe();
var actor = Sys.ActorOf(Props.Create(() =>
new SiteCommunicationActor(
"site1",
new CommunicationOptions { ApplicationHeartbeatInterval = TimeSpan.FromMilliseconds(50) },
dmProbe.Ref, () => true, null, transport)));
// Let the heartbeat timer fire a few times (each throws inside the transport).
Thread.Sleep(250);
// The actor is still alive and delegating: a subsequent send is handled normally.
actor.Tell(new NotificationSubmit(
"after-heartbeat", "Operators", "S", "B", "site1", null, null, DateTimeOffset.UtcNow), TestActor);
AwaitAssert(() => transport.Received(1).SubmitNotification(
Arg.Is<NotificationSubmit>(m => m.NotificationId == "after-heartbeat"), Arg.Is(TestActor)));
}
private static SiteHealthReport MinimalHealthReport(long sequence) => new(
SiteId: "site1",
SequenceNumber: sequence,
ReportTimestamp: DateTimeOffset.UtcNow,
DataConnectionStatuses: new Dictionary<string, ConnectionHealth>(),
TagResolutionCounts: new Dictionary<string, TagResolutionStatus>(),
ScriptErrorCount: 0,
AlarmEvaluationErrorCount: 0,
StoreAndForwardBufferDepths: new Dictionary<string, int>(),
DeadLetterCount: 0,
DeployedInstanceCount: 0,
EnabledInstanceCount: 0,
DisabledInstanceCount: 0);
}
@@ -104,4 +104,57 @@ public class CommunicationOptionsValidatorTests
Assert.True(result.Failed); Assert.True(result.Failed);
Assert.Contains("LiveAlarmCachePublishCoalesce", result.FailureMessage); Assert.Contains("LiveAlarmCachePublishCoalesce", result.FailureMessage);
} }
// ── T1A.3: gRPC central transport endpoints (required only when selected) ────
[Fact]
public void GrpcTransport_WithNoEndpoints_IsRejected()
{
var result = Validate(new CommunicationOptions
{
CentralTransport = CentralTransportMode.Grpc,
CentralGrpcEndpoints = new List<string>(),
});
Assert.True(result.Failed);
Assert.Contains("CentralGrpcEndpoints", result.FailureMessage);
}
[Fact]
public void GrpcTransport_WithBlankEndpoint_IsRejected()
{
var result = Validate(new CommunicationOptions
{
CentralTransport = CentralTransportMode.Grpc,
CentralGrpcEndpoints = new List<string> { " " },
});
Assert.True(result.Failed);
Assert.Contains("CentralGrpcEndpoints", result.FailureMessage);
}
[Fact]
public void GrpcTransport_WithEndpoints_IsValid()
{
var result = Validate(new CommunicationOptions
{
CentralTransport = CentralTransportMode.Grpc,
CentralGrpcEndpoints = new List<string>
{
"http://scadabridge-central-a:8083",
"http://scadabridge-central-b:8083",
},
});
Assert.True(result.Succeeded, result.FailureMessage);
}
[Fact]
public void AkkaTransport_IgnoresMissingGrpcEndpoints()
{
// The default transport must not be forced to declare gRPC endpoints it never dials.
var result = Validate(new CommunicationOptions
{
CentralTransport = CentralTransportMode.Akka,
CentralGrpcEndpoints = new List<string>(),
});
Assert.True(result.Succeeded, result.FailureMessage);
}
} }
@@ -0,0 +1,643 @@
using Google.Protobuf;
using ZB.MOM.WW.ScadaBridge.Commons.Entities.Audit;
using ZB.MOM.WW.ScadaBridge.Commons.Messages.Audit;
using ZB.MOM.WW.ScadaBridge.Commons.Messages.Deployment;
using ZB.MOM.WW.ScadaBridge.Commons.Messages.Health;
using ZB.MOM.WW.ScadaBridge.Commons.Messages.Notification;
using ZB.MOM.WW.ScadaBridge.Commons.Types;
using ZB.MOM.WW.ScadaBridge.Commons.Types.Audit;
using ZB.MOM.WW.ScadaBridge.Commons.Types.Enums;
using ZB.MOM.WW.ScadaBridge.Communication.Grpc;
namespace ZB.MOM.WW.ScadaBridge.Communication.Tests.Grpc;
/// <summary>
/// Round-trip golden tests for <see cref="CentralControlDtoMapper"/> — the DTO bridge
/// for the seven <c>CentralControlService</c> RPCs (Phase 1A of the ClusterClient→gRPC
/// migration).
/// </summary>
/// <remarks>
/// <para>
/// Every message gets a pair: a FULLY-POPULATED case that proves no field is dropped,
/// and a MINIMAL case that proves every nullable/optional/empty-collection member comes
/// back as null-or-empty rather than as a zero-valued stand-in. The minimal cases are
/// the ones that matter: a per-field unit test happily passes while a whole optional
/// branch is silently never written, which is exactly the class of bug the round-trip
/// guard in PLAN-05 T8 caught five times over.
/// </para>
/// <para>
/// When a round-trip here fails, the mapper is wrong — not the test.
/// </para>
/// </remarks>
public class CentralControlDtoMapperTests
{
private static readonly DateTimeOffset SampleInstant =
new(2026, 7, 22, 9, 30, 15, 250, TimeSpan.Zero);
// -----------------------------------------------------------------------
// NotificationSubmit / Ack
// -----------------------------------------------------------------------
[Fact]
public void NotificationSubmit_RoundTrip_FullyPopulated_PreservesEveryField()
{
var original = new NotificationSubmit(
NotificationId: Guid.NewGuid().ToString(),
ListName: "plant-ops",
Subject: "Tank 4 overfill",
Body: "Level exceeded 95% for 5 minutes.",
SourceSiteId: "site-a",
SourceInstanceId: "Tank04",
SourceScript: "OnLevelHigh",
SiteEnqueuedAt: SampleInstant,
OriginExecutionId: Guid.NewGuid(),
OriginParentExecutionId: Guid.NewGuid(),
SourceNode: "node-b");
var roundTripped = CentralControlDtoMapper.FromDto(Wire(CentralControlDtoMapper.ToDto(original)));
// Every member is a scalar, so record value-equality is a true deep compare.
Assert.Equal(original, roundTripped);
}
[Fact]
public void NotificationSubmit_RoundTrip_AllOptionalsNull_StayNull()
{
var original = new NotificationSubmit(
NotificationId: Guid.NewGuid().ToString(),
ListName: "plant-ops",
Subject: "s",
Body: "b",
SourceSiteId: "site-a",
SourceInstanceId: null,
SourceScript: null,
SiteEnqueuedAt: SampleInstant,
OriginExecutionId: null,
OriginParentExecutionId: null,
SourceNode: null);
var roundTripped = CentralControlDtoMapper.FromDto(Wire(CentralControlDtoMapper.ToDto(original)));
Assert.Equal(original, roundTripped);
Assert.Null(roundTripped.SourceInstanceId);
Assert.Null(roundTripped.SourceScript);
Assert.Null(roundTripped.OriginExecutionId);
Assert.Null(roundTripped.OriginParentExecutionId);
Assert.Null(roundTripped.SourceNode);
}
[Fact]
public void NotificationSubmit_NonUtcOffset_NormalizesToTheSameInstant()
{
// Documented contract: a protobuf Timestamp is an instant, so the offset
// component of a DateTimeOffset does not survive. Every producer stamps UTC
// (Notify.Send uses DateTimeOffset.UtcNow), so the instant is what matters.
var melbourne = new DateTimeOffset(2026, 7, 22, 19, 30, 15, TimeSpan.FromHours(10));
var original = new NotificationSubmit(
"id", "list", "s", "b", "site-a", null, null, melbourne);
var roundTripped = CentralControlDtoMapper.FromDto(Wire(CentralControlDtoMapper.ToDto(original)));
Assert.Equal(melbourne.UtcDateTime, roundTripped.SiteEnqueuedAt.UtcDateTime);
Assert.Equal(TimeSpan.Zero, roundTripped.SiteEnqueuedAt.Offset);
}
[Fact]
public void NotificationSubmitAck_RoundTrip_Accepted_And_Rejected()
{
var accepted = new NotificationSubmitAck("n1", Accepted: true, Error: null);
var rejected = new NotificationSubmitAck("n2", Accepted: false, Error: "list not found");
Assert.Equal(accepted, CentralControlDtoMapper.FromDto(Wire(CentralControlDtoMapper.ToDto(accepted))));
Assert.Equal(rejected, CentralControlDtoMapper.FromDto(Wire(CentralControlDtoMapper.ToDto(rejected))));
Assert.Null(CentralControlDtoMapper.FromDto(Wire(CentralControlDtoMapper.ToDto(accepted))).Error);
}
// -----------------------------------------------------------------------
// NotificationStatusQuery / Response
// -----------------------------------------------------------------------
[Fact]
public void NotificationStatusQuery_RoundTrip_PreservesEveryField()
{
var original = new NotificationStatusQuery("corr-1", Guid.NewGuid().ToString());
Assert.Equal(original, CentralControlDtoMapper.FromDto(Wire(CentralControlDtoMapper.ToDto(original))));
}
[Fact]
public void NotificationStatusResponse_RoundTrip_FullyPopulated_PreservesEveryField()
{
var original = new NotificationStatusResponse(
CorrelationId: "corr-1",
Found: true,
Status: "Delivered",
RetryCount: 3,
LastError: "smtp 421",
DeliveredAt: SampleInstant);
Assert.Equal(original, CentralControlDtoMapper.FromDto(Wire(CentralControlDtoMapper.ToDto(original))));
}
[Fact]
public void NotificationStatusResponse_RoundTrip_NotFound_LeavesOptionalsNull()
{
var original = new NotificationStatusResponse(
CorrelationId: "corr-1",
Found: false,
Status: "Unknown",
RetryCount: 0,
LastError: null,
DeliveredAt: null);
var roundTripped = CentralControlDtoMapper.FromDto(Wire(CentralControlDtoMapper.ToDto(original)));
Assert.Equal(original, roundTripped);
Assert.Null(roundTripped.LastError);
Assert.Null(roundTripped.DeliveredAt);
}
// -----------------------------------------------------------------------
// Audit ingest envelopes (rows delegate to AuditEventDtoMapper / SiteCallDtoMapper)
// -----------------------------------------------------------------------
[Fact]
public void IngestAuditEventsCommand_RoundTrip_PreservesOrderAndRows()
{
var first = NewAuditEvent(sourceScript: "OnDemand");
var second = NewAuditEvent(sourceScript: "OnTrigger");
var original = new IngestAuditEventsCommand([first, second]);
var roundTripped = CentralControlDtoMapper.FromDto(Wire(CentralControlDtoMapper.ToDto(original)));
Assert.Equal(2, roundTripped.Events.Count);
Assert.Equal(first.AsRow().EventId, roundTripped.Events[0].AsRow().EventId);
Assert.Equal("OnDemand", roundTripped.Events[0].AsRow().SourceScript);
Assert.Equal(second.AsRow().EventId, roundTripped.Events[1].AsRow().EventId);
Assert.Equal("OnTrigger", roundTripped.Events[1].AsRow().SourceScript);
}
[Fact]
public void IngestAuditEventsCommand_RoundTrip_EmptyBatch_StaysEmpty()
{
var roundTripped = CentralControlDtoMapper.FromDto(
Wire(CentralControlDtoMapper.ToDto(new IngestAuditEventsCommand([]))));
Assert.Empty(roundTripped.Events);
}
[Fact]
public void IngestCachedTelemetryCommand_RoundTrip_PreservesBothHalvesOfEachPacket()
{
var audit = NewAuditEvent(sourceScript: "OnDemand");
var siteCall = NewSiteCall();
var original = new IngestCachedTelemetryCommand([new CachedTelemetryEntry(audit, siteCall)]);
var roundTripped = CentralControlDtoMapper.FromDto(Wire(CentralControlDtoMapper.ToDto(original)));
var entry = Assert.Single(roundTripped.Entries);
Assert.Equal(audit.AsRow().EventId, entry.Audit.AsRow().EventId);
// IngestedAtUtc is central-set inside the dual-write transaction and is
// deliberately off the wire, so it is the one member excluded from the compare.
Assert.Equal(siteCall with { IngestedAtUtc = default }, entry.SiteCall with { IngestedAtUtc = default });
}
[Fact]
public void IngestCachedTelemetryCommand_RoundTrip_NullableSiteCallFields_StayNull()
{
var siteCall = NewSiteCall() with
{
SourceNode = null,
LastError = null,
HttpStatus = null,
TerminalAtUtc = null,
};
var original = new IngestCachedTelemetryCommand(
[new CachedTelemetryEntry(NewAuditEvent(), siteCall)]);
var roundTripped = CentralControlDtoMapper.FromDto(Wire(CentralControlDtoMapper.ToDto(original)));
var entry = Assert.Single(roundTripped.Entries);
Assert.Null(entry.SiteCall.SourceNode);
Assert.Null(entry.SiteCall.LastError);
Assert.Null(entry.SiteCall.HttpStatus);
Assert.Null(entry.SiteCall.TerminalAtUtc);
}
[Fact]
public void IngestCachedTelemetryCommand_RoundTrip_EmptyBatch_StaysEmpty()
{
var roundTripped = CentralControlDtoMapper.FromDto(
Wire(CentralControlDtoMapper.ToDto(new IngestCachedTelemetryCommand([]))));
Assert.Empty(roundTripped.Entries);
}
[Fact]
public void IngestAck_RoundTrip_PreservesIdsAndOrder()
{
var ids = new[] { Guid.NewGuid(), Guid.NewGuid(), Guid.NewGuid() };
var roundTripped = CentralControlDtoMapper.FromIngestAck(
Wire(CentralControlDtoMapper.ToIngestAck(ids)));
Assert.Equal(ids, roundTripped);
}
[Fact]
public void IngestAck_RoundTrip_NothingAccepted_StaysEmpty()
{
// An empty ack is meaningful — "zero rows persisted" leaves the site's rows
// Pending — so it must not be indistinguishable from a dropped field.
Assert.Empty(CentralControlDtoMapper.FromIngestAck(
Wire(CentralControlDtoMapper.ToIngestAck([]))));
}
// -----------------------------------------------------------------------
// Startup reconciliation
// -----------------------------------------------------------------------
[Fact]
public void ReconcileSiteRequest_RoundTrip_PreservesInventoryMap()
{
var original = new ReconcileSiteRequest(
SiteIdentifier: "site-a",
NodeId: "node-a",
LocalNameToRevisionHash: new Dictionary<string, string>
{
["Plant.Tank04"] = "hash-1",
["Plant.Pump01"] = "hash-2",
});
var roundTripped = CentralControlDtoMapper.FromDto(Wire(CentralControlDtoMapper.ToDto(original)));
Assert.Equal(original.SiteIdentifier, roundTripped.SiteIdentifier);
Assert.Equal(original.NodeId, roundTripped.NodeId);
Assert.Equal(
original.LocalNameToRevisionHash.OrderBy(kv => kv.Key),
roundTripped.LocalNameToRevisionHash.OrderBy(kv => kv.Key));
}
[Fact]
public void ReconcileSiteRequest_RoundTrip_EmptyInventory_StaysEmpty()
{
// A node with nothing deployed is the normal first-boot case, and central
// must see an empty inventory rather than a missing one.
var roundTripped = CentralControlDtoMapper.FromDto(Wire(CentralControlDtoMapper.ToDto(
new ReconcileSiteRequest("site-a", "node-a", new Dictionary<string, string>()))));
Assert.Empty(roundTripped.LocalNameToRevisionHash);
}
[Fact]
public void ReconcileSiteResponse_RoundTrip_PreservesGapOrphansAndUrl()
{
var original = new ReconcileSiteResponse(
Gap:
[
new ReconcileGapItem("Plant.Tank04", "dep-1", "hash-1", IsEnabled: true, "token-1"),
new ReconcileGapItem("Plant.Pump01", "dep-2", "hash-2", IsEnabled: false, "token-2"),
],
OrphanNames: ["Plant.Retired01", "Plant.Retired02"],
CentralFetchBaseUrl: "http://scadabridge-central-a:5000");
var roundTripped = CentralControlDtoMapper.FromDto(Wire(CentralControlDtoMapper.ToDto(original)));
Assert.Equal(original.Gap, roundTripped.Gap);
Assert.Equal(original.OrphanNames, roundTripped.OrphanNames);
Assert.Equal(original.CentralFetchBaseUrl, roundTripped.CentralFetchBaseUrl);
}
[Fact]
public void ReconcileSiteResponse_RoundTrip_NoGap_StaysEmpty()
{
var roundTripped = CentralControlDtoMapper.FromDto(Wire(CentralControlDtoMapper.ToDto(
new ReconcileSiteResponse([], [], "http://central:5000"))));
Assert.Empty(roundTripped.Gap);
Assert.Empty(roundTripped.OrphanNames);
}
// -----------------------------------------------------------------------
// Site health
// -----------------------------------------------------------------------
[Fact]
public void SiteHealthReport_RoundTrip_FullyPopulated_PreservesEveryField()
{
var original = FullyPopulatedHealthReport();
var roundTripped = CentralControlDtoMapper.FromDto(Wire(CentralControlDtoMapper.ToDto(original)));
AssertHealthReportsEqual(original, roundTripped);
}
[Fact]
public void SiteHealthReport_RoundTrip_MinimalReport_LeavesEveryOptionalNullOrEmpty()
{
// The shape a producer emits before any reporter has run: no endpoints map, no
// tag-quality map, no cluster-node list, no audit backlog, no nullable gauges.
var original = new SiteHealthReport(
SiteId: "site-a",
SequenceNumber: 1,
ReportTimestamp: SampleInstant,
DataConnectionStatuses: new Dictionary<string, ConnectionHealth>(),
TagResolutionCounts: new Dictionary<string, TagResolutionStatus>(),
ScriptErrorCount: 0,
AlarmEvaluationErrorCount: 0,
StoreAndForwardBufferDepths: new Dictionary<string, int>(),
DeadLetterCount: 0,
DeployedInstanceCount: 0,
EnabledInstanceCount: 0,
DisabledInstanceCount: 0);
var roundTripped = CentralControlDtoMapper.FromDto(Wire(CentralControlDtoMapper.ToDto(original)));
AssertHealthReportsEqual(original, roundTripped);
Assert.Empty(roundTripped.DataConnectionStatuses);
Assert.Empty(roundTripped.TagResolutionCounts);
Assert.Empty(roundTripped.StoreAndForwardBufferDepths);
Assert.Null(roundTripped.DataConnectionEndpoints);
Assert.Null(roundTripped.DataConnectionTagQuality);
Assert.Null(roundTripped.ClusterNodes);
Assert.Null(roundTripped.SiteAuditBacklog);
Assert.Null(roundTripped.OldestParkedMessageAgeSeconds);
Assert.Null(roundTripped.ScriptOldestBusyAgeSeconds);
Assert.Null(roundTripped.LocalDbReplicationConnected);
Assert.Null(roundTripped.LocalDbOplogBacklog);
}
[Fact]
public void SiteHealthReport_RoundTrip_EmptyNullableCollections_StayEmptyNotNull()
{
// The distinction the three wrapper messages exist for. Null means "this node
// does not report the signal"; empty means "it reports it, and there is
// nothing in it". proto3 cannot express presence on repeated/map fields, so
// collapsing one into the other here would be invisible until an operator
// misread the health page.
var original = FullyPopulatedHealthReport() with
{
DataConnectionEndpoints = new Dictionary<string, string>(),
DataConnectionTagQuality = new Dictionary<string, TagQualityCounts>(),
ClusterNodes = [],
};
var roundTripped = CentralControlDtoMapper.FromDto(Wire(CentralControlDtoMapper.ToDto(original)));
Assert.NotNull(roundTripped.DataConnectionEndpoints);
Assert.Empty(roundTripped.DataConnectionEndpoints);
Assert.NotNull(roundTripped.DataConnectionTagQuality);
Assert.Empty(roundTripped.DataConnectionTagQuality);
Assert.NotNull(roundTripped.ClusterNodes);
Assert.Empty(roundTripped.ClusterNodes);
}
[Fact]
public void SiteHealthReport_RoundTrip_FalseAndZeroGauges_StayFalseAndZero()
{
// The mirror of the null case: LocalDbReplicationConnected=false ("configured
// but disconnected") and LocalDbOplogBacklog=0 ("healthy, nothing queued") are
// real values that must not decay into null on the wire.
var original = FullyPopulatedHealthReport() with
{
LocalDbReplicationConnected = false,
LocalDbOplogBacklog = 0,
OldestParkedMessageAgeSeconds = 0d,
ScriptOldestBusyAgeSeconds = 0d,
};
var roundTripped = CentralControlDtoMapper.FromDto(Wire(CentralControlDtoMapper.ToDto(original)));
Assert.False(roundTripped.LocalDbReplicationConnected);
Assert.Equal(0L, roundTripped.LocalDbOplogBacklog);
Assert.Equal(0d, roundTripped.OldestParkedMessageAgeSeconds);
Assert.Equal(0d, roundTripped.ScriptOldestBusyAgeSeconds);
}
[Fact]
public void SiteHealthReport_RoundTrip_BacklogWithEmptyQueue_KeepsNullOldestPending()
{
var original = FullyPopulatedHealthReport() with
{
SiteAuditBacklog = new SiteAuditBacklogSnapshot(0, null, 4096),
};
var roundTripped = CentralControlDtoMapper.FromDto(Wire(CentralControlDtoMapper.ToDto(original)));
Assert.Equal(new SiteAuditBacklogSnapshot(0, null, 4096), roundTripped.SiteAuditBacklog);
}
[Fact]
public void SiteHealthReportAck_RoundTrip_Accepted_And_Rejected()
{
var accepted = new SiteHealthReportAck("site-a", 42, Accepted: true);
var rejected = new SiteHealthReportAck("site-a", 43, Accepted: false, Error: "aggregator down");
Assert.Equal(accepted, CentralControlDtoMapper.FromDto(Wire(CentralControlDtoMapper.ToDto(accepted))));
Assert.Equal(rejected, CentralControlDtoMapper.FromDto(Wire(CentralControlDtoMapper.ToDto(rejected))));
Assert.Null(CentralControlDtoMapper.FromDto(Wire(CentralControlDtoMapper.ToDto(accepted))).Error);
}
// -----------------------------------------------------------------------
// Heartbeat + ConnectionHealth enum
// -----------------------------------------------------------------------
[Theory]
[InlineData(true)]
[InlineData(false)]
public void HeartbeatMessage_RoundTrip_PreservesEveryField(bool isActive)
{
var original = new HeartbeatMessage("site-a", "scadabridge-site-a-node-b", isActive, SampleInstant);
Assert.Equal(original, CentralControlDtoMapper.FromDto(Wire(CentralControlDtoMapper.ToDto(original))));
}
[Theory]
[InlineData(ConnectionHealth.Connected)]
[InlineData(ConnectionHealth.Disconnected)]
[InlineData(ConnectionHealth.Connecting)]
[InlineData(ConnectionHealth.Error)]
public void ConnectionHealth_RoundTrip_EveryValueSurvives(ConnectionHealth health)
{
Assert.Equal(health, CentralControlDtoMapper.FromDto(CentralControlDtoMapper.ToDto(health)));
}
[Fact]
public void ConnectionHealth_Unspecified_DecodesToErrorNotConnected()
{
// Fail-safe direction: a wire value this build does not recognise must never
// render as "healthy" on the central health page.
Assert.Equal(
ConnectionHealth.Error,
CentralControlDtoMapper.FromDto(ConnectionHealthEnum.ConnectionHealthUnspecified));
}
[Fact]
public void ConnectionHealth_EveryEnumMemberIsMapped()
{
// Guards the ArgumentOutOfRangeException path: adding a ConnectionHealth value
// without extending the mapper should fail here, not silently on a rig.
foreach (var health in Enum.GetValues<ConnectionHealth>())
{
var wire = CentralControlDtoMapper.ToDto(health);
Assert.NotEqual(ConnectionHealthEnum.ConnectionHealthUnspecified, wire);
}
}
// -----------------------------------------------------------------------
// Fixtures
// -----------------------------------------------------------------------
private static SiteHealthReport FullyPopulatedHealthReport() =>
new(
SiteId: "site-a",
SequenceNumber: 987654321L,
ReportTimestamp: SampleInstant,
DataConnectionStatuses: new Dictionary<string, ConnectionHealth>
{
["opc-main"] = ConnectionHealth.Connected,
["mx-gateway"] = ConnectionHealth.Error,
["opc-backup"] = ConnectionHealth.Connecting,
["legacy"] = ConnectionHealth.Disconnected,
},
TagResolutionCounts: new Dictionary<string, TagResolutionStatus>
{
["opc-main"] = new(TotalSubscribed: 120, SuccessfullyResolved: 118),
},
ScriptErrorCount: 4,
AlarmEvaluationErrorCount: 2,
StoreAndForwardBufferDepths: new Dictionary<string, int>
{
["erp"] = 17,
["mes"] = 0,
},
DeadLetterCount: 5,
DeployedInstanceCount: 30,
EnabledInstanceCount: 28,
DisabledInstanceCount: 2,
NodeRole: "Active",
NodeHostname: "scadabridge-site-a-node-a",
DataConnectionEndpoints: new Dictionary<string, string>
{
["opc-main"] = "opc.tcp://opcua:4840",
},
DataConnectionTagQuality: new Dictionary<string, TagQualityCounts>
{
["opc-main"] = new(Good: 100, Bad: 3, Uncertain: 15),
},
ParkedMessageCount: 6,
ClusterNodes:
[
new NodeStatus("scadabridge-site-a-node-a", IsOnline: true, "Active"),
new NodeStatus("scadabridge-site-a-node-b", IsOnline: false, "Standby"),
],
SiteAuditWriteFailures: 7,
AuditRedactionFailure: 8,
SiteAuditBacklog: new SiteAuditBacklogSnapshot(
PendingCount: 42,
OldestPendingUtc: new DateTime(2026, 7, 22, 8, 0, 0, DateTimeKind.Utc),
OnDiskBytes: 1_234_567L),
SiteEventLogWriteFailures: 9L,
OldestParkedMessageAgeSeconds: 3600.5d)
{
ScriptQueueDepth = 11,
ScriptBusyThreads = 3,
ScriptOldestBusyAgeSeconds = 12.25d,
LocalDbReplicationConnected = true,
LocalDbOplogBacklog = 250L,
};
/// <summary>
/// Compares two reports member by member. <see cref="SiteHealthReport"/> is a record,
/// but its dictionary/list members compare by reference under record equality, so
/// <c>Assert.Equal(a, b)</c> would pass trivially for the scalars and never look
/// inside the collections — the exact blind spot these goldens exist to close.
/// </summary>
private static void AssertHealthReportsEqual(SiteHealthReport expected, SiteHealthReport actual)
{
Assert.Equal(expected.SiteId, actual.SiteId);
Assert.Equal(expected.SequenceNumber, actual.SequenceNumber);
Assert.Equal(expected.ReportTimestamp, actual.ReportTimestamp);
Assert.Equal(
expected.DataConnectionStatuses.OrderBy(kv => kv.Key),
actual.DataConnectionStatuses.OrderBy(kv => kv.Key));
Assert.Equal(
expected.TagResolutionCounts.OrderBy(kv => kv.Key),
actual.TagResolutionCounts.OrderBy(kv => kv.Key));
Assert.Equal(expected.ScriptErrorCount, actual.ScriptErrorCount);
Assert.Equal(expected.AlarmEvaluationErrorCount, actual.AlarmEvaluationErrorCount);
Assert.Equal(
expected.StoreAndForwardBufferDepths.OrderBy(kv => kv.Key),
actual.StoreAndForwardBufferDepths.OrderBy(kv => kv.Key));
Assert.Equal(expected.DeadLetterCount, actual.DeadLetterCount);
Assert.Equal(expected.DeployedInstanceCount, actual.DeployedInstanceCount);
Assert.Equal(expected.EnabledInstanceCount, actual.EnabledInstanceCount);
Assert.Equal(expected.DisabledInstanceCount, actual.DisabledInstanceCount);
Assert.Equal(expected.NodeRole, actual.NodeRole);
Assert.Equal(expected.NodeHostname, actual.NodeHostname);
Assert.Equal(
expected.DataConnectionEndpoints?.OrderBy(kv => kv.Key),
actual.DataConnectionEndpoints?.OrderBy(kv => kv.Key));
Assert.Equal(
expected.DataConnectionTagQuality?.OrderBy(kv => kv.Key),
actual.DataConnectionTagQuality?.OrderBy(kv => kv.Key));
Assert.Equal(expected.ParkedMessageCount, actual.ParkedMessageCount);
Assert.Equal(expected.ClusterNodes, actual.ClusterNodes);
Assert.Equal(expected.SiteAuditWriteFailures, actual.SiteAuditWriteFailures);
Assert.Equal(expected.AuditRedactionFailure, actual.AuditRedactionFailure);
Assert.Equal(expected.SiteAuditBacklog, actual.SiteAuditBacklog);
Assert.Equal(expected.SiteEventLogWriteFailures, actual.SiteEventLogWriteFailures);
Assert.Equal(expected.OldestParkedMessageAgeSeconds, actual.OldestParkedMessageAgeSeconds);
Assert.Equal(expected.ScriptQueueDepth, actual.ScriptQueueDepth);
Assert.Equal(expected.ScriptBusyThreads, actual.ScriptBusyThreads);
Assert.Equal(expected.ScriptOldestBusyAgeSeconds, actual.ScriptOldestBusyAgeSeconds);
Assert.Equal(expected.LocalDbReplicationConnected, actual.LocalDbReplicationConnected);
Assert.Equal(expected.LocalDbOplogBacklog, actual.LocalDbOplogBacklog);
}
/// <summary>
/// Serializes a wire message and parses it back, so every round-trip below crosses a
/// real protobuf encode/decode rather than only exercising the mapper's object graph.
/// This is what proves the three collection wrapper messages keep their presence
/// when empty — an empty message encodes as a tag with zero-length payload, and a
/// mapper that used a bare <c>repeated</c>/<c>map</c> field instead would decode an
/// empty collection back as null with no test-visible difference at the object level.
/// </summary>
/// <typeparam name="T">Generated protobuf message type.</typeparam>
/// <param name="message">The message to send through an encode/decode cycle.</param>
/// <returns>An independent instance parsed from <paramref name="message"/>'s bytes.</returns>
private static T Wire<T>(T message) where T : IMessage<T>, new() =>
new MessageParser<T>(() => new T()).ParseFrom(message.ToByteArray());
private static ZB.MOM.WW.Audit.AuditEvent NewAuditEvent(string? sourceScript = null) =>
ScadaBridgeAuditEventFactory.Create(
channel: AuditChannel.ApiOutbound,
kind: AuditKind.ApiCallCached,
status: AuditStatus.Forwarded,
eventId: Guid.NewGuid(),
occurredAtUtc: new DateTime(2026, 7, 22, 9, 0, 0, DateTimeKind.Utc),
target: "ERP.GetOrder",
sourceSiteId: "site-a",
sourceNode: "node-a",
sourceScript: sourceScript);
private static SiteCall NewSiteCall() => new()
{
TrackedOperationId = TrackedOperationId.New(),
Channel = "ApiOutbound",
Target = "ERP.GetOrder",
SourceSite = "site-a",
SourceNode = "node-a",
Status = "Delivered",
RetryCount = 2,
LastError = "transient 503",
HttpStatus = 200,
CreatedAtUtc = new DateTime(2026, 7, 22, 8, 0, 0, DateTimeKind.Utc),
UpdatedAtUtc = new DateTime(2026, 7, 22, 8, 5, 0, DateTimeKind.Utc),
TerminalAtUtc = new DateTime(2026, 7, 22, 8, 10, 0, DateTimeKind.Utc),
IngestedAtUtc = new DateTime(2026, 7, 22, 8, 10, 1, DateTimeKind.Utc),
};
}
@@ -0,0 +1,181 @@
using Akka.Actor;
using Akka.TestKit;
using Akka.TestKit.Xunit2;
using Grpc.Core;
using Microsoft.Extensions.Logging.Abstractions;
using Microsoft.Extensions.Options;
using NSubstitute;
using ZB.MOM.WW.ScadaBridge.Commons.Messages.Audit;
using ZB.MOM.WW.ScadaBridge.Commons.Messages.Notification;
using ZB.MOM.WW.ScadaBridge.Communication;
using ZB.MOM.WW.ScadaBridge.Communication.Grpc;
namespace ZB.MOM.WW.ScadaBridge.Communication.Tests.Grpc;
/// <summary>
/// Behaviour of <see cref="CentralControlGrpcService"/> that does not need a real gRPC pipeline:
/// the readiness gate, the fire-and-forget heartbeat contract, the ingest-timeout status
/// mapping, and the shared audit-ingest timeout constant. The auth interceptor + real transport
/// are proven separately in <c>CentralControlEndToEndTests</c> (Host.Tests).
/// </summary>
public class CentralControlGrpcServiceTests : TestKit
{
private static ServerCallContext NewContext(CancellationToken ct = default)
{
var context = Substitute.For<ServerCallContext>();
context.CancellationToken.Returns(ct);
return context;
}
private CentralControlGrpcService CreateService(CommunicationOptions? options = null)
=> new(
NullLogger<CentralControlGrpcService>.Instance,
Options.Create(options ?? new CommunicationOptions()));
[Fact]
public async Task BeforeSetReady_AUnaryCall_IsUnavailable()
{
// Nothing was dispatched, so Unavailable is the right status — it tells a site transport
// the call never ran and a cross-node retry is safe.
var service = CreateService();
var ex = await Assert.ThrowsAsync<RpcException>(
() => service.SubmitNotification(new NotificationSubmitDto(), NewContext()));
Assert.Equal(StatusCode.Unavailable, ex.StatusCode);
}
[Fact]
public async Task BeforeSetReady_ANonEmptyIngestBatch_IsUnavailable()
{
var service = CreateService();
var batch = new AuditEventBatch();
batch.Events.Add(NewAuditDto());
var ex = await Assert.ThrowsAsync<RpcException>(
() => service.IngestAuditEvents(batch, NewContext()));
Assert.Equal(StatusCode.Unavailable, ex.StatusCode);
}
[Fact]
public async Task AfterSetReady_AUnaryCall_ReachesTheActor()
{
var stub = Sys.ActorOf(Props.Create(() => new StubActor()));
var service = CreateService();
service.SetReady(stub);
var ack = await service.SubmitNotification(NewNotificationDto(), NewContext());
Assert.True(ack.Accepted);
}
[Fact]
public async Task Heartbeat_BeforeSetReady_SucceedsAndIsDropped()
{
// Fire-and-forget end-to-end: a heartbeat must never fault the site's timer, so even
// with no actor wired the call returns OK rather than Unavailable.
var service = CreateService();
var reply = await service.Heartbeat(NewHeartbeatDto(), NewContext());
Assert.NotNull(reply);
}
[Fact]
public async Task Heartbeat_AfterSetReady_TellsTheActor_AndNeverAsks()
{
// A black-hole actor that never replies would hang an Ask forever; the heartbeat still
// returns immediately, proving it is a Tell, not an Ask.
var blackHole = Sys.ActorOf(Props.Create(() => new NeverRepliesActor()));
var service = CreateService();
service.SetReady(blackHole);
var reply = await service.Heartbeat(NewHeartbeatDto(), NewContext());
Assert.NotNull(reply);
}
[Fact]
public async Task WhenTheActorNeverReplies_TheCall_IsDeadlineExceeded_NotUnavailable()
{
// The message WAS delivered (Unavailable would wrongly invite a duplicate retry on the
// peer node); a timeout is DeadlineExceeded, which callers never cross-node-retry.
var blackHole = Sys.ActorOf(Props.Create(() => new NeverRepliesActor()));
var service = CreateService(new CommunicationOptions
{
NotificationForwardTimeout = TimeSpan.FromMilliseconds(200),
});
service.SetReady(blackHole);
var ex = await Assert.ThrowsAsync<RpcException>(
() => service.SubmitNotification(NewNotificationDto(), NewContext()));
Assert.Equal(StatusCode.DeadlineExceeded, ex.StatusCode);
}
[Fact]
public async Task EmptyIngestBatch_ShortCircuits_EvenBeforeSetReady()
{
// An empty batch is a no-op the actor need never see; it must not depend on readiness.
var service = CreateService();
var ack = await service.IngestAuditEvents(new AuditEventBatch(), NewContext());
Assert.Empty(ack.AcceptedEventIds);
}
[Fact]
public void TheAuditIngestTimeout_IsTheOneSharedConstant()
{
// The plan calls SiteStreamGrpcServer.AuditIngestAskTimeout "one source of truth" shared
// between the two audit-ingest transports; the central service must not re-declare 30s.
Assert.Equal(TimeSpan.FromSeconds(30), SiteStreamGrpcServer.AuditIngestAskTimeout);
}
// The mapper reads SiteEnqueuedAt unconditionally, so a DTO that reaches mapping must carry
// a timestamp. Only DTOs that get past the readiness/auth gate map, so the negative tests
// above can pass a bare DTO.
private static NotificationSubmitDto NewNotificationDto() => new()
{
NotificationId = Guid.NewGuid().ToString(),
ListName = "ops",
SiteEnqueuedAt = Google.Protobuf.WellKnownTypes.Timestamp.FromDateTime(
DateTime.SpecifyKind(new DateTime(2026, 5, 20, 10, 0, 0), DateTimeKind.Utc)),
};
private static HeartbeatDto NewHeartbeatDto() => new()
{
SiteId = "site-a",
NodeHostname = "node-a",
IsActive = true,
Timestamp = Google.Protobuf.WellKnownTypes.Timestamp.FromDateTime(
DateTime.SpecifyKind(new DateTime(2026, 5, 20, 10, 0, 0), DateTimeKind.Utc)),
};
private static AuditEventDto NewAuditDto() => new()
{
EventId = Guid.NewGuid().ToString(),
OccurredAtUtc = Google.Protobuf.WellKnownTypes.Timestamp.FromDateTime(
DateTime.SpecifyKind(new DateTime(2026, 5, 20, 10, 0, 0), DateTimeKind.Utc)),
Channel = "ApiOutbound",
Kind = "ApiCall",
Status = "Delivered",
SourceSiteId = "site-a",
};
private sealed class StubActor : ReceiveActor
{
public StubActor()
{
Receive<NotificationSubmit>(msg =>
Sender.Tell(new NotificationSubmitAck(msg.NotificationId, Accepted: true, Error: null)));
}
}
/// <summary>Swallows every message and never replies, so an Ask against it times out.</summary>
private sealed class NeverRepliesActor : ReceiveActor
{
public NeverRepliesActor() => ReceiveAny(_ => { });
}
}
@@ -41,7 +41,7 @@ public class SiteStreamGrpcClientFactoryDisposeTests
public IReadOnlyCollection<TrackingClient> Created => _created.ToList(); public IReadOnlyCollection<TrackingClient> Created => _created.ToList();
protected override SiteStreamGrpcClient CreateClient(string grpcEndpoint) protected override SiteStreamGrpcClient CreateClient(string siteIdentifier, string grpcEndpoint)
{ {
var client = new TrackingClient(); var client = new TrackingClient();
_created.Add(client); _created.Add(client);
@@ -155,7 +155,7 @@ public class SiteStreamGrpcClientFactoryTests
{ {
public TrackingEndpointFactory() : base(NullLoggerFactory.Instance) { } public TrackingEndpointFactory() : base(NullLoggerFactory.Instance) { }
public int CreatedCount { get; private set; } public int CreatedCount { get; private set; }
protected override SiteStreamGrpcClient CreateClient(string grpcEndpoint) protected override SiteStreamGrpcClient CreateClient(string siteIdentifier, string grpcEndpoint)
{ {
CreatedCount++; CreatedCount++;
return new TrackingEndpointClient(grpcEndpoint); return new TrackingEndpointClient(grpcEndpoint);
@@ -0,0 +1,197 @@
using Grpc.Core;
using Microsoft.Extensions.Logging.Abstractions;
using ZB.MOM.WW.ScadaBridge.Communication.Grpc;
namespace ZB.MOM.WW.ScadaBridge.Host.Tests;
/// <summary>
/// Central's inbound gate for the site→central gRPC control plane (T1A.2). The mirror of
/// <see cref="ControlPlaneAuthInterceptorTests"/>, but central's verification model is inverted:
/// it looks up the key for the site named in the <c>x-scadabridge-site</c> header and checks the
/// bearer token against THAT, rather than against one own-key.
/// </summary>
/// <remarks>
/// Central genuinely needs a distinct verification model, so it is a separate class with its own
/// single public constructor — not a variant ctor on <see cref="ControlPlaneAuthInterceptor"/>,
/// whose two-public-constructor form once silently disabled the gate. The one-public-ctor
/// invariant is pinned below exactly as the sibling pins it.
/// </remarks>
public class CentralControlAuthInterceptorTests
{
private const string ControlMethod =
"/scadabridge.centralcontrol.v1.CentralControlService/SubmitNotification";
private const string SiteStreamMethod = "/sitestream.SiteStreamService/SubscribeInstance";
private const string SiteA = "site-a";
private const string SiteAKey = "site-a-preshared-key";
private const string SiteB = "site-b";
private const string SiteBKey = "site-b-preshared-key";
/// <summary>An <see cref="ISitePskProvider"/> backed by a fixed site→key map; throws for unknown sites.</summary>
private sealed class MapPskProvider(IReadOnlyDictionary<string, string> keys) : ISitePskProvider
{
public ValueTask<string> GetAsync(string siteId, CancellationToken ct)
=> keys.TryGetValue(siteId, out var key)
? new ValueTask<string>(key)
: throw new InvalidOperationException($"no key for '{siteId}'");
public void Invalidate(string siteId) { }
}
private static CentralControlAuthInterceptor CreateInterceptor()
=> new(
new MapPskProvider(new Dictionary<string, string> { [SiteA] = SiteAKey, [SiteB] = SiteBKey }),
NullLogger<CentralControlAuthInterceptor>.Instance);
private static ServerCallContext CreateContext(
string method, string? siteHeader, string? authorizationHeader)
{
var headers = new Metadata();
if (siteHeader is not null)
headers.Add(ControlPlaneCredentials.SiteHeader, siteHeader);
if (authorizationHeader is not null)
headers.Add(ControlPlaneCredentials.AuthorizationHeader, authorizationHeader);
return new FakeServerCallContext(method, headers);
}
private sealed class FakeServerCallContext(string method, Metadata requestHeaders)
: ServerCallContext
{
protected override string MethodCore => method;
protected override string HostCore => "localhost";
protected override string PeerCore => "ipv4:127.0.0.1:12345";
protected override DateTime DeadlineCore => DateTime.UtcNow.AddMinutes(1);
protected override Metadata RequestHeadersCore => requestHeaders;
protected override CancellationToken CancellationTokenCore => CancellationToken.None;
protected override Metadata ResponseTrailersCore { get; } = [];
protected override Status StatusCore { get; set; }
protected override WriteOptions? WriteOptionsCore { get; set; }
protected override AuthContext AuthContextCore { get; } =
new(null, new Dictionary<string, List<AuthProperty>>());
protected override ContextPropagationToken CreatePropagationTokenCore(
ContextPropagationOptions? options)
=> throw new NotSupportedException();
protected override Task WriteResponseHeadersAsyncCore(Metadata responseHeaders)
=> Task.CompletedTask;
}
private static Task<string> Invoke(
CentralControlAuthInterceptor interceptor, ServerCallContext context)
=> interceptor.UnaryServerHandler<string, string>(
"request", context, (_, _) => Task.FromResult("ok"));
[Fact]
public async Task NonGatedMethod_PassesThrough()
{
// Central's interceptor gates only CentralControlService; anything else on the listener
// (there is nothing today) is not its concern.
var interceptor = CreateInterceptor();
var context = CreateContext(SiteStreamMethod, siteHeader: null, authorizationHeader: null);
Assert.Equal("ok", await Invoke(interceptor, context));
}
[Fact]
public async Task CorrectSiteAndKey_IsAccepted()
{
var interceptor = CreateInterceptor();
var context = CreateContext(ControlMethod, SiteA, $"Bearer {SiteAKey}");
Assert.Equal("ok", await Invoke(interceptor, context));
}
[Fact]
public async Task MissingSiteHeader_IsDenied()
{
// Fail-closed: no header means no per-site key to verify against, so there is nothing to
// pass through TO. A present bearer token does not rescue it.
var interceptor = CreateInterceptor();
var context = CreateContext(ControlMethod, siteHeader: null, $"Bearer {SiteAKey}");
var ex = await Assert.ThrowsAsync<RpcException>(() => Invoke(interceptor, context));
Assert.Equal(StatusCode.PermissionDenied, ex.StatusCode);
}
[Fact]
public async Task BlankSiteHeader_IsDenied()
{
var interceptor = CreateInterceptor();
var context = CreateContext(ControlMethod, siteHeader: " ", $"Bearer {SiteAKey}");
var ex = await Assert.ThrowsAsync<RpcException>(() => Invoke(interceptor, context));
Assert.Equal(StatusCode.PermissionDenied, ex.StatusCode);
}
[Fact]
public async Task UnknownSite_IsDenied_NotPassedThrough()
{
// The provider throws for an unknown site; the interceptor must turn that into a denial,
// never swallow it and let the call proceed unauthenticated.
var interceptor = CreateInterceptor();
var context = CreateContext(ControlMethod, "site-nonexistent", "Bearer whatever");
var ex = await Assert.ThrowsAsync<RpcException>(() => Invoke(interceptor, context));
Assert.Equal(StatusCode.PermissionDenied, ex.StatusCode);
}
[Fact]
public async Task NoBearerToken_IsDenied()
{
var interceptor = CreateInterceptor();
var context = CreateContext(ControlMethod, SiteA, authorizationHeader: null);
var ex = await Assert.ThrowsAsync<RpcException>(() => Invoke(interceptor, context));
Assert.Equal(StatusCode.PermissionDenied, ex.StatusCode);
}
[Fact]
public async Task WrongKeyForTheSite_IsDenied()
{
// site-a presents site-b's key. Both keys are valid keys; the point is the token must
// match the key for THIS site, not just be a key central knows.
var interceptor = CreateInterceptor();
var context = CreateContext(ControlMethod, SiteA, $"Bearer {SiteBKey}");
var ex = await Assert.ThrowsAsync<RpcException>(() => Invoke(interceptor, context));
Assert.Equal(StatusCode.PermissionDenied, ex.StatusCode);
}
[Fact]
public async Task EachSiteIsVerifiedAgainstItsOwnKey()
{
// site-b with site-b's key is accepted by the same interceptor instance that rejected
// site-a-with-site-b's-key above — proving the per-site lookup, not a single shared key.
var interceptor = CreateInterceptor();
var context = CreateContext(ControlMethod, SiteB, $"Bearer {SiteBKey}");
Assert.Equal("ok", await Invoke(interceptor, context));
}
[Fact]
public void TheInterceptorHasExactlyOnePublicConstructor()
{
// Grpc.AspNetCore registers this interceptor BY TYPE; InterceptorRegistration.GetFactory()
// throws "Multiple constructors accepting all given argument types have been found" the
// moment a second public constructor is applicable, and the throw lands inside the
// pipeline on every call — a gate that authorizes nothing while looking like a handler
// bug. The explicit-prefix constructor is internal to keep it from recurring; this pins
// that. (Same invariant as ControlPlaneAuthInterceptorTests.)
var publicCtors = typeof(CentralControlAuthInterceptor).GetConstructors();
Assert.Single(publicCtors);
}
[Fact]
public void DefaultGatedPrefixes_MatchTheRealCentralControlServicePath()
{
// A typo here disables the whole gate silently: every call would pass through. Pin it
// against the generated service descriptor, not the proto text.
var method = CentralControlService.Descriptor.FullName;
Assert.Contains(
CentralControlAuthInterceptor.DefaultGatedPrefixes,
p => p == $"/{method}/");
}
}
@@ -0,0 +1,256 @@
using Akka.Actor;
using Google.Protobuf.WellKnownTypes;
using Grpc.Core;
using Grpc.Net.Client;
using Microsoft.AspNetCore.Builder;
using Microsoft.AspNetCore.Hosting;
using Microsoft.AspNetCore.TestHost;
using Microsoft.Extensions.DependencyInjection;
using Microsoft.Extensions.Hosting;
using Microsoft.Extensions.Logging.Abstractions;
using Microsoft.Extensions.Options;
using ZB.MOM.WW.ScadaBridge.Commons.Messages.Audit;
using ZB.MOM.WW.ScadaBridge.Commons.Messages.Notification;
using ZB.MOM.WW.ScadaBridge.Communication;
using ZB.MOM.WW.ScadaBridge.Communication.Grpc;
namespace ZB.MOM.WW.ScadaBridge.Host.Tests;
/// <summary>
/// End-to-end proof of the central-hosted gRPC control plane (T1A.2): the real
/// <see cref="CentralControlAuthInterceptor"/> AND the real
/// <see cref="CentralControlGrpcService"/> over a real gRPC stack, the service Asking a real
/// (stub) <c>CentralCommunicationActor</c> stand-in.
/// </summary>
/// <remarks>
/// <para>
/// The interceptor is registered <b>BY TYPE on <c>AddGrpc</c></b>, exactly as <c>Program.cs</c>
/// does, and is deliberately NOT placed in DI as a singleton — pre-registering it lets DI hand
/// the instance back and bypasses <c>Grpc.AspNetCore</c>'s own activation path, which is how the
/// two-public-constructor defect escaped a green suite once before. This harness copies the
/// shape of <see cref="ControlPlaneAuthEndToEndTests"/> for the same reason.
/// </para>
/// <para>
/// Runs in-process over <see cref="TestServer"/>: no ports, no containers. A tiny Akka actor
/// stands in for <c>CentralCommunicationActor</c> so the test never touches MSSQL or a cluster;
/// the method paths, message types and mapper are the real generated/production ones.
/// </para>
/// </remarks>
public class CentralControlEndToEndTests : IAsyncLifetime
{
private const string SiteA = "site-a";
private const string SiteAKey = "site-a-preshared-key";
private const string SiteB = "site-b";
private const string SiteBKey = "site-b-preshared-key";
// The deterministic id the stub actor "accepts" for every ingest batch, so the ingest
// bridge can be asserted without extracting ids out of a decoded AuditEvent.
private static readonly Guid AcceptedId = Guid.Parse("11111111-1111-1111-1111-111111111111");
private IHost _host = null!;
private TestServer _server = null!;
private ActorSystem _actorSystem = null!;
private CentralControlGrpcService _service = null!;
/// <inheritdoc />
public async Task InitializeAsync()
{
_actorSystem = ActorSystem.Create("centralcontrol-e2e-test");
var stub = _actorSystem.ActorOf(Props.Create(() => new StubCentralActor(AcceptedId)), "central-stub");
_service = new CentralControlGrpcService(
NullLogger<CentralControlGrpcService>.Instance,
Options.Create(new CommunicationOptions()));
_service.SetReady(stub);
var pskProvider = new MapPskProvider(new Dictionary<string, string>
{
[SiteA] = SiteAKey,
[SiteB] = SiteBKey,
});
_host = await new HostBuilder()
.ConfigureWebHost(web => web
.UseTestServer()
.ConfigureServices(services =>
{
// BY TYPE, and NOT also in DI — see the class remarks.
services.AddGrpc(o => o.Interceptors.Add<CentralControlAuthInterceptor>());
services.AddSingleton<ISitePskProvider>(pskProvider);
services.AddSingleton(_service);
})
.Configure(app =>
{
app.UseRouting();
app.UseEndpoints(e => e.MapGrpcService<CentralControlGrpcService>());
}))
.StartAsync();
_server = _host.GetTestServer();
}
/// <inheritdoc />
public async Task DisposeAsync()
{
await _host.StopAsync();
_host.Dispose();
await _actorSystem.Terminate();
}
/// <summary>Builds a channel credentialed for <paramref name="siteId"/> with <paramref name="key"/>.</summary>
private GrpcChannel Channel(string? key, string siteId)
{
var options = new GrpcChannelOptions { HttpHandler = _server.CreateHandler() };
if (key is not null)
{
options.WithSiteCredentials(new FixedPskProvider(key), siteId);
}
return GrpcChannel.ForAddress(_server.BaseAddress, options);
}
private CentralControlService.CentralControlServiceClient Client(string? key, string siteId)
=> new(Channel(key, siteId));
// ---- Auth positives / negatives, all through the real pipeline ----
[Fact]
public async Task CorrectSiteAndKey_ReachesTheService_OnAUnaryCall()
{
var client = Client(SiteAKey, SiteA);
var ack = await client.SubmitNotificationAsync(NewNotificationDto());
Assert.True(ack.Accepted);
}
[Fact]
public async Task NoCredentialsAtAll_IsRejected_WithPermissionDenied()
{
var client = Client(key: null, SiteA);
var ex = await Assert.ThrowsAsync<RpcException>(
async () => await client.SubmitNotificationAsync(new NotificationSubmitDto()));
Assert.Equal(StatusCode.PermissionDenied, ex.StatusCode);
}
[Fact]
public async Task WrongKeyForTheSite_IsRejected_WithPermissionDenied()
{
// site-a presents site-b's (valid, but wrong-for-this-site) key.
var client = Client(SiteBKey, SiteA);
var ex = await Assert.ThrowsAsync<RpcException>(
async () => await client.SubmitNotificationAsync(new NotificationSubmitDto()));
Assert.Equal(StatusCode.PermissionDenied, ex.StatusCode);
}
[Fact]
public async Task UnknownSite_IsRejected_WithPermissionDenied()
{
var client = Client("any-key", "site-nonexistent");
var ex = await Assert.ThrowsAsync<RpcException>(
async () => await client.SubmitNotificationAsync(new NotificationSubmitDto()));
Assert.Equal(StatusCode.PermissionDenied, ex.StatusCode);
}
[Fact]
public async Task TheAcceptAndRejectPathsAreDistinguishable()
{
// A gate whose accept and reject paths produce the same observable result is not a gate.
// Correct key → the service answers (Accepted:true); wrong key → PermissionDenied,
// never reaching the service. These are two different outcomes, which is the whole point.
var ok = await Client(SiteAKey, SiteA).SubmitNotificationAsync(NewNotificationDto());
Assert.True(ok.Accepted);
var ex = await Assert.ThrowsAsync<RpcException>(
async () => await Client("wrong", SiteA).SubmitNotificationAsync(new NotificationSubmitDto()));
Assert.Equal(StatusCode.PermissionDenied, ex.StatusCode);
}
// ---- One RPC per shape: unary (above) + the ingest bridge ----
[Fact]
public async Task IngestAuditEvents_DecodesTheBatch_AsksTheActor_EncodesTheAck()
{
var client = Client(SiteAKey, SiteA);
var batch = new AuditEventBatch();
batch.Events.Add(NewAuditDto());
batch.Events.Add(NewAuditDto());
var ack = await client.IngestAuditEventsAsync(batch);
// The stub actor accepts one deterministic id per non-empty batch; its presence proves
// the full DTO→command→Ask→reply→ack bridge ran, gated call and all.
Assert.Contains(AcceptedId.ToString(), ack.AcceptedEventIds);
}
[Fact]
public async Task IngestAuditEvents_EmptyBatch_ShortCircuits_WithoutAskingTheActor()
{
// Even the empty-batch fast path is behind the gate — it still needs a valid key.
var client = Client(SiteAKey, SiteA);
var ack = await client.IngestAuditEventsAsync(new AuditEventBatch());
Assert.Empty(ack.AcceptedEventIds);
}
// The mapper reads SiteEnqueuedAt unconditionally; only DTOs that clear the gate reach it,
// so the accept-path tests carry a timestamp while the negative tests can pass a bare DTO.
private static NotificationSubmitDto NewNotificationDto() => new()
{
NotificationId = Guid.NewGuid().ToString(),
ListName = "ops",
SiteEnqueuedAt = Timestamp.FromDateTime(
DateTime.SpecifyKind(new DateTime(2026, 5, 20, 10, 0, 0), DateTimeKind.Utc)),
};
private static AuditEventDto NewAuditDto() => new()
{
EventId = Guid.NewGuid().ToString(),
OccurredAtUtc = Timestamp.FromDateTime(
DateTime.SpecifyKind(new DateTime(2026, 5, 20, 10, 0, 0), DateTimeKind.Utc)),
Channel = "ApiOutbound",
Kind = "ApiCall",
Status = "Delivered",
SourceSiteId = SiteA,
};
private sealed class FixedPskProvider(string key) : ISitePskProvider
{
public ValueTask<string> GetAsync(string siteId, CancellationToken ct) => new(key);
public void Invalidate(string siteId) { }
}
private sealed class MapPskProvider(IReadOnlyDictionary<string, string> keys) : ISitePskProvider
{
public ValueTask<string> GetAsync(string siteId, CancellationToken ct)
=> keys.TryGetValue(siteId, out var key)
? new ValueTask<string>(key)
: throw new InvalidOperationException($"no key for '{siteId}'");
public void Invalidate(string siteId) { }
}
/// <summary>
/// Minimal stand-in for <c>CentralCommunicationActor</c>: answers the two RPC shapes this
/// test exercises. Replies straight to the Ask's temp sender, exactly as the real actor's
/// Forward/PipeTo paths do.
/// </summary>
private sealed class StubCentralActor : ReceiveActor
{
public StubCentralActor(Guid acceptedId)
{
Receive<NotificationSubmit>(msg =>
Sender.Tell(new NotificationSubmitAck(msg.NotificationId, Accepted: true, Error: null)));
Receive<IngestAuditEventsCommand>(_ =>
Sender.Tell(new IngestAuditEventsReply(new[] { acceptedId })));
}
}
}
@@ -0,0 +1,56 @@
using Xunit;
namespace ZB.MOM.WW.ScadaBridge.Host.Tests;
/// <summary>
/// Pins <see cref="Program.ParseHttpBindPorts"/>, which re-declares central's HTTP surface
/// after the gRPC listener switches Kestrel into explicit-endpoints mode.
/// </summary>
/// <remarks>
/// This exists because the regression it guards is invisible to every other test: a central
/// node that binds only the gRPC port boots clean, joins the cluster and serves gRPC, while
/// the Central UI, Management/Inbound API and <c>/health/*</c> are silently dead. TestServer
/// never binds real Kestrel, so the parser is the one seam that can be asserted in-process.
/// The rig caught the original defect; this keeps it caught.
/// </remarks>
public class CentralHttpBindPortsTests
{
[Theory]
[InlineData("http://+:5000", new[] { 5000 })]
[InlineData("http://0.0.0.0:5000", new[] { 5000 })]
[InlineData("http://[::]:5000", new[] { 5000 })]
[InlineData("http://localhost:5000", new[] { 5000 })]
[InlineData("http://*:8085", new[] { 8085 })]
[InlineData("http://+:5000;http://+:5001", new[] { 5000, 5001 })]
[InlineData("http://+:5000 ; http://+:5001", new[] { 5000, 5001 })]
[InlineData("http://+:5000;http://+:5000", new[] { 5000 })] // de-duped
public void ParsesPortsFromServerUrls(string urls, int[] expected)
{
Assert.Equal(expected, Program.ParseHttpBindPorts(urls));
}
[Theory]
[InlineData("https://+:443", 443)] // scheme default when no explicit port
[InlineData("http://+:80", 80)]
public void HandlesSchemeDefaultAndExplicitPort(string url, int expected)
{
Assert.Equal(new[] { expected }, Program.ParseHttpBindPorts(url));
}
[Theory]
[InlineData(null)]
[InlineData("")]
[InlineData(" ")]
public void EmptyOrNull_YieldsNoPorts(string? urls)
{
Assert.Empty(Program.ParseHttpBindPorts(urls));
}
[Fact]
public void UnparseableEntry_IsSkipped_NotThrown()
{
// A malformed URL must not take the node down at boot; the good one still binds.
var ports = Program.ParseHttpBindPorts("not-a-url;http://+:5000");
Assert.Equal(new[] { 5000 }, ports);
}
}
@@ -0,0 +1,231 @@
using Grpc.Core;
using Grpc.Net.Client;
using Microsoft.AspNetCore.Builder;
using Microsoft.AspNetCore.Hosting;
using Microsoft.AspNetCore.TestHost;
using Microsoft.Extensions.DependencyInjection;
using Microsoft.Extensions.Hosting;
using Microsoft.Extensions.Logging.Abstractions;
using Microsoft.Extensions.Options;
using ZB.MOM.WW.ScadaBridge.Communication;
using ZB.MOM.WW.ScadaBridge.Communication.Grpc;
namespace ZB.MOM.WW.ScadaBridge.Host.Tests;
/// <summary>
/// End-to-end proof that the two halves of the control-plane PSK actually interoperate:
/// <see cref="ControlPlaneCredentials"/> on the client and
/// <see cref="ControlPlaneAuthInterceptor"/> on the server, over a real gRPC stack.
/// </summary>
/// <remarks>
/// <para>
/// The unit tests either side of this file each test one half against a hand-built input, and
/// would both stay green if the halves disagreed — if the credentials never attached to a
/// streaming call, if the metadata key case differed, or if attaching call credentials to a
/// plaintext channel were rejected outright (gRPC refuses that by default; the code opts in with
/// <c>UnsafeUseInsecureChannelCallCredentials</c>, and nothing but a real call proves the opt-in
/// works). Getting that wrong takes down every site's streaming and audit-pull path at once,
/// which is a bad thing to discover on the rig.
/// </para>
/// <para>
/// Runs entirely in-process over <see cref="TestServer"/>: no ports, no containers. The service
/// is a stub rather than the real <c>SiteStreamGrpcServer</c> — this is a test of the auth
/// pipeline, and the real server would drag in an actor system for no added coverage. The method
/// paths and message types are the real generated ones.
/// </para>
/// </remarks>
public class ControlPlaneAuthEndToEndTests : IAsyncLifetime
{
private IHost _host = null!;
private TestServer _server = null!;
/// <summary>Boots the in-process gRPC host with the real interceptor.</summary>
public async Task InitializeAsync()
{
_host = await new HostBuilder()
.ConfigureWebHost(web => web
.UseTestServer()
.ConfigureServices(services =>
{
// Registered exactly as Program.cs does: by TYPE on AddGrpc, with the
// interceptor itself NOT in DI. That is load-bearing. An earlier version of
// this test added it as a singleton, which let DI hand back the instance and
// bypassed Grpc.AspNetCore's own activation — hiding a defect where the
// interceptor had two public constructors and
// InterceptorRegistration.GetFactory() threw on every single call. The rig
// caught it; this test did not. Do not pre-register it.
services.AddGrpc(o => o.Interceptors.Add<ControlPlaneAuthInterceptor>());
services.AddSingleton(Options.Create(
new CommunicationOptions { GrpcPsk = SiteKey }));
services.AddSingleton<EchoSiteStreamService>();
})
.Configure(app =>
{
app.UseRouting();
app.UseEndpoints(e => e.MapGrpcService<EchoSiteStreamService>());
}))
.StartAsync();
_server = _host.GetTestServer();
}
/// <inheritdoc />
public async Task DisposeAsync()
{
await _host.StopAsync();
_host.Dispose();
}
private const string SiteKey = "the-site-a-preshared-key";
/// <summary>
/// Builds a channel through the test server, credentialed exactly as production does.
/// </summary>
private GrpcChannel Channel(string? key, string siteId = "site-a")
{
var options = new GrpcChannelOptions { HttpHandler = _server.CreateHandler() };
if (key is not null)
{
options.WithSiteCredentials(new FixedPskProvider(key), siteId);
}
return GrpcChannel.ForAddress(_server.BaseAddress, options);
}
private sealed class FixedPskProvider(string key) : ISitePskProvider
{
public ValueTask<string> GetAsync(string siteId, CancellationToken ct) => new(key);
public void Invalidate(string siteId) { }
}
/// <summary>Stub service: echoes back what the auth pipeline let through.</summary>
private sealed class EchoSiteStreamService : SiteStreamService.SiteStreamServiceBase
{
/// <summary>The site header the last accepted call carried.</summary>
public string? LastSiteHeader { get; private set; }
public override Task<PullAuditEventsResponse> PullAuditEvents(
PullAuditEventsRequest request, ServerCallContext context)
{
LastSiteHeader = context.RequestHeaders
.FirstOrDefault(h => h.Key == ControlPlaneCredentials.SiteHeader)?.Value;
return Task.FromResult(new PullAuditEventsResponse { MoreAvailable = false });
}
public override async Task SubscribeInstance(
InstanceStreamRequest request,
IServerStreamWriter<SiteStreamEvent> responseStream,
ServerCallContext context)
{
await responseStream.WriteAsync(new SiteStreamEvent { CorrelationId = request.CorrelationId });
}
}
[Fact]
public async Task CorrectKey_IsAccepted_OnAUnaryCall()
{
using var channel = Channel(SiteKey);
var client = new SiteStreamService.SiteStreamServiceClient(channel);
var reply = await client.PullAuditEventsAsync(new PullAuditEventsRequest { BatchSize = 1 });
Assert.False(reply.MoreAvailable);
}
[Fact]
public async Task WrongKey_IsRejected_WithPermissionDenied()
{
using var channel = Channel("some-other-sites-key");
var client = new SiteStreamService.SiteStreamServiceClient(channel);
var ex = await Assert.ThrowsAsync<RpcException>(
async () => await client.PullAuditEventsAsync(new PullAuditEventsRequest()));
Assert.Equal(StatusCode.PermissionDenied, ex.StatusCode);
}
[Fact]
public async Task NoCredentialsAtAll_IsRejected()
{
// The pre-T0.3 client shape. This is the case that proves the gap is actually closed.
using var channel = Channel(key: null);
var client = new SiteStreamService.SiteStreamServiceClient(channel);
var ex = await Assert.ThrowsAsync<RpcException>(
async () => await client.PullAuditEventsAsync(new PullAuditEventsRequest()));
Assert.Equal(StatusCode.PermissionDenied, ex.StatusCode);
}
[Fact]
public async Task CredentialsApplyToStreamingCalls_NotJustUnaryOnes()
{
// CallCredentials cover every call on the channel; a client interceptor that only
// handled the unary path would pass the test above and still break every subscription.
using var channel = Channel(SiteKey);
var client = new SiteStreamService.SiteStreamServiceClient(channel);
using var call = client.SubscribeInstance(
new InstanceStreamRequest { CorrelationId = "c1", InstanceUniqueName = "i1" });
Assert.True(await call.ResponseStream.MoveNext(CancellationToken.None));
Assert.Equal("c1", call.ResponseStream.Current.CorrelationId);
}
[Fact]
public async Task WrongKey_IsRejected_OnStreamingCallsToo()
{
using var channel = Channel("wrong");
var client = new SiteStreamService.SiteStreamServiceClient(channel);
using var call = client.SubscribeInstance(new InstanceStreamRequest { CorrelationId = "c1" });
var ex = await Assert.ThrowsAsync<RpcException>(
async () => await call.ResponseStream.MoveNext(CancellationToken.None));
Assert.Equal(StatusCode.PermissionDenied, ex.StatusCode);
}
[Fact]
public async Task AnUnresolvableKey_FailsTheCall_RatherThanDialingWithoutOne()
{
// SitePskProvider throws when a site has no key anywhere. What matters here is that the
// throw stops the call: the alternative — swallowing it and sending the request
// unauthenticated — is the exact failure this design exists to prevent. The status code
// is gRPC's choice, so assert the RpcException and record what it actually is rather
// than pinning a guess: callers already treat every non-OK status as a failed call, and
// the diagnosable signal is SitePskProvider's own LogError, not this code.
var options = new GrpcChannelOptions { HttpHandler = _server.CreateHandler() }
.WithSiteCredentials(new ThrowingPskProvider(), "site-a");
using var channel = GrpcChannel.ForAddress(_server.BaseAddress, options);
var client = new SiteStreamService.SiteStreamServiceClient(channel);
var ex = await Assert.ThrowsAsync<RpcException>(
async () => await client.PullAuditEventsAsync(new PullAuditEventsRequest()));
Assert.NotEqual(StatusCode.OK, ex.StatusCode);
// And nothing reached the service.
Assert.Null(_host.Services.GetRequiredService<EchoSiteStreamService>().LastSiteHeader);
}
private sealed class ThrowingPskProvider : ISitePskProvider
{
public ValueTask<string> GetAsync(string siteId, CancellationToken ct)
=> throw new InvalidOperationException($"no key for '{siteId}'");
public void Invalidate(string siteId) { }
}
[Fact]
public async Task TheSiteHeaderTravels_SoCentralCanPickAPerSiteKeyInPhase1A()
{
// Central's own interceptor (T1A.2) verifies against the key for the site named in this
// header. Shipping it now means Phase 1A adds a lookup, not a wire change.
using var channel = Channel(SiteKey, siteId: "site-a");
var client = new SiteStreamService.SiteStreamServiceClient(channel);
await client.PullAuditEventsAsync(new PullAuditEventsRequest());
var service = _host.Services.GetRequiredService<EchoSiteStreamService>();
Assert.Equal("site-a", service.LastSiteHeader);
}
}
@@ -0,0 +1,234 @@
using Grpc.Core;
using Microsoft.Extensions.Logging.Abstractions;
using Microsoft.Extensions.Options;
using ZB.MOM.WW.ScadaBridge.Communication;
using ZB.MOM.WW.ScadaBridge.Communication.Grpc;
namespace ZB.MOM.WW.ScadaBridge.Host.Tests;
/// <summary>
/// The site↔central gRPC control plane's inbound gate (ClusterClient→gRPC migration, T0.3).
/// </summary>
/// <remarks>
/// <para>
/// <c>SiteStreamService</c> shipped unauthenticated: plaintext h2c with no interceptor, so
/// anything that could reach a site node's gRPC port could open a live data stream or pull audit
/// rows back with <c>PullAuditEvents</c>/<c>PullSiteCalls</c>. These tests pin the gate that
/// closes it, and — just as importantly — pin that it does NOT gate LocalDb sync, which has its
/// own interceptor and its own key.
/// </para>
/// <para>
/// Sibling of <see cref="LocalDbSyncAuthInterceptorTests"/>; the two interceptors share a shape
/// deliberately, so the cases mirror each other.
/// </para>
/// </remarks>
public class ControlPlaneAuthInterceptorTests
{
// Real method paths: package `sitestream`, service `SiteStreamService` (sitestream.proto).
private const string SubscribeMethod = "/sitestream.SiteStreamService/SubscribeInstance";
private const string PullAuditMethod = "/sitestream.SiteStreamService/PullAuditEvents";
private const string LocalDbSyncMethod = "/localdb_sync.v1.LocalDbSync/Sync";
private static ControlPlaneAuthInterceptor CreateInterceptor(string? psk)
=> new(
Options.Create(new CommunicationOptions { GrpcPsk = psk ?? "" }),
NullLogger<ControlPlaneAuthInterceptor>.Instance);
private static ServerCallContext CreateContext(string method, string? authorizationHeader)
{
var headers = new Metadata();
if (authorizationHeader is not null)
headers.Add("authorization", authorizationHeader);
return new FakeServerCallContext(method, headers);
}
/// <summary>
/// Minimal <see cref="ServerCallContext"/> carrying just a method name and request headers —
/// the only two things the interceptor reads. Hand-rolled for the same reason the LocalDb
/// sibling hand-rolls one: <c>Grpc.Core.Testing.TestServerCallContext</c> lives in the
/// retired native package and does not exist on the grpc-dotnet stack.
/// </summary>
private sealed class FakeServerCallContext(string method, Metadata requestHeaders)
: ServerCallContext
{
protected override string MethodCore => method;
protected override string HostCore => "localhost";
protected override string PeerCore => "ipv4:127.0.0.1:12345";
protected override DateTime DeadlineCore => DateTime.UtcNow.AddMinutes(1);
protected override Metadata RequestHeadersCore => requestHeaders;
protected override CancellationToken CancellationTokenCore => CancellationToken.None;
protected override Metadata ResponseTrailersCore { get; } = [];
protected override Status StatusCore { get; set; }
protected override WriteOptions? WriteOptionsCore { get; set; }
protected override AuthContext AuthContextCore { get; } =
new(null, new Dictionary<string, List<AuthProperty>>());
protected override ContextPropagationToken CreatePropagationTokenCore(
ContextPropagationOptions? options)
=> throw new NotSupportedException();
protected override Task WriteResponseHeadersAsyncCore(Metadata responseHeaders)
=> Task.CompletedTask;
}
/// <summary>Invokes the interceptor's unary path with a trivial continuation.</summary>
private static Task<string> Invoke(
ControlPlaneAuthInterceptor interceptor, ServerCallContext context)
=> interceptor.UnaryServerHandler<string, string>(
"request", context, (_, _) => Task.FromResult("ok"));
[Fact]
public async Task LocalDbSyncMethod_PassesThrough_BecauseItHasItsOwnGateAndItsOwnKey()
{
// Both interceptors sit on the same site AddGrpc pipeline and see every call. If this
// one also gated sync, a site would need its central-facing key to equal its pair-replication
// key — collapsing two distinct trust relationships into one secret.
var interceptor = CreateInterceptor("the-site-key");
var context = CreateContext(LocalDbSyncMethod, authorizationHeader: null);
Assert.Equal("ok", await Invoke(interceptor, context));
}
[Fact]
public async Task GatedMethod_WithNoKeyConfigured_IsDenied_EvenWithABearerToken()
{
// Fail-closed, and this is the case that differs in consequence from LocalDb's: an
// unset key here does not disable an optional feature, it closes the site's entire
// central-facing surface. Loud refusal beats silent unauthenticated service.
var interceptor = CreateInterceptor(psk: null);
var context = CreateContext(SubscribeMethod, "Bearer anything-at-all");
var ex = await Assert.ThrowsAsync<RpcException>(() => Invoke(interceptor, context));
Assert.Equal(StatusCode.PermissionDenied, ex.StatusCode);
}
[Fact]
public async Task GatedMethod_WithNoBearerToken_IsDenied()
{
var interceptor = CreateInterceptor("the-site-key");
var context = CreateContext(SubscribeMethod, authorizationHeader: null);
var ex = await Assert.ThrowsAsync<RpcException>(() => Invoke(interceptor, context));
Assert.Equal(StatusCode.PermissionDenied, ex.StatusCode);
}
[Fact]
public async Task GatedMethod_WithWrongBearerToken_IsDenied()
{
var interceptor = CreateInterceptor("the-site-key");
var context = CreateContext(SubscribeMethod, "Bearer some-other-sites-key");
var ex = await Assert.ThrowsAsync<RpcException>(() => Invoke(interceptor, context));
Assert.Equal(StatusCode.PermissionDenied, ex.StatusCode);
}
[Fact]
public async Task GatedMethod_WithCorrectBearerToken_PassesThrough()
{
var interceptor = CreateInterceptor("the-site-key");
var context = CreateContext(SubscribeMethod, "Bearer the-site-key");
Assert.Equal("ok", await Invoke(interceptor, context));
}
[Fact]
public async Task GatedMethod_WithCorrectKey_ButNoBearerScheme_IsDenied()
{
// A raw key with no "Bearer " prefix is not what ControlPlaneCredentials sends;
// accepting it would widen the accepted credential shape for nothing.
var interceptor = CreateInterceptor("the-site-key");
var context = CreateContext(SubscribeMethod, "the-site-key");
var ex = await Assert.ThrowsAsync<RpcException>(() => Invoke(interceptor, context));
Assert.Equal(StatusCode.PermissionDenied, ex.StatusCode);
}
[Fact]
public async Task GatedMethod_TokenComparison_IsNotAPrefixMatch()
{
// A StartsWith comparison would accept a truncated key and make the secret recoverable
// one character at a time. FixedTimeEquals also rejects on length.
var interceptor = CreateInterceptor("the-site-key");
var context = CreateContext(SubscribeMethod, "Bearer the-site-ke");
var ex = await Assert.ThrowsAsync<RpcException>(() => Invoke(interceptor, context));
Assert.Equal(StatusCode.PermissionDenied, ex.StatusCode);
}
[Fact]
public async Task ServerStreaming_IsGated_BecauseThatIsHowSubscriptionsActuallyRun()
{
// SubscribeInstance/SubscribeSite are server-streaming. Gating only the unary path
// would leave the live data feed wide open while every unary test still passed.
var interceptor = CreateInterceptor("the-site-key");
var context = CreateContext(SubscribeMethod, "Bearer the-wrong-key");
var ex = await Assert.ThrowsAsync<RpcException>(() =>
interceptor.ServerStreamingServerHandler<string, string>(
"request",
responseStream: null!,
context,
(_, _, _) => Task.CompletedTask));
Assert.Equal(StatusCode.PermissionDenied, ex.StatusCode);
}
[Fact]
public async Task PullRpcs_AreGated_BecauseTheyReturnAuditRows()
{
// The strongest reason this gate exists: PullAuditEvents/PullSiteCalls hand back audit
// content to anyone who asks. Unary, so it would be easy to miss in a streaming-focused
// reading of the service.
var interceptor = CreateInterceptor("the-site-key");
var context = CreateContext(PullAuditMethod, authorizationHeader: null);
var ex = await Assert.ThrowsAsync<RpcException>(() => Invoke(interceptor, context));
Assert.Equal(StatusCode.PermissionDenied, ex.StatusCode);
}
[Fact]
public async Task GatedPrefixes_AreConstructorProvided_SoLaterPhasesAddServicesNotInterceptors()
{
// Phases 1A/1B add CentralControlService and SiteCommandService to this same gate.
var interceptor = new ControlPlaneAuthInterceptor(
Options.Create(new CommunicationOptions { GrpcPsk = "k" }),
NullLogger<ControlPlaneAuthInterceptor>.Instance,
new[] { "/scadabridge.sitecommand.v1.SiteCommandService/" });
var gated = CreateContext("/scadabridge.sitecommand.v1.SiteCommandService/ExecuteQuery", null);
var ex = await Assert.ThrowsAsync<RpcException>(() => Invoke(interceptor, gated));
Assert.Equal(StatusCode.PermissionDenied, ex.StatusCode);
// ...and the default set is no longer implied once an explicit set is supplied.
var notGated = CreateContext(SubscribeMethod, authorizationHeader: null);
Assert.Equal("ok", await Invoke(interceptor, notGated));
}
[Fact]
public void TheInterceptorHasExactlyOnePublicConstructor()
{
// Grpc.AspNetCore registers this interceptor BY TYPE, and
// InterceptorRegistration.GetFactory() throws "Multiple constructors accepting all given
// argument types have been found" the moment a second public constructor is applicable.
// The throw lands inside the pipeline on every call, so the symptom is not a startup
// failure but a gate that authorizes nothing and fails everything with
// Unknown / "Exception was thrown by handler" — a shape that looks like a handler bug,
// not an auth bug. This shipped once and was caught only on the docker rig; the
// prefix-set constructor is internal now to keep it from recurring.
var publicCtors = typeof(ControlPlaneAuthInterceptor).GetConstructors();
Assert.Single(publicCtors);
}
[Fact]
public void DefaultGatedPrefixes_MatchTheRealSiteStreamServicePath()
{
// A typo here disables the whole gate silently: every call would simply pass through.
// Pin it against a path taken from the generated service, not from the proto text.
var method = SiteStreamService.Descriptor.FullName;
Assert.Contains(
ControlPlaneAuthInterceptor.DefaultGatedPrefixes,
p => p == $"/{method}/");
}
}
@@ -0,0 +1,373 @@
using System.Collections.Concurrent;
using Akka.Actor;
using Microsoft.AspNetCore.Builder;
using Microsoft.AspNetCore.Hosting;
using Microsoft.AspNetCore.TestHost;
using Microsoft.Extensions.DependencyInjection;
using Microsoft.Extensions.Hosting;
using Microsoft.Extensions.Logging.Abstractions;
using Microsoft.Extensions.Options;
using ZB.MOM.WW.ScadaBridge.Commons.Messages.Notification;
using ZB.MOM.WW.ScadaBridge.Communication;
using ZB.MOM.WW.ScadaBridge.Communication.Grpc;
namespace ZB.MOM.WW.ScadaBridge.Host.Tests;
/// <summary>
/// T1A.3: <see cref="GrpcCentralTransport"/> + <see cref="CentralChannelProvider"/> over a real
/// gRPC stack (two in-process <see cref="TestServer"/> central nodes, the real
/// <see cref="CentralControlGrpcService"/> and <see cref="CentralControlAuthInterceptor"/>). Proves
/// the sticky failover/failback policy, the PSK + site-header attachment, the per-call deadline,
/// and — the hard rule — no cross-node retry on <c>DeadlineExceeded</c>.
/// </summary>
/// <remarks>
/// A "down" node is modelled by a <see cref="ToggleHandler"/> that throws before reaching the
/// TestServer, so BOTH the unary call and the failback <c>Heartbeat</c> probe see it as
/// <c>Unavailable</c> — the honest shape of a refused connection, and the only class the transport
/// fails over on. Readiness is always set, so a node that is "up" answers everything.
/// </remarks>
public class GrpcCentralTransportTests : IAsyncLifetime
{
private const string SiteA = "site-a";
private const string SiteAKey = "site-a-preshared-key";
private const string EndpointA = "http://central-a/";
private const string EndpointB = "http://central-b/";
private ActorSystem _system = null!;
private CentralNode _nodeA = null!;
private CentralNode _nodeB = null!;
/// <inheritdoc />
public async Task InitializeAsync()
{
_system = ActorSystem.Create("grpc-central-transport-test");
_nodeA = await CentralNode.StartAsync(_system, "A", SiteA, SiteAKey, repliesToSubmit: true);
_nodeB = await CentralNode.StartAsync(_system, "B", SiteA, SiteAKey, repliesToSubmit: true);
}
/// <inheritdoc />
public async Task DisposeAsync()
{
await _nodeA.DisposeAsync();
await _nodeB.DisposeAsync();
await _system.Terminate();
}
private CentralChannelProvider NewProvider(string? pskKey = SiteAKey) => new(
new[] { EndpointA, EndpointB },
new FixedPskProvider(pskKey),
SiteA,
new CommunicationOptions(),
NullLogger<CentralChannelProvider>.Instance,
handlerFactory: HandlerFor,
probeDeadline: TimeSpan.FromSeconds(2),
backoffBase: TimeSpan.FromMilliseconds(50),
backoffCap: TimeSpan.FromMilliseconds(200));
private HttpMessageHandler HandlerFor(string endpoint) => endpoint == EndpointA
? new ToggleHandler(_nodeA.Server.CreateHandler(), () => _nodeA.IsUp)
: new ToggleHandler(_nodeB.Server.CreateHandler(), () => _nodeB.IsUp);
private GrpcCentralTransport NewTransport(CentralChannelProvider provider, CommunicationOptions? options = null)
=> new(provider, options ?? new CommunicationOptions(), NullLogger<GrpcCentralTransport>.Instance);
[Fact]
public async Task HappyPath_ReachesThePreferredNode_AndRoutesTheAckBack()
{
using var provider = NewProvider();
var transport = NewTransport(provider);
var inbox = new Capture(_system);
transport.SubmitNotification(NewSubmit("n1"), inbox.Ref);
var ack = Assert.IsType<NotificationSubmitAck>(inbox.Receive(TimeSpan.FromSeconds(5)));
Assert.True(ack.Accepted);
Assert.Equal("n1", ack.NotificationId);
Assert.Equal(0, provider.CurrentIndex); // stayed on preferred
Assert.Equal(1, _nodeA.SubmitCount);
Assert.Equal(0, _nodeB.SubmitCount);
}
[Fact]
public async Task Sticky_StaysOnThePreferredNode_WhileHealthy()
{
using var provider = NewProvider();
var transport = NewTransport(provider);
for (var i = 0; i < 4; i++)
{
var inbox = new Capture(_system);
transport.SubmitNotification(NewSubmit($"n{i}"), inbox.Ref);
Assert.IsType<NotificationSubmitAck>(inbox.Receive(TimeSpan.FromSeconds(5)));
}
Assert.Equal(0, provider.CurrentIndex);
Assert.Equal(4, _nodeA.SubmitCount);
Assert.Equal(0, _nodeB.SubmitCount);
}
[Fact]
public async Task Failover_FlipsToThePeer_WhenThePreferredIsUnavailable()
{
using var provider = NewProvider();
var transport = NewTransport(provider);
_nodeA.IsUp = false; // preferred refuses connections
var inbox = new Capture(_system);
transport.SubmitNotification(NewSubmit("n1"), inbox.Ref);
var ack = Assert.IsType<NotificationSubmitAck>(inbox.Receive(TimeSpan.FromSeconds(5)));
Assert.True(ack.Accepted);
Assert.Equal(1, provider.CurrentIndex); // flipped to the peer
Assert.Equal(0, _nodeA.SubmitCount);
Assert.Equal(1, _nodeB.SubmitCount);
}
[Fact]
public async Task Failback_ReturnsToThePreferred_OnceItIsReachableAgain()
{
using var provider = NewProvider();
var transport = NewTransport(provider);
// Take the preferred down and drive one call so we flip to the peer + arm the failback probe.
_nodeA.IsUp = false;
var inbox = new Capture(_system);
transport.SubmitNotification(NewSubmit("n1"), inbox.Ref);
Assert.IsType<NotificationSubmitAck>(inbox.Receive(TimeSpan.FromSeconds(5)));
Assert.Equal(1, provider.CurrentIndex);
// Bring the preferred back; the background probe should fail us back within a few backoffs.
_nodeA.IsUp = true;
await WaitUntil(() => provider.CurrentIndex == 0, TimeSpan.FromSeconds(5));
Assert.Equal(0, provider.CurrentIndex);
// New calls resume on the preferred node.
var inbox2 = new Capture(_system);
transport.SubmitNotification(NewSubmit("n2"), inbox2.Ref);
Assert.IsType<NotificationSubmitAck>(inbox2.Receive(TimeSpan.FromSeconds(5)));
Assert.True(_nodeA.SubmitCount >= 1);
}
[Fact]
public async Task PskAndSiteHeader_AreAttached_SoTheGatedCallReachesTheService()
{
// The service is gated by CentralControlAuthInterceptor; a call that reaches it (and gets
// Accepted) proves both the bearer PSK and the x-scadabridge-site header were attached.
using var provider = NewProvider(pskKey: SiteAKey);
var transport = NewTransport(provider);
var inbox = new Capture(_system);
transport.SubmitNotification(NewSubmit("n1"), inbox.Ref);
var ack = Assert.IsType<NotificationSubmitAck>(inbox.Receive(TimeSpan.FromSeconds(5)));
Assert.True(ack.Accepted);
}
[Fact]
public async Task WrongPsk_IsRejected_AndNotRetriedOnThePeer()
{
// PermissionDenied is not a connect failure — the transport surfaces it as a transient
// Status.Failure without flipping to the peer.
using var provider = NewProvider(pskKey: "the-wrong-key");
var transport = NewTransport(provider);
var inbox = new Capture(_system);
transport.SubmitNotification(NewSubmit("n1"), inbox.Ref);
Assert.IsType<Status.Failure>(inbox.Receive(TimeSpan.FromSeconds(5)));
Assert.Equal(0, provider.CurrentIndex); // no flip
Assert.Equal(0, _nodeB.SubmitCount); // peer never tried
}
[Fact]
public async Task DeadlineExceeded_IsNotRetriedOnThePeer()
{
// THE hard rule. Node A is UP but never replies, so the call deadlines. The transport must
// surface Status.Failure and must NOT try node B (the call may already have executed).
_nodeA.SetBlackHole();
var shortDeadline = new CommunicationOptions { NotificationForwardTimeout = TimeSpan.FromMilliseconds(300) };
using var provider = NewProvider();
var transport = NewTransport(provider, shortDeadline);
var inbox = new Capture(_system);
transport.SubmitNotification(NewSubmit("n1"), inbox.Ref);
// A per-call deadline is applied (the call returns fast instead of hanging on the black hole).
Assert.IsType<Status.Failure>(inbox.Receive(TimeSpan.FromSeconds(5)));
Assert.Equal(0, provider.CurrentIndex); // no failover on a deadline
Assert.Equal(0, _nodeB.SubmitCount); // peer never tried
}
private static NotificationSubmit NewSubmit(string id) => new(
NotificationId: id,
ListName: "ops",
Subject: "s",
Body: "b",
SourceSiteId: SiteA,
SourceInstanceId: null,
SourceScript: null,
SiteEnqueuedAt: DateTimeOffset.UtcNow);
private static async Task WaitUntil(Func<bool> condition, TimeSpan timeout)
{
var deadline = DateTime.UtcNow + timeout;
while (DateTime.UtcNow < deadline)
{
if (condition())
{
return;
}
await Task.Delay(25);
}
}
/// <summary>
/// A raw message sink used as the transport's <c>replyTo</c>. Unlike Akka's <c>Inbox</c>, which
/// rethrows a <see cref="Status.Failure"/>'s cause on receive, this captures every message
/// verbatim so a test can assert on the <see cref="Status.Failure"/> itself.
/// </summary>
private sealed class Capture
{
private readonly BlockingCollection<object> _messages = new();
public Capture(ActorSystem system)
{
Ref = system.ActorOf(Props.Create(() => new CaptureActor(_messages)));
}
public IActorRef Ref { get; }
public object Receive(TimeSpan timeout)
=> _messages.TryTake(out var message, timeout)
? message
: throw new TimeoutException("No message captured within the timeout.");
private sealed class CaptureActor : ReceiveActor
{
public CaptureActor(BlockingCollection<object> messages) => ReceiveAny(messages.Add);
}
}
/// <summary>A gRPC channel handler that throws (a refused connection) while its node is "down".</summary>
private sealed class ToggleHandler : DelegatingHandler
{
private readonly Func<bool> _isUp;
public ToggleHandler(HttpMessageHandler inner, Func<bool> isUp)
{
InnerHandler = inner;
_isUp = isUp;
}
protected override async Task<HttpResponseMessage> SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
{
if (!_isUp())
{
throw new HttpRequestException("simulated central node down");
}
return await base.SendAsync(request, cancellationToken).ConfigureAwait(false);
}
}
private sealed class FixedPskProvider(string? key) : ISitePskProvider
{
public ValueTask<string> GetAsync(string siteId, CancellationToken ct)
=> key is null ? throw new InvalidOperationException("no key") : new ValueTask<string>(key);
public void Invalidate(string siteId) { }
}
/// <summary>One in-process central node: TestServer + real service/interceptor + a stub actor.</summary>
private sealed class CentralNode : IAsyncDisposable
{
private IHost _host = null!;
private IActorRef _stub = null!;
private readonly StubCounters _counters = new();
public TestServer Server { get; private set; } = null!;
public volatile bool IsUp = true;
public int SubmitCount => _counters.Submits;
public static async Task<CentralNode> StartAsync(
ActorSystem system, string label, string site, string key, bool repliesToSubmit)
{
var node = new CentralNode();
node._stub = system.ActorOf(
Props.Create(() => new StubCentralActor(node._counters, repliesToSubmit)), $"stub-{label}");
var service = new CentralControlGrpcService(
NullLogger<CentralControlGrpcService>.Instance,
Options.Create(new CommunicationOptions()));
service.SetReady(node._stub);
var psk = new MapPskProvider(new Dictionary<string, string> { [site] = key });
node._host = await new HostBuilder()
.ConfigureWebHost(web => web
.UseTestServer()
.ConfigureServices(services =>
{
services.AddGrpc(o => o.Interceptors.Add<CentralControlAuthInterceptor>());
services.AddSingleton<ISitePskProvider>(psk);
services.AddSingleton(service);
})
.Configure(app =>
{
app.UseRouting();
app.UseEndpoints(e => e.MapGrpcService<CentralControlGrpcService>());
}))
.StartAsync();
node.Server = node._host.GetTestServer();
return node;
}
/// <summary>Switches the node's actor to a black hole that counts but never replies.</summary>
public void SetBlackHole() => _counters.BlackHole = true;
public async ValueTask DisposeAsync()
{
await _host.StopAsync();
_host.Dispose();
}
private sealed class StubCounters
{
private int _submits;
public int Submits => Volatile.Read(ref _submits);
public void IncrementSubmits() => Interlocked.Increment(ref _submits);
public volatile bool BlackHole;
}
private sealed class StubCentralActor : ReceiveActor
{
public StubCentralActor(StubCounters counters, bool repliesToSubmit)
{
Receive<NotificationSubmit>(msg =>
{
counters.IncrementSubmits();
if (repliesToSubmit && !counters.BlackHole)
{
Sender.Tell(new NotificationSubmitAck(msg.NotificationId, Accepted: true, Error: null));
}
});
// Heartbeat lands here as a Tell (the failback probe); ignore it, no reply expected.
ReceiveAny(_ => { });
}
}
private sealed class MapPskProvider(IReadOnlyDictionary<string, string> keys) : ISitePskProvider
{
public ValueTask<string> GetAsync(string siteId, CancellationToken ct)
=> keys.TryGetValue(siteId, out var key)
? new ValueTask<string>(key)
: throw new InvalidOperationException($"no key for '{siteId}'");
public void Invalidate(string siteId) { }
}
}
}
@@ -0,0 +1,179 @@
using Microsoft.Extensions.Logging.Abstractions;
using Microsoft.Extensions.Options;
using NSubstitute;
using ZB.MOM.WW.ScadaBridge.Communication;
using ZB.MOM.WW.Secrets.Abstractions;
namespace ZB.MOM.WW.ScadaBridge.Host.Tests;
/// <summary>
/// Central's per-site gRPC preshared key resolution (ClusterClient→gRPC migration, T0.3).
/// </summary>
/// <remarks>
/// The property that matters most here is the negative one: a site whose key cannot be found
/// must produce a throw, never an unauthenticated channel. Everything else — the two sources,
/// the caching, the invalidation — exists to make that behaviour usable in practice.
/// </remarks>
public class SitePskProviderTests
{
private static SitePskProvider Create(
ISecretResolver resolver, CommunicationOptions? options = null)
=> new(
resolver,
new StaticOptionsMonitor(options ?? new CommunicationOptions()),
NullLogger<SitePskProvider>.Instance);
private sealed class StaticOptionsMonitor(CommunicationOptions value)
: IOptionsMonitor<CommunicationOptions>
{
public CommunicationOptions CurrentValue => value;
public CommunicationOptions Get(string? name) => value;
public IDisposable? OnChange(Action<CommunicationOptions, string?> listener) => null;
}
[Fact]
public async Task ResolvesFromTheSecretStore_UnderTheSiteQualifiedName()
{
var resolver = Substitute.For<ISecretResolver>();
resolver.GetAsync(new SecretName("SB-GRPC-PSK-site-a"), Arg.Any<CancellationToken>())
.Returns("key-for-a");
var psk = await Create(resolver).GetAsync("site-a", CancellationToken.None);
Assert.Equal("key-for-a", psk);
}
[Fact]
public async Task ConfiguredMapWins_SoAHostWithNoMasterKeyCanStillDial()
{
// The development rig runs with no secrets master key at all — every credential
// arrives as an environment override. Without this source, the rig could not use the
// gated control plane and the fail-closed design would be untestable there.
var resolver = Substitute.For<ISecretResolver>();
resolver.GetAsync(Arg.Any<SecretName>(), Arg.Any<CancellationToken>())
.Returns("from-the-store");
var options = new CommunicationOptions();
options.SitePsks["site-a"] = "from-config";
var psk = await Create(resolver, options).GetAsync("site-a", CancellationToken.None);
Assert.Equal("from-config", psk);
await resolver.DidNotReceive().GetAsync(Arg.Any<SecretName>(), Arg.Any<CancellationToken>());
}
[Fact]
public async Task StoreIsStillConsulted_ForASiteMissingFromTheMap()
{
// Sites are added at runtime from the Central UI, so the map can never be complete.
var resolver = Substitute.For<ISecretResolver>();
resolver.GetAsync(new SecretName("SB-GRPC-PSK-site-b"), Arg.Any<CancellationToken>())
.Returns("key-for-b");
var options = new CommunicationOptions();
options.SitePsks["site-a"] = "from-config";
var psk = await Create(resolver, options).GetAsync("site-b", CancellationToken.None);
Assert.Equal("key-for-b", psk);
}
[Fact]
public async Task MissingSecret_Throws_AndNeverYieldsAnEmptyKey()
{
// Fail-closed. The alternative — returning "" — would build a channel that presents
// "Bearer " and gets PermissionDenied anyway, but with a far less diagnosable error.
var resolver = Substitute.For<ISecretResolver>();
resolver.GetAsync(Arg.Any<SecretName>(), Arg.Any<CancellationToken>())
.Returns((string?)null);
var ex = await Assert.ThrowsAsync<InvalidOperationException>(
async () => await Create(resolver).GetAsync("site-a", CancellationToken.None));
// The message must name both sources — either one fixes it.
Assert.Contains("SB-GRPC-PSK-site-a", ex.Message);
Assert.Contains("SitePsks", ex.Message);
}
[Fact]
public async Task EmptySecret_IsTreatedAsMissing()
{
var resolver = Substitute.For<ISecretResolver>();
resolver.GetAsync(Arg.Any<SecretName>(), Arg.Any<CancellationToken>()).Returns("");
await Assert.ThrowsAsync<InvalidOperationException>(
async () => await Create(resolver).GetAsync("site-a", CancellationToken.None));
}
[Fact]
public async Task AFaultingStore_SurfacesAsKeyNotConfigured_NotAsACryptoError()
{
// A host with no master key throws from the resolver. The operator's problem is the
// same either way — "this site has no key" — so the diagnosis must say that.
var resolver = Substitute.For<ISecretResolver>();
resolver.GetAsync(Arg.Any<SecretName>(), Arg.Any<CancellationToken>())
.Returns<string?>(_ => throw new InvalidOperationException("no master key configured"));
var ex = await Assert.ThrowsAsync<InvalidOperationException>(
async () => await Create(resolver).GetAsync("site-a", CancellationToken.None));
Assert.Contains("SB-GRPC-PSK-site-a", ex.Message);
}
[Fact]
public async Task SuccessfulResolvesAreCached_SoEveryCallDoesNotHitTheStore()
{
var resolver = Substitute.For<ISecretResolver>();
resolver.GetAsync(Arg.Any<SecretName>(), Arg.Any<CancellationToken>()).Returns("k");
var provider = Create(resolver);
await provider.GetAsync("site-a", CancellationToken.None);
await provider.GetAsync("site-a", CancellationToken.None);
await provider.GetAsync("site-a", CancellationToken.None);
await resolver.Received(1).GetAsync(Arg.Any<SecretName>(), Arg.Any<CancellationToken>());
}
[Fact]
public async Task FailuresAreNotCached_SoASeededKeyIsPickedUpWithoutARestart()
{
var resolver = Substitute.For<ISecretResolver>();
resolver.GetAsync(Arg.Any<SecretName>(), Arg.Any<CancellationToken>())
.Returns((string?)null, "seeded-later");
var provider = Create(resolver);
await Assert.ThrowsAsync<InvalidOperationException>(
async () => await provider.GetAsync("site-a", CancellationToken.None));
Assert.Equal("seeded-later", await provider.GetAsync("site-a", CancellationToken.None));
}
[Fact]
public async Task Invalidate_DropsTheCachedKey_SoARotatedKeyIsRead()
{
var resolver = Substitute.For<ISecretResolver>();
resolver.GetAsync(Arg.Any<SecretName>(), Arg.Any<CancellationToken>())
.Returns("old", "rotated");
var provider = Create(resolver);
Assert.Equal("old", await provider.GetAsync("site-a", CancellationToken.None));
provider.Invalidate("site-a");
Assert.Equal("rotated", await provider.GetAsync("site-a", CancellationToken.None));
}
[Fact]
public async Task KeysAreScopedPerSite_SoOneSiteNeverPresentsAnothersKey()
{
// The whole reason the design rejected a single fleet-wide key: blast radius.
var resolver = Substitute.For<ISecretResolver>();
resolver.GetAsync(new SecretName("SB-GRPC-PSK-site-a"), Arg.Any<CancellationToken>())
.Returns("key-a");
resolver.GetAsync(new SecretName("SB-GRPC-PSK-site-b"), Arg.Any<CancellationToken>())
.Returns("key-b");
var provider = Create(resolver);
Assert.Equal("key-a", await provider.GetAsync("site-a", CancellationToken.None));
Assert.Equal("key-b", await provider.GetAsync("site-b", CancellationToken.None));
}
}
@@ -38,6 +38,9 @@ public class StartupValidatorTests
["ScadaBridge:Database:SiteDbPath"] = "./data/scadabridge.db", ["ScadaBridge:Database:SiteDbPath"] = "./data/scadabridge.db",
["ScadaBridge:Cluster:SeedNodes:0"] = "akka.tcp://scadabridge@site-a-node1:8082", ["ScadaBridge:Cluster:SeedNodes:0"] = "akka.tcp://scadabridge@site-a-node1:8082",
["ScadaBridge:Cluster:SeedNodes:1"] = "akka.tcp://scadabridge@site-a-node2:8082", ["ScadaBridge:Cluster:SeedNodes:1"] = "akka.tcp://scadabridge@site-a-node2:8082",
// T0.3: the gRPC control plane is fail-closed, so a Site node without a preshared
// key serves nothing while still looking healthy. Required at boot for that reason.
["ScadaBridge:Communication:GrpcPsk"] = "test-site-control-plane-key",
}; };
[Fact] [Fact]
@@ -56,6 +59,44 @@ public class StartupValidatorTests
Assert.Null(ex); Assert.Null(ex);
} }
[Fact]
public void SiteWithoutGrpcPsk_FailsValidation()
{
// The failure this prevents is silent: ControlPlaneAuthInterceptor refuses every
// SiteStream call without a key, so the node boots, joins its pair, reports healthy —
// and serves no live streams, no audit pulls and no cached-telemetry ingest. Central
// sees a site that is up and answering heartbeats but never sends anything.
var values = ValidSiteConfig();
values.Remove("ScadaBridge:Communication:GrpcPsk");
var config = BuildConfig(values);
var ex = Assert.Throws<InvalidOperationException>(() => StartupValidator.Validate(config));
Assert.Contains("GrpcPsk", ex.Message);
}
[Fact]
public void SiteWithBlankGrpcPsk_FailsValidation()
{
// Whitespace is not a key. An empty-string value would otherwise satisfy a
// key-present check while leaving the interceptor in its fail-closed state.
var values = ValidSiteConfig();
values["ScadaBridge:Communication:GrpcPsk"] = " ";
var config = BuildConfig(values);
var ex = Assert.Throws<InvalidOperationException>(() => StartupValidator.Validate(config));
Assert.Contains("GrpcPsk", ex.Message);
}
[Fact]
public void CentralWithoutGrpcPsk_PassesValidation()
{
// Central holds one key PER SITE (SitePsks / the secret store), not a single key of
// its own, so this setting is meaningless there and must not be required.
var config = BuildConfig(ValidCentralConfig());
Assert.Null(Record.Exception(() => StartupValidator.Validate(config)));
}
[Fact] [Fact]
public void MissingRole_FailsValidation() public void MissingRole_FailsValidation()
{ {
@@ -3643,7 +3643,8 @@ public class ManagementActorTests : TestKit, IDisposable
private sealed class TrackingGrpcFactory : SiteStreamGrpcClientFactory private sealed class TrackingGrpcFactory : SiteStreamGrpcClientFactory
{ {
public TrackingGrpcFactory() : base(NullLoggerFactory.Instance) { } public TrackingGrpcFactory() : base(NullLoggerFactory.Instance) { }
protected override SiteStreamGrpcClient CreateClient(string grpcEndpoint) => new TrackingClient(grpcEndpoint); protected override SiteStreamGrpcClient CreateClient(string siteIdentifier, string grpcEndpoint)
=> new TrackingClient(grpcEndpoint);
} }
/// <summary> /// <summary>