dohertj2/ScadaBridge
1
0
Fork 0
Code Issues 2 Pull Requests 1 Actions Packages Projects Releases Wiki Activity

Compare commits

base: dohertj2/ScadaBridge:fix/cached-telemetry-drain-hot-loop
Branches Tags
dohertj2/ScadaBridge:main dohertj2/ScadaBridge:feat/grpc-sitecommand dohertj2/ScadaBridge:feat/grpc-central-control dohertj2/ScadaBridge:feat/grpc-phase0-psk dohertj2/ScadaBridge:fix/cached-telemetry-drain-hot-loop dohertj2/ScadaBridge:feat/localdb-phase2 dohertj2/ScadaBridge:feat/localdb-phase1 dohertj2/ScadaBridge:feat/secrets-0.2.0
..
compare: dohertj2/ScadaBridge:f7c781194039c6ced08d77c2de0edc3e4cab7b59
Branches Tags
dohertj2/ScadaBridge:main dohertj2/ScadaBridge:feat/grpc-sitecommand dohertj2/ScadaBridge:feat/grpc-central-control dohertj2/ScadaBridge:feat/grpc-phase0-psk dohertj2/ScadaBridge:fix/cached-telemetry-drain-hot-loop dohertj2/ScadaBridge:feat/localdb-phase2 dohertj2/ScadaBridge:feat/localdb-phase1 dohertj2/ScadaBridge:feat/secrets-0.2.0

24 Commits
fix/cached-telemetry-drain-hot-loop .. f7c7811940

Author SHA1 Message Date
Joseph Doherty f7c7811940 docs(grpc): Phase 1A live gate — PASS (3 RPCs + restart-reconcile + coexistence); records the Kestrel-drop defect 2026-07-22 19:55:09 -04:00
Joseph Doherty 0e162cb250 fix(grpc): central Kestrel gRPC listener was dropping the :5000 HTTP surface (T1A.2 regression) 
Caught on the Phase 1A rig proof: central-a logged only "Now listening on:
http://[::]:8083" and nothing else. Central UI, the Management + Inbound API, and
the /health/* endpoints Traefik + IActiveNodeGate depend on were all gone, with no
startup error — the node booted, joined the cluster and served gRPC fine.

Cause: calling options.ListenAnyIP in ConfigureKestrel puts Kestrel into
explicit-endpoints mode, which SUPPRESSES the URLs from ASPNETCORE_URLS/--urls
entirely — it is not additive, contrary to the comment T1A.2 shipped. Central's whole
HTTP/1 surface lives on that URL (http://+:5000 on the rig, a different port in prod),
so binding only the gRPC port silently deleted it. The site branch has the same shape
but no ASPNETCORE_URLS surface to lose — it binds every port it needs explicitly.

Fix: parse the port(s) from the configured URLs and re-declare them (Http1AndHttp2)
alongside the gRPC port (Http2) in the one ConfigureKestrel call. New
Program.ParseHttpBindPorts + CentralHttpBindPortsTests (14 cases: wildcard/ipv6/
hostname hosts, multi-URL, de-dupe, scheme-default, null/blank, unparseable-skipped).

Why no test caught it originally: unit/E2E tests use TestServer, which never binds
real Kestrel. Only a live node exposes the missing listener — which is exactly what
the rig proof is for.
 2026-07-22 19:50:32 -04:00
Joseph Doherty 33b15f10a4 feat(grpc): site-side ICentralTransport seam + gRPC transport (T1A.3) 
Introduce ICentralTransport as the site->central choke point inside
SiteCommunicationActor. The seven site->central sends (notification submit/
status, audit + cached-telemetry ingest, reconcile, health, heartbeat) now
delegate to an injected transport instead of owning ClusterClient.Send inline.

- AkkaCentralTransport: verbatim extraction of today's ClusterClient.Send path,
  including the exact sender-forwarding that routes central's reply straight
  back to the waiting Ask. Default when no transport is injected -> behaviour
  unchanged, existing SiteCommunicationActorTests pass as-is.
- GrpcCentralTransport + CentralChannelProvider: dial CentralControlService with
  sticky failover + background failback (1s-doubling-cap-60s), PSK +
  x-scadabridge-site via ControlPlaneCredentials, per-call deadlines mirroring
  today's Ask timeouts. Cross-node retry ONLY on provably-unsent
  connect failures; never on DeadlineExceeded. Heartbeat stays fire-and-forget.
- StaticSitePskProvider: site's single own-key provider (fail-closed).
- CommunicationOptions: CentralTransport flag (default Akka) + CentralGrpcEndpoints
  (validator: required when transport=Grpc). Host selects the impl; the
  ClusterClient is created only on the Akka path.

Tests: actor-with-fake-transport (7 delegations + fault routing + heartbeat
no-fault), AkkaCentralTransport sender-forwarding, GrpcCentralTransport over
in-process TestServer (failover flip, sticky, failback, PSK+header, deadline,
no-retry-on-deadline), validator. Communication.Tests 371 green, Host.Tests 391
green; the three above-seam suites pass unmodified.
 2026-07-22 19:29:14 -04:00
Joseph Doherty 780bb9c369 feat(grpc): host CentralControlService on the central node (T1A.2) 
Central now ALSO listens for the seven site→central control messages over
gRPC, alongside the existing ClusterClient path. Nothing flips to gRPC yet —
sites keep CentralTransport=Akka (T1A.3's job); central simply starts also
accepting.

- CentralControlGrpcService (Communication.Grpc): decodes each RPC onto the
  SAME in-process message the ClusterClient path carries, Asks the existing
  CentralCommunicationActor (zero handler-logic changes), encodes the reply via
  the T1A.1 mapper. Readiness-gated like SiteStreamGrpcServer.SetReady —
  Unavailable until AkkaHostedService hands the actor over. Heartbeat stays
  fire-and-forget (Tell, always-OK, never gated on readiness). Ingest reuses the
  shared SiteStreamGrpcServer.AuditIngestAskTimeout constant. Fault→status
  mapping is retry-aware: Unavailable (never dispatched, safe to cross-node
  retry) vs DeadlineExceeded/Internal (it ran, do not re-send elsewhere).

- CentralControlAuthInterceptor (Host): a SEPARATE interceptor class, not a
  variant constructor on ControlPlaneAuthInterceptor. Central's model is per-site
  (verify the Bearer token against the key for the site in the required
  x-scadabridge-site header, via ISitePskProvider) where a site verifies its one
  own-key — a genuinely different model. Fail-closed on every branch: missing or
  blank header, unresolvable key, and mismatched token all → PermissionDenied,
  never pass-through. One public constructor only (the explicit-prefix ctor is
  internal), pinned by a reflection test — a second public ctor makes
  Grpc.AspNetCore's GetFactory() throw per-call and silently disables the gate.

- Explicit Kestrel h2c listener on new option ScadaBridge:Node:CentralGrpcPort
  (default 8083, symmetric with sites), mirroring the Site branch. Additive to
  central's :5000 HTTP/1 surface, which is untouched — gRPC does NOT go through
  Traefik (HTTP/1 only). Registered by type on AddGrpc; service mapped with
  MapGrpcService. Port range-validated by NodeOptionsValidator.

- Rig: publish the central gRPC port 9013:8083 / 9014:8083 on both central nodes
  so a later task can exercise it.

Tests: CentralControlEndToEndTests (Host.Tests, TestServer + real interceptor +
real service over a stub actor) proves auth positives/negatives are
distinguishable and covers unary + the ingest bridge shapes; the interceptor is
registered BY TYPE, never in DI. CentralControlAuthInterceptorTests pins the
per-site gate + one-public-ctor invariant. CentralControlGrpcServiceTests
(Communication.Tests, TestKit) covers the readiness gate, fire-and-forget
heartbeat, and the DeadlineExceeded-vs-Unavailable status mapping. No active
<Protobuf> item. Communication.Tests (356) + Host.Tests (384) green.
 2026-07-22 18:53:59 -04:00
Joseph Doherty d7455577a8 feat(grpc): central_control.proto + DTO mapper for the 7 site→central control RPCs (T1A.1) 
Phase 1A of the ClusterClient→gRPC migration needs a wire contract for the
seven messages SiteCommunicationActor forwards to /user/central-communication.
This lands the contract and its mapper only — hosting (T1A.2) and the site-side
transport seam (T1A.3) follow.

`Protos/central_control.proto` (package scadabridge.centralcontrol.v1, service
CentralControlService) declares SubmitNotification, QueryNotificationStatus,
IngestAuditEvents, IngestCachedTelemetry, ReconcileSite, ReportSiteHealth and
Heartbeat. Note the direction is the inverse of SiteStreamService: here the site
dials and central serves.

Decisions worth recording:

- The two ingest RPCs IMPORT sitestream.proto and reuse AuditEventBatch /
  CachedTelemetryBatch / IngestAck rather than redeclaring them. The site
  telemetry actor already builds those messages, so a second copy would fork one
  wire contract into two kept in lockstep by hand. ForwardState / IngestedAtUtc
  stay off-wire exactly as they are today.
- Heartbeat replies google.protobuf.Empty — it is fire-and-forget and must never
  surface a fault onto the heartbeat timer path.
- The three NULLABLE SiteHealthReport collections travel inside single-field
  wrapper messages (ConnectionEndpointMapDto / TagQualityMapDto /
  NodeStatusListDto). proto3 cannot express presence on repeated/map fields, but
  null and empty genuinely differ here — SiteHealthCollector emits
  `ClusterNodes: _clusterNodes?.ToList()` and the central health surface reads
  null as "not reported", not as "reported empty". Same reasoning drives the
  BoolValue/Int64Value/DoubleValue wrappers on LocalDbReplicationConnected,
  LocalDbOplogBacklog and the two age gauges, whose docs are explicit that null
  is not zero/false.
- ConnectionHealthEnum reserves 0 for UNSPECIFIED instead of mapping Connected
  onto it, and the decoder resolves anything unknown to ConnectionHealth.Error.
  An unrecognised connection state must not render as "healthy".
- Guid? execution ids travel as "D" strings with empty meaning null; a malformed
  non-empty value throws rather than being laundered into "no correlation".
- DateTimeOffset normalizes to a UTC instant (a protobuf Timestamp has no
  offset). Lossless in practice — every producer stamps UTC — and documented +
  asserted rather than left implicit.

SiteCallDtoMapper gains a ToDto(SiteCall) overload. Its doc comment previously
asserted such a method "would be dead code"; that held only while ClusterClient
was the sole path from IngestCachedTelemetryCommand (which carries SiteCall, not
SiteCallOperational) to central. Comment corrected alongside.

Golden tests round-trip every message through a real protobuf encode/decode —
DTO → proto → bytes → proto → DTO — with a fully-populated case and a
null/empty/minimal case for every optional member. Verified to have teeth by
mutation: dropping a scalar, collapsing an empty nullable collection to absent,
and nulling a gauge each fail a test.

Codegen stays CHECKED IN under CentralControlGrpc/ (protoc segfaults in the
linux_arm64 Docker image); no active <Protobuf> item is committed.
docker/regen-proto.sh is generalized to `regen-proto.sh [sitestream|
centralcontrol|all]` — it now injects the ItemGroup rather than unwrapping a
comment, so it no longer depends on there being exactly one Protobuf line, and
it restores the csproj verbatim on every exit path.
 2026-07-22 18:28:27 -04:00
Joseph Doherty aa7c5cd138 docs(plans): tick Phase 0 DoD — PR #25 merged to main @ 3fa95555 2026-07-22 18:12:01 -04:00
Joseph Doherty 3fa955556d docs(grpc): record the Playwright result and root-cause both failures 
Phase 0's gate doc now carries the full suite picture, not just the rig checks.

Playwright: 170 pass / 2 fail / 1 skip of 173. Both failures were run down to
root cause and both are pre-existing on main, unrelated to this branch (which
touches no EF, CentralUI, Transport or ManagementService file):

- TransportImportTests is a REAL production bug: BundleImporter.cs:1298 opens a
  user-initiated transaction while the central context has EnableRetryOnFailure,
  so SqlServerRetryingExecutionStrategy refuses the split query inside it and
  bundle import fails against real MS SQL. The unit/integration suite cannot see
  it -- the in-memory EF provider has no retrying strategy and BeginTransaction
  is a no-op there.

- SmsNotificationE2ETests is a stale fixture: SID 'ACtest123' (2026-06-19) vs the
  ^AC[0-9a-fA-F]{32}$ guard added 2026-07-10 (40088a21). Failing since then, which
  has also silenced everything after the toast assertion -- including the
  secret-non-leak check on the Auth Token.

Also records that the earlier 44-failure run is void: a concurrent deploy.sh was
recreating the cluster underneath it.

Neither is fixed here; both are out of scope for a PSK-auth branch.
 2026-07-22 18:09:07 -04:00
Joseph Doherty 6ef8c7d70a docs(grpc): Phase 0 live gate PASS — record results, the inert-gate defect, and the trap for phases 1A/1B 
The gate's first run failed on a defect the green suite could not see: two public
constructors on ControlPlaneAuthInterceptor made Grpc.AspNetCore's activation
throw per call, so correct key, wrong key and no key all produced identical
errors. Recorded in full because the symptom (Unknown / "Exception was thrown by
handler") points at the handler, not at auth, and because phases 1A/1B both add
services to this same interceptor — they must extend DefaultGatedPrefixes rather
than add a second public constructor.

Also records what the gate does NOT cover: live streaming under load, key
rotation on a running pair, and docker-env2 (keyed but neither redeployed nor
gated).
 2026-07-22 18:01:11 -04:00
Joseph Doherty 228ff8b428 fix(grpc): one public constructor on ControlPlaneAuthInterceptor — two made the gate inert 
Caught by the Phase 0 live gate, not by the suite.

Grpc.AspNetCore registers the interceptor BY TYPE, and
InterceptorRegistration.GetFactory() throws "Multiple constructors accepting all
given argument types have been found" when more than one public constructor is
applicable. The interceptor had two: the DI one and a prefix-set overload added
for later phases.

The failure mode is nasty. The throw happens inside the interceptor pipeline on
every call, so nothing fails at startup — the site node boots, joins, reports
healthy. Every gated call then dies with Unknown / "Exception was thrown by
handler", which reads as a handler bug rather than an auth bug. And it fails
OPEN in the sense that matters least and closed in the sense that matters most:
no call is ever authorized, but no call is ever correctly REFUSED either, so the
rig showed identical errors for a correct key, a wrong key and no key at all.
Live evidence, site-a: three PullAuditEvents calls, three identical
InvalidOperationExceptions in the node log.

Fix: the prefix-set constructor is internal (Host.Tests already has
InternalsVisibleTo). Later phases extend DefaultGatedPrefixes rather than adding
a second public registration shape.

Why the tests missed it, and what changed: ControlPlaneAuthEndToEndTests
registered the interceptor with AddSingleton alongside AddGrpc, so DI handed
back the instance and Grpc.AspNetCore's activation path — the thing that throws
— never ran. The harness now registers exactly as Program.cs does, by type and
not in DI. Plus a direct reflection assertion that the type has exactly one
public constructor, since that is the real invariant and it is cheap to pin.
 2026-07-22 17:56:51 -04:00
Joseph Doherty 2ee84af1c0 feat(grpc): PSK-authenticate the site gRPC control plane; drop the vestigial management receptionist registration 
Phase 0 of the ClusterClient→gRPC migration
(docs/plans/2026-07-22-clusterclient-to-grpc-plan.md). Standalone hardening: it
closes a gap that exists today and is a precondition for moving command/control
onto gRPC in later phases.

T0.1 — delete the ManagementActor ClusterClientReceptionist registration.
It was built for an out-of-cluster CLI that was never written: the shipped CLI
speaks HTTP Basic to /management, which asks the actor in-process through
ManagementActorHolder. Nothing in the repo ever sent to /user/management. The
actor still runs there; only the cross-boundary advertisement is gone. Six
documents claimed the CLI used ClusterClient — including the CLI's own README
"Architecture Notes" — and are corrected here rather than left to rot.

T0.2 — record, do not port, the dead integration-routing path.
IntegrationCallRequest is unwired at BOTH ends: RouteIntegrationCallAsync has
zero callers anywhere, and RegisterLocalHandler(Integration, …) appears only in
a test, so production always answers "Integration handler not available". It is
excluded from the gRPC contract (28 of 29 commands migrate) rather than
enshrined on an additive-only wire format, and deleting it during a
transport migration would mix a behavioural change into a change whose whole
value is that behaviour is identical. See
docs/known-issues/2026-07-22-integration-call-routing-is-dead-code.md.

T0.3 — preshared-key authentication on SiteStreamService.
The service shipped with no auth at all: plaintext h2c, no interceptor, so
anything that could reach a site node's :8083 could open a live data stream or
read audit rows back via PullAuditEvents/PullSiteCalls. ControlPlaneAuthInterceptor
now gates /sitestream.SiteStreamService/ — modeled on LocalDbSyncAuthInterceptor
(constant-time compare, fail-closed, PermissionDenied) but gating a SET of
service prefixes so phases 1A/1B add services rather than interceptors. LocalDb
sync keeps its own separate key: it authenticates the pair partner, not central,
and collapsing the two would make a site's central-facing key also admit writes
into its database.

Keys are per site (SB-GRPC-PSK-<siteId>), never fleet-wide, so a compromised
site yields only its own. Central attaches them through ControlPlaneCredentials,
which binds CallCredentials to the channel — covering unary and streaming
uniformly, and letting the key resolve asynchronously, which a client
interceptor could not do without blocking. All three central→site channel
creation sites go through it (SiteStreamGrpcClient and both audit pull invokers);
the pull invokers' channel caches are re-keyed by (site, endpoint) because
credentials are per-site and bound to the channel.

Two decisions beyond the plan:

  * StartupValidator now requires GrpcPsk on Site nodes. The plan specified only
    the runtime gate, but fail-closed with no boot check produces a node that
    joins, answers heartbeats and reports healthy while refusing every stream,
    audit pull and telemetry ingest — silent and total. Same reasoning as the
    existing inbound API-key pepper rule.

  * Added Communication:SitePsks as a central-side key map. The plan assumed
    central would read the store, seeded via a dev KEK; the docker rig
    deliberately boots with no master key, so store-only resolution would leave
    it unable to dial its own sites. The store stays primary — it is the only
    source that can serve a site added at runtime — with the map covering
    key-less hosts and one-off pins. Neither source falling back to
    "unauthenticated" is the invariant.

T0.4 — dev keys on both rigs and tests.
34 tests. The seven that matter most exercise a real in-process gRPC stack over
TestServer: the unit tests on either side of the wire would both stay green if
the halves disagreed, and gRPC refuses call credentials on a plaintext channel
by default — the UnsafeUseInsecureChannelCallCredentials opt-in is only provable
by making a real call. They confirm correct key passes on unary AND streaming,
wrong key and no-credentials both get PermissionDenied, and an unresolvable key
fails the call with nothing reaching the service.

OPERATIONAL: a site node upgraded to this build without a key will not boot.
That includes the gitignored deploy/wonder-app-vd03/ overlay.
 2026-07-22 17:51:09 -04:00
Joseph Doherty f1ad967083 docs(plans): ClusterClient→gRPC-only migration plan (phases 0–5 + tasklist) 
Complete executable plan: PSK-from-Secrets auth (Phase 0), central_control +
site_command proto contracts behind transport seams inside the two
communication actors (1A ∥ 1B in worktrees), per-direction cutover flags,
ClusterClient/receptionist deletion, 8-check live gate. Design doc:
scadaproj/scadabridge_clusterclient_to_grpc.md (§7 = deep-dive corrections).
 2026-07-22 17:10:33 -04:00
Joseph Doherty 654df8abc2 docs(cluster): site-pair manual failover runbook + component spec 2026-07-22 07:49:51 -04:00
Joseph Doherty c8e2f4da02 feat(cluster): site-pair manual failover relayed from the central UI (Task 10) 
Central and each site are SEPARATE Akka clusters, so central cannot act on a
site's membership -- it asks. New TriggerSiteFailover/SiteFailoverAck contract
travels the existing ClusterClient command/control channel (mirroring the
RetryParkedOperation relay); the site's own SiteCommunicationActor performs the
graceful Leave and acks the outcome.

- ClusterFailoverCoordinator moved out of Host into Communication/ClusterState,
  beside ActiveNodeEvaluator. Both paths now share ONE oldest-Up implementation;
  SiteCommunicationActor cannot reference Host, and the two definitions must not
  drift or the node asked to leave stops being the singleton host.
- Site scope is the SITE-SPECIFIC role (site-{SiteId}), not the base Site role --
  site singletons are placed on the former, so the base role would move the wrong
  node. Pinned by a unit test asserting the role string and by a real-cluster test.
- Site-side guards: refuses a command addressed to another site (a misroute must
  never fail over a site the operator did not select), refuses when there is no
  peer, and reports a fault as an ack rather than throwing into supervision --
  a restart there would drop central's Ask into a bare timeout and lose the reason.
- Ack is sent before the Leave takes effect so it still reaches central.
- UI: the same control now serves both scopes via a SiteId parameter. The site
  confirmation deliberately does NOT claim the admin's page will disconnect --
  it won't, and crying wolf there devalues the central warning that is real. A
  site refusal and an unreachable site surface distinctly.
- Rolling upgrade: a site on an older binary has no handler, so the message
  dead-letters and the Ask times out, reported as "site did not respond". That
  is the honest outcome; documented on the contract.

Fallout fixed: HealthPageTests now renders the page inside
CascadingAuthenticationState with the real policy set and IAuthorizationService,
because the cards embed an AuthorizeView. That mirrors production, where the
layout supplies the cascading value.
 2026-07-22 07:48:56 -04:00
Joseph Doherty d66e0d585f chore(plans): mark self-first ordering + manual failover tasks complete (live gate PASS) 2026-07-22 07:14:11 -04:00
Joseph Doherty caf14a3e03 feat(ui): admin manual-failover control on the health page 
CentralFailoverControl lives in Components/Health/ (matching AuditKpiTiles /
SiteCallKpiTiles) rather than inline in Health.razor, so it is testable without
standing up the whole dashboard's DI graph.

- Admin-gated via AuthorizeView + RequireAdmin. The Health page itself is
  intentionally all-roles, so the gate belongs on the control, not the page.
- Disabled with an explanatory title when the pair has no online standby;
  the authoritative guard remains server-side against live cluster membership.
- Confirmation dialog (IDialogService, the page idiom) warns that singletons
  hand over, in-flight work on the active node is interrupted, and THIS PAGE
  will disconnect and reconnect against the new active node -- Traefik routes
  the UI to the node being restarted, so a working failover otherwise reads as
  a crash the admin caused.
- A refused failover (service returns null) surfaces the refusal; the UI never
  reports a failover that did not happen.

7 bUnit tests. Two harness requirements that bit first: AuthorizeView needs a
cascading AuthenticationState (the app supplies it from the layout), and
BunitContext pre-registers a placeholder IAuthorizationService that throws on
policy evaluation -- both handled the same way NavMenuTests documents.

Runbook paragraphs added to Component-ClusterInfrastructure.md (new Manual
Failover section) and docker/README.md.
 2026-07-22 07:00:10 -04:00
Joseph Doherty f679d5c749 feat(cluster): manual central failover service — graceful Leave of the oldest Up member 
Admin-triggered failover of the central pair. IManualFailoverService is declared in
CentralUI (plain strings, so that project stays Akka-free); AkkaManualFailoverService
implements it in the Host and is registered only in the Central branch.

- Leave, never Down: singletons hand over instead of being killed.
- Target = oldest Up member with the Central role, mirroring ActiveNodeEvaluator, so
  the node acted on is exactly the one hosting the singletons (never the leader,
  whose address-ordered definition diverges from singleton placement after a restart).
- Peer guard: returns null when fewer than 2 Up Central members — failing over a lone
  node is an outage, not a failover.
- Audited BEFORE the Leave is issued via ICentralAuditWriter: the acting node can be
  the one that goes away, and an audit written after could be lost to the shutdown it
  describes. Best-effort — audit failure never blocks the failover.

New audit taxonomy: AuditChannel.Cluster + AuditKind.ManualFailover (operator-initiated
topology actions are not script trust-boundary crossings, but are exactly what an audit
log exists to attribute). Lock-in tests updated 5->6 channels, 16->17 kinds.

alog.md §4 updated per the lock-in tests' contract. Both tables were already stale --
the Channel row omitted SecuredWrite and the Kind table claimed 10 while the code had
16 -- so they are completed here, not merely appended to.

ManualFailoverTests: 3 real-cluster tests (oldest leaves + survivor takes over, peer
guard refuses with a positive still-running assert, dry-run probe does not perturb).
 2026-07-22 06:55:15 -04:00
Joseph Doherty eca69505bc docs(plans): record the self-form watchdog rejection and the self-first ordering that replaced it 2026-07-22 06:33:28 -04:00
Joseph Doherty 4a6341d871 feat(cluster): self-first seed ordering closes the boot-alone outage gap 
Every node now lists ITSELF as seed-nodes[0] and its partner second. Akka runs
FirstSeedNodeProcess -- the only bootstrap path that can form a NEW cluster when
no peer answers InitJoin -- exclusively for seed-nodes[0]; every other node runs
JoinSeedNodeProcess and retries InitJoin forever. That is why a lone cold-starting
central-b never came Up (the "registered outage gap"), and self-first ordering
closes it using Akka's own protocol.

- 6 node appsettings swapped (the *-node-b configs; the -a nodes were already
  self-first). All 14 shipped node configs now satisfy the invariant.
- StartupValidator enforces it at boot, comparing host AND port -- the invariant
  fails silently when broken, so it is enforced loudly. NOTE: the gitignored
  deploy/wonder-app-vd03/ overlay must be reordered before its next deploy or
  that node will refuse to boot.
- SelfFirstSeedBootstrapTests: real in-process clusters at production
  failure-detection timings, incl. a falsifiability control proving the OLD
  peer-first ordering never forms.

Rejected alternative (implemented, measured, discarded): an external self-form
timer calling Cluster.Join(SelfAddress) after a window. It sits outside Akka's
join handshake and so cannot tell "no seed answered" from "a seed answered and
the join is in flight". On a routine standby restart the peer is alive but the
join stalls behind removal of the node's own stale incarnation; a Join(self)
during TryingToJoin abandons the in-flight join and forms a second cluster at
the same address -- still split after 90s. Docs that claimed self-first ordering
was unsafe for simultaneous cold start are corrected: while mutually reachable
the InitJoin handshake converges them to one cluster (measured).
 2026-07-22 06:32:00 -04:00
Joseph Doherty 69b3ccfc37 docs(components): correct the three component docs against the code they describe 
CLAUDE.md was corrected in 34227991 but the component docs were not, so the
same understatements — plus several outright wrong statements — survived where
a reader is most likely to meet them.

ClusterInfrastructure.md carried the worst of it. It described active/standby as
cluster leadership and showed an IsActiveNode snippet doing
`cluster.State.Leader == self.Address`. No such code exists: ActiveNodeGate
returns ClusterActivityEvaluator.SelfIsOldest, and ActiveNodeEvaluator's own doc
comment says "never cluster.State.Leader". A second snippet (siteCallAuditShutdown
.AddTask) described a hand-rolled drain that has since been folded into
SingletonRegistrar.Start. Both snippets are replaced with what the code does. The
central singleton table listed 3 of the 7 registered singletons. The downing
section still described keep-oldest as the strategy and omitted
downing-provider-class entirely; it now shows the branching block cf3bd52f
introduced, with the accepted dual-active trade stated rather than the old
"impossible to boot" framing. Note the requirements-side spec,
docs/requirements/Component-ClusterInfrastructure.md, was already rewritten by
cf3bd52f — this is the components-side doc, which was untouched.

Communication.md described two transports; there are three — the deployment
config fetch over HTTP was missing. Each row now names which side dials, because
the gRPC entry gave a data direction (site to central) without saying who hosts
the server, which reads as the opposite of the truth: the only MapGrpcService in
the tree is in Program.cs's Site branch, and central dials in. The SiteEnvelope
snippet called DeployInstanceAsync, which is not a member of CommunicationService;
the heartbeat bullet claimed a cluster-leadership check; the proto summary listed
4 of 6 RPCs.

DeploymentManager.md still said the pipeline sends DeployInstanceCommand carrying
FlattenedConfigurationJson. It stages a PendingDeployment and sends a
RefreshDeploymentCommand; the config travels over the HTTP fetch. That is the
change the 128 KB frame-size issue forced, and the doc predated it.

Two of the five briefed drifts turned out not to be errors: nothing claimed the
transports were authenticated, and nothing asserted per-site ActorSystem names —
both were simply unstated. They are now stated, since silence about an
unauthenticated boundary is its own problem.

Docs only; no code changed.

Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
 2026-07-21 18:45:56 -04:00
Joseph Doherty cf3bd52f93 feat(cluster): auto-down downing strategy — either-node crash now fails over (owner decision 2026-07-21: availability over partition-safety) 
Two-node keep-oldest could NEVER survive a crash of the oldest/active node:
Akka.NET 1.5.62 KeepOldest.OldestDecision only lets down-if-alone rescue a
side with >= 2 members, so the 1-vs-1 survivor takes DownReachable and downs
ITSELF — proven live on the rig ('SBR took decision ... including myself')
before this change. static-quorum(1) is worse (IsTooManyMembers -> DownAll);
keep-majority just re-keys the fatal crash to the lowest address.

SplitBrainResolverStrategy gains 'auto-down' (new default): BuildHocon emits
Akka's AutoDowning provider with auto-down-unreachable-after = StableAfter.
The leader among the REACHABLE members downs the unreachable peer, so the
survivor takes over singletons and /health/active in ~25s regardless of which
node died. Accepted trade (explicit owner decision): a real network partition
runs dual-active until an operator restarts one side. keep-oldest remains
supported; DownIfAlone validation is now scoped to it.

Live drill on the rebuilt rig: active-crash TAKEOVER in 28s (victim still
down; all 7 singletons Younger->Oldest), standby-crash removal 27s with 0
routing blips; victims rejoin as standby in 2s. New real-cluster tests pin
both directions (SbrFailoverTests.AutoDown_*); TwoNodeClusterFixture gains a
strategy knob. All 16 appsettings flipped (src, docker, docker-env2, and the
gitignored wonder-app-vd03 overlay on disk — owner must sync to the host).
Docs: decision record docs/plans/2026-07-21-auto-down-availability-decision.md,
Component-ClusterInfrastructure downing section rewritten, drill + README
reworked (active mode now asserts takeover), deferred-work SBR row resolved.
 2026-07-21 10:53:40 -04:00
Joseph Doherty dced0d2794 fix(deps): pin System.Security.Cryptography.Xml 10.0.10 — four new NU1903 advisories on the 10.0.7 DataProtection transitive broke fresh restores (docker image build) 
Same pattern as the SQLitePCLRaw pin: direct PackageReference at the chain's entry
project (ConfigurationDatabase). Bumping the DataProtection parent instead was tried
and rejected — 10.0.10 floors Microsoft.Extensions.*/EF at 10.0.10 (NU1605 cascade).
 2026-07-21 10:53:26 -04:00
Joseph Doherty 34227991ea docs(claude): correct five understatements about the inter-cluster boundary 
Found while OtOpcUa researched this repo as the model for its own per-cluster
mesh work. Each was verified against the code, not inferred from the doc.

Three transports cross the boundary, not two: ClusterClient, gRPC, and plain
token-gated HTTP for the deploy config fetch (DeploymentConfigEndpoints,
X-Deployment-Token, AllowAnonymous with the per-deployment token as the entire
security boundary).

The gRPC direction is inverted from the data flow. Data moves site to central,
but each SITE hosts the gRPC server and central dials in — MapGrpcService appears
only in the Site branch of Program.cs. There is no gRPC server on central, which
is why the two Ingest* RPCs are dead in practice. Also records the 6-RPC surface
(the doc implied 2), the (site, endpoint) factory key that fixed an arch-review
High, and the vendored-generated-code caveat.

Active/standby is ActiveNodeEvaluator.SelfIsOldestUp — the OLDEST Up member, and
explicitly never cluster.State.Leader, because leadership diverges from singleton
placement permanently after a restart-and-rejoin and both sides claim it during a
partition. The equivalence oldest-Up == singleton placement is the design, and it
was not stated anywhere in this file.

All clusters share one ActorSystem name ("scadabridge", hardcoded at
AkkaHostedService.cs:191); they are separate clusters only by seed partitioning.
Required, not incidental — Akka.Remote address matching means ClusterClient could
not reach a differently-named system. Site nodes also carry two roles, base plus
site-{SiteId}, with singletons scoped to the site-specific one.

Neither inter-cluster transport is authenticated or encrypted: no Akka TLS or
secure cookie, gRPC is h2c, and LocalDbSyncAuthInterceptor gates only the LocalDb
sync path — so the whole SiteStreamService surface, including the two Pull RPCs
returning audit rows, is reachable by anyone who can hit :8083. The boundary
assumes a trusted network; that assumption deserves to be explicit.

Also promotes two things from docs/ into CLAUDE.md because they change decisions:
the default 128 KB Akka frame size with log-frame-size-exceeding off and no custom
serializer (a silent single-message drop that leaves the association healthy), and
the registered two-node keep-oldest total-outage gap, whose own drill has a mode
existing "to make the registered gap observable — not to pretend it is covered."

Committed to a branch rather than main; merge at your discretion.
 2026-07-21 10:00:01 -04:00
Joseph Doherty 9d925c3347 Merge chore/localdb-0.1.3: pin ZB.MOM.WW.LocalDb 0.1.3 
Picks up two rebuilt-peer replication fixes found by the OtOpcUa LocalDb
Phase 1 live gate. They matter more here than in OtOpcUa, which replicates 2
tables to this repo's 10:

- 0.1.2 — a converged pair prunes its oplog to empty on ack, and snapshot
  detection read that as 'no gap possible', so a node whose database was lost
  rejoined empty and stayed empty until the next deploy.
- 0.1.3 — with back-fill working, the rebuilt node's own writes were silently
  dropped until its restarted seq counter climbed past the peer's stale
  watermark.

Build clean; SiteRuntime 512 green.

Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
 2026-07-21 02:19:13 -04:00
Joseph Doherty 8c6fc2f886 chore(localdb): pin ZB.MOM.WW.LocalDb 0.1.3 (rebuilt-peer replication fixes) 
Two fixes, both about a node whose LocalDb file is lost. They matter more here
than in OtOpcUa, which replicates 2 tables to this repo's 10.

0.1.2 — a converged pair prunes every oplog row on ack, and snapshot detection
read an empty oplog as "no gap possible". The steady state of a healthy pair
was the one state from which a rebuilt node could never be healed: it rejoined
empty and stayed empty until the next deploy.

0.1.3 — with back-fill working, the rebuilt node's OWN writes turned out to be
silently dropped: last_applied_remote_seq is a watermark in the peer's seq
space, and a rebuilt peer numbers from 1, so the healthy node's stale watermark
made the sender skip its whole oplog.

Both found by the OtOpcUa LocalDb Phase 1 live gate on the docker-dev rig; the
second only became reachable once the first was fixed.

Build clean; SiteRuntime 512, SiteEventLogging 70 green (Host 330,
HealthMonitoring 97 green on 0.1.2, unchanged by the pin).

Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
 2026-07-21 02:18:02 -04:00
117 changed files with 17102 additions and 886 deletions
Expand all files Collapse all files
CLAUDE.md
+11 -4
View File
@@ -134,8 +134,13 @@ Other peers in the `scadaproj` family (see `scadaproj/CLAUDE.md` for details): `
- **Operational constraints (read before upgrading a site pair):** stop and start both nodes TOGETHER — rolling one at a time is no longer supported, since the legacy `SfBufferSnapshot` compatibility handler went with the replicator. And a node offline longer than `LocalDb:Replication:TombstoneRetention` (default 7 days) can resurrect deleted rows on rejoin. See `docs/deployment/topology-guide.md`.
- `LocalDb:Replication:MaxBatchSize` batches by ROW COUNT, not bytes, against a 4 MB gRPC cap — the rig pins it to **16** (~70 KB worst-case `config_json` x 16 ~= 1.1 MB). The 500 default would allow ~35 MB.
- All timestamps are UTC throughout the system.
- Inter-cluster communication uses two transports: ClusterClient for command/control (deployments, lifecycle, subscribe/unsubscribe handshake, snapshots) and gRPC server-streaming for real-time data (attribute values, alarm states). Both CentralCommunicationActor and SiteCommunicationActor registered with receptionist. Central creates one ClusterClient per site using NodeA/NodeB as contact points. Sites configure multiple central contact points for failover. Addresses cached in CentralCommunicationActor, refreshed periodically (60s) and on admin changes. Heartbeats serve health monitoring only.
- gRPC streaming channel: SiteStreamGrpcServer on each site node (Kestrel HTTP/2, port 8083); central creates per-site SiteStreamGrpcClient via SiteStreamGrpcClientFactory. Site entity has GrpcNodeAAddress/GrpcNodeBAddress fields. Proto: sitestream.proto with SiteStreamService, SiteStreamEvent (oneof: AttributeValueUpdate, AlarmStateUpdate). DebugStreamEvent message removed (no longer flows through ClusterClient).
- Inter-cluster communication uses **three** transports, not two: **ClusterClient** for command/control (deployments, lifecycle, subscribe/unsubscribe handshake, snapshots); **gRPC** server-streaming for real-time data (attribute values, alarm states); and **plain token-gated HTTP** for the deployment config itself — notify-and-fetch, the site pulls the config from `DeploymentConfigEndpoints` (`ManagementService/DeploymentConfigEndpoints.cs`) with an `X-Deployment-Token` header, `AllowAnonymous` with the per-deployment token as the entire security boundary. Both CentralCommunicationActor and SiteCommunicationActor registered with receptionist (**per node, not as a singleton** — contact rotation reaches whichever node answers). Central creates one ClusterClient per site using NodeA/NodeB as contact points. Sites configure multiple central contact points for failover. Addresses cached in CentralCommunicationActor, refreshed periodically (60s) and on admin changes. Heartbeats serve health monitoring only. **Discovery is asymmetric by design:** central discovers sites from the *database* (`Site.NodeAAddress`/`NodeBAddress`, refreshable at runtime), sites discover central from *appsettings* (`ScadaBridge:Communication:CentralContactPoints`, static — restart required). **Central never buffers for an unreachable site** — the send is dropped with a warning and the caller's Ask times out; a `ConnectionStateChanged` mechanism built for this was deleted as dead code.
- **All clusters share ONE ActorSystem name**, `"scadabridge"` — hardcoded at `AkkaHostedService.cs:191`. Central and each site are separate clusters *only* by seed-node partitioning. This is required, not incidental: Akka.Remote address matching means a ClusterClient could not reach a differently-named system.
- **`ActiveNodeEvaluator.SelfIsOldestUp` is THE single definition of "active node"** (`Communication/ClusterState/ActiveNodeEvaluator.cs`) — the **oldest Up member** in a role scope, and explicitly **never `cluster.State.Leader`**: leadership (lowest address) is an Akka-internal concept that diverges from singleton placement permanently once the original first node restarts and rejoins, and both sides claim it during a partition. The equivalence *oldest-Up == where `ClusterSingletonManager` places singletons* **is** the design. `ClusterActivityEvaluator.SelfIsOldest`, the S&F delivery gate, `/health/active` and the heartbeat `IsActive` stamp all delegate here.
- Site nodes carry **two Akka roles**: the base `Site` plus a site-specific `site-{SiteId}` (`BuildRoles`, `AkkaHostedService.cs:386`). Singletons scope to the **site-specific** role.
- **The gRPC boundary is authenticated (PSK) as of 2026-07-22; Akka remoting still is not, and nothing is encrypted.** Akka remoting sets no `enable-ssl`, no secure cookie, no `trusted-selection-paths` — so the ClusterClient command/control path remains open to anyone who can reach the remoting port, and the boundary still assumes a trusted network. The gRPC listener stays **h2c**, but `SiteStreamService` is no longer open: `ControlPlaneAuthInterceptor` (`Host/ControlPlaneAuthInterceptor.cs`) gates `/sitestream.SiteStreamService/` — including the `PullAuditEvents`/`PullSiteCalls` RPCs that return audit rows — against a **per-site preshared key**, fail-closed, constant-time compared, alongside the separate `LocalDbSyncAuthInterceptor` on `/localdb_sync.v1.LocalDbSync/` with its own separate key. **Site side:** `ScadaBridge:Communication:GrpcPsk`, in production `${secret:SB-GRPC-PSK-<siteId>}`, and **`StartupValidator` refuses to boot a site node without it** (an unset key would leave the node healthy-looking but serving nothing). **Central side:** `SitePskProvider` resolves `SB-GRPC-PSK-{siteId}` from the secrets store at channel-build time (sites are added at runtime, so no boot-time expansion is possible), with `ScadaBridge:Communication:SitePsks:{siteId}` as an override for hosts running without a master key — the docker rig uses the latter. One key per site, never fleet-wide. A bearer token over h2c is readable and replayable on-path; TLS is the follow-on hardening and needs no change to this design. Introduced by Phase 0 of the ClusterClient→gRPC migration (`docs/plans/2026-07-22-clusterclient-to-grpc-plan.md`).
- **Akka frame size is the default 128 KB with `log-frame-size-exceeding` off**, and no custom serializer is configured — so payload carried over ClusterClient is JSON-escaped a second time by the default Newtonsoft serializer, roughly doubling it. Over the limit the transport drops **that one message** without tearing down the association (heartbeats keep flowing, the site still reports healthy) and the central Ask simply times out. See `docs/known-issues/2026-06-26-deploy-config-exceeds-akka-frame-size.md`; `DeployArtifactsCommand` still carries payload and remains exposed.
- gRPC streaming channel — **note the direction is inverted from the data flow**: data moves site→central, but each **site node hosts the gRPC server** (`SiteStreamGrpcServer`, Kestrel h2c, port 8083, mapped **only in the Site branch** of `Program.cs`) and **central is the client**, dialling in. There is **no gRPC server on central at all**, which is why the two `Ingest*` unary RPCs — documented as a "central-side ingest surface" — are dead in practice (acknowledged in `AkkaHostedService.cs:510-519`); sites reach central over ClusterClient instead. Central creates per-site `SiteStreamGrpcClient` via `SiteStreamGrpcClientFactory`, keyed **`(siteId, endpoint)`** — the key was widened from site-only to fix an arch-review High where one session's NodeA→NodeB flip disposed a channel another session was still using. Site entity has GrpcNodeAAddress/GrpcNodeBAddress fields. Proto: `sitestream.proto`, **6 RPCs** (2 server-streaming: `SubscribeInstance`, `SubscribeSite` (site-wide, alarm-only); 4 unary: `IngestAuditEvents`, `IngestCachedTelemetry`, `PullAuditEvents`, `PullSiteCalls`), `SiteStreamEvent` (oneof: AttributeValueUpdate, AlarmStateUpdate). Field numbers are never reused; evolution is additive only (`AlarmStateUpdate` grew 7→23 fields for the native-alarm mirror). Generated C# is **vendored** under `Communication/SiteStreamGrpc/` with the `<Protobuf>` include commented out — regeneration is a manual toggle-build-copy-untoggle. DebugStreamEvent message removed (no longer flows through ClusterClient).
- Native alarms: a read-only mirror of native alarms from OPC UA Alarms & Conditions servers and the MxAccess Gateway, unified onto an A&C-style condition model (`AlarmConditionState`: orthogonal Active/Acked/Confirmed/Shelved/Suppressed + 0–1000 severity) plus an `AlarmKind` discriminator (Computed/NativeOpcUa/NativeMxAccess). New DCL capability seam `IAlarmSubscribableConnection` (implemented by the OPC UA and MxGateway adapters); the `DataConnectionActor` opens ONE alarm feed per connection and routes transitions to instances by source-object reference. A `NativeAlarmActor` (peer to the computed `AlarmActor` under `InstanceActor`) mirrors one source binding: snapshot atomic-swap on (re)subscribe, retention (drops once inactive+acked), per-source cap, and site SQLite persistence (`native_alarm_state`, survives failover, cleared on redeploy/undeploy — mirrors static overrides). State streams to central over the additively-enriched gRPC `AlarmStateUpdate` (the existing computed `AlarmStateChanged` was enriched additively) and seeds via the DebugView snapshot. Authoring: `TemplateNativeAlarmSource` / `InstanceNativeAlarmSourceOverride` entities flatten to `ResolvedNativeAlarmSource` (inherit/compose/override); management commands + ManagementActor handlers + CLI (`template/instance native-alarm-source`) + Central UI (template editor tab + instance override panel) + enriched DebugView alarm table. Read-only — no ack-back; no central tables.
- OPC UA / MxGateway UX (M7): operator **Alarm Summary** page (`/monitoring/alarms`, RequireDeployment, read-only) fans out the existing per-instance `DebugViewSnapshot` Ask (SemaphoreSlim-capped, partial-results tolerant) and aggregates client-side — no central alarm store; shared `AlarmStateBadges` component. **Aggregated live stream shipped 2026-07-10** (`docs/plans/2026-07-10-aggregated-live-alarm-stream-plan.md`): a transient in-memory per-site central live cache (`ISiteAlarmLiveCache`) fed by a site-wide, alarm-only `SubscribeSite` gRPC stream (seed-then-stream), pushing near-real-time deltas to the page over the Blazor circuit with the 15s poll kept as fallback + NotReporting authority — still no persisted central alarm store. OPC UA node browser gains `BrowseNext` continuation paging ("Load more"), a bounded recursive address-space **search** (`IAddressSpaceSearchable` seam; depth + result caps; substring on DisplayName/path), and **type-info** (DataType/ValueRank/Writable on `BrowseNode` for Variables). Attribute-override **CSV bulk import** (`OverrideCsvParser`, all-or-nothing) via InstanceConfigure `InputFile` + CLI `instance import-overrides --file` (native-alarm-source-override CSV deferred). **Verify-endpoint** probe (temporary `RealOpcUaClient`, short timeout, captures an untrusted server cert but NEVER trusts it) + **site-local cert trust**: per-node `CertStoreActor` (runs on every site node, not a singleton) writing the `.der` into the node's OPC UA trusted-peer PKI store; DeploymentManager broadcasts `TrustServerCertCommand`/`RemoveServerCertCommand` to BOTH site nodes so PKI stores stay consistent across failover; Admin-gated cert-management UI (`/design/connections/{id}/certificates`). No central persistence of cert trust (follow-up).
@@ -208,11 +213,13 @@ Other peers in the `scadaproj` family (see `scadaproj/CLAUDE.md` for details): `
- Two-person MxGateway secured writes (M7): two new global roles — `Operator` (initiates) + `Verifier` (approves) — added alongside the canonical `Administrator`/`Designer`/`Deployer`/`Viewer`, with `RequireOperator`/`RequireVerifier` policies. An Operator submits a secured write from the Central UI Secured Writes page (`/operations/secured-writes`); it stays a `Pending` `PendingSecuredWrite` row until a *distinct* Verifier approves it (no-self-approval enforced server-side in the ManagementActor, plus a compare-and-swap race guard). Approval relays a `WriteTagRequest` to the site MxGateway; MxGateway-protocol connections only; each lifecycle event (submit/approve/reject/execute) emits a best-effort `AuditChannel.SecuredWrite` / `AuditKind.SecuredWrite*` central-direct-write row sharing the row id as `CorrelationId`. (SecuredWrite audit rows stamp `SourceNode` via `ICentralAuditWriter`/`INodeIdentityProvider`.) Pending secured writes expire server-side after a configurable TTL (`ManagementServiceOptions.SecuredWritePendingTtl`, default 24 h): an overdue `Pending` row is CAS'd to `Expired` (never relayed) — enforced at approve/reject and swept opportunistically on list (arch-review S2, `AuditKind.SecuredWriteExpire`).
### Cluster & Failover
- Keep-oldest split-brain resolver with `down-if-alone = on`, 15s stable-after.
- **`auto-down` downing strategy (decision 2026-07-21 — availability over partition-safety).** Akka's `AutoDowning` provider, `auto-down-unreachable-after` = 15s: the leader among the REACHABLE members downs the unreachable peer, so a hard crash of EITHER node (active/oldest included) fails over to the survivor in ~25s. Accepted trade: a real partition → dual-active until an operator restarts one side. `keep-oldest` remains a supported `SplitBrainResolverStrategy` value (partition-safe, but an oldest-crash is a total outage — Akka's `down-if-alone` only rescues a side with ≥2 members, proven live + in 1.5.62 source). Decision record: `docs/plans/2026-07-21-auto-down-availability-decision.md`.
- Both nodes are seed nodes. `min-nr-of-members = 1`.
- Failure detection: 2s heartbeat, 10s threshold. Total failover ~25s.
- Failure detection: 2s heartbeat, 10s threshold. Total failover ~25s (drill-measured 2026-07-21 under auto-down: active-crash TAKEOVER in 28s, standby-crash removal in 27s with 0 routing blips — `docker/failover-drill.sh`).
- CoordinatedShutdown for graceful singleton handover.
- Automatic dual-node recovery from persistent storage.
- **Active/standby is decided by `ActiveNodeEvaluator.SelfIsOldestUp`, never by cluster leadership** — see the Architecture note above. `/health/active` is **central-only** (site nodes map no `/health/*` at all) and backs both Traefik's active-node routing and `IActiveNodeGate`, so the proxy and the Inbound API always agree on which node is active. Central never needs to know which *site* node is active: ClusterClient contact rotation reaches either receptionist and the site-internal `ClusterSingletonProxy` lands the work on the active node for free. The **exception is gRPC**, which picks `GrpcNodeAAddress`/`GrpcNodeBAddress` explicitly and flips on error.
- **Seed-node ordering: every node lists ITSELF first (decision 2026-07-22) — the boot-alone gap is CLOSED.** Only `seed-nodes[0]` may self-join to form a new cluster (Akka runs `FirstSeedNodeProcess` for it, `JoinSeedNodeProcess` — which can never form one — for everyone else). All 14 shipped node appsettings now lead with the node's own address, so any node can cold-start alone and become operational unattended (~5s, `seed-node-timeout`); `StartupValidator` fails the boot if the ordering is broken (compares host AND port; Akka does no DNS canonicalisation). Two nodes cold-starting together while mutually reachable converge on ONE cluster via the `InitJoin` handshake — they split only under a genuine boot-time partition, the same class auto-down accepts. **An external self-form timer (`Cluster.Join(SelfAddress)` after a window) was implemented and REJECTED:** it sits outside the join handshake, so on a routine standby restart — where the peer is alive but the join is stalled behind removal of the node's own stale incarnation — it fires mid-join and permanently splits the pair (measured: still split after 90s). Regression tests: `SelfFirstSeedBootstrapTests`. The keep-oldest active-crash total outage was separately closed by the auto-down decision. See `docs/requirements/Component-ClusterInfrastructure.md` → Seed Node Ordering.
### UI & Monitoring
- Central UI: Blazor Server (ASP.NET Core + SignalR) with Bootstrap CSS. No third-party component frameworks (no Blazorise, MudBlazor, Radzen, etc.). Build custom Blazor components for tables, grids, forms, etc.
Directory.Packages.props
+24 -3
View File
@@ -108,9 +108,9 @@
<PackageVersion Include="ZB.MOM.WW.Secrets.Abstractions" Version="0.2.3" />
<PackageVersion Include="ZB.MOM.WW.Secrets.Ui" Version="0.2.3" />
<PackageVersion Include="ZB.MOM.WW.Secrets.Replicator.SqlServer" Version="0.2.3" />
<PackageVersion Include="ZB.MOM.WW.LocalDb" Version="0.1.1" />
<PackageVersion Include="ZB.MOM.WW.LocalDb.Replication" Version="0.1.1" />
<PackageVersion Include="ZB.MOM.WW.LocalDb.Contracts" Version="0.1.1" />
<PackageVersion Include="ZB.MOM.WW.LocalDb" Version="0.1.3" />
<PackageVersion Include="ZB.MOM.WW.LocalDb.Replication" Version="0.1.3" />
<PackageVersion Include="ZB.MOM.WW.LocalDb.Contracts" Version="0.1.3" />
</ItemGroup>
<!--
@@ -145,6 +145,27 @@
<PackageVersion Include="SQLitePCLRaw.lib.e_sqlite3" Version="2.1.12" />
</ItemGroup>
<!--
Four NU1903 high-severity advisories (GHSA-23rf-6693-g89p, GHSA-8q5v-6pqq-x66h,
GHSA-cvvh-rhrc-wg4q, GHSA-g8r8-53c2-pm3f) landed in the NuGet audit data against
System.Security.Cryptography.Xml 10.0.7, pulled in TRANSITIVELY by
Microsoft.AspNetCore.DataProtection 10.0.7 (ConfigurationDatabase's DataProtection
key storage). With TreatWarningsAsErrors any FRESH restore — notably the docker
image build — went red (surfaced 2026-07-21; local builds had cached audit data).
Same pattern as SQLitePCLRaw above: pin the vulnerable transitive package to its
patched version (10.0.10) with an explicit <PackageReference> in the one project
where the chain enters (ConfigurationDatabase; every other resolver — AuditLog,
SiteCallAudit, Transport, PerformanceTests, tests — reaches it through that
ProjectReference). Bumping the DataProtection parent instead was tried and
rejected: 10.0.10 floors Microsoft.Extensions.* and (via the EFCore adapter)
Microsoft.EntityFrameworkCore at 10.0.10, forcing a family-wide servicing bump
(NU1605 downgrade errors) that belongs in its own reviewed commit.
-->
<ItemGroup>
<PackageVersion Include="System.Security.Cryptography.Xml" Version="10.0.10" />
</ItemGroup>
<!--
GHSA-pgww-w46g-26qg (NU1902, moderate) on AngleSharp, reached only transitively via bunit
in ZB.MOM.WW.ScadaBridge.CentralUI.Tests. With TreatWarningsAsErrors it made the WHOLE
alog.md
+9 -2
View File
@@ -108,7 +108,7 @@ Single wide table, polymorphic by `Channel` + `Kind` discriminators, JSON payloa
| `EventId` | `uniqueidentifier` PK | Generated where the event originates (site or central). Idempotency key. |
| `OccurredAtUtc` | `datetime2` | When the event happened (call returned, retry attempted, etc.). |
| `IngestedAtUtc` | `datetime2` | When central persisted the row (lags `OccurredAtUtc` for site-originated rows). |
| `Channel` | `varchar(32)` | `ApiOutbound` \| `DbOutbound` \| `Notification` \| `ApiInbound`. |
| `Channel` | `varchar(32)` | `ApiOutbound` \| `DbOutbound` \| `Notification` \| `ApiInbound` \| `SecuredWrite` \| `Cluster`. The last two are not script trust-boundary crossings: `SecuredWrite` records the two-person write lifecycle, and `Cluster` records operator-initiated topology actions (admin-triggered manual failover, decision 2026-07-22). |
| `Kind` | `varchar(32)` | Event kind discriminator (see kinds list below). |
| `CorrelationId` | `uniqueidentifier` NULL | Ties multi-event operations together. `TrackedOperationId` for cached calls, `NotificationId` for notifications, request-id for inbound API. NULL for sync one-shot calls. |
| `SourceSiteId` | `varchar(64)` NULL | NULL for central-originated events (inbound API, central notification dispatch). |
@@ -135,7 +135,7 @@ Single wide table, polymorphic by `Channel` + `Kind` discriminators, JSON payloa
- `IX_AuditLog_Target_Occurred (Target, OccurredAtUtc)` — "what did we send to system X."
- Partitioning by month on `OccurredAtUtc` from day one (purge becomes a partition switch instead of a delete storm).
**`Kind` values (flat — 10 discriminators across all channels):**
**`Kind` values (flat — 17 discriminators across all channels; pinned by `AuditEnumTests`):**
| Kind | Fires when |
|---|---|
@@ -149,6 +149,13 @@ Single wide table, polymorphic by `Channel` + `Kind` discriminators, JSON payloa
| `InboundAuthFailure` | An inbound API request was rejected at the auth boundary (bad/missing key). One row, `Status=Failed`, `HttpStatus=401`. |
| `CachedSubmit` | Script-side enqueue of a cached call (`ExternalSystem.CachedCall` / `Database.CachedWrite`); first row in the cached-call lifecycle, written to site SQLite before any forward attempt. |
| `CachedResolve` | Terminal row for a cached operation — `Status` = `Delivered` / `Failed` / `Parked` / `Discarded`. |
| `SecuredWriteSubmit` | An Operator submitted a two-person secured write; row written after the `PendingSecuredWrite` is persisted so it carries the store-assigned id as `CorrelationId`. |
| `SecuredWriteApprove` | A distinct Verifier approved a pending secured write (no self-approval; enforced server-side). |
| `SecuredWriteReject` | A Verifier rejected a pending secured write. |
| `SecuredWriteExecute` | An approved secured write was relayed to the site MxGateway connection. |
| `SecuredWriteExpire` | A `Pending` secured write aged past its server-side TTL and was transitioned to `Expired` without executing — emitted by the system (no verifier). |
| `ReconciliationAbandoned` | A reconciliation pull row failed to insert up to the permanent-abandon threshold and central advanced its cursor past it; one synthetic row so the loss is queryable in the Audit Log itself. |
| `ManualFailover` | An administrator triggered a manual failover of the central pair from the Health page; one row per invocation, written BEFORE the graceful `Cluster.Leave` is issued. `Target` = the leaving node's address. |
### Site: `AuditLog` (SQLite)
docker-env2/central-node-a/appsettings.Central.json
+1 -1
View File
@@ -11,7 +11,7 @@
"akka.tcp://scadabridge@scadabridge-env2-central-a:8081",
"akka.tcp://scadabridge@scadabridge-env2-central-b:8081"
],
"SplitBrainResolverStrategy": "keep-oldest",
"SplitBrainResolverStrategy": "auto-down",
"StableAfter": "00:00:15",
"HeartbeatInterval": "00:00:02",
"FailureDetectionThreshold": "00:00:10",
docker-env2/central-node-b/appsettings.Central.json
+3 -3
View File
@@ -8,10 +8,10 @@
},
"Cluster": {
"SeedNodes": [
"akka.tcp://scadabridge@scadabridge-env2-central-a:8081",
"akka.tcp://scadabridge@scadabridge-env2-central-b:8081"
"akka.tcp://scadabridge@scadabridge-env2-central-b:8081",
"akka.tcp://scadabridge@scadabridge-env2-central-a:8081"
],
"SplitBrainResolverStrategy": "keep-oldest",
"SplitBrainResolverStrategy": "auto-down",
"StableAfter": "00:00:15",
"HeartbeatInterval": "00:00:02",
"FailureDetectionThreshold": "00:00:10",
docker-env2/docker-compose.yml
+8
View File
@@ -16,6 +16,10 @@ services:
# pepper per the "different per environment" guidance; real deployments inject a
# true secret out-of-band, never from source control. Both Central nodes share it.
ScadaBridge__InboundApi__ApiKeyPepper: "dev-only-insecure-pepper-env2-cluster-0001"
# DEV-ONLY gRPC control-plane preshared key for site-x — NOT a real secret.
# Must match ScadaBridge:Communication:GrpcPsk in site-x-node-*/appsettings.Site.json.
# Production seeds SB-GRPC-PSK-<siteId> into the secret store instead.
ScadaBridge__Communication__SitePsks__site-x: "dev-grpc-psk-docker-env2-site-x"
ports:
- "9101:5000" # Web UI + Inbound API
- "9111:8081" # Akka remoting
@@ -43,6 +47,10 @@ services:
# pepper per the "different per environment" guidance; real deployments inject a
# true secret out-of-band, never from source control. Both Central nodes share it.
ScadaBridge__InboundApi__ApiKeyPepper: "dev-only-insecure-pepper-env2-cluster-0001"
# DEV-ONLY gRPC control-plane preshared key for site-x — NOT a real secret.
# Must match ScadaBridge:Communication:GrpcPsk in site-x-node-*/appsettings.Site.json.
# Production seeds SB-GRPC-PSK-<siteId> into the secret store instead.
ScadaBridge__Communication__SitePsks__site-x: "dev-grpc-psk-docker-env2-site-x"
ports:
- "9102:5000" # Web UI + Inbound API
- "9112:8081" # Akka remoting
docker-env2/site-x-node-a/appsettings.Site.json
+8 -1
View File
@@ -13,7 +13,7 @@
"akka.tcp://scadabridge@scadabridge-env2-site-x-a:8082",
"akka.tcp://scadabridge@scadabridge-env2-site-x-b:8082"
],
"SplitBrainResolverStrategy": "keep-oldest",
"SplitBrainResolverStrategy": "auto-down",
"StableAfter": "00:00:15",
"HeartbeatInterval": "00:00:02",
"FailureDetectionThreshold": "00:00:10",
@@ -40,6 +40,13 @@
"SqliteDbPath": "/app/data/store-and-forward.db"
},
"Communication": {
// DEV-ONLY control-plane preshared key — NOT a real secret. Must be
// IDENTICAL on both nodes of the pair and match the central-side entry in
// ScadaBridge__Communication__SitePsks__<siteId> (docker-compose.yml).
// Production supplies this as ${secret:SB-GRPC-PSK-<siteId>}. Without it the
// node fails StartupValidator: the gate is fail-closed, so an unset key would
// refuse every SiteStream call while the node still looked healthy.
"GrpcPsk": "dev-grpc-psk-docker-env2-site-x",
"CentralContactPoints": [
"akka.tcp://scadabridge@scadabridge-env2-central-a:8081",
"akka.tcp://scadabridge@scadabridge-env2-central-b:8081"
docker-env2/site-x-node-b/appsettings.Site.json
+10 -3
View File
@@ -10,10 +10,10 @@
},
"Cluster": {
"SeedNodes": [
"akka.tcp://scadabridge@scadabridge-env2-site-x-a:8082",
"akka.tcp://scadabridge@scadabridge-env2-site-x-b:8082"
"akka.tcp://scadabridge@scadabridge-env2-site-x-b:8082",
"akka.tcp://scadabridge@scadabridge-env2-site-x-a:8082"
],
"SplitBrainResolverStrategy": "keep-oldest",
"SplitBrainResolverStrategy": "auto-down",
"StableAfter": "00:00:15",
"HeartbeatInterval": "00:00:02",
"FailureDetectionThreshold": "00:00:10",
@@ -40,6 +40,13 @@
"SqliteDbPath": "/app/data/store-and-forward.db"
},
"Communication": {
// DEV-ONLY control-plane preshared key — NOT a real secret. Must be
// IDENTICAL on both nodes of the pair and match the central-side entry in
// ScadaBridge__Communication__SitePsks__<siteId> (docker-compose.yml).
// Production supplies this as ${secret:SB-GRPC-PSK-<siteId>}. Without it the
// node fails StartupValidator: the gate is fail-closed, so an unset key would
// refuse every SiteStream call while the node still looked healthy.
"GrpcPsk": "dev-grpc-psk-docker-env2-site-x",
"CentralContactPoints": [
"akka.tcp://scadabridge@scadabridge-env2-central-a:8081",
"akka.tcp://scadabridge@scadabridge-env2-central-b:8081"
docker/README.md
+66 -12
View File
@@ -120,6 +120,31 @@ docker/
└── logs/
```
## gRPC control-plane keys (dev)
The site gRPC service (`SiteStreamService` on 8083 — live subscriptions, audit pull,
cached-telemetry ingest) is gated by a preshared key, and the gate is **fail-closed**: a site node
with no key refuses every call, and `StartupValidator` refuses to boot it at all. So the rig
carries dev keys, one per site:
| Where | Setting | Value |
|---|---|---|
| `site-{a,b,c}-node-*/appsettings.Site.json` | `ScadaBridge:Communication:GrpcPsk` | `dev-grpc-psk-docker-site-{a,b,c}` |
| `docker-compose.yml`, both central nodes | `ScadaBridge__Communication__SitePsks__site-{a,b,c}` | same value |
Both nodes of a pair carry the same key; each site's key is different from the others'. The
central half lives in compose env rather than the mounted `appsettings.Central.json`, which by
convention holds no plaintext credentials. Production uses `${secret:SB-GRPC-PSK-<siteId>}` on
the site and the matching secret in central's store — see
[`docs/deployment/topology-guide.md`](../docs/deployment/topology-guide.md).
**These are not real secrets and are committed deliberately**, exactly like the LocalDb sync key
(`dev-site-a-localdb-sync-key`) beside them. The two are separate keys on purpose: the LocalDb one
authenticates the *pair partner* for database replication, not central.
If you add a site to the rig, add its key in both places or its streams will fail with
`PermissionDenied`.
## Commands
### Initial Setup
@@ -273,29 +298,43 @@ All test passwords are `password`. See `infra/glauth/config.toml` for the full l
### Automated failover drill (`failover-drill.sh`)
```bash
DRILL_MODE=standby bash docker/failover-drill.sh # default — survivable younger-node crash
DRILL_MODE=active bash docker/failover-drill.sh # oldest-node crash — measures the registered outage gap
DRILL_MODE=standby bash docker/failover-drill.sh # default — younger-node crash, active untouched
DRILL_MODE=active bash docker/failover-drill.sh # oldest-node crash — survivor must TAKE OVER
```
The scripted drill (`docker kill` = SIGKILL, the hard-crash path — a `docker stop` would take the graceful `CoordinatedShutdown` path and would not prove crash recovery) has **two modes**, because under the unified oldest-member semantics the *active* node IS the oldest, i.e. the one crash two-node keep-oldest cannot survive:
The scripted drill (`docker kill` = SIGKILL, the hard-crash path — a `docker stop` would take the graceful `CoordinatedShutdown` path and would not prove crash recovery) has **two modes**, and since the **auto-down decision (2026-07-21)** both expect recovery — the cluster runs Akka's `AutoDowning` provider (`auto-down-unreachable-after` = 15s), under which the leader among the *reachable* members downs the unreachable peer, so a crash of either node fails over:
- **`DRILL_MODE=standby` (default) — kills the STANDBY (younger) central node.** The survivable direction: SBR downs the crashed member and the active node keeps its singletons. Expected result: **no routing outage at all** (the active node is never touched, so `/health/active` blips = 0) and member removal on the survivor within **~25s** (10s failure-detection threshold + 15s stable-after; the 2s heartbeat interval is not additive). PASS = the survivor logs the member removal within `TIMEOUT_S` (default 90s) while routing stays up.
- **`DRILL_MODE=active` — kills the ACTIVE (oldest) central node.** Expected result: a **total central outage** until the victim container is restarted — this is the registered deferred keep-oldest decision (master tracker 2026-07-08): keep-oldest downs the partition *without* the oldest, so the younger survivor downs itself, and it cannot re-form a cluster alone (see the seed-node constraint below). The drill confirms the dark window, then recovery within ~2 min of restarting the victim. The mode exists to make the registered gap *observable*, not to pretend it is covered.
- **`DRILL_MODE=standby` (default) — kills the STANDBY (younger) central node.** The active node is untouched: expected result is **no routing outage at all** (`/health/active` blips = 0) and member removal on the survivor within **~25s** (10s failure-detection threshold + 15s auto-down window; the 2s heartbeat interval is not additive). PASS = the survivor logs the downing/removal within `TIMEOUT_S` (default 90s) while routing stays up.
- **`DRILL_MODE=active` — kills the ACTIVE (oldest) central node.** The survivor must **take over while the victim is still down**: it auto-downs the dead oldest, becomes the oldest member itself, re-hosts all singletons, and its `/health/active` goes 200. PASS = survivor active within `TIMEOUT_S`, then Traefik routing to it. (Under the pre-2026-07-21 `keep-oldest` strategy this direction was a proven total outage — the younger survivor took `DownReachable` and downed itself, because Akka's `down-if-alone` only rescues a side with ≥ 2 members.)
The drill exercises S1 (SBR downing on hard crash), S3 (single active node routed through Traefik), and the Task 20 restart/rejoin contract. Requires a running cluster (`bash docker/deploy.sh`) and `curl` + `docker` on the host.
Both modes finish by restarting the victim and confirming it rejoins as a ready standby. The drill exercises downing-on-hard-crash, S3 (single active node routed through Traefik), and the Task 20 restart/rejoin contract. Requires a running cluster (`bash docker/deploy.sh`) and `curl` + `docker` on the host.
**Seed-node bootstrap constraint.** Only the FIRST seed in `Cluster:SeedNodes` may self-join to form a *new* cluster. Both central nodes list `scadabridge-central-a` first (`docker/central-node-a/appsettings.Central.json`, `docker/central-node-b/appsettings.Central.json`), so a lone restarted `central-b` (with `central-a` still down) loops on `InitJoin` forever — it never reaches `Up`, and `/health/active` never returns 200. Operator recovery actions: **(1)** restart the dead first-seed node (`central-a`) — preferred; or **(2)** restart the survivor with a self-first seed override (env `ScadaBridge__Cluster__SeedNodes__0=akka.tcp://scadabridge@<self-host>:8081`, `ScadaBridge__Cluster__SeedNodes__1=<peer>`). The repo deliberately does NOT ship self-first ordering per node: with *both* nodes self-first, a simultaneous cold start can let each self-join independently → two one-node clusters that never merge (the cold-start split-brain the identical-seed-order convention exists to prevent). The real remedy is the pending keep-oldest topology/strategy decision (deferred, owner: user).
**Partition trade (accepted).** Auto-down is availability-first: in a *real network partition* (both nodes alive, link cut) each side downs the other and both run active — dual-active until an operator restarts one side after the partition heals. This was an explicit owner decision (2026-07-21): site pairs have no shared lease infrastructure to arbitrate, and a stalled system is a bigger risk than a rare partition. See `docs/plans/2026-07-21-auto-down-availability-decision.md`.
> **Observed results** (plan R2-01 T3):
**Seed-node ordering — every node lists ITSELF first (decision 2026-07-22).** Akka runs `FirstSeedNodeProcess` — the only bootstrap path that can form a *new* cluster when no peer answers `InitJoin` — exclusively when `seed-nodes[0]` is the node's own address; every other node runs `JoinSeedNodeProcess`, which retries `InitJoin` forever and can never form a cluster. Each shipped node config therefore lists itself first and its partner second (`docker/central-node-b/appsettings.Central.json` leads with `scadabridge-central-b`), and `StartupValidator` fails the boot if that ordering is ever broken. This closes the former **registered outage gap**, where a lone cold-starting `central-b` (with `central-a` down) never came `Up` and recovery was operator-driven.
Self-first ordering is safe, and the three interesting cases are covered by `SelfFirstSeedBootstrapTests` (real in-process clusters at production failure-detection timings):
| Scenario | Behavior |
|---|---|
| Lone cold-start, peer dead | Forms alone in ~5s (`seed-node-timeout`) — operational, unattended |
| Restart into a **live** peer | `InitJoinAck` answers, node rejoins; never islands |
| Both cold-start simultaneously (mutually reachable) | The `InitJoin` handshake resolves it *before* either self-joins → **one** 2-member cluster |
> An earlier revision of this README claimed the repo deliberately avoided self-first ordering because simultaneous cold start would produce "two one-node clusters that never merge". That is **not** what happens while the nodes are mutually reachable — the handshake converges them (measured, row 3 above). Only a genuine boot-time *partition* splits them, which is the same class `auto-down` already accepts.
> **Rejected alternative — an external self-form timer.** A watchdog that waits N seconds for membership and then calls `Cluster.Join(SelfAddress)` was implemented and discarded: it cannot see Akka's join handshake, so it cannot distinguish "no seed answered" from "a seed answered and the join is in flight". On a routine standby restart the peer is alive but the join stalls behind removal of the restarting node's own stale incarnation; a `Join(self)` issued during `TryingToJoin` abandons the in-flight join and forms a second cluster at the same address — a **permanent** split (measured: still split after 90s). Akka's own first-seed process has no such race because it *is* part of the handshake.
> **Observed results** (auto-down decision verification):
>
> **Run 2026-07-13** against a freshly-deployed cluster on `main` @ `99544985` (round-2 merged image; `active=central-a`). Both directions behaved exactly as the design predicts.
> **Run 2026-07-21** against a freshly-deployed cluster with `SplitBrainResolverStrategy: auto-down` (first drill: `active=central-a`). Both directions recovered.
>
> | Direction (`DRILL_MODE`) | Outcome | Measured |
> |--------------------------|---------|----------|
> | `standby` (younger-node crash) | **PASS** — SBR downed+removed the crashed `central-b`; active `central-a` kept all 7 singletons; recovered on restart. | Member removed in **27s** (budget ~25s: 10s detection + 15s stable-after); **0** `/health/active` routing blips (active node never touched); routable **0s** after victim restart. |
> | `active` (oldest-node crash) | **Outage as designed** — killing the oldest/active `central-a` made the younger `central-b` self-down (total central outage — the registered keep-oldest gap); recovered after restarting the victim, `central-b` then assuming Oldest and re-hosting all singletons. | Outage confirmed at **9s**; central routable again **4s** after restarting `central-a`. |
> | `active` (oldest-node crash) | **PASS — TAKEOVER** — `central-b` auto-downed the dead oldest, went `Younger -> Oldest` on all 7 singletons, and served `/health/active` **while the victim was still down**; restarted victim rejoined as standby. | Survivor active + Traefik routing in **28s** (budget ~25s: 10s detection + 15s auto-down + hand-over); victim ready **2s** after restart. |
> | `standby` (younger-node crash) | **PASS** — active node untouched; survivor downed+removed the crashed member; restarted victim rejoined as standby. | Member removed in **27s**; **0** `/health/active` routing blips; victim ready **2s** after restart. |
>
> Notes: the `standby` PASS shows the survivable direction is clean end-to-end (SBR `DownUnreachable` decision + per-singleton "Member removed" in the survivor log, zero routing interruption). The `active` result **empirically confirms the deferred keep-oldest topology gap** (master tracker 2026-07-08 / `docs/plans/2026-07-08-deferred-work-register.md`): a hard crash of the active/oldest central node is a total outage until that node (the first seed) is restarted — the remedy remains the pending topology/strategy decision. In-process envelope (`FailoverTimingTests`, plan R2-01 T4) independently measured full failover at **33.7s**.
> Historical baseline (keep-oldest, run 2026-07-13 on `99544985`): `standby` PASS with member removal in 27s / 0 routing blips; `active` was a **total outage** — `central-b` self-downed ~20s after the kill (live SBR log 2026-07-21: `SBR took decision Akka.Cluster.SBR.DownReachable … including myself`) and could not re-bootstrap until `central-a` returned. That result is what motivated the auto-down decision. In-process envelope (`FailoverTimingTests`) measured full failover at **33.7s**.
### Central Failover
@@ -313,6 +352,14 @@ open http://localhost:9002
docker start scadabridge-central-a
```
**Manual failover from the UI (admin-only).** Instead of stopping a container, an Administrator can trigger a planned role swap from the **Trigger failover** button on the central-cluster card at `/monitoring/health` (via Traefik, `http://localhost:9000`). The active (oldest Up) node leaves the cluster **gracefully**, so singletons hand over rather than being killed; the node then restarts under `restart: unless-stopped` and rejoins as the standby.
- The button is disabled when the pair has no online standby — the same guard is re-enforced server-side, since failing over a lone node is an outage, not a failover.
- Triggering it **disconnects the page you clicked it on**: Traefik routes the UI to the active node, which is the node being restarted. The page reconnects against the new active node.
- Each invocation writes one `Cluster` / `ManualFailover` row to `dbo.AuditLog` naming the admin and the target address, written before the Leave is issued.
To verify on the rig: press the button, watch `central-a` restart and `central-b`'s badge flip to Primary, then confirm the audit row landed.
### Site Failover
```bash
@@ -329,3 +376,10 @@ docker start scadabridge-site-a-a
Same pattern applies for site-b (`scadabridge-site-b-a`/`scadabridge-site-b-b`) and site-c (`scadabridge-site-c-a`/`scadabridge-site-c-b`).
Failover takes approximately 25 seconds (2s heartbeat + 10s detection threshold + 15s stable-after for split-brain resolver).
**Manual site failover from the UI (admin-only).** Each site card on `/monitoring/health` carries the same **Trigger failover** button as the central card. Central and each site are separate Akka clusters, so this is a *request* relayed over the ClusterClient command/control channel — the site's own communication actor performs the graceful `Leave` against its `site-{SiteId}` role and acks the result.
- Unlike central failover, this does **not** disconnect your page — a site is a different cluster.
- A refusal from the site (no standby, or a command addressed to a different site) reads differently from an unreachable site (Ask timeout); the UI shows the site's own reason. Only the timeout leaves any doubt about whether the failover took effect.
- A site running an older binary has no handler for the command, so it dead-letters and you see "site did not respond".
- Each invocation writes a `Cluster` / `ManualFailover` audit row stamped with the site id.
docker/central-node-a/appsettings.Central.json
+1 -1
View File
@@ -11,7 +11,7 @@
"akka.tcp://scadabridge@scadabridge-central-a:8081",
"akka.tcp://scadabridge@scadabridge-central-b:8081"
],
"SplitBrainResolverStrategy": "keep-oldest",
"SplitBrainResolverStrategy": "auto-down",
"StableAfter": "00:00:15",
"HeartbeatInterval": "00:00:02",
"FailureDetectionThreshold": "00:00:10",
docker/central-node-b/appsettings.Central.json
+3 -3
View File
@@ -8,10 +8,10 @@
},
"Cluster": {
"SeedNodes": [
"akka.tcp://scadabridge@scadabridge-central-a:8081",
"akka.tcp://scadabridge@scadabridge-central-b:8081"
"akka.tcp://scadabridge@scadabridge-central-b:8081",
"akka.tcp://scadabridge@scadabridge-central-a:8081"
],
"SplitBrainResolverStrategy": "keep-oldest",
"SplitBrainResolverStrategy": "auto-down",
"StableAfter": "00:00:15",
"HeartbeatInterval": "00:00:02",
"FailureDetectionThreshold": "00:00:10",
docker/docker-compose.yml
+20
View File
@@ -27,9 +27,19 @@ services:
ScadaBridge__Database__MachineDataDb: "Server=scadabridge-mssql,1433;Database=ScadaBridgeMachineData;User Id=scadabridge_app;Password=ScadaBridge_Dev1#;TrustServerCertificate=true"
ScadaBridge__Security__Ldap__ServiceAccountPassword: "serviceaccount123"
ScadaBridge__Security__JwtSigningKey: "scadabridge-dev-jwt-signing-key-must-be-at-least-32-characters-long"
# DEV-ONLY gRPC control-plane preshared keys, one per site — NOT real secrets.
# Central verifies/presents these; each site node carries the same value as
# ScadaBridge:Communication:GrpcPsk in its mounted appsettings.Site.json. Kept as
# env overrides (not in the mounted central appsettings) so that file stays free of
# plaintext credentials. Production instead seeds SB-GRPC-PSK-<siteId> into the
# secret store, which is also the only source that can serve a site added at runtime.
ScadaBridge__Communication__SitePsks__site-a: "dev-grpc-psk-docker-site-a"
ScadaBridge__Communication__SitePsks__site-b: "dev-grpc-psk-docker-site-b"
ScadaBridge__Communication__SitePsks__site-c: "dev-grpc-psk-docker-site-c"
ports:
- "9001:5000" # Web UI + Inbound API
- "9011:8081" # Akka remoting (host access for CLI/debugging)
- "9013:8083" # gRPC control plane (CentralControlService, T1A.2)
volumes:
- ./central-node-a/appsettings.Central.json:/app/appsettings.Central.json:ro
- ./central-node-a/logs:/app/logs
@@ -65,9 +75,19 @@ services:
ScadaBridge__Database__MachineDataDb: "Server=scadabridge-mssql,1433;Database=ScadaBridgeMachineData;User Id=scadabridge_app;Password=ScadaBridge_Dev1#;TrustServerCertificate=true"
ScadaBridge__Security__Ldap__ServiceAccountPassword: "serviceaccount123"
ScadaBridge__Security__JwtSigningKey: "scadabridge-dev-jwt-signing-key-must-be-at-least-32-characters-long"
# DEV-ONLY gRPC control-plane preshared keys, one per site — NOT real secrets.
# Central verifies/presents these; each site node carries the same value as
# ScadaBridge:Communication:GrpcPsk in its mounted appsettings.Site.json. Kept as
# env overrides (not in the mounted central appsettings) so that file stays free of
# plaintext credentials. Production instead seeds SB-GRPC-PSK-<siteId> into the
# secret store, which is also the only source that can serve a site added at runtime.
ScadaBridge__Communication__SitePsks__site-a: "dev-grpc-psk-docker-site-a"
ScadaBridge__Communication__SitePsks__site-b: "dev-grpc-psk-docker-site-b"
ScadaBridge__Communication__SitePsks__site-c: "dev-grpc-psk-docker-site-c"
ports:
- "9002:5000" # Web UI + Inbound API
- "9012:8081" # Akka remoting
- "9014:8083" # gRPC control plane (CentralControlService, T1A.2)
volumes:
- ./central-node-b/appsettings.Central.json:/app/appsettings.Central.json:ro
- ./central-node-b/logs:/app/logs
docker/failover-drill.sh
+52 -39
View File
@@ -1,33 +1,33 @@
#!/usr/bin/env bash
# Failover drill against the running docker cluster (bash docker/deploy.sh first).
#
# ROUND-2 REWRITE (arch-review 01 round 2, N1). The original drill killed the
# ACTIVE central node — but under the unified oldest-member semantics the
# active node IS the oldest, i.e. the one crash two-node keep-oldest CANNOT
# survive (registered deferred user decision, master tracker 2026-07-08;
# SbrFailoverTests.cs XML doc). Two modes:
# AUTO-DOWN REWRITE (decision 2026-07-21). The cluster now runs the 'auto-down'
# downing strategy (availability-first): the leader among the REACHABLE members
# downs the unreachable peer after StableAfter, so a hard crash of EITHER
# central node — the active/oldest included — fails over to the survivor. The
# accepted trade (made explicitly by the owner) is dual-active during a real
# network partition. Both drill directions therefore expect RECOVERY:
#
# DRILL_MODE=standby (default) — kills the STANDBY (younger) central node.
# The survivable direction: SBR downs the crashed member, the active node
# keeps its singletons, and Traefik routing never goes dark. PASS = the
# survivor logs the member removal within TIMEOUT_S (budget ~25s+: 10s
# failure detection + 15s stable-after) while /health/active stays up.
# The active node is untouched: expect zero /health/active routing blips
# and member removal on the survivor within ~25s (10s failure detection +
# 15s auto-down-unreachable-after).
#
# DRILL_MODE=active — kills the ACTIVE (oldest) central node. THE EXPECTED
# OUTCOME IS A TOTAL CENTRAL OUTAGE: keep-oldest downs the partition
# without the oldest, so the younger survivor downs ITSELF (down-if-alone
# cannot help — the alone-oldest is dead and cannot down itself), and the
# self-downed survivor cannot re-form a cluster alone unless it is the
# FIRST seed (both nodes list central-a first; only the first seed may
# self-join). This mode measures the dark window and PASSes only when
# central recovers AFTER the victim container is restarted. It exists to
# make the registered gap observable — not to pretend it is covered.
# DRILL_MODE=active — kills the ACTIVE (oldest) central node. THE SURVIVOR
# MUST TAKE OVER: it downs the dead oldest, becomes oldest itself, hosts
# the singletons, and /health/active goes 200 on the survivor WHILE THE
# VICTIM IS STILL DOWN. Budget ~25s + singleton hand-over + health-probe
# margin. (Under the pre-2026-07-21 keep-oldest strategy this direction
# was a total outage — the younger survivor downed ITSELF, verified live;
# Akka's down-if-alone only rescues a side with >= 2 members.)
#
# Both modes finish by restarting the victim and confirming it rejoins as a
# fresh incarnation (standby).
set -euo pipefail
TRAEFIK_URL="${TRAEFIK_URL:-http://localhost:9000}"
TIMEOUT_S="${TIMEOUT_S:-90}"
DRILL_MODE="${DRILL_MODE:-standby}"
OUTAGE_CONFIRM_S="${OUTAGE_CONFIRM_S:-60}"
active_container() {
if curl -sf -o /dev/null "http://localhost:9001/health/active"; then echo scadabridge-central-a
@@ -35,6 +35,7 @@ active_container() {
else echo "ERROR: no active central node found" >&2; exit 1; fi
}
peer_of() { [ "$1" = scadabridge-central-a ] && echo scadabridge-central-b || echo scadabridge-central-a; }
port_of() { [ "$1" = scadabridge-central-a ] && echo 9001 || echo 9002; }
case "$DRILL_MODE" in
standby|active) ;;
@@ -47,6 +48,7 @@ if [ "$DRILL_MODE" = standby ]; then
else
VICTIM="$ACTIVE"; SURVIVOR=$(peer_of "$ACTIVE")
fi
SURVIVOR_PORT=$(port_of "$SURVIVOR")
echo "mode=${DRILL_MODE} active=${ACTIVE} victim=${VICTIM} survivor=${SURVIVOR}"
KILL_AT=$(date -u +%Y-%m-%dT%H:%M:%SZ)
@@ -54,60 +56,71 @@ docker kill "${VICTIM}" > /dev/null
START=$(date +%s)
if [ "$DRILL_MODE" = standby ]; then
echo "Standby crash: waiting for ${SURVIVOR} to DOWN+REMOVE the dead member (SBR budget ~25s)..."
echo "Standby crash: waiting for ${SURVIVOR} to DOWN+REMOVE the dead member (budget ~25s)..."
BLIPS=0
while true; do
ELAPSED=$(( $(date +%s) - START ))
curl -sf -o /dev/null "${TRAEFIK_URL}/health/active" || BLIPS=$((BLIPS + 1))
if docker logs --since "${KILL_AT}" "${SURVIVOR}" 2>&1 | grep -Eiq "marking.*node.*down|member removed|is removed"; then
echo "PASS: survivor removed the crashed member in ${ELAPSED}s (budget ~25s: 10s detection + 15s stable-after)."
if docker logs --since "${KILL_AT}" "${SURVIVOR}" 2>&1 | grep -Eiq "auto-downing|marking.*node.*down|member removed|is removed"; then
echo "PASS: survivor downed/removed the crashed member in ${ELAPSED}s (budget ~25s: 10s detection + 15s auto-down)."
echo "Active-node routing blips during the drill: ${BLIPS} (expected 0 — the active node was never touched)."
break
fi
if (( ELAPSED > TIMEOUT_S )); then
echo "FAIL: no downing/removal evidence on ${SURVIVOR} after ${ELAPSED}s — SBR did not act" >&2
echo "FAIL: no downing/removal evidence on ${SURVIVOR} after ${ELAPSED}s — auto-down did not act" >&2
docker start "${VICTIM}" > /dev/null
exit 1
fi
sleep 1
done
else
echo "Active crash: EXPECTING a central outage (registered keep-oldest gap). Watching /health/active..."
DARK_STREAK=0
echo "Active crash: waiting for ${SURVIVOR} to take over as the active node (victim stays DOWN; budget ~25s + hand-over)..."
while true; do
ELAPSED=$(( $(date +%s) - START ))
if curl -sf -o /dev/null "${TRAEFIK_URL}/health/active"; then DARK_STREAK=0; else DARK_STREAK=$((DARK_STREAK + 1)); fi
if (( DARK_STREAK >= 10 )); then
echo "Outage confirmed at ${ELAPSED}s: no active central node — the younger survivor self-downed"
echo "(keep-oldest downs the partition WITHOUT the oldest; this is the registered deferred gap)."
if curl -sf -o /dev/null "http://localhost:${SURVIVOR_PORT}/health/active"; then
echo "PASS: ${SURVIVOR} took over as active in ${ELAPSED}s with the victim still down"
echo "(downed the dead oldest via auto-down, assumed Oldest, re-hosted the singletons)."
break
fi
if (( ELAPSED > OUTAGE_CONFIRM_S )); then
echo "NOTE: /health/active stayed reachable ${ELAPSED}s after killing the oldest — better than the"
echo "registered gap predicts. Do NOT celebrate: capture both nodes' logs and investigate before trusting it."
break
if (( ELAPSED > TIMEOUT_S )); then
echo "FAIL: ${SURVIVOR} never became active within ${ELAPSED}s of killing the oldest — takeover did not happen." >&2
docker logs --since "${KILL_AT}" "${SURVIVOR}" 2>&1 | grep -Ei "sbr|downing|oldest|shutting down|terminated" | tail -20 >&2 || true
docker start "${VICTIM}" > /dev/null
exit 1
fi
sleep 1
done
echo "Confirming Traefik routes to the new active node..."
TR_START=$(date +%s)
while ! curl -sf -o /dev/null "${TRAEFIK_URL}/health/active"; do
if (( $(date +%s) - TR_START > 60 )); then
echo "FAIL: survivor is active but not routable through Traefik after 60s" >&2
docker start "${VICTIM}" > /dev/null
exit 1
fi
sleep 1
done
echo "Traefik routing recovered $(( $(date +%s) - START ))s after the kill."
fi
echo "Restarting ${VICTIM}..."
docker start "${VICTIM}" > /dev/null
RESTART_AT=$(date +%s)
echo "Waiting for central to be routable again through Traefik (${TRAEFIK_URL}/health/active)..."
echo "Waiting for the restarted victim to rejoin as a ready standby (${VICTIM} /health/ready)..."
VICTIM_PORT=$(port_of "$VICTIM")
while true; do
ELAPSED=$(( $(date +%s) - RESTART_AT ))
if curl -sf -o /dev/null "${TRAEFIK_URL}/health/active"; then
echo "Recovered: an active central node is routable ${ELAPSED}s after the victim restart."
if curl -sf -o /dev/null "http://localhost:${VICTIM_PORT}/health/ready"; then
echo "Recovered: ${VICTIM} is ready (rejoined as a fresh incarnation) ${ELAPSED}s after restart."
break
fi
if (( ELAPSED > 120 )); then
echo "FAIL: central not routable 120s after restarting ${VICTIM}" >&2
echo "FAIL: ${VICTIM} not ready 120s after restart" >&2
exit 1
fi
sleep 1
done
echo "Survivor singleton/downing evidence (last 20 matching log lines from ${SURVIVOR}):"
docker logs "${SURVIVOR}" 2>&1 | grep -Ei "singleton|oldest|downing|removed" | tail -20 || true
echo "Survivor downing/singleton evidence (last 20 matching log lines from ${SURVIVOR}):"
docker logs "${SURVIVOR}" 2>&1 | grep -Ei "auto-downing|singleton|oldest|downing|removed" | tail -20 || true
echo "Drill complete (${DRILL_MODE}). Verify on the Health dashboard that both nodes show Up and exactly one is Primary."
docker/regen-proto.sh
+75 -49
View File
@@ -1,92 +1,118 @@
#!/usr/bin/env bash
#
# Regenerates the gRPC C# files from sitestream.proto.
# Regenerates the gRPC C# files from the Communication project's .proto files.
#
# Background: protoc (linux/arm64) segfaults inside our Docker build container
# (Grpc.Tools 2.71.0). As a workaround the generated Sitestream.cs +
# SitestreamGrpc.cs are checked into src/ZB.MOM.WW.ScadaBridge.Communication/SiteStreamGrpc/
# and the Protobuf ItemGroup in the .csproj is commented out — Docker just
# compiles the checked-in C# files.
# (Grpc.Tools 2.71.0). As a workaround the generated C# is checked into
# src/ZB.MOM.WW.ScadaBridge.Communication/ — SiteStreamGrpc/ for sitestream.proto
# and CentralControlGrpc/ for central_control.proto — and the Protobuf ItemGroup
# in the .csproj is commented out, so Docker just compiles the checked-in files.
#
# Run this script ON YOUR DEV MACHINE whenever Protos/sitestream.proto changes:
# Run this script ON YOUR DEV MACHINE whenever a .proto changes:
#
# 1. Temporarily uncomments the Protobuf ItemGroup so Grpc.Tools runs.
# 2. dotnet build (regen writes fresh files to obj/).
# 3. Copies the regenerated files back into SiteStreamGrpc/.
# 4. Re-comments the Protobuf ItemGroup so Docker builds stay safe.
# docker/regen-proto.sh [sitestream|centralcontrol|all] (default: all)
#
# 1. Injects a Protobuf ItemGroup for the selected proto(s) so Grpc.Tools runs.
# 2. Deletes the stale checked-in C# so a failed regen is obvious.
# 3. dotnet build (regen writes fresh files to obj/).
# 4. Copies the regenerated files back into the source tree.
# 5. Restores the original csproj so no active Protobuf item is left behind.
#
# Only the SELECTED protos get a Protobuf item. Enabling one whose generated C#
# is still checked in would define every generated type twice, which is why the
# per-proto selection exists. central_control.proto imports sitestream.proto,
# but protoc resolves that from the project-relative path — the import needs no
# Protobuf item of its own.
#
# Once we move to a Dockerfile base image that ships a working linux/arm64
# protoc, this script can be retired and Docker can regen the proto on every
# protoc, this script can be retired and Docker can regen the protos on every
# build like every other normal .NET project.
set -euo pipefail
TARGET="${1:-all}"
case "$TARGET" in
sitestream|centralcontrol|all) ;;
*) echo "usage: $0 [sitestream|centralcontrol|all]" >&2; exit 2 ;;
esac
SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)"
REPO_ROOT="$(cd "$SCRIPT_DIR/.." && pwd)"
COMM_DIR="$REPO_ROOT/src/ZB.MOM.WW.ScadaBridge.Communication"
CSPROJ="$COMM_DIR/ZB.MOM.WW.ScadaBridge.Communication.csproj"
GEN_DIR="$COMM_DIR/SiteStreamGrpc"
GEN="$COMM_DIR/obj/Debug/net10.0/Protos"
echo "=== Regenerating gRPC files from sitestream.proto ==="
echo "=== Regenerating gRPC files ($TARGET) ==="
if [[ ! -f "$CSPROJ" ]]; then
echo "ERROR: csproj not found at $CSPROJ" >&2
exit 1
fi
# Backup so we can always restore the comment state on failure.
# Backup so we can always restore the comment state on failure. Leaving the
# csproj with an active Protobuf item is the one outcome that breaks Docker, so
# every exit path restores this copy.
BACKUP="$(mktemp)"
cp "$CSPROJ" "$BACKUP"
trap 'cp "$BACKUP" "$CSPROJ"; rm -f "$BACKUP"; echo "Restored csproj from backup."' ERR
# 1. Uncomment the Protobuf ItemGroup (strip the surrounding <!-- ... --> wrapper).
python3 - <<PY
import re, pathlib
p = pathlib.Path("$CSPROJ")
src = p.read_text()
# Find the commented Protobuf block and unwrap it.
new = re.sub(
r"<!--\s*\n(\s*<ItemGroup>\s*\n\s*<Protobuf [^>]*/>\s*\n\s*</ItemGroup>)\s*\n\s*-->",
r"\1",
src,
count=1,
)
if new == src:
raise SystemExit("Couldn't find commented Protobuf ItemGroup to enable.")
p.write_text(new)
# 1. Inject an ItemGroup holding just the selected protos, immediately before
# the closing </Project>. The documented commented-out block is left alone.
python3 - "$CSPROJ" "$TARGET" <<'PY'
import pathlib, sys
csproj, target = pathlib.Path(sys.argv[1]), sys.argv[2]
protos = []
if target in ("sitestream", "all"):
protos.append("sitestream.proto")
if target in ("centralcontrol", "all"):
protos.append("central_control.proto")
items = "\n".join(
f' <Protobuf Include="Protos\\{p}" GrpcServices="Both" />' for p in protos)
block = f" <ItemGroup>\n{items}\n </ItemGroup>\n\n</Project>"
src = csproj.read_text()
if "</Project>" not in src:
raise SystemExit("Couldn't find </Project> to inject the Protobuf ItemGroup before.")
csproj.write_text(src.replace("</Project>", block, 1))
PY
# 2. Delete the stale files so any failure to regen is obvious.
rm -f "$GEN_DIR/Sitestream.cs" "$GEN_DIR/SitestreamGrpc.cs"
if [[ "$TARGET" == "sitestream" || "$TARGET" == "all" ]]; then
rm -f "$COMM_DIR/SiteStreamGrpc/Sitestream.cs" "$COMM_DIR/SiteStreamGrpc/SitestreamGrpc.cs"
fi
if [[ "$TARGET" == "centralcontrol" || "$TARGET" == "all" ]]; then
rm -f "$COMM_DIR/CentralControlGrpc/CentralControl.cs" \
"$COMM_DIR/CentralControlGrpc/CentralControlGrpc.cs"
fi
# 3. Regenerate by building.
echo "Building Communication project (regen)..."
dotnet build "$CSPROJ" --nologo -v minimal | tail -5
# 4. Copy generated files back into the source tree.
mkdir -p "$GEN_DIR"
cp "$COMM_DIR/obj/Debug/net10.0/Protos/Sitestream.cs" "$GEN_DIR/Sitestream.cs"
cp "$COMM_DIR/obj/Debug/net10.0/Protos/SitestreamGrpc.cs" "$GEN_DIR/SitestreamGrpc.cs"
echo "Copied regenerated files to $GEN_DIR/"
# 5. Re-comment the Protobuf ItemGroup so Docker builds keep working.
python3 - <<PY
import re, pathlib
p = pathlib.Path("$CSPROJ")
src = p.read_text()
new = re.sub(
r"(\s*<ItemGroup>\s*\n\s*<Protobuf [^>]*/>\s*\n\s*</ItemGroup>)",
r"\n <!--\1\n -->",
src,
count=1,
)
p.write_text(new)
PY
if [[ "$TARGET" == "sitestream" || "$TARGET" == "all" ]]; then
mkdir -p "$COMM_DIR/SiteStreamGrpc"
cp "$GEN/Sitestream.cs" "$GEN/SitestreamGrpc.cs" "$COMM_DIR/SiteStreamGrpc/"
echo "Copied regenerated files to SiteStreamGrpc/"
fi
if [[ "$TARGET" == "centralcontrol" || "$TARGET" == "all" ]]; then
mkdir -p "$COMM_DIR/CentralControlGrpc"
cp "$GEN/CentralControl.cs" "$GEN/CentralControlGrpc.cs" "$COMM_DIR/CentralControlGrpc/"
echo "Copied regenerated files to CentralControlGrpc/"
fi
# 5. Restore the backed-up csproj — i.e. drop the injected ItemGroup — so Docker
# builds keep working.
cp "$BACKUP" "$CSPROJ"
rm -f "$BACKUP"
trap - ERR
echo ""
echo "Done. Review and commit:"
echo " git diff src/ZB.MOM.WW.ScadaBridge.Communication/Protos/sitestream.proto"
echo " git diff src/ZB.MOM.WW.ScadaBridge.Communication/Protos/"
echo " git diff src/ZB.MOM.WW.ScadaBridge.Communication/SiteStreamGrpc/"
echo " git diff src/ZB.MOM.WW.ScadaBridge.Communication/CentralControlGrpc/"
echo " git diff -- src/ZB.MOM.WW.ScadaBridge.Communication/*.csproj # must be EMPTY"
docker/site-a-node-a/appsettings.Site.json
+8 -1
View File
@@ -14,7 +14,7 @@
"akka.tcp://scadabridge@scadabridge-site-a-a:8082",
"akka.tcp://scadabridge@scadabridge-site-a-b:8082"
],
"SplitBrainResolverStrategy": "keep-oldest",
"SplitBrainResolverStrategy": "auto-down",
"StableAfter": "00:00:15",
"HeartbeatInterval": "00:00:02",
"FailureDetectionThreshold": "00:00:10",
@@ -41,6 +41,13 @@
"SqliteDbPath": "/app/data/store-and-forward.db"
},
"Communication": {
// DEV-ONLY control-plane preshared key — NOT a real secret. Must be
// IDENTICAL on both nodes of the pair and match the central-side entry in
// ScadaBridge__Communication__SitePsks__<siteId> (docker-compose.yml).
// Production supplies this as ${secret:SB-GRPC-PSK-<siteId>}. Without it the
// node fails StartupValidator: the gate is fail-closed, so an unset key would
// refuse every SiteStream call while the node still looked healthy.
"GrpcPsk": "dev-grpc-psk-docker-site-a",
"CentralContactPoints": [
"akka.tcp://scadabridge@scadabridge-central-a:8081",
"akka.tcp://scadabridge@scadabridge-central-b:8081"
docker/site-a-node-b/appsettings.Site.json
+10 -3
View File
@@ -11,10 +11,10 @@
},
"Cluster": {
"SeedNodes": [
"akka.tcp://scadabridge@scadabridge-site-a-a:8082",
"akka.tcp://scadabridge@scadabridge-site-a-b:8082"
"akka.tcp://scadabridge@scadabridge-site-a-b:8082",
"akka.tcp://scadabridge@scadabridge-site-a-a:8082"
],
"SplitBrainResolverStrategy": "keep-oldest",
"SplitBrainResolverStrategy": "auto-down",
"StableAfter": "00:00:15",
"HeartbeatInterval": "00:00:02",
"FailureDetectionThreshold": "00:00:10",
@@ -41,6 +41,13 @@
"SqliteDbPath": "/app/data/store-and-forward.db"
},
"Communication": {
// DEV-ONLY control-plane preshared key — NOT a real secret. Must be
// IDENTICAL on both nodes of the pair and match the central-side entry in
// ScadaBridge__Communication__SitePsks__<siteId> (docker-compose.yml).
// Production supplies this as ${secret:SB-GRPC-PSK-<siteId>}. Without it the
// node fails StartupValidator: the gate is fail-closed, so an unset key would
// refuse every SiteStream call while the node still looked healthy.
"GrpcPsk": "dev-grpc-psk-docker-site-a",
"CentralContactPoints": [
"akka.tcp://scadabridge@scadabridge-central-a:8081",
"akka.tcp://scadabridge@scadabridge-central-b:8081"
docker/site-b-node-a/appsettings.Site.json
+8 -1
View File
@@ -14,7 +14,7 @@
"akka.tcp://scadabridge@scadabridge-site-b-a:8082",
"akka.tcp://scadabridge@scadabridge-site-b-b:8082"
],
"SplitBrainResolverStrategy": "keep-oldest",
"SplitBrainResolverStrategy": "auto-down",
"StableAfter": "00:00:15",
"HeartbeatInterval": "00:00:02",
"FailureDetectionThreshold": "00:00:10",
@@ -41,6 +41,13 @@
"SqliteDbPath": "/app/data/store-and-forward.db"
},
"Communication": {
// DEV-ONLY control-plane preshared key — NOT a real secret. Must be
// IDENTICAL on both nodes of the pair and match the central-side entry in
// ScadaBridge__Communication__SitePsks__<siteId> (docker-compose.yml).
// Production supplies this as ${secret:SB-GRPC-PSK-<siteId>}. Without it the
// node fails StartupValidator: the gate is fail-closed, so an unset key would
// refuse every SiteStream call while the node still looked healthy.
"GrpcPsk": "dev-grpc-psk-docker-site-b",
"CentralContactPoints": [
"akka.tcp://scadabridge@scadabridge-central-a:8081",
"akka.tcp://scadabridge@scadabridge-central-b:8081"
docker/site-b-node-b/appsettings.Site.json
+10 -3
View File
@@ -11,10 +11,10 @@
},
"Cluster": {
"SeedNodes": [
"akka.tcp://scadabridge@scadabridge-site-b-a:8082",
"akka.tcp://scadabridge@scadabridge-site-b-b:8082"
"akka.tcp://scadabridge@scadabridge-site-b-b:8082",
"akka.tcp://scadabridge@scadabridge-site-b-a:8082"
],
"SplitBrainResolverStrategy": "keep-oldest",
"SplitBrainResolverStrategy": "auto-down",
"StableAfter": "00:00:15",
"HeartbeatInterval": "00:00:02",
"FailureDetectionThreshold": "00:00:10",
@@ -41,6 +41,13 @@
"SqliteDbPath": "/app/data/store-and-forward.db"
},
"Communication": {
// DEV-ONLY control-plane preshared key — NOT a real secret. Must be
// IDENTICAL on both nodes of the pair and match the central-side entry in
// ScadaBridge__Communication__SitePsks__<siteId> (docker-compose.yml).
// Production supplies this as ${secret:SB-GRPC-PSK-<siteId>}. Without it the
// node fails StartupValidator: the gate is fail-closed, so an unset key would
// refuse every SiteStream call while the node still looked healthy.
"GrpcPsk": "dev-grpc-psk-docker-site-b",
"CentralContactPoints": [
"akka.tcp://scadabridge@scadabridge-central-a:8081",
"akka.tcp://scadabridge@scadabridge-central-b:8081"
docker/site-c-node-a/appsettings.Site.json
+8 -1
View File
@@ -14,7 +14,7 @@
"akka.tcp://scadabridge@scadabridge-site-c-a:8082",
"akka.tcp://scadabridge@scadabridge-site-c-b:8082"
],
"SplitBrainResolverStrategy": "keep-oldest",
"SplitBrainResolverStrategy": "auto-down",
"StableAfter": "00:00:15",
"HeartbeatInterval": "00:00:02",
"FailureDetectionThreshold": "00:00:10",
@@ -41,6 +41,13 @@
"SqliteDbPath": "/app/data/store-and-forward.db"
},
"Communication": {
// DEV-ONLY control-plane preshared key — NOT a real secret. Must be
// IDENTICAL on both nodes of the pair and match the central-side entry in
// ScadaBridge__Communication__SitePsks__<siteId> (docker-compose.yml).
// Production supplies this as ${secret:SB-GRPC-PSK-<siteId>}. Without it the
// node fails StartupValidator: the gate is fail-closed, so an unset key would
// refuse every SiteStream call while the node still looked healthy.
"GrpcPsk": "dev-grpc-psk-docker-site-c",
"CentralContactPoints": [
"akka.tcp://scadabridge@scadabridge-central-a:8081",
"akka.tcp://scadabridge@scadabridge-central-b:8081"
docker/site-c-node-b/appsettings.Site.json
+10 -3
View File
@@ -11,10 +11,10 @@
},
"Cluster": {
"SeedNodes": [
"akka.tcp://scadabridge@scadabridge-site-c-a:8082",
"akka.tcp://scadabridge@scadabridge-site-c-b:8082"
"akka.tcp://scadabridge@scadabridge-site-c-b:8082",
"akka.tcp://scadabridge@scadabridge-site-c-a:8082"
],
"SplitBrainResolverStrategy": "keep-oldest",
"SplitBrainResolverStrategy": "auto-down",
"StableAfter": "00:00:15",
"HeartbeatInterval": "00:00:02",
"FailureDetectionThreshold": "00:00:10",
@@ -41,6 +41,13 @@
"SqliteDbPath": "/app/data/store-and-forward.db"
},
"Communication": {
// DEV-ONLY control-plane preshared key — NOT a real secret. Must be
// IDENTICAL on both nodes of the pair and match the central-side entry in
// ScadaBridge__Communication__SitePsks__<siteId> (docker-compose.yml).
// Production supplies this as ${secret:SB-GRPC-PSK-<siteId>}. Without it the
// node fails StartupValidator: the gate is fail-closed, so an unset key would
// refuse every SiteStream call while the node still looked healthy.
"GrpcPsk": "dev-grpc-psk-docker-site-c",
"CentralContactPoints": [
"akka.tcp://scadabridge@scadabridge-central-a:8081",
"akka.tcp://scadabridge@scadabridge-central-b:8081"
docs/components/ClusterInfrastructure.md
+96 -61
View File
@@ -1,23 +1,33 @@
# Cluster Infrastructure
The Cluster Infrastructure component manages Akka.NET cluster formation, active/standby failover, split-brain resolution, and the singleton hosting that all other ScadaBridge components depend on. Every site and central cluster is a two-node active/standby pair governed by the same configuration contract and bootstrap logic.
The Cluster Infrastructure component manages Akka.NET cluster formation, active/standby failover, the downing strategy for unreachable members, and the singleton hosting that all other ScadaBridge components depend on. Every site and central cluster is a two-node active/standby pair governed by the same configuration contract and bootstrap logic.
## Overview
Cluster Infrastructure (#13) is a **design responsibility** spanning two projects rather than a single buildable project:
- **`src/ZB.MOM.WW.ScadaBridge.ClusterInfrastructure/`** owns the cluster configuration contract: `ClusterOptions` (seed nodes, failure-detection timings, split-brain settings), `ClusterOptionsValidator`, and the `AddClusterInfrastructure` DI extension that registers the validator. It does not start an actor system.
- **`src/ZB.MOM.WW.ScadaBridge.ClusterInfrastructure/`** owns the cluster configuration contract: `ClusterOptions` (seed nodes, failure-detection timings, downing strategy), `ClusterOptionsValidator`, and the `AddClusterInfrastructure` DI extension that registers the validator. It does not start an actor system.
- **`src/ZB.MOM.WW.ScadaBridge.Host/`** owns the cluster bootstrap and runtime wiring: `AkkaHostedService` builds the Akka HOCON from `ClusterOptions` and `NodeOptions`, starts the `ActorSystem`, wires `CoordinatedShutdown`, and creates all role-specific actors including the cluster singletons.
This split is deliberate. The Host is the single deployable binary and the only project that performs Akka.NET bootstrap, so all cluster bring-up lives there. `ClusterInfrastructure` is the portable configuration contract that the Host consumes — it can be referenced by tests and other components without pulling in the Host.
Both central and site clusters run this same topology: two nodes, one active (cluster leader), one standby, with automatic failover and no manual intervention required for dual-node recovery.
Both central and site clusters run this same topology: two nodes, one active (the oldest `Up` member), one standby, with automatic failover and no manual intervention required for dual-node recovery.
## Key Concepts
### Active/standby via cluster leadership
### One `ActorSystem` name for every cluster
Akka.NET cluster leadership determines which node is "active". The cluster leader is the oldest node in the cluster, as tracked by the keep-oldest split-brain resolver. `ActiveNodeGate` (in the Host) exposes `IsActiveNode` by checking whether `cluster.SelfMember.Status == MemberStatus.Up` and `cluster.State.Leader == cluster.SelfAddress`. Cluster singletons — which run on the oldest `Up` member — automatically migrate to the surviving node on failover.
Every node in every cluster — central and all sites — joins an `ActorSystem` named **`"scadabridge"`**, hardcoded at `AkkaHostedService.cs:191` (`ActorSystem.Create("scadabridge", config)`). Central and each site are separate clusters *only* by seed-node partitioning, not by system name. This is required rather than incidental: Akka.Remote matches addresses including the system name, so a `ClusterClient` could not reach a differently-named system.
### Active/standby is the oldest `Up` member — never the cluster leader
A node is "active" when it is the **oldest `Up` member** of its role scope — the member `ClusterSingletonManager` places singletons on. Akka's *cluster leader* (lowest address) is a different, Akka-internal concept: it diverges from singleton placement permanently once the original first node restarts and rejoins. Every product-level active/standby decision therefore goes through one evaluator and never reads `cluster.State.Leader`:
- `ActiveNodeEvaluator.SelfIsOldestUp(Cluster, string? role)` (`Communication/ClusterState/ActiveNodeEvaluator.cs:35`) is the single implementation — self is `Up`, carries the role when one is given, and no other `Up` member in that scope is older (`self.IsOlderThan(m)`).
- `ClusterActivityEvaluator.SelfIsOldest` (`Host/Health/ClusterActivityEvaluator.cs:23`) delegates to it, and is what `ActiveNodeGate.IsActiveNode` (`Host/Health/ActiveNodeGate.cs:48`), `OldestNodeActiveHealthCheck`, and `AkkaClusterNodeProvider.SelfIsPrimary` all call.
- `SiteCommunicationActor` stamps its heartbeat's `IsActive` from the same evaluator (`Communication/Actors/SiteCommunicationActor.cs:517-518`).
Cluster singletons automatically migrate to the surviving node on failover, and because "active" is defined as the singleton-placement member, the health/routing view and the singleton view can never disagree.
### Configuration contract vs. bootstrap split
@@ -33,22 +43,26 @@ Cluster Infrastructure provides the hosting platform; each singleton is owned an
`AkkaHostedService.BuildHocon` constructs the Akka HOCON document from the bound options at startup. All interpolated values pass through `QuoteHocon` (string escaping) and `DurationHocon` (millisecond rendering) so the document is never corrupted by hostnames or timing values containing special characters or sub-second precision.
The snippet below is abbreviated to highlight the cluster stanzas. The full method also emits three additional stanzas: `akka.extensions` (registers `DistributedPubSubExtensionProvider`), `akka.remote.dot-netty.tcp` (binds `NodeOptions.NodeHostname` and `NodeOptions.RemotingPort`), and `akka.remote.transport-failure-detector` (heartbeat interval and acceptable-heartbeat-pause from `CommunicationOptions.TransportHeartbeatInterval` / `TransportFailureThreshold`).
The snippet below is abbreviated to highlight the cluster stanzas. The full method also emits `akka.extensions` (registers `DistributedPubSubExtensionProvider`), `akka.remote.dot-netty.tcp` (binds `NodeOptions.NodeHostname` and `NodeOptions.RemotingPort`), and `akka.remote.transport-failure-detector` (heartbeat interval and acceptable-heartbeat-pause from `CommunicationOptions.TransportHeartbeatInterval` / `TransportFailureThreshold`).
The downing block is **not** a fixed stanza — `BuildHocon` branches on `ClusterOptions.SplitBrainResolverStrategy` and emits one of two shapes (`AkkaHostedService.cs:275-286`):
```csharp
// Abbreviated — see AkkaHostedService.BuildHocon for the full method.
public static string BuildHocon(
NodeOptions nodeOptions,
ClusterOptions clusterOptions,
IEnumerable<string> roles,
TimeSpan transportHeartbeat,
TimeSpan transportFailure)
{
var seedNodesStr = string.Join(",",
clusterOptions.SeedNodes.Select(QuoteHocon));
var rolesStr = string.Join(",", roles.Select(QuoteHocon));
var downingBlock = string.Equals(
clusterOptions.SplitBrainResolverStrategy, "auto-down", StringComparison.OrdinalIgnoreCase)
? $@"downing-provider-class = ""Akka.Cluster.AutoDowning, Akka.Cluster""
auto-down-unreachable-after = {DurationHocon(clusterOptions.StableAfter)}"
: $@"downing-provider-class = ""Akka.Cluster.SBR.SplitBrainResolverProvider, Akka.Cluster""
split-brain-resolver {{
active-strategy = {QuoteHocon(clusterOptions.SplitBrainResolverStrategy)}
stable-after = {DurationHocon(clusterOptions.StableAfter)}
keep-oldest {{
down-if-alone = {(clusterOptions.DownIfAlone ? "on" : "off")}
}}
}}";
return $@"
return $@"
audit-telemetry-dispatcher {{
type = ForkJoinDispatcher
throughput = 100
@@ -66,13 +80,7 @@ akka {{
seed-nodes = [{seedNodesStr}]
roles = [{rolesStr}]
min-nr-of-members = {clusterOptions.MinNrOfMembers}
split-brain-resolver {{
active-strategy = {QuoteHocon(clusterOptions.SplitBrainResolverStrategy)}
stable-after = {DurationHocon(clusterOptions.StableAfter)}
keep-oldest {{
down-if-alone = {(clusterOptions.DownIfAlone ? "on" : "off")}
}}
}}
{downingBlock}
failure-detector {{
heartbeat-interval = {DurationHocon(clusterOptions.HeartbeatInterval)}
acceptable-heartbeat-pause = {DurationHocon(clusterOptions.FailureDetectionThreshold)}
@@ -83,23 +91,35 @@ akka {{
run-by-clr-shutdown-hook = on
}}
}}";
}
```
A `downing-provider-class` is always named explicitly. Akka defaults to `NoDowning`, under which the downing configuration is inert and singletons never migrate on a hard crash or partition; naming the provider is what activates automatic downing.
The HOCON also defines the `audit-telemetry-dispatcher` (a two-thread `ForkJoinDispatcher`) so `SiteAuditTelemetryActor`'s SQLite reads and gRPC pushes never contend with the default dispatcher used by hot-path actors.
### Split-brain resolution
Nothing in the emitted document enables remoting TLS or an Akka secure cookie — there is no `enable-ssl`, no `require-cookie`, no `trusted-selection-paths`. Akka remoting between nodes and from a `ClusterClient` is plaintext and unauthenticated; the deployment is assumed to sit on a trusted network.
The keep-oldest strategy is the only strategy `ClusterOptionsValidator` permits for ScadaBridge's two-node clusters. Quorum strategies (`keep-majority`, `static-quorum`) cannot distinguish a crash from a partition with two nodes — both sides would be below quorum and both would shut down. Keep-oldest with `down-if-alone = on` ensures at most one node runs the cluster at any time:
### Downing strategy (auto-down — availability-first)
- On a network partition, the older node stays active; the younger node downs itself.
- If the oldest node finds itself alone (no reachable members), it downs itself rather than running in isolation. Without `down-if-alone`, the oldest node could run as a single-node cluster while the younger node forms its own — producing two live clusters with divergent singleton state.
**Decision 2026-07-21** (`docs/plans/2026-07-21-auto-down-availability-decision.md`): the default strategy is **`auto-down`** — Akka's `AutoDowning` provider with `auto-down-unreachable-after` = `StableAfter` (15 s). The leader among the *reachable* members downs the unreachable peer once the stability window elapses.
- **Either-node crash is survivable.** If the standby crashes, the active node downs it and continues. If the **active/oldest** node crashes, the younger survivor downs the dead oldest, becomes the oldest itself, re-hosts every cluster singleton, and `/health/active` flips to it — no operator action and no victim restart.
- **The accepted trade is dual-active during a real network partition.** With both nodes alive but the link cut, each side downs the other and continues as a one-node cluster; both claim active until an operator restarts one side after the partition heals. This was chosen deliberately — pairs run one node per VM with no shared lease store (no Kubernetes, no site-side SQL) to arbitrate, and a stalled system is a bigger operational risk than a rare LAN partition.
- **`StableAfter` is the debounce**, not a resolver phase: 15 s of sustained unreachability before downing, which absorbs startup, rolling restarts, and transient blips.
`keep-oldest` remains a supported value (`ClusterOptionsValidator` allows exactly `auto-down` and `keep-oldest`) for deployments that prefer partition-safety, but it **cannot survive a crash of the oldest node in a two-node cluster**: Akka's `down-if-alone` only rescues the survivor when its own side has ≥ 2 members, so a 1-vs-1 survivor takes `DownReachable` and downs *itself*. Quorum strategies are rejected outright — `static-quorum` with quorum 1 trips Akka's `IsTooManyMembers` guard and downs *all* members on any unreachability, and `keep-majority` merely moves the fatal crash from the oldest node to the lowest-address node.
### Downed-node recovery
`run-coordinated-shutdown-when-down = on` means a downed node runs `CoordinatedShutdown` and terminates its own `ActorSystem`. The Host watches `ActorSystem.WhenTerminated`; a termination that is not the host's own `StopAsync` calls `IHostApplicationLifetime.StopApplication()` so the process exits and the service supervisor (docker `restart: unless-stopped`, Windows service recovery) restarts it as a fresh incarnation (`AkkaHostedService.cs:203-218`).
**Seed-node ordering (decision 2026-07-22).** Only the *first* seed listed in `Cluster:SeedNodes` may self-join to form a new cluster — Akka runs `FirstSeedNodeProcess` for it and `JoinSeedNodeProcess` (which can never form one) for everyone else. Every node therefore lists **itself** first and its partner second, so any node can boot alone and become operational unattended; `StartupValidator` fails the boot if that ordering is broken. Until this change all nodes shared one first seed, and a node that had to boot alone looped on `InitJoin` until its peer returned — the registered outage gap. See `docs/requirements/Component-ClusterInfrastructure.md` → Seed Node Ordering for the scenario table and for why an external self-form timer was rejected.
### Failure detection and failover timeline
Detection uses two independent Akka heartbeat channels:
- **Cluster failure detector** (`akka.cluster.failure-detector`): monitors membership, triggers `Unreachable` events that the split-brain resolver acts on.
- **Cluster failure detector** (`akka.cluster.failure-detector`): monitors membership, triggers the `Unreachable` events the downing provider acts on.
- **Transport failure detector** (`akka.remote.transport-failure-detector`): monitors the underlying TCP transport between nodes; configured separately from `CommunicationOptions.TransportHeartbeatInterval` / `TransportFailureThreshold`.
With the defaults in `ClusterOptions`, the total failover budget is approximately 25 seconds:
@@ -107,28 +127,33 @@ With the defaults in `ClusterOptions`, the total failover budget is approximatel
| Phase | Duration | Source |
|-------|----------|--------|
| Failure detection (`acceptable-heartbeat-pause`) | 10 s | `ClusterOptions.FailureDetectionThreshold` |
| Split-brain stable-after | 15 s | `ClusterOptions.StableAfter` |
| Downing window (`auto-down-unreachable-after`) | 15 s | `ClusterOptions.StableAfter` |
| Singleton restart | < 1 s | Actor `PreStart` |
The docker failover drill (`docker/failover-drill.sh`) measures both directions — `standby` mode kills the younger node, `active` mode kills the active/oldest node and asserts the survivor takes over while the victim is still down.
### Graceful shutdown and singleton handover
When a node is stopped cleanly, `CoordinatedShutdown` runs before the CLR exits (`run-by-clr-shutdown-hook = on`). The cluster-leave phase signals Akka to migrate singletons before the actor system terminates, so handover happens in seconds rather than waiting for the full failure-detection timeout. `SiteCallAuditActor` has an explicit graceful-stop task registered on `PhaseClusterLeave` with a 10-second timeout to drain any in-flight EF Core upsert before handover opens:
When a node is stopped cleanly, `CoordinatedShutdown` runs before the CLR exits (`run-by-clr-shutdown-hook = on`). The cluster-leave phase signals Akka to migrate singletons before the actor system terminates, so handover happens in seconds rather than waiting for the full failure-detection timeout.
Every singleton is created through the shared `SingletonRegistrar.Start` helper (`Host/Actors/SingletonRegistrar.cs`), so the drain is uniform rather than per-singleton boilerplate. The registrar applies the canonical `{name}-singleton` / `{name}-proxy` naming, a `PoisonPill` termination message, an optional `.WithRole(role)` on both the manager and proxy settings, and a `PhaseClusterLeave` task that `GracefulStop`s the manager (10-second default) so in-flight EF Core (central) or SQLite (site) work completes before handover opens:
```csharp
siteCallAuditShutdown.AddTask(
// SingletonRegistrar.Start — the drain task registered for every singleton
Akka.Actor.CoordinatedShutdown.Get(system).AddTask(
Akka.Actor.CoordinatedShutdown.PhaseClusterLeave,
"drain-site-call-audit-singleton",
$"drain-{name}-singleton",
async () =>
{
try
{
await siteCallAuditSingletonManager.GracefulStop(TimeSpan.FromSeconds(10));
await manager.GracefulStop(timeout);
}
catch (Exception ex)
{
_logger.LogWarning(ex,
"SiteCallAudit singleton did not drain within the graceful-stop "
+ "timeout; falling through to PoisonPill handover");
logger.LogWarning(ex,
"{Singleton} singleton did not drain within the graceful-stop timeout; "
+ "falling through to PoisonPill handover", name);
}
return Akka.Done.Instance;
});
@@ -136,25 +161,29 @@ siteCallAuditShutdown.AddTask(
### Cluster roles and singleton scoping
Each node carries one or more cluster roles set in the HOCON `roles` list. Site nodes carry both a base `"Site"` role and a site-specific role (`"site-{SiteId}"`, e.g. `"site-site-a"`). Singletons on site clusters are scoped to the site-specific role so each site's singleton runs on exactly one node of that site's cluster, not on any other site's nodes. Central singletons use no role scope — all central nodes share the `"Central"` role.
Each node carries one or more cluster roles set in the HOCON `roles` list, built by `AkkaHostedService.BuildRoles` (`AkkaHostedService.cs:406-417`). Site nodes carry **two** roles: the base `"Site"` role plus a site-specific `"site-{SiteId}"` (a node with `SiteId: "site-a"` gets `"site-site-a"`). Singletons on site clusters are scoped to the site-specific role so each site's singleton runs on exactly one node of that site's cluster. Central singletons pass no role to the registrar and so are unscoped — all central nodes share the `"Central"` role.
### Dual-node recovery
Because both nodes are configured as seed nodes, whichever node starts first after a simultaneous failure forms a new cluster; the second joins when it comes up. No startup ordering dependency exists, and no manual intervention is required. The keep-oldest resolver handles the "both starting fresh" case naturally — there is no pre-existing cluster to conflict with.
Because both nodes are configured as seed nodes **and each lists itself first**, whichever node starts first after a simultaneous failure forms a new cluster; the second joins when it comes up. There is no pre-existing cluster to conflict with, so the "both starting fresh" case needs no downing decision at all. Since 2026-07-22 there is no remaining ordering dependency: a node that must boot *alone* forms a cluster regardless of which node it is. Two nodes cold-starting at the same moment converge on one cluster via the `InitJoin` handshake — they split only under a genuine boot-time partition, the same class `auto-down` already accepts.
### Cluster singletons hosted
The Host wires the following singletons. Cluster Infrastructure provides the `ClusterSingletonManager` / `ClusterSingletonProxy` pattern; each singleton's behaviour is documented in the owning component.
The Host wires the following singletons through `SingletonRegistrar.Start`. Cluster Infrastructure provides the `ClusterSingletonManager` / `ClusterSingletonProxy` pattern and the drain hook; each singleton's behaviour is documented in the owning component.
**Central singletons (active central node, no role scope):**
**Central singletons (oldest `Up` central node, no role scope):**
| Singleton name | Actor class | Owner |
|----------------|-------------|-------|
| `notification-outbox` | `NotificationOutboxActor` | Notification Outbox (#21) |
| `audit-log-ingest` | `AuditLogIngestActor` | Audit Log (#23) |
| `site-call-audit` | `SiteCallAuditActor` | Site Call Audit (#22) |
| `audit-log-purge` | `AuditLogPurgeActor` | Audit Log (#23) |
| `site-audit-reconciliation` | `SiteAuditReconciliationActor` | Audit Log (#23) |
| `kpi-history-recorder` | `KpiHistoryRecorderActor` | KPI History |
| `pending-deployment-purge` | `PendingDeploymentPurgeActor` | Deployment Manager (#2) |
**Site singletons (active site node, scoped to `"site-{SiteId}"` role):**
**Site singletons (oldest `Up` node of that site, scoped to the `"site-{SiteId}"` role):**
| Singleton name | Actor class | Owner |
|----------------|-------------|-------|
@@ -173,29 +202,29 @@ Every host calls `AddClusterInfrastructure` to register `ClusterOptionsValidator
services.AddClusterInfrastructure();
```
This registers `ClusterOptionsValidator` as an `IValidateOptions<ClusterOptions>` singleton. Because the Host binds `ClusterOptions` with `ValidateOnStart`, a misconfigured `ScadaBridge:Cluster` section (wrong strategy, `MinNrOfMembers != 1`, `DownIfAlone = false`, fewer than two seed nodes) throws an `OptionsValidationException` at startup rather than booting into a broken cluster.
This registers `ClusterOptionsValidator` as an `IValidateOptions<ClusterOptions>` singleton. Because the Host binds `ClusterOptions` with `ValidateOnStart`, a misconfigured `ScadaBridge:Cluster` section throws an `OptionsValidationException` at startup rather than booting into a broken cluster. The validator rejects: a strategy other than `auto-down` or `keep-oldest`; `MinNrOfMembers != 1`; a non-positive `StableAfter`, `HeartbeatInterval` or `FailureDetectionThreshold`; a `HeartbeatInterval` not below `FailureDetectionThreshold`; fewer than two seed nodes unless `AllowSingleNodeCluster = true`; and `DownIfAlone = false` **only when the strategy is `keep-oldest`** (the flag is inert under `auto-down`, so any value passes there).
### Checking active-node status
Components that must run only on the active node resolve `IActiveNodeGate` (registered by the Host's Central composition root):
Components that must run only on the active node resolve `IActiveNodeGate` (registered by the Host's Central composition root). The gate is a thin wrapper over the oldest-`Up` evaluator — it never inspects cluster leadership:
```csharp
// Host/Health/ActiveNodeGate.cs
public bool IsActiveNode
{
get
{
var system = _akkaService.ActorSystem;
if (system == null) return false;
if (system == null)
return false;
var cluster = Cluster.Get(system);
var self = cluster.SelfMember;
if (self.Status != MemberStatus.Up) return false;
var leader = cluster.State.Leader;
return leader != null && leader == self.Address;
return ClusterActivityEvaluator.SelfIsOldest(cluster);
}
}
```
This returns `false` while the actor system is warming up — the safe-by-default answer matching the standby case. The Inbound API uses this gate to return HTTP 503 on standby nodes.
This returns `false` while the actor system is warming up, and `SelfIsOldest` returns `false` unless the node has reached `MemberStatus.Up` — the safe-by-default answer matching the standby case. The Inbound API uses this gate to return HTTP 503 on standby nodes, and `OldestNodeActiveHealthCheck` backs `/health/active` off the same evaluator, so the proxy's routing decision and the API's gating decision can never disagree.
## Configuration
@@ -205,13 +234,14 @@ This returns `false` while the actor system is warming up — the safe-by-defaul
| Key | Type | Default | Description |
|-----|------|---------|-------------|
| `SeedNodes` | `List<string>` | (required) | Akka seed-node URIs. Must contain at least 2 entries; both nodes list both themselves and their partner. |
| `SplitBrainResolverStrategy` | `string` | `"keep-oldest"` | Must be `"keep-oldest"`. Quorum strategies are rejected by `ClusterOptionsValidator`. |
| `StableAfter` | `TimeSpan` | `00:00:15` | Cluster must be stable for this duration before the resolver acts to down unreachable nodes. |
| `SeedNodes` | `List<string>` | (required) | Akka seed-node URIs. Must contain at least 2 entries (1 with `AllowSingleNodeCluster`); both nodes list both themselves and their partner. Only the **first** entry may self-form a new cluster. |
| `SplitBrainResolverStrategy` | `string` | `"auto-down"` | `"auto-down"` or `"keep-oldest"`. Quorum strategies are rejected by `ClusterOptionsValidator`. See downing strategy above. |
| `StableAfter` | `TimeSpan` | `00:00:15` | Sustained unreachability before downing. Emitted as `auto-down-unreachable-after` under `auto-down`, as the SBR `stable-after` under `keep-oldest`. |
| `HeartbeatInterval` | `TimeSpan` | `00:00:02` | Cluster failure-detector heartbeat frequency. Must be less than `FailureDetectionThreshold`. |
| `FailureDetectionThreshold` | `TimeSpan` | `00:00:10` | `acceptable-heartbeat-pause` for the cluster failure detector. |
| `MinNrOfMembers` | `int` | `1` | Must be `1`. A value of `2` blocks the cluster singleton after failover. |
| `DownIfAlone` | `bool` | `true` | Must be `true`. See split-brain resolution above. |
| `DownIfAlone` | `bool` | `true` | `keep-oldest` only — inert under `auto-down`. Validated as `true` only when the strategy is `keep-oldest`. |
| `AllowSingleNodeCluster` | `bool` | `false` | Acknowledges a deliberate single-node install: permits exactly one seed node instead of the usual two. |
### `ScadaBridge:Node`
@@ -241,7 +271,7 @@ This returns `false` while the actor system is warming up — the safe-by-defaul
"akka.tcp://scadabridge@scadabridge-central-a:8081",
"akka.tcp://scadabridge@scadabridge-central-b:8081"
],
"SplitBrainResolverStrategy": "keep-oldest",
"SplitBrainResolverStrategy": "auto-down",
"StableAfter": "00:00:15",
"HeartbeatInterval": "00:00:02",
"FailureDetectionThreshold": "00:00:10",
@@ -251,7 +281,7 @@ This returns `false` while the actor system is warming up — the safe-by-defaul
}
```
`DownIfAlone` is not present in the docker files because its default value of `true` is correct and `ClusterOptionsValidator` rejects `false`.
`DownIfAlone` is not present in the docker files because it is a `keep-oldest`-only knob and every shipped deployment runs `auto-down`, under which the flag is inert.
## Dependencies & Interactions
@@ -261,22 +291,26 @@ This returns `false` while the actor system is warming up — the safe-by-defaul
- [Site Runtime (#3)](./SiteRuntime.md) — the Deployment Manager singleton is the most operationally critical singleton this infrastructure hosts. It re-creates the full Instance Actor hierarchy from local SQLite on failover. Staggered Instance Actor startup after failover is Site Runtime's responsibility; this component provides the singleton placement guarantee.
- [Notification Outbox (#21)](./NotificationOutbox.md), [Site Call Audit (#22)](./SiteCallAudit.md), [Audit Log (#23)](./AuditLog.md) — each hosts one or more central singletons wired by `RegisterCentralActors`. Cluster Infrastructure provides the `ClusterSingletonManager`/`ClusterSingletonProxy` boilerplate and the graceful-shutdown hooks; the business logic lives in the owning component.
- [Central–Site Communication (#5)](./Communication.md) — `CentralCommunicationActor` and `SiteCommunicationActor` are created and registered with `ClusterClientReceptionist` inside the same `AkkaHostedService` startup, making them addressable by remote `ClusterClient` instances. The transport-level heartbeat (`TransportHeartbeatInterval`, `TransportFailureThreshold`) is configured separately from the cluster failure-detector and comes from `CommunicationOptions`.
- [Inbound API (#14)](./InboundAPI.md) — resolves `IActiveNodeGate` to return HTTP 503 on standby central nodes. Gate returns `false` until the actor system is `Up` and this node is the cluster leader.
- [Inbound API (#14)](./InboundAPI.md) — resolves `IActiveNodeGate` to return HTTP 503 on standby central nodes. Gate returns `false` until the actor system is `Up` and this node is the oldest `Up` member.
- Design spec: [Component-ClusterInfrastructure.md](../requirements/Component-ClusterInfrastructure.md).
## Troubleshooting
### Node fails to join cluster on startup
`ClusterOptionsValidator` rejects fewer than two seed nodes, a non-`keep-oldest` strategy, `MinNrOfMembers != 1`, or `DownIfAlone = false` at startup with an `OptionsValidationException`. Check that both seed-node URIs reference the Akka remoting port, not the gRPC port (8083) or metrics port (8084) — on site nodes, `StartupValidator` explicitly rejects seed entries whose port matches `GrpcPort`.
`ClusterOptionsValidator` rejects fewer than two seed nodes (without `AllowSingleNodeCluster`), a strategy outside `auto-down` / `keep-oldest`, `MinNrOfMembers != 1`, or `DownIfAlone = false` under `keep-oldest`, at startup with an `OptionsValidationException`. Check that both seed-node URIs reference the Akka remoting port, not the gRPC port (8083) or metrics port (8084) — on site nodes, `StartupValidator` explicitly rejects seed entries whose port matches `GrpcPort`.
A node that boots, logs no validation error, but never reaches `Up` was — before 2026-07-22 — usually hitting the seed-node bootstrap constraint: it was not the first entry in `SeedNodes` and the first seed was down, so it looped on `InitJoin` waiting for a peer that could form the cluster. Self-first ordering plus the `StartupValidator` rule that enforces it should make this unreachable; if you still see it, check that `seed-nodes[0]` really resolves to this node's own `NodeHostname:RemotingPort` (the validator compares host *and* port, and Akka does no DNS canonicalisation — `node-a` and `node-a.example.com` are different seed identities).
### Singleton not starting after failover
If the surviving node is `Up` but singletons do not start, `MinNrOfMembers` is the first thing to check. A value of `2` keeps the surviving node waiting for a second member indefinitely. The validator enforces `1`, but a manually patched `appsettings.json` that bypasses the validator could produce this.
### Two live clusters (split-brain)
### Two live clusters (dual-active)
If `DownIfAlone = false` were accepted (the validator rejects it), the oldest node could run alone while the younger forms its own cluster, producing two live clusters with divergent singleton state and dual MS SQL writers on central. `ClusterOptionsValidator` makes this configuration impossible to boot.
Under `auto-down` this is the **accepted trade, not a misconfiguration**: during a real network partition each side downs the other and continues as a one-node cluster, so both nodes are oldest-`Up`, both host a full set of singletons, and both answer `/health/active` with 200 — including dual MS SQL writers on central. Monitoring surfaces it directly (both nodes stamp `IsActive` on their heartbeats; the Health dashboard shows two Primaries). The two sides do **not** merge on their own — the mutual downing quarantines the association. Recovery is operator-driven: once the link is restored, restart **one** side; it rejoins its peer as a fresh incarnation and comes back as standby.
Deployments that would rather lose availability than run dual-active should set `SplitBrainResolverStrategy: "keep-oldest"` (with `DownIfAlone = true`), accepting that a crash of the oldest node is then a total outage.
### Graceful shutdown takes longer than expected
@@ -285,6 +319,7 @@ If a clean node stop takes up to 25 seconds instead of seconds, `CoordinatedShut
## Related Documentation
- [Cluster Infrastructure design specification](../requirements/Component-ClusterInfrastructure.md)
- [Auto-down downing strategy — availability over partition-safety (decision, 2026-07-21)](../plans/2026-07-21-auto-down-availability-decision.md)
- [Host](./Host.md)
- [Site Runtime](./SiteRuntime.md)
- [Health Monitoring](./HealthMonitoring.md)
docs/components/Communication.md
+60 -24
View File
@@ -1,6 +1,6 @@
# Central–Site Communication
The Central–Site Communication component is the transport layer that connects the central cluster to every site cluster. It provides two independent transports — Akka.NET `ClusterClient` for command/control and gRPC server-streaming for real-time data — wired together through a pair of actors that each cluster registers with the `ClusterClientReceptionist`.
The Central–Site Communication component is the transport layer that connects the central cluster to every site cluster. It provides three independent transports — Akka.NET `ClusterClient` for command/control, gRPC server-streaming for real-time data, and plain token-gated HTTP for the deployment-config fetch — anchored by a pair of actors that each cluster registers with the `ClusterClientReceptionist`.
## Overview
@@ -18,15 +18,39 @@ DI registration is called from the Host composition root via `AddCommunication`.
## Key Concepts
### Two transports, two concerns
### Three transports, three concerns
| Transport | Direction | Purpose |
|-----------|-----------|---------|
| Akka.NET `ClusterClient` | bidirectional (command/control) | Deployments, lifecycle, subscribe/unsubscribe handshake, snapshots, heartbeats, health reports, telemetry, notifications |
| gRPC server-streaming (`SiteStreamService`) | site → central | Real-time attribute value and alarm state changes |
| Transport | Who dials | Data direction | Purpose |
|-----------|-----------|----------------|---------|
| Akka.NET `ClusterClient` | both (central → site per site; site → central) | bidirectional | Deploy notifies, lifecycle, subscribe/unsubscribe handshake, snapshots, heartbeats, health reports, telemetry, notifications |
| gRPC (`SiteStreamService`) | **central dials the site** | mostly site → central | Real-time attribute value and alarm state changes (server-streaming), plus the audit ingest/pull unary RPCs |
| HTTP `GET` (`/api/internal/deployments/{id}/config`) | **site dials central** | central → site | The flattened deployment config itself (notify-and-fetch), gated by a per-deployment `X-Deployment-Token` |
The transports are independent. A gRPC stream interruption does not affect in-flight `ClusterClient` commands, and vice versa.
**The gRPC dial direction is inverted from its data direction.** Values flow site → central, but each **site node hosts the gRPC server** and **central is the client**. `MapGrpcService<SiteStreamGrpcServer>()` appears exactly once in the tree, inside the Site branch of `Program.cs` (`Host/Program.cs:542`); there is **no gRPC server on a central node at all**. That is why the two `Ingest*` unary RPCs — nominally a central-side ingest surface — are dead in the shipped topology (acknowledged in `AkkaHostedService.cs:505-508`: "when the gRPC server is not registered (current central topology)"); sites push audit telemetry to central over `ClusterClient` instead, and central pulls with `PullAuditEvents` / `PullSiteCalls` by dialling the site.
**No transport carries transport encryption; two of the three now carry authentication.**
- **Akka remoting / `ClusterClient` — unauthenticated.** `BuildHocon` emits no `enable-ssl`, no secure cookie and no `trusted-selection-paths`, so the command/control path is plaintext and open to anything that can reach the remoting port.
- **gRPC — authenticated by preshared key since 2026-07-22.** The listener is still **h2c** — `ListenAnyIP(grpcPort, o => o.Protocols = HttpProtocols.Http2)` with no `UseHttps` — but `ControlPlaneAuthInterceptor` gates every method under `/sitestream.SiteStreamService/`, including the `PullAuditEvents` / `PullSiteCalls` RPCs that return audit rows. It is fail-closed (no key ⇒ everything refused, and `StartupValidator` will not boot a site node in that state), compares with `CryptographicOperations.FixedTimeEquals`, and rejects with `PermissionDenied`. Central attaches the key via `ControlPlaneCredentials`, which binds `CallCredentials` to each channel so unary and streaming calls are covered uniformly — `SiteStreamGrpcClient`, `GrpcPullAuditEventsInvoker` and `GrpcPullSiteCallsInvoker` all build their channels through it. Keys are **per site** (`SB-GRPC-PSK-<siteId>`), so a compromised site yields only its own. `LocalDbSyncAuthInterceptor` shares the listener and keeps its own separate key on `/localdb_sync.v1.LocalDbSync/` — the two authenticate different peers (central vs. the pair partner) and are never shared.
- **HTTP config fetch — token-authenticated.** Its per-deployment token is the *entire* security boundary (the endpoint is `AllowAnonymous`).
A bearer PSK over plaintext h2c is readable and replayable by anyone on the path, so the design still assumes a trusted network between central and sites — but the bar is now "read the traffic" rather than "reach the port". TLS on these listeners is the follow-on hardening and would not change the key design. Operational detail: [`docs/deployment/topology-guide.md`](../deployment/topology-guide.md).
### Notify-and-fetch: the deployment-config HTTP path
An instance deployment does not carry its flattened configuration inside the Akka message. Central stages a `PendingDeployment` row (config JSON + a freshly generated `DeploymentFetchToken` + a TTL) and sends only a small `RefreshDeploymentCommand` over `ClusterClient`, carrying the deployment id, revision hash, `CentralFetchBaseUrl` and the fetch token. The site's Deployment Manager singleton then calls back to central over plain HTTP:
```csharp
// SiteRuntime/Deployment/HttpDeploymentConfigFetcher.cs
var url = $"{centralFetchBaseUrl.TrimEnd('/')}/api/internal/deployments/{Uri.EscapeDataString(deploymentId)}/config";
using var req = new HttpRequestMessage(HttpMethod.Get, url);
req.Headers.Add("X-Deployment-Token", token);
```
`DeploymentConfigEndpoints.Resolve` (`ManagementService/DeploymentConfigEndpoints.cs:101`) checks existence and TTL *before* the token, so unknown, superseded and expired deployments are all indistinguishable `404`s; a live row with a wrong or missing token is `401`. The token comparison is constant-time. This exists because a flattened config can exceed the default 128 KB Akka frame size, which drops the single oversized message without tearing down the association — heartbeats keep flowing, the site still reports healthy, and the deploy just hangs to its Ask timeout. See `docs/known-issues/2026-06-26-deploy-config-exceeds-akka-frame-size.md`. `DeployArtifactsCommand` was **not** moved to this path and still carries its payload inline.
### Hub-and-spoke topology
Sites do not communicate with each other. All inter-cluster traffic flows through central. Central maintains one `ClusterClient` per site; each site maintains a single `ClusterClient` pointed at both central nodes.
@@ -36,9 +60,9 @@ Sites do not communicate with each other. All inter-cluster traffic flows throug
Central-side callers wrap outbound messages in a `SiteEnvelope(SiteId, Message)`. `CentralCommunicationActor` resolves the site's `ClusterClient` by `SiteId` and forwards the inner message to `/user/site-communication` on the site:
```csharp
// CommunicationService.cs — deployment pattern
public async Task<DeploymentStatusResponse> DeployInstanceAsync(
string siteId, DeployInstanceCommand command, CancellationToken cancellationToken = default)
// CommunicationService.cs — deployment pattern (notify-and-fetch)
public async Task<DeploymentStatusResponse> RefreshDeploymentAsync(
string siteId, RefreshDeploymentCommand command, CancellationToken cancellationToken = default)
{
var envelope = new SiteEnvelope(siteId, command);
return await GetActor().Ask<DeploymentStatusResponse>(
@@ -86,7 +110,7 @@ If a site is unreachable when a command arrives, the caller's Ask times out. Cen
`SiteCommunicationActor` is a `ReceiveActor` created at `/user/site-communication` and registered with `ClusterClientReceptionist`. It owns:
- An `IActorRef? _centralClient` — the site's outbound `ClusterClient` to central. Injected post-construction via `RegisterCentralClient`.
- A `Timers`-based heartbeat (default 5-second interval, first tick after 1 second). Each tick sends a `HeartbeatMessage` with `IsActive` stamped from the Akka `Cluster` leader check — the node is active when its `MemberStatus` is `Up` and it holds cluster leadership.
- A `Timers`-based heartbeat on `CommunicationOptions.ApplicationHeartbeatInterval` (default 5 s; deliberately distinct from the Akka.Remote `TransportHeartbeatInterval`, so retuning the transport failure detector cannot silently retune the health heartbeat). Each tick sends a `HeartbeatMessage` whose `IsActive` is stamped from `ActiveNodeEvaluator.SelfIsOldestUp` — the node is active when it is the **oldest `Up` member**, *not* when it holds cluster leadership (`SiteCommunicationActor.cs:517-518`). A throwing active-check is caught and reported as `IsActive = false`.
- Dispatch to local handlers for every inbound command pattern. Handlers for event-log, parked-message, integration, and artifact patterns are registered post-construction via `RegisterLocalHandler`; unregistered patterns receive an inline error reply so the central Ask does not stall.
Site-to-central messages (health reports, audit batches, notification submissions) are sent via:
@@ -108,9 +132,9 @@ A malformed address for one site does not abort the refresh loop — the actor c
### gRPC real-time data transport
Real-time attribute value and alarm state changes are delivered over `SiteStreamService`, a gRPC server-streaming service defined in `sitestream.proto`.
Real-time attribute value and alarm state changes are delivered over `SiteStreamService`, defined in `sitestream.proto`. The **server runs on every site node and the client runs on central** — central dials in to receive the stream (see the transport table above).
**Site-side** — `SiteStreamGrpcServer` (Kestrel HTTP/2, port 8083):
**Site-side** — `SiteStreamGrpcServer` (Kestrel h2c, HTTP/2 only, port 8083):
- Implements `SiteStreamService.SiteStreamServiceBase`.
- For each `SubscribeInstance` call, creates a `StreamRelayActor` (named `stream-relay-{correlationId}-{seq}`) and subscribes it to `ISiteStreamSubscriber` (implemented by `SiteStreamManager` in the Site Runtime project — `SiteStreamGrpcServer` holds only the interface so it does not reference `SiteRuntime` directly).
@@ -144,7 +168,7 @@ private void HandleAttributeValueChanged(AttributeValueChanged msg)
**Central-side** — `SiteStreamGrpcClient` / `SiteStreamGrpcClientFactory`:
- `SiteStreamGrpcClientFactory` (singleton) caches one `SiteStreamGrpcClient` per site identifier. On `GetOrCreate`, it compares the cached client's `Endpoint` to the requested endpoint and atomically replaces a stale client (different endpoint — NodeA→NodeB failover flip, or an edited address) with a fresh one.
- `SiteStreamGrpcClientFactory` (singleton) caches one `SiteStreamGrpcClient` per **`(site, endpoint)` pair** — a `ConcurrentDictionary<(string Site, string Endpoint), SiteStreamGrpcClient>`. The key was widened from site-only to fix an arch-review High: with a site-only key, one debug session's NodeA→NodeB failover flip disposed a channel another session was still using. `GetOrCreate` therefore no longer disposes on endpoint mismatch; both of a site's node channels coexist, and site *removal* (`RemoveSiteAsync`) is the only shared-disposal path. The trade-off is that an edited gRPC address leaves the old endpoint's idle channel cached until site removal or process shutdown — bounded at a handful of entries per site.
- `SiteStreamGrpcClient` opens a `GrpcChannel` with HTTP/2 keepalive (`KeepAlivePingDelay` default 15 s, `KeepAlivePingTimeout` default 10 s, `KeepAlivePingPolicy.Always`). `SubscribeAsync` is a plain `async Task` that calls `SubscribeInstance` and reads the response stream with `await foreach`, invoking `onEvent` for each received event and `onError` on any non-cancellation exception. The caller (`DebugStreamBridgeActor.OpenGrpcStream`) launches it inside a `Task.Run` so the long-running stream loop runs off the actor thread.
### Debug stream session lifecycle
@@ -162,18 +186,22 @@ private void HandleAttributeValueChanged(AttributeValueChanged msg)
### Proto definition summary
```proto
// Protos/sitestream.proto
// Protos/sitestream.proto — six RPCs, all served by the SITE
service SiteStreamService {
rpc SubscribeInstance(InstanceStreamRequest) returns (stream SiteStreamEvent);
rpc SubscribeSite(SiteStreamRequest) returns (stream SiteStreamEvent);
rpc IngestAuditEvents(AuditEventBatch) returns (IngestAck);
rpc IngestCachedTelemetry(CachedTelemetryBatch) returns (IngestAck);
rpc PullAuditEvents(PullAuditEventsRequest) returns (PullAuditEventsResponse);
rpc PullSiteCalls(PullSiteCallsRequest) returns (PullSiteCallsResponse);
}
```
`SubscribeInstance` carries the real-time data stream. The other three RPCs (`IngestAuditEvents`, `IngestCachedTelemetry`, `PullAuditEvents`) serve the Audit Log component's gRPC telemetry push and reconciliation pull paths — `SiteStreamGrpcServer` hosts them on the same port because sites already listen there.
Two are server-streaming: `SubscribeInstance` carries the per-instance real-time stream; `SubscribeSite` is the **site-wide, alarm-only** stream (no instance filter, attribute updates never carried) that feeds the aggregated central live alarm cache. The four unary RPCs serve the Audit Log and Site Call Audit push/pull paths — `SiteStreamGrpcServer` hosts them on the same port because sites already listen there. As noted above, the two `Ingest*` RPCs are dead in the shipped topology (no central gRPC server exists for a site to dial); the two `Pull*` RPCs are live, with central as the caller.
`SiteStreamEvent` uses a `oneof event { AttributeValueUpdate, AlarmStateUpdate }` discriminator. `AlarmStateUpdate` carries the full native alarm condition (fields 8–21) alongside the base computed-alarm fields (1–7), added additively so old clients ignoring unknown fields continue to work.
`SiteStreamEvent` uses a `oneof event { AttributeValueUpdate, AlarmStateUpdate }` discriminator. `AlarmStateUpdate` carries the full native alarm condition (fields 8–23) alongside the base computed-alarm fields (1–7), added additively so old clients ignoring unknown fields continue to work. Field numbers are never reused and evolution is additive only.
The generated C# is **vendored** under `Communication/SiteStreamGrpc/` with the `<Protobuf>` include commented out, so editing `sitestream.proto` does not regenerate on build — regeneration is a manual toggle-build-copy-untoggle.
## Usage
@@ -181,7 +209,7 @@ Central callers interact through `CommunicationService`, which wraps each comman
| Pattern | Method | Timeout |
|---------|--------|---------|
| Instance deployment | `DeployInstanceAsync` | 120 s |
| Instance deployment (notify-and-fetch) | `RefreshDeploymentAsync` | 120 s |
| Instance lifecycle | `DisableInstanceAsync`, `EnableInstanceAsync`, `DeleteInstanceAsync` | 30 s |
| Artifact deployment | `DeployArtifactsAsync` | 60 s |
| Integration routing | `RouteIntegrationCallAsync` | 30 s |
@@ -197,11 +225,11 @@ For real-time streaming, callers use `DebugStreamService.StartStreamAsync`, whic
## Configuration
All options are bound from the `Communication` section via `CommunicationOptions`:
All options are bound from the `ScadaBridge:Communication` section via `CommunicationOptions`:
| Key | Default | Description |
|-----|---------|-------------|
| `DeploymentTimeout` | `00:02:00` | Ask timeout for instance deployment commands. |
| `DeploymentTimeout` | `00:02:00` | Ask timeout for the `RefreshDeploymentCommand` round-trip (covers the site's HTTP config fetch and apply). |
| `LifecycleTimeout` | `00:00:30` | Ask timeout for lifecycle commands (disable, enable, delete). |
| `ArtifactDeploymentTimeout` | `00:01:00` | Ask timeout for system-wide artifact deployment. |
| `QueryTimeout` | `00:00:30` | Ask timeout for remote queries and management commands. |
@@ -213,8 +241,12 @@ All options are bound from the `Communication` section via `CommunicationOptions
| `GrpcKeepAlivePingTimeout` | `00:00:10` | HTTP/2 keepalive PING timeout. |
| `GrpcMaxStreamLifetime` | `04:00:00` | Per-stream session timeout; forces reconnect of zombie streams. |
| `GrpcMaxConcurrentStreams` | `100` | Max concurrent `SubscribeInstance` streams per site node. |
| `TransportHeartbeatInterval` | `00:00:05` | `SiteCommunicationActor` heartbeat cadence. |
| `ApplicationHeartbeatInterval` | `00:00:05` | `SiteCommunicationActor` site→central heartbeat cadence. |
| `TransportHeartbeatInterval` | `00:00:05` | Akka.Remote transport failure-detector heartbeat interval (emitted into the HOCON by the Host). Distinct from the application heartbeat above. |
| `TransportFailureThreshold` | `00:00:15` | Akka remoting failure-detection threshold. |
| `CentralFetchBaseUrl` | `""` | Base URL (Traefik/LB) the site uses to fetch deploy configs from central. Carried in `RefreshDeploymentCommand` so sites need no standing config; **empty makes a deploy impossible** — `DeploymentService` fails fast. |
| `PendingDeploymentTtl` | `00:05:00` | How long a staged `PendingDeployment` row and its fetch token stay valid. Must comfortably cover both site nodes' fetches within one deploy window. |
| `PendingDeploymentPurgeInterval` | `01:00:00` | Cadence of the central `pending-deployment-purge` singleton that sweeps TTL-expired staging rows. Hygiene only — the fetch endpoint already enforces the TTL. |
Three layers of dead-client detection protect the gRPC stream path:
@@ -227,16 +259,16 @@ Three layers of dead-client detection protect the gRPC stream path:
## Dependencies & Interactions
- [Commons (#16)](./Commons.md) — owns all message contracts used by this component: `DeployInstanceCommand`, `SiteEnvelope`, `HeartbeatMessage`, `SiteHealthReport`, `SiteHealthReportReplica`, `RegisterNotificationOutbox`, `RegisterAuditIngest`, `IngestAuditEventsCommand`, `IngestCachedTelemetryCommand`, and all other request/response records. Commons does not hold an Akka package reference, so `RegisterAuditIngest` (which carries an `IActorRef`) lives in this project.
- [Cluster Infrastructure (#13)](./ClusterInfrastructure.md) — provides `ClusterClientReceptionist` registration and the active/standby leader model that `SiteCommunicationActor`'s `IsActive` check and `CentralCommunicationActor`'s `DistributedPubSub` fanout both depend on.
- [Cluster Infrastructure (#13)](./ClusterInfrastructure.md) — provides `ClusterClientReceptionist` registration and the oldest-`Up` active/standby model that `SiteCommunicationActor`'s `IsActive` stamp depends on, plus the single `"scadabridge"` `ActorSystem` name that makes cross-cluster `ClusterClient` addressing possible at all. `CentralCommunicationActor`'s `DistributedPubSub` fanout keeps both central nodes in sync regardless of which one a site's report landed on.
- [Configuration Database (#17)](./ConfigurationDatabase.md) — provides `ISiteRepository.GetAllSitesAsync` for address loading; site records carry `NodeAAddress`, `NodeBAddress`, `GrpcNodeAAddress`, `GrpcNodeBAddress`.
- [Deployment Manager (#2)](./DeploymentManager.md) — the primary consumer of command/control patterns 1–3. `CommunicationService` is injected into the Deployment Manager actor to send deployments, lifecycle commands, and artifact deployments to sites.
- [Deployment Manager (#2)](./DeploymentManager.md) — the primary consumer of command/control patterns 1–3. `CommunicationService` is injected into the Deployment Manager actor to send deploy notifies, lifecycle commands, and artifact deployments to sites. It also owns the staging half of the notify-and-fetch HTTP path (`PendingDeployment` rows + fetch tokens); the endpoint itself is served by the Management Service.
- [Site Runtime (#3)](./SiteRuntime.md) — `SiteCommunicationActor` forwards inbound commands to the `DeploymentManager` singleton proxy. `SiteStreamManager` (in Site Runtime) implements `ISiteStreamSubscriber` so `SiteStreamGrpcServer` can subscribe relay actors to instance event feeds without referencing Site Runtime directly.
- [Health Monitoring (#11)](./HealthMonitoring.md) — `CentralCommunicationActor` calls `ICentralHealthAggregator.MarkHeartbeat` and `ProcessReport` for every inbound heartbeat and health report. `DistributedPubSub` fanout keeps both central nodes' aggregators in sync.
- [Audit Log (#23)](./AuditLog.md) — `SiteStreamGrpcServer` hosts `IngestAuditEvents`, `IngestCachedTelemetry`, and `PullAuditEvents` RPCs. `CentralCommunicationActor` routes `IngestAuditEventsCommand` / `IngestCachedTelemetryCommand` ClusterClient messages to the `AuditLogIngestActor` proxy.
- [Audit Log (#23)](./AuditLog.md) — `SiteStreamGrpcServer` hosts the `IngestAuditEvents`, `IngestCachedTelemetry`, `PullAuditEvents` and `PullSiteCalls` RPCs. Because there is no central gRPC server, the `Ingest*` pair is unused in the shipped topology: sites push audit telemetry over ClusterClient, and `CentralCommunicationActor` routes `IngestAuditEventsCommand` / `IngestCachedTelemetryCommand` to the `AuditLogIngestActor` proxy. The `Pull*` reconciliation RPCs run the other way, with the central `site-audit-reconciliation` singleton dialling each site.
- [Notification Outbox (#21)](./NotificationOutbox.md) — `CentralCommunicationActor` routes `NotificationSubmit` / `NotificationStatusQuery` messages from sites to the `NotificationOutboxActor` proxy. `CommunicationService` Asks the proxy directly for central-UI outbox management calls.
- [Site Call Audit (#22)](./SiteCallAudit.md) — `CommunicationService` Asks the `SiteCallAuditActor` proxy directly for query and relay operations. `SiteCallAuditActor` issues `RetryParkedOperation` / `DiscardParkedOperation` relay commands to sites via `SiteEnvelope`; `SiteCommunicationActor` dispatches them to `_parkedMessageHandler`.
- [Store-and-Forward Engine (#6)](./StoreAndForward.md) — the site S&F Engine drives `NotificationSubmit` forwarding and cached-call telemetry emission through `SiteCommunicationActor`. Parked-message queries and retry/discard relay commands flow back the other way.
- [Management Service (#18)](./ManagementService.md) — `ManagementActor` is registered with `ClusterClientReceptionist` at `/user/management` on central; the CLI connects via its own separate `ClusterClient`. This is a distinct `ClusterClient` usage from the inter-cluster hub-and-spoke connections managed by this component.
- [Management Service (#18)](./ManagementService.md) — `ManagementActor` runs at `/user/management` on central and is reached **in-process** through `ManagementActorHolder`; the CLI connects over HTTP, not `ClusterClient`. (It was `ClusterClientReceptionist`-registered until 2026-07-22, for a CLI that was never built that way.) So this component's `ClusterClient` usage is exclusively the inter-cluster hub-and-spoke connections. Management Service also hosts `DeploymentConfigEndpoints` — the `GET /api/internal/deployments/{id}/config` route that terminates the third (HTTP) transport, mapped in the central-role block alongside `/api/audit/*` and `/management`.
- Design spec: [Component-Communication.md](../requirements/Component-Communication.md).
## Troubleshooting
@@ -257,6 +289,10 @@ A `Warning` at the `Status.Failure` handler in `CentralCommunicationActor` means
After a site node failover, the `DebugStreamBridgeActor` attempts to reconnect to the other node endpoint (`_useNodeA` flips on each error). If both nodes are unreachable, the actor exhausts its 3-retry budget and calls `onTerminated`. The engineer must restart the debug session.
### Deployments fail immediately with a config-fetch error
The site received the `RefreshDeploymentCommand` over ClusterClient but could not complete the HTTP leg. Check `CentralFetchBaseUrl` first — it must be reachable *from the site*, so a value that only resolves inside the central network fails every deploy. A `404` from the fetch means the staged row was unknown, superseded, or past `PendingDeploymentTtl`; a `401` means the row is live but the token did not match. Because the endpoint hides existence, a `404` cannot distinguish "wrong id" from "expired".
### Heartbeats arrive but health reports do not
`SiteCommunicationActor` sends heartbeats and health reports via separate paths. Health reports are sent only when the site's `HealthReportSender` publishes them (every 30 s by default). If heartbeats arrive but reports do not, the health-report sender on the site may have faulted — check site-side logs for errors in `HealthReportSender`.
docs/components/DeploymentManager.md
+20 -6
View File
@@ -26,10 +26,20 @@ Every instance deployment carries two correlated identifiers:
- **`DeploymentId`** — a new `Guid` (formatted `"N"`) minted by `DeploymentService` at the start of each `DeployInstanceAsync` call.
- **`RevisionHash`** — computed by the Template Engine's `RevisionHashService` over the fully resolved `FlattenedConfiguration`. The hash captures the template state at the moment of flattening, so concurrent last-write-wins template edits do not affect an in-flight deployment.
The pair travels inside `DeployInstanceCommand` to the site. The site uses the `DeploymentId` to detect an already-applied identical command (idempotent re-delivery) and uses the `RevisionHash` to reject a stale configuration that predates what is already running.
The pair travels to the site inside the `RefreshDeploymentCommand` notify and is echoed back on the fetched config. The site uses the `DeploymentId` to detect an already-applied identical command (idempotent re-delivery) and uses the `RevisionHash` to reject a stale configuration that predates what is already running.
Central stores the `RevisionHash` on `DeploymentRecord` and, after a confirmed success, on `DeployedConfigSnapshot`. Comparing the snapshot hash against the current-template hash determines whether an instance is stale without a site round-trip.
### Notify-and-fetch: the config does not travel in the Akka message
A deployment crosses the central↔site boundary over **two** transports, not one. Central stages the flattened configuration in a `PendingDeployment` row (config JSON, a generated `DeploymentFetchToken`, and an expiry of `CommunicationOptions.PendingDeploymentTtl`, default 5 minutes) and then sends only a small `RefreshDeploymentCommand` over ClusterClient carrying the deployment id, instance name, revision hash, `CentralFetchBaseUrl` and the fetch token. The site's Deployment Manager singleton fetches the config back over plain HTTP — `GET {CentralFetchBaseUrl}/api/internal/deployments/{deploymentId}/config` with an `X-Deployment-Token` header — and only then runs its normal apply path.
This exists because a flattened configuration can exceed the default 128 KB Akka frame size, and an over-limit message is dropped silently without tearing down the association — the deploy then simply hangs to its Ask timeout. `CentralFetchBaseUrl` is therefore mandatory: `DeployInstanceAsync` fails fast with "CentralFetchBaseUrl is not configured — required for deployment (notify-and-fetch)" rather than attempting a deploy that cannot complete. Note that `DeployArtifactsCommand` was **not** moved to this path — artifact deployment still carries its payload inline and remains exposed to the frame limit.
Staged rows are cleaned up by **TTL only** — they are deliberately not deleted on success or in the failure path. Three things keep that safe: `AddPendingDeploymentAsync` supersedes (deletes) any prior pending row for the same instance before inserting, so at most one row exists per instance; the fetch endpoint enforces the TTL itself, so an un-purged row is not a usable one; and the central `pending-deployment-purge` singleton sweeps expired rows on `PendingDeploymentPurgeInterval` (default 1 hour).
The site's **startup reconciliation** path uses the same endpoint but stages its own rows: a site node reports its local instance→revision-hash map on boot, and central's `ReconcileService` diffs it against the expected deployed set, stages a fresh `PendingDeployment` (with a new token) for each missing or stale instance, and returns the gap plus `CentralFetchBaseUrl` for the node to fetch. Intra-site replication to the standby node does **not** use this path — `deployed_configurations` is a replicated LocalDb table, so the active node's write reaches the peer as an ordinary row change.
### Per-instance operation lock
`OperationLockManager` holds a `Dictionary<string, LockEntry>` keyed by instance `UniqueName`. Each `LockEntry` wraps a `SemaphoreSlim(1,1)` with a reference count so the semaphore is created on first contention and disposed when the last waiter clears. The lock covers all four mutating operations — deploy, disable, enable, delete — so they can never interleave on a single instance. Operations on different instances proceed in parallel.
@@ -67,7 +77,7 @@ The operation lock is in-memory. If the active central node fails mid-deployment
3. **Flatten and validate** — `IFlatteningPipeline.FlattenAndValidateAsync` runs the Template Engine pipeline and returns a `FlatteningPipelineResult` containing the `FlattenedConfiguration`, `RevisionHash`, and a `ValidationResult`. Semantic validation failures (call targets, argument types, trigger operand types, connection binding completeness) are returned to the caller before any record is written.
4. **Pre-deploy site reconciliation** — when the prior `DeploymentRecord` for the instance is `InProgress` or `Failed` with a timeout marker (`"Communication failure:"`), the service queries the site via `CommunicationService.QueryDeploymentStateAsync`. If the site already holds the target revision hash, the prior record is updated to `Success` and no new deployment is sent.
5. **Write `InProgress` record** — a single `DeploymentRecord` insert directly at `InProgress` status (no transient `Pending` hop). `IDeploymentStatusNotifier.NotifyStatusChanged` fires to push the status to the UI.
6. **Send `DeployInstanceCommand`** — the command carries `DeploymentId`, `InstanceUniqueName`, `RevisionHash`, `FlattenedConfigurationJson`, `DeployedBy`, and `Timestamp`.
6. **Stage and notify** — insert a `PendingDeployment` row holding the flattened config JSON and a fresh fetch token, then send `RefreshDeploymentCommand` (`DeploymentId`, `InstanceUniqueName`, `RevisionHash`, `DeployedBy`, staging timestamp, `CentralFetchBaseUrl`, `FetchToken`) via `CommunicationService.RefreshDeploymentAsync`. The site fetches the config over HTTP and replies with the same `DeploymentStatusResponse` as before.
7. **Commit terminal status** — the `DeploymentRecord` is updated to `Success` or `Failed` and saved before any post-success side effects run. This ordering ensures the recorded outcome can never be lost if a post-success write fails.
8. **Post-success side effects** — `ApplyPostSuccessSideEffectsAsync` sets `Instance.State = Enabled` (or preserves `Disabled` on the reconciliation path) and upserts the `DeployedConfigSnapshot`. These writes are best-effort: a failure here is logged at `Error` but does not flip the already-committed `Success` record back to `Failed`.
9. **Audit log** — `IAuditService.LogAsync` records `Deploy` / `DeployFailed` / `DeployReconciled` with the `DeploymentId`, status, and user.
@@ -76,7 +86,7 @@ Any exception in the site round-trip (steps 6–7) writes `DeploymentStatus.Fail
```csharp
// DeploymentService.DeployInstanceAsync — exception handler
var isTimeout = ex is TimeoutException or OperationCanceledException;
var isTimeout = ex is TimeoutException or OperationCanceledException or Akka.Actor.AskTimeoutException;
record.Status = DeploymentStatus.Failed;
record.ErrorMessage = isTimeout
@@ -171,11 +181,11 @@ Options are registered via `AddDeploymentManager` and bound from `ScadaBridge:De
- [Template Engine (#1)](./TemplateEngine.md) — `FlatteningPipeline` delegates to `FlatteningService`, `ValidationService`, and `RevisionHashService`. Template state is captured at flatten time; last-write-wins edits made after flatten do not affect the in-flight deployment. `DiffService.ComputeDiff` powers the deployment diff view.
- [Configuration Database (#17)](./ConfigurationDatabase.md) — owns the EF Core implementation of `IDeploymentManagerRepository`, which stores `DeploymentRecord`, `DeployedConfigSnapshot`, and `SystemArtifactDeploymentRecord`. `IAuditService` (also registered by the Configuration Database component) writes all deployment audit rows.
- [Central–Site Communication (#5)](./Communication.md) — `CommunicationService` provides `DeployInstanceAsync`, `QueryDeploymentStateAsync`, `DeployArtifactsAsync`, `DisableInstanceAsync`, `EnableInstanceAsync`, and `DeleteInstanceAsync`. The communication layer routes by `SiteIdentifier` (string), not DB id; `DeploymentService.ResolveSiteIdentifierAsync` resolves the numeric `SiteId` before each cross-cluster call and treats a missing site row as a hard failure.
- [Commons (#16)](./Commons.md) — owns `DeploymentRecord`, `DeployedConfigSnapshot`, `SystemArtifactDeploymentRecord`, `DeploymentStatus`, `InstanceState`, `DeployInstanceCommand`, `DeployArtifactsCommand`, `DeploymentStateQueryRequest/Response`, `InstanceLifecycleResponse`, and the `IDeploymentManagerRepository` interface.
- [Central–Site Communication (#5)](./Communication.md) — `CommunicationService` provides `RefreshDeploymentAsync`, `QueryDeploymentStateAsync`, `DeployArtifactsAsync`, `DisableInstanceAsync`, `EnableInstanceAsync`, and `DeleteInstanceAsync`, all over the ClusterClient command/control transport. The communication layer routes by `SiteIdentifier` (string), not DB id; `DeploymentService.ResolveSiteIdentifierAsync` resolves the numeric `SiteId` before each cross-cluster call and treats a missing site row as a hard failure. `CommunicationOptions.CentralFetchBaseUrl` / `PendingDeploymentTtl` (also owned by that component) parameterise the notify-and-fetch HTTP leg.
- [Commons (#16)](./Commons.md) — owns `DeploymentRecord`, `DeployedConfigSnapshot`, `SystemArtifactDeploymentRecord`, `PendingDeployment`, `DeploymentFetchToken`, `DeploymentStatus`, `InstanceState`, `RefreshDeploymentCommand`, `DeployInstanceCommand` (retained as the site-side in-process apply DTO), `DeployArtifactsCommand`, `DeploymentStateQueryRequest/Response`, `InstanceLifecycleResponse`, and the `IDeploymentManagerRepository` interface.
- [Site Runtime (#3)](./SiteRuntime.md) — receives `DeployInstanceCommand` and `DeployArtifactsCommand` via the Communication Layer. Site-side apply is all-or-nothing per instance: the Deployment Manager singleton at the site stores the config, compiles all scripts, and creates or replaces the Instance Actor as a unit. A failure at any step is reported back with the specific error message and the previous configuration remains active.
- [Central UI (#9)](./CentralUI.md) — engineers trigger deployments, view diffs, manage instance lifecycle, and deploy system-wide artifacts through the UI. The deployment status page subscribes to `IDeploymentStatusNotifier.StatusChanged` for real-time push updates via Blazor Server SignalR.
- [Management Service (#18)](./ManagementService.md) — the actor-layer entry point for deployment commands received over ClusterClient. It resolves `DeploymentService` and `ArtifactDeploymentService` from a per-message DI scope and forwards `MgmtDeployArtifactsCommand`, `GetDeploymentDiffCommand`, and instance lifecycle requests.
- [Management Service (#18)](./ManagementService.md) — the actor-layer entry point for deployment commands received over ClusterClient. It resolves `DeploymentService` and `ArtifactDeploymentService` from a per-message DI scope and forwards `MgmtDeployArtifactsCommand`, `GetDeploymentDiffCommand`, and instance lifecycle requests. It also hosts `DeploymentConfigEndpoints` — the `GET /api/internal/deployments/{id}/config` route a site calls to fetch a staged config. That endpoint is `AllowAnonymous`; the per-deployment token, compared in constant time, is the entire security boundary, and existence/TTL are checked before the token so unknown, superseded and expired ids are indistinguishable `404`s.
- [Security & Auth (#10)](./Security.md) — the Deployment role is required for all deploy and artifact operations; site-scoped permissions are enforced by the Central UI and Management Service before commands reach `DeploymentService`.
## Troubleshooting
@@ -188,6 +198,10 @@ The operation lock is in-memory. On failover the new active node has no lock ent
The site round-trip timed out or was cancelled before a response arrived. The site may or may not have applied the config. On the next deploy attempt the reconciliation query determines the ground truth. If the query also fails (site unreachable), a new `DeployInstanceCommand` is sent; the site rejects it with "already applied" if it ran the previous one.
### A deployment fails with a config-fetch error
The notify reached the site but the HTTP leg did not complete. `CentralFetchBaseUrl` must be resolvable and reachable **from the site** — a value that only works inside the central network fails every deploy. A `404` from the fetch means the staged row was unknown, superseded, or past `PendingDeploymentTtl` (existence is hidden, so those are indistinguishable); a `401` means the row is live but the presented token did not match. A fetch failure applies nothing, and the site replies `Failed` rather than letting central's Ask hang to timeout.
### DeleteOrphaned audit entry
The site destroyed the Instance Actor but the central DB removal failed. The instance record exists in the central DB but has no corresponding site actor. It cannot be deleted through the normal UI path (the site will reject the delete command because the instance does not exist). Reconcile by removing the central record directly via the Management API or database, referencing the `CommandId` in the audit entry.
docs/components/ManagementService.md
+7 -4
View File
@@ -56,17 +56,18 @@ Mutating handlers that call repositories directly invoke `AuditAsync` (backed by
### Actor lifecycle and registration
`AkkaHostedService` (in the Host) creates the `ManagementActor` under the path `/user/management` and registers it with `ClusterClientReceptionist`:
`AkkaHostedService` (in the Host) creates the `ManagementActor` under the path `/user/management` and publishes it to `ManagementActorHolder`, which is the only way anything reaches it:
```csharp
var mgmtActor = _actorSystem!.ActorOf(
Props.Create(() => new ManagementActor(_serviceProvider, mgmtLogger)),
"management");
ClusterClientReceptionist.Get(_actorSystem).RegisterService(mgmtActor);
var mgmtHolder = _serviceProvider.GetRequiredService<ManagementActorHolder>();
mgmtHolder.ActorRef = mgmtActor;
```
A `ClusterClientReceptionist.Get(_actorSystem).RegisterService(mgmtActor)` call sat between those two statements until 2026-07-22. It was deleted because nothing ever sent to it: the CLI it was built for uses HTTP, not ClusterClient.
`ClusterClientReceptionist` advertises the actor to `ClusterClient` senders without requiring them to join the Akka cluster. The `ManagementActorHolder.ActorRef` property is then the bridge from the HTTP endpoint (which runs in ASP.NET Core middleware) into the Akka actor world.
The actor declares an explicit supervisor strategy — one-for-one with Resume and no retry limit — to match the coordinator-actor convention and remain correct if child actors are added later.
@@ -154,9 +155,11 @@ Content-Type: application/json
A successful response is HTTP 200 with the JSON result. An authorization failure is HTTP 403 with `{ "error": "...", "code": "UNAUTHORIZED" }`.
### Sending a command via ClusterClient
### Sending a command in-process
The `ManagementActor` is also reachable from any `ClusterClient` that has a contact point into the central cluster. The actor is registered under `/system/receptionist` with the path `/user/management`. Callers construct and `Tell` a `ManagementEnvelope` and expect one of `ManagementSuccess`, `ManagementError`, or `ManagementUnauthorized` in reply.
`ManagementEnvelope` is also the in-process contract: a caller holding `ManagementActorHolder.ActorRef` asks the actor directly and expects one of `ManagementSuccess`, `ManagementError`, or `ManagementUnauthorized` in reply. `ManagementEndpoints` is that caller.
There is **no** out-of-process actor path. The actor was advertised via `ClusterClientReceptionist` until 2026-07-22, so a `ClusterClient` with a contact point into the central cluster could `Tell` it a `ManagementEnvelope`; no caller ever did, and the registration is gone. The HTTP endpoints above are the only remote management surface.
## Command Groups
docs/deployment/topology-guide.md
+46 -6
View File
@@ -88,17 +88,19 @@ Both central nodes must be configured as seed nodes for each other:
},
"Cluster": {
"SeedNodes": [
"akka.tcp://scadabridge@central-01.example.com:8081",
"akka.tcp://scadabridge@central-02.example.com:8081"
"akka.tcp://scadabridge@central-02.example.com:8081",
"akka.tcp://scadabridge@central-01.example.com:8081"
]
}
}
}
```
> **Seed order is load-bearing — each node lists ITSELF first** (decision 2026-07-22). Note Node B's list is the reverse of Node A's. Akka only lets `seed-nodes[0]` form a *new* cluster, so a node listing its partner first can never boot while that partner is down. `StartupValidator` rejects the boot if the ordering is wrong, comparing host **and** port; use the same spelling of the hostname in `NodeHostname` and in the seed URI, since Akka does no DNS canonicalisation (`central-02` and `central-02.example.com` are different seed identities). See `docs/requirements/Component-ClusterInfrastructure.md` → Seed Node Ordering.
### Cluster Behavior
- **Split-brain resolver**: Keep-oldest with `down-if-alone = on`, 15-second stable-after.
- **Split-brain resolver**: `auto-down` (`AutoDowning` provider, `auto-down-unreachable-after` = 15s) since the 2026-07-21 availability-over-partition-safety decision — the leader among the *reachable* members downs the unreachable peer, so a hard crash of **either** node fails over. Accepted trade: a real partition leaves both sides active until an operator restarts one. `keep-oldest` (with `down-if-alone = on`) remains a supported `SplitBrainResolverStrategy` value, but in a two-node cluster it cannot survive a crash of the oldest node. See `docs/plans/2026-07-21-auto-down-availability-decision.md`.
- **Minimum members**: `min-nr-of-members = 1` — a single node can form a cluster.
- **Failure detection**: 2-second heartbeat interval, 10-second threshold.
- **Total failover time**: ~25 seconds from node failure to singleton migration.
@@ -145,6 +147,8 @@ Each site has its own two-node cluster:
}
```
> **Site Node B reverses this list** — `site-01-b` first, `site-01-a` second — per the self-first seed rule above. It applies to site pairs exactly as it does to the central pair: without it, `site-01-b` cannot boot while `site-01-a` is down.
### Site Cluster Behavior
- Same split-brain resolver as central (keep-oldest).
@@ -174,9 +178,45 @@ database from its peer rather than letting it rejoin.
### Central-Site Communication
- Sites connect to central via Akka.NET remoting.
- The `Communication:CentralSeedNode` setting in the site config points to one of the central nodes.
- If that central node is down, the site's communication actor will retry until it connects to the active central node.
Three transports cross the boundary, not one:
- **Akka ClusterClient** — command/control. Sites list every central node in
`ScadaBridge:Communication:CentralContactPoints`; contact rotation reaches whichever node
answers, so no "active central" needs to be identified. (There is no `Communication:CentralSeedNode`
setting — earlier revisions of this guide named one that never existed in the code.)
- **gRPC** — real-time data and audit pull. Note the direction is inverted from the data flow:
each **site node hosts the gRPC server** on `GrpcPort` (default 8083, h2c) and central dials in.
- **Plain HTTP** — the deploy config itself, fetched by the site with a per-deployment token.
#### gRPC control-plane preshared key (required)
Every site node must set `ScadaBridge:Communication:GrpcPsk`, and central must hold the same
value for that site. **`StartupValidator` refuses to boot a site node without it**, deliberately:
the gate is fail-closed, so an unset key would leave the node joined, healthy-looking and
answering heartbeats while refusing every gRPC call — no live subscriptions, no audit pull, no
cached-telemetry ingest.
| Side | Where the key lives |
|---|---|
| Site node (both nodes of the pair, identical) | `ScadaBridge:Communication:GrpcPsk`, in production `${secret:SB-GRPC-PSK-<siteId>}` |
| Central | secret `SB-GRPC-PSK-<siteId>` in its store — **or** `ScadaBridge:Communication:SitePsks:<siteId>` |
The store is the source that matters in production, because sites are added at runtime and their
keys cannot be enumerated in configuration at boot; `SitePsks` covers a host running without a
master key (the docker rig) and one-off pins.
One key **per site**, never one for the fleet: a compromised site must not yield another site's
key. And never share it with `LocalDb:Replication:ApiKey` — that authenticates the *pair partner*
for database replication, a different trust relationship on the same listener.
**Rotation:** set the new value on both sides, then restart the pair (pairs restart together
anyway — see above). **Upgrading to a build that has this gate requires seeding the key first**,
including in the on-host `deploy/` overlays.
The key is a bearer token over plaintext h2c, so it is readable and replayable by anyone on the
path. That is the accepted posture today — the same trusted-network assumption the boundary
already made, now with authentication rather than none. TLS on these listeners is follow-on
hardening and needs no change to the key design.
## Scaling Guidelines
docs/known-issues/2026-07-20-cached-telemetry-drain-hot-loop.md
+1 -48
View File
@@ -1,55 +1,8 @@
# Cached-telemetry drain hot-loops forever on a row whose tracking snapshot is gone
**Date:** 2026-07-20 · **Status:** RESOLVED (2026-07-20) · **Severity:** High — revised up from Medium on investigation (permanent stall of the cached-telemetry path, not just a log flood)
**Date:** 2026-07-20 · **Status:** OPEN · **Severity:** Medium (log flood + wasted I/O; no data loss)
· **Area:** AuditLog / Site Telemetry
## Resolution (2026-07-20)
Fixed by letting an unresolvable row LEAVE the queue.
`SiteAuditTelemetryActor.OnCachedDrainAsync` now abandons the operational half of a cached row
whose tracking snapshot is still unresolvable after a grace period
(`SiteAuditTelemetryOptions.CachedTrackingGraceSeconds`, default **300 s**) and marks it
`Forwarded`. A row with no `CorrelationId` at all is abandoned immediately — it can never resolve.
**Marking `Forwarded` does not drop the audit data.** That state's role in this machine is "no
longer owed by the drain, still eligible for reconciliation", which is exactly the situation:
`ISiteAuditQueue.ReadPendingSinceAsync` covers `Forwarded` rows as well as `Pending` ones and
central dedups on `EventId`, so the reconciliation pull still delivers them. What is genuinely lost
is the operational (`SiteCalls`) half — unrecoverable regardless, since the tracking row it would
have been built from no longer exists.
Three further behaviours, each deliberate:
- **Inside the grace window the row is still retried.** A missing snapshot is normally a brief
write race (the audit row lands microseconds before the tracking row); abandoning on the first
failed lookup would discard the operational half of every cached call that lost that race.
- **A tracking-store THROW never abandons.** A throw is a store fault (locked, corrupt,
mid-restore), not a verdict about the row. Those rows stay `Pending` however old they are.
- **Logging is per drain pass, not per row.** One summary line each for abandoned, lookup-failed
(with the first exception attached) and deferred rows. The deferred case dropped to Debug — being
inside the grace window is normal operation, not a warning.
### Severity was revised UP during the fix
The original write-up called this a log flood with "no data loss", which understated it. The queue
is read **oldest-first with a fixed `BatchSize` (default 256)**, so once a batch's worth of
permanently-unresolvable rows collects at the head, every drain re-reads exactly those rows, fails
identically, and **never reaches the newer rows behind them**. The cached-telemetry path stalls
permanently. The audit half still reached central by reconciliation, so "no data loss" held — but
"Medium" did not.
**Fixed in:** `SiteAuditTelemetryActor.cs` (`AbandonUnresolvableAsync` + the rewritten skip
branches), `SiteAuditTelemetryOptions.cs` (`CachedTrackingGraceSeconds`).
**Tests:** `SiteAuditTelemetryActorTests` — `CachedDrain_OrphanRow_PastGrace_IsAbandoned_...`,
`..._InsideGrace_IsRetried_NotAbandoned`, `..._FullBatchOfUnresolvableRows_DoesNotStarveTheQueue`,
`..._TrackingStoreThrow_DoesNotAbandonTheRow`. The pre-existing
`CachedDrain_OrphanRow_NoTrackingSnapshot_IsSkipped_DoesNotCrash` was **retargeted, not deleted** —
it had pinned the defect ("skipped and stays Pending"), so its orphan assertion is inverted while
the half that still holds is kept verbatim. Verified non-vacuous: with abandonment disabled the two
abandonment tests go red; the two must-NOT-abandon guards stay green in both directions.
The diagnosis below is retained as the record of how it was found.
## Summary
`SiteAuditTelemetryActor`'s cached-telemetry drain reads Pending audit rows, looks up each row's
docs/known-issues/2026-07-22-integration-call-routing-is-dead-code.md
+59
View File
@@ -0,0 +1,59 @@
# Integration call routing (`IntegrationCallRequest`) is dead on both ends
**Date:** 2026-07-22 · **Status:** OPEN (decision needed: wire or delete) · **Severity:** Low (no
runtime impact — the path cannot be reached) · **Area:** Central–Site Communication
## What
"Pattern 4: Integration Routing" — `CommunicationService.RouteIntegrationCallAsync` →
`SiteEnvelope(IntegrationCallRequest)` → `SiteCommunicationActor` → an integration handler — is
plumbed end to end but connected at neither end.
- **No producer.** `RouteIntegrationCallAsync` (`CommunicationService.cs`, "Pattern 4") has **zero
callers** in `src/` or `tests/`. It is the only one of `CommunicationService`'s command methods
with none.
- **No handler.** `SiteCommunicationActor` forwards to `_integrationHandler` when one is
registered, but `RegisterLocalHandler(LocalHandlerType.Integration, …)` appears **only** in
`SiteCommunicationActorTests.cs`. `AkkaHostedService` registers the other three handler types
(`Artifacts`, `EventLog`, `ParkedMessages`) and never this one.
So if anything ever did call it, the site would answer
`IntegrationCallResponse(Success: false, Error: "Integration handler not available")`
(`SiteCommunicationActor.cs`, Pattern 4) — and the two tests that exercise the path both register
the handler themselves first, which is why the suite has never noticed.
Do not confuse this with the **Inbound API**'s routed-site-script path, which is live, tested, and
uses different messages entirely. This is a separate, unused routing pattern that predates it.
## Why it is recorded rather than fixed
Found during the recon for the ClusterClient→gRPC transport migration
([`docs/plans/2026-07-22-clusterclient-to-grpc-plan.md`](../plans/2026-07-22-clusterclient-to-grpc-plan.md),
T0.2), which had to enumerate every command crossing the site↔central boundary. Of the **29**
command types, this is the one that is excluded: **28 migrate to the gRPC contract.**
Porting it would mean designing a proto contract, a `oneof` slot and round-trip mapper tests for a
verb no caller can invoke and no site can service — and enshrining it on a wire format whose
evolution rules are additive-only, so an unused RPC slot is permanent. Deleting it during a
transport migration would mix an unrelated behavioural change into a change whose whole value is
that behaviour is identical. Hence: excluded from the contract, behaviour untouched, decision
deferred to its own change.
## Decision needed
Either:
1. **Delete** — remove `RouteIntegrationCallAsync`, the `IntegrationCallRequest`/`Response`
messages, the `SiteCommunicationActor` receive block, `LocalHandlerType.Integration`, and the
three tests that cover them. This is the default if no consumer is planned.
2. **Wire** — register a real integration handler on site nodes and give the method a caller. This
only makes sense if there is a requirement it serves; none is recorded in
`docs/requirements/`.
Whichever is chosen, do it **before Phase 4** of the migration, since Phase 4 deletes the Akka
transport underneath this path. If it is still dead at that point, option 1 is forced.
## Filing
To be filed as a Gitea issue on `dohertj2/scadabridge` by the repo owner — this note is the
in-repo record of the finding and of the migration exclusion it justifies.
docs/plans/2026-07-08-deferred-work-register.md
+1 -1
View File
@@ -48,5 +48,5 @@ Two live items previously tracked ONLY in `archreview/plans/00-MASTER-TRACKER.md
| # | Item | Where noted | Rationale for deferral | Revisit trigger |
|---|------|-------------|------------------------|-----------------|
| SBR | **SBR oldest-crash total-outage gap** — 2-node `keep-oldest` downs the partition *without* the oldest, so a hard crash of the ACTIVE (oldest) central node makes the standby self-down (~10s) → total central outage until the crashed node restarts; only a younger-node crash fails over. | `archreview/plans/00-MASTER-TRACKER.md:194` + auto-memory `sbr-keep-oldest-2node-active-crash-gap` | Remedy is a production SBR topology/strategy decision (keep-majority + a 3rd/lighthouse seed node, static-quorum, or an accepted-risk note) — **owner: user decision**, not silently changeable. | Before the next production deployment that adds a central node, or the first real active-node crash. |
| SBR | ~~**SBR oldest-crash total-outage gap**~~ **RESOLVED 2026-07-21 (owner decision — availability over partition-safety).** All clusters switched from `keep-oldest` to the `auto-down` downing strategy (Akka `AutoDowning`, `auto-down-unreachable-after` = 15s): a hard crash of EITHER node — active/oldest included — now fails over to the survivor in ~25s with no operator action. Accepted trade: a real network partition produces dual-active until an operator restarts one side. Decision record + evidence (live keep-oldest `DownReachable … including myself` log, Akka.NET 1.5.62 `KeepOldest.OldestDecision` source, rejected alternatives incl. the static-quorum-1 `DownAll` trap): `docs/plans/2026-07-21-auto-down-availability-decision.md`. | `archreview/plans/00-MASTER-TRACKER.md:194` + auto-memory `sbr-keep-oldest-2node-active-crash-gap` (both now historical) | — | Closed. Residual: seed-node boot-alone constraint (unchanged, documented in `Component-ClusterInfrastructure.md`); dual-active recovery is operator-driven. |
| vd03 | **`deploy/wonder-app-vd03/` overlay edits unapplied** — `appsettings.Central.json` needs `AllowSingleNodeCluster: true` + phantom-seed removal + `NodeName: central-a`; `install.ps1` needs `sc.exe failure` recovery actions. The `deploy/wonder-app-vd03/` artifact directory is intentionally untracked (production config out of source control), so the repo cannot ship the fix. | `archreview/plans/00-MASTER-TRACKER.md:198` (PLAN-01 T16/T20/T23) | Needs on-host access; without `NodeName` that deployment's audit rows stamp NULL `SourceNode` — **partially mitigated once PLAN-R2-08 Task 7 lands: the host now FAILS AT BOOT with a key-naming error instead of silently NULLing, so applying the overlay becomes mandatory at the next upgrade.** Owner: whoever maintains the host (user). | Next wonder-app-vd03 deployment/upgrade — **the Task 7 validator makes this row unskippable then.** |
docs/plans/2026-07-21-auto-down-availability-decision.md
+108
View File
@@ -0,0 +1,108 @@
# Auto-Down Downing Strategy — Availability Over Partition-Safety (Decision, 2026-07-21)
**Status: DECIDED and implemented (owner decision, 2026-07-21).** Resolves the registered
deferred "keep-oldest topology/strategy" question (master tracker 2026-07-08;
`docs/plans/2026-07-08-deferred-work-register.md` → SBR row).
## The decision
All two-node ScadaBridge clusters (central and every site pair) switch their downing
strategy from the SBR **keep-oldest** resolver to Akka's **`AutoDowning`** provider
(`ClusterOptions.SplitBrainResolverStrategy: "auto-down"`, now the default):
- `downing-provider-class = "Akka.Cluster.AutoDowning, Akka.Cluster"`
- `auto-down-unreachable-after` = `ClusterOptions.StableAfter` (15s production)
The leader among the **reachable** members downs the unreachable peer after the
stability window. Consequence: a hard crash of **either** node — the active/oldest
included — fails over to the survivor in ~25s (10s failure detection + 15s window),
with no operator action and no victim restart required.
**The owner's stated rationale, verbatim in effect:** the pairs run one node per VM at
each site with no Kubernetes and no SQL available site-side, and "network partitions are
less of a risk than if this stops working." Availability wins.
## The accepted trade (read this before debugging a dual-active)
In a **real network partition** (both nodes alive, link cut) each side downs the other
and continues as a one-node cluster: **both run active** — two oldest-Up members, two
sets of singletons, `/health/active` = 200 on both. The pre-decision keep-oldest
resolver would instead have sacrificed the younger side. Recovery from dual-active is
operator-driven: after the partition heals, restart ONE side; the restarted node rejoins
its peer as a fresh incarnation and becomes standby. (The two sides do not merge on
their own — the mutual downing quarantines the association.)
## Why the crashed-oldest direction was unsurvivable before (evidence)
Live drill on the docker rig, 2026-07-21, `keep-oldest` + `down-if-alone = on`
(config verified live): killing the active/oldest `central-a` produced, on `central-b`:
```
SBR took decision Akka.Cluster.SBR.DownReachable and is downing
[akka.tcp://scadabridge@scadabridge-central-b:8081] including myself,
[1] unreachable of [2] members
```
The survivor downed ITSELF, exited (`run-coordinated-shutdown-when-down`), and its
restarted incarnation looped on `InitJoin` (non-first-seed cannot self-form) until the
victim returned. Root cause in Akka.NET 1.5.62 `KeepOldest.OldestDecision`
(`src/core/Akka.Cluster/SBR/DowningStrategy.cs`):
```csharp
// oldest is on the OTHER (unreachable) side:
if (DownIfAlone && otherSide == 1 && thisSide >= 2) // survivor side must be >= 2
return DownUnreachable.Instance;
return DownReachable.Instance; // 1-vs-1 → down MYSELF
```
`down-if-alone` is designed for ≥3-node clusters; with 1-vs-1 it deliberately keeps the
oldest side ("the node on the other side is no better" — upstream comment). So two-node
keep-oldest can never survive an oldest crash. This corrected an earlier
mis-explanation in the repo ("the alone-oldest is dead and cannot down itself").
## Alternatives rejected
| Option | Why not |
|---|---|
| keep-oldest (status quo) | Oldest crash = total outage (proven above). Remains a supported `SplitBrainResolverStrategy` value for deployments preferring partition-safety. |
| static-quorum, quorum 1 | Akka's `IsTooManyMembers` guard (`2 > 2*1-1`) returns **DownAll** on any unreachability — total shutdown, strictly worse. |
| static-quorum, quorum 2 | Survivor (1 < 2) downs itself on any crash. |
| keep-majority | 1-vs-1 tie keeps the lowest-address side — moves the fatal crash from "oldest" to "lowest address", same hole. |
| lease-majority | Needs a shared lease store (K8s API, SQL, …) reachable by both nodes — not available at sites. |
| third arbiter node | Would make `down-if-alone` work, but there is no third VM at sites. |
| custom downing provider | Would reimplement exactly what `AutoDowning` already does, tested upstream. If a future Akka.NET release removes `AutoDowning`, port it then. |
## What changed (implementation slice, same session)
- `ClusterOptions`: `SplitBrainResolverStrategy` default → `"auto-down"`; docs rewritten.
`DownIfAlone` kept (keep-oldest-only knob, validated only under keep-oldest).
- `ClusterOptionsValidator`: allows `auto-down` | `keep-oldest`; `DownIfAlone` requirement
scoped to keep-oldest.
- `AkkaHostedService.BuildHocon`: downing block branches on the strategy (AutoDowning
provider + `auto-down-unreachable-after` vs the SBR block).
- All 16 `appsettings` (src Host ×2, `docker/` ×8, `docker-env2/` ×4, and the gitignored
`deploy/wonder-app-vd03/` ×2 on-disk overlay) flipped to `auto-down`.
**Owner action: sync the wonder-app-vd03 overlay to the host and restart both services
together.**
- `docker/failover-drill.sh`: `active` mode now asserts the survivor TAKES OVER while
the victim is down (previously it asserted the outage).
- Tests: HOCON emission (`HoconBuilderTests`), validator/default tests, and two new
real-cluster tests in `SbrFailoverTests` — `AutoDown_HardCrashOfOldestNode_
YoungerSurvivorTakesOverSingleton` (the direction keep-oldest could never pass) and
`AutoDown_HardCrashOfYoungerNode_OldestKeepsSingleton`. `TwoNodeClusterFixture` gained
a `strategy` parameter (default `auto-down`).
- Docs: `Component-ClusterInfrastructure.md` (Downing Strategy section rewritten),
`docker/README.md` (drill docs + results), `CLAUDE.md`, deferred-work register entry
resolved.
## Residual operational notes
- **Seed-node bootstrap constraint still applies to boot-alone**: only the first seed
may self-form a cluster. Auto-down removes the active-crash outage (the survivor never
restarts), but a node that must BOOT alone while its peer is dead (cold start of only
the non-first-seed VM, or the survivor crashing while the peer is still down) still
waits in `InitJoin` for its peer. Operator recovery unchanged (restart first seed, or
self-first seed override).
- Monitoring already surfaces dual-active if it ever happens: both nodes report
`IsActive` in heartbeats / both `/health/active` = 200 — the Health dashboard shows
two Primaries.
docs/plans/2026-07-22-clusterclient-to-grpc-live-gate.md
+198
View File
@@ -0,0 +1,198 @@
# ClusterClient → gRPC migration — live gate results
Rig: `docker/` (2 central + 3×2 site + traefik), rebuilt from the branch under test via
`bash docker/deploy.sh`. Recorded check-by-check in the family's live-gate format. Phase 5's
full eight-check gate is recorded further down as those phases land; this file starts with
Phase 0, whose DoD has its own smaller gate.
---
## Phase 0 — PSK auth + dead-code removal — **PASS** (2026-07-22)
Branch `feat/grpc-phase0-psk` @ `228ff8b4`. Image rebuilt, all 9 containers recreated.
### Baseline (pre-change build, same rig)
An unauthenticated call from the host to a site's audit-pull RPC was **accepted**:
```
$ grpcurl -plaintext -d '{"batch_size":1}' localhost:9023 sitestream.SiteStreamService/PullAuditEvents
{}
```
That is the gap Phase 0 closes, reproduced rather than assumed.
### Checks
| # | Check | Result |
|---|---|---|
| 1 | All 8 nodes boot with keys configured (new `StartupValidator` rule) | **PASS** — all recreated and reached ready |
| 2 | Unauthenticated `PullAuditEvents` ⇒ `PermissionDenied`, all 3 sites | **PASS** |
| 3 | Wrong key (site-b's key presented to site-a) ⇒ `PermissionDenied` | **PASS** — per-site scoping is real, not decorative |
| 4 | Correct key ⇒ success, all 3 sites | **PASS** |
| 5 | Central's own authenticated paths still work | **PASS** — 14 successful `PullAuditEvents` from central to site-a; **0** auth failures in either central's log |
| 6 | LocalDb sync unaffected by the new interceptor | **PASS** — **0** control-plane rejections and **0** sync auth failures on the passive peer; session connected after one boot-order retry |
| 7 | No interceptor-activation errors | **PASS** — 0 (see the defect below) |
Evidence for 2–4:
```
=== NO CREDENTIALS ===
:9023 -> ERROR: Code: PermissionDenied Message: Control plane authentication failed.
:9033 -> ERROR: Code: PermissionDenied Message: Control plane authentication failed.
:9043 -> ERROR: Code: PermissionDenied Message: Control plane authentication failed.
=== WRONG KEY (site-b's key against site-a) ===
:9023 -> ERROR: Code: PermissionDenied Message: Control plane authentication failed.
=== CORRECT KEY ===
site-a :9023 -> {}
site-b :9033 -> {}
site-c :9043 -> {}
```
Site-a rejected exactly **2** calls — the two deliberate probes above — and nothing else.
### Defect the gate caught that the test suite did not
**First run of this gate FAILED**, and is worth recording because the failure mode is
deceptive.
`Grpc.AspNetCore` activates a type-registered interceptor through
`InterceptorRegistration.GetFactory()`, which throws when more than one public constructor is
applicable. `ControlPlaneAuthInterceptor` shipped with two — the DI one and a prefix-set
overload intended for later phases.
The throw happens **inside the pipeline, per call**, so:
- nothing failed at startup; the node booted, joined its pair and reported healthy;
- every gated call died with `Unknown / "Exception was thrown by handler"`, which reads as a
handler bug rather than an auth bug;
- **correct key, wrong key and no key produced identical errors** — the tell. A gate that
cannot distinguish those is not authenticating anything.
Site-a's log at the time: three `PullAuditEvents` calls, three identical
`System.InvalidOperationException: Multiple constructors accepting all given argument types
have been found in type 'ControlPlaneAuthInterceptor'`.
The full suite was green when this shipped — **29 suites, 6,872 tests, 0 failures**. The
in-process end-to-end test missed it because it registered the interceptor with
`AddSingleton` alongside `AddGrpc`, so DI returned the instance and gRPC's activation path
never ran.
Fixed in `228ff8b4`: the prefix-set constructor is `internal`; the end-to-end harness now
registers exactly as `Program.cs` does (by type, not in DI); and a reflection assertion pins
"exactly one public constructor", since that is the actual invariant.
**Lesson for phases 1A/1B, which both add services to this interceptor:** extend
`DefaultGatedPrefixes`; do not add a second public constructor. And any in-process harness for
a DI-activated component must mirror the production registration shape or it proves less than
it appears to.
### Test suite alongside the gate
Non-Playwright: **29 suites, 6,872 tests, 0 failures**.
Playwright (against this rig): **170 passed, 2 failed, 1 skipped** of 173. Both failures were
run down to root cause and **both are pre-existing on `main`, unrelated to Phase 0** — this
branch touches no EF, CentralUI, Transport or ManagementService file (`git diff --stat
main...HEAD -- src/` is 15 files, all Communication/Host/AuditLog gRPC plumbing).
An earlier run of this suite reported 44 failures. That run is **void**: a `docker/deploy.sh`
was recreating the cluster underneath it, so the fast `LoginTests`/`NavigationTests` failures
were "app unreachable", not defects.
**1. `TransportImportTests.ImportSyntheticBundle_AppliesAndShowsAuditDrillIn` — a real
production bug, not a test defect.** Central's log during the failure:
```
[ERR] An exception occurred while iterating over the results of a query ...
System.InvalidOperationException: The configured execution strategy
'SqlServerRetryingExecutionStrategy' does not support user-initiated transactions.
at Microsoft.EntityFrameworkCore.Query.Internal.SplitQueryingEnumerable`1.AsyncEnumerator.MoveNextAsync()
```
`BundleImporter.cs:1298` opens a user-initiated transaction; the central context is configured
with `EnableRetryOnFailure` (`ConfigurationDatabase/ServiceCollectionExtensions.cs:33`). SQL
Server's retrying strategy refuses to run a split query inside a caller's transaction, so
**bundle import fails against real MS SQL**. The fix is the one the exception names: wrap the
transaction in `Database.CreateExecutionStrategy().ExecuteAsync(...)`.
Why the whole unit/integration suite is green on it: those tests use the in-memory EF provider,
which has no retrying execution strategy — and `BeginTransactionAsync` is a no-op there. The
comment directly above line 1298 documents that divergence without drawing the conclusion. Only
a rig-backed test can see this.
**2. `SmsNotificationE2ETests.SmsConfigPage_CreateOrRender_NeverLeaksAuthToken` — a stale test
fixture.** No server-side error at all: the page renders 200, and no `INSERT INTO
SmsConfigurations` is ever issued. The test's fixture SID is `ACtest123` (`d6ead8ae`,
2026-06-19). `SmsConfiguration.razor:231` rejects anything not matching `^AC[0-9a-fA-F]{32}$`,
added by `40088a21` (2026-07-10) to close an un-escaped URI-interpolation hole. `Save()` sets
`_formError` and returns — no toast, exactly as observed. The fixture was never updated.
This has been failing since 2026-07-10, and it matters more than a red line: everything after
the toast assertion — including **the secret-non-leak assertion that the Auth Token value never
reaches the page HTML** — has not executed since. Fix is a valid 32-hex SID in the fixture.
### Not covered by this gate
- Streaming subscriptions were exercised in-process (TestServer), not over the rig. The
interceptor is path-scoped, not method-scoped, so the rig's `PullAuditEvents` evidence covers
the same code path — but a live `SubscribeInstance` under load is untested here.
- Key **rotation** on a live pair.
- `docker-env2` was updated with its own key but not redeployed or gated.
---
## Phase 1A — central control plane (site→central over gRPC) — **PASS** (2026-07-22)
Branch `feat/grpc-central-control` @ `0e162cb2`. Rig rebuilt; **site-a flipped to
`CentralTransport=Grpc`** with `CentralGrpcEndpoints=[central-a:8083, central-b:8083]`,
**site-b/c left on Akka** (default) to prove coexistence. The site-a flag flip was a
DoD-test-only rig edit — reverted from the branch, never committed (the plan keeps the default
`Akka` until Phase 4).
### The defect this gate caught (T1A.2 shipped it; fixed in `0e162cb2`)
**First rebuild: central's entire HTTP surface was gone.** central-a logged only
`Now listening on: http://[::]:8083` — no `:5000`. Central UI, the Management + Inbound API,
and every `/health/*` endpoint (Traefik routing + `IActiveNodeGate` both depend on them) were
dead. The node booted, joined the cluster and served gRPC fine; **no startup error.**
Cause: `builder.WebHost.ConfigureKestrel(o => o.ListenAnyIP(8083, Http2))` puts Kestrel into
explicit-endpoints mode, which **suppresses the URLs from `ASPNETCORE_URLS`/`--urls`** — it is
not additive, contrary to the comment T1A.2 shipped. Central's whole HTTP/1 surface lives on
that URL (`http://+:5000` on the rig; a different port in production). The site branch has the
same `ConfigureKestrel` shape but nothing on `ASPNETCORE_URLS` to lose — it binds every port it
needs explicitly — which is why the pattern looked safe.
Every unit + E2E test uses `TestServer`, which never binds real Kestrel, so the whole suite
(6,872) stayed green. Only a live node exposes a missing listener. Fix: parse the port(s) from
the configured URLs and re-declare them (`Http1AndHttp2`) alongside the gRPC port (`Http2`) in
the one `ConfigureKestrel` call — `Program.ParseHttpBindPorts` + `CentralHttpBindPortsTests`
(14 cases). After the fix: `Now listening on: http://[::]:5000` **and** `:8083`; 9001 ready
`200`/active Healthy, 9002 standby, Traefik LB `200`, CLI over the LB works.
### Checks (post-fix rebuild)
| # | Check | Result |
|---|---|---|
| 1 | site-a rides authenticated gRPC to `CentralControlService` | **PASS** — Heartbeat, `ReportSiteHealth`, `ReconcileSite` all HTTP/2 → 200; **0** auth failures on either central |
| 2 | Heartbeat drives the active flag | **PASS** — 144+ heartbeats over gRPC; site-a `online=True` at central |
| 3 | Health page live | **PASS** — `ReportSiteHealth` lands; central shows site-a `online=True`, sequence advancing, alongside Akka site-b/c |
| 4 | Reconcile works after site restart | **PASS** — restarted `site-a-a`; it logged `Site→central transport: gRPC to 2 central endpoint(s)`, then `Reconcile pass … complete: 0 fetched, 0 failed, 0 orphan(s)` |
| 5 | Coexistence | **PASS** — site-b/c log `Created ClusterClient to central`; both `online=True` — Akka and gRPC sites side by side |
| 6 | Central HTTP surface intact under the new gRPC listener | **PASS** — after the fix (see above) |
### Not independently exercised on this gate
- **Notification e2e (`SubmitNotification`/`QueryNotificationStatus`) and audit ingest
(`IngestAuditEvents`/`IngestCachedTelemetry`)** were NOT driven live: the rig has templates
but **no deployed instance**, so nothing emits site→central notifications or audit rows on its
own. These four RPCs traverse the identical `CentralControlGrpcService` →
`CentralCommunicationActor` Ask path that checks 1–3 proved live under real auth, and their
payload mappers carry 32 round-trip goldens — but the payloads themselves were not put over
the wire here. **Phase 2's central-kill S&F soak is where they get their live workout;** flag
for a fuller 1A proof if a deployed-instance rig is set up before then.
- Cross-node failover/failback of the site→central channel under a central-node kill (unit-proven
via TestServer; not exercised on the rig at 1A).
docs/plans/2026-07-22-clusterclient-to-grpc-plan.md
+355
View File
@@ -0,0 +1,355 @@
# ClusterClient → gRPC-only cross-cluster transport — implementation plan
**Date:** 2026-07-22. **Design:** `~/Desktop/scadaproj/scadabridge_clusterclient_to_grpc.md`
(read it first, §7 especially — it contains the deep-dive corrections this plan builds on).
**Goal:** all site↔central traffic rides gRPC with PSK auth from `ZB.MOM.WW.Secrets`;
`ClusterClient`/`ClusterClientReceptionist` are deleted; Akka remoting never crosses the
site↔central boundary.
Everything below was code-verified 2026-07-22. If a cited line has drifted, re-locate by the
quoted identifier, never by line number.
## How to execute this plan (read before starting)
- **Phase order:** 0 → (1A ∥ 1B) → (2 ∥ 3) → 4 → 5. Phases marked ∥ are parallel-safe **only in
separate git worktrees** (`git worktree add ../ScadaBridge-1B feat/grpc-sitecommand`) — never
run two agents against one working tree (destructive git races are a known family incident).
Designated merge order when tracks meet: 1A lands first, 1B rebases (expected conflicts are
confined to `ZB.MOM.WW.ScadaBridge.Communication.csproj` proto ItemGroups and `Program.cs`
service/Map blocks — resolve as union).
- **Branches:** `feat/grpc-phase0-psk`, `feat/grpc-central-control` (1A),
`feat/grpc-sitecommand` (1B), then per-phase branches; PR each phase to `main` with its DoD met.
- **Build/test:** `dotnet build ZB.MOM.WW.ScadaBridge.slnx` (0 warnings — TreatWarningsAsErrors),
`dotnet test ZB.MOM.WW.ScadaBridge.slnx`. Tests are xunit **v2** (2.9.3) + `Akka.TestKit.Xunit2`
1.5.62 + NSubstitute. TestKit tests inherit `TestKit` — never hand-roll
`ActorSystem.Create`+join.
- **Proto codegen is CHECKED-IN, not build-time** (protoc segfaults in the linux_arm64 image).
For every new/changed proto follow the sitestream recipe documented in
`src/ZB.MOM.WW.ScadaBridge.Communication/ZB.MOM.WW.ScadaBridge.Communication.csproj`:
temporarily uncomment/add the `<Protobuf Include=... GrpcServices="Both" />` item, delete stale
generated files, `dotnet build` on macOS, copy `obj/**/Protos/*.cs` into a committed folder
(mirror `SiteStreamGrpc/`), re-comment the item. `docker/regen-proto.sh` exists.
- **Rig:** `docker/deploy.sh` rebuilds the 2-central + 3×2-site cluster (+ traefik). Central UI
`9001/9002:5000`, site gRPC `90x3/90x4:8083`. **External sites↔central rules:** never
host-`sqlite3` a live WAL DB; `aspnet:10.0` has no `curl`; seeding via `docker/seed-sites.sh`.
- **Coexistence rule:** every migrated path sits behind a config flag with the Akka
implementation as default until Phase 4. Rollback at any point = flip the flag.
## The two choke points (where all transport code changes)
- **Site→central:** `SiteCommunicationActor`
(`src/ZB.MOM.WW.ScadaBridge.Communication/Actors/SiteCommunicationActor.cs`) — 7 message types,
all `ClusterClient.Send` to `/user/central-communication` (Ask, except `HeartbeatMessage` Tell).
- **Central→site:** `CentralCommunicationActor` (same dir) — the `SiteEnvelope` handler
(`:452-467`) unwraps and routes via per-site ClusterClient. ALL producers
(`CommunicationService`'s 27 + `SiteCallAuditActor`'s 2 relays) go through it.
**`CommunicationService` gets NO interface extraction; its ~20 consumers are untouched.**
---
## Phase 0 — PSK auth + dead-code removal (standalone hardening; parallel-safe with 1A/1B prep)
### T0.1 Delete the vestigial `/user/management` receptionist registration
`AkkaHostedService.cs:459` (`RegisterService` of `ManagementActor`) — delete the registration
only (the actor stays; `ManagementEndpoints.cs:117` asks it in-process via
`ManagementActorHolder`). Grep-verify nothing sends to `/user/management`. Update
`docs/requirements/Component-Host.md` REQ-HOST-6a to record the removal.
### T0.2 File the dead-integration issue
`IntegrationCallRequest` (`CommunicationService.cs:238`) is unwired in production —
`RegisterLocalHandler(Integration, …)` exists only in `SiteCommunicationActorTests.cs:102`, so
production always replies "Integration handler not available" (`SiteCommunicationActor.cs:134-136`).
File a Gitea issue (decide: wire or delete); **exclude it from the gRPC contract** (28 of 29
commands migrate). Do not change its behavior in this program.
### T0.3 PSK interceptor + options
New `src/ZB.MOM.WW.ScadaBridge.Host/ControlPlaneAuthInterceptor.cs`, copied from
`LocalDbSyncAuthInterceptor.cs` (same file layout: seal, 4 server handlers → shared
`Authorize`, `authorization: Bearer` metadata extraction, `CryptographicOperations.FixedTimeEquals`
over UTF-8, **fail-closed when the expected key is unset**, reject with
`StatusCode.PermissionDenied`). Differences from the template:
- **Path scope:** gates a *set* of service prefixes (constructor-provided), initially the
sitestream service prefix (read the real package/service name from `sitestream.proto` —
`/…SiteStreamService/`); later phases add the new services. LocalDb sync keeps its own
interceptor + key untouched.
- **Key source, site side (server on `:8083`):** new option
`CommunicationOptions.GrpcPsk` (`ScadaBridge:Communication:GrpcPsk`), production value
`${secret:SB-GRPC-PSK-<siteId>}` resolved by the existing pre-host `SecretReferenceExpander`
(`Program.cs:56-65`) — zero new resolution code.
- **Key source, central side (clients today; server in 1A):** central's site set is dynamic, so
central resolves secret **`SB-GRPC-PSK-{siteId}`** at channel-build time via the runtime
`ISecretResolver` (copy the fail-closed lazy pattern from
`DataConnectionLayer/Adapters/MxGatewayDataConnection.cs:82-98`), cached per site, cache
invalidated on site remove. New helper `SitePskProvider` in Communication (interface) +
Host (implementation over `ISecretResolver`).
Wire: add the interceptor to the site's existing `AddGrpc` (`Program.cs:526-527`, alongside the
LocalDb one). Attach the PSK on central's existing site-dialing clients as call-level
`Metadata` `Authorization: Bearer <psk>` (the LocalDb sync client pattern,
`SyncBackgroundService.cs:82-86`): `SiteStreamGrpcClient` (subscribe calls),
`GrpcPullAuditEventsClient`, `GrpcPullSiteCallsClient` — each already flows through a channel/
invoker creation point where the site id is known.
### T0.4 Rig + tests
- Rig: mirror the LocalDb dev-key pattern — literal `ScadaBridge:Communication:GrpcPsk:
"dev-grpc-psk-site-a"` in BOTH `docker/site-a-node-*/appsettings.Site.json` (and site-b/c with
their own keys, since unlike LocalDb this is not optional), plus central-side dev secrets: seed
`SB-GRPC-PSK-site-{a,b,c}` into both centrals' secret stores (secret CLI seed flow, or dev-KEK
env as the rig's Secrets setup already does).
- Tests: interceptor unit tests (wrong key / missing header / unset expected key ⇒
`PermissionDenied`; non-gated service path passes; constant-time compare exercised);
`SitePskProvider` fail-closed test; one Host wiring test asserting the interceptor is
registered.
**Phase 0 DoD:** suite green; on the rig, an unauthenticated `grpcurl`/test client gets
`PermissionDenied` on SiteStream, authenticated streaming + pull still work end-to-end. PR merged.
---
## Phase 1A — Central control plane (site→central) — worktree A
### T1A.1 `central_control.proto`
New `Protos/central_control.proto` in the Communication project (checked-in codegen per recipe),
package `scadabridge.centralcontrol.v1`, service `CentralControlService`:
| RPC | Wraps DTO (source file under `Commons/Messages/`) | Notes |
|---|---|---|
| `SubmitNotification` | `NotificationSubmit`/`NotificationSubmitAck` (`Notification/NotificationMessages.cs:30,47`) | 11 fields incl. `Guid?` execution ids → string |
| `QueryNotificationStatus` | `NotificationStatusQuery`/`Response` (`:55,62`) | |
| `IngestAuditEvents` | reuse existing `AuditEventBatch`/`IngestAck` from sitestream.proto | import, don't duplicate; `ForwardState`/`IngestedAtUtc` stay off-wire (`AuditEventDtoMapper.cs:23-27`) |
| `IngestCachedTelemetry` | reuse `CachedTelemetryBatch`/`IngestAck` | same |
| `ReconcileSite` | `ReconcileSiteRequest`/`Response` (`Deployment/ReconcileSiteRequest.cs`, `ReconcileSiteResponse.cs` incl. `ReconcileGapItem`) | map<string,string> for name→hash |
| `ReportSiteHealth` | `SiteHealthReport`/`SiteHealthReportAck` (`Health/SiteHealthReport.cs`) | the big one: ~30 fields incl. maps of `ConnectionHealth` enum, `TagResolutionStatus`, `TagQualityCounts`, `NodeStatus` list, `SiteAuditBacklogSnapshot`; model nullable ints/doubles with wrappers; keep `SequenceNumber` |
| `Heartbeat` | `HeartbeatMessage` (`Health/HeartbeatMessage.cs`) → `google.protobuf.Empty` reply | fire-and-forget semantics preserved client-side (don't await failure into caller) |
Mappers in `Communication/Grpc/CentralControlDtoMapper.cs` with **round-trip golden tests**
(construct DTO → proto → DTO, assert deep-equal; include null-optional cases).
### T1A.2 Central hosting
Central branch of `Program.cs` (~`:262` services, `:449-469` Map block):
`builder.Services.AddGrpc(o => o.Interceptors.Add<ControlPlaneAuthInterceptor>())` — central's
interceptor variant verifies `Bearer` against the per-site PSK looked up by the **required
`x-scadabridge-site` metadata header** via `SitePskProvider` (fail-closed: missing header ⇒
`PermissionDenied`). Add an **explicit Kestrel listener** for h2c gRPC: new option
`ScadaBridge:Node:CentralGrpcPort` (default **8083**, symmetric with sites), configured like the
site branch does (`Program.cs:507-512`, `HttpProtocols.Http2`); central's `:5000` stays as-is
(Traefik is HTTP/1 — gRPC does NOT go through traefik). `MapGrpcService<CentralControlGrpcService>()`.
`CentralControlGrpcService` (Host or Communication): decode proto → the SAME message types →
`Ask` the existing `CentralCommunicationActor` handlers (they already handle all 7 — zero
handler logic changes) → encode reply. Reuse the readiness convention: reject `Unavailable`
until the central actor system is up (mirror `SiteStreamGrpcServer.SetReady`).
### T1A.3 Site-side client + transport seam
- New `ICentralTransport` (Communication): one method per the 7 sends. Implementations:
`AkkaCentralTransport` (extracted verbatim from today's `SiteCommunicationActor` send blocks)
and `GrpcCentralTransport` (new).
- `GrpcCentralTransport`: **channel pair with sticky failover + failback** per design §3.5 — new
shared `CentralChannelProvider`: endpoints from new option
`ScadaBridge:Communication:CentralGrpcEndpoints` (List<string>, e.g.
`["http://scadabridge-central-a:8083","http://scadabridge-central-b:8083"]`; validator: required
when transport=Grpc); sticky-until-failure; flip on connect-fail/`Unavailable`; background
failback probe every 30–60 s (gRPC health or a `Heartbeat` ping); reconnect backoff copied from
`SyncBackgroundService.cs:151` (1 s doubling, cap 60 s). Attach PSK (`GrpcPsk` option) +
`x-scadabridge-site` on every call; per-call deadlines from the matching `CommunicationOptions`
timeout (`NotificationForwardTimeout`, `HealthReportTimeout`, etc. — today's Ask timeouts,
unchanged values). **Cross-node auto-retry only on connect-fail/`Unavailable`** — never on
`DeadlineExceeded`.
- `SiteCommunicationActor` selects the implementation from new option
`ScadaBridge:Communication:CentralTransport` (`Akka` | `Grpc`, **default `Akka`**). The 7
handler bodies delegate to the injected transport; reply/fault semantics identical (timeout or
non-OK status ⇒ same `Status.Failure` the S&F/audit layers already treat as transient).
### T1A.4 Tests
Extend `Communication.Tests` (TestKit): `SiteCommunicationActor` with an NSubstitute
`ICentralTransport` — all 7 paths, fault propagation (transport throw ⇒ same failure the S&F
tests expect). `GrpcCentralTransport` unit tests with an in-process `TestServer` gRPC host:
failover flip, sticky behavior, failback probe, PSK attached, deadline set, no-retry-on-deadline.
Reuse/extend `DirectActorSiteStreamAuditClient` for the ingest integration harness.
`NotificationForwarderTests`/`SiteAuditTelemetryActorTests`/`HealthReportSenderTests` must pass
unmodified (they sit above the seam — if they need edits, the seam is wrong).
**1A DoD:** suite green; on the rig with site-a flipped to `CentralTransport=Grpc`
(central gRPC port published, e.g. `9013/9014:8083`): notification e2e, audit rows land, health
page live, heartbeat drives active flag, reconcile works after site restart — while site-b/c
still run Akka (coexistence proven).
---
## Phase 1B — Site command plane (central→site) — worktree B
### T1B.1 `site_command.proto`
Package `scadabridge.sitecommand.v1`, service `SiteCommandService` — **28 commands** (29 minus
dead `IntegrationCallRequest`), grouped into domain RPCs with `oneof` request/response
envelopes (full command list + reply types + `CommunicationService.cs` line refs in the design
doc §7 / recon inventory):
| RPC | Commands (count) | Deadline source |
|---|---|---|
| `ExecuteLifecycle` | RefreshDeployment, Enable/Disable/DeleteInstance, DeploymentStateQuery, DeployArtifacts (6) | `DeploymentTimeout`/`LifecycleTimeout`/`ArtifactDeploymentTimeout` |
| `ExecuteOpcUa` | BrowseNode, SearchAddressSpace, ReadTagValues, VerifyEndpoint, Trust/List/RemoveServerCert, WriteTag (8) | `QueryTimeout` (browse/search per existing Ask usage) |
| `ExecuteQuery` | EventLogQuery, DebugSnapshot, Subscribe/UnsubscribeDebugView (4) | `QueryTimeout`/`DebugViewTimeout` |
| `ExecuteParked` | ParkedMessageQuery/Retry/Discard, RetryParkedOperation, DiscardParkedOperation (5) | `QueryTimeout`; relay callers keep `RelayTimeout`(10s) < `QueryTimeout`(30s) ordering |
| `ExecuteRoute` | RouteToCall/GetAttributes/SetAttributes/WaitForAttribute (4) | `IntegrationTimeout`; WaitForAttribute uses its dynamic timeout |
| `TriggerFailover` | TriggerSiteFailover (1) | `LifecycleTimeout` |
Mappers `SiteCommandDtoMapper.cs` + round-trip golden tests for every command/reply (the bulk of
this track — budget accordingly; enums, `TrackedOperationId` struct → string guid, nullable
wrappers).
### T1B.2 Site server: shared dispatcher
Refactor `SiteCommunicationActor`'s receive table into `SiteCommandDispatcher` (pure routing:
message → `_deploymentManagerProxy` / `_artifactHandler` / `_eventLogHandler` /
`_parkedMessageHandler` / failover handler — preserving EXACTLY today's targets, including the
node-local parked-message handler; see design §7.3, the replicated-store semantics are
deliberate). The actor and a new `SiteCommandGrpcService` (mapped in the site branch next to
`SiteStreamGrpcServer`, gated by `ControlPlaneAuthInterceptor` + the readiness flag) both call
the dispatcher — one routing truth, both transports.
### T1B.3 Central client + transport seam
`ISiteCommandTransport` (send `SiteEnvelope`-equivalent, Ask or Tell) injected into
**`CentralCommunicationActor`**; implementations `AkkaSiteTransport` (today's per-site
ClusterClient path, extracted) and `GrpcSiteTransport`. `GrpcSiteTransport` uses a new shared
**`SitePairChannelProvider`**: addresses from the `Site` entity's existing
`GrpcNodeAAddress`/`GrpcNodeBAddress` (the streaming path's columns — do NOT invent
`LoadSiteAddressesFromDb`, it doesn't exist; reuse the `ISiteRepository` reads +
`CentralCommunicationActor`'s existing DB-driven cache-refresh loop `:532-598` to build/refresh
channels instead of ClusterClients), sticky failover/failback per §3.5, PSK from
`SitePskProvider` + deadlines per the table above. Config flag
`ScadaBridge:Communication:SiteTransport` (`Akka` | `Grpc`, default `Akka`) selected inside
`CentralCommunicationActor` — `CommunicationService` and `SiteCallAuditActor` unchanged.
### T1B.4 Tests
TestKit: `CentralCommunicationActor` with substitute `ISiteCommandTransport` (envelope routing,
Ask-sender reply plumbing, per-site transport lifecycle on site add/remove/change);
`SiteCommandDispatcher` unit tests (every command → correct target, incl. parked→local handler
and failover→local); `SiteCommandGrpcService` via TestServer (auth, readiness, one command per
oneof group); existing `CommunicationServiceTests`/`CentralCommunicationActor*Tests` pass with
the Akka implementation as default.
**1B DoD:** suite green; rig central flipped to `SiteTransport=Grpc` for site-a only: from
CentralUI — deploy refresh, enable/disable instance, browse, read tag, write tag, event-log
query, parked query/retry/discard (run retry against the STANDBY site node explicitly —
replicated-store semantics), `TriggerSiteFailover` — all work; site-b/c untouched on Akka.
---
## Phase 2 (after 1A) ∥ Phase 3 (after 1B) — full cutover on the rig + hardening
- **Phase 2:** flip all sites to `CentralTransport=Grpc`. Soak: S&F drain under central-a kill
(rows stay Pending, resume without loss or duplicates — sequence/dedup layers unchanged),
failback observed when central-a returns, health/heartbeat cadence unchanged in
`CentralHealthAggregator` (no sequence regressions logged).
- **Phase 3:** flip central to `SiteTransport=Grpc` for all sites. Soak: full CentralUI command
matrix against each site; site-node kill mid-command returns a clean error (no hang beyond
deadline); site pair failover mid-stream of commands.
- Both phases: watch for `PermissionDenied` noise (would indicate PSK drift), and confirm zero
ClusterClient log activity on flipped paths.
## Phase 4 — deletion + config cutover (sequential, after 2+3)
1. Flip both flag defaults to `Grpc`; rig + docs updated; one soak cycle.
2. Delete: `AkkaCentralTransport`/`AkkaSiteTransport`, ClusterClient creation
(`AkkaHostedService.cs:942-953`), `DefaultSiteClientFactory` (+ its tests), per-site
ClusterClient cache in `CentralCommunicationActor` (keep the DB refresh loop — it now feeds
`SitePairChannelProvider`), receptionist registrations `:436` and `:935` (T0.1 already removed
`:459`), `CommunicationOptions.CentralContactPoints` (+ validator + rig configs), then the
flags themselves.
3. Grep-gates: `rg -i "clusterclient|receptionist" src tests docker docs` → only historical docs;
`rg "CentralContactPoints"` → empty.
4. Docs: update `grpc_streams.md` (its "ClusterClient keeps command/control" split is
superseded; also fix its `LoadSiteAddressesFromDb` doc-vs-code gap), `Component-Host.md`,
`Component-StoreAndForward.md:137`, and add a `docs/known-issues` cross-ref note that the
frame-size class is retired. Keep `Akka.Cluster.Tools` (ClusterSingleton still used).
## Phase 5 — live gate (rig; sequential; every check PASS required)
1. **PSK negative:** no key / wrong key / missing `x-scadabridge-site` ⇒ `PermissionDenied`;
unset server key ⇒ all rejected (fail-closed); LocalDb sync key unaffected.
2. **Site→central matrix:** notification e2e (delivered + status query), audit + cached-telemetry
rows in `dbo.AuditLog`/site-calls, `/monitoring/health` live per site, heartbeat → active
flag, reconcile self-heal after site-node restart.
3. **Central→site matrix:** all 6 RPC groups exercised from CentralUI, parked retry/discard on
the standby node, failover command drains cleanly.
4. **Failover/failback:** kill central-a → sites flip to central-b sticky (S&F uninterrupted);
restart central-a → failback within probe cadence; same for a site node from central's side;
booting node rejects `Unavailable` until ready and the client fails over.
5. **Mid-drain kill:** kill central during an S&F drain burst — zero loss, zero duplicates.
6. **Frame-class retirement:** issue a command/reply > 128 KB (large browse/event-log result) —
succeeds over gRPC (impossible before).
7. **Boundary check:** with everything on gRPC, verify NO Akka association exists between any
site container and central (`netstat`/Akka logs) — remoting is pair-internal only.
8. **Restart discipline:** full-rig restart (pairs together) comes up clean; no receptionist/
ClusterClient log lines anywhere.
Record results in `docs/plans/2026-07-22-clusterclient-to-grpc-live-gate.md` (check-by-check,
the family's live-gate format).
## Effort & parallelization summary
| Track | Est. | Parallel with |
|---|---|---|
| Phase 0 | 2–3 d | 1A/1B proto authoring |
| 1A | 1.5–2 wk | 1B (separate worktrees; 1A merges first) |
| 1B | 2–3 wk (mapper-heavy) | 1A |
| 2, 3 | 2–4 d each | each other (independent flags/paths) |
| 4 | 2–3 d | — |
| 5 | 2–3 d | — |
Critical path ≈ 1B: **~4–6 weeks total**, matching the design estimate.
## Task checklist (tick as you go; IDs reference the sections above)
**Phase 0 — PSK + dead code** (branch `feat/grpc-phase0-psk`)
- [x] T0.1 Delete `/user/management` receptionist registration (`AkkaHostedService.cs:459`) + Component-Host.md update
- [x] T0.2 File dead-`IntegrationCallRequest` issue; record exclusion (28 of 29 migrate)
- [x] T0.3 `ControlPlaneAuthInterceptor` + `CommunicationOptions.GrpcPsk` + `SitePskProvider`; gate SiteStream; PSK attached on central's streaming + pull clients
- [x] T0.4 Rig dev keys (all 3 sites + central store seeds) + interceptor/provider/wiring tests
- [x] Phase 0 DoD: suite green; rig unauthenticated ⇒ `PermissionDenied`, authenticated paths work; PR merged (#25, ff to `main` @ `3fa95555`; gate PASS in `2026-07-22-clusterclient-to-grpc-live-gate.md`)
**Phase 1A — central control plane** (worktree, `feat/grpc-central-control`)
- [ ] T1A.1 `central_control.proto` (7 RPCs; checked-in codegen) + `CentralControlDtoMapper` + round-trip golden tests
- [ ] T1A.2 Central hosting: `AddGrpc` + per-site-PSK interceptor (`x-scadabridge-site`), `CentralGrpcPort` h2c listener (8083), `CentralControlGrpcService` (Ask existing handlers), readiness gate
- [ ] T1A.3 `ICentralTransport` (Akka extract + Grpc impl), `CentralChannelProvider` (sticky failover/failback, backoff, deadlines, PSK), `CentralTransport` flag default `Akka`, `CentralGrpcEndpoints` option + validator
- [ ] T1A.4 Tests: actor-with-fake-transport ×7, TestServer transport tests, S&F/audit/health suites pass unmodified
- [ ] 1A DoD: rig site-a on `Grpc` proves all 5 site→central paths while site-b/c stay Akka; PR merged (before 1B)
**Phase 1B — site command plane** (worktree, `feat/grpc-sitecommand`)
- [ ] T1B.1 `site_command.proto` (6 oneof RPCs / 28 commands) + `SiteCommandDtoMapper` + round-trip golden tests (all 28 + replies)
- [ ] T1B.2 `SiteCommandDispatcher` refactor (actor + new `SiteCommandGrpcService` share it; parked stays node-local)
- [ ] T1B.3 `ISiteCommandTransport` in `CentralCommunicationActor` (Akka extract + Grpc impl), `SitePairChannelProvider` (Site entity Grpc columns + DB refresh loop), `SiteTransport` flag default `Akka`
- [ ] T1B.4 Tests: dispatcher routing ×28, actor envelope/reply plumbing, TestServer service tests, existing Communication suites green
- [ ] 1B DoD: rig central on `Grpc` for site-a proves full command matrix incl. standby parked retry; rebased on 1A; PR merged
**Phase 2 ∥ 3 — cutover + soak**
- [ ] P2 All sites `CentralTransport=Grpc`; central-kill S&F soak (no loss/dupes), failback observed, health sequences clean
- [ ] P3 Central `SiteTransport=Grpc` all sites; full UI command matrix per site; site-kill mid-command clean; no PSK noise; zero ClusterClient log activity on flipped paths
**Phase 4 — deletion**
- [ ] Defaults flip to `Grpc` + soak; then delete Akka transports, ClusterClient creation, `DefaultSiteClientFactory`, receptionist registrations, `CentralContactPoints`, then the flags
- [ ] Grep-gates pass (`clusterclient|receptionist` → historical docs only; `CentralContactPoints` → empty)
- [ ] Docs updated: `grpc_streams.md`, `Component-Host.md`, `Component-StoreAndForward.md:137`, known-issues cross-ref
**Phase 5 — live gate** (record in `2026-07-22-clusterclient-to-grpc-live-gate.md`)
- [ ] 1 PSK negatives · [ ] 2 site→central matrix · [ ] 3 central→site matrix · [ ] 4 failover/failback both directions · [ ] 5 mid-drain kill · [ ] 6 >128 KB frame-class proof · [ ] 7 no cross-boundary Akka association · [ ] 8 full-rig restart clean
## Gotchas for the executor (will bite; read twice)
- Generated proto C# is committed; never add an active `<Protobuf>` item to the csproj in a
final commit (linux_arm64 protoc segfault breaks the Docker build).
- `HeartbeatMessage` must stay fire-and-forget end-to-end — don't let a gRPC failure surface as
a fault to the heartbeat timer path.
- Deadline ≠ retry: no automatic cross-node retry on `DeadlineExceeded` for WriteTag/Deploy/
Failover; only on provably-unsent failures.
- The parked-message handler is node-local **on purpose** (replicated store); do not "fix" it
onto the singleton proxy.
- Ack-before-Leave on `TriggerSiteFailover` (`SiteCommunicationActor.cs:569`): the gRPC reply
must be written before the node leaves — verify the response completes under failover.
- Inner-before-outer timeouts: `RelayTimeout`(10 s) < `QueryTimeout`(30 s) must survive the
deadline mapping (`CommunicationService.cs:779-786` documents why).
- `SiteStreamGrpcServer.AuditIngestAskTimeout` (30 s) is "one source of truth" shared with
`CentralCommunicationActor` — keep the new central service on the same constant.
- Rig sites reach central by container name (`scadabridge-central-{a,b}:8083`), NOT via traefik
(HTTP/1 only).
- xunit v2: use `Xunit.SkippableFact` for env-gated tests, not `Assert.Skip`.
docs/plans/2026-07-22-clusterclient-to-grpc-plan.md.tasks.json
+241
View File
@@ -0,0 +1,241 @@
{
"plan": "docs/plans/2026-07-22-clusterclient-to-grpc-plan.md",
"design": "~/Desktop/scadaproj/scadabridge_clusterclient_to_grpc.md",
"worktrees": {
"feat/grpc-phase0-psk": "/Users/dohertj2/Desktop/ScadaBridge-phase0"
},
"tasks": [
{
"id": "T0.1",
"phase": "0",
"subject": "Delete the vestigial /user/management receptionist registration",
"status": "completed",
"activeForm": "Deleting the /user/management receptionist registration",
"files": [
"src/ZB.MOM.WW.ScadaBridge.Host/Actors/AkkaHostedService.cs",
"docs/requirements/Component-Host.md",
"docs/requirements/Component-ManagementService.md",
"docs/requirements/Component-Communication.md",
"docs/components/ManagementService.md",
"docs/components/Communication.md",
"src/ZB.MOM.WW.ScadaBridge.CLI/README.md"
],
"notes": "Actor stays; only the ClusterClientReceptionist.RegisterService call goes. Verified: CLI is Akka-free, so 6 docs claiming the CLI reaches ManagementActor over ClusterClient are stale and must be corrected in the same change."
},
{
"id": "T0.2",
"phase": "0",
"subject": "Record the dead IntegrationCallRequest exclusion (28 of 29 commands migrate)",
"status": "completed",
"activeForm": "Recording the dead IntegrationCallRequest exclusion",
"files": [
"docs/known-issues/"
],
"notes": "Plan says 'file a Gitea issue' \u2014 outward-facing, handed to the user. In-repo half is a known-issues note + the exclusion record."
},
{
"id": "T0.3",
"phase": "0",
"subject": "ControlPlaneAuthInterceptor + CommunicationOptions.GrpcPsk + SitePskProvider; gate SiteStream; attach PSK on central's clients",
"status": "completed",
"activeForm": "Building the control-plane PSK auth",
"files": [
"src/ZB.MOM.WW.ScadaBridge.Host/ControlPlaneAuthInterceptor.cs",
"src/ZB.MOM.WW.ScadaBridge.Communication/CommunicationOptions.cs",
"src/ZB.MOM.WW.ScadaBridge.Communication/Grpc/ISitePskProvider.cs",
"src/ZB.MOM.WW.ScadaBridge.Host/SitePskProvider.cs",
"src/ZB.MOM.WW.ScadaBridge.Communication/Grpc/SiteStreamGrpcClient.cs",
"src/ZB.MOM.WW.ScadaBridge.Communication/Grpc/SiteStreamGrpcClientFactory.cs",
"src/ZB.MOM.WW.ScadaBridge.AuditLog/Central/GrpcPullAuditEventsClient.cs",
"src/ZB.MOM.WW.ScadaBridge.AuditLog/Central/GrpcPullSiteCallsClient.cs",
"src/ZB.MOM.WW.ScadaBridge.Host/Program.cs"
],
"notes": "Service prefix verified from sitestream.proto: /sitestream.SiteStreamService/. Grpc 2.76 -> CallCredentials.FromInterceptor + UnsafeUseInsecureChannelCallCredentials is the async-safe attach path on h2c."
},
{
"id": "T0.4",
"phase": "0",
"subject": "Rig dev keys (3 sites + central secret seeds) + interceptor/provider/wiring tests",
"status": "completed",
"activeForm": "Seeding rig dev keys and writing the auth tests",
"files": [
"docker/site-a-node-a/appsettings.Site.json",
"docker/site-a-node-b/appsettings.Site.json",
"docker/site-b-node-a/appsettings.Site.json",
"docker/site-b-node-b/appsettings.Site.json",
"docker/site-c-node-a/appsettings.Site.json",
"docker/site-c-node-b/appsettings.Site.json",
"tests/ZB.MOM.WW.ScadaBridge.Host.Tests/ControlPlaneAuthInterceptorTests.cs"
],
"notes": "Fail-closed from day one \u2014 every environment (incl. docker-env2 and the gitignored deploy/wonder-app-vd03 overlay) needs its key before upgrade. Ops item for the user."
},
{
"id": "P0.DoD",
"phase": "0",
"subject": "Phase 0 DoD: suite green; rig unauthenticated => PermissionDenied, authenticated paths work; PR merged",
"status": "completed",
"activeForm": "Verifying the Phase 0 DoD",
"blockedBy": [
"T0.1",
"T0.2",
"T0.3",
"T0.4"
],
"notes": "Live gate PASS 2026-07-22 (all 7 checks, docs/plans/2026-07-22-clusterclient-to-grpc-live-gate.md). Suite: 29 non-Playwright suites / 6872 tests / 0 failures. Playwright 170 pass / 2 fail / 1 skip of 173 - BOTH failures root-caused and PRE-EXISTING on main, unrelated to Phase 0 (branch touches no EF/CentralUI/Transport/ManagementService file): (1) TransportImportTests - REAL production bug, BundleImporter.cs:1298 user-initiated transaction + EnableRetryOnFailure => import broken on real MS SQL, hidden by the in-memory EF provider; (2) SmsNotificationE2ETests - stale fixture SID 'ACtest123' vs the ^AC[0-9a-fA-F]{32}$ guard added 2026-07-10 (40088a21); failing since then, which also silences its secret-non-leak assertion. The earlier 44-failure run is VOID (concurrent rig rebuild). REMAINING: PR + merge, HELD for the user per the plan's stop-at-DoD rule."
},
{
"id": "T1A.1",
"phase": "1A",
"subject": "central_control.proto (7 RPCs, checked-in codegen) + CentralControlDtoMapper + round-trip golden tests",
"status": "in_progress",
"activeForm": "Authoring central_control.proto and its mappers",
"blockedBy": [
"P0.DoD"
]
},
{
"id": "T1A.2",
"phase": "1A",
"subject": "Central hosting: AddGrpc + per-site-PSK interceptor, CentralGrpcPort h2c listener, CentralControlGrpcService, readiness gate",
"status": "pending",
"activeForm": "Hosting CentralControlService on central",
"blockedBy": [
"T1A.1"
]
},
{
"id": "T1A.3",
"phase": "1A",
"subject": "ICentralTransport (Akka extract + Grpc impl), CentralChannelProvider, CentralTransport flag, CentralGrpcEndpoints option",
"status": "pending",
"activeForm": "Building the site->central transport seam",
"blockedBy": [
"T1A.1"
]
},
{
"id": "T1A.4",
"phase": "1A",
"subject": "Tests: actor-with-fake-transport x7, TestServer transport tests, S&F/audit/health suites pass unmodified",
"status": "pending",
"activeForm": "Testing the central control plane",
"blockedBy": [
"T1A.2",
"T1A.3"
]
},
{
"id": "P1A.DoD",
"phase": "1A",
"subject": "1A DoD: rig site-a on Grpc proves all 5 site->central paths while site-b/c stay Akka; PR merged before 1B",
"status": "pending",
"activeForm": "Verifying the 1A DoD",
"blockedBy": [
"T1A.4"
]
},
{
"id": "T1B.1",
"phase": "1B",
"subject": "site_command.proto (6 oneof RPCs / 28 commands) + SiteCommandDtoMapper + round-trip golden tests",
"status": "in_progress",
"activeForm": "Authoring site_command.proto and its mappers",
"blockedBy": [
"P0.DoD"
]
},
{
"id": "T1B.2",
"phase": "1B",
"subject": "SiteCommandDispatcher refactor (actor + SiteCommandGrpcService share it; parked stays node-local)",
"status": "pending",
"activeForm": "Extracting the site command dispatcher",
"blockedBy": [
"T1B.1"
]
},
{
"id": "T1B.3",
"phase": "1B",
"subject": "ISiteCommandTransport in CentralCommunicationActor (Akka extract + Grpc impl), SitePairChannelProvider, SiteTransport flag",
"status": "pending",
"activeForm": "Building the central->site transport seam",
"blockedBy": [
"T1B.1"
]
},
{
"id": "T1B.4",
"phase": "1B",
"subject": "Tests: dispatcher routing x28, actor envelope/reply plumbing, TestServer service tests, existing suites green",
"status": "pending",
"activeForm": "Testing the site command plane",
"blockedBy": [
"T1B.2",
"T1B.3"
]
},
{
"id": "P1B.DoD",
"phase": "1B",
"subject": "1B DoD: rig central on Grpc for site-a proves full command matrix incl. standby parked retry; rebased on 1A; PR merged",
"status": "pending",
"activeForm": "Verifying the 1B DoD",
"blockedBy": [
"T1B.4",
"P1A.DoD"
]
},
{
"id": "P2",
"phase": "2",
"subject": "All sites CentralTransport=Grpc; central-kill S&F soak, failback observed, health sequences clean",
"status": "pending",
"activeForm": "Running the site->central cutover soak",
"blockedBy": [
"P1A.DoD"
]
},
{
"id": "P3",
"phase": "3",
"subject": "Central SiteTransport=Grpc all sites; full UI command matrix; site-kill mid-command clean; zero ClusterClient activity",
"status": "pending",
"activeForm": "Running the central->site cutover soak",
"blockedBy": [
"P1B.DoD"
]
},
{
"id": "P4.1",
"phase": "4",
"subject": "Flip both flag defaults to Grpc + soak; delete Akka transports, ClusterClient creation, DefaultSiteClientFactory, receptionist registrations, CentralContactPoints, then the flags",
"status": "pending",
"activeForm": "Deleting the ClusterClient transport",
"blockedBy": [
"P2",
"P3"
]
},
{
"id": "P4.2",
"phase": "4",
"subject": "Grep-gates pass + docs updated (grpc_streams.md, Component-Host.md, Component-StoreAndForward.md, known-issues cross-ref)",
"status": "pending",
"activeForm": "Running the deletion grep-gates and doc sweep",
"blockedBy": [
"P4.1"
]
},
{
"id": "P5",
"phase": "5",
"subject": "Live gate, 8 checks, recorded in docs/plans/2026-07-22-clusterclient-to-grpc-live-gate.md",
"status": "pending",
"activeForm": "Running the live gate",
"blockedBy": [
"P4.2"
]
}
]
}
docs/plans/2026-07-22-selfform-fallback-and-manual-failover.md
+661
View File
@@ -0,0 +1,661 @@
# ScadaBridge: InitJoin Self-Form Fallback + Manual Failover Control — Implementation Plan
> **For Claude:** REQUIRED SUB-SKILL: Use superpowers-extended-cc:executing-plans to implement this plan task-by-task.
>
> Shared cross-repo design: `~/Desktop/scadaproj/docs/plans/2026-07-22-initjoin-selfform-fallback.md` (design rationale, MNTR assessment, behavior spec). The OtOpcUa half lives in `~/Desktop/OtOpcUa/docs/plans/2026-07-22-selfform-fallback-and-manual-failover.md`. This plan is self-contained for execution.
**Goal:** (1) Either node of a 2-node ScadaBridge cluster can cold-start alone and become operational, unattended. (2) An admin-only "Trigger failover" control on the Health page performs a graceful, audited role swap of the central pair.
> ## ⚠️ ARCHITECTURE REVISED DURING EXECUTION (2026-07-22)
>
> **Part 1 shipped as self-first seed ordering, NOT the `SelfFormAfter` watchdog described below.** Tasks 1–7 as originally written are superseded; what was actually built is in "Part 1 as executed". Tasks 8–10 (manual failover) are unaffected and still apply as written.
>
> **Why.** A code review of Task 2 raised, and a written test then confirmed, that the watchdog's success signal ("am I `Up` within the window?") cannot distinguish *no seed answered InitJoin* from *a seed answered and the join is in flight* — it sits outside Akka's join handshake. On a routine standby restart the peer is alive but the join stalls behind removal of the restarting node's own stale incarnation; a `Join(self)` issued during `TryingToJoin` abandons the in-flight join and forms a second cluster at the same address. **Measured: a permanent split, still unhealed after 90 s** — converting a routine restart into an outage of the previously-healthy node. That is strictly worse than the gap being closed, and it is not the boot-partition trade the design accepted.
>
> Akka's own `FirstSeedNodeProcess` already implements exactly the intended semantics — InitJoin the other seeds, self-join only if nobody answers — and, being part of the handshake, has no such race. It runs only when `seed-nodes[0]` is the node's own address. So the fix is seed **ordering**, not new runtime code.
>
> The plan's stated safety property — "a booting node only self-forms when NO seed answers InitJoin" — is true of Akka's native first-seed rule and **false** of the watchdog. That claim also appears in the shared cross-repo design doc (`scadaproj/docs/plans/2026-07-22-initjoin-selfform-fallback.md`) and in the OtOpcUa half; **both still need correcting** (owner deferred, 2026-07-22).
## Part 1 as executed — self-first seed ordering
**Architecture:** Every node lists ITSELF as `seed-nodes[0]` and its partner second. No new runtime code, no timer, no new option. `StartupValidator` enforces the ordering at boot (host **and** port comparison) because a broken ordering fails silently. Manual failover (Part 2) is unchanged: graceful `Cluster.Leave(oldest Up member)` via a new `IManualFailoverService` (CentralUI seam, Host implementation) — singleton drain, watchdog process-exit, supervisor restart, rejoin as youngest.
**Behavior** (all rows covered by `SelfFirstSeedBootstrapTests` — real in-process clusters from production `BuildHocon` at production failure-detection timings):
| Scenario | Behavior |
|---|---|
| Lone cold-start, peer dead | Self-joins after `seed-node-timeout` (~5 s) — operational, unattended |
| Restart into a **live** peer | Peer answers `InitJoinAck`; node rejoins, never islands |
| Both cold-start simultaneously, mutually reachable | `InitJoin` handshake converges them → **one** 2-member cluster |
| Both cold-start during a genuine boot **partition** | Each forms its own cluster — same dual-active class `auto-down` already accepts |
| Peer-first ordering (the old config) | Never forms — retained as a falsifiability control in the test suite |
**Shipped:**
- 6 node appsettings reordered (the `*-node-b` configs; `-a` nodes were already self-first). All 14 satisfy the invariant.
- `StartupValidator` self-first rule + 3 tests; `SelfFirstSeedBootstrapTests` (4 tests).
- Docs corrected: `docker/README.md`, `docs/requirements/Component-ClusterInfrastructure.md` (new **Seed Node Ordering** section), `docs/components/ClusterInfrastructure.md` (3 passages), `docs/deployment/topology-guide.md` (incl. the stale keep-oldest claim Task 6 flagged), `CLAUDE.md`.
- Several docs had asserted self-first ordering was *unsafe* because a simultaneous cold start would produce two clusters that never merge. Disproved by test (row 3) and corrected.
**⚠️ Ops action:** the gitignored `deploy/wonder-app-vd03/` overlay must have its `SeedNodes` reordered self-first before its next deploy, or the node will now **refuse to boot**. The validator rule is a hard gate deliberately — the alternative is the silent wedge it replaces.
---
<details>
<summary>Original Part 1 architecture (SUPERSEDED — kept for the decision record)</summary>
**Architecture:** New `ClusterOptions.SelfFormAfter` (`TimeSpan?`, default 10 s, `null`/`≤0` disables, appsettings-bound) arms `ClusterBootstrapFallback` right after ActorSystem creation: wait for membership via `RegisterOnMemberUp`; on expiry, `Cluster.Join(SelfAddress)`. Safety gate: fires only when this node's own address is in its own seed list.
</details>
**Tech Stack:** .NET 10, Akka.NET 1.5.62, Blazor Server (CentralUI), bUnit, xunit. No new packages.
**Branch:** `feat/selfform-fallback` off `main`.
---
## Design essentials (from the shared design doc)
**The defect:** Akka only lets the FIRST listed seed self-join to form a *new* cluster; every other node loops on `InitJoin` forever. Both docker central nodes list `central-a` first, so a lone cold-starting `central-b` never comes Up (the "registered outage gap" — `docker/README.md:289`). `ClusterOptions.SeedNodes`' doc comment claims "either can start first", which the deployed configs do not deliver — this plan makes it true and fixes the comment.
**Behavior spec:**
| Scenario | Behavior with fallback |
|---|---|
| Peer alive (any boot order) | Normal seed join in ms — fallback never fires |
| Lone cold-start, self IS in own seed list | After `SelfFormAfter`: warn log + `Cluster.Join(SelfAddress)` → Up alone, singletons start (`min-nr-of-members=1`) |
| Lone cold-start, self NOT in own seed list | Fallback inert (info log) — self-forming would island the node from the real seeds |
| Peer boots after survivor self-formed | Peer's InitJoin is answered → joins as youngest/standby. No island. |
| Both cold-start simultaneously, mutually unreachable | Both self-form → dual-active (same partition class auto-down accepts; restart one side) |
| `SelfFormAfter` null/`≤0` | Disabled — today's wait-forever behavior |
| Window expires mid-join-handshake | Benign: Akka ignores `Join` once joined |
**Manual failover rules:** graceful `Leave`, never `Down`; admin-only (`AuthorizationPolicies.RequireAdmin`); peer guard (disabled when <2 Up `Central` members); confirmation dialog warning the Blazor circuit will drop (Traefik routes the UI to the active node — triggering failover disconnects your own page, which reconnects against the new active); audited via the app's **central** audit writer (`ICentralAuditWriter` — NOT the shared seam, see the dual-seam gotcha) before the Leave is issued. No interplay with `SelfFormAfter` (the peer is alive on this path, so the restarted node rejoins normally).
**Multi-node TestKit:** assessed and NOT used — in-process real clusters via `TwoNodeClusterFixture` (production `BuildHocon`) cover every deterministic scenario; MNTR would need a dedicated no-parallelization test project for no added coverage. See the shared design doc for the full verdict.
---
### Task 1 (A1): `SelfFormAfter` option + validator
**Classification:** small
**Estimated implement time:** ~4 min
**Parallelizable with:** none (first task)
**Files:**
- Modify: `src/ZB.MOM.WW.ScadaBridge.ClusterInfrastructure/ClusterOptions.cs`
- Modify: `src/ZB.MOM.WW.ScadaBridge.ClusterInfrastructure/ClusterOptionsValidator.cs`
- Test: `tests/ZB.MOM.WW.ScadaBridge.ClusterInfrastructure.Tests/ClusterOptionsTests.cs`
- Test: `tests/ZB.MOM.WW.ScadaBridge.ClusterInfrastructure.Tests/ClusterOptionsValidatorTests.cs`
**Step 1: Create the branch**
```bash
cd ~/Desktop/ScadaBridge && git checkout main && git checkout -b feat/selfform-fallback
```
**Step 2: Write the failing tests** (append to the existing test classes, matching their assertion style — read them first)
```csharp
// ClusterOptionsTests.cs
[Fact]
public void SelfFormAfter_defaults_to_ten_seconds()
{
new ClusterOptions().SelfFormAfter.ShouldBe(TimeSpan.FromSeconds(10));
}
// ClusterOptionsValidatorTests.cs
[Fact] public void SelfFormAfter_null_passes_validation() { /* valid options + null → Succeeded */ }
[Fact] public void SelfFormAfter_zero_passes_validation() { /* zero = explicit disable → Succeeded */ }
[Fact] public void SelfFormAfter_negative_fails_validation() { /* -1s → Failed, message mentions SelfFormAfter */ }
```
**Step 3: Run to verify failure**
```bash
dotnet test tests/ZB.MOM.WW.ScadaBridge.ClusterInfrastructure.Tests --filter "FullyQualifiedName~SelfFormAfter"
```
Expected: FAIL (compile error — property doesn't exist).
**Step 4: Implement.** `ClusterOptions.cs` — add after `AllowSingleNodeCluster` (line 104):
```csharp
/// <summary>
/// Bootstrap self-form fallback window (decision 2026-07-22, scadaproj/akka_failover.md §6.1).
/// Akka only lets the FIRST listed seed form a new cluster; a non-first seed cold-starting
/// while its peer is down loops on InitJoin forever. When this node has waited longer than
/// this window without becoming a cluster member, it forms a cluster on itself
/// (<c>Cluster.Join(SelfAddress)</c>) — but ONLY if its own address is in its seed list
/// (a non-seed node self-forming would create a permanent island). Default 10s: the pair
/// shares a datacenter, so a live peer answers InitJoin in milliseconds and waiting longer
/// buys nothing. <c>null</c> or a non-positive value disables the fallback (wait-forever).
/// Accepted trade: both nodes cold-starting within the window while mutually unreachable
/// form two clusters — the same partition class the auto-down strategy already accepts.
/// </summary>
public TimeSpan? SelfFormAfter { get; set; } = TimeSpan.FromSeconds(10);
```
`ClusterOptionsValidator.cs` — inside `Validate`, after the `FailureDetectionThreshold` rules:
```csharp
builder.RequireThat(options.SelfFormAfter is null || options.SelfFormAfter.Value >= TimeSpan.Zero,
"ClusterOptions.SelfFormAfter must be null (disabled), zero (disabled) or a positive duration; "
+ "a negative value is always a configuration mistake.");
```
**Step 5: Run tests → PASS**
```bash
dotnet test tests/ZB.MOM.WW.ScadaBridge.ClusterInfrastructure.Tests
```
**Step 6: Commit**
```bash
git add -A && git commit -m "feat(cluster): SelfFormAfter option — bootstrap self-form fallback window"
```
---
### Task 2 (A2): `ClusterBootstrapFallback` + first integration test
**Classification:** high-risk (cluster formation behavior)
**Estimated implement time:** ~5 min
**Parallelizable with:** none
**Files:**
- Create: `src/ZB.MOM.WW.ScadaBridge.Host/Actors/ClusterBootstrapFallback.cs`
- Create: `tests/ZB.MOM.WW.ScadaBridge.IntegrationTests/Cluster/SelfFormBootstrapTests.cs`
**Step 1: Write the failing test**
```csharp
using Akka.Actor;
using Akka.Cluster;
using Akka.Configuration;
using Microsoft.Extensions.Logging.Abstractions;
using ZB.MOM.WW.ScadaBridge.ClusterInfrastructure;
using ZB.MOM.WW.ScadaBridge.Host;
using ZB.MOM.WW.ScadaBridge.Host.Actors;
namespace ZB.MOM.WW.ScadaBridge.IntegrationTests.Cluster;
/// <summary>
/// Guards the InitJoin self-form fallback (decision 2026-07-22): Akka only lets the FIRST
/// listed seed form a new cluster, so without the fallback a non-first seed cold-starting
/// alone waits on InitJoin forever — the "registered outage gap". These tests build REAL
/// single/dual-node clusters from the production BuildHocon output, exactly like
/// TwoNodeClusterFixture, and arm the production fallback.
/// </summary>
public sealed class SelfFormBootstrapTests : IAsyncLifetime
{
private readonly List<ActorSystem> _systems = new();
/// <summary>Starts a node whose seed list puts the PEER first (self second, or absent),
/// so Akka's own first-seed rule can never self-form it — only the fallback can.</summary>
private ActorSystem StartNode(int selfPort, int peerPort, TimeSpan? selfFormAfter, bool selfInSeeds = true)
{
var nodeOptions = new NodeOptions { Role = "Central", NodeHostname = "127.0.0.1", RemotingPort = selfPort };
var clusterOptions = new ClusterOptions
{
SeedNodes = selfInSeeds
? new List<string>
{
$"akka.tcp://scadabridge@127.0.0.1:{peerPort}",
$"akka.tcp://scadabridge@127.0.0.1:{selfPort}",
}
: new List<string> { $"akka.tcp://scadabridge@127.0.0.1:{peerPort}" },
SelfFormAfter = selfFormAfter,
StableAfter = TimeSpan.FromSeconds(3),
HeartbeatInterval = TimeSpan.FromMilliseconds(500),
FailureDetectionThreshold = TimeSpan.FromSeconds(2),
MinNrOfMembers = 1,
AllowSingleNodeCluster = !selfInSeeds,
};
var hocon = AkkaHostedService.BuildHocon(
nodeOptions, clusterOptions, new[] { "Central" },
TimeSpan.FromSeconds(1), TimeSpan.FromSeconds(3));
var system = ActorSystem.Create("scadabridge", ConfigurationFactory.ParseString(hocon));
_systems.Add(system);
ClusterBootstrapFallback.Arm(system, clusterOptions, NullLogger.Instance);
return system;
}
[Fact]
public async Task Lone_non_first_seed_self_forms_after_the_window()
{
var selfPort = TwoNodeClusterFixture.GetFreeTcpPort();
var deadPeerPort = TwoNodeClusterFixture.GetFreeTcpPort(); // nothing listening
var node = StartNode(selfPort, deadPeerPort, selfFormAfter: TimeSpan.FromSeconds(2));
// Without the fallback this waits forever (Akka first-seed rule). With it, the node
// must be a 1-member Up cluster shortly after the 2s window.
await TwoNodeClusterFixture.WaitForMembersUp(node, 1, TimeSpan.FromSeconds(20));
Cluster.Get(node).SelfMember.Status.ShouldBe(MemberStatus.Up);
}
public Task InitializeAsync() => Task.CompletedTask;
public async Task DisposeAsync()
{
foreach (var s in _systems)
{
try { await s.Terminate().WaitAsync(TimeSpan.FromSeconds(10)); } catch { /* teardown */ }
}
}
}
```
(Match the assertion library to `SbrFailoverTests.cs` — Shouldly vs xunit `Assert`.)
**Step 2: Run to verify failure** — compile error (`ClusterBootstrapFallback` doesn't exist):
```bash
dotnet test tests/ZB.MOM.WW.ScadaBridge.IntegrationTests --filter "FullyQualifiedName~SelfFormBootstrapTests"
```
**Step 3: Implement `ClusterBootstrapFallback.cs`**
```csharp
using Akka.Actor;
using ZB.MOM.WW.ScadaBridge.ClusterInfrastructure;
namespace ZB.MOM.WW.ScadaBridge.Host.Actors;
/// <summary>
/// InitJoin self-form fallback (decision 2026-07-22, scadaproj/akka_failover.md §6.1).
/// Akka only lets the FIRST listed seed form a NEW cluster; every other node retries InitJoin
/// forever. So "both nodes are seed nodes" (ClusterOptions.SeedNodes) does NOT mean either can
/// cold-start alone — a non-first seed booting while its peer is down waits indefinitely (the
/// "registered outage gap", docker/README.md). This watchdog waits <see cref="ClusterOptions.SelfFormAfter"/>
/// for membership; on expiry it forms a cluster on itself.
///
/// <para><b>Island safety.</b> Fires ONLY when this node's own address is in its own seed list.
/// A node that is not a seed (never legitimately first) must keep waiting: if it self-formed,
/// a later-booting real seed would form a second cluster and the two can never merge. For nodes
/// that ARE seeds, sequential recovery is island-free — Akka's join protocol prefers an existing
/// cluster (a booting node only self-forms when NO seed answers InitJoin), so a peer booting
/// after this node self-formed simply joins it.</para>
///
/// <para><b>Races are benign.</b> If the join completes between window expiry and
/// <c>Cluster.Join(SelfAddress)</c>, Akka ignores the join — a node joins a cluster at most once
/// per incarnation. The residual risk is both nodes cold-starting inside the window while
/// mutually unreachable (a boot-time partition): both self-form, the same dual-active class the
/// auto-down downing strategy already accepts, with the same recovery (restart one side).</para>
/// </summary>
public static class ClusterBootstrapFallback
{
public static void Arm(ActorSystem system, ClusterOptions options, ILogger logger)
{
if (options.SelfFormAfter is not { } window || window <= TimeSpan.Zero)
{
logger.LogInformation(
"Cluster self-form fallback disabled (SelfFormAfter not set) — a node cold-starting "
+ "while its peer is down will wait on InitJoin indefinitely.");
return;
}
var cluster = Akka.Cluster.Cluster.Get(system);
var self = cluster.SelfAddress;
var isSeed = options.SeedNodes.Any(s => TryParseAddress(s, out var a) && a.Equals(self));
if (!isSeed)
{
logger.LogInformation(
"Cluster self-form fallback inactive: this node ({Self}) is not in its own seed list "
+ "[{Seeds}] — self-forming here would island it from the real seeds.",
self, string.Join(", ", options.SeedNodes));
return;
}
var joined = new TaskCompletionSource(TaskCreationOptions.RunContinuationsAsynchronously);
cluster.RegisterOnMemberUp(() => joined.TrySetResult());
_ = Task.Run(async () =>
{
var winner = await Task.WhenAny(joined.Task, Task.Delay(window));
if (winner == joined.Task || system.WhenTerminated.IsCompleted)
return;
logger.LogWarning(
"No cluster membership after {Window} — no seed answered InitJoin (peer down at boot). "
+ "Self-forming a cluster at {Self} so this node becomes operational; if the peer was "
+ "merely partitioned (not dead), the pair is now dual-active — restart one side after "
+ "the partition heals (accepted availability-first trade, decision 2026-07-22).",
window, self);
cluster.Join(self);
});
}
private static bool TryParseAddress(string seed, out Address address)
{
try { address = Address.Parse(seed); return true; }
catch { address = default!; return false; }
}
}
```
(Add the `Microsoft.Extensions.Logging` using. If non-generic `TaskCompletionSource` is unavailable, use `TaskCompletionSource<bool>` + `TrySetResult(true)`.)
**Step 4: Run → PASS** (~5–10 s). **Step 5: Commit**
```bash
git add -A && git commit -m "feat(cluster): InitJoin self-form fallback — lone non-first seed becomes Up"
```
---
### Task 3 (A3): Remaining fallback tests — disabled / late-peer merge / non-seed
**Classification:** standard
**Estimated implement time:** ~5 min
**Parallelizable with:** Task 4
**Files:**
- Modify: `tests/ZB.MOM.WW.ScadaBridge.IntegrationTests/Cluster/SelfFormBootstrapTests.cs`
**Step 1: Append three tests** (each absence assertion carries an in-test positive control — repo convention):
```csharp
[Fact]
public async Task Disabled_fallback_keeps_waiting_and_the_node_was_otherwise_formable()
{
var selfPort = TwoNodeClusterFixture.GetFreeTcpPort();
var deadPeerPort = TwoNodeClusterFixture.GetFreeTcpPort();
var node = StartNode(selfPort, deadPeerPort, selfFormAfter: null);
await Task.Delay(TimeSpan.FromSeconds(6)); // 3x the window used in the enabled test
var cluster = Cluster.Get(node);
cluster.State.Members.ShouldBeEmpty(); // still InitJoin-looping — today's behavior
// POSITIVE CONTROL: prove the node COULD have formed; only the fallback was missing.
cluster.Join(cluster.SelfAddress);
await TwoNodeClusterFixture.WaitForMembersUp(node, 1, TimeSpan.FromSeconds(20));
}
[Fact]
public async Task Peer_booting_after_self_form_joins_the_existing_cluster_no_island()
{
var portA = TwoNodeClusterFixture.GetFreeTcpPort();
var portB = TwoNodeClusterFixture.GetFreeTcpPort();
// B cold-starts alone (A dead), self-forms after 2s.
var nodeB = StartNode(portB, peerPort: portA, selfFormAfter: TimeSpan.FromSeconds(2));
await TwoNodeClusterFixture.WaitForMembersUp(nodeB, 1, TimeSpan.FromSeconds(20));
// A boots later with the pair seed list. B answers InitJoin, so A must JOIN B's
// cluster instead of islanding. Generous window on A so its fallback can't race.
var nodeA = StartNode(portA, peerPort: portB, selfFormAfter: TimeSpan.FromSeconds(30));
await TwoNodeClusterFixture.WaitForMembersUp(nodeA, 2, TimeSpan.FromSeconds(20));
await TwoNodeClusterFixture.WaitForMembersUp(nodeB, 2, TimeSpan.FromSeconds(20));
}
[Fact]
public async Task Non_seed_node_never_self_forms()
{
var selfPort = TwoNodeClusterFixture.GetFreeTcpPort();
var deadSeedPort = TwoNodeClusterFixture.GetFreeTcpPort();
var node = StartNode(selfPort, deadSeedPort, selfFormAfter: TimeSpan.FromSeconds(1), selfInSeeds: false);
await Task.Delay(TimeSpan.FromSeconds(5)); // 5x the window
var cluster = Cluster.Get(node);
cluster.State.Members.ShouldBeEmpty(); // guard refused to island a non-seed
// Positive control: the guard (not the environment) prevented formation.
cluster.Join(cluster.SelfAddress);
await TwoNodeClusterFixture.WaitForMembersUp(node, 1, TimeSpan.FromSeconds(20));
}
```
**Step 2: Run** — all 4 PASS. **Step 3: Commit** `test(cluster): self-form fallback — disabled, late-peer merge, non-seed guard`.
---
### Task 4 (A4): Production wiring + fix the misleading `SeedNodes` doc comment
**Classification:** small
**Estimated implement time:** ~3 min
**Parallelizable with:** Task 3
**Files:**
- Modify: `src/ZB.MOM.WW.ScadaBridge.Host/Actors/AkkaHostedService.cs` (inside `GetOrCreateActorSystem`, after the `WhenTerminated` continuation ending ~line 218, before `_actorSystem = system;`)
- Modify: `src/ZB.MOM.WW.ScadaBridge.ClusterInfrastructure/ClusterOptions.cs:33-38`
**Step 1: Wire the fallback:**
```csharp
// InitJoin self-form fallback (decision 2026-07-22): without it a non-first seed
// cold-starting while its peer is down loops on InitJoin forever — auto-down closed
// the crash-failover gap, this closes the cold-start-alone gap. Guarded inside Arm:
// disabled when SelfFormAfter is unset, inert when this node is not its own seed.
ClusterBootstrapFallback.Arm(system, _clusterOptions, _logger);
```
**Step 2: Fix the `SeedNodes` doc comment** (replace the `<summary>`):
```csharp
/// <summary>
/// Akka.NET cluster seed nodes. Both nodes are seed nodes — each node lists itself and its
/// partner. NOTE: listing both is NOT sufficient for "either can start first": Akka only lets
/// the FIRST listed seed form a new cluster, so a lone non-first seed waits on InitJoin until
/// <see cref="SelfFormAfter"/> expires and the self-form fallback fires (decision 2026-07-22).
/// Must contain at least one entry.
/// </summary>
```
**Step 3: Build + targeted tests**
```bash
dotnet build ZB.MOM.WW.ScadaBridge.slnx # 0 warnings (TreatWarningsAsErrors)
dotnet test tests/ZB.MOM.WW.ScadaBridge.Host.Tests
dotnet test tests/ZB.MOM.WW.ScadaBridge.ClusterInfrastructure.Tests
```
**Step 4: Commit** `feat(cluster): arm self-form fallback at ActorSystem creation; honest SeedNodes doc`.
---
### Task 5 (A5): appsettings sweep
**Classification:** trivial
**Estimated implement time:** ~3 min
**Parallelizable with:** Task 6
Add `"SelfFormAfter": "00:00:10"` next to `"SplitBrainResolverStrategy"` in the `ScadaBridge:Cluster` section of each (explicit for operator visibility; matches the code default):
- `src/ZB.MOM.WW.ScadaBridge.Host/appsettings.Central.json`, `appsettings.Site.json`
- `docker/central-node-a/appsettings.Central.json`, `docker/central-node-b/appsettings.Central.json`
- `docker/site-a-node-a/appsettings.Site.json`, `docker/site-a-node-b/appsettings.Site.json`
- `docker/site-b-node-a/appsettings.Site.json`, `docker/site-b-node-b/appsettings.Site.json`
- `docker/site-c-node-a/appsettings.Site.json`, `docker/site-c-node-b/appsettings.Site.json`
- `docker-env2/` — the 4 node appsettings files
Then `dotnet test tests/ZB.MOM.WW.ScadaBridge.Host.Tests` and commit `config(cluster): SelfFormAfter=10s explicit in all node appsettings`.
> Ops note (do NOT edit here): the gitignored `deploy/wonder-app-vd03/` overlay gets the same key on the next production deploy.
---
### Task 6 (A6): Docs
**Classification:** small
**Estimated implement time:** ~5 min
**Parallelizable with:** Task 5
- `docs/requirements/Component-ClusterInfrastructure.md` — replace the "registered outage gap" recovery text (env-var override) with `SelfFormAfter` semantics + island guard + boot-partition trade.
- `docker/README.md` (~line 289) — same replacement; keep the partition-trade note.
- `docs/deployment/topology-guide.md:101` — **fix the stale keep-oldest claim** (still says "Keep-oldest with `down-if-alone = on`"): rewrite to auto-down default + `SelfFormAfter`, pointing at the decision records.
- `CLAUDE.md` (~line 222) — update the boot-order note: pairs no longer require the first seed for cold start; note the 10 s window.
Commit: `docs(cluster): SelfFormAfter fallback; fix stale keep-oldest note in topology guide`.
---
### Task 7 (A7): Full verification + optional docker live gate
**Classification:** standard
**Estimated implement time:** ~5 min (suite runtime dominates)
**Parallelizable with:** none
**Step 1:**
```bash
cd ~/Desktop/ScadaBridge
dotnet build ZB.MOM.WW.ScadaBridge.slnx # expect 0 warnings
dotnet test ZB.MOM.WW.ScadaBridge.slnx # expect green vs pre-existing baseline
```
**Step 2 (LIVE GATE — run if the docker rig is available; else record deferred-live):**
```bash
cd docker && bash deploy.sh # rebuild with the fallback
docker compose stop central-a central-b # verify service names in docker-compose.yml first
docker compose start central-b # cold-start ONLY the non-first seed
docker compose logs -f central-b | grep -m1 "Self-forming a cluster" # ≈10s after start
curl -fsS http://localhost:9002/health/active # expect 200
docker compose start central-a # first seed returns → must JOIN, not island
docker compose logs central-a | grep -i "Welcome"
```
Expected: `central-b` self-forms ≈10 s and serves (previously a permanent wedge); `central-a` rejoins as youngest/standby.
---
### Task 8 (D1): `IManualFailoverService` + cluster-level test
**Classification:** high-risk (cluster behavior)
**Estimated implement time:** ~5 min
**Parallelizable with:** none
**Files:**
- Create: `src/ZB.MOM.WW.ScadaBridge.CentralUI/Services/IManualFailoverService.cs`
- Create: `src/ZB.MOM.WW.ScadaBridge.Host/Health/AkkaManualFailoverService.cs`
- Modify: `src/ZB.MOM.WW.ScadaBridge.Host/Program.cs` (register in the Central branch, next to `IActiveNodeGate` ~line 330)
- Test: `tests/ZB.MOM.WW.ScadaBridge.IntegrationTests/Cluster/ManualFailoverTests.cs`
**Step 1: Failing tests** on `TwoNodeClusterFixture`:
```csharp
[Fact]
public async Task Failover_makes_the_oldest_leave_and_the_survivor_take_over()
{
await using var f = await TwoNodeClusterFixture.StartAsync();
var oldest = Akka.Cluster.Cluster.Get(f.NodeA); // NodeA started first = oldest
var target = AkkaManualFailoverService.FailOverCore(f.NodeB, "Central"); // issued from the OTHER node
target.ShouldBe(oldest.SelfAddress);
// Graceful exit path: the left node's own ActorSystem terminates…
await f.NodeA.WhenTerminated.WaitAsync(TimeSpan.FromSeconds(30));
// …and the survivor becomes a 1-member cluster and the oldest-Up active node.
await TwoNodeClusterFixture.WaitForMemberRemoved(f.NodeB, oldest.SelfAddress, TimeSpan.FromSeconds(30));
ActiveNodeEvaluator.SelfIsOldestUp(Akka.Cluster.Cluster.Get(f.NodeB)).ShouldBeTrue();
}
[Fact]
public void Failover_refuses_when_no_peer_exists()
{ /* 1-node cluster → FailOverCore returns null, node still Up afterwards (positive assert) */ }
```
**Step 2:** Run → FAIL (service missing).
**Step 3: Implement.** Interface in CentralUI (CentralUI stays Akka-free):
```csharp
public interface IManualFailoverService
{
/// <summary>Gracefully fails over the central cluster: the current active (oldest Up)
/// member leaves, restarts via its supervisor, and rejoins as standby. Returns the
/// address string acted on, or null when there is no peer to fail over to.</summary>
Task<string?> FailOverCentralAsync(string actor);
}
```
Host implementation — static testable core + thin DI wrapper:
```csharp
public sealed class AkkaManualFailoverService : IManualFailoverService
{
// ctor: (AkkaHostedService akka, ICentralAuditWriter audit, ILogger<AkkaManualFailoverService> logger)
public async Task<string?> FailOverCentralAsync(string actor)
{
var system = _akka.GetOrCreateActorSystem();
var target = FailOverCore(system, role: "Central", dryRun: true);
if (target is null) return null; // peer guard
await _audit.WriteAsync(/* canonical AuditEvent: Action=cluster.manual-failover,
Actor=actor, DetailsJson={"target": target}, Outcome=Success — copy the exact
call shape from an existing audited admin action (e.g. the Sites admin service);
use the CENTRAL audit writer, not the shared seam */);
FailOverCore(system, role: "Central"); // Cluster.Leave(target)
_logger.LogWarning("Manual failover triggered by {Actor}: {Target} is leaving the cluster.", actor, target);
return target.ToString();
}
/// <summary>Oldest Up member with the role leaves — mirrors ActiveNodeEvaluator's oldest-Up
/// rule so the node acted on is exactly the one hosting the singletons. Returns null when
/// fewer than 2 Up members carry the role (no peer = failover would be an outage).</summary>
public static Address? FailOverCore(ActorSystem system, string role, bool dryRun = false)
{
var cluster = Akka.Cluster.Cluster.Get(system);
var withRole = cluster.State.Members
.Where(m => m.Status == MemberStatus.Up && m.HasRole(role))
.OrderBy(m => m, Member.AgeOrdering)
.ToList();
if (withRole.Count < 2) return null;
var oldest = withRole[0];
if (!dryRun) cluster.Leave(oldest.Address);
return oldest.Address;
}
}
```
**Step 4:** Register in the Central branch of `Program.cs`. **Step 5:** Tests PASS → commit `feat(ui): manual central failover service — graceful Leave of the oldest Up member`.
---
### Task 9 (D2): Health page button + bUnit tests + runbook
**Classification:** standard
**Estimated implement time:** ~5 min
**Parallelizable with:** none
**Files:**
- Modify: `src/ZB.MOM.WW.ScadaBridge.CentralUI/Components/Pages/Monitoring/Health.razor` (central-cluster card, near the Nodes column ~line 233)
- Test: `tests/ZB.MOM.WW.ScadaBridge.CentralUI.Tests/HealthFailoverButtonTests.cs` (follow the project's existing bUnit page-test pattern)
- Modify: `docs/requirements/Component-ClusterInfrastructure.md` + `docker/README.md` — manual-failover runbook paragraph
**Step 1: Failing bUnit tests:** (a) button absent without the admin policy; (b) present + enabled for admin with ≥2 online central nodes; (c) disabled with tooltip at 1 node; (d) confirm flow calls `IManualFailoverService.FailOverCentralAsync` exactly once (fake service).
**Step 2: Implement:**
```razor
<AuthorizeView Policy="@AuthorizationPolicies.RequireAdmin">
<button class="btn btn-outline-warning btn-sm"
disabled="@(!CentralHasPeer)"
title="@(CentralHasPeer ? "Gracefully restart the active node; the standby takes over."
: "No standby available — failover would be an outage.")"
@onclick="() => _showFailoverConfirm = true">
Trigger failover
</button>
</AuthorizeView>
```
plus a confirmation modal (copy the page's existing dialog idiom). The warning text MUST state: the active node restarts, roles swap, and this page will briefly disconnect and reconnect against the new active node (Traefik routes to the active). On confirm: call the service with the authenticated user name; surface the returned target address.
**Step 3:** Tests PASS → commit `feat(ui): admin manual-failover control on the health page`.
**Live check (fold into the Task 7 gate when the rig is up):** press the button, watch `central-a` restart and `central-b`'s badge flip to Primary; audit row lands in `dbo.AuditLog`.
---
### Task 10 (D3, OPTIONAL — confirm with the user before executing): site-pair failover from the central UI
**Classification:** high-risk (new cross-cluster message contract)
**Estimated implement time:** exploration first; likely 2–3 tasks if approved
The Health page also shows per-site node cards (Primary/Standby from heartbeats), but central and sites are **separate Akka clusters** — a site failover needs a `TriggerSiteFailover(siteId)` command over the existing central→site transport (same channel as the Retry/Discard relay), handled on the site's active node by `Cluster.Leave(SelfAddress)`. That adds a versioned message contract (rolling-upgrade surface). If approved: explore `ZB.MOM.WW.ScadaBridge.Communication` for the command path, mirror an existing command end-to-end, per-site button with the same guard/confirm/audit rules. Otherwise: file a follow-up issue and skip.
---
## Completion
- Merge decision via the finishing-a-development-branch flow (family convention: ff-merge to `main` + push to gitea, or PR — ask the user).
- After merge: update `scadaproj/akka_failover.md` §6.1 status, `scadaproj/CLAUDE.md` index row, and memory `ha-availability-over-partition-safety` (tracked as the family-docs task in the scadaproj index plan).
- Verification-before-completion applies throughout: no task is done without its command output; live gates may be recorded deferred-live if the rig is down.
docs/plans/2026-07-22-selfform-fallback-and-manual-failover.md.tasks.json
+61
View File
@@ -0,0 +1,61 @@
{
"planPath": "docs/plans/2026-07-22-selfform-fallback-and-manual-failover.md",
"tasks": [
{
"id": 0,
"subject": "Part 1 (SUPERSEDED Tasks 1-4): SelfFormAfter watchdog - implemented, review+test rejected it, reverted",
"status": "completed"
},
{
"id": 1,
"subject": "Part 1 as executed: self-first seed ordering + StartupValidator rule + SelfFirstSeedBootstrapTests + 14-config sweep + docs",
"status": "completed",
"blockedBy": [
0
]
},
{
"id": 2,
"subject": "Task 7: full solution verification + optional docker live gate",
"status": "completed",
"blockedBy": [
1
]
},
{
"id": 3,
"subject": "Task 8: IManualFailoverService + cluster-level test",
"status": "completed",
"blockedBy": [
2
]
},
{
"id": 4,
"subject": "Task 9: Health page failover button + bUnit tests + runbook",
"status": "completed",
"blockedBy": [
3
]
},
{
"id": 5,
"subject": "Task 10: site-pair failover via central\u2192site transport (BUILT \u2014 user approved 2026-07-22)",
"status": "completed",
"blockedBy": [
4
]
},
{
"id": 6,
"subject": "FOLLOW-UP: OtOpcUa converged on self-first seed ordering (2 local commits, unpushed); scadaproj CLAUDE.md index rows still owner-held",
"status": "completed"
},
{
"id": 7,
"subject": "OPS: reorder SeedNodes self-first in gitignored deploy/wonder-app-vd03/ before next deploy (hard boot gate)",
"status": "pending"
}
],
"lastUpdated": "2026-07-22T00:00:00Z"
}
docs/requirements/Component-ClusterInfrastructure.md
+68 -12
View File
@@ -92,27 +92,83 @@ Akka.NET cluster singletons run on the active node of their cluster and migrate
- Health reporting resumes from the new active node.
- Alarm states are re-evaluated from incoming values (alarm state is in-memory only).
## Split-Brain Resolution
## Downing Strategy (auto-down — availability-first)
The system uses the Akka.NET **keep-oldest** split-brain resolver strategy:
**Decision 2026-07-21 (owner decision, resolves the deferred keep-oldest topology/strategy question):** the clusters run the **`auto-down`** downing strategy — Akka's `AutoDowning` provider with `auto-down-unreachable-after` = `StableAfter` (15s). The leader among the *reachable* members downs the unreachable peer after the stability window:
- On a network partition, the node that has been in the cluster longest remains active. The younger node downs itself.
- **Stable-after duration**: 15 seconds. The cluster membership must remain stable (no changes) for 15 seconds before the resolver acts to down unreachable nodes. This prevents premature downing during startup or rolling restarts.
- **`down-if-alone = on`**: The keep-oldest resolver is configured with `down-if-alone` enabled. If the oldest node finds itself alone (no other reachable members), it downs itself rather than continuing as a single-node cluster. This prevents the oldest node from running in isolation during a network partition while the younger node also forms its own cluster.
- **Why keep-oldest**: With only two nodes, quorum-based strategies (static-quorum, keep-majority) cannot distinguish "one node crashed" from "network partition" — both sides see fewer than quorum and both would down themselves, resulting in total cluster shutdown. Keep-oldest with `down-if-alone` provides safe singleton ownership — at most one node runs the cluster singleton at any time.
- **Either-node crash is survivable.** If the standby crashes, the active node downs it and continues (as before). If the **active/oldest** node crashes, the younger survivor becomes leader among the reachable members, downs the dead oldest, becomes the oldest itself, re-hosts every cluster singleton, and `/health/active` flips to it — **no operator action and no victim restart required**. This closes the keep-oldest total-outage gap.
- **The accepted trade: dual-active during a real network partition.** With both nodes alive but partitioned, each side downs the other and continues as a one-node cluster — both claim active until the partition heals and an operator restarts one side (the restarted node rejoins the other as a fresh incarnation). This trade was chosen deliberately: site pairs run one node per VM with no shared lease infrastructure (no Kubernetes, no SQL at sites) to arbitrate, and a stalled system is a bigger operational risk than a rare LAN partition.
- **Stable-after duration**: 15 seconds of sustained unreachability before downing. This prevents premature downing during startup, rolling restarts, or transient network blips.
### Down-if-alone recovery
### Why not the alternatives (all verified against Akka.NET 1.5.62 source, 2026-07-21)
When a node downs itself (via `down-if-alone`, or any other SBR decision), the resolver is configured with `run-coordinated-shutdown-when-down = on`, so the self-down runs `CoordinatedShutdown` and **terminates that node's `ActorSystem`**. The Host process must not keep running with a dead actor system — it would serve nothing and be restarted by nobody. The recovery contract is:
- **keep-oldest** (used until 2026-07-21): partition-safe, but in a two-node cluster a crash of the **oldest** is a total outage. `KeepOldest.OldestDecision` only lets `down-if-alone` rescue the survivor when the surviving side has **≥ 2 members** (`otherSide == 1 && thisSide >= 2`); with 1-vs-1 the younger survivor takes `DownReachable` — it downs *itself*. Verified live on the docker rig: the survivor logged `SBR took decision Akka.Cluster.SBR.DownReachable … including myself`, exited, and looped on `InitJoin`. The strategy remains supported in `ClusterOptions` (`SplitBrainResolverStrategy: keep-oldest`) for deployments that prefer partition-safety over availability.
- **static-quorum (quorum-size 1)**: Akka's `IsTooManyMembers` guard (`members > quorum*2-1`, i.e. `2 > 1`) returns **DownAll** on any unreachability — total shutdown, strictly worse.
- **keep-majority**: a 1-vs-1 split keeps the side with the lowest address, which just moves the fatal crash from "the oldest" to "the lowest-address node".
- **lease-majority**: needs a shared lease store (Kubernetes API, SQL, …) reachable from both nodes — not available at sites (one node per VM, no shared infrastructure).
1. Self-down ⇒ `CoordinatedShutdown` ⇒ `ActorSystem` termination.
### Downed-node recovery
When a node is downed (auto-downed by its peer after a partition heals, or a keep-oldest self-down where that strategy is configured), `run-coordinated-shutdown-when-down = on` runs `CoordinatedShutdown` and **terminates that node's `ActorSystem`**. The Host process must not keep running with a dead actor system — it would serve nothing and be restarted by nobody. The recovery contract is:
1. Down ⇒ `CoordinatedShutdown` ⇒ `ActorSystem` termination.
2. The Host watches `ActorSystem.WhenTerminated`; a termination that is **not** the host's own `StopAsync` triggers `IHostApplicationLifetime.StopApplication()`, so **the process exits**.
3. The service supervisor restarts it — docker `restart: unless-stopped`, or Windows service recovery actions (`sc.exe failure … restart/…`).
4. The restarted process rejoins as a **fresh incarnation** (the keep-oldest resolver handles the rejoin cleanly; there is no stale membership to reconcile) — **but only while a peer still holding cluster state is reachable**. A lone restarted node that is *not* the first seed cannot re-form a cluster on its own (see the seed-node bootstrap constraint below); it waits for its peer.
4. The restarted process rejoins as a **fresh incarnation**, and can re-form the cluster on its own if no peer is reachable — see Seed Node Ordering below.
**Seed-node bootstrap constraint.** Only the FIRST seed listed in `Cluster:SeedNodes` may self-join to form a *new* cluster. All nodes list the same first seed (e.g. `scadabridge-central-a`), so a lone restarted non-first-seed node (with the first seed still down) loops on `InitJoin` forever — never `Up`, never routable. This is why the two-node keep-oldest **oldest/active-node crash is a total-outage gap**: after the oldest dies the younger survivor self-downs, and it cannot re-bootstrap alone. Recovery is operator-driven — either restart the dead first-seed node (preferred) or restart the survivor with a self-first seed override (`ScadaBridge__Cluster__SeedNodes__0` = self, `__1` = peer). The repo does not ship self-first ordering per node: with both nodes self-first a simultaneous cold start risks two independent one-node clusters that never merge. Removing the gap itself is the **registered deferred keep-oldest topology/strategy decision** (master tracker 2026-07-08, owner: user).
### Seed Node Ordering
The docker failover drill (`docker/failover-drill.sh`) exercises this per direction: `standby` mode proves SBR downing + singleton continuity on the oldest; `active` mode measures the registered total-outage gap and the recovery-on-restart path.
**Every node lists ITSELF as `seed-nodes[0]` and its partner second (decision 2026-07-22).**
Akka runs two different bootstrap processes depending on that first entry. When `seed-nodes[0]` is the node's own address it runs `FirstSeedNodeProcess`: it `InitJoin`s the *other* seeds and self-joins only after `seed-node-timeout` elapses with nobody answering. When it is not, the node runs `JoinSeedNodeProcess`, which can never form a new cluster — it retries `InitJoin` indefinitely.
Until 2026-07-22 every node listed the same first seed, so a node that had to **boot alone** — a cold start of only the non-first-seed VM, or the survivor crashing while its peer was still dead — never reached `Up` and was never routable. That was the **registered outage gap**, and recovery was operator-driven. Self-first ordering closes it using Akka's own protocol, and `StartupValidator` fails the boot if a node config ever breaks the ordering (the invariant is silent when violated, so it is enforced loudly).
Behavior, covered by `SelfFirstSeedBootstrapTests` (real in-process clusters built from `BuildHocon` at production failure-detection timings):
| Scenario | Behavior |
|---|---|
| Lone cold-start, peer dead | Self-joins after `seed-node-timeout` (~5s) — operational, unattended |
| Restart into a **live** peer | The peer answers `InitJoinAck`; the node joins the existing cluster and never islands |
| Both cold-start simultaneously, mutually reachable | The `InitJoin` handshake resolves it *before* either self-joins → **one** 2-member cluster |
| Both cold-start during a genuine boot-time **partition** | Each forms its own cluster — the same dual-active class `auto-down` already accepts, same recovery (restart one side) |
**Rejected alternative — an external self-form timer.** A watchdog that waited a configurable window for membership and then called `Cluster.Join(SelfAddress)` was implemented and discarded. Its success signal ("am I `Up` yet?") cannot distinguish *no seed answered* from *a seed answered and the join is in flight*, because it sits outside Akka's join handshake. On a routine standby restart the peer is alive but the join stalls behind removal of the restarting node's own stale incarnation (the peer must down it, then wait for the failure detector and a leader action); a `Join(self)` issued during `TryingToJoin` abandons the in-flight join and forms a second cluster at the same address. Measured: a permanent split that had not healed after 90s, converting a routine restart into an outage of the previously-healthy node. Akka's first-seed process has no such race because it is part of the handshake, which is why the ordering — not a timer — is the mechanism.
The docker failover drill (`docker/failover-drill.sh`) proves both directions: `standby` mode kills the younger node (active untouched, zero routing blips); `active` mode kills the active/oldest node and asserts the survivor **takes over while the victim is still down**.
### Manual Failover (admin-triggered)
An Administrator can swap the central pair's roles deliberately — for planned maintenance on the active node, or to move singletons off a node behaving badly without waiting for a crash. The control is the **Trigger failover** button on the central-cluster card of the Health dashboard (`/monitoring/health`).
Semantics:
- **Graceful `Leave`, never `Down`.** The active node leaves the cluster so `ClusterSingletonManager` hands its singletons to the standby before the member is removed. A `Down` would skip the hand-off.
- **Target = the oldest Up member with the `Central` role**, the same rule `ActiveNodeEvaluator` uses, so the node acted on is exactly the one hosting the singletons — never Akka's cluster *leader*, whose address-ordered definition diverges from singleton placement after a restart.
- **Peer guard.** Refused when fewer than two Up `Central` members exist: failing over a lone node is an outage, not a failover. Enforced server-side in `AkkaManualFailoverService`; the button is also disabled client-side when the pair has no online standby.
- **Admin-only** (`AuthorizationPolicies.RequireAdmin`). The Health page itself is all-roles, so the gate lives on the control.
- **Audited before acting.** One `AuditChannel.Cluster` / `AuditKind.ManualFailover` row is written through `ICentralAuditWriter` *before* the Leave is issued, naming the actor and the target address — the acting node can be the one that goes away, and an audit written afterwards could be lost to the shutdown it describes. Audit failure never blocks the failover.
- **Your own page will disconnect.** Traefik routes the UI to the active node, so triggering a failover drops the admin's Blazor circuit; it reconnects against the new active node. The confirmation dialog says so.
After the Leave the node's `ActorSystem` terminates, the `WhenTerminated` watchdog exits the process, the service supervisor restarts it, and it rejoins as the youngest member (the new standby). Recovery is the normal restart contract above — no interaction with seed ordering, since the peer is alive on this path.
#### Site-pair failover
The same control appears on each **site** card. Central and each site are separate Akka clusters, so central cannot act on a site's membership — it *asks*, over the existing ClusterClient command/control channel:
1. `CommunicationService.TriggerSiteFailoverAsync` sends a `TriggerSiteFailover` inside a `SiteEnvelope`.
2. The site's `SiteCommunicationActor` (registered per node, so contact rotation reaches whichever answers) resolves the target from cluster state and issues the graceful `Leave` locally.
3. It replies `SiteFailoverAck` — sent **before** the Leave takes effect, so the ack still arrives when the acking node is the one leaving.
Differences from central failover, all deliberate:
- **Role scope is `site-{SiteId}`, not the base `Site` role.** Site singletons (the Deployment Manager) are placed on the site-specific role; using the base role would move the wrong node. Pinned by both a unit test asserting the role string and a real-cluster test.
- **Misroute is refused.** A command whose `SiteId` does not match the receiving node's is rejected rather than acted on — a misroute must never fail over a site the operator did not select.
- **A fault is acked, not thrown.** Reporting through the ack keeps the reason; letting it reach supervision would restart the communication actor and reduce central's Ask to a bare timeout.
- **The operator's own session is unaffected** — a different cluster entirely. The confirmation dialog therefore does *not* carry the "this page will disconnect" warning that the central one does.
- **Refusal vs unreachable are distinct.** A `false` ack is a definitive answer from the site (peer guard, misroute); a timeout means the site never answered. Both surface to the operator with their own wording, because only one of them means "nothing happened".
**Rolling upgrade.** A site running a binary older than this contract has no handler for `TriggerSiteFailover`; the message dead-letters and central's Ask times out, reported as "site did not respond". That is the correct user-facing outcome — an old site genuinely cannot honour the request. Message evolution stays additive-only.
## Single-Node Operation
docs/requirements/Component-Communication.md
+28 -1
View File
@@ -109,6 +109,31 @@ The streaming protocol is defined in `sitestream.proto` (`src/ZB.MOM.WW.ScadaBri
- The `oneof event` pattern is extensible — future event types (health metrics, connection state changes) are added as new fields without breaking existing consumers.
- Proto field numbers are never reused; new RPCs and message fields are appended additively. Old clients ignore unknown `oneof` variants.
##### Authentication (preshared key, 2026-07-22)
Every RPC on this service — streaming and unary alike — requires a preshared key, presented as
`authorization: Bearer <key>` metadata and verified by `ControlPlaneAuthInterceptor` on the site
node. The gate is **fail-closed**: with no key configured, every call is refused with
`PermissionDenied`, and `StartupValidator` refuses to boot a site node in that state (an unset key
would otherwise leave the node joined and reporting healthy while serving nothing).
Keys are scoped **one per site** — secret `SB-GRPC-PSK-<siteId>`, so a compromised site never
yields another site's key. The site node reads its own key from
`ScadaBridge:Communication:GrpcPsk` (in production a `${secret:}` reference expanded before the
host is built); central resolves each site's key at channel-build time via `SitePskProvider`,
because sites are created at runtime and cannot be enumerated in configuration at boot.
`ControlPlaneCredentials` binds the key to the channel as `CallCredentials`, so no individual call
site can omit it.
This is distinct from `LocalDb:Replication:ApiKey`, which gates
`/localdb_sync.v1.LocalDbSync/` on the same listener via its own interceptor: that key
authenticates the *pair partner* for database replication, a different trust relationship, and the
two are never shared.
The transport remains h2c, so the key is readable and replayable by anyone on the path — the
boundary still assumes a trusted network, with the bar raised from "can reach the port" to "can
read the traffic". TLS is follow-on hardening and does not change this design.
#### Enriched AlarmStateUpdate (Native Alarm Mirror)
`AlarmStateUpdate` carries the read-only native alarm mirror (Computed, native OPC UA, and native MxAccess Gateway alarms) to central over the **existing gRPC real-time stream** — no new transport, no command/control round-trip. The message was extended **additively**: existing fields 1–7 are unchanged, and fields 8–23 carry the enriched native-alarm state. Old clients that only read fields 1–7 continue to work; new fields are populated only where the source provides them.
@@ -306,7 +331,9 @@ Akka.NET guarantees message ordering between a specific sender/receiver actor pa
## ManagementActor and ClusterClient
The ManagementActor is registered at the well-known path `/user/management` on central nodes and advertised via **ClusterClientReceptionist**. External tools (primarily the CLI) connect using Akka.NET ClusterClient, which contacts the receptionist to discover the ManagementActor. This is a separate ClusterClient usage from the inter-cluster ClusterClient connections used for central-site messaging — the CLI does not participate in cluster membership or affect the hub-and-spoke topology.
The ManagementActor runs at the well-known path `/user/management` on central nodes. It is **not** advertised via ClusterClientReceptionist, and no ClusterClient reaches it.
That registration existed until 2026-07-22 for an out-of-cluster CLI that was never built. The shipped CLI speaks **HTTP Basic to the central `/management` endpoints**; those endpoints ask the ManagementActor **in-process** through `ManagementActorHolder` (`ManagementEndpoints.cs`). Nothing in the repo ever sent to `/user/management` across the boundary, so the registration was deleted — leaving exactly one receptionist registration per cluster role (`CentralCommunicationActor` on central, `SiteCommunicationActor` on sites), both of which serve inter-cluster central↔site messaging and are themselves scheduled for removal by the gRPC transport migration (`docs/plans/2026-07-22-clusterclient-to-grpc-plan.md`).
## Connection Failure Behavior
docs/requirements/Component-Host.md
+5 -1
View File
@@ -125,7 +125,11 @@ The Host bootstraps the Akka.NET actor system from a **hand-assembled, injection
### REQ-HOST-6a: ClusterClientReceptionist (Central Only)
On central nodes, the Host must configure the Akka.NET **ClusterClientReceptionist** and register the ManagementActor with it. This allows external processes (e.g., the CLI) to discover and communicate with the ManagementActor via ClusterClient without joining the cluster as full members. The receptionist is started as part of the Akka.NET bootstrap (REQ-HOST-6) on central nodes only.
On central nodes, the Host must configure the Akka.NET **ClusterClientReceptionist** and register the **CentralCommunicationActor** with it, so that site clusters' ClusterClients can reach the central command/control endpoint without joining the central cluster.
**The ManagementActor is NOT registered with the receptionist** (removed 2026-07-22). That registration was written for an out-of-cluster CLI that was never built: the shipped CLI speaks HTTP Basic to the central `/management` endpoints, which ask the ManagementActor **in-process** via `ManagementActorHolder` (`ManagementEndpoints.cs`). No sender to `/user/management` existed anywhere in the repo, so the registration only widened the cluster-client surface for nothing. The actor itself still runs at `/user/management`; only its cross-boundary advertisement is gone.
> **Migration note.** This receptionist registration — and the `CentralCommunicationActor` one that remains — are scheduled for deletion once the site↔central transport moves to gRPC. See `docs/plans/2026-07-22-clusterclient-to-grpc-plan.md`.
### REQ-HOST-7: ASP.NET Web Endpoints
docs/requirements/Component-ManagementService.md
+1 -1
View File
@@ -24,7 +24,7 @@ Central cluster only. The ManagementActor runs as a plain actor on **every** cen
### ManagementActor
The central actor that receives and processes all management commands. Registered at a well-known actor path (`/user/management`) and with ClusterClientReceptionist.
The central actor that receives and processes all management commands. Created at the well-known actor path (`/user/management`) and handed to `ManagementActorHolder`, which is how every in-process caller (notably `ManagementEndpoints`) reaches it. It is **not** advertised via ClusterClientReceptionist — see the note under "ManagementActor and ClusterClient" in Component-Communication.
### ManagementEndpoints
src/ZB.MOM.WW.ScadaBridge.AuditLog/Central/GrpcPullAuditEventsClient.cs
+34 -11
View File
@@ -145,7 +145,7 @@ public sealed class GrpcPullAuditEventsClient : IPullAuditEventsClient
{
try
{
var reply = await _invoker.InvokeAsync(endpoint, request, ct).ConfigureAwait(false);
var reply = await _invoker.InvokeAsync(siteId, endpoint, request, ct).ConfigureAwait(false);
return (reply, false);
}
catch (RpcException ex) when (IsTolerable(ex.StatusCode))
@@ -226,11 +226,17 @@ public sealed class GrpcPullAuditEventsClient : IPullAuditEventsClient
/// May throw <see cref="RpcException"/> / <see cref="HttpRequestException"/>
/// on transport faults — the caller classifies and swallows tolerable ones.
/// </summary>
/// <param name="siteId">
/// The site being pulled from. Selects which preshared key the call presents —
/// <c>PullAuditEvents</c> is gated by the site's <c>ControlPlaneAuthInterceptor</c>, and
/// keys are per-site, so the endpoint alone is not enough to authenticate.
/// </param>
/// <param name="endpoint">The site gRPC authority (e.g. <c>http://site-a:8083</c>).</param>
/// <param name="request">The wire-format pull request.</param>
/// <param name="ct">Cancellation token.</param>
/// <returns>The wire-format pull response.</returns>
Task<ProtoPullResponse> InvokeAsync(string endpoint, ProtoPullRequest request, CancellationToken ct);
Task<ProtoPullResponse> InvokeAsync(
string siteId, string endpoint, ProtoPullRequest request, CancellationToken ct);
}
}
@@ -251,8 +257,9 @@ public sealed class GrpcPullAuditEventsClient : IPullAuditEventsClient
public sealed class GrpcPullAuditEventsInvoker
: GrpcPullAuditEventsClient.IPullAuditEventsInvoker, IDisposable
{
private readonly ConcurrentDictionary<string, GrpcChannel> _channels = new(StringComparer.Ordinal);
private readonly ConcurrentDictionary<(string Site, string Endpoint), GrpcChannel> _channels = new();
private readonly CommunicationOptions _options;
private readonly ISitePskProvider? _pskProvider;
/// <summary>
/// Creates the invoker using default <see cref="CommunicationOptions"/>.
@@ -268,15 +275,27 @@ public sealed class GrpcPullAuditEventsInvoker
/// </summary>
/// <param name="options">Communication options supplying gRPC keepalive timings.</param>
public GrpcPullAuditEventsInvoker(CommunicationOptions options)
: this(options, pskProvider: null)
{
}
/// <summary>
/// Creates the invoker with per-site call credentials, the production shape: the site's
/// <c>ControlPlaneAuthInterceptor</c> refuses an unauthenticated <c>PullAuditEvents</c>.
/// </summary>
/// <param name="options">Communication options supplying gRPC keepalive timings.</param>
/// <param name="pskProvider">Resolves each site's preshared key; null dials unauthenticated.</param>
public GrpcPullAuditEventsInvoker(CommunicationOptions options, ISitePskProvider? pskProvider)
{
_options = options ?? throw new ArgumentNullException(nameof(options));
_pskProvider = pskProvider;
}
/// <inheritdoc />
public async Task<ProtoPullResponse> InvokeAsync(
string endpoint, ProtoPullRequest request, CancellationToken ct)
string siteId, string endpoint, ProtoPullRequest request, CancellationToken ct)
{
var channel = GetOrCreateChannel(endpoint);
var channel = GetOrCreateChannel(siteId, endpoint);
var client = new SiteStreamService.SiteStreamServiceClient(channel);
using var call = client.PullAuditEventsAsync(request, cancellationToken: ct);
return await call.ResponseAsync.ConfigureAwait(false);
@@ -288,12 +307,13 @@ public sealed class GrpcPullAuditEventsInvoker
// pool) and the loser would leak. Create-then-GetOrAdd-then-dispose-if-lost
// mirrors SiteStreamGrpcClientFactory: only the channel actually installed
// survives; a channel that lost the race is disposed immediately.
private GrpcChannel GetOrCreateChannel(string endpoint)
private GrpcChannel GetOrCreateChannel(string siteId, string endpoint)
{
if (!_channels.TryGetValue(endpoint, out var channel))
var key = (siteId, endpoint);
if (!_channels.TryGetValue(key, out var channel))
{
var created = CreateChannel(endpoint);
channel = _channels.GetOrAdd(endpoint, created);
var created = CreateChannel(siteId, endpoint);
channel = _channels.GetOrAdd(key, created);
if (!ReferenceEquals(channel, created))
{
created.Dispose();
@@ -302,7 +322,10 @@ public sealed class GrpcPullAuditEventsInvoker
return channel;
}
private GrpcChannel CreateChannel(string endpoint) =>
// Keyed by (site, endpoint) rather than endpoint alone: the call credentials are bound to
// the channel, and they are per-site, so two sites sharing an endpoint string would
// otherwise share one channel carrying the first site's key.
private GrpcChannel CreateChannel(string siteId, string endpoint) =>
GrpcChannel.ForAddress(endpoint, new GrpcChannelOptions
{
HttpHandler = new SocketsHttpHandler
@@ -311,7 +334,7 @@ public sealed class GrpcPullAuditEventsInvoker
KeepAlivePingTimeout = _options.GrpcKeepAlivePingTimeout,
KeepAlivePingPolicy = HttpKeepAlivePingPolicy.Always,
},
});
}.WithSiteCredentials(_pskProvider, siteId));
/// <summary>Disposes all cached channels.</summary>
public void Dispose()
src/ZB.MOM.WW.ScadaBridge.AuditLog/Central/GrpcPullSiteCallsClient.cs
+34 -11
View File
@@ -174,7 +174,7 @@ public sealed class GrpcPullSiteCallsClient : IPullSiteCallsClient
{
try
{
var reply = await _invoker.InvokeAsync(endpoint, request, ct).ConfigureAwait(false);
var reply = await _invoker.InvokeAsync(siteId, endpoint, request, ct).ConfigureAwait(false);
return (reply, false);
}
catch (RpcException ex) when (IsTolerable(ex.StatusCode))
@@ -254,11 +254,17 @@ public sealed class GrpcPullSiteCallsClient : IPullSiteCallsClient
/// May throw <see cref="RpcException"/> / <see cref="HttpRequestException"/>
/// on transport faults — the caller classifies and swallows tolerable ones.
/// </summary>
/// <param name="siteId">
/// The site being pulled from. Selects which preshared key the call presents —
/// <c>PullSiteCalls</c> is gated by the site's <c>ControlPlaneAuthInterceptor</c>, and
/// keys are per-site, so the endpoint alone is not enough to authenticate.
/// </param>
/// <param name="endpoint">The site gRPC authority (e.g. <c>http://site-a:8083</c>).</param>
/// <param name="request">The wire-format pull request.</param>
/// <param name="ct">Cancellation token.</param>
/// <returns>The wire-format pull response.</returns>
Task<ProtoPullResponse> InvokeAsync(string endpoint, ProtoPullRequest request, CancellationToken ct);
Task<ProtoPullResponse> InvokeAsync(
string siteId, string endpoint, ProtoPullRequest request, CancellationToken ct);
}
}
@@ -277,8 +283,9 @@ public sealed class GrpcPullSiteCallsClient : IPullSiteCallsClient
public sealed class GrpcPullSiteCallsInvoker
: GrpcPullSiteCallsClient.IPullSiteCallsInvoker, IDisposable
{
private readonly ConcurrentDictionary<string, GrpcChannel> _channels = new(StringComparer.Ordinal);
private readonly ConcurrentDictionary<(string Site, string Endpoint), GrpcChannel> _channels = new();
private readonly CommunicationOptions _options;
private readonly ISitePskProvider? _pskProvider;
/// <summary>Creates the invoker using default <see cref="CommunicationOptions"/>.</summary>
public GrpcPullSiteCallsInvoker()
@@ -292,15 +299,27 @@ public sealed class GrpcPullSiteCallsInvoker
/// </summary>
/// <param name="options">Communication options supplying gRPC keepalive timings.</param>
public GrpcPullSiteCallsInvoker(CommunicationOptions options)
: this(options, pskProvider: null)
{
}
/// <summary>
/// Creates the invoker with per-site call credentials, the production shape: the site's
/// <c>ControlPlaneAuthInterceptor</c> refuses an unauthenticated <c>PullSiteCalls</c>.
/// </summary>
/// <param name="options">Communication options supplying gRPC keepalive timings.</param>
/// <param name="pskProvider">Resolves each site's preshared key; null dials unauthenticated.</param>
public GrpcPullSiteCallsInvoker(CommunicationOptions options, ISitePskProvider? pskProvider)
{
_options = options ?? throw new ArgumentNullException(nameof(options));
_pskProvider = pskProvider;
}
/// <inheritdoc />
public async Task<ProtoPullResponse> InvokeAsync(
string endpoint, ProtoPullRequest request, CancellationToken ct)
string siteId, string endpoint, ProtoPullRequest request, CancellationToken ct)
{
var channel = GetOrCreateChannel(endpoint);
var channel = GetOrCreateChannel(siteId, endpoint);
var client = new SiteStreamService.SiteStreamServiceClient(channel);
using var call = client.PullSiteCallsAsync(request, cancellationToken: ct);
return await call.ResponseAsync.ConfigureAwait(false);
@@ -310,12 +329,13 @@ public sealed class GrpcPullSiteCallsInvoker
// concurrent first dials of the same endpoint can both build a GrpcChannel;
// only the channel actually installed survives, the loser is disposed.
// Mirrors SiteStreamGrpcClientFactory / GrpcPullAuditEventsInvoker.
private GrpcChannel GetOrCreateChannel(string endpoint)
private GrpcChannel GetOrCreateChannel(string siteId, string endpoint)
{
if (!_channels.TryGetValue(endpoint, out var channel))
var key = (siteId, endpoint);
if (!_channels.TryGetValue(key, out var channel))
{
var created = CreateChannel(endpoint);
channel = _channels.GetOrAdd(endpoint, created);
var created = CreateChannel(siteId, endpoint);
channel = _channels.GetOrAdd(key, created);
if (!ReferenceEquals(channel, created))
{
created.Dispose();
@@ -324,7 +344,10 @@ public sealed class GrpcPullSiteCallsInvoker
return channel;
}
private GrpcChannel CreateChannel(string endpoint) =>
// Keyed by (site, endpoint) rather than endpoint alone: the call credentials are bound to
// the channel, and they are per-site, so two sites sharing an endpoint string would
// otherwise share one channel carrying the first site's key.
private GrpcChannel CreateChannel(string siteId, string endpoint) =>
GrpcChannel.ForAddress(endpoint, new GrpcChannelOptions
{
HttpHandler = new SocketsHttpHandler
@@ -333,7 +356,7 @@ public sealed class GrpcPullSiteCallsInvoker
KeepAlivePingTimeout = _options.GrpcKeepAlivePingTimeout,
KeepAlivePingPolicy = HttpKeepAlivePingPolicy.Always,
},
});
}.WithSiteCredentials(_pskProvider, siteId));
/// <summary>Disposes all cached channels.</summary>
public void Dispose()
src/ZB.MOM.WW.ScadaBridge.AuditLog/ServiceCollectionExtensions.cs
+14 -4
View File
@@ -511,9 +511,14 @@ public static class ServiceCollectionExtensions
var options = sp
.GetService<Microsoft.Extensions.Options.IOptions<
ZB.MOM.WW.ScadaBridge.Communication.CommunicationOptions>>();
// The PSK provider is central-only and optional in DI, so GetService (not
// GetRequiredService): a host without one dials unauthenticated and the site
// refuses it, which is the fail-closed outcome we want rather than a
// resolution crash at composition time.
var psk = sp.GetService<ZB.MOM.WW.ScadaBridge.Communication.Grpc.ISitePskProvider>();
return options is null
? new GrpcPullAuditEventsInvoker()
: new GrpcPullAuditEventsInvoker(options.Value);
? new GrpcPullAuditEventsInvoker(new ZB.MOM.WW.ScadaBridge.Communication.CommunicationOptions(), psk)
: new GrpcPullAuditEventsInvoker(options.Value, psk);
});
services.TryAddSingleton<GrpcPullAuditEventsClient.IPullAuditEventsInvoker>(
sp => sp.GetRequiredService<GrpcPullAuditEventsInvoker>());
@@ -536,9 +541,14 @@ public static class ServiceCollectionExtensions
var options = sp
.GetService<Microsoft.Extensions.Options.IOptions<
ZB.MOM.WW.ScadaBridge.Communication.CommunicationOptions>>();
// The PSK provider is central-only and optional in DI, so GetService (not
// GetRequiredService): a host without one dials unauthenticated and the site
// refuses it, which is the fail-closed outcome we want rather than a
// resolution crash at composition time.
var psk = sp.GetService<ZB.MOM.WW.ScadaBridge.Communication.Grpc.ISitePskProvider>();
return options is null
? new GrpcPullSiteCallsInvoker()
: new GrpcPullSiteCallsInvoker(options.Value);
? new GrpcPullSiteCallsInvoker(new ZB.MOM.WW.ScadaBridge.Communication.CommunicationOptions(), psk)
: new GrpcPullSiteCallsInvoker(options.Value, psk);
});
services.TryAddSingleton<GrpcPullSiteCallsClient.IPullSiteCallsInvoker>(
sp => sp.GetRequiredService<GrpcPullSiteCallsInvoker>());
src/ZB.MOM.WW.ScadaBridge.AuditLog/Site/Telemetry/SiteAuditTelemetryActor.cs
+23 -114
View File
@@ -261,18 +261,6 @@ public class SiteAuditTelemetryActor : ReceiveActor
var batch = new CachedTelemetryBatch();
var emittedEventIds = new List<Guid>(pending.Count);
// Rows whose operational half can never be built. They are marked
// Forwarded so they LEAVE this queue — see AbandonUnresolvableAsync
// for why that is safe and why leaving them Pending is not.
var abandonedEventIds = new List<Guid>();
// Counted, not logged per row: a whole batch can fail identically
// (tracking store down, tracking retention elapsed), and one line
// per row per drain is what turned this into a log flood.
var deferredNoSnapshot = 0;
var lookupFailures = 0;
Exception? firstLookupFailure = null;
var graceSeconds = Math.Max(0, _options.CachedTrackingGraceSeconds);
var abandonBefore = DateTime.UtcNow - TimeSpan.FromSeconds(graceSeconds);
foreach (var auditRow in pending)
{
@@ -280,9 +268,13 @@ public class SiteAuditTelemetryActor : ReceiveActor
{
// CorrelationId carries the TrackedOperationId for cached
// rows — see CachedCallLifecycleBridge.BuildPacket. Without
// it there is nothing to look up, ever, so this row is
// abandoned immediately rather than after the grace period.
abandonedEventIds.Add(auditRow.EventId);
// it we can't look up the tracking row; log + skip so the
// bad row doesn't block the rest of the batch. The audit
// row stays Pending (still not in emittedEventIds) and
// central reconciliation will pick it up.
_logger.LogWarning(
"Cached-telemetry drain: audit row {EventId} ({Action}) has no CorrelationId; skipping.",
auditRow.EventId, auditRow.Action);
continue;
}
@@ -296,32 +288,24 @@ public class SiteAuditTelemetryActor : ReceiveActor
catch (Exception ex)
{
// A tracking-store throw must NOT abort the rest of the
// batch — the audit half is best-effort. A throw is a
// STORE fault (locked, corrupt) rather than a verdict about
// this row, so the row is never abandoned on this path: it
// stays Pending and retries once the store recovers.
lookupFailures++;
firstLookupFailure ??= ex;
// batch — the audit half is best-effort. Log and skip
// this row; it stays Pending for the next drain.
_logger.LogWarning(ex,
"Cached-telemetry drain: tracking lookup threw for {EventId} (TrackedOperationId {Tid}, sqlite {SqliteError}); skipping.",
auditRow.EventId, auditRow.CorrelationId, SqliteErrorCodes.Describe(ex));
continue;
}
if (snapshot is null)
{
// No tracking row. Within the grace window this is the
// ordinary write race (the audit row landed first), so
// retry. Past it the snapshot is gone for good — tracking
// retention elapsed, or the two stores were reset
// independently — and retrying forever would wedge the
// queue behind these rows.
if (auditRow.OccurredAtUtc <= abandonBefore)
{
abandonedEventIds.Add(auditRow.EventId);
}
else
{
deferredNoSnapshot++;
}
// No tracking row — possible if the audit row is older
// than the tracking retention window, or the tracking
// store was reset. The audit half remains valid and will
// be picked up by central reconciliation; skip the
// combined push for this row.
_logger.LogWarning(
"Cached-telemetry drain: no tracking snapshot for {EventId} (TrackedOperationId {Tid}); skipping.",
auditRow.EventId, auditRow.CorrelationId);
continue;
}
@@ -330,33 +314,11 @@ public class SiteAuditTelemetryActor : ReceiveActor
emittedEventIds.Add(auditRow.EventId);
}
if (lookupFailures > 0)
{
_logger.LogWarning(firstLookupFailure,
"Cached-telemetry drain: tracking lookup failed for {Count} of {Total} rows " +
"(sqlite {SqliteError}); they stay Pending and retry next drain. First failure attached.",
lookupFailures, pending.Count, SqliteErrorCodes.Describe(firstLookupFailure!));
}
if (deferredNoSnapshot > 0)
{
_logger.LogDebug(
"Cached-telemetry drain: {Count} row(s) have no tracking snapshot yet and are inside the " +
"{Grace}s grace window; retrying next drain.",
deferredNoSnapshot, graceSeconds);
}
if (abandonedEventIds.Count > 0)
{
await AbandonUnresolvableAsync(abandonedEventIds, graceSeconds, ct)
.ConfigureAwait(false);
}
if (batch.Packets.Count == 0)
{
// Nothing resolvable in this read. Any permanently-unresolvable
// rows have just been marked Forwarded above, so the next drain
// sees past them rather than re-reading the same head of queue.
// Every row in this read was skipped (no CorrelationId / no
// tracking snapshot). Leave them Pending and try again next
// drain — the underlying race normally resolves on its own.
return;
}
@@ -396,59 +358,6 @@ public class SiteAuditTelemetryActor : ReceiveActor
}
}
/// <summary>
/// Marks cached rows whose operational half can never be built as
/// Forwarded, so they leave the cached-drain queue.
/// </summary>
/// <remarks>
/// <para>
/// <b>Why they must leave.</b> Leaving an unresolvable row Pending — the
/// previous behaviour — is not a harmless skip. The queue is read
/// oldest-first with a fixed <c>BatchSize</c>, so once a batch's worth of
/// unresolvable rows collects at the head, every drain re-reads exactly
/// those rows, fails identically, and never sees the newer rows behind
/// them. The cached-telemetry path stalls permanently, and each pass logs
/// once per row (measured at ~2 800 warnings/minute on a rig).
/// </para>
/// <para>
/// <b>Why Forwarded is the honest state.</b> Its role in this state machine
/// is "no longer owed by the drain, still eligible for reconciliation" —
/// which is exactly the situation here. The audit half is NOT dropped:
/// <c>ReadPendingSinceAsync</c> covers Forwarded rows as well as Pending
/// ones and central dedups on EventId, so the reconciliation pull still
/// delivers them. What is genuinely lost is the operational
/// (<c>SiteCalls</c>) half — and that is unrecoverable regardless, because
/// the tracking row it would have been built from no longer exists.
/// </para>
/// <para>
/// A failure to mark is swallowed: the rows simply stay Pending and the
/// next drain retries. Escalating here would take down the audit drain over
/// a best-effort cleanup.
/// </para>
/// </remarks>
private async Task AbandonUnresolvableAsync(
IReadOnlyList<Guid> eventIds, int graceSeconds, CancellationToken ct)
{
try
{
await _queue.MarkForwardedAsync(eventIds, ct).ConfigureAwait(false);
_logger.LogWarning(
"Cached-telemetry drain: abandoned the operational half of {Count} row(s) with no " +
"resolvable tracking snapshot after {Grace}s; marked Forwarded so they no longer block " +
"the queue. The audit half still reaches central via the reconciliation pull. This is " +
"expected after tracking retention elapses or the tracking store is reset independently " +
"of the audit store.",
eventIds.Count, graceSeconds);
}
catch (Exception ex)
{
_logger.LogWarning(ex,
"Cached-telemetry drain: could not mark {Count} unresolvable row(s) Forwarded " +
"(sqlite {SqliteError}); they stay Pending and will be retried.",
eventIds.Count, SqliteErrorCodes.Describe(ex));
}
}
private static AuditEventBatch BuildBatch(IReadOnlyList<AuditEvent> events)
{
var batch = new AuditEventBatch();
src/ZB.MOM.WW.ScadaBridge.AuditLog/Site/Telemetry/SiteAuditTelemetryOptions.cs
-29
View File
@@ -25,33 +25,4 @@ public sealed class SiteAuditTelemetryOptions
/// Longer interval avoids hammering an idle SQLite + gRPC channel.
/// </summary>
public int IdleIntervalSeconds { get; set; } = 30;
/// <summary>
/// How long a cached-telemetry audit row may go without a resolvable
/// <c>OperationTracking</c> snapshot before the drain ABANDONS its
/// operational half and marks the row Forwarded. Default: 300 s.
/// </summary>
/// <remarks>
/// <para>
/// A missing tracking snapshot is normally a brief write race — the audit
/// row lands microseconds before the tracking row — so the drain retries
/// within this grace period. Past it the snapshot is almost certainly gone
/// for good (tracking retention elapsed while central was unreachable, or
/// the two stores were reset independently), and retrying forever is
/// actively harmful: the queue is read oldest-first with a fixed
/// <see cref="BatchSize"/>, so a batch's worth of permanently-unresolvable
/// rows at the head STARVES every newer cached row behind them.
/// </para>
/// <para>
/// Abandoning costs only the operational (<c>SiteCalls</c>) half, which is
/// unrecoverable anyway once the tracking row is gone. The audit half is
/// NOT lost: the reconciliation pull covers Forwarded rows as well as
/// Pending ones, and central dedups on EventId.
/// </para>
/// <para>
/// Set to 0 to abandon on the first failed lookup. Negative values are
/// treated as 0.
/// </para>
/// </remarks>
public int CachedTrackingGraceSeconds { get; set; } = 300;
}
src/ZB.MOM.WW.ScadaBridge.CLI/README.md
+4 -2
View File
@@ -1987,9 +1987,11 @@ scadabridge --url <url> cached-call discard --site-id <string> --tracked-operati
## Architecture Notes
The CLI connects to the Central cluster using Akka.NET's `ClusterClient`. It does not join the cluster — it contacts the `ClusterClientReceptionist` on one of the configured Central nodes and sends commands to the `ManagementActor` at path `/user/management`.
The CLI connects to the Central cluster over **HTTP** (`ManagementHttpClient`), posting to the `/management` endpoints at the configured `managementUrl` — normally the Traefik load balancer, which routes to the active central node. It carries HTTP **Basic** credentials from `--username`/`--password`. Central's endpoint handler asks the `ManagementActor` in-process via `ManagementActorHolder`.
The connection is established per-command invocation and torn down cleanly via `CoordinatedShutdown` when the command completes.
There is no Akka dependency in the CLI at all: it does not join the cluster, does not use `ClusterClient`, and does not contact a `ClusterClientReceptionist`. (Earlier revisions of this document described a ClusterClient transport that was never built.)
An `HttpClient` is created per command invocation and disposed when the command completes.
Role enforcement is applied by the ManagementActor on the server side. The CLI authenticates against LDAP using `--username` / `--password`, resolves LDAP group memberships, then maps groups to ScadaBridge roles (Admin, Design, Deployment) via role mappings configured in the security settings. Operations require the appropriate role — for example, creating templates requires `Design`, deploying requires `Deployment`. In the test environment, use the `multi-role` user (password: `password`) which has all three roles.
src/ZB.MOM.WW.ScadaBridge.CentralUI/Components/Health/CentralFailoverControl.razor
+149
View File
@@ -0,0 +1,149 @@
@using Microsoft.AspNetCore.Components.Authorization
@using ZB.MOM.WW.ScadaBridge.CentralUI.Components.Shared
@using ZB.MOM.WW.ScadaBridge.CentralUI.Services
@using ZB.MOM.WW.ScadaBridge.Security
@inject IManualFailoverService Failover
@inject IDialogService Dialog
@* Admin-only manual failover of the central pair (decision 2026-07-22). Rendered on the
Health dashboard's central-cluster card. The page itself is all-roles, so the gate lives
here rather than on the page's [Authorize] attribute. *@
<AuthorizeView Policy="@AuthorizationPolicies.RequireAdmin">
<span class="d-inline-flex align-items-center gap-2">
<button class="btn btn-outline-warning btn-sm"
disabled="@(!HasPeer || _busy)"
title="@(HasPeer
? (IsSite
? $"Gracefully restart site {SiteId}'s active node; its standby takes over."
: "Gracefully restart the active node; the standby takes over.")
: "No standby available — failing over a lone node would be an outage, not a failover.")"
@onclick="TriggerAsync">
@(_busy ? "Failing over…" : "Trigger failover")
</button>
@if (_message is not null)
{
<small class="@(_failed ? "text-danger" : "text-muted")" role="status">@_message</small>
}
</span>
</AuthorizeView>
@code {
/// <summary>
/// Number of central nodes currently reporting online. Fewer than two means there is no
/// standby to take over, so the control is disabled — the guard is enforced again
/// server-side in the failover service, which is the authoritative check.
/// </summary>
[Parameter]
public int OnlineCentralNodeCount { get; set; }
/// <summary>
/// When set, this control fails over the named SITE pair instead of the central pair.
/// Sites are separate Akka clusters reached over the command/control relay, so the site
/// path cannot disturb the admin's own session — which is why the confirmation text
/// differs. <c>null</c> (the default) means the central pair.
/// </summary>
[Parameter]
public string? SiteId { get; set; }
private bool IsSite => !string.IsNullOrWhiteSpace(SiteId);
[CascadingParameter]
private Task<AuthenticationState>? AuthState { get; set; }
private bool HasPeer => OnlineCentralNodeCount >= 2;
private bool _busy;
private bool _failed;
private string? _message;
private async Task TriggerAsync()
{
// Central: the admin is almost certainly connected THROUGH the node about to restart
// (Traefik routes to the active node), so the dialog must set that expectation or a
// working failover reads as a crash they caused.
// Site: a different cluster entirely — this page is unaffected, and claiming otherwise
// would train operators to ignore the warning that does matter.
var confirmed = IsSite
? await Dialog.ConfirmAsync(
$"Trigger failover for site {SiteId}?",
$"The active node of site {SiteId} will leave its cluster and restart; the site's "
+ "standby takes over and its Deployment Manager singleton hands over gracefully. "
+ "In-flight work on that site node is interrupted, and the site is briefly "
+ "unavailable while the handover completes.",
danger: true)
: await Dialog.ConfirmAsync(
"Trigger central failover?",
"The active central node will leave the cluster and restart; the standby takes over "
+ "and becomes active. Cluster singletons hand over gracefully, but in-flight work on "
+ "the active node is interrupted. This page is served by the active node, so it will "
+ "briefly disconnect and reconnect against the new active node.",
danger: true);
if (!confirmed)
{
return;
}
_busy = true;
_failed = false;
_message = null;
try
{
var actor = await ResolveActorAsync();
if (IsSite)
{
var outcome = await Failover.FailOverSiteAsync(SiteId!, actor);
if (outcome.Accepted)
{
_message = $"Failover triggered — {outcome.TargetAddress} is leaving the site cluster.";
}
else
{
// Carry the site's own words through: a peer-guard refusal and an
// unreachable site are different situations for the operator.
_failed = true;
_message = outcome.ErrorMessage ?? "Refused by the site.";
}
}
else
{
var target = await Failover.FailOverCentralAsync(actor);
if (target is null)
{
// The server-side peer guard refused. Never report a failover that did not happen.
_failed = true;
_message = "Refused: no standby available to take over.";
}
else
{
_message = $"Failover triggered — {target} is leaving the cluster.";
}
}
}
catch (Exception ex)
{
_failed = true;
_message = $"Failover failed: {ex.Message}";
}
finally
{
_busy = false;
}
}
/// <summary>Authenticated user name recorded on the audit row.</summary>
private async Task<string> ResolveActorAsync()
{
if (AuthState is null)
{
return "unknown";
}
var state = await AuthState;
return state.User.Identity?.Name
?? state.User.FindFirst(JwtTokenService.UsernameClaimType)?.Value
?? "unknown";
}
}
src/ZB.MOM.WW.ScadaBridge.CentralUI/Components/Pages/Monitoring/Health.razor
+21 -5
View File
@@ -218,11 +218,18 @@
<small class="text-muted ms-2">offline since @changedAt.ToString("u")</small>
}
</div>
<small class="text-muted">
Last report: <TimestampDisplay Value="@state.LastReportReceivedAt" Format="HH:mm:ss" NullText="awaiting first report" />
| Last heartbeat: <TimestampDisplay Value="@state.LastHeartbeatAt" Format="HH:mm:ss" />
| Seq: @state.LastSequenceNumber
</small>
<div class="d-flex align-items-center gap-3">
@* Admin-only; the control gates itself, and disables when the pair has
no online standby to take over. Central acts on the local cluster;
a site is asked over the command/control relay. *@
<CentralFailoverControl OnlineCentralNodeCount="@OnlineNodeCount(state)"
SiteId="@(isCentral ? null : siteId)" />
<small class="text-muted">
Last report: <TimestampDisplay Value="@state.LastReportReceivedAt" Format="HH:mm:ss" NullText="awaiting first report" />
| Last heartbeat: <TimestampDisplay Value="@state.LastHeartbeatAt" Format="HH:mm:ss" />
| Seq: @state.LastSequenceNumber
</small>
</div>
</div>
<div class="card-body p-3">
@if (state.LatestReport != null)
@@ -442,6 +449,15 @@
private string StaleTimeoutDisplay =>
FormatDuration(HealthOptions.Value.MetricsStaleTimeout);
// Online central nodes, from the same ClusterNodes list the Nodes column renders.
// Fewer than two means no standby, which disables the manual-failover control. This is
// a display-side guard only — AkkaManualFailoverService re-checks against live cluster
// membership, which is the authoritative answer.
private static int OnlineNodeCount(SiteHealthState state) =>
state.LatestReport?.ClusterNodes is { Count: > 0 } nodes
? nodes.Count(n => n.IsOnline)
: (state.IsOnline ? 1 : 0);
private static string FormatDuration(TimeSpan span) =>
span.TotalMinutes >= 1 && span == TimeSpan.FromMinutes(Math.Round(span.TotalMinutes))
? $"{span.TotalMinutes:0} minute{(span.TotalMinutes == 1 ? "" : "s")}"
src/ZB.MOM.WW.ScadaBridge.CentralUI/Services/IManualFailoverService.cs
+48
View File
@@ -0,0 +1,48 @@
namespace ZB.MOM.WW.ScadaBridge.CentralUI.Services;
/// <summary>
/// Admin-triggered manual failover of the central pair. Declared here — and in terms of
/// plain strings — so CentralUI stays Akka-free; the Akka implementation lives in the Host
/// (<c>AkkaManualFailoverService</c>) and is registered only in the Central branch.
/// </summary>
public interface IManualFailoverService
{
/// <summary>
/// Gracefully fails over the central cluster: the current active (oldest Up) member
/// leaves, restarts via its supervisor, and rejoins as standby. The Leave is graceful,
/// never a Down, so cluster singletons hand over instead of being killed.
/// <para>
/// The caller is usually connected THROUGH the node being failed over (Traefik routes
/// the UI to the active node), so the calling Blazor circuit should expect to drop and
/// reconnect against the new active node.
/// </para>
/// </summary>
/// <param name="actor">Authenticated user name, recorded on the audit row.</param>
/// <returns>The address acted on, or <c>null</c> when there is no peer to fail over to
/// (failing over a lone node would be an outage, not a failover).</returns>
Task<string?> FailOverCentralAsync(string actor);
/// <summary>
/// Asks a SITE to gracefully fail over its own two-node pair. Central and each site are
/// separate Akka clusters, so this is a request relayed over the command/control channel —
/// the site performs the Leave itself and reports the outcome. Unlike central failover,
/// this does NOT disturb the caller's own UI session.
/// </summary>
/// <param name="siteId">The site whose pair should fail over.</param>
/// <param name="actor">Authenticated user name, recorded on the audit row.</param>
/// <returns>The site's outcome — accepted with a target, or refused with a reason.</returns>
Task<SiteFailoverOutcome> FailOverSiteAsync(string siteId, string actor);
}
/// <summary>
/// Outcome of a site-pair failover request, in UI terms. Deliberately distinct from the wire
/// ack so CentralUI stays free of the Akka message contract.
/// </summary>
/// <param name="Accepted"><c>true</c> when the site issued the graceful Leave.</param>
/// <param name="TargetAddress">Address of the leaving site node; <c>null</c> when refused.</param>
/// <param name="ErrorMessage">
/// Why the request was refused or failed. A refusal (peer guard, misroute) is a definitive
/// answer FROM the site; an unreachable site surfaces here as a timeout message. Both are
/// reported to the operator rather than being flattened into a generic failure.
/// </param>
public sealed record SiteFailoverOutcome(bool Accepted, string? TargetAddress, string? ErrorMessage);
src/ZB.MOM.WW.ScadaBridge.ClusterInfrastructure/ClusterOptions.cs
+40 -10
View File
@@ -31,19 +31,46 @@ public class ClusterOptions
// when the binding sites can be updated in the same commit.
/// <summary>
/// Akka.NET cluster seed nodes. Both nodes are seed nodes — each node lists
/// itself and its partner — so either can start first and form the cluster.
/// Akka.NET cluster seed nodes. Both nodes are seed nodes — each node lists itself and its
/// partner.
/// <para>
/// <b>ORDER IS LOAD-BEARING (decision 2026-07-22): every node must list ITSELF first.</b>
/// Akka runs <c>FirstSeedNodeProcess</c> — the only bootstrap path that can form a NEW
/// cluster when no peer answers <c>InitJoin</c> — exclusively when <c>seed-nodes[0]</c> is
/// this node's own address; any other node runs <c>JoinSeedNodeProcess</c> and retries
/// <c>InitJoin</c> forever. So merely listing both nodes does NOT mean either can start
/// first: a node that lists its partner first can never cold-start while that partner is
/// down (the "registered outage gap", <c>docker/README.md</c>). Self-first ordering closes
/// it using Akka's own protocol, which — unlike an external self-form timer — is part of
/// the join handshake and so cannot mistake an in-flight join for an absent peer.
/// Enforced at boot by <c>StartupValidator</c>.
/// </para>
/// Must contain at least one entry.
/// </summary>
public List<string> SeedNodes { get; set; } = new();
/// <summary>
/// Split-brain resolver strategy. Must be <c>keep-oldest</c> for the two-node
/// clusters ScadaBridge uses: quorum strategies (<c>keep-majority</c>,
/// <c>static-quorum</c>) cannot distinguish a crash from a partition with only
/// two nodes and would shut down the whole cluster.
/// Downing strategy for unreachable members. Two supported values:
/// <list type="bullet">
/// <item><c>auto-down</c> (default, decision 2026-07-21) — availability-first: each
/// side downs the unreachable peer after <see cref="StableAfter"/>, so a hard crash
/// of EITHER node (oldest included) fails over to the survivor. The accepted trade:
/// a true network partition produces two live one-node clusters (dual-active) until
/// an operator restarts one side. Chosen because ScadaBridge pairs run one node per
/// VM with no shared lease infrastructure, and a stalled system is a bigger risk
/// than a rare partition.</item>
/// <item><c>keep-oldest</c> — partition-safe SBR: downs the side without the oldest
/// member. In a TWO-node cluster this makes a crash of the oldest/active node a
/// total outage: Akka's <c>down-if-alone</c> only rescues the survivor when its own
/// side has ≥2 members (verified against Akka.NET 1.5.62 <c>KeepOldest.Decide</c>
/// and live on the docker rig, 2026-07-21).</item>
/// </list>
/// Other SBR strategies are rejected: <c>static-quorum</c> with quorum 1 hits Akka's
/// <c>IsTooManyMembers</c> guard (2 &gt; 2*1-1) and downs ALL on any unreachability;
/// <c>keep-majority</c> just moves the fatal crash from the oldest to the
/// lowest-address node.
/// </summary>
public string SplitBrainResolverStrategy { get; set; } = "keep-oldest";
public string SplitBrainResolverStrategy { get; set; } = "auto-down";
/// <summary>
/// Time the cluster membership must remain stable before the split-brain
@@ -71,9 +98,12 @@ public class ClusterOptions
public int MinNrOfMembers { get; set; } = 1;
/// <summary>
/// The keep-oldest resolver's <c>down-if-alone</c> flag. When <c>true</c> (the
/// design-doc requirement), the oldest node downs itself if it finds it has no
/// other reachable members, rather than running as an isolated single-node cluster.
/// The keep-oldest resolver's <c>down-if-alone</c> flag; only consulted when
/// <see cref="SplitBrainResolverStrategy"/> is <c>keep-oldest</c>. When <c>true</c>,
/// the oldest node downs itself if it finds it has no other reachable members,
/// rather than running as an isolated single-node cluster. Note that in a two-node
/// cluster this does NOT let the younger survivor take over from a crashed oldest —
/// Akka's alone-check requires the surviving side to have ≥2 members.
/// </summary>
public bool DownIfAlone { get; set; } = true;
src/ZB.MOM.WW.ScadaBridge.ClusterInfrastructure/ClusterOptionsValidator.cs
+18 -4
View File
@@ -12,9 +12,18 @@ namespace ZB.MOM.WW.ScadaBridge.ClusterInfrastructure;
/// </summary>
public sealed class ClusterOptionsValidator : OptionsValidatorBase<ClusterOptions>
{
/// <summary>Split-brain resolver strategies safe for ScadaBridge's two-node clusters.</summary>
/// <summary>
/// Downing strategies supported for ScadaBridge's two-node clusters.
/// <c>auto-down</c> (default) survives a crash of either node at the accepted cost
/// of dual-active during a real partition; <c>keep-oldest</c> is partition-safe but
/// cannot survive a crash of the oldest node. Quorum strategies are rejected:
/// <c>static-quorum</c> quorum-size 1 trips Akka's IsTooManyMembers guard (DownAll
/// on any unreachability in a 2-node cluster) and <c>keep-majority</c> keys the
/// fatal crash to the lowest-address node instead of the oldest.
/// </summary>
private static readonly HashSet<string> AllowedStrategies = new(StringComparer.OrdinalIgnoreCase)
{
"auto-down",
"keep-oldest"
};
@@ -37,8 +46,9 @@ public sealed class ClusterOptionsValidator : OptionsValidatorBase<ClusterOption
builder.RequireThat(
!string.IsNullOrWhiteSpace(options.SplitBrainResolverStrategy)
&& AllowedStrategies.Contains(options.SplitBrainResolverStrategy),
$"ClusterOptions.SplitBrainResolverStrategy must be 'keep-oldest' for a two-node cluster; " +
$"'{options.SplitBrainResolverStrategy}' would risk a total cluster shutdown on a partition.");
$"ClusterOptions.SplitBrainResolverStrategy must be 'auto-down' or 'keep-oldest' for a " +
$"two-node cluster; '{options.SplitBrainResolverStrategy}' would risk a total cluster " +
"shutdown on a partition or an unreachability event.");
builder.RequireThat(options.MinNrOfMembers == 1,
$"ClusterOptions.MinNrOfMembers must be 1 (was {options.MinNrOfMembers}); " +
@@ -58,7 +68,11 @@ public sealed class ClusterOptionsValidator : OptionsValidatorBase<ClusterOption
$"FailureDetectionThreshold ({options.FailureDetectionThreshold}); otherwise nodes are " +
"declared unreachable before a heartbeat can arrive.");
builder.RequireThat(options.DownIfAlone,
// DownIfAlone is a keep-oldest knob; under auto-down each side downs the
// unreachable peer regardless, so the flag is inert and any value is fine.
var isKeepOldest = string.Equals(
options.SplitBrainResolverStrategy, "keep-oldest", StringComparison.OrdinalIgnoreCase);
builder.RequireThat(!isKeepOldest || options.DownIfAlone,
"ClusterOptions.DownIfAlone must be true for the keep-oldest resolver "
+ "(Component-ClusterInfrastructure.md → Split-Brain Resolution); with it false the "
+ "oldest node can run as an isolated single-node cluster during a partition while the "
src/ZB.MOM.WW.ScadaBridge.Commons/Messages/RemoteQuery/SiteFailoverRelayMessages.cs
+53
View File
@@ -0,0 +1,53 @@
namespace ZB.MOM.WW.ScadaBridge.Commons.Messages.RemoteQuery;
/// <summary>
/// Central → site relay command: gracefully fail over the owning site's two-node cluster.
/// Sent over the command/control channel when an Administrator clicks "Trigger failover" on a
/// site card in the Central UI Health dashboard.
/// </summary>
/// <remarks>
/// <para>
/// Central and each site are SEPARATE Akka clusters, so central cannot act on a site's
/// membership directly — it can only ask. The site's own <c>SiteCommunicationActor</c> performs
/// the <c>Cluster.Leave</c> against its site-specific <c>site-{SiteId}</c> role, which is the
/// role its singletons (the Deployment Manager) are scoped to.
/// </para>
/// <para>
/// Either site node may receive this: <c>SiteCommunicationActor</c> is registered per node
/// (not as a singleton), and ClusterClient contact rotation reaches whichever answers. That is
/// fine — <c>Cluster.Leave(address)</c> is valid from any member, and the target is resolved
/// from cluster state rather than from who received the message.
/// </para>
/// <para>
/// <b>Rolling upgrade:</b> a site running a binary older than this contract has no handler for
/// this message; it becomes an unhandled message / dead letter and central's Ask times out.
/// The timeout is reported to the operator as "site did not respond", which is the correct
/// user-facing outcome — an old site genuinely cannot honour the request. Message evolution
/// stays additive-only.
/// </para>
/// </remarks>
/// <param name="CorrelationId">Correlation id echoed on the ack.</param>
/// <param name="SiteId">The site whose pair should fail over. Carried explicitly so a
/// misrouted command is detectable at the site rather than silently acted on.</param>
public sealed record TriggerSiteFailover(
string CorrelationId,
string SiteId);
/// <summary>
/// Site → central ack for a <see cref="TriggerSiteFailover"/> relay command.
/// </summary>
/// <param name="CorrelationId">Correlation id of the originating relay command.</param>
/// <param name="Accepted">
/// <c>true</c> when the site resolved a target and issued the graceful Leave.
/// <c>false</c> is a definitive refusal from the site — most often the peer guard (fewer than
/// two Up members in the site role, so a failover would be an outage) or a site-id mismatch.
/// A <c>false</c> ack is NOT a transport failure and must be distinguished from an
/// unreachable-site timeout.
/// </param>
/// <param name="TargetAddress">Address of the node that is leaving; <c>null</c> when refused.</param>
/// <param name="ErrorMessage">Reason for a refusal, or a fault message; <c>null</c> on success.</param>
public sealed record SiteFailoverAck(
string CorrelationId,
bool Accepted,
string? TargetAddress,
string? ErrorMessage);
src/ZB.MOM.WW.ScadaBridge.Commons/Types/Enums/AuditChannel.cs
+12 -2
View File
@@ -3,7 +3,8 @@ namespace ZB.MOM.WW.ScadaBridge.Commons.Types.Enums;
/// <summary>
/// Top-level Audit Log channel — the trust boundary the audited action crosses.
/// One of: outbound API call, outbound DB write, notification send/deliver, inbound API request,
/// or a two-person ("secured") write through its submit/approve/reject/execute lifecycle.
/// a two-person ("secured") write through its submit/approve/reject/execute lifecycle, or an
/// operator-initiated cluster topology action.
/// </summary>
public enum AuditChannel
{
@@ -11,5 +12,14 @@ public enum AuditChannel
DbOutbound,
Notification,
ApiInbound,
SecuredWrite
SecuredWrite,
/// <summary>
/// An operator-initiated change to cluster topology — currently only the admin-triggered
/// manual failover of the central pair. Distinct from the script trust boundary the other
/// channels describe: nothing here crosses into user script, but a human deliberately
/// restarted the active node, which is exactly the kind of act an audit log exists to
/// attribute. (decision 2026-07-22)
/// </summary>
Cluster
}
src/ZB.MOM.WW.ScadaBridge.Commons/Types/Enums/AuditKind.cs
+10 -1
View File
@@ -40,5 +40,14 @@ public enum AuditKind
/// <c>EventId</c>, source site, and final error) so the loss is queryable in
/// the Audit Log itself, not only in a rotating log file.
/// </summary>
ReconciliationAbandoned
ReconciliationAbandoned,
/// <summary>
/// An administrator triggered a manual failover of the central pair from the Health page:
/// the active (oldest Up) node was asked to leave the cluster gracefully so its singletons
/// hand over and the standby takes over. One row per invocation, written BEFORE the Leave
/// is issued (the acting node is usually not the one that goes away, but the audit must
/// survive either outcome). (decision 2026-07-22)
/// </summary>
ManualFailover
}
src/ZB.MOM.WW.ScadaBridge.Communication/Actors/AkkaCentralTransport.cs
+185
View File
@@ -0,0 +1,185 @@
using Akka.Actor;
using Akka.Cluster.Tools.Client;
using Akka.Event;
using ZB.MOM.WW.ScadaBridge.Commons.Messages.Audit;
using ZB.MOM.WW.ScadaBridge.Commons.Messages.Deployment;
using ZB.MOM.WW.ScadaBridge.Commons.Messages.Health;
using ZB.MOM.WW.ScadaBridge.Commons.Messages.Notification;
namespace ZB.MOM.WW.ScadaBridge.Communication.Actors;
/// <summary>
/// The <see cref="ICentralTransport"/> that carries the seven site→central sends over Akka
/// <c>ClusterClient</c> — the transport in production today, and the default. Every method is a
/// verbatim lift of the corresponding <c>SiteCommunicationActor</c> send block: it forwards a
/// <see cref="ClusterClient.Send"/> to <c>/user/central-communication</c> with the
/// <paramref name="replyTo"/> as the send's sender, so central's reply routes straight back to the
/// waiting Ask rather than through the site communication actor.
/// </summary>
/// <remarks>
/// The <c>ClusterClient</c> reference arrives after construction via
/// <see cref="SetCentralClient"/> (the actor forwards the <c>RegisterCentralClient</c> message it
/// receives once the Host builds the client). Until then — and if central contact points are not
/// configured at all — the client is null and each method answers the same transient-failure reply
/// the old inline handlers did.
/// </remarks>
public sealed class AkkaCentralTransport : ICentralTransport
{
/// <summary>The receptionist-registered path of the central communication actor.</summary>
private const string CentralPath = "/user/central-communication";
private readonly ILoggingAdapter? _log;
private IActorRef? _centralClient;
/// <summary>Creates the transport with no logging adapter (behaviourally identical; warnings are dropped).</summary>
public AkkaCentralTransport()
{
}
/// <summary>Creates the transport bound to the site communication actor's logging adapter.</summary>
/// <param name="log">Logging adapter used for the "no ClusterClient registered" warnings.</param>
public AkkaCentralTransport(ILoggingAdapter log)
{
_log = log;
}
/// <summary>
/// Registers the central <c>ClusterClient</c> once the Host has built it. Called from the site
/// communication actor's <c>RegisterCentralClient</c> handler.
/// </summary>
/// <param name="centralClient">The ClusterClient reaching the central cluster.</param>
public void SetCentralClient(IActorRef centralClient)
{
_centralClient = centralClient;
_log?.Info("Registered central ClusterClient");
}
/// <inheritdoc />
public void SubmitNotification(NotificationSubmit message, IActorRef replyTo)
{
if (_centralClient == null)
{
// No ClusterClient registered yet (e.g. central contact points not
// configured, or registration not yet completed). A non-accepted ack
// makes the S&F forwarder treat this as transient and retry later.
_log?.Warning(
"Cannot forward NotificationSubmit {0} — no central ClusterClient registered",
message.NotificationId);
replyTo.Tell(new NotificationSubmitAck(
message.NotificationId, Accepted: false, Error: "Central ClusterClient not registered"));
return;
}
_log?.Debug("Forwarding NotificationSubmit {0} to central", message.NotificationId);
_centralClient.Tell(new ClusterClient.Send(CentralPath, message), replyTo);
}
/// <inheritdoc />
public void QueryNotificationStatus(NotificationStatusQuery message, IActorRef replyTo)
{
if (_centralClient == null)
{
// No ClusterClient registered yet. Reply Found: false so Notify.Status
// falls back to the site S&F buffer to decide Forwarding vs Unknown.
_log?.Warning(
"Cannot forward NotificationStatusQuery {0} — no central ClusterClient registered",
message.NotificationId);
replyTo.Tell(new NotificationStatusResponse(
message.CorrelationId, Found: false, Status: "Unknown",
RetryCount: 0, LastError: null, DeliveredAt: null));
return;
}
_log?.Debug("Forwarding NotificationStatusQuery {0} to central", message.NotificationId);
_centralClient.Tell(new ClusterClient.Send(CentralPath, message), replyTo);
}
/// <inheritdoc />
public void IngestAuditEvents(IngestAuditEventsCommand message, IActorRef replyTo)
{
if (_centralClient == null)
{
// No ClusterClient registered yet. Faulting the Ask makes the
// SiteAuditTelemetryActor drain loop treat this as transient and keep
// the rows Pending for the next tick.
_log?.Warning(
"Cannot forward IngestAuditEventsCommand ({0} events) — no central ClusterClient registered",
message.Events.Count);
replyTo.Tell(new Status.Failure(
new InvalidOperationException("Central ClusterClient not registered")));
return;
}
_log?.Debug("Forwarding IngestAuditEventsCommand ({0} events) to central", message.Events.Count);
_centralClient.Tell(new ClusterClient.Send(CentralPath, message), replyTo);
}
/// <inheritdoc />
public void IngestCachedTelemetry(IngestCachedTelemetryCommand message, IActorRef replyTo)
{
if (_centralClient == null)
{
_log?.Warning(
"Cannot forward IngestCachedTelemetryCommand ({0} entries) — no central ClusterClient registered",
message.Entries.Count);
replyTo.Tell(new Status.Failure(
new InvalidOperationException("Central ClusterClient not registered")));
return;
}
_log?.Debug("Forwarding IngestCachedTelemetryCommand ({0} entries) to central", message.Entries.Count);
_centralClient.Tell(new ClusterClient.Send(CentralPath, message), replyTo);
}
/// <inheritdoc />
public void ReconcileSite(ReconcileSiteRequest message, IActorRef replyTo)
{
if (_centralClient == null)
{
// No ClusterClient registered yet. Faulting the Ask makes the
// SiteReconciliationActor treat the pass as best-effort-failed; it
// logs a warning and retries reconcile on the next node startup.
_log?.Warning(
"Cannot forward ReconcileSiteRequest for site {0} node {1} — no central ClusterClient registered",
message.SiteIdentifier, message.NodeId);
replyTo.Tell(new Status.Failure(
new InvalidOperationException("Central ClusterClient not registered")));
return;
}
_log?.Debug(
"Forwarding ReconcileSiteRequest for site {0} node {1} ({2} local instance(s)) to central",
message.SiteIdentifier, message.NodeId, message.LocalNameToRevisionHash.Count);
_centralClient.Tell(new ClusterClient.Send(CentralPath, message), replyTo);
}
/// <inheritdoc />
public void ReportSiteHealth(SiteHealthReport message, IActorRef replyTo)
{
if (_centralClient == null)
{
// No ClusterClient registered yet. A non-accepted ack makes the
// sender's counter-restore path treat this tick as a loss.
_log?.Warning(
"Cannot forward SiteHealthReport #{0} — no central ClusterClient registered",
message.SequenceNumber);
replyTo.Tell(new SiteHealthReportAck(
message.SiteId, message.SequenceNumber, Accepted: false,
Error: "Central ClusterClient not registered"));
return;
}
_centralClient.Tell(new ClusterClient.Send(CentralPath, message), replyTo);
}
/// <inheritdoc />
public void SendHeartbeat(HeartbeatMessage message, IActorRef self)
{
if (_centralClient == null)
{
return;
}
_centralClient.Tell(new ClusterClient.Send(CentralPath, message), self);
}
}
src/ZB.MOM.WW.ScadaBridge.Communication/Actors/ICentralTransport.cs
+78
View File
@@ -0,0 +1,78 @@
using Akka.Actor;
using ZB.MOM.WW.ScadaBridge.Commons.Messages.Audit;
using ZB.MOM.WW.ScadaBridge.Commons.Messages.Deployment;
using ZB.MOM.WW.ScadaBridge.Commons.Messages.Health;
using ZB.MOM.WW.ScadaBridge.Commons.Messages.Notification;
namespace ZB.MOM.WW.ScadaBridge.Communication.Actors;
/// <summary>
/// The site→central transport seam: one method per the seven messages
/// <see cref="SiteCommunicationActor"/> sends to <c>/user/central-communication</c> today.
/// </summary>
/// <remarks>
/// <para>
/// The actor's receive handlers no longer own the wire plumbing — they capture the current
/// <c>Sender</c> and hand it to the transport as <paramref name="replyTo"/>. Two implementations
/// exist behind the <c>ScadaBridge:Communication:CentralTransport</c> flag: the default
/// <see cref="AkkaCentralTransport"/> (verbatim of the old <c>ClusterClient.Send</c> path,
/// including the exact sender-forwarding that routes central's reply straight back to the waiting
/// Ask) and <see cref="Grpc.GrpcCentralTransport"/> (a gRPC dial of <c>CentralControlService</c>).
/// </para>
/// <para>
/// <b>Reply/fault contract, identical on both transports.</b> Each Ask-returning method (all but
/// the heartbeat) guarantees exactly one reply eventually lands at <paramref name="replyTo"/>:
/// either the real reply type the caller Asks for, or a transient-failure signal. The Akka path
/// sends a not-accepted ack / <see cref="Status.Failure"/> when no ClusterClient is registered;
/// the gRPC path sends <see cref="Status.Failure"/> on any non-OK status (a timeout or an
/// <c>Unavailable</c> that could not be failed over). Both are what the S&amp;F / audit / health
/// layers above the seam already treat as transient — rows stay buffered, counters restore, the
/// pass re-runs.
/// </para>
/// <para>
/// <b>The heartbeat stays fire-and-forget end-to-end.</b> <see cref="SendHeartbeat"/> takes no
/// <c>replyTo</c> and never surfaces a fault: a transport failure is swallowed and logged, exactly
/// as the old <c>Tell</c> dropped it. A failing heartbeat must never fault the site's heartbeat
/// timer path.
/// </para>
/// </remarks>
public interface ICentralTransport
{
/// <summary>Forwards a buffered notification; central replies <see cref="NotificationSubmitAck"/> to <paramref name="replyTo"/>.</summary>
/// <param name="message">The notification submission.</param>
/// <param name="replyTo">The actor (the S&amp;F forwarder's Ask) the ack routes back to.</param>
void SubmitNotification(NotificationSubmit message, IActorRef replyTo);
/// <summary>Forwards a Notify.Status query; central replies <see cref="NotificationStatusResponse"/> to <paramref name="replyTo"/>.</summary>
/// <param name="message">The status query.</param>
/// <param name="replyTo">The actor (the Notify helper's Ask) the response routes back to.</param>
void QueryNotificationStatus(NotificationStatusQuery message, IActorRef replyTo);
/// <summary>Pushes a batch of audit events; central replies <see cref="IngestAuditEventsReply"/> to <paramref name="replyTo"/>.</summary>
/// <param name="message">The audit-event ingest command.</param>
/// <param name="replyTo">The actor (the telemetry drain's Ask) the reply routes back to.</param>
void IngestAuditEvents(IngestAuditEventsCommand message, IActorRef replyTo);
/// <summary>Pushes a batch of combined cached-call telemetry; central replies <see cref="IngestCachedTelemetryReply"/> to <paramref name="replyTo"/>.</summary>
/// <param name="message">The cached-telemetry ingest command.</param>
/// <param name="replyTo">The actor (the telemetry drain's Ask) the reply routes back to.</param>
void IngestCachedTelemetry(IngestCachedTelemetryCommand message, IActorRef replyTo);
/// <summary>Reports a node's startup inventory; central replies <see cref="ReconcileSiteResponse"/> to <paramref name="replyTo"/>.</summary>
/// <param name="message">The reconcile request.</param>
/// <param name="replyTo">The actor (the reconciliation Ask) the response routes back to.</param>
void ReconcileSite(ReconcileSiteRequest message, IActorRef replyTo);
/// <summary>Reports periodic site health; central replies <see cref="SiteHealthReportAck"/> to <paramref name="replyTo"/>.</summary>
/// <param name="message">The health report.</param>
/// <param name="replyTo">The actor (the health transport's Ask) the ack routes back to.</param>
void ReportSiteHealth(SiteHealthReport message, IActorRef replyTo);
/// <summary>
/// Sends an application heartbeat, fire-and-forget. Never replies and never faults; a failure
/// is swallowed and logged.
/// </summary>
/// <param name="message">The heartbeat.</param>
/// <param name="self">The site communication actor, used as the sender on the Akka path (ignored on gRPC).</param>
void SendHeartbeat(HeartbeatMessage message, IActorRef self);
}
src/ZB.MOM.WW.ScadaBridge.Communication/Actors/SiteCommunicationActor.cs
+132 -155
View File
@@ -1,6 +1,5 @@
using Akka.Actor;
using Akka.Cluster;
using Akka.Cluster.Tools.Client;
using Akka.Event;
using ZB.MOM.WW.ScadaBridge.Commons.Messages.Artifacts;
using ZB.MOM.WW.ScadaBridge.Commons.Messages.Audit;
@@ -37,6 +36,7 @@ public class SiteCommunicationActor : ReceiveActor, IWithTimers
/// do not need a real cluster.
/// </summary>
private readonly Func<bool> _isActiveCheck;
private readonly Func<string, string?> _failOverRole;
/// <summary>
/// Reference to the local Deployment Manager singleton proxy.
@@ -44,10 +44,14 @@ public class SiteCommunicationActor : ReceiveActor, IWithTimers
private readonly IActorRef _deploymentManagerProxy;
/// <summary>
/// ClusterClient reference for sending messages to the central cluster.
/// Set via RegisterCentralClient message.
/// The site→central transport. Finalized in <see cref="PreStart"/> to the injected instance,
/// or a default <see cref="AkkaCentralTransport"/> (ClusterClient) when none is supplied — so
/// the seven site→central sends delegate here rather than owning the wire plumbing inline.
/// </summary>
private IActorRef? _centralClient;
private ICentralTransport _transport;
/// <summary>The transport supplied by the Host (null selects the default Akka transport).</summary>
private readonly ICentralTransport? _injectedTransport;
/// <summary>
/// Local actor references for routing specific message patterns.
@@ -72,22 +76,36 @@ public class SiteCommunicationActor : ReceiveActor, IWithTimers
/// pass a stub so they do not need to load Akka.Cluster into the <c>TestKit</c>
/// ActorSystem.
/// </param>
/// <param name="transport">
/// The site→central transport. <c>null</c> (the default, used by every existing test and by
/// the Host's Akka path) selects an <see cref="AkkaCentralTransport"/> over ClusterClient; the
/// Host injects a <see cref="Grpc.GrpcCentralTransport"/> when
/// <c>ScadaBridge:Communication:CentralTransport</c> is <c>Grpc</c>.
/// </param>
public SiteCommunicationActor(
string siteId,
CommunicationOptions options,
IActorRef deploymentManagerProxy,
Func<bool>? isActiveCheck = null)
Func<bool>? isActiveCheck = null,
Func<string, string?>? failOverRole = null,
ICentralTransport? transport = null)
{
_siteId = siteId;
_options = options;
_deploymentManagerProxy = deploymentManagerProxy;
_isActiveCheck = isActiveCheck ?? DefaultIsActiveCheck;
_failOverRole = failOverRole ?? DefaultFailOverRole;
_injectedTransport = transport;
// Finalized in PreStart (where _log is usable for the default transport); assigned here
// too so the field is definitely-assigned for the constructor's Receive closures.
_transport = transport!;
// Registration
// Registration. Feeding the ClusterClient into the transport is a no-op unless the
// default/Akka transport is in use — the gRPC transport dials configured endpoints and
// never receives this message (the Host does not create a ClusterClient for it).
Receive<RegisterCentralClient>(msg =>
{
_centralClient = msg.Client;
_log.Info("Registered central ClusterClient");
(_transport as AkkaCentralTransport)?.SetCentralClient(msg.Client);
});
Receive<RegisterLocalHandler>(HandleRegisterLocalHandler);
@@ -263,157 +281,41 @@ public class SiteCommunicationActor : ReceiveActor, IWithTimers
}
});
// Notification Outbox: forward a buffered notification submitted by the site
// Store-and-Forward Engine to the central cluster. The original Sender (the
// S&F forwarder's Ask) is forwarded as the ClusterClient.Send sender so the
// NotificationSubmitAck routes straight back to the waiting Ask, not here.
Receive<NotificationSubmit>(msg =>
{
if (_centralClient == null)
{
// No ClusterClient registered yet (e.g. central contact points not
// configured, or registration not yet completed). A non-accepted ack
// makes the S&F forwarder treat this as transient and retry later.
_log.Warning(
"Cannot forward NotificationSubmit {0} — no central ClusterClient registered",
msg.NotificationId);
Sender.Tell(new NotificationSubmitAck(
msg.NotificationId, Accepted: false, Error: "Central ClusterClient not registered"));
return;
}
// Central→site manual failover relay. Central and the site are separate clusters,
// so central can only ask — this node performs the graceful Leave locally, scoped to
// the SITE-SPECIFIC role, because that is what site singletons (the Deployment
// Manager) are placed on. Either node may receive this (the actor is per-node, not a
// singleton, and contact rotation picks whichever answers); the target is resolved
// from cluster state, not from who received the message.
Receive<TriggerSiteFailover>(HandleTriggerSiteFailover);
_log.Debug("Forwarding NotificationSubmit {0} to central", msg.NotificationId);
_centralClient.Tell(
new ClusterClient.Send("/user/central-communication", msg), Sender);
});
// The seven site→central sends now delegate to the injected transport (ClusterClient by
// default, gRPC when configured). Each handler captures the current Sender as the reply
// target so central's reply routes straight back to the waiting Ask, not through this
// actor — the exact sender-forwarding the ClusterClient path relied on. The per-message
// "no transport / not-accepted" fallbacks live inside the transport now.
// Notification Outbox: forward a Notify.Status query to the central cluster.
// The original Sender (the Notify helper's Ask) is forwarded as the
// ClusterClient.Send sender so the NotificationStatusResponse routes straight
// back to the waiting Ask, not here.
Receive<NotificationStatusQuery>(msg =>
{
if (_centralClient == null)
{
// No ClusterClient registered yet. Reply Found: false so Notify.Status
// falls back to the site S&F buffer to decide Forwarding vs Unknown.
_log.Warning(
"Cannot forward NotificationStatusQuery {0} — no central ClusterClient registered",
msg.NotificationId);
Sender.Tell(new NotificationStatusResponse(
msg.CorrelationId, Found: false, Status: "Unknown",
RetryCount: 0, LastError: null, DeliveredAt: null));
return;
}
// Notification Outbox: forward a buffered notification (S&F forwarder's Ask → ack back).
Receive<NotificationSubmit>(msg => _transport.SubmitNotification(msg, Sender));
_log.Debug("Forwarding NotificationStatusQuery {0} to central", msg.NotificationId);
_centralClient.Tell(
new ClusterClient.Send("/user/central-communication", msg), Sender);
});
// Notification Outbox: forward a Notify.Status query (Notify helper's Ask → response back).
Receive<NotificationStatusQuery>(msg => _transport.QueryNotificationStatus(msg, Sender));
// Audit Log: forward a batch of site-local audit events to the
// central cluster. The site SiteAuditTelemetryActor drains its SQLite
// Pending queue through the ClusterClientSiteAuditClient, which Asks
// this actor; the original Sender (that Ask) is passed as the
// ClusterClient.Send sender so the IngestAuditEventsReply routes
// straight back to the waiting Ask, not here. Mirrors NotificationSubmit.
Receive<IngestAuditEventsCommand>(msg =>
{
if (_centralClient == null)
{
// No ClusterClient registered yet (e.g. central contact points
// not configured, or registration not yet completed). Faulting
// the Ask makes the SiteAuditTelemetryActor drain loop treat
// this as transient and keep the rows Pending for the next tick.
_log.Warning(
"Cannot forward IngestAuditEventsCommand ({0} events) — no central ClusterClient registered",
msg.Events.Count);
Sender.Tell(new Status.Failure(
new InvalidOperationException("Central ClusterClient not registered")));
return;
}
// Audit Log: forward a batch of site-local audit events (telemetry drain's Ask → reply back).
Receive<IngestAuditEventsCommand>(msg => _transport.IngestAuditEvents(msg, Sender));
_log.Debug("Forwarding IngestAuditEventsCommand ({0} events) to central", msg.Events.Count);
_centralClient.Tell(
new ClusterClient.Send("/user/central-communication", msg), Sender);
});
// Audit Log: forward a batch of combined cached-call telemetry (telemetry drain's Ask → reply back).
Receive<IngestCachedTelemetryCommand>(msg => _transport.IngestCachedTelemetry(msg, Sender));
// Audit Log: forward a batch of combined cached-call telemetry
// packets to the central cluster. Same forward + reply-routing pattern
// as IngestAuditEventsCommand; central replies with an
// IngestCachedTelemetryReply.
Receive<IngestCachedTelemetryCommand>(msg =>
{
if (_centralClient == null)
{
_log.Warning(
"Cannot forward IngestCachedTelemetryCommand ({0} entries) — no central ClusterClient registered",
msg.Entries.Count);
Sender.Tell(new Status.Failure(
new InvalidOperationException("Central ClusterClient not registered")));
return;
}
// Site startup reconciliation: forward the node's local-inventory request (reconcile Ask → response back).
Receive<ReconcileSiteRequest>(msg => _transport.ReconcileSite(msg, Sender));
_log.Debug("Forwarding IngestCachedTelemetryCommand ({0} entries) to central", msg.Entries.Count);
_centralClient.Tell(
new ClusterClient.Send("/user/central-communication", msg), Sender);
});
// Site startup reconciliation: forward the node's local-inventory
// ReconcileSiteRequest to the central cluster. The original Sender (the
// SiteReconciliationActor's Ask) is passed as the ClusterClient.Send sender so
// the ReconcileSiteResponse routes straight back to the waiting Ask, not here.
// Mirrors IngestAuditEventsCommand.
Receive<ReconcileSiteRequest>(msg =>
{
if (_centralClient == null)
{
// No ClusterClient registered yet (e.g. central contact points not
// configured, or registration not yet completed). Faulting the Ask makes
// the SiteReconciliationActor treat the pass as best-effort-failed; it
// logs a warning and retries reconcile on the next node startup.
_log.Warning(
"Cannot forward ReconcileSiteRequest for site {0} node {1} — no central ClusterClient registered",
msg.SiteIdentifier, msg.NodeId);
Sender.Tell(new Status.Failure(
new InvalidOperationException("Central ClusterClient not registered")));
return;
}
_log.Debug(
"Forwarding ReconcileSiteRequest for site {0} node {1} ({2} local instance(s)) to central",
msg.SiteIdentifier, msg.NodeId, msg.LocalNameToRevisionHash.Count);
_centralClient.Tell(
new ClusterClient.Send("/user/central-communication", msg), Sender);
});
// Internal: send heartbeat tick
// Internal: send heartbeat tick.
Receive<SendHeartbeat>(_ => SendHeartbeatToCentral());
// Internal: forward health report to central. The original Sender (the
// AkkaHealthReportTransport's Ask) is forwarded as the ClusterClient.Send
// sender so the central SiteHealthReportAck routes straight back to the
// waiting Ask — making report delivery observable end-to-end (review 01
// [Medium]). Mirrors the NotificationSubmit ack pattern above.
Receive<SiteHealthReport>(msg =>
{
if (_centralClient == null)
{
// No ClusterClient registered yet. A non-accepted ack makes the
// sender's counter-restore path treat this tick as a loss.
_log.Warning(
"Cannot forward SiteHealthReport #{0} — no central ClusterClient registered",
msg.SequenceNumber);
Sender.Tell(new SiteHealthReportAck(
msg.SiteId, msg.SequenceNumber, Accepted: false,
Error: "Central ClusterClient not registered"));
return;
}
_centralClient.Tell(
new ClusterClient.Send("/user/central-communication", msg), Sender);
});
// Internal: forward the periodic health report (health transport's Ask → ack back), so a
// lost report is observable end-to-end and the sender can restore its per-interval counters.
Receive<SiteHealthReport>(msg => _transport.ReportSiteHealth(msg, Sender));
}
/// <inheritdoc />
@@ -432,6 +334,12 @@ public class SiteCommunicationActor : ReceiveActor, IWithTimers
/// <inheritdoc />
protected override void PreStart()
{
// Finalize the transport now that the actor context (and _log) exist. The default Akka
// transport is given this actor's logging adapter so the "no ClusterClient registered"
// warnings are preserved exactly. PreStart always runs before any message, so the Receive
// closures above see a non-null _transport.
_transport = _injectedTransport ?? new AkkaCentralTransport(_log);
_log.Info("SiteCommunicationActor started for site {0}", _siteId);
// Schedule periodic heartbeat to central. Uses the application heartbeat
@@ -467,9 +375,6 @@ public class SiteCommunicationActor : ReceiveActor, IWithTimers
private void SendHeartbeatToCentral()
{
if (_centralClient == null)
return;
var hostname = Environment.MachineName;
// Stamp HeartbeatMessage.IsActive with this node's
@@ -501,8 +406,17 @@ public class SiteCommunicationActor : ReceiveActor, IWithTimers
IsActive: isActive,
DateTimeOffset.UtcNow);
_centralClient.Tell(
new ClusterClient.Send("/user/central-communication", heartbeat), Self);
// Fire-and-forget on both transports: a failure here must never fault the heartbeat timer
// path. Both real transports swallow their own errors; this catch is a belt-and-braces
// guarantee that no transport (including a future one) can turn a heartbeat into a fault.
try
{
_transport.SendHeartbeat(heartbeat, Self);
}
catch (Exception ex)
{
_log.Debug(ex, "Heartbeat send for site {0} failed; swallowed (heartbeats are fire-and-forget)", _siteId);
}
}
/// <summary>
@@ -517,6 +431,69 @@ public class SiteCommunicationActor : ReceiveActor, IWithTimers
private bool DefaultIsActiveCheck() =>
ClusterState.ActiveNodeEvaluator.SelfIsOldestUp(Cluster.Get(Context.System));
/// <summary>
/// Handles a central-initiated site failover. Refuses a command addressed to a different
/// site (a misroute must never silently fail over a site the operator did not select) and
/// refuses when the site pair has no peer to take over. The ack is sent BEFORE the Leave
/// takes effect on the wire, so it still reaches central even when this node is the one
/// leaving.
/// </summary>
private void HandleTriggerSiteFailover(TriggerSiteFailover msg)
{
if (!string.Equals(msg.SiteId, _siteId, StringComparison.Ordinal))
{
_log.Warning(
"Refusing TriggerSiteFailover addressed to site {Requested}; this node serves {Actual}",
msg.SiteId, _siteId);
Sender.Tell(new SiteFailoverAck(
msg.CorrelationId, Accepted: false, TargetAddress: null,
ErrorMessage: $"Command addressed to site '{msg.SiteId}' but this node serves '{_siteId}'."));
return;
}
var role = $"site-{_siteId}";
try
{
var target = _failOverRole(role);
if (target is null)
{
_log.Warning(
"Refusing TriggerSiteFailover for {SiteId}: fewer than 2 Up members in role {Role}, "
+ "so there is no standby to take over", _siteId, role);
Sender.Tell(new SiteFailoverAck(
msg.CorrelationId, Accepted: false, TargetAddress: null,
ErrorMessage: "No standby available — failing over a lone node would be an outage."));
return;
}
_log.Warning(
"Manual failover requested by central for site {SiteId}: {Target} is leaving the "
+ "site cluster gracefully; its singletons hand over to the standby.", _siteId, target);
Sender.Tell(new SiteFailoverAck(msg.CorrelationId, Accepted: true, target, ErrorMessage: null));
}
catch (Exception ex)
{
// A fault here must be reported to the operator, not thrown into supervision —
// restarting the communication actor would drop central's Ask into a timeout and
// lose the reason.
_log.Error(ex, "TriggerSiteFailover for {SiteId} faulted", _siteId);
Sender.Tell(new SiteFailoverAck(
msg.CorrelationId, Accepted: false, TargetAddress: null, ErrorMessage: ex.Message));
}
}
/// <summary>
/// Production failover action: gracefully Leave the oldest Up member carrying
/// <paramref name="role"/>, via the shared <see cref="ClusterState.ClusterFailoverCoordinator"/>
/// so the central and site paths cannot drift. Injected in tests for the same reason
/// <see cref="DefaultIsActiveCheck"/> is — a real Leave needs Akka.Cluster in the
/// ActorSystem, which the TestKit system does not load.
/// </summary>
/// <param name="role">Site-specific role scope.</param>
/// <returns>Address of the leaving node, or null when there is no peer.</returns>
private string? DefaultFailOverRole(string role) =>
ClusterState.ClusterFailoverCoordinator.FailOverOldest(Context.System, role)?.ToString();
// ── Internal messages ──
internal record SendHeartbeat;
src/ZB.MOM.WW.ScadaBridge.Communication/CentralControlGrpc/CentralControl.cs
+6112
View File
File diff suppressed because it is too large Load Diff
src/ZB.MOM.WW.ScadaBridge.Communication/CentralControlGrpc/CentralControlGrpc.cs
+704
View File
@@ -0,0 +1,704 @@
// <auto-generated>
// Generated by the protocol buffer compiler. DO NOT EDIT!
// source: Protos/central_control.proto
// </auto-generated>
#pragma warning disable 0414, 1591, 8981, 0612
#region Designer generated code
using grpc = global::Grpc.Core;
namespace ZB.MOM.WW.ScadaBridge.Communication.Grpc {
/// <summary>
/// Central-hosted control plane (Phase 1A of the ClusterClient→gRPC migration).
///
/// Direction: SITE is the client, CENTRAL is the server — the inverse of
/// SiteStreamService, where central dials the site. That asymmetry is deliberate
/// and mirrors the direction the Akka ClusterClient traffic flows today: these
/// seven calls are exactly the seven messages SiteCommunicationActor sends to
/// /user/central-communication.
///
/// Every call is gated by ControlPlaneAuthInterceptor: `authorization: Bearer &lt;psk>`
/// plus the `x-scadabridge-site` metadata header naming which site's preshared key
/// central must verify against.
/// </summary>
public static partial class CentralControlService
{
static readonly string __ServiceName = "scadabridge.centralcontrol.v1.CentralControlService";
[global::System.CodeDom.Compiler.GeneratedCode("grpc_csharp_plugin", null)]
static void __Helper_SerializeMessage(global::Google.Protobuf.IMessage message, grpc::SerializationContext context)
{
#if !GRPC_DISABLE_PROTOBUF_BUFFER_SERIALIZATION
if (message is global::Google.Protobuf.IBufferMessage)
{
context.SetPayloadLength(message.CalculateSize());
global::Google.Protobuf.MessageExtensions.WriteTo(message, context.GetBufferWriter());
context.Complete();
return;
}
#endif
context.Complete(global::Google.Protobuf.MessageExtensions.ToByteArray(message));
}
[global::System.CodeDom.Compiler.GeneratedCode("grpc_csharp_plugin", null)]
static class __Helper_MessageCache<T>
{
public static readonly bool IsBufferMessage = global::System.Reflection.IntrospectionExtensions.GetTypeInfo(typeof(global::Google.Protobuf.IBufferMessage)).IsAssignableFrom(typeof(T));
}
[global::System.CodeDom.Compiler.GeneratedCode("grpc_csharp_plugin", null)]
static T __Helper_DeserializeMessage<T>(grpc::DeserializationContext context, global::Google.Protobuf.MessageParser<T> parser) where T : global::Google.Protobuf.IMessage<T>
{
#if !GRPC_DISABLE_PROTOBUF_BUFFER_SERIALIZATION
if (__Helper_MessageCache<T>.IsBufferMessage)
{
return parser.ParseFrom(context.PayloadAsReadOnlySequence());
}
#endif
return parser.ParseFrom(context.PayloadAsNewBuffer());
}
[global::System.CodeDom.Compiler.GeneratedCode("grpc_csharp_plugin", null)]
static readonly grpc::Marshaller<global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.NotificationSubmitDto> __Marshaller_scadabridge_centralcontrol_v1_NotificationSubmitDto = grpc::Marshallers.Create(__Helper_SerializeMessage, context => __Helper_DeserializeMessage(context, global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.NotificationSubmitDto.Parser));
[global::System.CodeDom.Compiler.GeneratedCode("grpc_csharp_plugin", null)]
static readonly grpc::Marshaller<global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.NotificationSubmitAckDto> __Marshaller_scadabridge_centralcontrol_v1_NotificationSubmitAckDto = grpc::Marshallers.Create(__Helper_SerializeMessage, context => __Helper_DeserializeMessage(context, global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.NotificationSubmitAckDto.Parser));
[global::System.CodeDom.Compiler.GeneratedCode("grpc_csharp_plugin", null)]
static readonly grpc::Marshaller<global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.NotificationStatusQueryDto> __Marshaller_scadabridge_centralcontrol_v1_NotificationStatusQueryDto = grpc::Marshallers.Create(__Helper_SerializeMessage, context => __Helper_DeserializeMessage(context, global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.NotificationStatusQueryDto.Parser));
[global::System.CodeDom.Compiler.GeneratedCode("grpc_csharp_plugin", null)]
static readonly grpc::Marshaller<global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.NotificationStatusResponseDto> __Marshaller_scadabridge_centralcontrol_v1_NotificationStatusResponseDto = grpc::Marshallers.Create(__Helper_SerializeMessage, context => __Helper_DeserializeMessage(context, global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.NotificationStatusResponseDto.Parser));
[global::System.CodeDom.Compiler.GeneratedCode("grpc_csharp_plugin", null)]
static readonly grpc::Marshaller<global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.AuditEventBatch> __Marshaller_sitestream_AuditEventBatch = grpc::Marshallers.Create(__Helper_SerializeMessage, context => __Helper_DeserializeMessage(context, global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.AuditEventBatch.Parser));
[global::System.CodeDom.Compiler.GeneratedCode("grpc_csharp_plugin", null)]
static readonly grpc::Marshaller<global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.IngestAck> __Marshaller_sitestream_IngestAck = grpc::Marshallers.Create(__Helper_SerializeMessage, context => __Helper_DeserializeMessage(context, global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.IngestAck.Parser));
[global::System.CodeDom.Compiler.GeneratedCode("grpc_csharp_plugin", null)]
static readonly grpc::Marshaller<global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.CachedTelemetryBatch> __Marshaller_sitestream_CachedTelemetryBatch = grpc::Marshallers.Create(__Helper_SerializeMessage, context => __Helper_DeserializeMessage(context, global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.CachedTelemetryBatch.Parser));
[global::System.CodeDom.Compiler.GeneratedCode("grpc_csharp_plugin", null)]
static readonly grpc::Marshaller<global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.ReconcileSiteRequestDto> __Marshaller_scadabridge_centralcontrol_v1_ReconcileSiteRequestDto = grpc::Marshallers.Create(__Helper_SerializeMessage, context => __Helper_DeserializeMessage(context, global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.ReconcileSiteRequestDto.Parser));
[global::System.CodeDom.Compiler.GeneratedCode("grpc_csharp_plugin", null)]
static readonly grpc::Marshaller<global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.ReconcileSiteResponseDto> __Marshaller_scadabridge_centralcontrol_v1_ReconcileSiteResponseDto = grpc::Marshallers.Create(__Helper_SerializeMessage, context => __Helper_DeserializeMessage(context, global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.ReconcileSiteResponseDto.Parser));
[global::System.CodeDom.Compiler.GeneratedCode("grpc_csharp_plugin", null)]
static readonly grpc::Marshaller<global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.SiteHealthReportDto> __Marshaller_scadabridge_centralcontrol_v1_SiteHealthReportDto = grpc::Marshallers.Create(__Helper_SerializeMessage, context => __Helper_DeserializeMessage(context, global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.SiteHealthReportDto.Parser));
[global::System.CodeDom.Compiler.GeneratedCode("grpc_csharp_plugin", null)]
static readonly grpc::Marshaller<global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.SiteHealthReportAckDto> __Marshaller_scadabridge_centralcontrol_v1_SiteHealthReportAckDto = grpc::Marshallers.Create(__Helper_SerializeMessage, context => __Helper_DeserializeMessage(context, global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.SiteHealthReportAckDto.Parser));
[global::System.CodeDom.Compiler.GeneratedCode("grpc_csharp_plugin", null)]
static readonly grpc::Marshaller<global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.HeartbeatDto> __Marshaller_scadabridge_centralcontrol_v1_HeartbeatDto = grpc::Marshallers.Create(__Helper_SerializeMessage, context => __Helper_DeserializeMessage(context, global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.HeartbeatDto.Parser));
[global::System.CodeDom.Compiler.GeneratedCode("grpc_csharp_plugin", null)]
static readonly grpc::Marshaller<global::Google.Protobuf.WellKnownTypes.Empty> __Marshaller_google_protobuf_Empty = grpc::Marshallers.Create(__Helper_SerializeMessage, context => __Helper_DeserializeMessage(context, global::Google.Protobuf.WellKnownTypes.Empty.Parser));
[global::System.CodeDom.Compiler.GeneratedCode("grpc_csharp_plugin", null)]
static readonly grpc::Method<global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.NotificationSubmitDto, global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.NotificationSubmitAckDto> __Method_SubmitNotification = new grpc::Method<global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.NotificationSubmitDto, global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.NotificationSubmitAckDto>(
grpc::MethodType.Unary,
__ServiceName,
"SubmitNotification",
__Marshaller_scadabridge_centralcontrol_v1_NotificationSubmitDto,
__Marshaller_scadabridge_centralcontrol_v1_NotificationSubmitAckDto);
[global::System.CodeDom.Compiler.GeneratedCode("grpc_csharp_plugin", null)]
static readonly grpc::Method<global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.NotificationStatusQueryDto, global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.NotificationStatusResponseDto> __Method_QueryNotificationStatus = new grpc::Method<global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.NotificationStatusQueryDto, global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.NotificationStatusResponseDto>(
grpc::MethodType.Unary,
__ServiceName,
"QueryNotificationStatus",
__Marshaller_scadabridge_centralcontrol_v1_NotificationStatusQueryDto,
__Marshaller_scadabridge_centralcontrol_v1_NotificationStatusResponseDto);
[global::System.CodeDom.Compiler.GeneratedCode("grpc_csharp_plugin", null)]
static readonly grpc::Method<global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.AuditEventBatch, global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.IngestAck> __Method_IngestAuditEvents = new grpc::Method<global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.AuditEventBatch, global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.IngestAck>(
grpc::MethodType.Unary,
__ServiceName,
"IngestAuditEvents",
__Marshaller_sitestream_AuditEventBatch,
__Marshaller_sitestream_IngestAck);
[global::System.CodeDom.Compiler.GeneratedCode("grpc_csharp_plugin", null)]
static readonly grpc::Method<global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.CachedTelemetryBatch, global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.IngestAck> __Method_IngestCachedTelemetry = new grpc::Method<global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.CachedTelemetryBatch, global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.IngestAck>(
grpc::MethodType.Unary,
__ServiceName,
"IngestCachedTelemetry",
__Marshaller_sitestream_CachedTelemetryBatch,
__Marshaller_sitestream_IngestAck);
[global::System.CodeDom.Compiler.GeneratedCode("grpc_csharp_plugin", null)]
static readonly grpc::Method<global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.ReconcileSiteRequestDto, global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.ReconcileSiteResponseDto> __Method_ReconcileSite = new grpc::Method<global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.ReconcileSiteRequestDto, global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.ReconcileSiteResponseDto>(
grpc::MethodType.Unary,
__ServiceName,
"ReconcileSite",
__Marshaller_scadabridge_centralcontrol_v1_ReconcileSiteRequestDto,
__Marshaller_scadabridge_centralcontrol_v1_ReconcileSiteResponseDto);
[global::System.CodeDom.Compiler.GeneratedCode("grpc_csharp_plugin", null)]
static readonly grpc::Method<global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.SiteHealthReportDto, global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.SiteHealthReportAckDto> __Method_ReportSiteHealth = new grpc::Method<global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.SiteHealthReportDto, global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.SiteHealthReportAckDto>(
grpc::MethodType.Unary,
__ServiceName,
"ReportSiteHealth",
__Marshaller_scadabridge_centralcontrol_v1_SiteHealthReportDto,
__Marshaller_scadabridge_centralcontrol_v1_SiteHealthReportAckDto);
[global::System.CodeDom.Compiler.GeneratedCode("grpc_csharp_plugin", null)]
static readonly grpc::Method<global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.HeartbeatDto, global::Google.Protobuf.WellKnownTypes.Empty> __Method_Heartbeat = new grpc::Method<global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.HeartbeatDto, global::Google.Protobuf.WellKnownTypes.Empty>(
grpc::MethodType.Unary,
__ServiceName,
"Heartbeat",
__Marshaller_scadabridge_centralcontrol_v1_HeartbeatDto,
__Marshaller_google_protobuf_Empty);
/// <summary>Service descriptor</summary>
public static global::Google.Protobuf.Reflection.ServiceDescriptor Descriptor
{
get { return global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.CentralControlReflection.Descriptor.Services[0]; }
}
/// <summary>Base class for server-side implementations of CentralControlService</summary>
[grpc::BindServiceMethod(typeof(CentralControlService), "BindService")]
public abstract partial class CentralControlServiceBase
{
/// <summary>
/// Store-and-forward handoff of one notification for central delivery. The
/// ack is idempotent on notification_id — a duplicate submit after a lost ack
/// must not produce a second delivery.
/// </summary>
/// <param name="request">The request received from the client.</param>
/// <param name="context">The context of the server-side call handler being invoked.</param>
/// <returns>The response to send back to the client (wrapped by a task).</returns>
[global::System.CodeDom.Compiler.GeneratedCode("grpc_csharp_plugin", null)]
public virtual global::System.Threading.Tasks.Task<global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.NotificationSubmitAckDto> SubmitNotification(global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.NotificationSubmitDto request, grpc::ServerCallContext context)
{
throw new grpc::RpcException(new grpc::Status(grpc::StatusCode.Unimplemented, ""));
}
/// <summary>
/// Notify.Status(id) round-trip for a notification that has already left the
/// site buffer. `found = false` sends the caller back to the site-local buffer
/// to decide Forwarding vs Unknown.
/// </summary>
/// <param name="request">The request received from the client.</param>
/// <param name="context">The context of the server-side call handler being invoked.</param>
/// <returns>The response to send back to the client (wrapped by a task).</returns>
[global::System.CodeDom.Compiler.GeneratedCode("grpc_csharp_plugin", null)]
public virtual global::System.Threading.Tasks.Task<global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.NotificationStatusResponseDto> QueryNotificationStatus(global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.NotificationStatusQueryDto request, grpc::ServerCallContext context)
{
throw new grpc::RpcException(new grpc::Status(grpc::StatusCode.Unimplemented, ""));
}
/// <summary>
/// Audit Log (#23) push telemetry. Reuses the SiteStreamService messages: the
/// batch a site drains from its SQLite hot path is byte-identical whichever
/// transport carries it.
/// </summary>
/// <param name="request">The request received from the client.</param>
/// <param name="context">The context of the server-side call handler being invoked.</param>
/// <returns>The response to send back to the client (wrapped by a task).</returns>
[global::System.CodeDom.Compiler.GeneratedCode("grpc_csharp_plugin", null)]
public virtual global::System.Threading.Tasks.Task<global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.IngestAck> IngestAuditEvents(global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.AuditEventBatch request, grpc::ServerCallContext context)
{
throw new grpc::RpcException(new grpc::Status(grpc::StatusCode.Unimplemented, ""));
}
/// <summary>
/// Audit Log (#23) M3 combined cached-call telemetry (audit row + SiteCalls
/// operational upsert, written in one central transaction).
/// </summary>
/// <param name="request">The request received from the client.</param>
/// <param name="context">The context of the server-side call handler being invoked.</param>
/// <returns>The response to send back to the client (wrapped by a task).</returns>
[global::System.CodeDom.Compiler.GeneratedCode("grpc_csharp_plugin", null)]
public virtual global::System.Threading.Tasks.Task<global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.IngestAck> IngestCachedTelemetry(global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.CachedTelemetryBatch request, grpc::ServerCallContext context)
{
throw new grpc::RpcException(new grpc::Status(grpc::StatusCode.Unimplemented, ""));
}
/// <summary>
/// Node-startup self-heal: the node's local deployed inventory in, fetch
/// tokens for whatever it is missing or stale out.
/// </summary>
/// <param name="request">The request received from the client.</param>
/// <param name="context">The context of the server-side call handler being invoked.</param>
/// <returns>The response to send back to the client (wrapped by a task).</returns>
[global::System.CodeDom.Compiler.GeneratedCode("grpc_csharp_plugin", null)]
public virtual global::System.Threading.Tasks.Task<global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.ReconcileSiteResponseDto> ReconcileSite(global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.ReconcileSiteRequestDto request, grpc::ServerCallContext context)
{
throw new grpc::RpcException(new grpc::Status(grpc::StatusCode.Unimplemented, ""));
}
/// <summary>
/// Periodic site health report (30 s cadence). The ack makes delivery
/// observable end-to-end so the sender can restore its per-interval counters
/// when a report is lost.
/// </summary>
/// <param name="request">The request received from the client.</param>
/// <param name="context">The context of the server-side call handler being invoked.</param>
/// <returns>The response to send back to the client (wrapped by a task).</returns>
[global::System.CodeDom.Compiler.GeneratedCode("grpc_csharp_plugin", null)]
public virtual global::System.Threading.Tasks.Task<global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.SiteHealthReportAckDto> ReportSiteHealth(global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.SiteHealthReportDto request, grpc::ServerCallContext context)
{
throw new grpc::RpcException(new grpc::Status(grpc::StatusCode.Unimplemented, ""));
}
/// <summary>
/// Application heartbeat. Returns Empty because the message is
/// fire-and-forget: nothing on the site consumes a reply, and a failure here
/// must never surface as a fault on the heartbeat timer path.
/// </summary>
/// <param name="request">The request received from the client.</param>
/// <param name="context">The context of the server-side call handler being invoked.</param>
/// <returns>The response to send back to the client (wrapped by a task).</returns>
[global::System.CodeDom.Compiler.GeneratedCode("grpc_csharp_plugin", null)]
public virtual global::System.Threading.Tasks.Task<global::Google.Protobuf.WellKnownTypes.Empty> Heartbeat(global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.HeartbeatDto request, grpc::ServerCallContext context)
{
throw new grpc::RpcException(new grpc::Status(grpc::StatusCode.Unimplemented, ""));
}
}
/// <summary>Client for CentralControlService</summary>
public partial class CentralControlServiceClient : grpc::ClientBase<CentralControlServiceClient>
{
/// <summary>Creates a new client for CentralControlService</summary>
/// <param name="channel">The channel to use to make remote calls.</param>
[global::System.CodeDom.Compiler.GeneratedCode("grpc_csharp_plugin", null)]
public CentralControlServiceClient(grpc::ChannelBase channel) : base(channel)
{
}
/// <summary>Creates a new client for CentralControlService that uses a custom <c>CallInvoker</c>.</summary>
/// <param name="callInvoker">The callInvoker to use to make remote calls.</param>
[global::System.CodeDom.Compiler.GeneratedCode("grpc_csharp_plugin", null)]
public CentralControlServiceClient(grpc::CallInvoker callInvoker) : base(callInvoker)
{
}
/// <summary>Protected parameterless constructor to allow creation of test doubles.</summary>
[global::System.CodeDom.Compiler.GeneratedCode("grpc_csharp_plugin", null)]
protected CentralControlServiceClient() : base()
{
}
/// <summary>Protected constructor to allow creation of configured clients.</summary>
/// <param name="configuration">The client configuration.</param>
[global::System.CodeDom.Compiler.GeneratedCode("grpc_csharp_plugin", null)]
protected CentralControlServiceClient(ClientBaseConfiguration configuration) : base(configuration)
{
}
/// <summary>
/// Store-and-forward handoff of one notification for central delivery. The
/// ack is idempotent on notification_id — a duplicate submit after a lost ack
/// must not produce a second delivery.
/// </summary>
/// <param name="request">The request to send to the server.</param>
/// <param name="headers">The initial metadata to send with the call. This parameter is optional.</param>
/// <param name="deadline">An optional deadline for the call. The call will be cancelled if deadline is hit.</param>
/// <param name="cancellationToken">An optional token for canceling the call.</param>
/// <returns>The response received from the server.</returns>
[global::System.CodeDom.Compiler.GeneratedCode("grpc_csharp_plugin", null)]
public virtual global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.NotificationSubmitAckDto SubmitNotification(global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.NotificationSubmitDto request, grpc::Metadata headers = null, global::System.DateTime? deadline = null, global::System.Threading.CancellationToken cancellationToken = default(global::System.Threading.CancellationToken))
{
return SubmitNotification(request, new grpc::CallOptions(headers, deadline, cancellationToken));
}
/// <summary>
/// Store-and-forward handoff of one notification for central delivery. The
/// ack is idempotent on notification_id — a duplicate submit after a lost ack
/// must not produce a second delivery.
/// </summary>
/// <param name="request">The request to send to the server.</param>
/// <param name="options">The options for the call.</param>
/// <returns>The response received from the server.</returns>
[global::System.CodeDom.Compiler.GeneratedCode("grpc_csharp_plugin", null)]
public virtual global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.NotificationSubmitAckDto SubmitNotification(global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.NotificationSubmitDto request, grpc::CallOptions options)
{
return CallInvoker.BlockingUnaryCall(__Method_SubmitNotification, null, options, request);
}
/// <summary>
/// Store-and-forward handoff of one notification for central delivery. The
/// ack is idempotent on notification_id — a duplicate submit after a lost ack
/// must not produce a second delivery.
/// </summary>
/// <param name="request">The request to send to the server.</param>
/// <param name="headers">The initial metadata to send with the call. This parameter is optional.</param>
/// <param name="deadline">An optional deadline for the call. The call will be cancelled if deadline is hit.</param>
/// <param name="cancellationToken">An optional token for canceling the call.</param>
/// <returns>The call object.</returns>
[global::System.CodeDom.Compiler.GeneratedCode("grpc_csharp_plugin", null)]
public virtual grpc::AsyncUnaryCall<global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.NotificationSubmitAckDto> SubmitNotificationAsync(global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.NotificationSubmitDto request, grpc::Metadata headers = null, global::System.DateTime? deadline = null, global::System.Threading.CancellationToken cancellationToken = default(global::System.Threading.CancellationToken))
{
return SubmitNotificationAsync(request, new grpc::CallOptions(headers, deadline, cancellationToken));
}
/// <summary>
/// Store-and-forward handoff of one notification for central delivery. The
/// ack is idempotent on notification_id — a duplicate submit after a lost ack
/// must not produce a second delivery.
/// </summary>
/// <param name="request">The request to send to the server.</param>
/// <param name="options">The options for the call.</param>
/// <returns>The call object.</returns>
[global::System.CodeDom.Compiler.GeneratedCode("grpc_csharp_plugin", null)]
public virtual grpc::AsyncUnaryCall<global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.NotificationSubmitAckDto> SubmitNotificationAsync(global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.NotificationSubmitDto request, grpc::CallOptions options)
{
return CallInvoker.AsyncUnaryCall(__Method_SubmitNotification, null, options, request);
}
/// <summary>
/// Notify.Status(id) round-trip for a notification that has already left the
/// site buffer. `found = false` sends the caller back to the site-local buffer
/// to decide Forwarding vs Unknown.
/// </summary>
/// <param name="request">The request to send to the server.</param>
/// <param name="headers">The initial metadata to send with the call. This parameter is optional.</param>
/// <param name="deadline">An optional deadline for the call. The call will be cancelled if deadline is hit.</param>
/// <param name="cancellationToken">An optional token for canceling the call.</param>
/// <returns>The response received from the server.</returns>
[global::System.CodeDom.Compiler.GeneratedCode("grpc_csharp_plugin", null)]
public virtual global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.NotificationStatusResponseDto QueryNotificationStatus(global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.NotificationStatusQueryDto request, grpc::Metadata headers = null, global::System.DateTime? deadline = null, global::System.Threading.CancellationToken cancellationToken = default(global::System.Threading.CancellationToken))
{
return QueryNotificationStatus(request, new grpc::CallOptions(headers, deadline, cancellationToken));
}
/// <summary>
/// Notify.Status(id) round-trip for a notification that has already left the
/// site buffer. `found = false` sends the caller back to the site-local buffer
/// to decide Forwarding vs Unknown.
/// </summary>
/// <param name="request">The request to send to the server.</param>
/// <param name="options">The options for the call.</param>
/// <returns>The response received from the server.</returns>
[global::System.CodeDom.Compiler.GeneratedCode("grpc_csharp_plugin", null)]
public virtual global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.NotificationStatusResponseDto QueryNotificationStatus(global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.NotificationStatusQueryDto request, grpc::CallOptions options)
{
return CallInvoker.BlockingUnaryCall(__Method_QueryNotificationStatus, null, options, request);
}
/// <summary>
/// Notify.Status(id) round-trip for a notification that has already left the
/// site buffer. `found = false` sends the caller back to the site-local buffer
/// to decide Forwarding vs Unknown.
/// </summary>
/// <param name="request">The request to send to the server.</param>
/// <param name="headers">The initial metadata to send with the call. This parameter is optional.</param>
/// <param name="deadline">An optional deadline for the call. The call will be cancelled if deadline is hit.</param>
/// <param name="cancellationToken">An optional token for canceling the call.</param>
/// <returns>The call object.</returns>
[global::System.CodeDom.Compiler.GeneratedCode("grpc_csharp_plugin", null)]
public virtual grpc::AsyncUnaryCall<global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.NotificationStatusResponseDto> QueryNotificationStatusAsync(global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.NotificationStatusQueryDto request, grpc::Metadata headers = null, global::System.DateTime? deadline = null, global::System.Threading.CancellationToken cancellationToken = default(global::System.Threading.CancellationToken))
{
return QueryNotificationStatusAsync(request, new grpc::CallOptions(headers, deadline, cancellationToken));
}
/// <summary>
/// Notify.Status(id) round-trip for a notification that has already left the
/// site buffer. `found = false` sends the caller back to the site-local buffer
/// to decide Forwarding vs Unknown.
/// </summary>
/// <param name="request">The request to send to the server.</param>
/// <param name="options">The options for the call.</param>
/// <returns>The call object.</returns>
[global::System.CodeDom.Compiler.GeneratedCode("grpc_csharp_plugin", null)]
public virtual grpc::AsyncUnaryCall<global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.NotificationStatusResponseDto> QueryNotificationStatusAsync(global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.NotificationStatusQueryDto request, grpc::CallOptions options)
{
return CallInvoker.AsyncUnaryCall(__Method_QueryNotificationStatus, null, options, request);
}
/// <summary>
/// Audit Log (#23) push telemetry. Reuses the SiteStreamService messages: the
/// batch a site drains from its SQLite hot path is byte-identical whichever
/// transport carries it.
/// </summary>
/// <param name="request">The request to send to the server.</param>
/// <param name="headers">The initial metadata to send with the call. This parameter is optional.</param>
/// <param name="deadline">An optional deadline for the call. The call will be cancelled if deadline is hit.</param>
/// <param name="cancellationToken">An optional token for canceling the call.</param>
/// <returns>The response received from the server.</returns>
[global::System.CodeDom.Compiler.GeneratedCode("grpc_csharp_plugin", null)]
public virtual global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.IngestAck IngestAuditEvents(global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.AuditEventBatch request, grpc::Metadata headers = null, global::System.DateTime? deadline = null, global::System.Threading.CancellationToken cancellationToken = default(global::System.Threading.CancellationToken))
{
return IngestAuditEvents(request, new grpc::CallOptions(headers, deadline, cancellationToken));
}
/// <summary>
/// Audit Log (#23) push telemetry. Reuses the SiteStreamService messages: the
/// batch a site drains from its SQLite hot path is byte-identical whichever
/// transport carries it.
/// </summary>
/// <param name="request">The request to send to the server.</param>
/// <param name="options">The options for the call.</param>
/// <returns>The response received from the server.</returns>
[global::System.CodeDom.Compiler.GeneratedCode("grpc_csharp_plugin", null)]
public virtual global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.IngestAck IngestAuditEvents(global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.AuditEventBatch request, grpc::CallOptions options)
{
return CallInvoker.BlockingUnaryCall(__Method_IngestAuditEvents, null, options, request);
}
/// <summary>
/// Audit Log (#23) push telemetry. Reuses the SiteStreamService messages: the
/// batch a site drains from its SQLite hot path is byte-identical whichever
/// transport carries it.
/// </summary>
/// <param name="request">The request to send to the server.</param>
/// <param name="headers">The initial metadata to send with the call. This parameter is optional.</param>
/// <param name="deadline">An optional deadline for the call. The call will be cancelled if deadline is hit.</param>
/// <param name="cancellationToken">An optional token for canceling the call.</param>
/// <returns>The call object.</returns>
[global::System.CodeDom.Compiler.GeneratedCode("grpc_csharp_plugin", null)]
public virtual grpc::AsyncUnaryCall<global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.IngestAck> IngestAuditEventsAsync(global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.AuditEventBatch request, grpc::Metadata headers = null, global::System.DateTime? deadline = null, global::System.Threading.CancellationToken cancellationToken = default(global::System.Threading.CancellationToken))
{
return IngestAuditEventsAsync(request, new grpc::CallOptions(headers, deadline, cancellationToken));
}
/// <summary>
/// Audit Log (#23) push telemetry. Reuses the SiteStreamService messages: the
/// batch a site drains from its SQLite hot path is byte-identical whichever
/// transport carries it.
/// </summary>
/// <param name="request">The request to send to the server.</param>
/// <param name="options">The options for the call.</param>
/// <returns>The call object.</returns>
[global::System.CodeDom.Compiler.GeneratedCode("grpc_csharp_plugin", null)]
public virtual grpc::AsyncUnaryCall<global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.IngestAck> IngestAuditEventsAsync(global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.AuditEventBatch request, grpc::CallOptions options)
{
return CallInvoker.AsyncUnaryCall(__Method_IngestAuditEvents, null, options, request);
}
/// <summary>
/// Audit Log (#23) M3 combined cached-call telemetry (audit row + SiteCalls
/// operational upsert, written in one central transaction).
/// </summary>
/// <param name="request">The request to send to the server.</param>
/// <param name="headers">The initial metadata to send with the call. This parameter is optional.</param>
/// <param name="deadline">An optional deadline for the call. The call will be cancelled if deadline is hit.</param>
/// <param name="cancellationToken">An optional token for canceling the call.</param>
/// <returns>The response received from the server.</returns>
[global::System.CodeDom.Compiler.GeneratedCode("grpc_csharp_plugin", null)]
public virtual global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.IngestAck IngestCachedTelemetry(global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.CachedTelemetryBatch request, grpc::Metadata headers = null, global::System.DateTime? deadline = null, global::System.Threading.CancellationToken cancellationToken = default(global::System.Threading.CancellationToken))
{
return IngestCachedTelemetry(request, new grpc::CallOptions(headers, deadline, cancellationToken));
}
/// <summary>
/// Audit Log (#23) M3 combined cached-call telemetry (audit row + SiteCalls
/// operational upsert, written in one central transaction).
/// </summary>
/// <param name="request">The request to send to the server.</param>
/// <param name="options">The options for the call.</param>
/// <returns>The response received from the server.</returns>
[global::System.CodeDom.Compiler.GeneratedCode("grpc_csharp_plugin", null)]
public virtual global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.IngestAck IngestCachedTelemetry(global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.CachedTelemetryBatch request, grpc::CallOptions options)
{
return CallInvoker.BlockingUnaryCall(__Method_IngestCachedTelemetry, null, options, request);
}
/// <summary>
/// Audit Log (#23) M3 combined cached-call telemetry (audit row + SiteCalls
/// operational upsert, written in one central transaction).
/// </summary>
/// <param name="request">The request to send to the server.</param>
/// <param name="headers">The initial metadata to send with the call. This parameter is optional.</param>
/// <param name="deadline">An optional deadline for the call. The call will be cancelled if deadline is hit.</param>
/// <param name="cancellationToken">An optional token for canceling the call.</param>
/// <returns>The call object.</returns>
[global::System.CodeDom.Compiler.GeneratedCode("grpc_csharp_plugin", null)]
public virtual grpc::AsyncUnaryCall<global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.IngestAck> IngestCachedTelemetryAsync(global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.CachedTelemetryBatch request, grpc::Metadata headers = null, global::System.DateTime? deadline = null, global::System.Threading.CancellationToken cancellationToken = default(global::System.Threading.CancellationToken))
{
return IngestCachedTelemetryAsync(request, new grpc::CallOptions(headers, deadline, cancellationToken));
}
/// <summary>
/// Audit Log (#23) M3 combined cached-call telemetry (audit row + SiteCalls
/// operational upsert, written in one central transaction).
/// </summary>
/// <param name="request">The request to send to the server.</param>
/// <param name="options">The options for the call.</param>
/// <returns>The call object.</returns>
[global::System.CodeDom.Compiler.GeneratedCode("grpc_csharp_plugin", null)]
public virtual grpc::AsyncUnaryCall<global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.IngestAck> IngestCachedTelemetryAsync(global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.CachedTelemetryBatch request, grpc::CallOptions options)
{
return CallInvoker.AsyncUnaryCall(__Method_IngestCachedTelemetry, null, options, request);
}
/// <summary>
/// Node-startup self-heal: the node's local deployed inventory in, fetch
/// tokens for whatever it is missing or stale out.
/// </summary>
/// <param name="request">The request to send to the server.</param>
/// <param name="headers">The initial metadata to send with the call. This parameter is optional.</param>
/// <param name="deadline">An optional deadline for the call. The call will be cancelled if deadline is hit.</param>
/// <param name="cancellationToken">An optional token for canceling the call.</param>
/// <returns>The response received from the server.</returns>
[global::System.CodeDom.Compiler.GeneratedCode("grpc_csharp_plugin", null)]
public virtual global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.ReconcileSiteResponseDto ReconcileSite(global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.ReconcileSiteRequestDto request, grpc::Metadata headers = null, global::System.DateTime? deadline = null, global::System.Threading.CancellationToken cancellationToken = default(global::System.Threading.CancellationToken))
{
return ReconcileSite(request, new grpc::CallOptions(headers, deadline, cancellationToken));
}
/// <summary>
/// Node-startup self-heal: the node's local deployed inventory in, fetch
/// tokens for whatever it is missing or stale out.
/// </summary>
/// <param name="request">The request to send to the server.</param>
/// <param name="options">The options for the call.</param>
/// <returns>The response received from the server.</returns>
[global::System.CodeDom.Compiler.GeneratedCode("grpc_csharp_plugin", null)]
public virtual global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.ReconcileSiteResponseDto ReconcileSite(global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.ReconcileSiteRequestDto request, grpc::CallOptions options)
{
return CallInvoker.BlockingUnaryCall(__Method_ReconcileSite, null, options, request);
}
/// <summary>
/// Node-startup self-heal: the node's local deployed inventory in, fetch
/// tokens for whatever it is missing or stale out.
/// </summary>
/// <param name="request">The request to send to the server.</param>
/// <param name="headers">The initial metadata to send with the call. This parameter is optional.</param>
/// <param name="deadline">An optional deadline for the call. The call will be cancelled if deadline is hit.</param>
/// <param name="cancellationToken">An optional token for canceling the call.</param>
/// <returns>The call object.</returns>
[global::System.CodeDom.Compiler.GeneratedCode("grpc_csharp_plugin", null)]
public virtual grpc::AsyncUnaryCall<global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.ReconcileSiteResponseDto> ReconcileSiteAsync(global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.ReconcileSiteRequestDto request, grpc::Metadata headers = null, global::System.DateTime? deadline = null, global::System.Threading.CancellationToken cancellationToken = default(global::System.Threading.CancellationToken))
{
return ReconcileSiteAsync(request, new grpc::CallOptions(headers, deadline, cancellationToken));
}
/// <summary>
/// Node-startup self-heal: the node's local deployed inventory in, fetch
/// tokens for whatever it is missing or stale out.
/// </summary>
/// <param name="request">The request to send to the server.</param>
/// <param name="options">The options for the call.</param>
/// <returns>The call object.</returns>
[global::System.CodeDom.Compiler.GeneratedCode("grpc_csharp_plugin", null)]
public virtual grpc::AsyncUnaryCall<global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.ReconcileSiteResponseDto> ReconcileSiteAsync(global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.ReconcileSiteRequestDto request, grpc::CallOptions options)
{
return CallInvoker.AsyncUnaryCall(__Method_ReconcileSite, null, options, request);
}
/// <summary>
/// Periodic site health report (30 s cadence). The ack makes delivery
/// observable end-to-end so the sender can restore its per-interval counters
/// when a report is lost.
/// </summary>
/// <param name="request">The request to send to the server.</param>
/// <param name="headers">The initial metadata to send with the call. This parameter is optional.</param>
/// <param name="deadline">An optional deadline for the call. The call will be cancelled if deadline is hit.</param>
/// <param name="cancellationToken">An optional token for canceling the call.</param>
/// <returns>The response received from the server.</returns>
[global::System.CodeDom.Compiler.GeneratedCode("grpc_csharp_plugin", null)]
public virtual global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.SiteHealthReportAckDto ReportSiteHealth(global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.SiteHealthReportDto request, grpc::Metadata headers = null, global::System.DateTime? deadline = null, global::System.Threading.CancellationToken cancellationToken = default(global::System.Threading.CancellationToken))
{
return ReportSiteHealth(request, new grpc::CallOptions(headers, deadline, cancellationToken));
}
/// <summary>
/// Periodic site health report (30 s cadence). The ack makes delivery
/// observable end-to-end so the sender can restore its per-interval counters
/// when a report is lost.
/// </summary>
/// <param name="request">The request to send to the server.</param>
/// <param name="options">The options for the call.</param>
/// <returns>The response received from the server.</returns>
[global::System.CodeDom.Compiler.GeneratedCode("grpc_csharp_plugin", null)]
public virtual global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.SiteHealthReportAckDto ReportSiteHealth(global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.SiteHealthReportDto request, grpc::CallOptions options)
{
return CallInvoker.BlockingUnaryCall(__Method_ReportSiteHealth, null, options, request);
}
/// <summary>
/// Periodic site health report (30 s cadence). The ack makes delivery
/// observable end-to-end so the sender can restore its per-interval counters
/// when a report is lost.
/// </summary>
/// <param name="request">The request to send to the server.</param>
/// <param name="headers">The initial metadata to send with the call. This parameter is optional.</param>
/// <param name="deadline">An optional deadline for the call. The call will be cancelled if deadline is hit.</param>
/// <param name="cancellationToken">An optional token for canceling the call.</param>
/// <returns>The call object.</returns>
[global::System.CodeDom.Compiler.GeneratedCode("grpc_csharp_plugin", null)]
public virtual grpc::AsyncUnaryCall<global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.SiteHealthReportAckDto> ReportSiteHealthAsync(global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.SiteHealthReportDto request, grpc::Metadata headers = null, global::System.DateTime? deadline = null, global::System.Threading.CancellationToken cancellationToken = default(global::System.Threading.CancellationToken))
{
return ReportSiteHealthAsync(request, new grpc::CallOptions(headers, deadline, cancellationToken));
}
/// <summary>
/// Periodic site health report (30 s cadence). The ack makes delivery
/// observable end-to-end so the sender can restore its per-interval counters
/// when a report is lost.
/// </summary>
/// <param name="request">The request to send to the server.</param>
/// <param name="options">The options for the call.</param>
/// <returns>The call object.</returns>
[global::System.CodeDom.Compiler.GeneratedCode("grpc_csharp_plugin", null)]
public virtual grpc::AsyncUnaryCall<global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.SiteHealthReportAckDto> ReportSiteHealthAsync(global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.SiteHealthReportDto request, grpc::CallOptions options)
{
return CallInvoker.AsyncUnaryCall(__Method_ReportSiteHealth, null, options, request);
}
/// <summary>
/// Application heartbeat. Returns Empty because the message is
/// fire-and-forget: nothing on the site consumes a reply, and a failure here
/// must never surface as a fault on the heartbeat timer path.
/// </summary>
/// <param name="request">The request to send to the server.</param>
/// <param name="headers">The initial metadata to send with the call. This parameter is optional.</param>
/// <param name="deadline">An optional deadline for the call. The call will be cancelled if deadline is hit.</param>
/// <param name="cancellationToken">An optional token for canceling the call.</param>
/// <returns>The response received from the server.</returns>
[global::System.CodeDom.Compiler.GeneratedCode("grpc_csharp_plugin", null)]
public virtual global::Google.Protobuf.WellKnownTypes.Empty Heartbeat(global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.HeartbeatDto request, grpc::Metadata headers = null, global::System.DateTime? deadline = null, global::System.Threading.CancellationToken cancellationToken = default(global::System.Threading.CancellationToken))
{
return Heartbeat(request, new grpc::CallOptions(headers, deadline, cancellationToken));
}
/// <summary>
/// Application heartbeat. Returns Empty because the message is
/// fire-and-forget: nothing on the site consumes a reply, and a failure here
/// must never surface as a fault on the heartbeat timer path.
/// </summary>
/// <param name="request">The request to send to the server.</param>
/// <param name="options">The options for the call.</param>
/// <returns>The response received from the server.</returns>
[global::System.CodeDom.Compiler.GeneratedCode("grpc_csharp_plugin", null)]
public virtual global::Google.Protobuf.WellKnownTypes.Empty Heartbeat(global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.HeartbeatDto request, grpc::CallOptions options)
{
return CallInvoker.BlockingUnaryCall(__Method_Heartbeat, null, options, request);
}
/// <summary>
/// Application heartbeat. Returns Empty because the message is
/// fire-and-forget: nothing on the site consumes a reply, and a failure here
/// must never surface as a fault on the heartbeat timer path.
/// </summary>
/// <param name="request">The request to send to the server.</param>
/// <param name="headers">The initial metadata to send with the call. This parameter is optional.</param>
/// <param name="deadline">An optional deadline for the call. The call will be cancelled if deadline is hit.</param>
/// <param name="cancellationToken">An optional token for canceling the call.</param>
/// <returns>The call object.</returns>
[global::System.CodeDom.Compiler.GeneratedCode("grpc_csharp_plugin", null)]
public virtual grpc::AsyncUnaryCall<global::Google.Protobuf.WellKnownTypes.Empty> HeartbeatAsync(global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.HeartbeatDto request, grpc::Metadata headers = null, global::System.DateTime? deadline = null, global::System.Threading.CancellationToken cancellationToken = default(global::System.Threading.CancellationToken))
{
return HeartbeatAsync(request, new grpc::CallOptions(headers, deadline, cancellationToken));
}
/// <summary>
/// Application heartbeat. Returns Empty because the message is
/// fire-and-forget: nothing on the site consumes a reply, and a failure here
/// must never surface as a fault on the heartbeat timer path.
/// </summary>
/// <param name="request">The request to send to the server.</param>
/// <param name="options">The options for the call.</param>
/// <returns>The call object.</returns>
[global::System.CodeDom.Compiler.GeneratedCode("grpc_csharp_plugin", null)]
public virtual grpc::AsyncUnaryCall<global::Google.Protobuf.WellKnownTypes.Empty> HeartbeatAsync(global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.HeartbeatDto request, grpc::CallOptions options)
{
return CallInvoker.AsyncUnaryCall(__Method_Heartbeat, null, options, request);
}
/// <summary>Creates a new instance of client from given <c>ClientBaseConfiguration</c>.</summary>
[global::System.CodeDom.Compiler.GeneratedCode("grpc_csharp_plugin", null)]
protected override CentralControlServiceClient NewInstance(ClientBaseConfiguration configuration)
{
return new CentralControlServiceClient(configuration);
}
}
/// <summary>Creates service definition that can be registered with a server</summary>
/// <param name="serviceImpl">An object implementing the server-side handling logic.</param>
[global::System.CodeDom.Compiler.GeneratedCode("grpc_csharp_plugin", null)]
public static grpc::ServerServiceDefinition BindService(CentralControlServiceBase serviceImpl)
{
return grpc::ServerServiceDefinition.CreateBuilder()
.AddMethod(__Method_SubmitNotification, serviceImpl.SubmitNotification)
.AddMethod(__Method_QueryNotificationStatus, serviceImpl.QueryNotificationStatus)
.AddMethod(__Method_IngestAuditEvents, serviceImpl.IngestAuditEvents)
.AddMethod(__Method_IngestCachedTelemetry, serviceImpl.IngestCachedTelemetry)
.AddMethod(__Method_ReconcileSite, serviceImpl.ReconcileSite)
.AddMethod(__Method_ReportSiteHealth, serviceImpl.ReportSiteHealth)
.AddMethod(__Method_Heartbeat, serviceImpl.Heartbeat).Build();
}
/// <summary>Register service method with a service binder with or without implementation. Useful when customizing the service binding logic.
/// Note: this method is part of an experimental API that can change or be removed without any prior notice.</summary>
/// <param name="serviceBinder">Service methods will be bound by calling <c>AddMethod</c> on this object.</param>
/// <param name="serviceImpl">An object implementing the server-side handling logic.</param>
[global::System.CodeDom.Compiler.GeneratedCode("grpc_csharp_plugin", null)]
public static void BindService(grpc::ServiceBinderBase serviceBinder, CentralControlServiceBase serviceImpl)
{
serviceBinder.AddMethod(__Method_SubmitNotification, serviceImpl == null ? null : new grpc::UnaryServerMethod<global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.NotificationSubmitDto, global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.NotificationSubmitAckDto>(serviceImpl.SubmitNotification));
serviceBinder.AddMethod(__Method_QueryNotificationStatus, serviceImpl == null ? null : new grpc::UnaryServerMethod<global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.NotificationStatusQueryDto, global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.NotificationStatusResponseDto>(serviceImpl.QueryNotificationStatus));
serviceBinder.AddMethod(__Method_IngestAuditEvents, serviceImpl == null ? null : new grpc::UnaryServerMethod<global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.AuditEventBatch, global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.IngestAck>(serviceImpl.IngestAuditEvents));
serviceBinder.AddMethod(__Method_IngestCachedTelemetry, serviceImpl == null ? null : new grpc::UnaryServerMethod<global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.CachedTelemetryBatch, global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.IngestAck>(serviceImpl.IngestCachedTelemetry));
serviceBinder.AddMethod(__Method_ReconcileSite, serviceImpl == null ? null : new grpc::UnaryServerMethod<global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.ReconcileSiteRequestDto, global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.ReconcileSiteResponseDto>(serviceImpl.ReconcileSite));
serviceBinder.AddMethod(__Method_ReportSiteHealth, serviceImpl == null ? null : new grpc::UnaryServerMethod<global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.SiteHealthReportDto, global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.SiteHealthReportAckDto>(serviceImpl.ReportSiteHealth));
serviceBinder.AddMethod(__Method_Heartbeat, serviceImpl == null ? null : new grpc::UnaryServerMethod<global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.HeartbeatDto, global::Google.Protobuf.WellKnownTypes.Empty>(serviceImpl.Heartbeat));
}
}
}
#endregion
src/ZB.MOM.WW.ScadaBridge.Communication/ClusterState/ClusterFailoverCoordinator.cs
+56
View File
@@ -0,0 +1,56 @@
using Akka.Actor;
using Akka.Cluster;
namespace ZB.MOM.WW.ScadaBridge.Communication.ClusterState;
/// <summary>
/// Deliberate, operator-initiated failover of a two-node cluster: the current active member
/// leaves GRACEFULLY so <c>ClusterSingletonManager</c> hands its singletons to the survivor.
///
/// <para>Lives beside <see cref="ActiveNodeEvaluator"/> — the two share one definition of
/// "active" (oldest Up member in a role scope), and they must not drift: the node asked to
/// leave has to be exactly the node hosting the singletons. Placed in Communication rather
/// than Host because BOTH sides need it — the central pair fails over via the Host's
/// <c>AkkaManualFailoverService</c>, and a site pair fails over inside
/// <c>SiteCommunicationActor</c>, which cannot reference Host.</para>
///
/// <para><b>Leave, never Down.</b> A Down skips singleton hand-off and hands the outcome to
/// the downing strategy — the wrong tool for a planned role swap.</para>
/// </summary>
public static class ClusterFailoverCoordinator
{
/// <summary>
/// Asks the oldest Up member carrying <paramref name="role"/> to leave the cluster.
/// Mirrors <see cref="ActiveNodeEvaluator.SelfIsOldestUp"/>'s rule, so the node acted on is
/// the singleton host — never Akka's cluster <i>leader</i>, whose address-ordered definition
/// diverges from singleton placement once the original first node restarts and rejoins.
/// </summary>
/// <param name="system">Actor system whose cluster is acted on.</param>
/// <param name="role">Role scope. Central uses <c>Central</c>; a site pair uses its
/// site-specific <c>site-{SiteId}</c> role, since site singletons are scoped to that role.</param>
/// <param name="dryRun">Resolve and return the target WITHOUT issuing the Leave. Used to
/// name the target in an audit row before acting, and to answer "would this work?".</param>
/// <returns>The address that leaves (or would leave), or <c>null</c> when fewer than two Up
/// members carry the role — failing over a lone node is an outage, not a failover.</returns>
public static Address? FailOverOldest(ActorSystem system, string role, bool dryRun = false)
{
var cluster = Akka.Cluster.Cluster.Get(system);
var withRole = cluster.State.Members
.Where(m => m.Status == MemberStatus.Up && m.HasRole(role))
.OrderBy(m => m, Member.AgeOrdering)
.ToList();
if (withRole.Count < 2)
{
return null;
}
var oldest = withRole[0];
if (!dryRun)
{
cluster.Leave(oldest.Address);
}
return oldest.Address;
}
}
src/ZB.MOM.WW.ScadaBridge.Communication/CommunicationOptions.cs
+81
View File
@@ -1,11 +1,44 @@
namespace ZB.MOM.WW.ScadaBridge.Communication;
/// <summary>
/// Which transport carries the seven site→central control messages. Selected per node by
/// <c>ScadaBridge:Communication:CentralTransport</c>; the migration ships with
/// <see cref="Akka"/> as the default so nothing flips until a node opts in.
/// </summary>
public enum CentralTransportMode
{
/// <summary>Akka <c>ClusterClient</c> — the transport in production today, and the default.</summary>
Akka = 0,
/// <summary>gRPC dial of the central <c>CentralControlService</c> (Phase 1A migration target).</summary>
Grpc = 1,
}
/// <summary>
/// Configuration options for central-site communication, including per-pattern
/// timeouts and transport heartbeat settings.
/// </summary>
public class CommunicationOptions
{
/// <summary>
/// Which transport carries the site→central control messages. Default <see cref="CentralTransportMode.Akka"/>
/// (ClusterClient) — coexistence rule: a node flips to gRPC only by setting this to <c>Grpc</c>,
/// and rollback is flipping it back. Selecting <c>Grpc</c> requires <see cref="CentralGrpcEndpoints"/>.
/// </summary>
public CentralTransportMode CentralTransport { get; set; } = CentralTransportMode.Akka;
/// <summary>
/// Central control-plane gRPC endpoints (preferred first), e.g.
/// <c>["http://scadabridge-central-a:8083", "http://scadabridge-central-b:8083"]</c>. Dialled by
/// <see cref="Grpc.CentralChannelProvider"/> with sticky failover/failback. Required when
/// <see cref="CentralTransport"/> is <see cref="CentralTransportMode.Grpc"/>, ignored otherwise.
/// </summary>
/// <remarks>
/// Sites reach central by container/host name, NOT via Traefik (which is HTTP/1 only; gRPC is
/// h2c on the central node's dedicated <c>CentralGrpcPort</c>).
/// </remarks>
public List<string> CentralGrpcEndpoints { get; set; } = new();
/// <summary>Timeout for deployment commands (typically longest due to apply logic).</summary>
public TimeSpan DeploymentTimeout { get; set; } = TimeSpan.FromMinutes(2);
@@ -41,6 +74,54 @@ public class CommunicationOptions
/// </summary>
public List<string> CentralContactPoints { get; set; } = new();
/// <summary>
/// Preshared key authenticating this node's gRPC control plane — the site↔central
/// boundary. On a site node this is the key its inbound gate
/// (<c>ControlPlaneAuthInterceptor</c>) expects on every <c>SiteStreamService</c> call, and
/// which central must present; central resolves the matching value per site from its own
/// secret store under the name <c>SB-GRPC-PSK-{siteId}</c>.
/// </summary>
/// <remarks>
/// <para>
/// In production this is supplied as <c>${secret:SB-GRPC-PSK-&lt;siteId&gt;}</c> and expanded
/// out of the secrets store before the host is built, so the plaintext never sits in
/// appsettings. Development rigs set a literal, mirroring the LocalDb replication key.
/// </para>
/// <para>
/// <b>Empty means closed, not open.</b> With no key set the interceptor rejects every gated
/// call. This is not optional configuration: a node that ships without a key serves no
/// streams and no audit pulls.
/// </para>
/// <para>
/// Distinct from <c>LocalDb:Replication:ApiKey</c>, which authenticates the pair partner for
/// database replication over the same listener. The two are never shared.
/// </para>
/// </remarks>
public string GrpcPsk { get; set; } = "";
/// <summary>
/// Central-side per-site gRPC preshared keys, keyed by site identifier — the mirror image
/// of <see cref="GrpcPsk"/>, which is the single key a site node expects on its own inbound
/// gate. An entry here takes precedence over the secret store for that site.
/// </summary>
/// <remarks>
/// <para>
/// <b>Why both a config map and a secret store.</b> The store is the primary source and the
/// only one that works for the real case: sites are added at runtime from the Central UI, so
/// their keys cannot be enumerated in configuration at boot, and <c>SitePskProvider</c>
/// resolves <c>SB-GRPC-PSK-{siteId}</c> on demand. This map covers the cases the store
/// cannot or should not: a development rig that runs with no master key and injects every
/// credential as an environment override, and an operator pinning one site's key without
/// touching the store. Values may themselves be <c>${secret:…}</c> references, since a map
/// declared in configuration IS enumerable at boot.
/// </para>
/// <para>
/// Absence is not a fallback to "unauthenticated" in either source — a site with no key in
/// the map and none in the store cannot be dialed at all.
/// </para>
/// </remarks>
public Dictionary<string, string> SitePsks { get; set; } = new();
/// <summary>gRPC keepalive ping interval for streaming connections.</summary>
public TimeSpan GrpcKeepAlivePingDelay { get; set; } = TimeSpan.FromSeconds(15);
src/ZB.MOM.WW.ScadaBridge.Communication/CommunicationOptionsValidator.cs
+13
View File
@@ -66,6 +66,19 @@ public sealed class CommunicationOptionsValidator : OptionsValidatorBase<Communi
builder.RequireThat(options.GrpcMaxConcurrentStreams > 0,
$"ScadaBridge:Communication:GrpcMaxConcurrentStreams must be positive (was {options.GrpcMaxConcurrentStreams}).");
// The gRPC site→central transport needs at least one central endpoint to dial. Only
// enforced when that transport is selected — the default Akka path ignores the list, so a
// node on ClusterClient must not be forced to declare gRPC endpoints it never uses.
if (options.CentralTransport == CentralTransportMode.Grpc)
{
builder.RequireThat(
options.CentralGrpcEndpoints.Count > 0
&& options.CentralGrpcEndpoints.All(e => !string.IsNullOrWhiteSpace(e)),
"ScadaBridge:Communication:CentralGrpcEndpoints must list at least one non-empty "
+ "central gRPC endpoint when CentralTransport is Grpc "
+ $"(was {options.CentralGrpcEndpoints.Count} entr{(options.CentralGrpcEndpoints.Count == 1 ? "y" : "ies")}).");
}
// ── Aggregated live alarm cache (plan #10, Task 6) ───────────────────────
// Linger drives a Timer dueTime; TimeSpan.Zero is valid (stop the aggregator
// immediately when the last viewer leaves), only a negative value is invalid.
src/ZB.MOM.WW.ScadaBridge.Communication/CommunicationService.cs
+28
View File
@@ -809,6 +809,34 @@ public class CommunicationService
return await GetSiteCallAudit().Ask<DiscardSiteCallResponse>(
request, _options.QueryTimeout, cancellationToken);
}
/// <summary>
/// Asks a site to gracefully fail over its own two-node pair (Task 10). Central and each
/// site are SEPARATE Akka clusters, so central cannot act on site membership directly —
/// the site's own <c>SiteCommunicationActor</c> performs the <c>Cluster.Leave</c> against
/// its site-specific role and acks the outcome.
/// <para>
/// A site running a binary older than the <see cref="TriggerSiteFailover"/> contract has no
/// handler for it, so the message dead-letters and this Ask times out. That surfaces to the
/// operator as "site did not respond", which is the honest outcome — an old site genuinely
/// cannot honour the request.
/// </para>
/// </summary>
/// <param name="siteId">Target site.</param>
/// <param name="correlationId">Correlation id echoed on the ack.</param>
/// <param name="cancellationToken">Cancellation token.</param>
/// <returns>The site's ack.</returns>
public async Task<SiteFailoverAck> TriggerSiteFailoverAsync(
string siteId, string correlationId, CancellationToken cancellationToken = default)
{
_logger.LogWarning(
"Relaying TriggerSiteFailover to site {SiteId}, correlationId={CorrelationId}",
siteId, correlationId);
var envelope = new SiteEnvelope(siteId, new TriggerSiteFailover(correlationId, siteId));
return await GetActor().Ask<SiteFailoverAck>(
envelope, _options.QueryTimeout, cancellationToken);
}
}
/// <summary>
src/ZB.MOM.WW.ScadaBridge.Communication/Grpc/CentralChannelProvider.cs
+274
View File
@@ -0,0 +1,274 @@
using Google.Protobuf.WellKnownTypes;
using Grpc.Core;
using Grpc.Net.Client;
using Microsoft.Extensions.Logging;
namespace ZB.MOM.WW.ScadaBridge.Communication.Grpc;
/// <summary>
/// A pair (or more) of gRPC channels to the central cluster's control-plane nodes, with the
/// sticky-failover + background-failback policy of the design (§3.5). The site holds one channel
/// per central endpoint, prefers the first, and stays on it until a call proves it unreachable —
/// only then flipping to the next, and only <see cref="StatusCode.Unavailable"/> / connect
/// failures count (a <see cref="StatusCode.DeadlineExceeded"/> never flips or retries, because the
/// call may have run).
/// </summary>
/// <remarks>
/// <para>
/// <b>Sticky.</b> All calls go to the current channel; a healthy preferred endpoint never
/// ping-pongs. <see cref="ReportUnavailable"/> flips to the next endpoint (round-robin) when the
/// caller sees the current one refuse a connection.
/// </para>
/// <para>
/// <b>Failback.</b> While off the preferred endpoint a background probe (a cheap <c>Heartbeat</c>
/// ping — always answered, even by a not-yet-ready node) re-checks the preferred one. On the first
/// success new calls return to it; in-flight calls finish where they are. The probe is
/// event-driven: it arms on a flip and stops the moment we are back on the preferred endpoint, so
/// a steady healthy pair spends no cycles. Its cadence backs off exponentially — 1 s, doubling,
/// capped at 60 s — while the preferred endpoint stays down, so a genuinely dead node is not
/// probed hard.
/// </para>
/// <para>
/// <b>Auth.</b> Every channel carries the site's own preshared key and its
/// <c>x-scadabridge-site</c> identity via <see cref="ControlPlaneCredentials"/> — the same
/// insecure-h2c call-credentials shape the streaming client uses. The <paramref name="handlerFactory"/>
/// seam lets a test point a channel at an in-process <c>TestServer</c>; production uses a
/// keepalive-configured <see cref="SocketsHttpHandler"/>.
/// </para>
/// </remarks>
public sealed class CentralChannelProvider : IDisposable
{
private static readonly TimeSpan DefaultBackoffBase = TimeSpan.FromSeconds(1);
private static readonly TimeSpan DefaultBackoffCap = TimeSpan.FromSeconds(60);
private static readonly TimeSpan DefaultProbeDeadline = TimeSpan.FromSeconds(5);
private readonly IReadOnlyList<string> _endpoints;
private readonly GrpcChannel[] _channels;
private readonly CentralControlService.CentralControlServiceClient[] _clients;
private readonly ILogger _logger;
private readonly string _siteId;
private readonly TimeSpan _backoffBase;
private readonly TimeSpan _backoffCap;
private readonly TimeSpan _probeDeadline;
private readonly Timer? _failbackTimer;
private readonly object _gate = new();
private volatile int _current; // preferred == 0
private int _consecutiveProbeFailures;
private bool _disposed;
/// <summary>Creates the provider and opens one channel per endpoint.</summary>
/// <param name="endpoints">Central control-plane endpoints, preferred first (index 0). Must be non-empty.</param>
/// <param name="pskProvider">Resolves this site's preshared key (site-side: a single-key provider).</param>
/// <param name="siteId">This site's identity, sent as the <c>x-scadabridge-site</c> header.</param>
/// <param name="options">Communication options supplying gRPC keepalive settings.</param>
/// <param name="logger">Logger for flip/failback diagnostics.</param>
/// <param name="handlerFactory">Test seam: per-endpoint <see cref="HttpMessageHandler"/>; null uses a production socket handler.</param>
/// <param name="probeDeadline">Deadline for a failback probe. Null uses 5 s.</param>
/// <param name="backoffBase">Initial failback-probe backoff. Null uses 1 s.</param>
/// <param name="backoffCap">Maximum failback-probe backoff. Null uses 60 s.</param>
public CentralChannelProvider(
IReadOnlyList<string> endpoints,
ISitePskProvider pskProvider,
string siteId,
CommunicationOptions options,
ILogger logger,
Func<string, HttpMessageHandler>? handlerFactory = null,
TimeSpan? probeDeadline = null,
TimeSpan? backoffBase = null,
TimeSpan? backoffCap = null)
{
ArgumentNullException.ThrowIfNull(endpoints);
ArgumentNullException.ThrowIfNull(pskProvider);
ArgumentException.ThrowIfNullOrWhiteSpace(siteId);
ArgumentNullException.ThrowIfNull(options);
ArgumentNullException.ThrowIfNull(logger);
if (endpoints.Count == 0)
{
throw new ArgumentException("At least one central gRPC endpoint is required.", nameof(endpoints));
}
_endpoints = endpoints;
_logger = logger;
_siteId = siteId;
_backoffBase = backoffBase ?? DefaultBackoffBase;
_backoffCap = backoffCap ?? DefaultBackoffCap;
_probeDeadline = probeDeadline ?? DefaultProbeDeadline;
_channels = new GrpcChannel[endpoints.Count];
_clients = new CentralControlService.CentralControlServiceClient[endpoints.Count];
for (var i = 0; i < endpoints.Count; i++)
{
var channelOptions = new GrpcChannelOptions
{
HttpHandler = handlerFactory?.Invoke(endpoints[i]) ?? new SocketsHttpHandler
{
KeepAlivePingDelay = options.GrpcKeepAlivePingDelay,
KeepAlivePingTimeout = options.GrpcKeepAlivePingTimeout,
KeepAlivePingPolicy = HttpKeepAlivePingPolicy.Always,
EnableMultipleHttp2Connections = true,
},
}.WithSiteCredentials(pskProvider, siteId);
_channels[i] = GrpcChannel.ForAddress(endpoints[i], channelOptions);
_clients[i] = new CentralControlService.CentralControlServiceClient(_channels[i]);
}
// Only a multi-endpoint pair can ever fail over, so a lone endpoint needs no probe.
if (endpoints.Count > 1)
{
_failbackTimer = new Timer(_ => _ = FailbackTickAsync(), null, Timeout.Infinite, Timeout.Infinite);
}
}
/// <summary>The number of endpoints in the pair.</summary>
public int EndpointCount => _endpoints.Count;
/// <summary>The index of the endpoint calls are currently routed to (preferred == 0).</summary>
public int CurrentIndex => _current;
/// <summary>The endpoint address calls are currently routed to.</summary>
public string CurrentEndpoint => _endpoints[_current];
/// <summary>
/// The endpoint index and client calls should use right now. Captured together so a caller can
/// tell <see cref="ReportUnavailable"/> exactly which endpoint failed even if a concurrent flip
/// has already moved <see cref="CurrentIndex"/>.
/// </summary>
/// <returns>The current endpoint index and its client.</returns>
public (int Index, CentralControlService.CentralControlServiceClient Client) Current()
{
var idx = _current;
return (idx, _clients[idx]);
}
/// <summary>
/// Reports that the endpoint at <paramref name="failedIndex"/> refused a connection (an
/// <see cref="StatusCode.Unavailable"/> / connect failure). If it is still the current endpoint
/// and another exists, flips to the next one and — when now off the preferred endpoint — arms
/// the failback probe. Idempotent under a concurrent flip: a stale index is ignored.
/// </summary>
/// <param name="failedIndex">The endpoint index the caller's failed call used.</param>
public void ReportUnavailable(int failedIndex)
{
if (_endpoints.Count < 2)
{
return; // nothing to fail over to
}
lock (_gate)
{
if (_disposed || failedIndex != _current)
{
return; // a concurrent flip already moved us; do not double-flip
}
var next = (failedIndex + 1) % _endpoints.Count;
_current = next;
_logger.LogWarning(
"Central control-plane endpoint {Failed} is unavailable; site {SiteId} failed over to {Next}.",
_endpoints[failedIndex], _siteId, _endpoints[next]);
if (_current != 0)
{
_consecutiveProbeFailures = 0;
ArmFailback(_backoffBase);
}
}
}
private void ArmFailback(TimeSpan due)
{
if (_disposed)
{
return;
}
_failbackTimer?.Change(due, Timeout.InfiniteTimeSpan);
}
private async Task FailbackTickAsync()
{
int currentAtTick = _current;
if (_disposed || currentAtTick == 0)
{
return; // already back on the preferred endpoint (or shutting down)
}
var preferred = _clients[0];
try
{
await preferred.HeartbeatAsync(
new HeartbeatDto
{
SiteId = _siteId,
NodeHostname = "failback-probe",
IsActive = false,
Timestamp = Timestamp.FromDateTimeOffset(DateTimeOffset.UtcNow),
},
deadline: DateTime.UtcNow.Add(_probeDeadline)).ConfigureAwait(false);
// The preferred endpoint answered — return new calls to it.
lock (_gate)
{
if (_disposed)
{
return;
}
_current = 0;
_consecutiveProbeFailures = 0;
}
_logger.LogInformation(
"Central control-plane preferred endpoint {Preferred} is reachable again; site {SiteId} failed back.",
_endpoints[0], _siteId);
}
catch (Exception ex)
{
lock (_gate)
{
if (_disposed || _current == 0)
{
return;
}
_consecutiveProbeFailures++;
var backoff = NextBackoff(_consecutiveProbeFailures);
_logger.LogDebug(ex,
"Failback probe of preferred central endpoint {Preferred} failed; re-probing in {Backoff}.",
_endpoints[0], backoff);
ArmFailback(backoff);
}
}
}
private TimeSpan NextBackoff(int failures)
{
// 1 s, doubling, capped at 60 s. Guard the shift against overflow for a long outage.
var exponent = Math.Min(failures - 1, 20);
var scaled = _backoffBase.Ticks * (1L << exponent);
var cap = _backoffCap.Ticks;
return TimeSpan.FromTicks(scaled >= cap || scaled < 0 ? cap : scaled);
}
/// <inheritdoc />
public void Dispose()
{
lock (_gate)
{
if (_disposed)
{
return;
}
_disposed = true;
}
_failbackTimer?.Dispose();
foreach (var channel in _channels)
{
channel.Dispose();
}
}
}
src/ZB.MOM.WW.ScadaBridge.Communication/Grpc/CentralControlDtoMapper.cs
+764
View File
@@ -0,0 +1,764 @@
using ZB.MOM.WW.ScadaBridge.Commons.Messages.Audit;
using ZB.MOM.WW.ScadaBridge.Commons.Messages.Deployment;
using ZB.MOM.WW.ScadaBridge.Commons.Messages.Health;
using ZB.MOM.WW.ScadaBridge.Commons.Messages.Notification;
using ZB.MOM.WW.ScadaBridge.Commons.Types;
using ZB.MOM.WW.ScadaBridge.Commons.Types.Enums;
using Timestamp = Google.Protobuf.WellKnownTypes.Timestamp;
namespace ZB.MOM.WW.ScadaBridge.Communication.Grpc;
/// <summary>
/// Canonical bridge between the seven in-process messages the site sends to central
/// and the wire format of <c>CentralControlService</c> (<c>Protos/central_control.proto</c>).
/// </summary>
/// <remarks>
/// <para>
/// The seven pairs mirror, one for one, the seven messages
/// <c>SiteCommunicationActor</c> forwards to <c>/user/central-communication</c> over
/// Akka <c>ClusterClient</c> today. Both transports carry the SAME message types
/// end-to-end — central's handlers are untouched by the migration — so this mapper is
/// the only place the two representations meet, and a field that does not survive a
/// round-trip here is a field the gRPC transport silently drops.
/// </para>
/// <para>
/// <b>Ingest is deliberately absent from the message list.</b> <c>IngestAuditEvents</c>
/// and <c>IngestCachedTelemetry</c> reuse the <c>AuditEventBatch</c> /
/// <c>CachedTelemetryBatch</c> / <c>IngestAck</c> messages already defined for the
/// site-hosted <c>SiteStreamService</c>, so the per-row work is delegated to the
/// existing <see cref="AuditEventDtoMapper"/> and <see cref="SiteCallDtoMapper"/>; only
/// the batch/ack envelopes are assembled here.
/// </para>
///
/// <para><b>Conventions, applied uniformly across every method below.</b></para>
/// <list type="bullet">
/// <item>
/// <b>Nullable strings ↔ empty strings.</b> A proto3 scalar string cannot be absent,
/// so a null .NET string is written as <see cref="string.Empty"/> and an empty wire
/// string is read back as <see langword="null"/>. This is the convention already in
/// force on <see cref="AuditEventDtoMapper"/>, and it is why no field on this wire
/// may distinguish "null" from "deliberately empty".
/// </item>
/// <item>
/// <b>Nullable <see cref="Guid"/> ↔ string.</b> Execution ids travel as their "D"
/// string form; the empty string means <see langword="null"/>. A malformed non-empty
/// value throws out of <c>FromDto</c> rather than degrading to null — a corrupt
/// correlation id must not be laundered into "no correlation".
/// </item>
/// <item>
/// <b>Nullable numbers and booleans ↔ protobuf wrapper types.</b>
/// <c>Int32Value</c>/<c>Int64Value</c>/<c>DoubleValue</c>/<c>BoolValue</c> preserve
/// true null. Several health gauges (<c>LocalDbOplogBacklog</c>,
/// <c>LocalDbReplicationConnected</c>) are documented as "null means unknown, and
/// that is NOT the same as zero/false"; collapsing them to a bare scalar would
/// report a broken replication pair as healthy.
/// </item>
/// <item>
/// <b>Nullable collections ↔ wrapper messages.</b> proto3 cannot express presence on
/// a <c>repeated</c> or <c>map</c> field, so the three nullable
/// <see cref="SiteHealthReport"/> collections travel inside single-field wrapper
/// messages (<c>ConnectionEndpointMapDto</c>, <c>TagQualityMapDto</c>,
/// <c>NodeStatusListDto</c>). An absent wrapper is null; a present-but-empty wrapper
/// is an empty collection.
/// </item>
/// <item>
/// <b><see cref="DateTimeOffset"/> normalizes to a UTC instant.</b> A protobuf
/// <c>Timestamp</c> is an instant, not an offset-qualified local time, so the offset
/// component is dropped and the value round-trips with <c>Offset == TimeSpan.Zero</c>.
/// Every producer in this system stamps UTC (the repo-wide invariant; e.g.
/// <c>Notify.Send</c> uses <c>DateTimeOffset.UtcNow</c>), so this is lossless in
/// practice and the instant is preserved regardless.
/// </item>
/// </list>
/// </remarks>
public static class CentralControlDtoMapper
{
// -----------------------------------------------------------------------
// Notification Outbox (#21)
// -----------------------------------------------------------------------
/// <summary>Projects a <see cref="NotificationSubmit"/> onto its wire DTO.</summary>
/// <param name="msg">The notification submission to project.</param>
/// <returns>The wire-format DTO; null strings and null execution ids collapse to empty strings.</returns>
public static NotificationSubmitDto ToDto(NotificationSubmit msg)
{
ArgumentNullException.ThrowIfNull(msg);
return new NotificationSubmitDto
{
NotificationId = msg.NotificationId,
ListName = msg.ListName,
Subject = msg.Subject,
Body = msg.Body,
SourceSiteId = msg.SourceSiteId,
SourceInstanceId = msg.SourceInstanceId ?? string.Empty,
SourceScript = msg.SourceScript ?? string.Empty,
SiteEnqueuedAt = Timestamp.FromDateTimeOffset(msg.SiteEnqueuedAt),
OriginExecutionId = GuidToWire(msg.OriginExecutionId),
OriginParentExecutionId = GuidToWire(msg.OriginParentExecutionId),
SourceNode = msg.SourceNode ?? string.Empty,
};
}
/// <summary>Reconstructs a <see cref="NotificationSubmit"/> from its wire DTO.</summary>
/// <param name="dto">The wire-format DTO to reconstruct.</param>
/// <returns>The in-process message; empty strings rehydrate as null.</returns>
public static NotificationSubmit FromDto(NotificationSubmitDto dto)
{
ArgumentNullException.ThrowIfNull(dto);
return new NotificationSubmit(
NotificationId: dto.NotificationId,
ListName: dto.ListName,
Subject: dto.Subject,
Body: dto.Body,
SourceSiteId: dto.SourceSiteId,
SourceInstanceId: NullIfEmpty(dto.SourceInstanceId),
SourceScript: NullIfEmpty(dto.SourceScript),
SiteEnqueuedAt: dto.SiteEnqueuedAt.ToDateTimeOffset(),
OriginExecutionId: GuidFromWire(dto.OriginExecutionId),
OriginParentExecutionId: GuidFromWire(dto.OriginParentExecutionId),
SourceNode: NullIfEmpty(dto.SourceNode));
}
/// <summary>Projects a <see cref="NotificationSubmitAck"/> onto its wire DTO.</summary>
/// <param name="msg">The ack to project.</param>
/// <returns>The wire-format DTO; a null error collapses to an empty string.</returns>
public static NotificationSubmitAckDto ToDto(NotificationSubmitAck msg)
{
ArgumentNullException.ThrowIfNull(msg);
return new NotificationSubmitAckDto
{
NotificationId = msg.NotificationId,
Accepted = msg.Accepted,
Error = msg.Error ?? string.Empty,
};
}
/// <summary>Reconstructs a <see cref="NotificationSubmitAck"/> from its wire DTO.</summary>
/// <param name="dto">The wire-format DTO to reconstruct.</param>
/// <returns>The in-process ack; an empty error rehydrates as null.</returns>
public static NotificationSubmitAck FromDto(NotificationSubmitAckDto dto)
{
ArgumentNullException.ThrowIfNull(dto);
return new NotificationSubmitAck(
NotificationId: dto.NotificationId,
Accepted: dto.Accepted,
Error: NullIfEmpty(dto.Error));
}
/// <summary>Projects a <see cref="NotificationStatusQuery"/> onto its wire DTO.</summary>
/// <param name="msg">The status query to project.</param>
/// <returns>The wire-format DTO.</returns>
public static NotificationStatusQueryDto ToDto(NotificationStatusQuery msg)
{
ArgumentNullException.ThrowIfNull(msg);
return new NotificationStatusQueryDto
{
CorrelationId = msg.CorrelationId,
NotificationId = msg.NotificationId,
};
}
/// <summary>Reconstructs a <see cref="NotificationStatusQuery"/> from its wire DTO.</summary>
/// <param name="dto">The wire-format DTO to reconstruct.</param>
/// <returns>The in-process query.</returns>
public static NotificationStatusQuery FromDto(NotificationStatusQueryDto dto)
{
ArgumentNullException.ThrowIfNull(dto);
return new NotificationStatusQuery(
CorrelationId: dto.CorrelationId,
NotificationId: dto.NotificationId);
}
/// <summary>Projects a <see cref="NotificationStatusResponse"/> onto its wire DTO.</summary>
/// <param name="msg">The status response to project.</param>
/// <returns>The wire-format DTO; a null delivery timestamp leaves the field unset.</returns>
public static NotificationStatusResponseDto ToDto(NotificationStatusResponse msg)
{
ArgumentNullException.ThrowIfNull(msg);
var dto = new NotificationStatusResponseDto
{
CorrelationId = msg.CorrelationId,
Found = msg.Found,
Status = msg.Status,
RetryCount = msg.RetryCount,
LastError = msg.LastError ?? string.Empty,
};
if (msg.DeliveredAt.HasValue)
{
dto.DeliveredAt = Timestamp.FromDateTimeOffset(msg.DeliveredAt.Value);
}
return dto;
}
/// <summary>Reconstructs a <see cref="NotificationStatusResponse"/> from its wire DTO.</summary>
/// <param name="dto">The wire-format DTO to reconstruct.</param>
/// <returns>The in-process response; an unset delivery timestamp rehydrates as null.</returns>
public static NotificationStatusResponse FromDto(NotificationStatusResponseDto dto)
{
ArgumentNullException.ThrowIfNull(dto);
return new NotificationStatusResponse(
CorrelationId: dto.CorrelationId,
Found: dto.Found,
Status: dto.Status,
RetryCount: dto.RetryCount,
LastError: NullIfEmpty(dto.LastError),
DeliveredAt: dto.DeliveredAt?.ToDateTimeOffset());
}
// -----------------------------------------------------------------------
// Audit Log (#23) ingest — envelopes only; rows go through the existing mappers
// -----------------------------------------------------------------------
/// <summary>
/// Projects an <see cref="IngestAuditEventsCommand"/> onto the shared
/// <see cref="AuditEventBatch"/> wire message.
/// </summary>
/// <remarks>
/// The per-row projection is <see cref="AuditEventDtoMapper.ToDto"/>, which is lossy
/// by design: <c>ForwardState</c> is site-local storage state and <c>IngestedAtUtc</c>
/// is stamped centrally at ingest, so neither travels.
/// </remarks>
/// <param name="cmd">The ingest command to project.</param>
/// <returns>A batch carrying one DTO per audit event, in order.</returns>
public static AuditEventBatch ToDto(IngestAuditEventsCommand cmd)
{
ArgumentNullException.ThrowIfNull(cmd);
var batch = new AuditEventBatch();
foreach (var evt in cmd.Events)
{
batch.Events.Add(AuditEventDtoMapper.ToDto(evt));
}
return batch;
}
/// <summary>
/// Reconstructs an <see cref="IngestAuditEventsCommand"/> from the shared
/// <see cref="AuditEventBatch"/> wire message — the shape central's
/// <c>CentralCommunicationActor</c> already handles.
/// </summary>
/// <param name="batch">The wire batch to reconstruct.</param>
/// <returns>The in-process ingest command.</returns>
public static IngestAuditEventsCommand FromDto(AuditEventBatch batch)
{
ArgumentNullException.ThrowIfNull(batch);
var events = new List<ZB.MOM.WW.Audit.AuditEvent>(batch.Events.Count);
foreach (var dto in batch.Events)
{
events.Add(AuditEventDtoMapper.FromDto(dto));
}
return new IngestAuditEventsCommand(events);
}
/// <summary>
/// Projects an <see cref="IngestCachedTelemetryCommand"/> onto the shared
/// <see cref="CachedTelemetryBatch"/> wire message.
/// </summary>
/// <param name="cmd">The cached-telemetry ingest command to project.</param>
/// <returns>A batch carrying one packet (audit row + operational row) per entry, in order.</returns>
public static CachedTelemetryBatch ToDto(IngestCachedTelemetryCommand cmd)
{
ArgumentNullException.ThrowIfNull(cmd);
var batch = new CachedTelemetryBatch();
foreach (var entry in cmd.Entries)
{
batch.Packets.Add(new CachedTelemetryPacket
{
AuditEvent = AuditEventDtoMapper.ToDto(entry.Audit),
Operational = SiteCallDtoMapper.ToDto(entry.SiteCall),
});
}
return batch;
}
/// <summary>
/// Reconstructs an <see cref="IngestCachedTelemetryCommand"/> from the shared
/// <see cref="CachedTelemetryBatch"/> wire message.
/// </summary>
/// <param name="batch">The wire batch to reconstruct.</param>
/// <returns>The in-process dual-write ingest command.</returns>
public static IngestCachedTelemetryCommand FromDto(CachedTelemetryBatch batch)
{
ArgumentNullException.ThrowIfNull(batch);
var entries = new List<CachedTelemetryEntry>(batch.Packets.Count);
foreach (var packet in batch.Packets)
{
entries.Add(new CachedTelemetryEntry(
AuditEventDtoMapper.FromDto(packet.AuditEvent),
SiteCallDtoMapper.FromDto(packet.Operational)));
}
return new IngestCachedTelemetryCommand(entries);
}
/// <summary>
/// Projects the accepted-id list of an ingest reply onto the shared
/// <see cref="IngestAck"/> wire message. Shared by both ingest RPCs — the two
/// central reply types differ only in which handler produced them.
/// </summary>
/// <param name="acceptedEventIds">Ids central considers durably persisted.</param>
/// <returns>The wire ack carrying the ids in "D" string form, in order.</returns>
public static IngestAck ToIngestAck(IReadOnlyList<Guid> acceptedEventIds)
{
ArgumentNullException.ThrowIfNull(acceptedEventIds);
var ack = new IngestAck();
foreach (var id in acceptedEventIds)
{
ack.AcceptedEventIds.Add(id.ToString());
}
return ack;
}
/// <summary>
/// Reads the accepted-id list back out of an <see cref="IngestAck"/>.
/// </summary>
/// <param name="ack">The wire ack to read.</param>
/// <returns>The accepted event ids, in wire order.</returns>
public static IReadOnlyList<Guid> FromIngestAck(IngestAck ack)
{
ArgumentNullException.ThrowIfNull(ack);
var ids = new List<Guid>(ack.AcceptedEventIds.Count);
foreach (var id in ack.AcceptedEventIds)
{
ids.Add(Guid.Parse(id));
}
return ids;
}
// -----------------------------------------------------------------------
// Startup reconciliation
// -----------------------------------------------------------------------
/// <summary>Projects a <see cref="ReconcileSiteRequest"/> onto its wire DTO.</summary>
/// <param name="msg">The reconcile request to project.</param>
/// <returns>The wire-format DTO carrying the node's local name→revision-hash inventory.</returns>
public static ReconcileSiteRequestDto ToDto(ReconcileSiteRequest msg)
{
ArgumentNullException.ThrowIfNull(msg);
var dto = new ReconcileSiteRequestDto
{
SiteIdentifier = msg.SiteIdentifier,
NodeId = msg.NodeId,
};
foreach (var (name, hash) in msg.LocalNameToRevisionHash)
{
dto.LocalNameToRevisionHash[name] = hash;
}
return dto;
}
/// <summary>Reconstructs a <see cref="ReconcileSiteRequest"/> from its wire DTO.</summary>
/// <param name="dto">The wire-format DTO to reconstruct.</param>
/// <returns>The in-process reconcile request.</returns>
public static ReconcileSiteRequest FromDto(ReconcileSiteRequestDto dto)
{
ArgumentNullException.ThrowIfNull(dto);
return new ReconcileSiteRequest(
SiteIdentifier: dto.SiteIdentifier,
NodeId: dto.NodeId,
LocalNameToRevisionHash: new Dictionary<string, string>(dto.LocalNameToRevisionHash));
}
/// <summary>Projects a <see cref="ReconcileSiteResponse"/> onto its wire DTO.</summary>
/// <param name="msg">The reconcile response to project.</param>
/// <returns>The wire-format DTO carrying the gap items, orphan names and fetch base URL.</returns>
public static ReconcileSiteResponseDto ToDto(ReconcileSiteResponse msg)
{
ArgumentNullException.ThrowIfNull(msg);
var dto = new ReconcileSiteResponseDto
{
CentralFetchBaseUrl = msg.CentralFetchBaseUrl,
};
foreach (var item in msg.Gap)
{
dto.Gap.Add(new ReconcileGapItemDto
{
InstanceUniqueName = item.InstanceUniqueName,
DeploymentId = item.DeploymentId,
RevisionHash = item.RevisionHash,
IsEnabled = item.IsEnabled,
FetchToken = item.FetchToken,
});
}
dto.OrphanNames.AddRange(msg.OrphanNames);
return dto;
}
/// <summary>Reconstructs a <see cref="ReconcileSiteResponse"/> from its wire DTO.</summary>
/// <param name="dto">The wire-format DTO to reconstruct.</param>
/// <returns>The in-process reconcile response.</returns>
public static ReconcileSiteResponse FromDto(ReconcileSiteResponseDto dto)
{
ArgumentNullException.ThrowIfNull(dto);
var gap = new List<ReconcileGapItem>(dto.Gap.Count);
foreach (var item in dto.Gap)
{
gap.Add(new ReconcileGapItem(
InstanceUniqueName: item.InstanceUniqueName,
DeploymentId: item.DeploymentId,
RevisionHash: item.RevisionHash,
IsEnabled: item.IsEnabled,
FetchToken: item.FetchToken));
}
return new ReconcileSiteResponse(
Gap: gap,
OrphanNames: dto.OrphanNames.ToList(),
CentralFetchBaseUrl: dto.CentralFetchBaseUrl);
}
// -----------------------------------------------------------------------
// Health Monitoring (#11)
// -----------------------------------------------------------------------
/// <summary>Projects a <see cref="SiteHealthReport"/> onto its wire DTO.</summary>
/// <remarks>
/// The three nullable collections travel inside wrapper messages so a null stays
/// distinguishable from an empty collection; the nullable gauges travel in protobuf
/// wrapper types for the same reason.
/// </remarks>
/// <param name="msg">The health report to project.</param>
/// <returns>The wire-format DTO.</returns>
public static SiteHealthReportDto ToDto(SiteHealthReport msg)
{
ArgumentNullException.ThrowIfNull(msg);
var dto = new SiteHealthReportDto
{
SiteId = msg.SiteId,
SequenceNumber = msg.SequenceNumber,
ReportTimestamp = Timestamp.FromDateTimeOffset(msg.ReportTimestamp),
ScriptErrorCount = msg.ScriptErrorCount,
AlarmEvaluationErrorCount = msg.AlarmEvaluationErrorCount,
DeadLetterCount = msg.DeadLetterCount,
DeployedInstanceCount = msg.DeployedInstanceCount,
EnabledInstanceCount = msg.EnabledInstanceCount,
DisabledInstanceCount = msg.DisabledInstanceCount,
NodeRole = msg.NodeRole,
NodeHostname = msg.NodeHostname,
ParkedMessageCount = msg.ParkedMessageCount,
SiteAuditWriteFailures = msg.SiteAuditWriteFailures,
AuditRedactionFailure = msg.AuditRedactionFailure,
SiteEventLogWriteFailures = msg.SiteEventLogWriteFailures,
OldestParkedMessageAgeSeconds = msg.OldestParkedMessageAgeSeconds,
ScriptQueueDepth = msg.ScriptQueueDepth,
ScriptBusyThreads = msg.ScriptBusyThreads,
ScriptOldestBusyAgeSeconds = msg.ScriptOldestBusyAgeSeconds,
LocalDbReplicationConnected = msg.LocalDbReplicationConnected,
LocalDbOplogBacklog = msg.LocalDbOplogBacklog,
};
foreach (var (name, health) in msg.DataConnectionStatuses)
{
dto.DataConnectionStatuses[name] = ToDto(health);
}
foreach (var (name, resolution) in msg.TagResolutionCounts)
{
dto.TagResolutionCounts[name] = new TagResolutionStatusDto
{
TotalSubscribed = resolution.TotalSubscribed,
SuccessfullyResolved = resolution.SuccessfullyResolved,
};
}
foreach (var (name, depth) in msg.StoreAndForwardBufferDepths)
{
dto.StoreAndForwardBufferDepths[name] = depth;
}
if (msg.DataConnectionEndpoints is { } endpoints)
{
var wrapper = new ConnectionEndpointMapDto();
foreach (var (name, endpoint) in endpoints)
{
wrapper.Entries[name] = endpoint;
}
dto.DataConnectionEndpoints = wrapper;
}
if (msg.DataConnectionTagQuality is { } tagQuality)
{
var wrapper = new TagQualityMapDto();
foreach (var (name, counts) in tagQuality)
{
wrapper.Entries[name] = new TagQualityCountsDto
{
Good = counts.Good,
Bad = counts.Bad,
Uncertain = counts.Uncertain,
};
}
dto.DataConnectionTagQuality = wrapper;
}
if (msg.ClusterNodes is { } clusterNodes)
{
var wrapper = new NodeStatusListDto();
foreach (var node in clusterNodes)
{
wrapper.Nodes.Add(new NodeStatusDto
{
Hostname = node.Hostname,
IsOnline = node.IsOnline,
Role = node.Role,
});
}
dto.ClusterNodes = wrapper;
}
if (msg.SiteAuditBacklog is { } backlog)
{
var snapshot = new SiteAuditBacklogSnapshotDto
{
PendingCount = backlog.PendingCount,
OnDiskBytes = backlog.OnDiskBytes,
};
if (backlog.OldestPendingUtc.HasValue)
{
snapshot.OldestPendingUtc = Timestamp.FromDateTime(EnsureUtc(backlog.OldestPendingUtc.Value));
}
dto.SiteAuditBacklog = snapshot;
}
return dto;
}
/// <summary>Reconstructs a <see cref="SiteHealthReport"/> from its wire DTO.</summary>
/// <param name="dto">The wire-format DTO to reconstruct.</param>
/// <returns>The in-process health report; absent wrappers rehydrate as null, not as empty.</returns>
public static SiteHealthReport FromDto(SiteHealthReportDto dto)
{
ArgumentNullException.ThrowIfNull(dto);
var statuses = new Dictionary<string, ConnectionHealth>(dto.DataConnectionStatuses.Count);
foreach (var (name, health) in dto.DataConnectionStatuses)
{
statuses[name] = FromDto(health);
}
var resolution = new Dictionary<string, TagResolutionStatus>(dto.TagResolutionCounts.Count);
foreach (var (name, counts) in dto.TagResolutionCounts)
{
resolution[name] = new TagResolutionStatus(counts.TotalSubscribed, counts.SuccessfullyResolved);
}
var bufferDepths = new Dictionary<string, int>(dto.StoreAndForwardBufferDepths);
Dictionary<string, string>? endpoints = null;
if (dto.DataConnectionEndpoints is { } endpointWrapper)
{
endpoints = new Dictionary<string, string>(endpointWrapper.Entries);
}
Dictionary<string, TagQualityCounts>? tagQuality = null;
if (dto.DataConnectionTagQuality is { } tagQualityWrapper)
{
tagQuality = new Dictionary<string, TagQualityCounts>(tagQualityWrapper.Entries.Count);
foreach (var (name, counts) in tagQualityWrapper.Entries)
{
tagQuality[name] = new TagQualityCounts(counts.Good, counts.Bad, counts.Uncertain);
}
}
List<NodeStatus>? clusterNodes = null;
if (dto.ClusterNodes is { } nodeWrapper)
{
clusterNodes = new List<NodeStatus>(nodeWrapper.Nodes.Count);
foreach (var node in nodeWrapper.Nodes)
{
clusterNodes.Add(new NodeStatus(node.Hostname, node.IsOnline, node.Role));
}
}
SiteAuditBacklogSnapshot? backlog = null;
if (dto.SiteAuditBacklog is { } snapshot)
{
backlog = new SiteAuditBacklogSnapshot(
PendingCount: snapshot.PendingCount,
OldestPendingUtc: snapshot.OldestPendingUtc is null
? null
: DateTime.SpecifyKind(snapshot.OldestPendingUtc.ToDateTime(), DateTimeKind.Utc),
OnDiskBytes: snapshot.OnDiskBytes);
}
return new SiteHealthReport(
SiteId: dto.SiteId,
SequenceNumber: dto.SequenceNumber,
ReportTimestamp: dto.ReportTimestamp.ToDateTimeOffset(),
DataConnectionStatuses: statuses,
TagResolutionCounts: resolution,
ScriptErrorCount: dto.ScriptErrorCount,
AlarmEvaluationErrorCount: dto.AlarmEvaluationErrorCount,
StoreAndForwardBufferDepths: bufferDepths,
DeadLetterCount: dto.DeadLetterCount,
DeployedInstanceCount: dto.DeployedInstanceCount,
EnabledInstanceCount: dto.EnabledInstanceCount,
DisabledInstanceCount: dto.DisabledInstanceCount,
NodeRole: dto.NodeRole,
NodeHostname: dto.NodeHostname,
DataConnectionEndpoints: endpoints,
DataConnectionTagQuality: tagQuality,
ParkedMessageCount: dto.ParkedMessageCount,
ClusterNodes: clusterNodes,
SiteAuditWriteFailures: dto.SiteAuditWriteFailures,
AuditRedactionFailure: dto.AuditRedactionFailure,
SiteAuditBacklog: backlog,
SiteEventLogWriteFailures: dto.SiteEventLogWriteFailures,
OldestParkedMessageAgeSeconds: dto.OldestParkedMessageAgeSeconds)
{
// Init-only members: SiteHealthReport surfaces the scheduler and LocalDb
// gauges as init properties rather than positional parameters, so they
// cannot be passed to the constructor above.
ScriptQueueDepth = dto.ScriptQueueDepth,
ScriptBusyThreads = dto.ScriptBusyThreads,
ScriptOldestBusyAgeSeconds = dto.ScriptOldestBusyAgeSeconds,
LocalDbReplicationConnected = dto.LocalDbReplicationConnected,
LocalDbOplogBacklog = dto.LocalDbOplogBacklog,
};
}
/// <summary>Projects a <see cref="SiteHealthReportAck"/> onto its wire DTO.</summary>
/// <param name="msg">The ack to project.</param>
/// <returns>The wire-format DTO; a null error collapses to an empty string.</returns>
public static SiteHealthReportAckDto ToDto(SiteHealthReportAck msg)
{
ArgumentNullException.ThrowIfNull(msg);
return new SiteHealthReportAckDto
{
SiteId = msg.SiteId,
SequenceNumber = msg.SequenceNumber,
Accepted = msg.Accepted,
Error = msg.Error ?? string.Empty,
};
}
/// <summary>Reconstructs a <see cref="SiteHealthReportAck"/> from its wire DTO.</summary>
/// <param name="dto">The wire-format DTO to reconstruct.</param>
/// <returns>The in-process ack; an empty error rehydrates as null.</returns>
public static SiteHealthReportAck FromDto(SiteHealthReportAckDto dto)
{
ArgumentNullException.ThrowIfNull(dto);
return new SiteHealthReportAck(
SiteId: dto.SiteId,
SequenceNumber: dto.SequenceNumber,
Accepted: dto.Accepted,
Error: NullIfEmpty(dto.Error));
}
/// <summary>Projects a <see cref="HeartbeatMessage"/> onto its wire DTO.</summary>
/// <param name="msg">The heartbeat to project.</param>
/// <returns>The wire-format DTO.</returns>
public static HeartbeatDto ToDto(HeartbeatMessage msg)
{
ArgumentNullException.ThrowIfNull(msg);
return new HeartbeatDto
{
SiteId = msg.SiteId,
NodeHostname = msg.NodeHostname,
IsActive = msg.IsActive,
Timestamp = Timestamp.FromDateTimeOffset(msg.Timestamp),
};
}
/// <summary>Reconstructs a <see cref="HeartbeatMessage"/> from its wire DTO.</summary>
/// <param name="dto">The wire-format DTO to reconstruct.</param>
/// <returns>The in-process heartbeat.</returns>
public static HeartbeatMessage FromDto(HeartbeatDto dto)
{
ArgumentNullException.ThrowIfNull(dto);
return new HeartbeatMessage(
SiteId: dto.SiteId,
NodeHostname: dto.NodeHostname,
IsActive: dto.IsActive,
Timestamp: dto.Timestamp.ToDateTimeOffset());
}
/// <summary>Projects a <see cref="ConnectionHealth"/> onto its wire enum.</summary>
/// <param name="health">The connection health to project.</param>
/// <returns>The wire enum value. Never <c>ConnectionHealthUnspecified</c>.</returns>
/// <exception cref="ArgumentOutOfRangeException">
/// A <see cref="ConnectionHealth"/> value was added without extending this mapping.
/// Throwing beats a silent default: an unmapped state would otherwise be reported as
/// whichever value happened to be first.
/// </exception>
public static ConnectionHealthEnum ToDto(ConnectionHealth health) => health switch
{
ConnectionHealth.Connected => ConnectionHealthEnum.ConnectionHealthConnected,
ConnectionHealth.Disconnected => ConnectionHealthEnum.ConnectionHealthDisconnected,
ConnectionHealth.Connecting => ConnectionHealthEnum.ConnectionHealthConnecting,
ConnectionHealth.Error => ConnectionHealthEnum.ConnectionHealthError,
_ => throw new ArgumentOutOfRangeException(nameof(health), health, "Unmapped ConnectionHealth value"),
};
/// <summary>Reconstructs a <see cref="ConnectionHealth"/> from its wire enum.</summary>
/// <remarks>
/// An unspecified or unknown wire value decodes to <see cref="ConnectionHealth.Error"/>,
/// never to <see cref="ConnectionHealth.Connected"/>. A newer site sending a value this
/// build has never heard of must not have it rendered as "healthy" on the central
/// health page — the safe direction for an unknown connection state is "not working".
/// </remarks>
/// <param name="health">The wire enum value to reconstruct.</param>
/// <returns>The in-process connection health.</returns>
public static ConnectionHealth FromDto(ConnectionHealthEnum health) => health switch
{
ConnectionHealthEnum.ConnectionHealthConnected => ConnectionHealth.Connected,
ConnectionHealthEnum.ConnectionHealthDisconnected => ConnectionHealth.Disconnected,
ConnectionHealthEnum.ConnectionHealthConnecting => ConnectionHealth.Connecting,
_ => ConnectionHealth.Error,
};
private static string GuidToWire(Guid? value) =>
value?.ToString() ?? string.Empty;
private static Guid? GuidFromWire(string? value) =>
string.IsNullOrEmpty(value) ? null : Guid.Parse(value);
private static string? NullIfEmpty(string? value) =>
string.IsNullOrEmpty(value) ? null : value;
// All ScadaBridge timestamps are UTC by invariant; Timestamp.FromDateTime requires
// UTC kind. Specify (never convert) so a value read back from SQLite with Kind=Utc
// passes through and a defensively-unspecified one is treated as the UTC it already
// is. Mirrors AuditEventDtoMapper/SiteCallDtoMapper.EnsureUtc.
private static DateTime EnsureUtc(DateTime value) =>
value.Kind == DateTimeKind.Utc ? value : DateTime.SpecifyKind(value, DateTimeKind.Utc);
}
src/ZB.MOM.WW.ScadaBridge.Communication/Grpc/CentralControlGrpcService.cs
+299
View File
@@ -0,0 +1,299 @@
using Akka.Actor;
using Google.Protobuf.WellKnownTypes;
using Grpc.Core;
using Microsoft.Extensions.Logging;
using Microsoft.Extensions.Options;
using ZB.MOM.WW.ScadaBridge.Commons.Messages.Audit;
using ZB.MOM.WW.ScadaBridge.Commons.Messages.Deployment;
using ZB.MOM.WW.ScadaBridge.Commons.Messages.Health;
using ZB.MOM.WW.ScadaBridge.Commons.Messages.Notification;
using GrpcStatus = Grpc.Core.Status;
namespace ZB.MOM.WW.ScadaBridge.Communication.Grpc;
/// <summary>
/// Central-hosted gRPC face of the seven site→central control messages
/// (<c>Protos/central_control.proto</c>). Decodes each request onto the SAME in-process
/// message type the Akka <c>ClusterClient</c> path already carries, <c>Ask</c>s
/// <see cref="Actors.CentralCommunicationActor"/>, and encodes the reply back.
/// </summary>
/// <remarks>
/// <para>
/// <b>Direction is inverted from <see cref="SiteStreamGrpcServer"/>.</b> That server runs on a
/// site and central dials in; this one runs on CENTRAL and the site dials in. The two listen on
/// the same port number (8083) on their respective nodes, which is symmetry, not a collision —
/// a node is either central or a site, never both.
/// </para>
/// <para>
/// <b>Zero handler logic lives here.</b> Every RPC lands on a receive
/// <c>CentralCommunicationActor</c> already implements for the ClusterClient path, so the two
/// transports cannot drift in behaviour: the actor is the single implementation, and this class
/// is a codec plus an <c>Ask</c>. That is also why the service takes the actor through
/// <see cref="SetReady"/> rather than resolving anything from DI — the actor is created by the
/// host's Akka bootstrap, not by the container.
/// </para>
/// <para>
/// <b>Fault semantics deliberately differ from <see cref="SiteStreamGrpcServer"/>'s ingest
/// RPCs.</b> That server answers a failed audit ingest with an EMPTY <c>IngestAck</c>; this one
/// fails the call with a non-OK status. Both leave the site's rows <c>Pending</c> for the next
/// drain, so the outcome is the same — but this service replaces the ClusterClient path, whose
/// documented behaviour is to propagate the fault (<c>CentralCommunicationActor</c>'s
/// <c>HandleIngestAuditEvents</c> pipes a <c>Status.Failure</c> back), and preserving that keeps
/// a lost batch visible as a failure rather than as a successful call that acked nothing.
/// </para>
/// <para>
/// <b>Status mapping, and why it is not uniform.</b> A site transport may safely re-send a call
/// to the peer central node only when the call provably never ran. So:
/// </para>
/// <list type="bullet">
/// <item><see cref="StatusCode.Unavailable"/> — this node is not ready; nothing was dispatched,
/// so a cross-node retry is safe and correct.</item>
/// <item><see cref="StatusCode.DeadlineExceeded"/> — the <c>Ask</c> timed out. The message WAS
/// delivered and may have been processed; retrying it on the other node would duplicate work.</item>
/// <item><see cref="StatusCode.Internal"/> — the handler faulted (a piped
/// <see cref="Akka.Actor.Status.Failure"/>, e.g. a database error inside reconcile). Same
/// reasoning: it ran, so do not re-send it elsewhere.</item>
/// </list>
/// </remarks>
public sealed class CentralControlGrpcService : CentralControlService.CentralControlServiceBase
{
private readonly ILogger<CentralControlGrpcService> _logger;
private readonly CommunicationOptions _options;
// Null until the host's Akka bootstrap hands the actor over. Doubles as the readiness
// flag: a call arriving before then cannot be served and is refused with Unavailable.
// Volatile because SetReady runs on the startup thread while calls are served on
// Kestrel's thread pool.
private volatile IActorRef? _central;
/// <summary>
/// Creates the service. <b>This must remain the only public constructor</b> — see
/// <see cref="SetReady"/> for how the actor arrives, and the Host's
/// <c>CentralControlAuthInterceptor</c> for the interceptor-side version of the same rule.
/// </summary>
/// <param name="logger">Logger for readiness and fault diagnostics.</param>
/// <param name="options">Communication options supplying the per-RPC <c>Ask</c> timeouts.</param>
public CentralControlGrpcService(
ILogger<CentralControlGrpcService> logger,
IOptions<CommunicationOptions> options)
{
ArgumentNullException.ThrowIfNull(logger);
ArgumentNullException.ThrowIfNull(options);
_logger = logger;
_options = options.Value;
}
/// <summary>
/// Hands the <c>CentralCommunicationActor</c> to the service and opens the gate. Mirrors
/// <see cref="SiteStreamGrpcServer.SetReady"/>: the gRPC service is a DI singleton created
/// before the actor system exists, so the actor arrives post-construction.
/// </summary>
/// <remarks>
/// The contract is deliberately narrow, exactly as on the site side: it asserts that the
/// actor exists and can receive, NOT that every downstream singleton proxy
/// (<c>notification-outbox</c>, <c>audit-log-ingest</c>) has registered itself yet. Those
/// register moments later in the same startup path, and the actor already answers a call
/// that beats them with the same "not available, retry" reply it gives on the ClusterClient
/// path — so gating readiness on them would add nothing but a longer window in which sites
/// see <see cref="StatusCode.Unavailable"/>.
/// </remarks>
/// <param name="centralCommunicationActor">The central communication actor.</param>
public void SetReady(IActorRef centralCommunicationActor)
{
ArgumentNullException.ThrowIfNull(centralCommunicationActor);
_central = centralCommunicationActor;
}
/// <summary>Exposed for wiring assertions in tests.</summary>
internal bool IsReady => _central is not null;
/// <inheritdoc />
public override async Task<NotificationSubmitAckDto> SubmitNotification(
NotificationSubmitDto request, ServerCallContext context)
{
var central = RequireReady(context);
var ack = await AskAsync<NotificationSubmitAck>(
central,
CentralControlDtoMapper.FromDto(request),
_options.NotificationForwardTimeout,
context).ConfigureAwait(false);
return CentralControlDtoMapper.ToDto(ack);
}
/// <inheritdoc />
public override async Task<NotificationStatusResponseDto> QueryNotificationStatus(
NotificationStatusQueryDto request, ServerCallContext context)
{
var central = RequireReady(context);
var response = await AskAsync<NotificationStatusResponse>(
central,
CentralControlDtoMapper.FromDto(request),
_options.NotificationForwardTimeout,
context).ConfigureAwait(false);
return CentralControlDtoMapper.ToDto(response);
}
/// <inheritdoc />
public override async Task<IngestAck> IngestAuditEvents(
AuditEventBatch request, ServerCallContext context)
{
// An empty batch is a no-op the actor need never see; answering it here also means a
// site that drains an empty queue does not fail against a not-yet-ready central.
if (request.Events.Count == 0)
{
return new IngestAck();
}
var central = RequireReady(context);
var reply = await AskAsync<IngestAuditEventsReply>(
central,
CentralControlDtoMapper.FromDto(request),
SiteStreamGrpcServer.AuditIngestAskTimeout,
context).ConfigureAwait(false);
return CentralControlDtoMapper.ToIngestAck(reply.AcceptedEventIds);
}
/// <inheritdoc />
public override async Task<IngestAck> IngestCachedTelemetry(
CachedTelemetryBatch request, ServerCallContext context)
{
if (request.Packets.Count == 0)
{
return new IngestAck();
}
var central = RequireReady(context);
var reply = await AskAsync<IngestCachedTelemetryReply>(
central,
CentralControlDtoMapper.FromDto(request),
SiteStreamGrpcServer.AuditIngestAskTimeout,
context).ConfigureAwait(false);
return CentralControlDtoMapper.ToIngestAck(reply.AcceptedEventIds);
}
/// <inheritdoc />
public override async Task<ReconcileSiteResponseDto> ReconcileSite(
ReconcileSiteRequestDto request, ServerCallContext context)
{
var central = RequireReady(context);
var response = await AskAsync<ReconcileSiteResponse>(
central,
CentralControlDtoMapper.FromDto(request),
_options.QueryTimeout,
context).ConfigureAwait(false);
return CentralControlDtoMapper.ToDto(response);
}
/// <inheritdoc />
public override async Task<SiteHealthReportAckDto> ReportSiteHealth(
SiteHealthReportDto request, ServerCallContext context)
{
var central = RequireReady(context);
var ack = await AskAsync<SiteHealthReportAck>(
central,
CentralControlDtoMapper.FromDto(request),
_options.HealthReportTimeout,
context).ConfigureAwait(false);
return CentralControlDtoMapper.ToDto(ack);
}
/// <summary>
/// Application heartbeat — <b>always answers OK</b>, even when this node is not ready.
/// </summary>
/// <remarks>
/// The heartbeat is fire-and-forget on both sides: nothing on the site consumes the reply,
/// and the site's heartbeat timer must never take a fault (a failing heartbeat that raised
/// an error would be a self-inflicted outage on a purely informational signal). So this is
/// the one RPC that does not go through <see cref="RequireReady"/>: a heartbeat arriving
/// before the actor exists is logged and dropped, exactly as the actor itself drops one
/// that arrives before <c>ICentralHealthAggregator</c> is resolvable. Liveness is still
/// detected — the aggregator's offline timeout fires when the heartbeats stop landing.
/// </remarks>
/// <param name="request">The heartbeat.</param>
/// <param name="context">The gRPC call context.</param>
/// <returns>An empty reply, always.</returns>
public override Task<Empty> Heartbeat(HeartbeatDto request, ServerCallContext context)
{
var central = _central;
if (central is null)
{
_logger.LogDebug(
"Dropped a heartbeat from site {SiteId}: the central communication actor is not "
+ "ready yet. Heartbeats are fire-and-forget, so the call still succeeds.",
request.SiteId);
return Task.FromResult(new Empty());
}
// Tell, never Ask: the actor's HandleHeartbeat sends no reply.
central.Tell(CentralControlDtoMapper.FromDto(request), ActorRefs.NoSender);
return Task.FromResult(new Empty());
}
/// <summary>
/// Returns the central communication actor, or throws <see cref="StatusCode.Unavailable"/>
/// when the host has not finished bringing the actor system up.
/// </summary>
private IActorRef RequireReady(ServerCallContext context)
{
var central = _central;
if (central is not null)
{
return central;
}
_logger.LogWarning(
"Refused a control-plane call to {Method}: the central communication actor is not "
+ "ready yet. Nothing was dispatched, so the caller may retry (including against "
+ "the peer central node).",
context.Method);
throw new RpcException(new GrpcStatus(
StatusCode.Unavailable,
"Central control plane is not ready: the actor system is still starting."));
}
/// <summary>
/// Asks the central actor and maps a fault onto the status code that tells the caller
/// whether a cross-node retry is safe. See the class remarks for the mapping rationale.
/// </summary>
private async Task<TReply> AskAsync<TReply>(
IActorRef central, object message, TimeSpan timeout, ServerCallContext context)
{
try
{
return await central.Ask<TReply>(message, timeout, context.CancellationToken)
.ConfigureAwait(false);
}
catch (AskTimeoutException ex)
{
_logger.LogWarning(ex,
"Control-plane call {Method} timed out after {Timeout} waiting for the central "
+ "communication actor.",
context.Method, timeout);
throw new RpcException(new GrpcStatus(
StatusCode.DeadlineExceeded,
$"Central did not answer within {timeout}."));
}
catch (OperationCanceledException)
{
// The client gave up or its deadline expired; there is no one left to answer.
throw new RpcException(new GrpcStatus(
StatusCode.Cancelled, "The call was cancelled."));
}
catch (Exception ex)
{
_logger.LogError(ex,
"Control-plane call {Method} faulted inside the central communication actor.",
context.Method);
throw new RpcException(new GrpcStatus(
StatusCode.Internal, "Central failed to process the call."));
}
}
}
src/ZB.MOM.WW.ScadaBridge.Communication/Grpc/GrpcCentralTransport.cs
+258
View File
@@ -0,0 +1,258 @@
using Akka.Actor;
using Grpc.Core;
using Microsoft.Extensions.Logging;
using ZB.MOM.WW.ScadaBridge.Commons.Messages.Audit;
using ZB.MOM.WW.ScadaBridge.Commons.Messages.Deployment;
using ZB.MOM.WW.ScadaBridge.Commons.Messages.Health;
using ZB.MOM.WW.ScadaBridge.Commons.Messages.Notification;
using ZB.MOM.WW.ScadaBridge.Communication.Actors;
using AkkaStatus = Akka.Actor.Status;
namespace ZB.MOM.WW.ScadaBridge.Communication.Grpc;
/// <summary>
/// The <see cref="ICentralTransport"/> that carries the seven site→central sends over gRPC — the
/// migration target for the Akka <c>ClusterClient</c> path. Each method encodes the message with
/// <see cref="CentralControlDtoMapper"/>, dials <c>CentralControlService</c> through the sticky
/// <see cref="CentralChannelProvider"/>, and delivers the decoded reply (or a transient-failure
/// signal) to the waiting Ask.
/// </summary>
/// <remarks>
/// <para>
/// The actor handlers are synchronous; each method here kicks off the RPC as a detached task and
/// <c>Tell</c>s the result to <paramref name="replyTo"/> when it completes — <c>IActorRef.Tell</c>
/// is thread-safe, so the reply lands at the Ask exactly as central's ClusterClient reply did. On
/// any non-OK status the reply is <see cref="Status.Failure"/>, which the S&amp;F / audit / health
/// layers already treat as transient.
/// </para>
/// <para>
/// <b>Cross-node retry only on provably-unsent failures.</b> An <see cref="StatusCode.Unavailable"/>
/// (connection refused / node not ready) flips the channel pair and retries once on the peer.
/// A <see cref="StatusCode.DeadlineExceeded"/> is NEVER retried across nodes — a deploy / write /
/// failover may already have executed, and duplicating it is worse than surfacing a transient
/// failure the layer above tolerates.
/// </para>
/// <para>
/// <b>Per-call deadlines mirror today's Ask timeouts.</b> Notification submit/status →
/// <c>NotificationForwardTimeout</c> (30 s, the value the S&amp;F forwarder and the central service
/// both use); health → <c>HealthReportTimeout</c> (10 s); reconcile → <c>QueryTimeout</c> (30 s);
/// both ingest RPCs → <see cref="SiteStreamGrpcServer.AuditIngestAskTimeout"/> (the one shared 30 s
/// constant). The heartbeat, fire-and-forget with no server-side Ask, is merely bounded by
/// <c>HealthReportTimeout</c> and its failures are swallowed.
/// </para>
/// </remarks>
public sealed class GrpcCentralTransport : ICentralTransport
{
private readonly CentralChannelProvider _channels;
private readonly CommunicationOptions _options;
private readonly ILogger<GrpcCentralTransport> _logger;
/// <summary>Creates the transport over a channel pair.</summary>
/// <param name="channels">The sticky central channel pair.</param>
/// <param name="options">Communication options supplying the per-call deadlines.</param>
/// <param name="logger">Logger for failover/fault diagnostics.</param>
public GrpcCentralTransport(
CentralChannelProvider channels,
CommunicationOptions options,
ILogger<GrpcCentralTransport> logger)
{
ArgumentNullException.ThrowIfNull(channels);
ArgumentNullException.ThrowIfNull(options);
ArgumentNullException.ThrowIfNull(logger);
_channels = channels;
_options = options;
_logger = logger;
}
/// <inheritdoc />
public void SubmitNotification(NotificationSubmit message, IActorRef replyTo)
{
var dto = CentralControlDtoMapper.ToDto(message);
Dispatch(replyTo, _options.NotificationForwardTimeout,
(c, o) => c.SubmitNotificationAsync(dto, o),
ack => CentralControlDtoMapper.FromDto(ack));
}
/// <inheritdoc />
public void QueryNotificationStatus(NotificationStatusQuery message, IActorRef replyTo)
{
var dto = CentralControlDtoMapper.ToDto(message);
Dispatch(replyTo, _options.NotificationForwardTimeout,
(c, o) => c.QueryNotificationStatusAsync(dto, o),
response => CentralControlDtoMapper.FromDto(response));
}
/// <inheritdoc />
public void IngestAuditEvents(IngestAuditEventsCommand message, IActorRef replyTo)
{
var batch = CentralControlDtoMapper.ToDto(message);
Dispatch(replyTo, SiteStreamGrpcServer.AuditIngestAskTimeout,
(c, o) => c.IngestAuditEventsAsync(batch, o),
ack => new IngestAuditEventsReply(CentralControlDtoMapper.FromIngestAck(ack)));
}
/// <inheritdoc />
public void IngestCachedTelemetry(IngestCachedTelemetryCommand message, IActorRef replyTo)
{
var batch = CentralControlDtoMapper.ToDto(message);
Dispatch(replyTo, SiteStreamGrpcServer.AuditIngestAskTimeout,
(c, o) => c.IngestCachedTelemetryAsync(batch, o),
ack => new IngestCachedTelemetryReply(CentralControlDtoMapper.FromIngestAck(ack)));
}
/// <inheritdoc />
public void ReconcileSite(ReconcileSiteRequest message, IActorRef replyTo)
{
var dto = CentralControlDtoMapper.ToDto(message);
Dispatch(replyTo, _options.QueryTimeout,
(c, o) => c.ReconcileSiteAsync(dto, o),
response => CentralControlDtoMapper.FromDto(response));
}
/// <inheritdoc />
public void ReportSiteHealth(SiteHealthReport message, IActorRef replyTo)
{
var dto = CentralControlDtoMapper.ToDto(message);
Dispatch(replyTo, _options.HealthReportTimeout,
(c, o) => c.ReportSiteHealthAsync(dto, o),
ack => CentralControlDtoMapper.FromDto(ack));
}
/// <inheritdoc />
public void SendHeartbeat(HeartbeatMessage message, IActorRef self)
{
_ = SendHeartbeatAsync(message);
}
private async Task SendHeartbeatAsync(HeartbeatMessage message)
{
var (index, client) = _channels.Current();
var dto = CentralControlDtoMapper.ToDto(message);
try
{
var options = new CallOptions(deadline: DateTime.UtcNow.Add(_options.HealthReportTimeout));
using var call = client.HeartbeatAsync(dto, options);
await call.ResponseAsync.ConfigureAwait(false);
}
catch (RpcException ex) when (IsConnectFailure(ex))
{
// Nudge the pair so the next call tries the peer, but never fault: a heartbeat
// failure must not surface on the site's heartbeat timer path.
_channels.ReportUnavailable(index);
_logger.LogDebug(ex, "Heartbeat to central endpoint {Endpoint} was unavailable.", CurrentEndpointSafe());
}
catch (Exception ex)
{
_logger.LogDebug(ex, "Heartbeat to central failed (swallowed — heartbeats are fire-and-forget).");
}
}
/// <summary>
/// Runs a unary RPC on the current channel, delivers the decoded reply to
/// <paramref name="replyTo"/>, and applies the sticky-failover / no-retry-on-deadline policy.
/// </summary>
private void Dispatch<TWire>(
IActorRef replyTo,
TimeSpan timeout,
Func<CentralControlService.CentralControlServiceClient, CallOptions, AsyncUnaryCall<TWire>> call,
Func<TWire, object> decode)
{
_ = DispatchAsync(replyTo, timeout, call, decode);
}
private async Task DispatchAsync<TWire>(
IActorRef replyTo,
TimeSpan timeout,
Func<CentralControlService.CentralControlServiceClient, CallOptions, AsyncUnaryCall<TWire>> call,
Func<TWire, object> decode)
{
var (index, client) = _channels.Current();
try
{
var reply = await InvokeAsync(client, timeout, call).ConfigureAwait(false);
replyTo.Tell(decode(reply));
}
catch (RpcException ex) when (IsConnectFailure(ex))
{
// Provably unsent: the connection was refused / the node was not ready. Fail over
// to the peer and retry ONCE. This is the only status we retry across nodes.
_channels.ReportUnavailable(index);
var (retryIndex, retryClient) = _channels.Current();
if (retryIndex != index)
{
try
{
var reply = await InvokeAsync(retryClient, timeout, call).ConfigureAwait(false);
replyTo.Tell(decode(reply));
return;
}
catch (Exception retryEx)
{
_logger.LogWarning(retryEx,
"Central control-plane call failed on both endpoints; surfacing as transient.");
replyTo.Tell(new AkkaStatus.Failure(retryEx));
return;
}
}
replyTo.Tell(new AkkaStatus.Failure(ex));
}
catch (Exception ex)
{
// DeadlineExceeded / Internal / PermissionDenied / a PSK-resolution throw — do NOT
// retry across nodes (the call may have run). Surface as the transient failure the
// layer above already tolerates.
replyTo.Tell(new AkkaStatus.Failure(ex));
}
}
private static async Task<TWire> InvokeAsync<TWire>(
CentralControlService.CentralControlServiceClient client,
TimeSpan timeout,
Func<CentralControlService.CentralControlServiceClient, CallOptions, AsyncUnaryCall<TWire>> call)
{
var options = new CallOptions(deadline: DateTime.UtcNow.Add(timeout));
using var asyncCall = call(client, options);
return await asyncCall.ResponseAsync.ConfigureAwait(false);
}
/// <summary>
/// A failure that provably never reached a server — the only class safe to retry on the peer.
/// <see cref="StatusCode.DeadlineExceeded"/> is deliberately excluded (the call may have run).
/// </summary>
/// <remarks>
/// Two shapes qualify: a server-signalled <see cref="StatusCode.Unavailable"/> (e.g. a node
/// that returns Unavailable while it is still starting), and a client-side failure to even
/// start the call — Grpc.Net surfaces a refused/failed connection as
/// <see cref="StatusCode.Internal"/> "Error starting gRPC call" with the transport exception
/// attached, and there the request never left the client. Anything else — including a deadline,
/// a permission denial, or a generic server-side Internal after the call reached the server —
/// is NOT retried across nodes.
/// </remarks>
private static bool IsConnectFailure(RpcException ex)
{
if (ex.StatusCode == StatusCode.Unavailable)
{
return true;
}
// "Error starting gRPC call" is Grpc.Net's marker for a call that could not be sent — a
// refused/failed connection carrying an HttpRequestException. Provably unsent.
return ex.StatusCode == StatusCode.Internal
&& (ex.Status.DebugException is HttpRequestException
|| ex.Status.Detail.StartsWith("Error starting gRPC call", StringComparison.Ordinal));
}
private string CurrentEndpointSafe()
{
try
{
return _channels.CurrentEndpoint;
}
catch
{
return "(unknown)";
}
}
}
src/ZB.MOM.WW.ScadaBridge.Communication/Grpc/ISitePskProvider.cs
+132
View File
@@ -0,0 +1,132 @@
using Grpc.Core;
using Grpc.Net.Client;
namespace ZB.MOM.WW.ScadaBridge.Communication.Grpc;
/// <summary>
/// Resolves the preshared key that authenticates gRPC control-plane traffic for one site
/// relationship. Central holds one key per site; each site holds only its own.
/// </summary>
/// <remarks>
/// <para>
/// <b>Why per-site rather than one fleet-wide key.</b> A compromised site yields only its own
/// key, never another site's. A single shared key would be simpler to seed and strictly worse
/// on blast radius, which is why it was rejected in the design.
/// </para>
/// <para>
/// <b>Fail-closed.</b> Implementations throw when the key cannot be resolved. They must never
/// fall back to "no key means no authentication" — that is the failure mode this whole
/// mechanism exists to remove, and it would silently disable auth on exactly the default
/// configuration. A dial that cannot be authenticated does not happen.
/// </para>
/// <para>
/// The interface lives in Communication (not Host) because both sides need it: central's
/// site-dialing clients live here and in AuditLog, while the implementation over
/// <c>ISecretResolver</c> lives in Host, which owns the secrets container.
/// </para>
/// </remarks>
public interface ISitePskProvider
{
/// <summary>
/// Resolves the preshared key for <paramref name="siteId"/>, caching the result.
/// </summary>
/// <param name="siteId">Site identifier, as used in the <c>Site.SiteIdentifier</c> column.</param>
/// <param name="ct">Cancellation token.</param>
/// <returns>The preshared key. Never null or empty.</returns>
/// <exception cref="InvalidOperationException">
/// The key is not configured, not resolvable, or empty — the fail-closed path.
/// </exception>
ValueTask<string> GetAsync(string siteId, CancellationToken ct);
/// <summary>
/// Drops any cached key for <paramref name="siteId"/>, so the next
/// <see cref="GetAsync"/> re-reads the store. Called when a site is removed, and after a
/// key rotation.
/// </summary>
/// <param name="siteId">Site identifier whose cached key should be discarded.</param>
void Invalidate(string siteId);
}
/// <summary>
/// Builds the call credentials that carry a site's preshared key (and the site's own identity)
/// on every gRPC call central makes to that site — and, from Phase 1A, on the calls a site makes
/// to central.
/// </summary>
/// <remarks>
/// <para>
/// <b>Why <see cref="CallCredentials.FromInterceptor(AsyncAuthInterceptor)"/> rather than a
/// client <c>Interceptor</c>.</b> The key is resolved asynchronously from the secrets store, and
/// this is the one extension point in gRPC that is async by design. A client interceptor would
/// have to block on the resolve inside a synchronous <c>AsyncServerStreamingCall</c> path.
/// Credentials also apply uniformly to unary and streaming calls, so no call site can forget one.
/// </para>
/// <para>
/// <b>Why <c>UnsafeUseInsecureChannelCallCredentials</c>.</b> gRPC refuses to attach call
/// credentials to a plaintext channel by default, precisely because a bearer token on h2c is
/// readable and replayable by anyone on the path. That is a real and accepted limitation here:
/// these listeners are h2c today and the boundary assumes a trusted network. The PSK raises the
/// bar from "anyone who can reach the port" to "anyone who can read the traffic"; TLS on these
/// listeners is the follow-on hardening and requires no change to this code.
/// </para>
/// </remarks>
public static class ControlPlaneCredentials
{
/// <summary>
/// Metadata header naming the site a call belongs to. Central needs it to pick which
/// per-site key to verify against; a site's own inbound gate ignores it (a site has exactly
/// one key). Required by central's interceptor from Phase 1A.
/// </summary>
public const string SiteHeader = "x-scadabridge-site";
/// <summary>The bearer metadata header. Lowercase — gRPC lowercases header keys on the wire.</summary>
public const string AuthorizationHeader = "authorization";
/// <summary>
/// Creates call credentials that attach <c>authorization: Bearer &lt;psk&gt;</c> and
/// <c>x-scadabridge-site: &lt;siteId&gt;</c> to every call.
/// </summary>
/// <param name="provider">Resolves the site's preshared key.</param>
/// <param name="siteId">The site this channel talks to (or, site-side, this site's own id).</param>
/// <returns>Call credentials for a channel bound to that site.</returns>
public static CallCredentials ForSite(ISitePskProvider provider, string siteId)
{
ArgumentNullException.ThrowIfNull(provider);
ArgumentException.ThrowIfNullOrWhiteSpace(siteId);
return CallCredentials.FromInterceptor(async (context, metadata) =>
{
// A throw here fails the call, which is the point: an unauthenticated dial must
// not happen. Callers classify the resulting fault the same way they classify any
// other — the pull clients degrade to an empty batch and log, the streaming
// subscribers retry.
var psk = await provider.GetAsync(siteId, context.CancellationToken).ConfigureAwait(false);
metadata.Add(AuthorizationHeader, $"Bearer {psk}");
metadata.Add(SiteHeader, siteId);
});
}
/// <summary>
/// Applies per-site call credentials to channel options, if a provider is available.
/// A null provider leaves the options untouched — the shape used by test-only and
/// default constructors that never dial a gated endpoint.
/// </summary>
/// <param name="options">Channel options being built.</param>
/// <param name="provider">Key provider, or null to leave the channel unauthenticated.</param>
/// <param name="siteId">The site this channel talks to.</param>
/// <returns>The same options instance, for chaining.</returns>
public static GrpcChannelOptions WithSiteCredentials(
this GrpcChannelOptions options, ISitePskProvider? provider, string? siteId)
{
ArgumentNullException.ThrowIfNull(options);
if (provider is null || string.IsNullOrWhiteSpace(siteId))
{
return options;
}
options.Credentials = ChannelCredentials.Create(
ChannelCredentials.Insecure, ForSite(provider, siteId));
options.UnsafeUseInsecureChannelCallCredentials = true;
return options;
}
}
src/ZB.MOM.WW.ScadaBridge.Communication/Grpc/SiteCallDtoMapper.cs
+55 -8
View File
@@ -21,15 +21,17 @@ namespace ZB.MOM.WW.ScadaBridge.Communication.Grpc;
/// Mirrors the sibling <see cref="AuditEventDtoMapper"/>.
/// </para>
/// <para>
/// Two directions are provided. <see cref="FromDto"/> rehydrates the central
/// Three directions are provided. <see cref="FromDto"/> rehydrates the central
/// <see cref="SiteCall"/> entity central writes into the <c>SiteCalls</c> table.
/// <see cref="ToDto"/> projects a site-local <see cref="SiteCallOperational"/>
/// onto the wire — used by the Site Call Audit <c>PullSiteCalls</c>
/// reconciliation handler (the central→site self-heal pull). The
/// <see cref="SiteCall"/> entity itself is never mapped back onto the wire:
/// sites emit operational state from <see cref="SiteCallOperational"/>, never
/// from the central <see cref="SiteCall"/>, so a <c>SiteCall</c>→DTO method
/// would be dead code.
/// <see cref="ToDto(SiteCallOperational)"/> projects a site-local
/// <see cref="SiteCallOperational"/> onto the wire — used by the Site Call Audit
/// <c>PullSiteCalls</c> reconciliation handler (the central→site self-heal pull).
/// <see cref="ToDto(SiteCall)"/> projects the entity form back onto the wire; it
/// exists for the gRPC central control plane, whose site-side transport receives
/// an already-decoded <c>IngestCachedTelemetryCommand</c> (which carries
/// <see cref="SiteCall"/>, not <see cref="SiteCallOperational"/>) and must
/// re-encode it. It was previously documented here as necessarily dead code —
/// true only while ClusterClient was the sole path from that command to central.
/// </para>
/// <para>
/// String nullability convention: proto3 scalar strings cannot be absent, so the
@@ -120,6 +122,51 @@ public static class SiteCallDtoMapper
return dto;
}
/// <summary>
/// Projects a <see cref="SiteCall"/> entity onto its wire-format DTO — the
/// inverse of <see cref="FromDto"/>, so the pair round-trips exactly.
/// </summary>
/// <remarks>
/// <see cref="SiteCall.IngestedAtUtc"/> is deliberately NOT written: it is
/// central-set inside the dual-write transaction and the value carried on the
/// wire is informational only (<see cref="FromDto"/> stamps a placeholder that
/// the ingest actor overwrites). Every other field survives; null
/// <c>SourceNode</c>/<c>LastError</c> collapse to empty strings while the
/// nullable <c>HttpStatus</c>/<c>TerminalAtUtc</c> stay unset on the wire.
/// </remarks>
/// <param name="siteCall">The central operational-state entity to project.</param>
/// <returns>A populated <see cref="SiteCallOperationalDto"/> ready for transmission.</returns>
public static SiteCallOperationalDto ToDto(SiteCall siteCall)
{
ArgumentNullException.ThrowIfNull(siteCall);
var dto = new SiteCallOperationalDto
{
TrackedOperationId = siteCall.TrackedOperationId.ToString(),
Channel = siteCall.Channel,
Target = siteCall.Target,
SourceSite = siteCall.SourceSite,
SourceNode = siteCall.SourceNode ?? string.Empty,
Status = siteCall.Status,
RetryCount = siteCall.RetryCount,
LastError = siteCall.LastError ?? string.Empty,
CreatedAtUtc = Timestamp.FromDateTime(EnsureUtc(siteCall.CreatedAtUtc)),
UpdatedAtUtc = Timestamp.FromDateTime(EnsureUtc(siteCall.UpdatedAtUtc)),
};
if (siteCall.HttpStatus.HasValue)
{
dto.HttpStatus = siteCall.HttpStatus.Value;
}
if (siteCall.TerminalAtUtc.HasValue)
{
dto.TerminalAtUtc = Timestamp.FromDateTime(EnsureUtc(siteCall.TerminalAtUtc.Value));
}
return dto;
}
// All ScadaBridge timestamps are UTC by invariant; Timestamp.FromDateTime
// requires UTC kind. Specify (never convert) so a row read back from SQLite
// with Kind=Utc passes through and a defensively-unspecified value is
src/ZB.MOM.WW.ScadaBridge.Communication/Grpc/SiteStreamGrpcClient.cs
+22 -1
View File
@@ -60,6 +60,27 @@ public class SiteStreamGrpcClient : IAsyncDisposable, IDisposable
/// <param name="logger">Logger for diagnostics and errors.</param>
/// <param name="options">Communication options including keepalive settings.</param>
public SiteStreamGrpcClient(string endpoint, ILogger logger, CommunicationOptions options)
: this(endpoint, logger, options, pskProvider: null, siteIdentifier: null)
{
}
/// <summary>
/// Creates a client that authenticates every call with the site's preshared key.
/// This is the production shape: <c>SiteStreamService</c> is gated by
/// <c>ControlPlaneAuthInterceptor</c> on the site node, so a client without credentials
/// gets <see cref="StatusCode.PermissionDenied"/> on every call.
/// </summary>
/// <param name="endpoint">The gRPC endpoint address for the site.</param>
/// <param name="logger">Logger for diagnostics and errors.</param>
/// <param name="options">Communication options including keepalive settings.</param>
/// <param name="pskProvider">Resolves the site's preshared key; null leaves the channel unauthenticated.</param>
/// <param name="siteIdentifier">Site this channel talks to; null leaves the channel unauthenticated.</param>
public SiteStreamGrpcClient(
string endpoint,
ILogger logger,
CommunicationOptions options,
ISitePskProvider? pskProvider,
string? siteIdentifier)
{
Endpoint = endpoint;
KeepAlivePingDelay = options.GrpcKeepAlivePingDelay;
@@ -72,7 +93,7 @@ public class SiteStreamGrpcClient : IAsyncDisposable, IDisposable
KeepAlivePingTimeout = options.GrpcKeepAlivePingTimeout,
KeepAlivePingPolicy = HttpKeepAlivePingPolicy.Always
}
});
}.WithSiteCredentials(pskProvider, siteIdentifier));
_client = new SiteStreamService.SiteStreamServiceClient(_channel);
_logger = logger;
}
src/ZB.MOM.WW.ScadaBridge.Communication/Grpc/SiteStreamGrpcClientFactory.cs
+34 -7
View File
@@ -26,9 +26,11 @@ public class SiteStreamGrpcClientFactory : IAsyncDisposable, IDisposable
private readonly ConcurrentDictionary<(string Site, string Endpoint), SiteStreamGrpcClient> _clients = new();
private readonly ILoggerFactory _loggerFactory;
private readonly CommunicationOptions _options;
private readonly ISitePskProvider? _pskProvider;
/// <summary>
/// Test/default constructor — uses default <see cref="CommunicationOptions"/>.
/// Test/default constructor — uses default <see cref="CommunicationOptions"/> and creates
/// unauthenticated channels.
/// </summary>
/// <param name="loggerFactory">Logger factory passed to created clients.</param>
public SiteStreamGrpcClientFactory(ILoggerFactory loggerFactory)
@@ -37,16 +39,36 @@ public class SiteStreamGrpcClientFactory : IAsyncDisposable, IDisposable
}
/// <summary>
/// DI constructor — flows <see cref="CommunicationOptions"/> into every created
/// <see cref="SiteStreamGrpcClient"/> so the configured gRPC keepalive settings
/// are applied rather than hard-coded defaults.
/// Constructor without a key provider — creates unauthenticated channels, which a gated
/// site will refuse. Retained for tests and for hosts that never dial a site.
/// </summary>
/// <param name="loggerFactory">Logger factory passed to created clients.</param>
/// <param name="options">Communication options applied to each created client.</param>
public SiteStreamGrpcClientFactory(ILoggerFactory loggerFactory, IOptions<CommunicationOptions> options)
: this(loggerFactory, options, pskProvider: null)
{
}
/// <summary>
/// DI constructor — flows <see cref="CommunicationOptions"/> into every created
/// <see cref="SiteStreamGrpcClient"/> so the configured gRPC keepalive settings are applied
/// rather than hard-coded defaults, and attaches the per-site preshared key that the site's
/// <c>ControlPlaneAuthInterceptor</c> requires.
/// </summary>
/// <param name="loggerFactory">Logger factory passed to created clients.</param>
/// <param name="options">Communication options applied to each created client.</param>
/// <param name="pskProvider">
/// Resolves each site's preshared key. Optional in DI so a host that registers no provider
/// (a site node, which never dials another site) still resolves this factory.
/// </param>
public SiteStreamGrpcClientFactory(
ILoggerFactory loggerFactory,
IOptions<CommunicationOptions> options,
ISitePskProvider? pskProvider)
{
_loggerFactory = loggerFactory;
_options = options.Value;
_pskProvider = pskProvider;
}
/// <summary>
@@ -59,7 +81,7 @@ public class SiteStreamGrpcClientFactory : IAsyncDisposable, IDisposable
/// <param name="grpcEndpoint">gRPC endpoint (second half of the cache key) the client is bound to.</param>
/// <returns>The cached or newly-created client bound to <paramref name="grpcEndpoint"/>.</returns>
public virtual SiteStreamGrpcClient GetOrCreate(string siteIdentifier, string grpcEndpoint) =>
_clients.GetOrAdd((siteIdentifier, grpcEndpoint), _ => CreateClient(grpcEndpoint));
_clients.GetOrAdd((siteIdentifier, grpcEndpoint), key => CreateClient(key.Site, key.Endpoint));
/// <summary>
/// Returns the cached client for <c>(site, endpoint)</c>, or <c>null</c> — never creates.
@@ -77,12 +99,13 @@ public class SiteStreamGrpcClientFactory : IAsyncDisposable, IDisposable
/// can substitute a tracking client while still exercising the factory's real
/// caching and disposal machinery.
/// </summary>
/// <param name="siteIdentifier">Site the new client talks to; selects which preshared key it presents.</param>
/// <param name="grpcEndpoint">gRPC endpoint the new client will connect to.</param>
/// <returns>A new <see cref="SiteStreamGrpcClient"/> connected to <paramref name="grpcEndpoint"/>.</returns>
protected virtual SiteStreamGrpcClient CreateClient(string grpcEndpoint)
protected virtual SiteStreamGrpcClient CreateClient(string siteIdentifier, string grpcEndpoint)
{
var logger = _loggerFactory.CreateLogger<SiteStreamGrpcClient>();
return new SiteStreamGrpcClient(grpcEndpoint, logger, _options);
return new SiteStreamGrpcClient(grpcEndpoint, logger, _options, _pskProvider, siteIdentifier);
}
/// <summary>
@@ -99,6 +122,10 @@ public class SiteStreamGrpcClientFactory : IAsyncDisposable, IDisposable
if (_clients.TryRemove(key, out var client))
await client.DisposeAsync();
}
// Drop the cached preshared key too, so a site removed and re-added under the same
// identifier (with a rotated key) is not dialed with the stale one.
_pskProvider?.Invalidate(siteIdentifier);
}
/// <summary>
src/ZB.MOM.WW.ScadaBridge.Communication/Grpc/StaticSitePskProvider.cs
+43
View File
@@ -0,0 +1,43 @@
namespace ZB.MOM.WW.ScadaBridge.Communication.Grpc;
/// <summary>
/// Site-side <see cref="ISitePskProvider"/> over a single fixed key — the one key a site node
/// presents on every control-plane call it makes to central (<c>CommunicationOptions.GrpcPsk</c>).
/// Central's provider resolves a key <em>per site</em>; a site has exactly one, so it ignores the
/// requested <c>siteId</c> and returns its own key.
/// </summary>
/// <remarks>
/// <b>Fail-closed.</b> An empty key throws, matching the contract on <see cref="ISitePskProvider"/>
/// and the interceptor's own posture: a node shipped without a key must not degrade to an
/// unauthenticated dial.
/// </remarks>
public sealed class StaticSitePskProvider : ISitePskProvider
{
private readonly string _key;
/// <summary>Creates the provider bound to a site's own preshared key.</summary>
/// <param name="key">The site's <c>GrpcPsk</c>. Empty is permitted at construction but throws on use.</param>
public StaticSitePskProvider(string key)
{
_key = key ?? string.Empty;
}
/// <inheritdoc />
public ValueTask<string> GetAsync(string siteId, CancellationToken ct)
{
if (string.IsNullOrEmpty(_key))
{
throw new InvalidOperationException(
"No gRPC preshared key is configured for this site (ScadaBridge:Communication:GrpcPsk). "
+ "The control plane is fail-closed: an unauthenticated dial does not happen.");
}
return new ValueTask<string>(_key);
}
/// <inheritdoc />
public void Invalidate(string siteId)
{
// A single static key never changes for the process lifetime; nothing to drop.
}
}
src/ZB.MOM.WW.ScadaBridge.Communication/Protos/central_control.proto
+247
View File
@@ -0,0 +1,247 @@
syntax = "proto3";
option csharp_namespace = "ZB.MOM.WW.ScadaBridge.Communication.Grpc";
package scadabridge.centralcontrol.v1;
import "google/protobuf/empty.proto";
import "google/protobuf/timestamp.proto";
import "google/protobuf/wrappers.proto";
// The two ingest RPCs deliberately REUSE the batch/ack messages already defined
// for the site-hosted SiteStreamService rather than redeclaring them. The site
// telemetry actor builds AuditEventBatch / CachedTelemetryBatch today and hands
// them to ISiteStreamAuditClient; duplicating the shapes here would fork one
// wire contract into two that must be kept in lockstep by hand.
import "Protos/sitestream.proto";
// Central-hosted control plane (Phase 1A of the ClusterClient→gRPC migration).
//
// Direction: SITE is the client, CENTRAL is the server — the inverse of
// SiteStreamService, where central dials the site. That asymmetry is deliberate
// and mirrors the direction the Akka ClusterClient traffic flows today: these
// seven calls are exactly the seven messages SiteCommunicationActor sends to
// /user/central-communication.
//
// Every call is gated by ControlPlaneAuthInterceptor: `authorization: Bearer <psk>`
// plus the `x-scadabridge-site` metadata header naming which site's preshared key
// central must verify against.
service CentralControlService {
// Store-and-forward handoff of one notification for central delivery. The
// ack is idempotent on notification_id — a duplicate submit after a lost ack
// must not produce a second delivery.
rpc SubmitNotification(NotificationSubmitDto) returns (NotificationSubmitAckDto);
// Notify.Status(id) round-trip for a notification that has already left the
// site buffer. `found = false` sends the caller back to the site-local buffer
// to decide Forwarding vs Unknown.
rpc QueryNotificationStatus(NotificationStatusQueryDto) returns (NotificationStatusResponseDto);
// Audit Log (#23) push telemetry. Reuses the SiteStreamService messages: the
// batch a site drains from its SQLite hot path is byte-identical whichever
// transport carries it.
rpc IngestAuditEvents(sitestream.AuditEventBatch) returns (sitestream.IngestAck);
// Audit Log (#23) M3 combined cached-call telemetry (audit row + SiteCalls
// operational upsert, written in one central transaction).
rpc IngestCachedTelemetry(sitestream.CachedTelemetryBatch) returns (sitestream.IngestAck);
// Node-startup self-heal: the node's local deployed inventory in, fetch
// tokens for whatever it is missing or stale out.
rpc ReconcileSite(ReconcileSiteRequestDto) returns (ReconcileSiteResponseDto);
// Periodic site health report (30 s cadence). The ack makes delivery
// observable end-to-end so the sender can restore its per-interval counters
// when a report is lost.
rpc ReportSiteHealth(SiteHealthReportDto) returns (SiteHealthReportAckDto);
// Application heartbeat. Returns Empty because the message is
// fire-and-forget: nothing on the site consumes a reply, and a failure here
// must never surface as a fault on the heartbeat timer path.
rpc Heartbeat(HeartbeatDto) returns (google.protobuf.Empty);
}
// ---------------------------------------------------------------------------
// Notification Outbox (#21)
// ---------------------------------------------------------------------------
// Site -> Central: submit a buffered notification for central delivery.
// Mirrors Commons NotificationSubmit.
message NotificationSubmitDto {
string notification_id = 1; // GUID string, the idempotency key
string list_name = 2;
string subject = 3;
string body = 4;
string source_site_id = 5;
string source_instance_id = 6; // empty string represents null
string source_script = 7; // empty string represents null
google.protobuf.Timestamp site_enqueued_at = 8;
string origin_execution_id = 9; // GUID string; empty represents null
string origin_parent_execution_id = 10; // GUID string; empty represents null
string source_node = 11; // empty string represents null
}
// Central -> Site: ack sent after the Notifications row is persisted.
message NotificationSubmitAckDto {
string notification_id = 1;
bool accepted = 2;
string error = 3; // empty string represents null
}
// Site -> Central: Notify.Status(id) lookup against the central outbox.
message NotificationStatusQueryDto {
string correlation_id = 1;
string notification_id = 2;
}
// Central -> Site: current central delivery state for a queried notification.
message NotificationStatusResponseDto {
string correlation_id = 1;
bool found = 2;
string status = 3;
int32 retry_count = 4;
string last_error = 5; // empty string represents null
google.protobuf.Timestamp delivered_at = 6; // absent when null
}
// ---------------------------------------------------------------------------
// Startup reconciliation (Deployment Manager)
// ---------------------------------------------------------------------------
// Site -> Central: the node's local deployed inventory at startup.
message ReconcileSiteRequestDto {
string site_identifier = 1;
string node_id = 2;
// Instance unique name -> revision hash of the config the node currently holds.
map<string, string> local_name_to_revision_hash = 3;
}
// Central -> Site: the gap the node must (re)fetch, plus orphans to log.
message ReconcileSiteResponseDto {
repeated ReconcileGapItemDto gap = 1;
repeated string orphan_names = 2;
string central_fetch_base_url = 3;
}
// One instance the node must (re)fetch, with a freshly-minted short-TTL token.
message ReconcileGapItemDto {
string instance_unique_name = 1;
string deployment_id = 2;
string revision_hash = 3;
bool is_enabled = 4;
string fetch_token = 5;
}
// ---------------------------------------------------------------------------
// Health Monitoring (#11)
// ---------------------------------------------------------------------------
// Wire form of the Commons ConnectionHealth enum.
//
// CONNECTION_HEALTH_UNSPECIFIED exists only to keep the proto3 zero value from
// meaning something. Mapping Connected onto 0 would make an absent/garbled
// value decode as "healthy", which is precisely the wrong direction to fail;
// the mapper decodes UNSPECIFIED as Error instead and never emits it.
enum ConnectionHealthEnum {
CONNECTION_HEALTH_UNSPECIFIED = 0;
CONNECTION_HEALTH_CONNECTED = 1;
CONNECTION_HEALTH_DISCONNECTED = 2;
CONNECTION_HEALTH_CONNECTING = 3;
CONNECTION_HEALTH_ERROR = 4;
}
message TagResolutionStatusDto {
int32 total_subscribed = 1;
int32 successfully_resolved = 2;
}
message TagQualityCountsDto {
int32 good = 1;
int32 bad = 2;
int32 uncertain = 3;
}
message NodeStatusDto {
string hostname = 1;
bool is_online = 2;
string role = 3;
}
// Point-in-time snapshot of the site-local SQLite audit queue.
message SiteAuditBacklogSnapshotDto {
int32 pending_count = 1;
google.protobuf.Timestamp oldest_pending_utc = 2; // absent when the queue is empty
int64 on_disk_bytes = 3;
}
// The three collection wrappers below exist so null and empty stay
// distinguishable. proto3 cannot express presence on a `repeated` or `map`
// field — an unset one and an empty one are the same bytes — but the
// corresponding SiteHealthReport members are genuinely nullable
// (SiteHealthCollector emits `ClusterNodes: _clusterNodes?.ToList()`), and the
// central health surface reads null as "this producer doesn't report the
// signal" rather than "the signal is empty". Wrapping in a message restores
// message presence and makes the distinction survive the round-trip.
message ConnectionEndpointMapDto {
map<string, string> entries = 1;
}
message TagQualityMapDto {
map<string, TagQualityCountsDto> entries = 1;
}
message NodeStatusListDto {
repeated NodeStatusDto nodes = 1;
}
// Site -> Central: periodic site health report. Mirrors Commons SiteHealthReport.
// Additive-only evolution: field numbers are never reused.
message SiteHealthReportDto {
string site_id = 1;
int64 sequence_number = 2;
google.protobuf.Timestamp report_timestamp = 3;
map<string, ConnectionHealthEnum> data_connection_statuses = 4;
map<string, TagResolutionStatusDto> tag_resolution_counts = 5;
int32 script_error_count = 6;
int32 alarm_evaluation_error_count = 7;
map<string, int32> store_and_forward_buffer_depths = 8;
int32 dead_letter_count = 9;
int32 deployed_instance_count = 10;
int32 enabled_instance_count = 11;
int32 disabled_instance_count = 12;
string node_role = 13;
string node_hostname = 14;
ConnectionEndpointMapDto data_connection_endpoints = 15; // absent when null
TagQualityMapDto data_connection_tag_quality = 16; // absent when null
int32 parked_message_count = 17;
NodeStatusListDto cluster_nodes = 18; // absent when null
int32 site_audit_write_failures = 19;
int32 audit_redaction_failure = 20;
SiteAuditBacklogSnapshotDto site_audit_backlog = 21; // absent when no data yet
int64 site_event_log_write_failures = 22;
google.protobuf.DoubleValue oldest_parked_message_age_seconds = 23; // absent when nothing parked
int32 script_queue_depth = 24;
int32 script_busy_threads = 25;
google.protobuf.DoubleValue script_oldest_busy_age_seconds = 26; // absent when the pool is idle
// Nullable on purpose: absent means "replication not wired on this node",
// which is NOT the same as false ("wired but currently disconnected").
google.protobuf.BoolValue local_db_replication_connected = 27;
// Absent means UNKNOWN, never zero — a failed backlog read rendered as 0
// would report a broken replication pair as perfectly healthy.
google.protobuf.Int64Value local_db_oplog_backlog = 28;
}
// Central -> Site: health report ack, so a lost report is observable.
message SiteHealthReportAckDto {
string site_id = 1;
int64 sequence_number = 2;
bool accepted = 3;
string error = 4; // empty string represents null
}
// Site -> Central: application heartbeat (fire-and-forget; reply is Empty).
message HeartbeatDto {
string site_id = 1;
string node_hostname = 2;
bool is_active = 3;
google.protobuf.Timestamp timestamp = 4;
}
src/ZB.MOM.WW.ScadaBridge.Communication/ServiceCollectionExtensions.cs
+10 -1
View File
@@ -1,5 +1,6 @@
using Microsoft.Extensions.DependencyInjection;
using Microsoft.Extensions.DependencyInjection.Extensions;
using Microsoft.Extensions.Logging;
using Microsoft.Extensions.Options;
using ZB.MOM.WW.ScadaBridge.Communication.Grpc;
@@ -23,7 +24,15 @@ public static class ServiceCollectionExtensions
ServiceDescriptor.Singleton<IValidateOptions<CommunicationOptions>, CommunicationOptionsValidator>());
services.AddSingleton<CommunicationService>();
services.AddSingleton<SiteStreamGrpcClientFactory>();
// Explicit factory rather than AddSingleton<T>(): the ISitePskProvider dependency is
// optional (central registers one, a site node does not), and constructor selection
// over a nullable interface parameter is exactly the case the container cannot decide
// for itself — GetService returns null cleanly where constructor injection would throw.
services.AddSingleton(sp => new SiteStreamGrpcClientFactory(
sp.GetRequiredService<ILoggerFactory>(),
sp.GetRequiredService<IOptions<CommunicationOptions>>(),
sp.GetService<ISitePskProvider>()));
services.AddSingleton<DebugStreamService>();
// Aggregated live alarm cache (plan #10, Task 4): transient, in-memory, shared
src/ZB.MOM.WW.ScadaBridge.Communication/ZB.MOM.WW.ScadaBridge.Communication.csproj
+19 -9
View File
@@ -32,20 +32,30 @@
<ProjectReference Include="../ZB.MOM.WW.ScadaBridge.HealthMonitoring/ZB.MOM.WW.ScadaBridge.HealthMonitoring.csproj" />
</ItemGroup>
<!-- gRPC proto generation. The compiled C# is checked in under
SiteStreamGrpc/ (Sitestream.cs + SitestreamGrpc.cs) because protoc
segfaults inside our linux_arm64 Docker build image. To regenerate
after schema changes:
1. Temporarily uncomment the Protobuf ItemGroup below.
2. Delete SiteStreamGrpc/*.cs.
<!-- gRPC proto generation. The compiled C# is checked in — SiteStreamGrpc/
(Sitestream.cs + SitestreamGrpc.cs) for Protos/sitestream.proto, and
CentralControlGrpc/ (CentralControl.cs + CentralControlGrpc.cs) for
Protos/central_control.proto — because protoc segfaults inside our
linux_arm64 Docker build image. To regenerate after schema changes run
`docker/regen-proto.sh [sitestream|centralcontrol|all]`, which does all
of the following and always leaves this file as it found it:
1. Temporarily uncomment the Protobuf ItemGroup below (just the line
for the proto you changed — the other file's checked-in C# is
already compiled, so enabling both at once duplicates types).
2. Delete the matching checked-in *.cs.
3. `dotnet build` (on macOS) — Grpc.Tools writes fresh files to obj/.
4. Copy obj/Debug/net10.0/Protos/*.cs into SiteStreamGrpc/.
4. Copy obj/Debug/net10.0/Protos/*.cs into the matching folder.
5. Re-comment the ItemGroup.
Eventually we should switch the Docker build image to one with a
working protoc on arm64. -->
central_control.proto imports sitestream.proto, so protoc resolves it
from the project-relative path without sitestream.proto needing its own
Protobuf item.
An ACTIVE Protobuf item must never be committed — it breaks the Docker
image build. Eventually we should switch the Docker build image to one
with a working protoc on arm64. -->
<!--
<ItemGroup>
<Protobuf Include="Protos\sitestream.proto" GrpcServices="Both" />
<Protobuf Include="Protos\central_control.proto" GrpcServices="Both" />
</ItemGroup>
-->
src/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase.csproj
+4
View File
@@ -20,6 +20,10 @@
<PackageReference Include="Microsoft.Extensions.Options" />
<PackageReference Include="Microsoft.AspNetCore.DataProtection" />
<PackageReference Include="Microsoft.AspNetCore.DataProtection.EntityFrameworkCore" />
<!-- Direct pin of a transitive dependency of Microsoft.AspNetCore.DataProtection:
10.0.7 carries four NU1903 high-severity advisories that break fresh restores
under TreatWarningsAsErrors. See Directory.Packages.props for the rationale. -->
<PackageReference Include="System.Security.Cryptography.Xml" />
</ItemGroup>
<ItemGroup>
src/ZB.MOM.WW.ScadaBridge.Host/Actors/AkkaHostedService.cs
+97 -21
View File
@@ -8,6 +8,7 @@ using Microsoft.Extensions.Options;
using ZB.MOM.WW.ScadaBridge.ClusterInfrastructure;
using ZB.MOM.WW.ScadaBridge.Communication;
using ZB.MOM.WW.ScadaBridge.Communication.Actors;
using ZB.MOM.WW.ScadaBridge.Communication.Grpc;
using ZB.MOM.WW.ScadaBridge.Host.Actors;
using ZB.MOM.WW.ScadaBridge.SiteRuntime;
using ZB.MOM.WW.ScadaBridge.SiteRuntime.Actors;
@@ -200,8 +201,9 @@ public class AkkaHostedService : IHostedService
_communicationOptions.TransportHeartbeatInterval.TotalSeconds,
_communicationOptions.TransportFailureThreshold.TotalSeconds);
// Down-if-alone recovery watchdog: SBR's keep-oldest down-if-alone plus
// run-coordinated-shutdown-when-down means a self-downed node terminates
// Downed-node recovery watchdog: any downing decision against this node
// (auto-down by the peer, or an SBR self-down under keep-oldest) plus
// run-coordinated-shutdown-when-down means a downed node terminates
// its own ActorSystem. If that happens outside our StopAsync, the Host
// process must exit so the service supervisor (docker
// `restart: unless-stopped` / Windows service recovery) restarts it and
@@ -228,14 +230,21 @@ public class AkkaHostedService : IHostedService
/// seed-node URI, role or split-brain strategy containing a quote, backslash or
/// whitespace cannot corrupt the document or be silently misparsed.
///
/// The <c>keep-oldest down-if-alone</c> flag is emitted from
/// The downing block branches on <see cref="ClusterOptions.SplitBrainResolverStrategy"/>:
/// <c>auto-down</c> (default; decision 2026-07-21) installs Akka's
/// <c>AutoDowning</c> provider with <c>auto-down-unreachable-after</c> =
/// <see cref="ClusterOptions.StableAfter"/> — the leader among the REACHABLE members
/// downs the unreachable peer, so a crash of either node (oldest included) fails
/// over to the survivor; the accepted trade is dual-active during a real network
/// partition. Any other value takes the SBR path, where the
/// <c>keep-oldest down-if-alone</c> flag is emitted from
/// <see cref="ClusterOptions.DownIfAlone"/> rather than hard-coded, so the bound
/// configuration value is actually consumed.
///
/// The split-brain-resolver <c>downing-provider-class</c> is installed
/// explicitly: Akka defaults to <c>NoDowning</c>, under which the entire
/// split-brain-resolver section is inert and singletons never migrate on a hard
/// crash or partition. Naming the SBR provider is what activates automatic downing.
/// A <c>downing-provider-class</c> is always installed explicitly: Akka defaults
/// to <c>NoDowning</c>, under which the downing configuration is inert and
/// singletons never migrate on a hard crash or partition. Naming the provider is
/// what activates automatic downing.
///
/// Every duration is rendered via <see cref="DurationHocon"/> in
/// milliseconds, so sub-second cluster timing values (e.g. a 750ms heartbeat) are
@@ -258,6 +267,25 @@ public class AkkaHostedService : IHostedService
clusterOptions.SeedNodes.Select(QuoteHocon));
var rolesStr = string.Join(",", roles.Select(QuoteHocon));
// auto-down (default): AutoDowning provider — the leader among the reachable
// members downs the unreachable peer after StableAfter, so a crash of EITHER
// node fails over to the survivor (dual-active during a real partition is the
// accepted trade — decision 2026-07-21). Anything else: the SBR provider with
// the configured active-strategy (keep-oldest), which is partition-safe but
// cannot survive a crash of the oldest node in a two-node cluster.
var downingBlock = string.Equals(
clusterOptions.SplitBrainResolverStrategy, "auto-down", StringComparison.OrdinalIgnoreCase)
? $@"downing-provider-class = ""Akka.Cluster.AutoDowning, Akka.Cluster""
auto-down-unreachable-after = {DurationHocon(clusterOptions.StableAfter)}"
: $@"downing-provider-class = ""Akka.Cluster.SBR.SplitBrainResolverProvider, Akka.Cluster""
split-brain-resolver {{
active-strategy = {QuoteHocon(clusterOptions.SplitBrainResolverStrategy)}
stable-after = {DurationHocon(clusterOptions.StableAfter)}
keep-oldest {{
down-if-alone = {(clusterOptions.DownIfAlone ? "on" : "off")}
}}
}}";
return $@"
audit-telemetry-dispatcher {{
type = ForkJoinDispatcher
@@ -287,14 +315,7 @@ akka {{
seed-nodes = [{seedNodesStr}]
roles = [{rolesStr}]
min-nr-of-members = {clusterOptions.MinNrOfMembers}
downing-provider-class = ""Akka.Cluster.SBR.SplitBrainResolverProvider, Akka.Cluster""
split-brain-resolver {{
active-strategy = {QuoteHocon(clusterOptions.SplitBrainResolverStrategy)}
stable-after = {DurationHocon(clusterOptions.StableAfter)}
keep-oldest {{
down-if-alone = {(clusterOptions.DownIfAlone ? "on" : "off")}
}}
}}
{downingBlock}
failure-detector {{
heartbeat-interval = {DurationHocon(clusterOptions.HeartbeatInterval)}
acceptable-heartbeat-pause = {DurationHocon(clusterOptions.FailureDetectionThreshold)}
@@ -416,6 +437,19 @@ akka {{
ClusterClientReceptionist.Get(_actorSystem).RegisterService(centralCommActor);
_logger.LogInformation("CentralCommunicationActor registered with ClusterClientReceptionist");
// Hand the same actor to the central-hosted gRPC control plane (T1A.2) and open its
// readiness gate — the gRPC face Asks this exact actor, so both transports resolve to
// one handler implementation. Mirrors SiteStreamGrpcServer.SetReady on the site side:
// the service is a DI singleton created before the actor system exists, so the actor
// arrives here post-construction. Null on a host that did not register the service
// (e.g. an in-process test harness), so the wiring is a guarded no-op there.
var centralControlGrpc = _serviceProvider
.GetService<ZB.MOM.WW.ScadaBridge.Communication.Grpc.CentralControlGrpcService>();
centralControlGrpc?.SetReady(centralCommActor);
_logger.LogInformation(
"CentralControlGrpcService readiness set (service bound: {Bound})",
centralControlGrpc is not null);
// Wire up the CommunicationService with the actor reference
var commService = _serviceProvider.GetService<CommunicationService>();
commService?.SetCommunicationActor(centralCommActor);
@@ -430,16 +464,22 @@ akka {{
siteAlarmLiveCache?.SetActorSystem(_actorSystem!);
// Management Service — accessible via ClusterClient
// Management Service — reached IN-PROCESS only, via ManagementActorHolder.
//
// This actor used to be registered with the ClusterClientReceptionist as well, for
// an out-of-cluster CLI that was never built (REQ-HOST-6a). The shipped CLI speaks
// HTTP Basic to /management, which asks this actor through the holder below
// (ManagementEndpoints), so the registration had no sender anywhere in the repo —
// it only advertised a management surface across the cluster-client boundary for
// free. Removed 2026-07-22 (ClusterClient→gRPC migration, T0.1).
var mgmtLogger = _serviceProvider.GetRequiredService<ILoggerFactory>()
.CreateLogger<ZB.MOM.WW.ScadaBridge.ManagementService.ManagementActor>();
var mgmtActor = _actorSystem!.ActorOf(
Props.Create(() => new ZB.MOM.WW.ScadaBridge.ManagementService.ManagementActor(_serviceProvider, mgmtLogger)),
"management");
ClusterClientReceptionist.Get(_actorSystem).RegisterService(mgmtActor);
var mgmtHolder = _serviceProvider.GetRequiredService<ZB.MOM.WW.ScadaBridge.ManagementService.ManagementActorHolder>();
mgmtHolder.ActorRef = mgmtActor;
_logger.LogInformation("ManagementActor registered with ClusterClientReceptionist");
_logger.LogInformation("ManagementActor started at /user/management (in-process access via ManagementActorHolder)");
// Notification Outbox — cluster singleton so exactly one node owns ingest,
// the dispatch sweep and the purge loop. Central actors run on the base
@@ -793,13 +833,45 @@ akka {{
_logger, role: siteRole);
var dmProxy = dm.Proxy;
// Select the site→central transport behind the coexistence flag (default Akka
// ClusterClient). When gRPC is chosen the site dials CentralControlService directly with
// a sticky-failover channel pair, presenting its own preshared key; the ClusterClient
// below is then not created at all.
ICentralTransport? centralTransport = null;
if (_communicationOptions.CentralTransport == CentralTransportMode.Grpc)
{
var loggerFactory = _serviceProvider.GetRequiredService<ILoggerFactory>();
var channelProvider = new CentralChannelProvider(
_communicationOptions.CentralGrpcEndpoints,
new StaticSitePskProvider(_communicationOptions.GrpcPsk),
_nodeOptions.SiteId!,
_communicationOptions,
loggerFactory.CreateLogger<CentralChannelProvider>());
_trackedDisposables.Add(channelProvider);
centralTransport = new GrpcCentralTransport(
channelProvider,
_communicationOptions,
loggerFactory.CreateLogger<GrpcCentralTransport>());
_logger.LogInformation(
"Site→central transport: gRPC to {Count} central endpoint(s) for site {SiteId}",
_communicationOptions.CentralGrpcEndpoints.Count, _nodeOptions.SiteId);
}
else
{
_logger.LogInformation(
"Site→central transport: Akka ClusterClient (default) for site {SiteId}",
_nodeOptions.SiteId);
}
// Create SiteCommunicationActor for receiving messages from central
var siteCommActor = _actorSystem.ActorOf(
Props.Create(() => new SiteCommunicationActor(
_nodeOptions.SiteId!,
_communicationOptions,
dmProxy,
activeNodeCheck)),
activeNodeCheck,
null,
centralTransport)),
"site-communication");
// Register local handlers with SiteCommunicationActor
@@ -918,8 +990,12 @@ akka {{
"Site actors registered. DeploymentManager singleton scoped to role={SiteRole}, SiteCommunicationActor created.",
siteRole);
// Create ClusterClient to central if contact points are configured
if (_communicationOptions.CentralContactPoints.Count > 0)
// Create ClusterClient to central if contact points are configured — but only on the Akka
// transport. On the gRPC transport the SiteCommunicationActor already holds a
// GrpcCentralTransport and never receives RegisterCentralClient, so a ClusterClient here
// would be dead weight (and keep an unwanted cross-cluster Akka association alive).
if (_communicationOptions.CentralTransport == CentralTransportMode.Akka
&& _communicationOptions.CentralContactPoints.Count > 0)
{
var contacts = _communicationOptions.CentralContactPoints
.Select(cp => ActorPath.Parse($"{cp}/system/receptionist"))
src/ZB.MOM.WW.ScadaBridge.Host/CentralControlAuthInterceptor.cs
+245
View File
@@ -0,0 +1,245 @@
using System.Security.Cryptography;
using System.Text;
using Grpc.Core;
using Grpc.Core.Interceptors;
using Microsoft.Extensions.Logging;
using ZB.MOM.WW.ScadaBridge.Communication.Grpc;
namespace ZB.MOM.WW.ScadaBridge.Host;
/// <summary>
/// Gates the central-hosted gRPC control plane (<c>CentralControlService</c>) with each site's
/// preshared key. The sibling of <see cref="ControlPlaneAuthInterceptor"/>, but with the
/// verification model inverted: a site checks one bearer token against its own single key, whereas
/// central must check the presented token against the key belonging to the SITE that sent it.
/// </summary>
/// <remarks>
/// <para>
/// <b>Why a separate class rather than a second constructor on
/// <see cref="ControlPlaneAuthInterceptor"/>.</b> <c>Grpc.AspNetCore</c> registers a
/// type-registered interceptor through <c>InterceptorRegistration.GetFactory()</c>, which throws
/// <c>"Multiple constructors accepting all given argument types have been found"</c> the moment a
/// second public constructor is applicable. That throw lands inside the pipeline on every call and
/// surfaces as <c>Unknown / "Exception was thrown by handler"</c> — the node boots healthy and
/// every gated call dies looking like a handler bug, with correct/wrong/no key all producing the
/// identical error. It shipped once with a fully green suite and was caught only on the rig.
/// Central's model genuinely differs (per-site key by header, not the site's one own-key), so it
/// gets its own class with its own single public constructor rather than a variant ctor on the
/// site interceptor. Both classes are pinned by a reflection test asserting exactly one public
/// constructor.
/// </para>
/// <para>
/// <b>Fail-closed on every branch.</b> A gated call is refused with
/// <see cref="StatusCode.PermissionDenied"/> when: the <c>x-scadabridge-site</c> header is
/// missing or blank; no key can be resolved for that site (<see cref="ISitePskProvider"/> throws);
/// or the presented bearer token does not match. There is no pass-through — an unresolvable or
/// absent identity never degrades to "let it in". Non-gated services (should any share the
/// listener) return immediately, matching the site interceptor's shape.
/// </para>
/// <para>
/// The comparison is <see cref="CryptographicOperations.FixedTimeEquals"/> over UTF-8, the same
/// constant-time compare the site interceptor and LocalDb sync use.
/// </para>
/// </remarks>
public sealed class CentralControlAuthInterceptor : Interceptor
{
/// <summary>
/// Service prefixes gated by default — the one central-hosted control-plane service. Taken
/// from the generated <c>package scadabridge.centralcontrol.v1; service CentralControlService</c>.
/// </summary>
public static readonly IReadOnlyList<string> DefaultGatedPrefixes =
new[] { $"/{CentralControlService.Descriptor.FullName}/" };
private readonly IReadOnlyList<string> _gatedPrefixes;
private readonly ISitePskProvider _pskProvider;
private readonly ILogger<CentralControlAuthInterceptor> _logger;
/// <summary>
/// Creates the interceptor gating <see cref="DefaultGatedPrefixes"/>.
/// </summary>
/// <remarks>
/// <b>This must remain the ONLY public constructor</b> — see the class remarks for why a
/// second one silently disables the gate. Pinned by
/// <c>CentralControlAuthInterceptorTests.TheInterceptorHasExactlyOnePublicConstructor</c>.
/// </remarks>
/// <param name="pskProvider">Resolves each site's preshared key.</param>
/// <param name="logger">Logger for denial diagnostics.</param>
public CentralControlAuthInterceptor(
ISitePskProvider pskProvider,
ILogger<CentralControlAuthInterceptor> logger)
: this(pskProvider, logger, DefaultGatedPrefixes)
{
}
/// <summary>
/// Creates the interceptor gating an explicit prefix set. <b>Internal</b> — a public second
/// constructor would reintroduce the ambiguous-constructor defect described on the class.
/// </summary>
/// <param name="pskProvider">Resolves each site's preshared key.</param>
/// <param name="logger">Logger for denial diagnostics.</param>
/// <param name="gatedPrefixes">Method-path prefixes to gate.</param>
internal CentralControlAuthInterceptor(
ISitePskProvider pskProvider,
ILogger<CentralControlAuthInterceptor> logger,
IReadOnlyList<string> gatedPrefixes)
{
ArgumentNullException.ThrowIfNull(pskProvider);
ArgumentNullException.ThrowIfNull(logger);
ArgumentNullException.ThrowIfNull(gatedPrefixes);
_pskProvider = pskProvider;
_logger = logger;
_gatedPrefixes = gatedPrefixes;
}
/// <inheritdoc />
public override async Task<TResponse> UnaryServerHandler<TRequest, TResponse>(
TRequest request,
ServerCallContext context,
UnaryServerMethod<TRequest, TResponse> continuation)
{
await AuthorizeAsync(context).ConfigureAwait(false);
return await continuation(request, context).ConfigureAwait(false);
}
/// <inheritdoc />
public override async Task DuplexStreamingServerHandler<TRequest, TResponse>(
IAsyncStreamReader<TRequest> requestStream,
IServerStreamWriter<TResponse> responseStream,
ServerCallContext context,
DuplexStreamingServerMethod<TRequest, TResponse> continuation)
{
await AuthorizeAsync(context).ConfigureAwait(false);
await continuation(requestStream, responseStream, context).ConfigureAwait(false);
}
/// <inheritdoc />
public override async Task<TResponse> ClientStreamingServerHandler<TRequest, TResponse>(
IAsyncStreamReader<TRequest> requestStream,
ServerCallContext context,
ClientStreamingServerMethod<TRequest, TResponse> continuation)
{
await AuthorizeAsync(context).ConfigureAwait(false);
return await continuation(requestStream, context).ConfigureAwait(false);
}
/// <inheritdoc />
public override async Task ServerStreamingServerHandler<TRequest, TResponse>(
TRequest request,
IServerStreamWriter<TResponse> responseStream,
ServerCallContext context,
ServerStreamingServerMethod<TRequest, TResponse> continuation)
{
await AuthorizeAsync(context).ConfigureAwait(false);
await continuation(request, responseStream, context).ConfigureAwait(false);
}
/// <summary>
/// Throws <see cref="RpcException"/> with <see cref="StatusCode.PermissionDenied"/> unless a
/// gated call carries a valid <c>x-scadabridge-site</c> header AND a bearer token matching
/// that site's resolved key. Non-gated calls return immediately.
/// </summary>
private async Task AuthorizeAsync(ServerCallContext context)
{
if (!IsGated(context.Method))
{
return;
}
var siteId = ExtractSiteId(context.RequestHeaders);
if (string.IsNullOrWhiteSpace(siteId))
{
_logger.LogWarning(
"Rejected a central control-plane call to {Method}: the required "
+ "'{Header}' metadata header is missing or blank, so there is no per-site key to "
+ "verify against.",
context.Method, ControlPlaneCredentials.SiteHeader);
throw Denied("missing site identity header");
}
string expected;
try
{
expected = await _pskProvider.GetAsync(siteId, context.CancellationToken)
.ConfigureAwait(false);
}
catch (OperationCanceledException)
{
throw;
}
catch (Exception ex)
{
// Fail-closed: an unresolvable key is a denial, never a pass-through. The provider
// has already logged the specific cause (missing secret / missing config entry).
_logger.LogWarning(ex,
"Rejected a central control-plane call to {Method}: no preshared key could be "
+ "resolved for site {SiteId}.",
context.Method, siteId);
throw Denied("no key configured for the presented site");
}
var presented = ExtractBearerToken(context.RequestHeaders);
if (presented is null || !FixedTimeEquals(presented, expected))
{
_logger.LogWarning(
"Rejected a central control-plane call to {Method} from site {SiteId}: {Reason}.",
context.Method, siteId,
presented is null ? "no bearer token presented" : "bearer token did not match");
throw Denied("control plane authentication failed");
}
}
private static RpcException Denied(string reason)
=> new(new Status(StatusCode.PermissionDenied, $"Control plane authentication failed: {reason}."));
private bool IsGated(string method)
{
foreach (var prefix in _gatedPrefixes)
{
if (method.StartsWith(prefix, StringComparison.Ordinal))
{
return true;
}
}
return false;
}
private static string? ExtractSiteId(Metadata headers)
{
foreach (var entry in headers)
{
if (string.Equals(entry.Key, ControlPlaneCredentials.SiteHeader,
StringComparison.OrdinalIgnoreCase))
{
return entry.Value;
}
}
return null;
}
private static string? ExtractBearerToken(Metadata headers)
{
// gRPC lowercases header keys on the wire; compare case-insensitively so a hand-built
// Metadata in a test behaves the same as a real request.
foreach (var entry in headers)
{
if (!string.Equals(entry.Key, ControlPlaneCredentials.AuthorizationHeader,
StringComparison.OrdinalIgnoreCase))
{
continue;
}
var value = entry.Value;
if (value is not null && value.StartsWith("Bearer ", StringComparison.OrdinalIgnoreCase))
{
return value["Bearer ".Length..];
}
}
return null;
}
private static bool FixedTimeEquals(string presented, string expected)
=> CryptographicOperations.FixedTimeEquals(
Encoding.UTF8.GetBytes(presented), Encoding.UTF8.GetBytes(expected));
}
src/ZB.MOM.WW.ScadaBridge.Host/ControlPlaneAuthInterceptor.cs
+228
View File
@@ -0,0 +1,228 @@
using System.Security.Cryptography;
using System.Text;
using Grpc.Core;
using Grpc.Core.Interceptors;
using Microsoft.Extensions.Logging;
using Microsoft.Extensions.Options;
using ZB.MOM.WW.ScadaBridge.Communication;
using ZB.MOM.WW.ScadaBridge.Communication.Grpc;
namespace ZB.MOM.WW.ScadaBridge.Host;
/// <summary>
/// Gates the site↔central gRPC control plane with a preshared key.
/// </summary>
/// <remarks>
/// <para>
/// <b>The gap this closes.</b> <c>SiteStreamService</c> shipped with no authentication at all —
/// plaintext h2c, no interceptor. Anything that could reach a site node's gRPC port could open a
/// live data stream or call <c>PullAuditEvents</c>/<c>PullSiteCalls</c> and read audit rows back.
/// The only gated surface on that listener was LocalDb sync, and only for its own service. That
/// gap exists independently of the ClusterClient→gRPC migration; it becomes indefensible once
/// every command crosses this listener.
/// </para>
/// <para>
/// <b>Modeled on <see cref="LocalDbSyncAuthInterceptor"/>,</b> deliberately: same four server
/// handlers funnelling into one <c>Authorize</c>, same <c>authorization: Bearer</c> extraction,
/// same <see cref="CryptographicOperations.FixedTimeEquals"/> comparison, same fail-closed
/// posture, same <see cref="StatusCode.PermissionDenied"/> rejection. Two differences:
/// </para>
/// <list type="number">
/// <item>It gates a <b>set</b> of service prefixes rather than one, so later phases can add the
/// new command/control services without a second interceptor.</item>
/// <item>Its expected key comes from <see cref="CommunicationOptions.GrpcPsk"/> — the site's own
/// key, supplied in production as <c>${secret:SB-GRPC-PSK-&lt;siteId&gt;}</c> and expanded before
/// the host is built.</item>
/// </list>
/// <para>
/// <b>The two keys are separate on purpose.</b> LocalDb sync keeps its own
/// <c>LocalDb:Replication:ApiKey</c>, which authenticates a different peer (the pair partner, not
/// central) over a different trust relationship. Sharing one key would mean a site's central-facing
/// key also admits writes into its database.
/// </para>
/// <para>
/// <b>Fail-closed, and not optional.</b> With no <c>GrpcPsk</c> configured, every gated call is
/// rejected — including the ones that work today. That is a deliberate break: LocalDb replication
/// is an opt-in feature whose "off" state is "no peer", whereas streaming and audit pull are
/// core paths, so "no key" must not silently mean "no authentication". Every environment must
/// carry a key before upgrading to this build.
/// </para>
/// </remarks>
public sealed class ControlPlaneAuthInterceptor : Interceptor
{
/// <summary>
/// Service prefixes gated by default. Read from the generated <c>sitestream.proto</c>
/// package/service names — <c>package sitestream; service SiteStreamService</c>.
/// Later phases append their own services here.
/// </summary>
public static readonly IReadOnlyList<string> DefaultGatedPrefixes =
new[] { "/sitestream.SiteStreamService/" };
private readonly IReadOnlyList<string> _gatedPrefixes;
private readonly IOptions<CommunicationOptions> _options;
private readonly ILogger<ControlPlaneAuthInterceptor> _logger;
/// <summary>
/// Creates the interceptor gating <see cref="DefaultGatedPrefixes"/>.
/// </summary>
/// <remarks>
/// <b>This must remain the ONLY public constructor.</b> <c>AddGrpc</c> registers the
/// interceptor by type, and <c>Grpc.AspNetCore.Server.InterceptorRegistration.GetFactory()</c>
/// throws <c>"Multiple constructors accepting all given argument types have been found"</c>
/// when a second one is applicable. That throw happens per call, inside the pipeline, and
/// surfaces to the caller as <c>Unknown / "Exception was thrown by handler"</c> — so the gate
/// silently stops authorizing anything while still failing every call. A second public
/// constructor added here in a later phase reintroduces exactly that. Pinned by
/// <c>ControlPlaneAuthInterceptorTests.TheInterceptorHasExactlyOnePublicConstructor</c>.
/// </remarks>
/// <param name="options">Communication options; <c>GrpcPsk</c> is the expected bearer token.</param>
/// <param name="logger">Logger for denial diagnostics.</param>
public ControlPlaneAuthInterceptor(
IOptions<CommunicationOptions> options,
ILogger<ControlPlaneAuthInterceptor> logger)
: this(options, logger, DefaultGatedPrefixes)
{
}
/// <summary>
/// Creates the interceptor gating an explicit set of service prefixes. <b>Internal</b> —
/// see the public constructor's remarks for why this cannot be public. Phases that add a
/// service to the gate should extend <see cref="DefaultGatedPrefixes"/> rather than reach
/// for a second registration shape.
/// </summary>
/// <param name="options">Communication options; <c>GrpcPsk</c> is the expected bearer token.</param>
/// <param name="logger">Logger for denial diagnostics.</param>
/// <param name="gatedPrefixes">Method-path prefixes to gate, e.g. <c>/sitestream.SiteStreamService/</c>.</param>
internal ControlPlaneAuthInterceptor(
IOptions<CommunicationOptions> options,
ILogger<ControlPlaneAuthInterceptor> logger,
IReadOnlyList<string> gatedPrefixes)
{
ArgumentNullException.ThrowIfNull(options);
ArgumentNullException.ThrowIfNull(logger);
ArgumentNullException.ThrowIfNull(gatedPrefixes);
_options = options;
_logger = logger;
_gatedPrefixes = gatedPrefixes;
}
/// <inheritdoc />
public override Task<TResponse> UnaryServerHandler<TRequest, TResponse>(
TRequest request,
ServerCallContext context,
UnaryServerMethod<TRequest, TResponse> continuation)
{
Authorize(context);
return continuation(request, context);
}
/// <inheritdoc />
public override Task DuplexStreamingServerHandler<TRequest, TResponse>(
IAsyncStreamReader<TRequest> requestStream,
IServerStreamWriter<TResponse> responseStream,
ServerCallContext context,
DuplexStreamingServerMethod<TRequest, TResponse> continuation)
{
Authorize(context);
return continuation(requestStream, responseStream, context);
}
/// <inheritdoc />
public override Task<TResponse> ClientStreamingServerHandler<TRequest, TResponse>(
IAsyncStreamReader<TRequest> requestStream,
ServerCallContext context,
ClientStreamingServerMethod<TRequest, TResponse> continuation)
{
Authorize(context);
return continuation(requestStream, context);
}
/// <inheritdoc />
public override Task ServerStreamingServerHandler<TRequest, TResponse>(
TRequest request,
IServerStreamWriter<TResponse> responseStream,
ServerCallContext context,
ServerStreamingServerMethod<TRequest, TResponse> continuation)
{
Authorize(context);
return continuation(request, responseStream, context);
}
/// <summary>
/// Throws <see cref="RpcException"/> with <see cref="StatusCode.PermissionDenied"/> if this
/// is a gated call that does not carry the configured bearer token. Calls to services
/// outside <c>gatedPrefixes</c> — notably LocalDb sync, which has its own interceptor —
/// return immediately.
/// </summary>
private void Authorize(ServerCallContext context)
{
if (!IsGated(context.Method))
{
return;
}
var expected = _options.Value.GrpcPsk;
if (string.IsNullOrEmpty(expected))
{
_logger.LogWarning(
"Rejected a control-plane call to {Method}: no ScadaBridge:Communication:GrpcPsk is "
+ "configured, so the control plane is closed. Set the same key here (in production, "
+ "as ${{secret:SB-GRPC-PSK-<siteId>}}) and in central's secret store.",
context.Method);
throw new RpcException(new Status(
StatusCode.PermissionDenied,
"Control plane is not accepting calls: no preshared key is configured on this node."));
}
var presented = ExtractBearerToken(context.RequestHeaders);
if (presented is null || !FixedTimeEquals(presented, expected))
{
_logger.LogWarning(
"Rejected a control-plane call to {Method}: {Reason}.",
context.Method,
presented is null ? "no bearer token presented" : "bearer token did not match");
throw new RpcException(new Status(
StatusCode.PermissionDenied,
"Control plane authentication failed."));
}
}
private bool IsGated(string method)
{
foreach (var prefix in _gatedPrefixes)
{
if (method.StartsWith(prefix, StringComparison.Ordinal))
{
return true;
}
}
return false;
}
private static string? ExtractBearerToken(Metadata headers)
{
// gRPC lowercases header keys on the wire; compare case-insensitively anyway so a
// hand-built Metadata in a test behaves the same as a real request.
foreach (var entry in headers)
{
if (!string.Equals(entry.Key, ControlPlaneCredentials.AuthorizationHeader,
StringComparison.OrdinalIgnoreCase))
{
continue;
}
var value = entry.Value;
if (value is not null && value.StartsWith("Bearer ", StringComparison.OrdinalIgnoreCase))
{
return value["Bearer ".Length..];
}
}
return null;
}
private static bool FixedTimeEquals(string presented, string expected)
=> CryptographicOperations.FixedTimeEquals(
Encoding.UTF8.GetBytes(presented), Encoding.UTF8.GetBytes(expected));
}
src/ZB.MOM.WW.ScadaBridge.Host/Health/AkkaManualFailoverService.cs
+170
View File
@@ -0,0 +1,170 @@
using System.Text.Json;
using Akka.Actor;
using Akka.Cluster;
using Microsoft.Extensions.Logging;
using ZB.MOM.WW.ScadaBridge.CentralUI.Services;
using ZB.MOM.WW.ScadaBridge.Commons.Interfaces.Services;
using ZB.MOM.WW.ScadaBridge.Commons.Types.Audit;
using ZB.MOM.WW.ScadaBridge.Commons.Types.Enums;
using ZB.MOM.WW.ScadaBridge.Communication;
using ZB.MOM.WW.ScadaBridge.Communication.ClusterState;
using ZB.MOM.WW.ScadaBridge.Host.Actors;
namespace ZB.MOM.WW.ScadaBridge.Host.Health;
/// <summary>
/// Production <see cref="IManualFailoverService"/> backed by the running Akka.NET cluster.
/// Registered only in the Central-role branch of <c>Program.cs</c>.
///
/// <para><b>Leave, never Down.</b> The target is asked to leave gracefully, so
/// <c>ClusterSingletonManager</c> hands its singletons to the survivor before the member is
/// removed. A <c>Down</c> would skip that hand-off and leave the pair to the downing strategy
/// — the wrong tool for a deliberate, planned role swap.</para>
///
/// <para><b>The target is the oldest Up member, not the leader.</b> That mirrors
/// <c>ActiveNodeEvaluator</c>'s rule, which is where the singletons actually live; Akka's
/// cluster leadership is address-ordered and diverges from singleton placement after a
/// restart (review 01 [High]).</para>
///
/// <para><b>Audit before acting.</b> The row is written before the Leave is issued. The node
/// serving this call is usually NOT the one leaving, but it can be (an admin routed to the
/// active node fails that node over), and an audit written afterwards could be lost to the
/// very shutdown it describes.</para>
/// </summary>
public sealed class AkkaManualFailoverService : IManualFailoverService
{
private readonly AkkaHostedService _akka;
private readonly ICentralAuditWriter _audit;
private readonly CommunicationService _communication;
private readonly ILogger<AkkaManualFailoverService> _logger;
/// <summary>Initializes a new <see cref="AkkaManualFailoverService"/>.</summary>
/// <param name="akka">The Akka hosted service exposing the cluster's actor system.</param>
/// <param name="audit">Central direct-write audit writer.</param>
/// <param name="communication">Central→site command/control transport, used for site failover.</param>
/// <param name="logger">Logger.</param>
public AkkaManualFailoverService(
AkkaHostedService akka,
ICentralAuditWriter audit,
CommunicationService communication,
ILogger<AkkaManualFailoverService> logger)
{
_akka = akka;
_audit = audit;
_communication = communication;
_logger = logger;
}
/// <inheritdoc />
public async Task<string?> FailOverCentralAsync(string actor)
{
var system = _akka.GetOrCreateActorSystem();
// Resolve first so the audit row can name the target, and so the peer guard rejects
// before anything observable happens.
var target = FailOverCore(system, role: CentralRole, dryRun: true);
if (target is null)
{
_logger.LogWarning(
"Manual failover requested by {Actor} but refused: fewer than 2 Up '{Role}' members, "
+ "so there is no standby to take over.", actor, CentralRole);
return null;
}
await WriteAuditAsync(actor, target);
_logger.LogWarning(
"Manual failover triggered by {Actor}: {Target} is leaving the cluster gracefully; "
+ "its singletons hand over to the standby, it restarts via its supervisor and rejoins "
+ "as the youngest member.", actor, target);
FailOverCore(system, role: CentralRole);
return target.ToString();
}
/// <inheritdoc />
public async Task<SiteFailoverOutcome> FailOverSiteAsync(string siteId, string actor)
{
// Audit BEFORE relaying, for the same reason as central: the row must exist even if
// the outcome is never observed. Unlike central, the acting node is never the one
// leaving — but a relay can still time out, and an un-acked request that DID take
// effect at the site would otherwise be unattributed.
await WriteAuditAsync(actor, target: siteId, sourceSiteId: siteId);
_logger.LogWarning(
"Manual site failover triggered by {Actor} for site {SiteId}; relaying to the site cluster.",
actor, siteId);
try
{
var ack = await _communication.TriggerSiteFailoverAsync(siteId, Guid.NewGuid().ToString());
return new SiteFailoverOutcome(ack.Accepted, ack.TargetAddress, ack.ErrorMessage);
}
catch (Exception ex)
{
// Central never buffers for an unreachable site — the Ask simply times out. Report
// that distinctly from a refusal, which is a definitive answer FROM the site.
_logger.LogWarning(ex,
"Manual site failover for {SiteId} did not get an ack from the site.", siteId);
return new SiteFailoverOutcome(
Accepted: false,
TargetAddress: null,
ErrorMessage: $"Site did not respond: {ex.Message}");
}
}
/// <summary>The Akka role scoping central-cluster membership.</summary>
private const string CentralRole = "Central";
/// <summary>
/// Oldest Up member with the role leaves. Delegates to
/// <see cref="ClusterFailoverCoordinator.FailOverOldest"/>, which lives in Communication so
/// the site-pair failover path (inside <c>SiteCommunicationActor</c>, which cannot reference
/// Host) shares one implementation of the rule.
/// </summary>
/// <param name="system">The actor system whose cluster is acted on.</param>
/// <param name="role">Role scope for membership.</param>
/// <param name="dryRun">When true, resolve and return the target without issuing the Leave.</param>
/// <returns>The address that leaves (or would leave), or null when there is no peer.</returns>
public static Address? FailOverCore(ActorSystem system, string role, bool dryRun = false)
=> ClusterFailoverCoordinator.FailOverOldest(system, role, dryRun);
/// <summary>
/// Best-effort audit row. Audit failure must never block the failover the admin asked for
/// — the same rule the rest of the Audit Log follows (audit is best-effort; the
/// user-facing action's own success path is authoritative).
/// </summary>
private Task WriteAuditAsync(string actor, Address target)
=> WriteAuditAsync(actor, target.ToString(), sourceSiteId: null);
/// <inheritdoc cref="WriteAuditAsync(string, Address)"/>
private async Task WriteAuditAsync(string actor, string target, string? sourceSiteId)
{
try
{
var evt = ScadaBridgeAuditEventFactory.Create(
channel: AuditChannel.Cluster,
kind: AuditKind.ManualFailover,
status: AuditStatus.Submitted,
actor: actor,
target: target,
sourceSiteId: sourceSiteId,
extra: JsonSerializer.Serialize(new
{
target,
// Central rows name the Central role; site rows name the site, so a query
// can tell which pair an operator moved.
scope = sourceSiteId is null ? CentralRole : $"site-{sourceSiteId}"
}));
await _audit.WriteAsync(evt);
}
catch (Exception ex)
{
_logger.LogWarning(
ex,
"Best-effort manual-failover audit emission failed (actor={Actor}, target={Target}); "
+ "the failover itself proceeds.", actor, target);
}
}
}
src/ZB.MOM.WW.ScadaBridge.Host/NodeOptions.cs
+9
View File
@@ -21,6 +21,15 @@ public class NodeOptions
/// <summary>Gets or sets the gRPC port for the site stream server.</summary>
public int GrpcPort { get; set; } = 8083;
/// <summary>
/// HTTP/2 (h2c) port the CENTRAL node listens on for the site→central
/// <c>CentralControlService</c> gRPC control plane. Default 8083 — deliberately symmetric
/// with the site <see cref="GrpcPort"/>, since a node is either central or a site and the
/// two never share a process. This listener is distinct from central's <c>:5000</c> HTTP/1
/// surface (Central UI, Management/Inbound API), which stays exactly as-is: gRPC does NOT go
/// through Traefik (HTTP/1 only). Ignored on site nodes.
/// </summary>
public int CentralGrpcPort { get; set; } = 8083;
/// <summary>
/// HTTP/1.1 port serving the Prometheus /metrics scrape endpoint on site nodes.
/// Defaults to 8084 — deliberately distinct from <see cref="RemotingPort"/> (8082)
/// and <see cref="GrpcPort"/> (8083) so the Kestrel metrics listener never contends
src/ZB.MOM.WW.ScadaBridge.Host/NodeOptionsValidator.cs
+1
View File
@@ -23,6 +23,7 @@ public sealed class NodeOptionsValidator : OptionsValidatorBase<NodeOptions>
RequirePort(builder, options.RemotingPort, nameof(NodeOptions.RemotingPort));
RequirePort(builder, options.GrpcPort, nameof(NodeOptions.GrpcPort));
RequirePort(builder, options.MetricsPort, nameof(NodeOptions.MetricsPort));
RequirePort(builder, options.CentralGrpcPort, nameof(NodeOptions.CentralGrpcPort));
}
// 0 stays valid (dynamic-port request); reject only out-of-TCP-range values.
src/ZB.MOM.WW.ScadaBridge.Host/Program.cs
+148 -6
View File
@@ -97,9 +97,76 @@ try
// Windows Service support (no-op when not running as a Windows Service)
builder.Host.UseWindowsService();
// Explicit Kestrel h2c listener for the central-hosted gRPC control plane
// (CentralControlService): HTTP/2-only, on its own port (default 8083, symmetric
// with the site GrpcPort — a node is either central or a site, never both). gRPC
// does NOT go through Traefik (HTTP/1 only); sites reach this port by container name.
//
// WARNING — the trap the rig caught (T1A.2 shipped it, fixed here): calling
// options.Listen*/ListenAnyIP puts Kestrel into EXPLICIT-ENDPOINTS mode, which
// SUPPRESSES the URLs from ASPNETCORE_URLS/--urls entirely — it is NOT additive.
// Central's ENTIRE HTTP/1 surface (Central UI, Management + Inbound API, and the
// /health/* endpoints Traefik + IActiveNodeGate depend on) lives on that URL
// (http://+:5000 on the rig, a different port in production). If we bind only the
// gRPC port, :5000 vanishes and central is reachable over gRPC while the UI, the
// management API and health checks are all dead — with no startup error. Unit tests
// use TestServer and never bind real Kestrel, so only a live node exposes this.
// The site branch has the same ConfigureKestrel shape but no ASPNETCORE_URLS surface
// to lose (it binds every port it needs — gRPC + metrics — explicitly). Central must
// therefore RE-BIND its HTTP port(s) here alongside the gRPC port.
var centralGrpcPort = configuration.GetValue<int>("ScadaBridge:Node:CentralGrpcPort", 8083);
// "urls" is WebHostDefaults.ServerUrlsKey — the host setting ASPNETCORE_URLS/--urls
// populate. Read it as a literal so this needs no extra Hosting using.
var httpUrls = configuration["ASPNETCORE_URLS"]
?? builder.WebHost.GetSetting("urls")
?? "http://+:5000";
var httpPorts = ParseHttpBindPorts(httpUrls);
builder.WebHost.ConfigureKestrel(options =>
{
// The HTTP/1.1 (+ HTTP/2) surface from the configured URLs, re-declared so it
// survives the explicit-endpoints switch above.
foreach (var httpPort in httpPorts)
{
options.ListenAnyIP(httpPort, listenOptions =>
{
listenOptions.Protocols = Microsoft.AspNetCore.Server.Kestrel.Core.HttpProtocols.Http1AndHttp2;
});
}
// The gRPC control plane, HTTP/2 h2c only, on its own port.
options.ListenAnyIP(centralGrpcPort, listenOptions =>
{
listenOptions.Protocols = Microsoft.AspNetCore.Server.Kestrel.Core.HttpProtocols.Http2;
});
});
// Shared components
builder.Services.AddClusterInfrastructure();
builder.Services.AddCommunication();
// Per-site gRPC preshared keys. Central-only: it is the side that dials sites, and
// the only side whose key set is dynamic (sites come from the configuration
// database, so there is no fixed list of ${secret:} references to expand at boot —
// hence a runtime resolver rather than the pre-host SecretReferenceExpander a site
// node uses for its single key). Registered before the clients that consume it.
builder.Services.AddSingleton<
ZB.MOM.WW.ScadaBridge.Communication.Grpc.ISitePskProvider, SitePskProvider>();
// Central-hosted gRPC control plane (T1A.2). CentralControlAuthInterceptor is the
// per-site sibling of the site's ControlPlaneAuthInterceptor: it verifies the Bearer
// token against the key for the site named in the required x-scadabridge-site header,
// resolved through the ISitePskProvider registered just above. Registered BY TYPE on
// AddGrpc exactly as the site branch registers its interceptors — NOT as a DI singleton,
// which would let DI hand the instance back and bypass Grpc.AspNetCore's own activation
// path (the shape that once hid a two-public-constructor defect until the rig caught it).
// CentralControlGrpcService decodes each request onto the same in-process message the
// ClusterClient path carries and Asks CentralCommunicationActor — zero handler logic here.
builder.Services.AddGrpc(options =>
{
options.Interceptors.Add<CentralControlAuthInterceptor>();
});
builder.Services.AddSingleton<
ZB.MOM.WW.ScadaBridge.Communication.Grpc.CentralControlGrpcService>();
builder.Services.AddHealthMonitoring();
builder.Services.AddCentralHealthAggregation();
builder.Services.AddExternalSystemGateway();
@@ -329,6 +396,12 @@ try
// which node is active.
builder.Services.AddSingleton<ZB.MOM.WW.ScadaBridge.InboundAPI.IActiveNodeGate, ActiveNodeGate>();
// Admin-triggered manual failover of the central pair (Health page control,
// decision 2026-07-22). Central-only: the seam is declared in CentralUI so that
// project stays Akka-free, and only this branch has a cluster to act on.
builder.Services.AddSingleton<
ZB.MOM.WW.ScadaBridge.CentralUI.Services.IManualFailoverService, AkkaManualFailoverService>();
// Cluster node status provider scoped to the Central role — feeds the
// CentralHealthReportLoop so the central cluster appears on /monitoring/health.
builder.Services.AddSingleton<IClusterNodeProvider>(sp =>
@@ -448,6 +521,14 @@ try
// Requires endpoint routing (app.UseRouting() above).
app.MapZbMetrics();
// Central-hosted gRPC control plane (T1A.2) — the site→central CentralControlService.
// Runs on the dedicated h2c listener configured above (default :8083), gated by
// CentralControlAuthInterceptor and readiness-gated by the service itself (Unavailable
// until AkkaHostedService hands the CentralCommunicationActor over via SetReady). It
// shares the app's endpoint routing with the HTTP/1 surface; Kestrel steers each
// connection to the right pipeline by listener/protocol.
app.MapGrpcService<ZB.MOM.WW.ScadaBridge.Communication.Grpc.CentralControlGrpcService>();
app.MapStaticAssets();
app.MapCentralUI<ZB.MOM.WW.ScadaBridge.Host.Components.App>();
app.MapInboundAPI();
@@ -513,12 +594,21 @@ try
});
});
// gRPC server registration
// The interceptor gates ONLY /localdb_sync.v1.LocalDbSync/ — SiteStream calls on
// this same pipeline pass through untouched. It is fail-closed: with no
// LocalDb:Replication:ApiKey configured, no sync stream is accepted at all.
// gRPC server registration. Two interceptors, two disjoint service prefixes, two
// separate keys — neither one's absence weakens the other:
//
// LocalDbSyncAuthInterceptor gates /localdb_sync.v1.LocalDbSync/ (LocalDb:Replication:ApiKey)
// ControlPlaneAuthInterceptor gates /sitestream.SiteStreamService/ (ScadaBridge:Communication:GrpcPsk)
//
// Both are fail-closed: an unset key closes that surface rather than opening it. For
// LocalDb that means replication simply does not start; for the control plane it means
// this node serves no streams and no audit pulls until a key is configured, which is
// why every environment must carry one before running this build.
builder.Services.AddGrpc(options =>
options.Interceptors.Add<LocalDbSyncAuthInterceptor>());
{
options.Interceptors.Add<LocalDbSyncAuthInterceptor>();
options.Interceptors.Add<ControlPlaneAuthInterceptor>();
});
builder.Services.AddSingleton<ZB.MOM.WW.ScadaBridge.Communication.Grpc.SiteStreamGrpcServer>();
// Existing site service registrations (this is also where LocalDb and its
@@ -580,4 +670,56 @@ finally
/// <summary>
/// Exposes the auto-generated Program class for test infrastructure (e.g. WebApplicationFactory).
/// </summary>
public partial class Program { }
public partial class Program
{
/// <summary>
/// Extracts the distinct TCP ports from an ASP.NET Core server-URLs string (the
/// <c>ASPNETCORE_URLS</c> / <c>--urls</c> value, e.g. <c>"http://+:5000"</c> or a
/// semicolon-separated list). Used to re-declare central's HTTP surface after the gRPC
/// listener switches Kestrel into explicit-endpoints mode — see the call site's warning.
/// </summary>
/// <remarks>
/// Bind hosts (<c>+</c>, <c>*</c>, <c>0.0.0.0</c>, <c>[::]</c>, a hostname) are irrelevant
/// here because the caller re-binds via <c>ListenAnyIP</c>; only the port matters. A URL
/// with no explicit port falls back to the scheme default (80/443). Unparseable entries
/// are skipped rather than throwing — a bad URL should not take the node down at boot.
/// </remarks>
/// <param name="serverUrls">The server-URLs string; may be null/empty.</param>
/// <returns>The distinct ports, in first-seen order.</returns>
internal static IReadOnlyList<int> ParseHttpBindPorts(string? serverUrls)
{
var ports = new List<int>();
if (string.IsNullOrWhiteSpace(serverUrls))
{
return ports;
}
foreach (var raw in serverUrls.Split(';', StringSplitOptions.RemoveEmptyEntries | StringSplitOptions.TrimEntries))
{
int port;
// Uri can't parse the wildcard hosts Kestrel accepts (+, *), so normalize them
// to a placeholder host before parsing; the host is discarded anyway.
var normalized = raw.Replace("://+", "://placeholder").Replace("://*", "://placeholder");
if (Uri.TryCreate(normalized, UriKind.Absolute, out var uri))
{
port = uri.Port; // Uri fills the scheme default (80/443) when none is given.
}
else
{
// Last-ditch: pull the port after the final ':' (handles odd inputs Uri rejects).
var colon = raw.LastIndexOf(':');
if (colon < 0 || !int.TryParse(raw.AsSpan(colon + 1), out port))
{
continue;
}
}
if (port is > 0 and <= 65535 && !ports.Contains(port))
{
ports.Add(port);
}
}
return ports;
}
}
src/ZB.MOM.WW.ScadaBridge.Host/SitePskProvider.cs
+149
View File
@@ -0,0 +1,149 @@
using System.Collections.Concurrent;
using Microsoft.Extensions.Logging;
using Microsoft.Extensions.Options;
using ZB.MOM.WW.ScadaBridge.Communication;
using ZB.MOM.WW.ScadaBridge.Communication.Grpc;
using ZB.MOM.WW.Secrets.Abstractions;
namespace ZB.MOM.WW.ScadaBridge.Host;
/// <summary>
/// Central's <see cref="ISitePskProvider"/>: resolves each site's gRPC preshared key at
/// channel-build time — from <c>ScadaBridge:Communication:SitePsks</c> if the site is listed
/// there, otherwise from the secrets store under the name <c>SB-GRPC-PSK-{siteId}</c>.
/// </summary>
/// <remarks>
/// <para>
/// <b>Why resolved at runtime rather than expanded into config at boot.</b> Site nodes get their
/// single key through the pre-host <c>SecretReferenceExpander</c> — a <c>${secret:…}</c> in
/// appsettings, resolved once before the host is built. Central cannot do that: its set of sites
/// comes from the configuration database and changes while the process runs, so there is no
/// fixed list of references to expand at boot. This mirrors the pattern the MxGateway data
/// connection already uses for its per-connection API keys.
/// </para>
/// <para>
/// <b>Caching.</b> Successful resolves are cached indefinitely; a site's key changes only on
/// rotation, and rotation restarts the pair. Failures are deliberately NOT cached, so a key
/// seeded after the first (failed) dial is picked up on the next attempt instead of requiring a
/// restart. <see cref="Invalidate"/> drops an entry when a site is removed or a key is rotated
/// in place.
/// </para>
/// <para>
/// <b>Fail-closed.</b> A key absent from BOTH sources throws, and the dial that needed it fails.
/// It never degrades to an unauthenticated call. The message names the exact config key and the
/// exact secret name, so the fix is one setting or one <c>secret</c> CLI seed.
/// </para>
/// </remarks>
public sealed class SitePskProvider : ISitePskProvider
{
/// <summary>Prefix of the per-site secret name; the site identifier is appended verbatim.</summary>
public const string SecretNamePrefix = "SB-GRPC-PSK-";
private readonly ConcurrentDictionary<string, string> _cache = new(StringComparer.Ordinal);
private readonly ISecretResolver _resolver;
private readonly IOptionsMonitor<CommunicationOptions> _options;
private readonly ILogger<SitePskProvider> _logger;
/// <summary>Creates the provider over the host's secret resolver.</summary>
/// <param name="resolver">Runtime secret resolver from the host container.</param>
/// <param name="options">
/// Communication options supplying the optional <c>SitePsks</c> map. Read through an
/// <see cref="IOptionsMonitor{TOptions}"/> so a reloaded configuration is honoured on the
/// next uncached resolve rather than pinned at construction.
/// </param>
/// <param name="logger">Logger for resolution diagnostics.</param>
public SitePskProvider(
ISecretResolver resolver,
IOptionsMonitor<CommunicationOptions> options,
ILogger<SitePskProvider> logger)
{
ArgumentNullException.ThrowIfNull(resolver);
ArgumentNullException.ThrowIfNull(options);
ArgumentNullException.ThrowIfNull(logger);
_resolver = resolver;
_options = options;
_logger = logger;
}
/// <summary>Builds the secret name for a site.</summary>
/// <param name="siteId">Site identifier.</param>
/// <returns>The secret name, e.g. <c>SB-GRPC-PSK-site-a</c>.</returns>
public static string SecretNameFor(string siteId) => SecretNamePrefix + siteId;
/// <inheritdoc />
public async ValueTask<string> GetAsync(string siteId, CancellationToken ct)
{
ArgumentException.ThrowIfNullOrWhiteSpace(siteId);
if (_cache.TryGetValue(siteId, out var cached))
{
return cached;
}
// Configured map first: it is the explicit, operator-stated answer for this site, and
// it is the only source available to a host running without a secrets master key.
var configured = _options.CurrentValue.SitePsks;
if (configured.TryGetValue(siteId, out var fromConfig) && !string.IsNullOrEmpty(fromConfig))
{
_cache[siteId] = fromConfig;
return fromConfig;
}
// Otherwise the store, which is the only source that can serve a site added at runtime.
var secretName = SecretNameFor(siteId);
var value = await ResolveFromStoreAsync(secretName, siteId, ct).ConfigureAwait(false);
if (string.IsNullOrEmpty(value))
{
_logger.LogError(
"gRPC control-plane key for site {SiteId} could not be resolved: it is absent from "
+ "ScadaBridge:Communication:SitePsks and secret '{SecretName}' is missing, empty or "
+ "tombstoned. Calls to this site are refused until one of the two is set (the site "
+ "node must carry the same value in ScadaBridge:Communication:GrpcPsk).",
siteId, secretName);
throw new InvalidOperationException(
$"gRPC preshared key for site '{siteId}' could not be resolved from "
+ $"ScadaBridge:Communication:SitePsks or secret '{secretName}'.");
}
// Two concurrent first dials may both resolve; the read is idempotent and the values
// identical, so the loser simply overwrites with the same string.
_cache[siteId] = value;
return value;
}
/// <summary>
/// Reads the key from the secrets store, treating a store fault as "not found" rather than
/// letting it propagate. A host with no master key configured (the development rig) throws
/// from the resolver; that must produce the same clear "key not configured" error as a
/// missing secret, not an opaque cryptographic one.
/// </summary>
private async Task<string?> ResolveFromStoreAsync(string secretName, string siteId, CancellationToken ct)
{
try
{
return await _resolver.GetAsync(new SecretName(secretName), ct).ConfigureAwait(false);
}
catch (OperationCanceledException)
{
throw;
}
catch (Exception ex)
{
_logger.LogDebug(ex,
"Secret store lookup for '{SecretName}' (site {SiteId}) faulted; treating as not found.",
secretName, siteId);
return null;
}
}
/// <inheritdoc />
public void Invalidate(string siteId)
{
if (!string.IsNullOrWhiteSpace(siteId))
{
_cache.TryRemove(siteId, out _);
}
}
}
src/ZB.MOM.WW.ScadaBridge.Host/StartupValidator.cs
+62
View File
@@ -93,6 +93,22 @@ public static class StartupValidator
.Require("ScadaBridge:Cluster:SeedNodes",
_ => seedNodes != null && seedNodes.Count >= 2,
"must have at least 2 entries")
// Self-first seed ordering (decision 2026-07-22). Akka runs FirstSeedNodeProcess —
// the ONLY bootstrap path that can form a new cluster when no peer answers InitJoin —
// exclusively when seed-nodes[0] is this node's own address. Every other node runs
// JoinSeedNodeProcess and retries InitJoin forever, so a node listing its partner
// first cannot cold-start alone: that is the "registered outage gap"
// (docker/README.md), and it is a silent failure at boot rather than a loud one.
// Enforced here rather than in ClusterOptionsValidator because only this validator
// sees both the node identity and the seed list.
.Require("ScadaBridge:Cluster:SeedNodes",
_ => seedNodes is not { Count: >= 1 }
|| SeedNodeIsSelf(seedNodes[0], nodeSection["NodeHostname"], port),
"must list this node itself first: seed-nodes[0] has to be this node's own "
+ $"'akka.tcp://scadabridge@{nodeSection["NodeHostname"]}:{port}'. Akka only lets "
+ "seed-nodes[0] form a new cluster, so with the partner listed first this node "
+ "can never start while its peer is down (Component-ClusterInfrastructure.md → "
+ "Seed Node Ordering)")
// The big Site-only block: GrpcPort/MetricsPort validity + cross-field
// collisions + seed-node-port loop, in the original order.
.When(role == "Site", p =>
@@ -110,6 +126,23 @@ public static class StartupValidator
p.Require("ScadaBridge:Node:MetricsPort", _ => metricsPort != port, "must differ from RemotingPort");
p.Require("ScadaBridge:Node:MetricsPort", _ => metricsPort != grpcPort, "must differ from GrpcPort");
// The gRPC control-plane preshared key. Same argument as the inbound
// API-key pepper above: without it the node boots and looks healthy, but
// ControlPlaneAuthInterceptor is fail-closed, so every SiteStream call —
// live subscriptions, audit pulls, cached-telemetry ingest — is refused
// with PermissionDenied. A silent, total loss of the site's central-facing
// surface is far worse than a loud boot failure, so require it here.
// Central holds the matching value per site in its secret store as
// SB-GRPC-PSK-{SiteId}; production supplies this one as
// ${secret:SB-GRPC-PSK-<siteId>}, expanded before the host is built.
p.Require("ScadaBridge:Communication:GrpcPsk",
value => !string.IsNullOrWhiteSpace(value),
"is required for Site nodes: it is the preshared key the gRPC control "
+ "plane authenticates with, and the interceptor is fail-closed, so an "
+ "unset key refuses every SiteStream call. Set the same value here (in "
+ "production as ${secret:SB-GRPC-PSK-<siteId>}) and under the secret "
+ "name SB-GRPC-PSK-<siteId> in central's secret store");
// ScadaBridge:Database:SiteDbPath was required here until LocalDb
// Phase 2. The site's tables now live in the consolidated LocalDb
// database (LocalDb:Path, which SiteServiceRegistration requires),
@@ -155,4 +188,33 @@ public static class StartupValidator
return int.TryParse(seedNode[(lastColon + 1)..], out var port) ? port : -1;
}
/// <summary>
/// Extracts the host from an Akka seed-node address of the form
/// <c>akka.tcp://system@host:port</c>. Returns an empty string when no host can be parsed.
/// </summary>
private static string SeedNodeHost(string seedNode)
{
if (string.IsNullOrWhiteSpace(seedNode))
return string.Empty;
var at = seedNode.LastIndexOf('@');
var lastColon = seedNode.LastIndexOf(':');
if (at < 0 || lastColon <= at)
return string.Empty;
return seedNode[(at + 1)..lastColon];
}
/// <summary>
/// True when <paramref name="seedNode"/> addresses this node itself (host AND port).
/// Host comparison is case-insensitive because DNS names are; it is otherwise exact —
/// Akka does no DNS canonicalisation either, so <c>node-a</c> and
/// <c>node-a.example.com</c> are genuinely different seed identities to the cluster.
/// </summary>
private static bool SeedNodeIsSelf(string seedNode, string? nodeHostname, int remotingPort)
{
return SeedNodePort(seedNode) == remotingPort
&& string.Equals(SeedNodeHost(seedNode), nodeHostname, StringComparison.OrdinalIgnoreCase);
}
}
src/ZB.MOM.WW.ScadaBridge.Host/appsettings.Central.json
+1 -1
View File
@@ -13,7 +13,7 @@
"akka.tcp://scadabridge@localhost:8081",
"akka.tcp://scadabridge@localhost:8082"
],
"SplitBrainResolverStrategy": "keep-oldest",
"SplitBrainResolverStrategy": "auto-down",
"StableAfter": "00:00:15",
"HeartbeatInterval": "00:00:02",
"FailureDetectionThreshold": "00:00:10",
src/ZB.MOM.WW.ScadaBridge.Host/appsettings.Site.json
+3 -1
View File
@@ -16,7 +16,7 @@
"akka.tcp://scadabridge@localhost:8082",
"akka.tcp://scadabridge@localhost:8085"
],
"SplitBrainResolverStrategy": "keep-oldest",
"SplitBrainResolverStrategy": "auto-down",
"StableAfter": "00:00:15",
"HeartbeatInterval": "00:00:02",
"FailureDetectionThreshold": "00:00:10",
@@ -42,6 +42,8 @@
"SqliteDbPath": "./data/store-and-forward.db"
},
"Communication": {
"_grpcPsk": "REQUIRED on Site nodes (StartupValidator fails the boot without it). The preshared key the gRPC control plane authenticates with: ControlPlaneAuthInterceptor is fail-closed, so an unset key refuses every SiteStream call — live subscriptions, audit pulls, cached-telemetry ingest — while the node still reports healthy. Supply it as ${secret:SB-GRPC-PSK-<siteId>} so the plaintext never sits in this file, and seed the SAME value on central: either as the secret SB-GRPC-PSK-<siteId> in its store, or as ScadaBridge:Communication:SitePsks:<siteId>. BOTH nodes of the pair carry the same key. Distinct from LocalDb:Replication:ApiKey, which authenticates the pair partner, not central — never share the two.",
"GrpcPsk": "${secret:SB-GRPC-PSK-site-1}",
"_centralContactPoints": "Host-016: each entry MUST be a central node's remoting endpoint, NOT this site's own remoting port. The single dev-loopback default below points only at central-a (localhost:8081). In a multi-central deployment add the second central node here (e.g. 'akka.tcp://scadabridge@central-b-host:8081') so ClusterClient can fail over when central-a is down. The previous template listed localhost:8082 as the second contact — that is THIS site's own RemotingPort and is a permanent failure in the initial-contact rotation.",
"CentralContactPoints": [
"akka.tcp://scadabridge@localhost:8081"
tests/ZB.MOM.WW.ScadaBridge.AuditLog.Tests/Central/GrpcPullAuditEventsClientTests.cs
+2 -2
View File
@@ -58,7 +58,7 @@ public class GrpcPullAuditEventsClientTests
public static FakeInvoker Throwing(Exception ex) => new(null, ex);
public Task<ProtoPullResponse> InvokeAsync(
string endpoint, ProtoPullRequest request, CancellationToken ct)
string siteId, string endpoint, ProtoPullRequest request, CancellationToken ct)
{
CallCount++;
Endpoint = endpoint;
@@ -84,7 +84,7 @@ public class GrpcPullAuditEventsClientTests
_byEndpoint = byEndpoint;
public Task<ProtoPullResponse> InvokeAsync(
string endpoint, ProtoPullRequest request, CancellationToken ct)
string siteId, string endpoint, ProtoPullRequest request, CancellationToken ct)
{
Dialed.Add(endpoint);
return Task.FromResult(_byEndpoint[endpoint]());
tests/ZB.MOM.WW.ScadaBridge.AuditLog.Tests/Central/GrpcPullSiteCallsClientTests.cs
+2 -2
View File
@@ -50,7 +50,7 @@ public class GrpcPullSiteCallsClientTests
public static FakeInvoker Throwing(Exception ex) => new(null, ex);
public Task<ProtoPullResponse> InvokeAsync(
string endpoint, ProtoPullRequest request, CancellationToken ct)
string siteId, string endpoint, ProtoPullRequest request, CancellationToken ct)
{
CallCount++;
Endpoint = endpoint;
@@ -77,7 +77,7 @@ public class GrpcPullSiteCallsClientTests
_byEndpoint = byEndpoint;
public Task<ProtoPullResponse> InvokeAsync(
string endpoint, ProtoPullRequest request, CancellationToken ct)
string siteId, string endpoint, ProtoPullRequest request, CancellationToken ct)
{
Dialed.Add(endpoint);
// The Func may throw (transport fault) — the client's try/catch handles it.
tests/ZB.MOM.WW.ScadaBridge.AuditLog.Tests/Site/Telemetry/SiteAuditTelemetryActorTests.cs
+4 -137
View File
@@ -362,22 +362,11 @@ public class SiteAuditTelemetryActorTests : TestKit
}
[Fact]
public async Task CachedDrain_OrphanRow_PastGrace_IsAbandoned_AndTheValidRowStillFlows()
public async Task CachedDrain_OrphanRow_NoTrackingSnapshot_IsSkipped_DoesNotCrash()
{
// RETARGETED, not rewritten from scratch: this test previously asserted
// the orphan "is skipped and stays Pending", which is precisely the
// defect fixed here (known-issues/2026-07-20-cached-telemetry-drain-
// hot-loop.md) — a Pending row that can never resolve is re-read every
// drain forever and starves the queue behind it. The half of the test
// that still holds (the valid row flows, the drain does not crash) is
// kept verbatim; only the orphan's fate is inverted.
//
// NewCachedEvent stamps occurredAtUtc in the past, well beyond the
// default 300 s grace, so this orphan is past the abandon threshold.
//
// Arrange — two cached audit rows: one with a tracking snapshot, one
// orphaned (the tracking store returns null). The orphaned row must be
// abandoned without aborting the batch — the valid row still flows.
// skipped without aborting the batch — the valid row still flows.
var orphan = NewCachedEvent(AuditKind.CachedSubmit);
var valid = NewCachedEvent(AuditKind.CachedResolve);
@@ -414,8 +403,8 @@ public class SiteAuditTelemetryActorTests : TestKit
// Act
CreateActorWithCachedDrain();
// Assert — exactly one push containing ONLY the valid row: the orphan
// has no operational half to send.
// Assert — exactly one push containing ONLY the valid row; the orphan
// is skipped and stays Pending (not in MarkForwardedAsync's id list).
await AwaitAssertAsync(async () =>
{
await _client.Received(1).IngestCachedTelemetryAsync(
@@ -426,131 +415,9 @@ public class SiteAuditTelemetryActorTests : TestKit
Assert.Single(capturedBatch!.Packets);
Assert.Equal(valid.EventId.ToString(), capturedBatch.Packets[0].AuditEvent.EventId);
// The valid row is marked Forwarded because central ack'd it...
await _queue.Received(1).MarkForwardedAsync(
Arg.Is<IReadOnlyList<Guid>>(g => g.Count == 1 && g[0] == valid.EventId),
Arg.Any<CancellationToken>());
// ...and the orphan is marked Forwarded too, in its own call, so it
// LEAVES the cached queue. Its audit half is still delivered by the
// reconciliation pull, which covers Forwarded rows as well as Pending.
await _queue.Received(1).MarkForwardedAsync(
Arg.Is<IReadOnlyList<Guid>>(g => g.Count == 1 && g[0] == orphan.EventId),
Arg.Any<CancellationToken>());
}
[Fact]
public async Task CachedDrain_OrphanRow_InsideGrace_IsRetried_NotAbandoned()
{
// The other side of the grace window, and the reason abandonment is not
// immediate: a missing snapshot is normally a brief write race (the
// audit row lands microseconds before the tracking row). Abandoning on
// the first failed lookup would throw away the operational half of
// every cached call that happened to lose that race.
var fresh = ScadaBridgeAuditEventFactory.Create(
channel: AuditChannel.ApiOutbound,
kind: AuditKind.CachedSubmit,
status: AuditStatus.Submitted,
eventId: Guid.NewGuid(),
occurredAtUtc: DateTime.UtcNow, // inside the grace window
target: "ERP.GetOrder",
sourceSiteId: "site-1",
correlationId: Guid.NewGuid());
_queue.ReadPendingAsync(Arg.Any<int>(), Arg.Any<CancellationToken>())
.Returns(Task.FromResult<IReadOnlyList<AuditEvent>>(Array.Empty<AuditEvent>()));
_queue.ReadPendingCachedTelemetryAsync(Arg.Any<int>(), Arg.Any<CancellationToken>())
.Returns(Task.FromResult<IReadOnlyList<AuditEvent>>(new[] { fresh }));
_trackingStore.GetStatusAsync(
new TrackedOperationId(fresh.CorrelationId!.Value), Arg.Any<CancellationToken>())
.Returns(Task.FromResult<TrackingStatusSnapshot?>(null));
CreateActorWithCachedDrain();
// Give the drain several ticks to prove it keeps retrying rather than
// abandoning. A positive control guards against vacuity: the row must
// actually have been READ more than once, otherwise "never marked
// Forwarded" would also be true of a drain that never ran at all.
await AwaitAssertAsync(async () =>
{
await _queue.Received(Quantity.Within(2, int.MaxValue))
.ReadPendingCachedTelemetryAsync(Arg.Any<int>(), Arg.Any<CancellationToken>());
}, TimeSpan.FromSeconds(6));
await _queue.DidNotReceive().MarkForwardedAsync(
Arg.Any<IReadOnlyList<Guid>>(), Arg.Any<CancellationToken>());
}
[Fact]
public async Task CachedDrain_FullBatchOfUnresolvableRows_DoesNotStarveTheQueue()
{
// The actual regression. The queue is read oldest-first with a fixed
// BatchSize, so before the fix a batch's worth of permanently
// unresolvable rows at the head meant every subsequent drain re-read
// exactly those rows, failed identically, and never reached the newer
// rows behind them — a permanent stall of the cached-telemetry path,
// plus one log line per row per pass (~2,800/min measured on a rig).
var stuck = Enumerable.Range(0, 4).Select(_ => NewCachedEvent()).ToList();
_queue.ReadPendingAsync(Arg.Any<int>(), Arg.Any<CancellationToken>())
.Returns(Task.FromResult<IReadOnlyList<AuditEvent>>(Array.Empty<AuditEvent>()));
_queue.ReadPendingCachedTelemetryAsync(Arg.Any<int>(), Arg.Any<CancellationToken>())
.Returns(Task.FromResult<IReadOnlyList<AuditEvent>>(stuck));
foreach (var row in stuck)
{
_trackingStore.GetStatusAsync(
new TrackedOperationId(row.CorrelationId!.Value), Arg.Any<CancellationToken>())
.Returns(Task.FromResult<TrackingStatusSnapshot?>(null));
}
CreateActorWithCachedDrain();
// Every row is abandoned in ONE call, so the head of the queue clears
// and the next drain can see past it.
await AwaitAssertAsync(async () =>
{
await _queue.Received(1).MarkForwardedAsync(
Arg.Is<IReadOnlyList<Guid>>(g => g.Count == stuck.Count),
Arg.Any<CancellationToken>());
}, TimeSpan.FromSeconds(5));
// Nothing was pushed — there was no operational half to build for any
// of them — but the drain still made progress rather than spinning.
await _client.DidNotReceive().IngestCachedTelemetryAsync(
Arg.Any<CachedTelemetryBatch>(), Arg.Any<CancellationToken>());
}
[Fact]
public async Task CachedDrain_TrackingStoreThrow_DoesNotAbandonTheRow()
{
// A throw is a STORE fault (locked, corrupt, mid-restore), not a
// verdict about this row: the snapshot may well exist and be readable
// a second later. Abandoning here would discard the operational half
// of every in-flight cached call during a transient SQLite lock — so
// the throw path deliberately never abandons, however old the row is.
var row = NewCachedEvent(); // deliberately older than the grace window
_queue.ReadPendingAsync(Arg.Any<int>(), Arg.Any<CancellationToken>())
.Returns(Task.FromResult<IReadOnlyList<AuditEvent>>(Array.Empty<AuditEvent>()));
_queue.ReadPendingCachedTelemetryAsync(Arg.Any<int>(), Arg.Any<CancellationToken>())
.Returns(Task.FromResult<IReadOnlyList<AuditEvent>>(new[] { row }));
_trackingStore.GetStatusAsync(
new TrackedOperationId(row.CorrelationId!.Value), Arg.Any<CancellationToken>())
.ThrowsAsync(new InvalidOperationException("tracking store unavailable"));
CreateActorWithCachedDrain();
// Positive control again: prove the drain actually ran repeatedly, so
// "never abandoned" is a real observation rather than an artefact of a
// drain that never executed.
await AwaitAssertAsync(async () =>
{
await _queue.Received(Quantity.Within(2, int.MaxValue))
.ReadPendingCachedTelemetryAsync(Arg.Any<int>(), Arg.Any<CancellationToken>());
}, TimeSpan.FromSeconds(6));
await _queue.DidNotReceive().MarkForwardedAsync(
Arg.Any<IReadOnlyList<Guid>>(), Arg.Any<CancellationToken>());
}
[Fact]
tests/ZB.MOM.WW.ScadaBridge.CentralUI.Tests/Monitoring/HealthFailoverButtonTests.cs
+241
View File
@@ -0,0 +1,241 @@
using System.Security.Claims;
using Bunit;
using Microsoft.AspNetCore.Authorization;
using Microsoft.AspNetCore.Components;
using Microsoft.AspNetCore.Components.Authorization;
using Microsoft.Extensions.DependencyInjection;
using NSubstitute;
using ZB.MOM.WW.ScadaBridge.CentralUI.Components.Health;
using ZB.MOM.WW.ScadaBridge.CentralUI.Components.Shared;
using ZB.MOM.WW.ScadaBridge.CentralUI.Services;
using ZB.MOM.WW.ScadaBridge.Security;
namespace ZB.MOM.WW.ScadaBridge.CentralUI.Tests.Monitoring;
/// <summary>
/// bUnit tests for the admin "Trigger failover" control on the Health dashboard
/// (decision 2026-07-22). The control is destructive and privileged, so the three
/// things that must hold are: only an Administrator sees it, it is refused when
/// there is no standby to take over, and it never fires without an explicit
/// confirmation.
/// </summary>
public class HealthFailoverButtonTests : BunitContext
{
private readonly IManualFailoverService _failover = Substitute.For<IManualFailoverService>();
private readonly IDialogService _dialog = Substitute.For<IDialogService>();
private void Arrange(string role)
{
var claims = new[]
{
new Claim(JwtTokenService.UsernameClaimType, "tester"),
new Claim(JwtTokenService.RoleClaimType, role)
};
var user = new ClaimsPrincipal(new ClaimsIdentity(claims, "TestAuth"));
Services.AddSingleton<AuthenticationStateProvider>(new TestAuthStateProvider(user));
Services.AddAuthorizationCore();
AuthorizationPolicies.AddScadaBridgeAuthorization(Services);
// BunitContext pre-registers a placeholder IAuthorizationService that throws when
// AuthorizeView evaluates a policy. Force the real service so the admin gate is
// genuinely exercised rather than trivially satisfied (same note as NavMenuTests).
Services.AddSingleton<IAuthorizationService, DefaultAuthorizationService>();
Services.AddSingleton(_failover);
Services.AddSingleton(_dialog);
}
/// <summary>
/// AuthorizeView requires a cascading Task&lt;AuthenticationState&gt;, which the app
/// supplies from its layout; a bare component render has to provide it explicitly.
/// </summary>
private IRenderedComponent<CentralFailoverControl> Render(int onlineNodes)
{
var host = Render<CascadingAuthenticationState>(parameters => parameters
.Add(p => p.ChildContent, (RenderFragment)(builder =>
{
builder.OpenComponent<CentralFailoverControl>(0);
builder.AddAttribute(1, nameof(CentralFailoverControl.OnlineCentralNodeCount), onlineNodes);
builder.CloseComponent();
})));
return host.FindComponent<CentralFailoverControl>();
}
[Fact]
public void Button_is_hidden_for_a_non_admin()
{
Arrange("Viewer");
var cut = Render(onlineNodes: 2);
Assert.Empty(cut.FindAll("button"));
}
[Fact]
public void Button_is_rendered_and_enabled_for_an_admin_with_a_standby()
{
Arrange("Administrator");
var cut = Render(onlineNodes: 2);
var button = cut.Find("button");
Assert.False(button.HasAttribute("disabled"));
Assert.Contains("failover", button.TextContent, StringComparison.OrdinalIgnoreCase);
}
[Fact]
public void Button_is_disabled_with_an_explanatory_title_when_there_is_no_standby()
{
Arrange("Administrator");
var cut = Render(onlineNodes: 1);
var button = cut.Find("button");
Assert.True(button.HasAttribute("disabled"));
// The tooltip must say WHY, not just that it is unavailable.
Assert.Contains("outage", button.GetAttribute("title"), StringComparison.OrdinalIgnoreCase);
}
[Fact]
public async Task Confirming_triggers_the_failover_exactly_once()
{
Arrange("Administrator");
_dialog.ConfirmAsync(Arg.Any<string>(), Arg.Any<string>(), Arg.Any<bool>()).Returns(true);
_failover.FailOverCentralAsync(Arg.Any<string>())
.Returns(Task.FromResult<string?>("akka.tcp://scadabridge@central-a:8081"));
var cut = Render(onlineNodes: 2);
await cut.Find("button").ClickAsync(new());
await _failover.Received(1).FailOverCentralAsync("tester");
}
[Fact]
public async Task Cancelling_the_confirmation_does_not_trigger_a_failover()
{
Arrange("Administrator");
_dialog.ConfirmAsync(Arg.Any<string>(), Arg.Any<string>(), Arg.Any<bool>()).Returns(false);
var cut = Render(onlineNodes: 2);
await cut.Find("button").ClickAsync(new());
await _failover.DidNotReceive().FailOverCentralAsync(Arg.Any<string>());
}
[Fact]
public async Task Confirmation_warns_that_this_page_will_disconnect()
{
// Traefik routes the UI to the ACTIVE node — the very node being failed over — so
// the admin's own circuit drops. If the dialog does not say so, a working failover
// looks like a crash they caused.
Arrange("Administrator");
_dialog.ConfirmAsync(Arg.Any<string>(), Arg.Any<string>(), Arg.Any<bool>()).Returns(true);
var cut = Render(onlineNodes: 2);
await cut.Find("button").ClickAsync(new());
await _dialog.Received(1).ConfirmAsync(
Arg.Any<string>(),
Arg.Is<string>(m => m.Contains("reconnect", StringComparison.OrdinalIgnoreCase)),
Arg.Any<bool>());
}
[Fact]
public async Task A_refused_failover_surfaces_the_refusal_instead_of_claiming_success()
{
// The service returns null when the peer guard refuses. The UI must not report a
// failover that never happened.
Arrange("Administrator");
_dialog.ConfirmAsync(Arg.Any<string>(), Arg.Any<string>(), Arg.Any<bool>()).Returns(true);
_failover.FailOverCentralAsync(Arg.Any<string>()).Returns(Task.FromResult<string?>(null));
var cut = Render(onlineNodes: 2);
await cut.Find("button").ClickAsync(new());
Assert.Contains("no standby", cut.Markup, StringComparison.OrdinalIgnoreCase);
}
// ---- Site-pair failover (Task 10) ------------------------------------------
// Same control, SiteId set. The differences that matter: it calls the site path,
// and its confirmation must NOT claim the admin's own page will drop (it won't —
// the site is a different cluster).
private IRenderedComponent<CentralFailoverControl> RenderForSite(int onlineNodes, string siteId)
{
var host = Render<CascadingAuthenticationState>(parameters => parameters
.Add(p => p.ChildContent, (RenderFragment)(builder =>
{
builder.OpenComponent<CentralFailoverControl>(0);
builder.AddAttribute(1, nameof(CentralFailoverControl.OnlineCentralNodeCount), onlineNodes);
builder.AddAttribute(2, nameof(CentralFailoverControl.SiteId), siteId);
builder.CloseComponent();
})));
return host.FindComponent<CentralFailoverControl>();
}
[Fact]
public async Task Site_card_confirming_triggers_the_site_failover_path()
{
Arrange("Administrator");
_dialog.ConfirmAsync(Arg.Any<string>(), Arg.Any<string>(), Arg.Any<bool>()).Returns(true);
_failover.FailOverSiteAsync(Arg.Any<string>(), Arg.Any<string>())
.Returns(Task.FromResult(new SiteFailoverOutcome(true, "akka.tcp://scadabridge@site-a-a:8082", null)));
var cut = RenderForSite(onlineNodes: 2, siteId: "SiteA");
await cut.Find("button").ClickAsync(new());
await _failover.Received(1).FailOverSiteAsync("SiteA", "tester");
await _failover.DidNotReceive().FailOverCentralAsync(Arg.Any<string>());
}
[Fact]
public async Task Site_card_confirmation_does_not_claim_this_page_disconnects()
{
// Central failover drops the admin's circuit; a SITE failover does not, and saying
// otherwise would train operators to distrust the warning that does matter.
Arrange("Administrator");
_dialog.ConfirmAsync(Arg.Any<string>(), Arg.Any<string>(), Arg.Any<bool>()).Returns(true);
_failover.FailOverSiteAsync(Arg.Any<string>(), Arg.Any<string>())
.Returns(Task.FromResult(new SiteFailoverOutcome(true, "addr", null)));
var cut = RenderForSite(onlineNodes: 2, siteId: "SiteA");
await cut.Find("button").ClickAsync(new());
await _dialog.Received(1).ConfirmAsync(
Arg.Any<string>(),
Arg.Is<string>(m => !m.Contains("reconnect", StringComparison.OrdinalIgnoreCase)),
Arg.Any<bool>());
}
[Fact]
public async Task Site_card_surfaces_a_refusal_from_the_site()
{
Arrange("Administrator");
_dialog.ConfirmAsync(Arg.Any<string>(), Arg.Any<string>(), Arg.Any<bool>()).Returns(true);
_failover.FailOverSiteAsync(Arg.Any<string>(), Arg.Any<string>())
.Returns(Task.FromResult(new SiteFailoverOutcome(
false, null, "No standby available — failing over a lone node would be an outage.")));
var cut = RenderForSite(onlineNodes: 2, siteId: "SiteA");
await cut.Find("button").ClickAsync(new());
Assert.Contains("No standby available", cut.Markup);
}
[Fact]
public async Task Site_card_surfaces_an_unreachable_site_distinctly_from_a_refusal()
{
// A timeout is not a "no" from the site — the operator must be able to tell the
// difference, because the failover may or may not have taken effect.
Arrange("Administrator");
_dialog.ConfirmAsync(Arg.Any<string>(), Arg.Any<string>(), Arg.Any<bool>()).Returns(true);
_failover.FailOverSiteAsync(Arg.Any<string>(), Arg.Any<string>())
.Returns(Task.FromResult(new SiteFailoverOutcome(
false, null, "Site did not respond: Ask timed out")));
var cut = RenderForSite(onlineNodes: 2, siteId: "SiteA");
await cut.Find("button").ClickAsync(new());
Assert.Contains("did not respond", cut.Markup);
}
}
tests/ZB.MOM.WW.ScadaBridge.CentralUI.Tests/Pages/HealthPageTests.cs
+45 -11
View File
@@ -2,11 +2,14 @@ using System.Security.Claims;
using ZB.MOM.WW.ScadaBridge.Security;
using Akka.Actor;
using Bunit;
using Microsoft.AspNetCore.Authorization;
using Microsoft.AspNetCore.Components;
using Microsoft.AspNetCore.Components.Authorization;
using Microsoft.Extensions.DependencyInjection;
using Microsoft.Extensions.Logging.Abstractions;
using Microsoft.Extensions.Options;
using NSubstitute;
using ZB.MOM.WW.ScadaBridge.CentralUI.Components.Shared;
using ZB.MOM.WW.ScadaBridge.CentralUI.Services;
using ZB.MOM.WW.ScadaBridge.Commons.Entities.Sites;
using ZB.MOM.WW.ScadaBridge.Commons.Interfaces.Repositories;
@@ -105,12 +108,43 @@ public class HealthPageTests : BunitContext
var user = new ClaimsPrincipal(new ClaimsIdentity(claims, "TestAuth"));
Services.AddSingleton<AuthenticationStateProvider>(new TestAuthStateProvider(user));
Services.AddAuthorizationCore();
// The failover control's AuthorizeView evaluates the named RequireAdmin policy, so the
// real policy set has to be registered or the page throws on any card that renders it.
AuthorizationPolicies.AddScadaBridgeAuthorization(Services);
// BunitContext pre-registers a placeholder IAuthorizationService that throws when a
// policy is evaluated; force the real one (same note as NavMenuTests).
Services.AddSingleton<IAuthorizationService, DefaultAuthorizationService>();
// The cluster cards embed CentralFailoverControl (Task 10), which injects these two.
// The page's DI graph genuinely requires them in production, so the test supplies
// them rather than the component making its dependencies optional. Behaviour of the
// control itself is covered by HealthFailoverButtonTests.
Services.AddSingleton(Substitute.For<IManualFailoverService>());
Services.AddSingleton(Substitute.For<IDialogService>());
}
/// <summary>
/// Renders the dashboard the way the app does — inside a
/// <see cref="CascadingAuthenticationState"/>, which the real layout supplies. The
/// cluster cards contain an AuthorizeView (the Task 10 failover control), and
/// AuthorizeView throws without that cascading value.
/// </summary>
private IRenderedComponent<HealthPage> RenderHealthPage()
{
var host = Render<CascadingAuthenticationState>(parameters => parameters
.Add(p => p.ChildContent, (RenderFragment)(builder =>
{
builder.OpenComponent<HealthPage>(0);
builder.CloseComponent();
})));
return host.FindComponent<HealthPage>();
}
[Fact]
public void Renders_OutboxKpiTiles_WithValues()
{
var cut = Render<HealthPage>();
var cut = RenderHealthPage();
// KPI data arrives via an async actor Ask after first render.
cut.WaitForAssertion(() =>
@@ -129,7 +163,7 @@ public class HealthPageTests : BunitContext
[Fact]
public void RendersLinkToTheNotificationKpisPage()
{
var cut = Render<HealthPage>();
var cut = RenderHealthPage();
var link = cut.Find("a[href='/notifications/kpis']");
Assert.Contains("View details", link.TextContent);
}
@@ -148,7 +182,7 @@ public class HealthPageTests : BunitContext
AsOfUtc: DateTime.UtcNow)));
Services.AddSingleton(auditService);
var cut = Render<HealthPage>();
var cut = RenderHealthPage();
cut.WaitForAssertion(() =>
{
@@ -166,7 +200,7 @@ public class HealthPageTests : BunitContext
[Fact]
public void Renders_SiteCallKpiTiles_WithValues()
{
var cut = Render<HealthPage>();
var cut = RenderHealthPage();
// KPI data arrives via an async actor Ask after first render.
cut.WaitForAssertion(() =>
@@ -186,7 +220,7 @@ public class HealthPageTests : BunitContext
[Fact]
public void RendersLinkToTheSiteCallsReportPage()
{
var cut = Render<HealthPage>();
var cut = RenderHealthPage();
var link = cut.Find("a[href='/site-calls/report']");
Assert.Contains("View details", link.TextContent);
}
@@ -197,7 +231,7 @@ public class HealthPageTests : BunitContext
_siteCallKpiReply = new SiteCallKpiResponse(
"k", false, "site call repository unavailable", 0, 0, 0, 0, null, 0);
var cut = Render<HealthPage>();
var cut = RenderHealthPage();
cut.WaitForAssertion(() =>
{
@@ -216,7 +250,7 @@ public class HealthPageTests : BunitContext
_kpiReply = new NotificationKpiResponse(
"k", false, "outbox repository unavailable", 0, 0, 0, 0, null);
var cut = Render<HealthPage>();
var cut = RenderHealthPage();
cut.WaitForAssertion(() =>
{
@@ -234,7 +268,7 @@ public class HealthPageTests : BunitContext
// default-site load produces charts.
SeedSites("site-a");
var cut = Render<HealthPage>();
var cut = RenderHealthPage();
cut.WaitForAssertion(() =>
{
@@ -268,7 +302,7 @@ public class HealthPageTests : BunitContext
throw new InvalidOperationException("kpi history unavailable"));
Services.AddSingleton(faulting);
var cut = Render<HealthPage>();
var cut = RenderHealthPage();
cut.WaitForAssertion(() =>
{
@@ -294,7 +328,7 @@ public class HealthPageTests : BunitContext
LastReportReceivedAt = DateTimeOffset.UtcNow.AddMinutes(-5),
});
var cut = Render<HealthPage>();
var cut = RenderHealthPage();
cut.WaitForAssertion(() =>
{
@@ -316,7 +350,7 @@ public class HealthPageTests : BunitContext
LastStatusChangeAt = changedAt,
});
var cut = Render<HealthPage>();
var cut = RenderHealthPage();
cut.WaitForAssertion(() =>
{
tests/ZB.MOM.WW.ScadaBridge.ClusterInfrastructure.Tests/ClusterOptionsTests.cs
+4 -1
View File
@@ -10,7 +10,10 @@ public class ClusterOptionsTests
{
var options = new ClusterOptions();
Assert.Equal("keep-oldest", options.SplitBrainResolverStrategy);
// 'auto-down' is the default posture (decision 2026-07-21): a crash of either
// node fails over to the survivor; dual-active during a real partition is the
// accepted trade for pairs with no shared lease infrastructure.
Assert.Equal("auto-down", options.SplitBrainResolverStrategy);
Assert.Equal(TimeSpan.FromSeconds(15), options.StableAfter);
Assert.Equal(TimeSpan.FromSeconds(2), options.HeartbeatInterval);
Assert.Equal(TimeSpan.FromSeconds(10), options.FailureDetectionThreshold);
tests/ZB.MOM.WW.ScadaBridge.ClusterInfrastructure.Tests/ClusterOptionsValidatorTests.cs
+30 -1
View File
@@ -153,9 +153,10 @@ public class ClusterOptionsValidatorTests
}
[Fact]
public void DownIfAloneFalse_FailsValidation()
public void DownIfAloneFalse_UnderKeepOldest_FailsValidation()
{
var options = ValidOptions();
options.SplitBrainResolverStrategy = "keep-oldest";
options.DownIfAlone = false;
var result = new ClusterOptionsValidator().Validate(null, options);
@@ -164,6 +165,34 @@ public class ClusterOptionsValidatorTests
Assert.Contains("DownIfAlone", result.FailureMessage);
}
[Fact]
public void AutoDownStrategy_Passes()
{
// Decision 2026-07-21: 'auto-down' is the supported availability-first
// posture — either-node crash fails over; dual-active on a real partition
// is the accepted trade.
var options = ValidOptions();
options.SplitBrainResolverStrategy = "auto-down";
var result = new ClusterOptionsValidator().Validate(null, options);
Assert.True(result.Succeeded, result.FailureMessage);
}
[Fact]
public void DownIfAloneFalse_UnderAutoDown_Passes()
{
// DownIfAlone is a keep-oldest knob; under auto-down it is inert and must
// not block startup.
var options = ValidOptions();
options.SplitBrainResolverStrategy = "auto-down";
options.DownIfAlone = false;
var result = new ClusterOptionsValidator().Validate(null, options);
Assert.True(result.Succeeded, result.FailureMessage);
}
[Fact]
public void Validate_AccumulatesAllFailures()
{
tests/ZB.MOM.WW.ScadaBridge.Commons.Tests/Types/Enums/AuditEnumTests.cs
+4 -3
View File
@@ -12,7 +12,7 @@ public class AuditEnumTests
[Fact]
public void AuditChannel_HasExactlyExpectedMembers()
{
var expected = new[] { "ApiOutbound", "DbOutbound", "Notification", "ApiInbound", "SecuredWrite" };
var expected = new[] { "ApiOutbound", "DbOutbound", "Notification", "ApiInbound", "SecuredWrite", "Cluster" };
var actual = Enum.GetValues(typeof(AuditChannel))
.Cast<AuditChannel>()
.Select(x => x.ToString())
@@ -23,7 +23,7 @@ public class AuditEnumTests
}
[Fact]
public void AuditKind_HasExactlySixteenExpectedMembers()
public void AuditKind_HasExactlySeventeenExpectedMembers()
{
var expected = new[]
{
@@ -33,13 +33,14 @@ public class AuditEnumTests
"SecuredWriteSubmit", "SecuredWriteApprove", "SecuredWriteReject", "SecuredWriteExecute",
"SecuredWriteExpire",
"ReconciliationAbandoned",
"ManualFailover",
};
var actual = Enum.GetValues(typeof(AuditKind))
.Cast<AuditKind>()
.Select(x => x.ToString())
.ToArray();
Assert.Equal(16, actual.Length);
Assert.Equal(17, actual.Length);
Assert.Equal(expected, actual);
}
tests/ZB.MOM.WW.ScadaBridge.Communication.Tests/Actors/AkkaCentralTransportTests.cs
+67
View File
@@ -0,0 +1,67 @@
using Akka.Actor;
using Akka.Cluster.Tools.Client;
using Akka.TestKit.Xunit2;
using ZB.MOM.WW.ScadaBridge.Commons.Messages.Notification;
using ZB.MOM.WW.ScadaBridge.Communication.Actors;
namespace ZB.MOM.WW.ScadaBridge.Communication.Tests.Actors;
/// <summary>
/// T1A.3: the regression-prone bit of the Akka transport — that a forwarded
/// <see cref="ClusterClient.Send"/> carries the caller's sender, so central's reply routes
/// straight back to the waiting Ask rather than to the site communication actor — plus the
/// no-ClusterClient-yet fallback that keeps the S&amp;F layer treating the send as transient.
/// </summary>
public class AkkaCentralTransportTests : TestKit
{
[Fact]
public void SubmitNotification_ForwardsToClusterClient_WithReplyToAsSender()
{
var transport = new AkkaCentralTransport();
var clusterClient = CreateTestProbe();
var replyTo = CreateTestProbe();
transport.SetCentralClient(clusterClient.Ref);
var submit = new NotificationSubmit(
"notif-1", "Operators", "S", "B", "site1", null, null, DateTimeOffset.UtcNow);
transport.SubmitNotification(submit, replyTo.Ref);
// The ClusterClient receives a Send addressed to the central actor...
var send = clusterClient.ExpectMsg<ClusterClient.Send>();
Assert.Equal("/user/central-communication", send.Path);
Assert.IsType<NotificationSubmit>(send.Message);
// ...and replying to it lands at replyTo, proving the sender was forwarded (not the
// transport / actor). This is the routing the waiting Ask relies on.
clusterClient.Reply(new NotificationSubmitAck("notif-1", Accepted: true, Error: null));
replyTo.ExpectMsg<NotificationSubmitAck>(ack => ack.NotificationId == "notif-1" && ack.Accepted);
}
[Fact]
public void SubmitNotification_WithNoClusterClient_RepliesNonAcceptedToReplyTo()
{
var transport = new AkkaCentralTransport();
var replyTo = CreateTestProbe();
transport.SubmitNotification(
new NotificationSubmit("notif-2", "Operators", "S", "B", "site1", null, null, DateTimeOffset.UtcNow),
replyTo.Ref);
replyTo.ExpectMsg<NotificationSubmitAck>(ack => ack.NotificationId == "notif-2" && !ack.Accepted);
}
[Fact]
public void IngestAuditEvents_WithNoClusterClient_FaultsTheReplyTo()
{
// The audit drain treats a faulted Ask as transient and keeps rows Pending — so the
// no-client path must be a Status.Failure, not a silent drop.
var transport = new AkkaCentralTransport();
var replyTo = CreateTestProbe();
transport.IngestAuditEvents(
new Commons.Messages.Audit.IngestAuditEventsCommand(new List<ZB.MOM.WW.Audit.AuditEvent>()),
replyTo.Ref);
replyTo.ExpectMsg<Status.Failure>();
}
}
tests/ZB.MOM.WW.ScadaBridge.Communication.Tests/Actors/SiteCommunicationActorTransportTests.cs
+166
View File
@@ -0,0 +1,166 @@
using Akka.Actor;
using Akka.TestKit.Xunit2;
using NSubstitute;
using ZB.MOM.WW.Audit;
using ZB.MOM.WW.ScadaBridge.Commons.Messages.Audit;
using ZB.MOM.WW.ScadaBridge.Commons.Messages.Deployment;
using ZB.MOM.WW.ScadaBridge.Commons.Messages.Health;
using ZB.MOM.WW.ScadaBridge.Commons.Messages.Notification;
using ZB.MOM.WW.ScadaBridge.Commons.Types.Enums;
using ZB.MOM.WW.ScadaBridge.Communication.Actors;
namespace ZB.MOM.WW.ScadaBridge.Communication.Tests.Actors;
/// <summary>
/// T1A.3: the site communication actor delegates each of the seven site→central sends to the
/// injected <see cref="ICentralTransport"/>, preserving the current <c>Sender</c> as the reply
/// target — and a transport failure surfaces to that sender exactly as the S&amp;F / audit / health
/// layers already expect, while a heartbeat transport fault never faults the actor.
/// </summary>
public class SiteCommunicationActorTransportTests : TestKit
{
private readonly CommunicationOptions _options = new();
private (IActorRef actor, ICentralTransport transport) NewActor()
{
var transport = Substitute.For<ICentralTransport>();
var dmProbe = CreateTestProbe();
var actor = Sys.ActorOf(Props.Create(() =>
new SiteCommunicationActor("site1", _options, dmProbe.Ref, () => false, null, transport)));
return (actor, transport);
}
[Fact]
public void NotificationSubmit_DelegatesToTransport_WithSenderAsReplyTo()
{
var (actor, transport) = NewActor();
var submit = new NotificationSubmit(
"notif-1", "Operators", "Subj", "Body", "site1", "inst1", "alarmScript", DateTimeOffset.UtcNow);
actor.Tell(submit, TestActor);
AwaitAssert(() => transport.Received(1).SubmitNotification(
Arg.Is<NotificationSubmit>(m => m.NotificationId == "notif-1"), Arg.Is(TestActor)));
}
[Fact]
public void NotificationStatusQuery_DelegatesToTransport_WithSenderAsReplyTo()
{
var (actor, transport) = NewActor();
actor.Tell(new NotificationStatusQuery("corr-1", "notif-1"), TestActor);
AwaitAssert(() => transport.Received(1).QueryNotificationStatus(
Arg.Is<NotificationStatusQuery>(m => m.NotificationId == "notif-1"), Arg.Is(TestActor)));
}
[Fact]
public void IngestAuditEvents_DelegatesToTransport_WithSenderAsReplyTo()
{
var (actor, transport) = NewActor();
actor.Tell(new IngestAuditEventsCommand(new List<AuditEvent>()), TestActor);
AwaitAssert(() => transport.Received(1).IngestAuditEvents(Arg.Any<IngestAuditEventsCommand>(), Arg.Is(TestActor)));
}
[Fact]
public void IngestCachedTelemetry_DelegatesToTransport_WithSenderAsReplyTo()
{
var (actor, transport) = NewActor();
actor.Tell(new IngestCachedTelemetryCommand(new List<CachedTelemetryEntry>()), TestActor);
AwaitAssert(() => transport.Received(1).IngestCachedTelemetry(Arg.Any<IngestCachedTelemetryCommand>(), Arg.Is(TestActor)));
}
[Fact]
public void ReconcileSite_DelegatesToTransport_WithSenderAsReplyTo()
{
var (actor, transport) = NewActor();
actor.Tell(
new ReconcileSiteRequest("site1", "node-a", new Dictionary<string, string>()), TestActor);
AwaitAssert(() => transport.Received(1).ReconcileSite(
Arg.Is<ReconcileSiteRequest>(m => m.NodeId == "node-a"), Arg.Is(TestActor)));
}
[Fact]
public void ReportSiteHealth_DelegatesToTransport_WithSenderAsReplyTo()
{
var (actor, transport) = NewActor();
var report = MinimalHealthReport(sequence: 7);
actor.Tell(report, TestActor);
AwaitAssert(() => transport.Received(1).ReportSiteHealth(
Arg.Is<SiteHealthReport>(m => m.SequenceNumber == 7), Arg.Is(TestActor)));
}
[Fact]
public void TransportFailureReply_RoutesBackToTheWaitingSender()
{
// The seam preserves the reply-routing the S&F layer depends on: when the transport
// answers the captured replyTo with a Status.Failure (its transient-failure signal on a
// non-OK status), that failure reaches the original sender — here the test actor — so the
// waiting Ask faults, exactly as it did on the ClusterClient path.
var transport = Substitute.For<ICentralTransport>();
transport
.When(t => t.SubmitNotification(Arg.Any<NotificationSubmit>(), Arg.Any<IActorRef>()))
.Do(ci => ci.Arg<IActorRef>().Tell(
new Status.Failure(new InvalidOperationException("central unavailable"))));
var dmProbe = CreateTestProbe();
var actor = Sys.ActorOf(Props.Create(() =>
new SiteCommunicationActor("site1", _options, dmProbe.Ref, () => false, null, transport)));
actor.Tell(new NotificationSubmit(
"notif-x", "Operators", "S", "B", "site1", null, null, DateTimeOffset.UtcNow), TestActor);
var failure = ExpectMsg<Status.Failure>();
Assert.IsType<InvalidOperationException>(failure.Cause);
}
[Fact]
public void HeartbeatTransportThrow_DoesNotFaultTheActor()
{
// A transport whose SendHeartbeat throws must not fault the actor — heartbeats are
// fire-and-forget and their failure is swallowed. Prove the actor still serves messages
// after a heartbeat that threw.
var transport = Substitute.For<ICentralTransport>();
transport
.When(t => t.SendHeartbeat(Arg.Any<HeartbeatMessage>(), Arg.Any<IActorRef>()))
.Do(_ => throw new InvalidOperationException("boom"));
var dmProbe = CreateTestProbe();
var actor = Sys.ActorOf(Props.Create(() =>
new SiteCommunicationActor(
"site1",
new CommunicationOptions { ApplicationHeartbeatInterval = TimeSpan.FromMilliseconds(50) },
dmProbe.Ref, () => true, null, transport)));
// Let the heartbeat timer fire a few times (each throws inside the transport).
Thread.Sleep(250);
// The actor is still alive and delegating: a subsequent send is handled normally.
actor.Tell(new NotificationSubmit(
"after-heartbeat", "Operators", "S", "B", "site1", null, null, DateTimeOffset.UtcNow), TestActor);
AwaitAssert(() => transport.Received(1).SubmitNotification(
Arg.Is<NotificationSubmit>(m => m.NotificationId == "after-heartbeat"), Arg.Is(TestActor)));
}
private static SiteHealthReport MinimalHealthReport(long sequence) => new(
SiteId: "site1",
SequenceNumber: sequence,
ReportTimestamp: DateTimeOffset.UtcNow,
DataConnectionStatuses: new Dictionary<string, ConnectionHealth>(),
TagResolutionCounts: new Dictionary<string, TagResolutionStatus>(),
ScriptErrorCount: 0,
AlarmEvaluationErrorCount: 0,
StoreAndForwardBufferDepths: new Dictionary<string, int>(),
DeadLetterCount: 0,
DeployedInstanceCount: 0,
EnabledInstanceCount: 0,
DisabledInstanceCount: 0);
}
tests/ZB.MOM.WW.ScadaBridge.Communication.Tests/CommunicationOptionsValidatorTests.cs
+53
View File
@@ -104,4 +104,57 @@ public class CommunicationOptionsValidatorTests
Assert.True(result.Failed);
Assert.Contains("LiveAlarmCachePublishCoalesce", result.FailureMessage);
}
// ── T1A.3: gRPC central transport endpoints (required only when selected) ────
[Fact]
public void GrpcTransport_WithNoEndpoints_IsRejected()
{
var result = Validate(new CommunicationOptions
{
CentralTransport = CentralTransportMode.Grpc,
CentralGrpcEndpoints = new List<string>(),
});
Assert.True(result.Failed);
Assert.Contains("CentralGrpcEndpoints", result.FailureMessage);
}
[Fact]
public void GrpcTransport_WithBlankEndpoint_IsRejected()
{
var result = Validate(new CommunicationOptions
{
CentralTransport = CentralTransportMode.Grpc,
CentralGrpcEndpoints = new List<string> { " " },
});
Assert.True(result.Failed);
Assert.Contains("CentralGrpcEndpoints", result.FailureMessage);
}
[Fact]
public void GrpcTransport_WithEndpoints_IsValid()
{
var result = Validate(new CommunicationOptions
{
CentralTransport = CentralTransportMode.Grpc,
CentralGrpcEndpoints = new List<string>
{
"http://scadabridge-central-a:8083",
"http://scadabridge-central-b:8083",
},
});
Assert.True(result.Succeeded, result.FailureMessage);
}
[Fact]
public void AkkaTransport_IgnoresMissingGrpcEndpoints()
{
// The default transport must not be forced to declare gRPC endpoints it never dials.
var result = Validate(new CommunicationOptions
{
CentralTransport = CentralTransportMode.Akka,
CentralGrpcEndpoints = new List<string>(),
});
Assert.True(result.Succeeded, result.FailureMessage);
}
}
tests/ZB.MOM.WW.ScadaBridge.Communication.Tests/Grpc/CentralControlDtoMapperTests.cs
+643
View File
@@ -0,0 +1,643 @@
using Google.Protobuf;
using ZB.MOM.WW.ScadaBridge.Commons.Entities.Audit;
using ZB.MOM.WW.ScadaBridge.Commons.Messages.Audit;
using ZB.MOM.WW.ScadaBridge.Commons.Messages.Deployment;
using ZB.MOM.WW.ScadaBridge.Commons.Messages.Health;
using ZB.MOM.WW.ScadaBridge.Commons.Messages.Notification;
using ZB.MOM.WW.ScadaBridge.Commons.Types;
using ZB.MOM.WW.ScadaBridge.Commons.Types.Audit;
using ZB.MOM.WW.ScadaBridge.Commons.Types.Enums;
using ZB.MOM.WW.ScadaBridge.Communication.Grpc;
namespace ZB.MOM.WW.ScadaBridge.Communication.Tests.Grpc;
/// <summary>
/// Round-trip golden tests for <see cref="CentralControlDtoMapper"/> — the DTO bridge
/// for the seven <c>CentralControlService</c> RPCs (Phase 1A of the ClusterClient→gRPC
/// migration).
/// </summary>
/// <remarks>
/// <para>
/// Every message gets a pair: a FULLY-POPULATED case that proves no field is dropped,
/// and a MINIMAL case that proves every nullable/optional/empty-collection member comes
/// back as null-or-empty rather than as a zero-valued stand-in. The minimal cases are
/// the ones that matter: a per-field unit test happily passes while a whole optional
/// branch is silently never written, which is exactly the class of bug the round-trip
/// guard in PLAN-05 T8 caught five times over.
/// </para>
/// <para>
/// When a round-trip here fails, the mapper is wrong — not the test.
/// </para>
/// </remarks>
public class CentralControlDtoMapperTests
{
private static readonly DateTimeOffset SampleInstant =
new(2026, 7, 22, 9, 30, 15, 250, TimeSpan.Zero);
// -----------------------------------------------------------------------
// NotificationSubmit / Ack
// -----------------------------------------------------------------------
[Fact]
public void NotificationSubmit_RoundTrip_FullyPopulated_PreservesEveryField()
{
var original = new NotificationSubmit(
NotificationId: Guid.NewGuid().ToString(),
ListName: "plant-ops",
Subject: "Tank 4 overfill",
Body: "Level exceeded 95% for 5 minutes.",
SourceSiteId: "site-a",
SourceInstanceId: "Tank04",
SourceScript: "OnLevelHigh",
SiteEnqueuedAt: SampleInstant,
OriginExecutionId: Guid.NewGuid(),
OriginParentExecutionId: Guid.NewGuid(),
SourceNode: "node-b");
var roundTripped = CentralControlDtoMapper.FromDto(Wire(CentralControlDtoMapper.ToDto(original)));
// Every member is a scalar, so record value-equality is a true deep compare.
Assert.Equal(original, roundTripped);
}
[Fact]
public void NotificationSubmit_RoundTrip_AllOptionalsNull_StayNull()
{
var original = new NotificationSubmit(
NotificationId: Guid.NewGuid().ToString(),
ListName: "plant-ops",
Subject: "s",
Body: "b",
SourceSiteId: "site-a",
SourceInstanceId: null,
SourceScript: null,
SiteEnqueuedAt: SampleInstant,
OriginExecutionId: null,
OriginParentExecutionId: null,
SourceNode: null);
var roundTripped = CentralControlDtoMapper.FromDto(Wire(CentralControlDtoMapper.ToDto(original)));
Assert.Equal(original, roundTripped);
Assert.Null(roundTripped.SourceInstanceId);
Assert.Null(roundTripped.SourceScript);
Assert.Null(roundTripped.OriginExecutionId);
Assert.Null(roundTripped.OriginParentExecutionId);
Assert.Null(roundTripped.SourceNode);
}
[Fact]
public void NotificationSubmit_NonUtcOffset_NormalizesToTheSameInstant()
{
// Documented contract: a protobuf Timestamp is an instant, so the offset
// component of a DateTimeOffset does not survive. Every producer stamps UTC
// (Notify.Send uses DateTimeOffset.UtcNow), so the instant is what matters.
var melbourne = new DateTimeOffset(2026, 7, 22, 19, 30, 15, TimeSpan.FromHours(10));
var original = new NotificationSubmit(
"id", "list", "s", "b", "site-a", null, null, melbourne);
var roundTripped = CentralControlDtoMapper.FromDto(Wire(CentralControlDtoMapper.ToDto(original)));
Assert.Equal(melbourne.UtcDateTime, roundTripped.SiteEnqueuedAt.UtcDateTime);
Assert.Equal(TimeSpan.Zero, roundTripped.SiteEnqueuedAt.Offset);
}
[Fact]
public void NotificationSubmitAck_RoundTrip_Accepted_And_Rejected()
{
var accepted = new NotificationSubmitAck("n1", Accepted: true, Error: null);
var rejected = new NotificationSubmitAck("n2", Accepted: false, Error: "list not found");
Assert.Equal(accepted, CentralControlDtoMapper.FromDto(Wire(CentralControlDtoMapper.ToDto(accepted))));
Assert.Equal(rejected, CentralControlDtoMapper.FromDto(Wire(CentralControlDtoMapper.ToDto(rejected))));
Assert.Null(CentralControlDtoMapper.FromDto(Wire(CentralControlDtoMapper.ToDto(accepted))).Error);
}
// -----------------------------------------------------------------------
// NotificationStatusQuery / Response
// -----------------------------------------------------------------------
[Fact]
public void NotificationStatusQuery_RoundTrip_PreservesEveryField()
{
var original = new NotificationStatusQuery("corr-1", Guid.NewGuid().ToString());
Assert.Equal(original, CentralControlDtoMapper.FromDto(Wire(CentralControlDtoMapper.ToDto(original))));
}
[Fact]
public void NotificationStatusResponse_RoundTrip_FullyPopulated_PreservesEveryField()
{
var original = new NotificationStatusResponse(
CorrelationId: "corr-1",
Found: true,
Status: "Delivered",
RetryCount: 3,
LastError: "smtp 421",
DeliveredAt: SampleInstant);
Assert.Equal(original, CentralControlDtoMapper.FromDto(Wire(CentralControlDtoMapper.ToDto(original))));
}
[Fact]
public void NotificationStatusResponse_RoundTrip_NotFound_LeavesOptionalsNull()
{
var original = new NotificationStatusResponse(
CorrelationId: "corr-1",
Found: false,
Status: "Unknown",
RetryCount: 0,
LastError: null,
DeliveredAt: null);
var roundTripped = CentralControlDtoMapper.FromDto(Wire(CentralControlDtoMapper.ToDto(original)));
Assert.Equal(original, roundTripped);
Assert.Null(roundTripped.LastError);
Assert.Null(roundTripped.DeliveredAt);
}
// -----------------------------------------------------------------------
// Audit ingest envelopes (rows delegate to AuditEventDtoMapper / SiteCallDtoMapper)
// -----------------------------------------------------------------------
[Fact]
public void IngestAuditEventsCommand_RoundTrip_PreservesOrderAndRows()
{
var first = NewAuditEvent(sourceScript: "OnDemand");
var second = NewAuditEvent(sourceScript: "OnTrigger");
var original = new IngestAuditEventsCommand([first, second]);
var roundTripped = CentralControlDtoMapper.FromDto(Wire(CentralControlDtoMapper.ToDto(original)));
Assert.Equal(2, roundTripped.Events.Count);
Assert.Equal(first.AsRow().EventId, roundTripped.Events[0].AsRow().EventId);
Assert.Equal("OnDemand", roundTripped.Events[0].AsRow().SourceScript);
Assert.Equal(second.AsRow().EventId, roundTripped.Events[1].AsRow().EventId);
Assert.Equal("OnTrigger", roundTripped.Events[1].AsRow().SourceScript);
}
[Fact]
public void IngestAuditEventsCommand_RoundTrip_EmptyBatch_StaysEmpty()
{
var roundTripped = CentralControlDtoMapper.FromDto(
Wire(CentralControlDtoMapper.ToDto(new IngestAuditEventsCommand([]))));
Assert.Empty(roundTripped.Events);
}
[Fact]
public void IngestCachedTelemetryCommand_RoundTrip_PreservesBothHalvesOfEachPacket()
{
var audit = NewAuditEvent(sourceScript: "OnDemand");
var siteCall = NewSiteCall();
var original = new IngestCachedTelemetryCommand([new CachedTelemetryEntry(audit, siteCall)]);
var roundTripped = CentralControlDtoMapper.FromDto(Wire(CentralControlDtoMapper.ToDto(original)));
var entry = Assert.Single(roundTripped.Entries);
Assert.Equal(audit.AsRow().EventId, entry.Audit.AsRow().EventId);
// IngestedAtUtc is central-set inside the dual-write transaction and is
// deliberately off the wire, so it is the one member excluded from the compare.
Assert.Equal(siteCall with { IngestedAtUtc = default }, entry.SiteCall with { IngestedAtUtc = default });
}
[Fact]
public void IngestCachedTelemetryCommand_RoundTrip_NullableSiteCallFields_StayNull()
{
var siteCall = NewSiteCall() with
{
SourceNode = null,
LastError = null,
HttpStatus = null,
TerminalAtUtc = null,
};
var original = new IngestCachedTelemetryCommand(
[new CachedTelemetryEntry(NewAuditEvent(), siteCall)]);
var roundTripped = CentralControlDtoMapper.FromDto(Wire(CentralControlDtoMapper.ToDto(original)));
var entry = Assert.Single(roundTripped.Entries);
Assert.Null(entry.SiteCall.SourceNode);
Assert.Null(entry.SiteCall.LastError);
Assert.Null(entry.SiteCall.HttpStatus);
Assert.Null(entry.SiteCall.TerminalAtUtc);
}
[Fact]
public void IngestCachedTelemetryCommand_RoundTrip_EmptyBatch_StaysEmpty()
{
var roundTripped = CentralControlDtoMapper.FromDto(
Wire(CentralControlDtoMapper.ToDto(new IngestCachedTelemetryCommand([]))));
Assert.Empty(roundTripped.Entries);
}
[Fact]
public void IngestAck_RoundTrip_PreservesIdsAndOrder()
{
var ids = new[] { Guid.NewGuid(), Guid.NewGuid(), Guid.NewGuid() };
var roundTripped = CentralControlDtoMapper.FromIngestAck(
Wire(CentralControlDtoMapper.ToIngestAck(ids)));
Assert.Equal(ids, roundTripped);
}
[Fact]
public void IngestAck_RoundTrip_NothingAccepted_StaysEmpty()
{
// An empty ack is meaningful — "zero rows persisted" leaves the site's rows
// Pending — so it must not be indistinguishable from a dropped field.
Assert.Empty(CentralControlDtoMapper.FromIngestAck(
Wire(CentralControlDtoMapper.ToIngestAck([]))));
}
// -----------------------------------------------------------------------
// Startup reconciliation
// -----------------------------------------------------------------------
[Fact]
public void ReconcileSiteRequest_RoundTrip_PreservesInventoryMap()
{
var original = new ReconcileSiteRequest(
SiteIdentifier: "site-a",
NodeId: "node-a",
LocalNameToRevisionHash: new Dictionary<string, string>
{
["Plant.Tank04"] = "hash-1",
["Plant.Pump01"] = "hash-2",
});
var roundTripped = CentralControlDtoMapper.FromDto(Wire(CentralControlDtoMapper.ToDto(original)));
Assert.Equal(original.SiteIdentifier, roundTripped.SiteIdentifier);
Assert.Equal(original.NodeId, roundTripped.NodeId);
Assert.Equal(
original.LocalNameToRevisionHash.OrderBy(kv => kv.Key),
roundTripped.LocalNameToRevisionHash.OrderBy(kv => kv.Key));
}
[Fact]
public void ReconcileSiteRequest_RoundTrip_EmptyInventory_StaysEmpty()
{
// A node with nothing deployed is the normal first-boot case, and central
// must see an empty inventory rather than a missing one.
var roundTripped = CentralControlDtoMapper.FromDto(Wire(CentralControlDtoMapper.ToDto(
new ReconcileSiteRequest("site-a", "node-a", new Dictionary<string, string>()))));
Assert.Empty(roundTripped.LocalNameToRevisionHash);
}
[Fact]
public void ReconcileSiteResponse_RoundTrip_PreservesGapOrphansAndUrl()
{
var original = new ReconcileSiteResponse(
Gap:
[
new ReconcileGapItem("Plant.Tank04", "dep-1", "hash-1", IsEnabled: true, "token-1"),
new ReconcileGapItem("Plant.Pump01", "dep-2", "hash-2", IsEnabled: false, "token-2"),
],
OrphanNames: ["Plant.Retired01", "Plant.Retired02"],
CentralFetchBaseUrl: "http://scadabridge-central-a:5000");
var roundTripped = CentralControlDtoMapper.FromDto(Wire(CentralControlDtoMapper.ToDto(original)));
Assert.Equal(original.Gap, roundTripped.Gap);
Assert.Equal(original.OrphanNames, roundTripped.OrphanNames);
Assert.Equal(original.CentralFetchBaseUrl, roundTripped.CentralFetchBaseUrl);
}
[Fact]
public void ReconcileSiteResponse_RoundTrip_NoGap_StaysEmpty()
{
var roundTripped = CentralControlDtoMapper.FromDto(Wire(CentralControlDtoMapper.ToDto(
new ReconcileSiteResponse([], [], "http://central:5000"))));
Assert.Empty(roundTripped.Gap);
Assert.Empty(roundTripped.OrphanNames);
}
// -----------------------------------------------------------------------
// Site health
// -----------------------------------------------------------------------
[Fact]
public void SiteHealthReport_RoundTrip_FullyPopulated_PreservesEveryField()
{
var original = FullyPopulatedHealthReport();
var roundTripped = CentralControlDtoMapper.FromDto(Wire(CentralControlDtoMapper.ToDto(original)));
AssertHealthReportsEqual(original, roundTripped);
}
[Fact]
public void SiteHealthReport_RoundTrip_MinimalReport_LeavesEveryOptionalNullOrEmpty()
{
// The shape a producer emits before any reporter has run: no endpoints map, no
// tag-quality map, no cluster-node list, no audit backlog, no nullable gauges.
var original = new SiteHealthReport(
SiteId: "site-a",
SequenceNumber: 1,
ReportTimestamp: SampleInstant,
DataConnectionStatuses: new Dictionary<string, ConnectionHealth>(),
TagResolutionCounts: new Dictionary<string, TagResolutionStatus>(),
ScriptErrorCount: 0,
AlarmEvaluationErrorCount: 0,
StoreAndForwardBufferDepths: new Dictionary<string, int>(),
DeadLetterCount: 0,
DeployedInstanceCount: 0,
EnabledInstanceCount: 0,
DisabledInstanceCount: 0);
var roundTripped = CentralControlDtoMapper.FromDto(Wire(CentralControlDtoMapper.ToDto(original)));
AssertHealthReportsEqual(original, roundTripped);
Assert.Empty(roundTripped.DataConnectionStatuses);
Assert.Empty(roundTripped.TagResolutionCounts);
Assert.Empty(roundTripped.StoreAndForwardBufferDepths);
Assert.Null(roundTripped.DataConnectionEndpoints);
Assert.Null(roundTripped.DataConnectionTagQuality);
Assert.Null(roundTripped.ClusterNodes);
Assert.Null(roundTripped.SiteAuditBacklog);
Assert.Null(roundTripped.OldestParkedMessageAgeSeconds);
Assert.Null(roundTripped.ScriptOldestBusyAgeSeconds);
Assert.Null(roundTripped.LocalDbReplicationConnected);
Assert.Null(roundTripped.LocalDbOplogBacklog);
}
[Fact]
public void SiteHealthReport_RoundTrip_EmptyNullableCollections_StayEmptyNotNull()
{
// The distinction the three wrapper messages exist for. Null means "this node
// does not report the signal"; empty means "it reports it, and there is
// nothing in it". proto3 cannot express presence on repeated/map fields, so
// collapsing one into the other here would be invisible until an operator
// misread the health page.
var original = FullyPopulatedHealthReport() with
{
DataConnectionEndpoints = new Dictionary<string, string>(),
DataConnectionTagQuality = new Dictionary<string, TagQualityCounts>(),
ClusterNodes = [],
};
var roundTripped = CentralControlDtoMapper.FromDto(Wire(CentralControlDtoMapper.ToDto(original)));
Assert.NotNull(roundTripped.DataConnectionEndpoints);
Assert.Empty(roundTripped.DataConnectionEndpoints);
Assert.NotNull(roundTripped.DataConnectionTagQuality);
Assert.Empty(roundTripped.DataConnectionTagQuality);
Assert.NotNull(roundTripped.ClusterNodes);
Assert.Empty(roundTripped.ClusterNodes);
}
[Fact]
public void SiteHealthReport_RoundTrip_FalseAndZeroGauges_StayFalseAndZero()
{
// The mirror of the null case: LocalDbReplicationConnected=false ("configured
// but disconnected") and LocalDbOplogBacklog=0 ("healthy, nothing queued") are
// real values that must not decay into null on the wire.
var original = FullyPopulatedHealthReport() with
{
LocalDbReplicationConnected = false,
LocalDbOplogBacklog = 0,
OldestParkedMessageAgeSeconds = 0d,
ScriptOldestBusyAgeSeconds = 0d,
};
var roundTripped = CentralControlDtoMapper.FromDto(Wire(CentralControlDtoMapper.ToDto(original)));
Assert.False(roundTripped.LocalDbReplicationConnected);
Assert.Equal(0L, roundTripped.LocalDbOplogBacklog);
Assert.Equal(0d, roundTripped.OldestParkedMessageAgeSeconds);
Assert.Equal(0d, roundTripped.ScriptOldestBusyAgeSeconds);
}
[Fact]
public void SiteHealthReport_RoundTrip_BacklogWithEmptyQueue_KeepsNullOldestPending()
{
var original = FullyPopulatedHealthReport() with
{
SiteAuditBacklog = new SiteAuditBacklogSnapshot(0, null, 4096),
};
var roundTripped = CentralControlDtoMapper.FromDto(Wire(CentralControlDtoMapper.ToDto(original)));
Assert.Equal(new SiteAuditBacklogSnapshot(0, null, 4096), roundTripped.SiteAuditBacklog);
}
[Fact]
public void SiteHealthReportAck_RoundTrip_Accepted_And_Rejected()
{
var accepted = new SiteHealthReportAck("site-a", 42, Accepted: true);
var rejected = new SiteHealthReportAck("site-a", 43, Accepted: false, Error: "aggregator down");
Assert.Equal(accepted, CentralControlDtoMapper.FromDto(Wire(CentralControlDtoMapper.ToDto(accepted))));
Assert.Equal(rejected, CentralControlDtoMapper.FromDto(Wire(CentralControlDtoMapper.ToDto(rejected))));
Assert.Null(CentralControlDtoMapper.FromDto(Wire(CentralControlDtoMapper.ToDto(accepted))).Error);
}
// -----------------------------------------------------------------------
// Heartbeat + ConnectionHealth enum
// -----------------------------------------------------------------------
[Theory]
[InlineData(true)]
[InlineData(false)]
public void HeartbeatMessage_RoundTrip_PreservesEveryField(bool isActive)
{
var original = new HeartbeatMessage("site-a", "scadabridge-site-a-node-b", isActive, SampleInstant);
Assert.Equal(original, CentralControlDtoMapper.FromDto(Wire(CentralControlDtoMapper.ToDto(original))));
}
[Theory]
[InlineData(ConnectionHealth.Connected)]
[InlineData(ConnectionHealth.Disconnected)]
[InlineData(ConnectionHealth.Connecting)]
[InlineData(ConnectionHealth.Error)]
public void ConnectionHealth_RoundTrip_EveryValueSurvives(ConnectionHealth health)
{
Assert.Equal(health, CentralControlDtoMapper.FromDto(CentralControlDtoMapper.ToDto(health)));
}
[Fact]
public void ConnectionHealth_Unspecified_DecodesToErrorNotConnected()
{
// Fail-safe direction: a wire value this build does not recognise must never
// render as "healthy" on the central health page.
Assert.Equal(
ConnectionHealth.Error,
CentralControlDtoMapper.FromDto(ConnectionHealthEnum.ConnectionHealthUnspecified));
}
[Fact]
public void ConnectionHealth_EveryEnumMemberIsMapped()
{
// Guards the ArgumentOutOfRangeException path: adding a ConnectionHealth value
// without extending the mapper should fail here, not silently on a rig.
foreach (var health in Enum.GetValues<ConnectionHealth>())
{
var wire = CentralControlDtoMapper.ToDto(health);
Assert.NotEqual(ConnectionHealthEnum.ConnectionHealthUnspecified, wire);
}
}
// -----------------------------------------------------------------------
// Fixtures
// -----------------------------------------------------------------------
private static SiteHealthReport FullyPopulatedHealthReport() =>
new(
SiteId: "site-a",
SequenceNumber: 987654321L,
ReportTimestamp: SampleInstant,
DataConnectionStatuses: new Dictionary<string, ConnectionHealth>
{
["opc-main"] = ConnectionHealth.Connected,
["mx-gateway"] = ConnectionHealth.Error,
["opc-backup"] = ConnectionHealth.Connecting,
["legacy"] = ConnectionHealth.Disconnected,
},
TagResolutionCounts: new Dictionary<string, TagResolutionStatus>
{
["opc-main"] = new(TotalSubscribed: 120, SuccessfullyResolved: 118),
},
ScriptErrorCount: 4,
AlarmEvaluationErrorCount: 2,
StoreAndForwardBufferDepths: new Dictionary<string, int>
{
["erp"] = 17,
["mes"] = 0,
},
DeadLetterCount: 5,
DeployedInstanceCount: 30,
EnabledInstanceCount: 28,
DisabledInstanceCount: 2,
NodeRole: "Active",
NodeHostname: "scadabridge-site-a-node-a",
DataConnectionEndpoints: new Dictionary<string, string>
{
["opc-main"] = "opc.tcp://opcua:4840",
},
DataConnectionTagQuality: new Dictionary<string, TagQualityCounts>
{
["opc-main"] = new(Good: 100, Bad: 3, Uncertain: 15),
},
ParkedMessageCount: 6,
ClusterNodes:
[
new NodeStatus("scadabridge-site-a-node-a", IsOnline: true, "Active"),
new NodeStatus("scadabridge-site-a-node-b", IsOnline: false, "Standby"),
],
SiteAuditWriteFailures: 7,
AuditRedactionFailure: 8,
SiteAuditBacklog: new SiteAuditBacklogSnapshot(
PendingCount: 42,
OldestPendingUtc: new DateTime(2026, 7, 22, 8, 0, 0, DateTimeKind.Utc),
OnDiskBytes: 1_234_567L),
SiteEventLogWriteFailures: 9L,
OldestParkedMessageAgeSeconds: 3600.5d)
{
ScriptQueueDepth = 11,
ScriptBusyThreads = 3,
ScriptOldestBusyAgeSeconds = 12.25d,
LocalDbReplicationConnected = true,
LocalDbOplogBacklog = 250L,
};
/// <summary>
/// Compares two reports member by member. <see cref="SiteHealthReport"/> is a record,
/// but its dictionary/list members compare by reference under record equality, so
/// <c>Assert.Equal(a, b)</c> would pass trivially for the scalars and never look
/// inside the collections — the exact blind spot these goldens exist to close.
/// </summary>
private static void AssertHealthReportsEqual(SiteHealthReport expected, SiteHealthReport actual)
{
Assert.Equal(expected.SiteId, actual.SiteId);
Assert.Equal(expected.SequenceNumber, actual.SequenceNumber);
Assert.Equal(expected.ReportTimestamp, actual.ReportTimestamp);
Assert.Equal(
expected.DataConnectionStatuses.OrderBy(kv => kv.Key),
actual.DataConnectionStatuses.OrderBy(kv => kv.Key));
Assert.Equal(
expected.TagResolutionCounts.OrderBy(kv => kv.Key),
actual.TagResolutionCounts.OrderBy(kv => kv.Key));
Assert.Equal(expected.ScriptErrorCount, actual.ScriptErrorCount);
Assert.Equal(expected.AlarmEvaluationErrorCount, actual.AlarmEvaluationErrorCount);
Assert.Equal(
expected.StoreAndForwardBufferDepths.OrderBy(kv => kv.Key),
actual.StoreAndForwardBufferDepths.OrderBy(kv => kv.Key));
Assert.Equal(expected.DeadLetterCount, actual.DeadLetterCount);
Assert.Equal(expected.DeployedInstanceCount, actual.DeployedInstanceCount);
Assert.Equal(expected.EnabledInstanceCount, actual.EnabledInstanceCount);
Assert.Equal(expected.DisabledInstanceCount, actual.DisabledInstanceCount);
Assert.Equal(expected.NodeRole, actual.NodeRole);
Assert.Equal(expected.NodeHostname, actual.NodeHostname);
Assert.Equal(
expected.DataConnectionEndpoints?.OrderBy(kv => kv.Key),
actual.DataConnectionEndpoints?.OrderBy(kv => kv.Key));
Assert.Equal(
expected.DataConnectionTagQuality?.OrderBy(kv => kv.Key),
actual.DataConnectionTagQuality?.OrderBy(kv => kv.Key));
Assert.Equal(expected.ParkedMessageCount, actual.ParkedMessageCount);
Assert.Equal(expected.ClusterNodes, actual.ClusterNodes);
Assert.Equal(expected.SiteAuditWriteFailures, actual.SiteAuditWriteFailures);
Assert.Equal(expected.AuditRedactionFailure, actual.AuditRedactionFailure);
Assert.Equal(expected.SiteAuditBacklog, actual.SiteAuditBacklog);
Assert.Equal(expected.SiteEventLogWriteFailures, actual.SiteEventLogWriteFailures);
Assert.Equal(expected.OldestParkedMessageAgeSeconds, actual.OldestParkedMessageAgeSeconds);
Assert.Equal(expected.ScriptQueueDepth, actual.ScriptQueueDepth);
Assert.Equal(expected.ScriptBusyThreads, actual.ScriptBusyThreads);
Assert.Equal(expected.ScriptOldestBusyAgeSeconds, actual.ScriptOldestBusyAgeSeconds);
Assert.Equal(expected.LocalDbReplicationConnected, actual.LocalDbReplicationConnected);
Assert.Equal(expected.LocalDbOplogBacklog, actual.LocalDbOplogBacklog);
}
/// <summary>
/// Serializes a wire message and parses it back, so every round-trip below crosses a
/// real protobuf encode/decode rather than only exercising the mapper's object graph.
/// This is what proves the three collection wrapper messages keep their presence
/// when empty — an empty message encodes as a tag with zero-length payload, and a
/// mapper that used a bare <c>repeated</c>/<c>map</c> field instead would decode an
/// empty collection back as null with no test-visible difference at the object level.
/// </summary>
/// <typeparam name="T">Generated protobuf message type.</typeparam>
/// <param name="message">The message to send through an encode/decode cycle.</param>
/// <returns>An independent instance parsed from <paramref name="message"/>'s bytes.</returns>
private static T Wire<T>(T message) where T : IMessage<T>, new() =>
new MessageParser<T>(() => new T()).ParseFrom(message.ToByteArray());
private static ZB.MOM.WW.Audit.AuditEvent NewAuditEvent(string? sourceScript = null) =>
ScadaBridgeAuditEventFactory.Create(
channel: AuditChannel.ApiOutbound,
kind: AuditKind.ApiCallCached,
status: AuditStatus.Forwarded,
eventId: Guid.NewGuid(),
occurredAtUtc: new DateTime(2026, 7, 22, 9, 0, 0, DateTimeKind.Utc),
target: "ERP.GetOrder",
sourceSiteId: "site-a",
sourceNode: "node-a",
sourceScript: sourceScript);
private static SiteCall NewSiteCall() => new()
{
TrackedOperationId = TrackedOperationId.New(),
Channel = "ApiOutbound",
Target = "ERP.GetOrder",
SourceSite = "site-a",
SourceNode = "node-a",
Status = "Delivered",
RetryCount = 2,
LastError = "transient 503",
HttpStatus = 200,
CreatedAtUtc = new DateTime(2026, 7, 22, 8, 0, 0, DateTimeKind.Utc),
UpdatedAtUtc = new DateTime(2026, 7, 22, 8, 5, 0, DateTimeKind.Utc),
TerminalAtUtc = new DateTime(2026, 7, 22, 8, 10, 0, DateTimeKind.Utc),
IngestedAtUtc = new DateTime(2026, 7, 22, 8, 10, 1, DateTimeKind.Utc),
};
}
tests/ZB.MOM.WW.ScadaBridge.Communication.Tests/Grpc/CentralControlGrpcServiceTests.cs
+181
View File
@@ -0,0 +1,181 @@
using Akka.Actor;
using Akka.TestKit;
using Akka.TestKit.Xunit2;
using Grpc.Core;
using Microsoft.Extensions.Logging.Abstractions;
using Microsoft.Extensions.Options;
using NSubstitute;
using ZB.MOM.WW.ScadaBridge.Commons.Messages.Audit;
using ZB.MOM.WW.ScadaBridge.Commons.Messages.Notification;
using ZB.MOM.WW.ScadaBridge.Communication;
using ZB.MOM.WW.ScadaBridge.Communication.Grpc;
namespace ZB.MOM.WW.ScadaBridge.Communication.Tests.Grpc;
/// <summary>
/// Behaviour of <see cref="CentralControlGrpcService"/> that does not need a real gRPC pipeline:
/// the readiness gate, the fire-and-forget heartbeat contract, the ingest-timeout status
/// mapping, and the shared audit-ingest timeout constant. The auth interceptor + real transport
/// are proven separately in <c>CentralControlEndToEndTests</c> (Host.Tests).
/// </summary>
public class CentralControlGrpcServiceTests : TestKit
{
private static ServerCallContext NewContext(CancellationToken ct = default)
{
var context = Substitute.For<ServerCallContext>();
context.CancellationToken.Returns(ct);
return context;
}
private CentralControlGrpcService CreateService(CommunicationOptions? options = null)
=> new(
NullLogger<CentralControlGrpcService>.Instance,
Options.Create(options ?? new CommunicationOptions()));
[Fact]
public async Task BeforeSetReady_AUnaryCall_IsUnavailable()
{
// Nothing was dispatched, so Unavailable is the right status — it tells a site transport
// the call never ran and a cross-node retry is safe.
var service = CreateService();
var ex = await Assert.ThrowsAsync<RpcException>(
() => service.SubmitNotification(new NotificationSubmitDto(), NewContext()));
Assert.Equal(StatusCode.Unavailable, ex.StatusCode);
}
[Fact]
public async Task BeforeSetReady_ANonEmptyIngestBatch_IsUnavailable()
{
var service = CreateService();
var batch = new AuditEventBatch();
batch.Events.Add(NewAuditDto());
var ex = await Assert.ThrowsAsync<RpcException>(
() => service.IngestAuditEvents(batch, NewContext()));
Assert.Equal(StatusCode.Unavailable, ex.StatusCode);
}
[Fact]
public async Task AfterSetReady_AUnaryCall_ReachesTheActor()
{
var stub = Sys.ActorOf(Props.Create(() => new StubActor()));
var service = CreateService();
service.SetReady(stub);
var ack = await service.SubmitNotification(NewNotificationDto(), NewContext());
Assert.True(ack.Accepted);
}
[Fact]
public async Task Heartbeat_BeforeSetReady_SucceedsAndIsDropped()
{
// Fire-and-forget end-to-end: a heartbeat must never fault the site's timer, so even
// with no actor wired the call returns OK rather than Unavailable.
var service = CreateService();
var reply = await service.Heartbeat(NewHeartbeatDto(), NewContext());
Assert.NotNull(reply);
}
[Fact]
public async Task Heartbeat_AfterSetReady_TellsTheActor_AndNeverAsks()
{
// A black-hole actor that never replies would hang an Ask forever; the heartbeat still
// returns immediately, proving it is a Tell, not an Ask.
var blackHole = Sys.ActorOf(Props.Create(() => new NeverRepliesActor()));
var service = CreateService();
service.SetReady(blackHole);
var reply = await service.Heartbeat(NewHeartbeatDto(), NewContext());
Assert.NotNull(reply);
}
[Fact]
public async Task WhenTheActorNeverReplies_TheCall_IsDeadlineExceeded_NotUnavailable()
{
// The message WAS delivered (Unavailable would wrongly invite a duplicate retry on the
// peer node); a timeout is DeadlineExceeded, which callers never cross-node-retry.
var blackHole = Sys.ActorOf(Props.Create(() => new NeverRepliesActor()));
var service = CreateService(new CommunicationOptions
{
NotificationForwardTimeout = TimeSpan.FromMilliseconds(200),
});
service.SetReady(blackHole);
var ex = await Assert.ThrowsAsync<RpcException>(
() => service.SubmitNotification(NewNotificationDto(), NewContext()));
Assert.Equal(StatusCode.DeadlineExceeded, ex.StatusCode);
}
[Fact]
public async Task EmptyIngestBatch_ShortCircuits_EvenBeforeSetReady()
{
// An empty batch is a no-op the actor need never see; it must not depend on readiness.
var service = CreateService();
var ack = await service.IngestAuditEvents(new AuditEventBatch(), NewContext());
Assert.Empty(ack.AcceptedEventIds);
}
[Fact]
public void TheAuditIngestTimeout_IsTheOneSharedConstant()
{
// The plan calls SiteStreamGrpcServer.AuditIngestAskTimeout "one source of truth" shared
// between the two audit-ingest transports; the central service must not re-declare 30s.
Assert.Equal(TimeSpan.FromSeconds(30), SiteStreamGrpcServer.AuditIngestAskTimeout);
}
// The mapper reads SiteEnqueuedAt unconditionally, so a DTO that reaches mapping must carry
// a timestamp. Only DTOs that get past the readiness/auth gate map, so the negative tests
// above can pass a bare DTO.
private static NotificationSubmitDto NewNotificationDto() => new()
{
NotificationId = Guid.NewGuid().ToString(),
ListName = "ops",
SiteEnqueuedAt = Google.Protobuf.WellKnownTypes.Timestamp.FromDateTime(
DateTime.SpecifyKind(new DateTime(2026, 5, 20, 10, 0, 0), DateTimeKind.Utc)),
};
private static HeartbeatDto NewHeartbeatDto() => new()
{
SiteId = "site-a",
NodeHostname = "node-a",
IsActive = true,
Timestamp = Google.Protobuf.WellKnownTypes.Timestamp.FromDateTime(
DateTime.SpecifyKind(new DateTime(2026, 5, 20, 10, 0, 0), DateTimeKind.Utc)),
};
private static AuditEventDto NewAuditDto() => new()
{
EventId = Guid.NewGuid().ToString(),
OccurredAtUtc = Google.Protobuf.WellKnownTypes.Timestamp.FromDateTime(
DateTime.SpecifyKind(new DateTime(2026, 5, 20, 10, 0, 0), DateTimeKind.Utc)),
Channel = "ApiOutbound",
Kind = "ApiCall",
Status = "Delivered",
SourceSiteId = "site-a",
};
private sealed class StubActor : ReceiveActor
{
public StubActor()
{
Receive<NotificationSubmit>(msg =>
Sender.Tell(new NotificationSubmitAck(msg.NotificationId, Accepted: true, Error: null)));
}
}
/// <summary>Swallows every message and never replies, so an Ask against it times out.</summary>
private sealed class NeverRepliesActor : ReceiveActor
{
public NeverRepliesActor() => ReceiveAny(_ => { });
}
}
tests/ZB.MOM.WW.ScadaBridge.Communication.Tests/Grpc/SiteStreamGrpcClientFactoryDisposeTests.cs
+1 -1
View File
@@ -41,7 +41,7 @@ public class SiteStreamGrpcClientFactoryDisposeTests
public IReadOnlyCollection<TrackingClient> Created => _created.ToList();
protected override SiteStreamGrpcClient CreateClient(string grpcEndpoint)
protected override SiteStreamGrpcClient CreateClient(string siteIdentifier, string grpcEndpoint)
{
var client = new TrackingClient();
_created.Add(client);
tests/ZB.MOM.WW.ScadaBridge.Communication.Tests/Grpc/SiteStreamGrpcClientFactoryTests.cs
+1 -1
View File
@@ -155,7 +155,7 @@ public class SiteStreamGrpcClientFactoryTests
{
public TrackingEndpointFactory() : base(NullLoggerFactory.Instance) { }
public int CreatedCount { get; private set; }
protected override SiteStreamGrpcClient CreateClient(string grpcEndpoint)
protected override SiteStreamGrpcClient CreateClient(string siteIdentifier, string grpcEndpoint)
{
CreatedCount++;
return new TrackingEndpointClient(grpcEndpoint);

Some files were not shown because too many files have changed in this diff Show More