dohertj2/ScadaBridge
1
0
Fork 0
Code Issues 2 Pull Requests 1 Actions Packages Projects Releases Wiki Activity

feat(grpc): Phase 1A — site→central control plane over gRPC (default Akka) #26

Closed
dohertj2 wants to merge 0 commits from feat/grpc-central-control into main
pull from: feat/grpc-central-control
merge into: dohertj2:main
Conversation 1 Commits - Files Changed -
dohertj2 commented 2026-07-22 20:01:15 -04:00
Owner
Copy Link
Copy Source

Phase 1A of the ClusterClient→gRPC migration: the site→central control plane now has a gRPC implementation behind a flag. Default is still Akka — nothing moves off ClusterClient by default. Rollback = flip ScadaBridge:Communication:CentralTransport back.

What changed

  • central_control.proto (scadabridge.centralcontrol.v1, 7 RPCs) + CentralControlDtoMapper with 32 round-trip goldens (verified to have teeth by mutation). Checked-in codegen (no active <Protobuf> item — the arm64 Docker build stays intact).
  • Central hosts CentralControlService — decodes → Asks the existing CentralCommunicationActor (zero handler changes) → encodes. Gated by a new CentralControlAuthInterceptor that verifies the Bearer PSK against the per-site key (ISitePskProvider, keyed by the required x-scadabridge-site header), fail-closed. Own single public constructor, pinned by a reflection test.
  • Site-side ICentralTransport seam — AkkaCentralTransport (today's ClusterClient path, extracted verbatim) and GrpcCentralTransport (new: CentralChannelProvider with sticky failover/failback, PSK, per-call deadlines matched to today's Ask timeouts, no cross-node retry on DeadlineExceeded). Selected by CentralTransport (default Akka).
  • The three above-seam suites (NotificationForwarderTests, HealthReportSenderTests, SiteAuditTelemetryActorTests) pass unmodified — the plan's own signal the seam is placed right.

Live gate — PASS, and it caught a real regression

Full writeup in docs/plans/2026-07-22-clusterclient-to-grpc-live-gate.md. With site-a flipped to gRPC and site-b/c left on Akka: heartbeat drives the active flag, health lands (site-a online, advancing sequence), reconcile-after-restart works, 0 auth failures, and Akka + gRPC sites coexist.

The rig caught a central-down defect the whole 6,872-test suite missed (0e162cb2): T1A.2's ConfigureKestrel(ListenAnyIP(8083)) put Kestrel into explicit-endpoints mode, which suppresses ASPNETCORE_URLS — so binding only the gRPC port silently deleted central's entire :5000 HTTP surface (Central UI, Management/Inbound API, /health/* for Traefik + IActiveNodeGate), with no startup error. TestServer never binds real Kestrel, so no test could see it. Fixed by re-declaring the configured HTTP port(s) alongside the gRPC port (Program.ParseHttpBindPorts + 14 test cases).

Deferred (honest)

Notification-e2e and audit-ingest were not driven live — the rig has no deployed instance to emit them. Same transport + auth + actor path as the three proven RPCs, 32 goldens on the payloads; Phase 2's central-kill S&F soak exercises them for real.

Verification

dotnet build 0/0. Host.Tests 405, Communication.Tests 371 — full runs, green.

Phase 1A of the ClusterClient→gRPC migration: the **site→central control plane** now has a gRPC implementation behind a flag. Default is still `Akka` — nothing moves off ClusterClient by default. Rollback = flip `ScadaBridge:Communication:CentralTransport` back. ## What changed - **`central_control.proto`** (`scadabridge.centralcontrol.v1`, 7 RPCs) + `CentralControlDtoMapper` with **32 round-trip goldens** (verified to have teeth by mutation). Checked-in codegen (no active `<Protobuf>` item — the arm64 Docker build stays intact). - **Central hosts `CentralControlService`** — decodes → `Ask`s the existing `CentralCommunicationActor` (zero handler changes) → encodes. Gated by a **new** `CentralControlAuthInterceptor` that verifies the `Bearer` PSK against the **per-site** key (`ISitePskProvider`, keyed by the required `x-scadabridge-site` header), fail-closed. Own single public constructor, pinned by a reflection test. - **Site-side `ICentralTransport` seam** — `AkkaCentralTransport` (today's ClusterClient path, extracted verbatim) and `GrpcCentralTransport` (new: `CentralChannelProvider` with sticky failover/failback, PSK, per-call deadlines matched to today's Ask timeouts, no cross-node retry on `DeadlineExceeded`). Selected by `CentralTransport` (default `Akka`). - The three above-seam suites (`NotificationForwarderTests`, `HealthReportSenderTests`, `SiteAuditTelemetryActorTests`) pass **unmodified** — the plan's own signal the seam is placed right. ## Live gate — PASS, and it caught a real regression Full writeup in `docs/plans/2026-07-22-clusterclient-to-grpc-live-gate.md`. With site-a flipped to gRPC and site-b/c left on Akka: heartbeat drives the active flag, health lands (site-a online, advancing sequence), reconcile-after-restart works, 0 auth failures, and **Akka + gRPC sites coexist**. **The rig caught a central-down defect the whole 6,872-test suite missed** (`0e162cb2`): T1A.2's `ConfigureKestrel(ListenAnyIP(8083))` put Kestrel into explicit-endpoints mode, which **suppresses `ASPNETCORE_URLS`** — so binding only the gRPC port silently deleted central's entire `:5000` HTTP surface (Central UI, Management/Inbound API, `/health/*` for Traefik + `IActiveNodeGate`), with no startup error. TestServer never binds real Kestrel, so no test could see it. Fixed by re-declaring the configured HTTP port(s) alongside the gRPC port (`Program.ParseHttpBindPorts` + 14 test cases). ## Deferred (honest) Notification-e2e and audit-ingest were not driven live — the rig has no deployed instance to emit them. Same transport + auth + actor path as the three proven RPCs, 32 goldens on the payloads; Phase 2's central-kill S&F soak exercises them for real. ## Verification `dotnet build` 0/0. Host.Tests 405, Communication.Tests 371 — full runs, green.
dohertj2 added 5 commits 2026-07-22 20:01:16 -04:00
feat(grpc): central_control.proto + DTO mapper for the 7 site→central control RPCs (T1A.1) d7455577a8 
Phase 1A of the ClusterClient→gRPC migration needs a wire contract for the
seven messages SiteCommunicationActor forwards to /user/central-communication.
This lands the contract and its mapper only — hosting (T1A.2) and the site-side
transport seam (T1A.3) follow.

`Protos/central_control.proto` (package scadabridge.centralcontrol.v1, service
CentralControlService) declares SubmitNotification, QueryNotificationStatus,
IngestAuditEvents, IngestCachedTelemetry, ReconcileSite, ReportSiteHealth and
Heartbeat. Note the direction is the inverse of SiteStreamService: here the site
dials and central serves.

Decisions worth recording:

- The two ingest RPCs IMPORT sitestream.proto and reuse AuditEventBatch /
  CachedTelemetryBatch / IngestAck rather than redeclaring them. The site
  telemetry actor already builds those messages, so a second copy would fork one
  wire contract into two kept in lockstep by hand. ForwardState / IngestedAtUtc
  stay off-wire exactly as they are today.
- Heartbeat replies google.protobuf.Empty — it is fire-and-forget and must never
  surface a fault onto the heartbeat timer path.
- The three NULLABLE SiteHealthReport collections travel inside single-field
  wrapper messages (ConnectionEndpointMapDto / TagQualityMapDto /
  NodeStatusListDto). proto3 cannot express presence on repeated/map fields, but
  null and empty genuinely differ here — SiteHealthCollector emits
  `ClusterNodes: _clusterNodes?.ToList()` and the central health surface reads
  null as "not reported", not as "reported empty". Same reasoning drives the
  BoolValue/Int64Value/DoubleValue wrappers on LocalDbReplicationConnected,
  LocalDbOplogBacklog and the two age gauges, whose docs are explicit that null
  is not zero/false.
- ConnectionHealthEnum reserves 0 for UNSPECIFIED instead of mapping Connected
  onto it, and the decoder resolves anything unknown to ConnectionHealth.Error.
  An unrecognised connection state must not render as "healthy".
- Guid? execution ids travel as "D" strings with empty meaning null; a malformed
  non-empty value throws rather than being laundered into "no correlation".
- DateTimeOffset normalizes to a UTC instant (a protobuf Timestamp has no
  offset). Lossless in practice — every producer stamps UTC — and documented +
  asserted rather than left implicit.

SiteCallDtoMapper gains a ToDto(SiteCall) overload. Its doc comment previously
asserted such a method "would be dead code"; that held only while ClusterClient
was the sole path from IngestCachedTelemetryCommand (which carries SiteCall, not
SiteCallOperational) to central. Comment corrected alongside.

Golden tests round-trip every message through a real protobuf encode/decode —
DTO → proto → bytes → proto → DTO — with a fully-populated case and a
null/empty/minimal case for every optional member. Verified to have teeth by
mutation: dropping a scalar, collapsing an empty nullable collection to absent,
and nulling a gauge each fail a test.

Codegen stays CHECKED IN under CentralControlGrpc/ (protoc segfaults in the
linux_arm64 Docker image); no active <Protobuf> item is committed.
docker/regen-proto.sh is generalized to `regen-proto.sh [sitestream|
centralcontrol|all]` — it now injects the ItemGroup rather than unwrapping a
comment, so it no longer depends on there being exactly one Protobuf line, and
it restores the csproj verbatim on every exit path.
feat(grpc): host CentralControlService on the central node (T1A.2) 780bb9c369 
Central now ALSO listens for the seven site→central control messages over
gRPC, alongside the existing ClusterClient path. Nothing flips to gRPC yet —
sites keep CentralTransport=Akka (T1A.3's job); central simply starts also
accepting.

- CentralControlGrpcService (Communication.Grpc): decodes each RPC onto the
  SAME in-process message the ClusterClient path carries, Asks the existing
  CentralCommunicationActor (zero handler-logic changes), encodes the reply via
  the T1A.1 mapper. Readiness-gated like SiteStreamGrpcServer.SetReady —
  Unavailable until AkkaHostedService hands the actor over. Heartbeat stays
  fire-and-forget (Tell, always-OK, never gated on readiness). Ingest reuses the
  shared SiteStreamGrpcServer.AuditIngestAskTimeout constant. Fault→status
  mapping is retry-aware: Unavailable (never dispatched, safe to cross-node
  retry) vs DeadlineExceeded/Internal (it ran, do not re-send elsewhere).

- CentralControlAuthInterceptor (Host): a SEPARATE interceptor class, not a
  variant constructor on ControlPlaneAuthInterceptor. Central's model is per-site
  (verify the Bearer token against the key for the site in the required
  x-scadabridge-site header, via ISitePskProvider) where a site verifies its one
  own-key — a genuinely different model. Fail-closed on every branch: missing or
  blank header, unresolvable key, and mismatched token all → PermissionDenied,
  never pass-through. One public constructor only (the explicit-prefix ctor is
  internal), pinned by a reflection test — a second public ctor makes
  Grpc.AspNetCore's GetFactory() throw per-call and silently disables the gate.

- Explicit Kestrel h2c listener on new option ScadaBridge:Node:CentralGrpcPort
  (default 8083, symmetric with sites), mirroring the Site branch. Additive to
  central's :5000 HTTP/1 surface, which is untouched — gRPC does NOT go through
  Traefik (HTTP/1 only). Registered by type on AddGrpc; service mapped with
  MapGrpcService. Port range-validated by NodeOptionsValidator.

- Rig: publish the central gRPC port 9013:8083 / 9014:8083 on both central nodes
  so a later task can exercise it.

Tests: CentralControlEndToEndTests (Host.Tests, TestServer + real interceptor +
real service over a stub actor) proves auth positives/negatives are
distinguishable and covers unary + the ingest bridge shapes; the interceptor is
registered BY TYPE, never in DI. CentralControlAuthInterceptorTests pins the
per-site gate + one-public-ctor invariant. CentralControlGrpcServiceTests
(Communication.Tests, TestKit) covers the readiness gate, fire-and-forget
heartbeat, and the DeadlineExceeded-vs-Unavailable status mapping. No active
<Protobuf> item. Communication.Tests (356) + Host.Tests (384) green.
feat(grpc): site-side ICentralTransport seam + gRPC transport (T1A.3) 33b15f10a4 
Introduce ICentralTransport as the site->central choke point inside
SiteCommunicationActor. The seven site->central sends (notification submit/
status, audit + cached-telemetry ingest, reconcile, health, heartbeat) now
delegate to an injected transport instead of owning ClusterClient.Send inline.

- AkkaCentralTransport: verbatim extraction of today's ClusterClient.Send path,
  including the exact sender-forwarding that routes central's reply straight
  back to the waiting Ask. Default when no transport is injected -> behaviour
  unchanged, existing SiteCommunicationActorTests pass as-is.
- GrpcCentralTransport + CentralChannelProvider: dial CentralControlService with
  sticky failover + background failback (1s-doubling-cap-60s), PSK +
  x-scadabridge-site via ControlPlaneCredentials, per-call deadlines mirroring
  today's Ask timeouts. Cross-node retry ONLY on provably-unsent
  connect failures; never on DeadlineExceeded. Heartbeat stays fire-and-forget.
- StaticSitePskProvider: site's single own-key provider (fail-closed).
- CommunicationOptions: CentralTransport flag (default Akka) + CentralGrpcEndpoints
  (validator: required when transport=Grpc). Host selects the impl; the
  ClusterClient is created only on the Akka path.

Tests: actor-with-fake-transport (7 delegations + fault routing + heartbeat
no-fault), AkkaCentralTransport sender-forwarding, GrpcCentralTransport over
in-process TestServer (failover flip, sticky, failback, PSK+header, deadline,
no-retry-on-deadline), validator. Communication.Tests 371 green, Host.Tests 391
green; the three above-seam suites pass unmodified.
fix(grpc): central Kestrel gRPC listener was dropping the :5000 HTTP surface (T1A.2 regression) 0e162cb250 
Caught on the Phase 1A rig proof: central-a logged only "Now listening on:
http://[::]:8083" and nothing else. Central UI, the Management + Inbound API, and
the /health/* endpoints Traefik + IActiveNodeGate depend on were all gone, with no
startup error — the node booted, joined the cluster and served gRPC fine.

Cause: calling options.ListenAnyIP in ConfigureKestrel puts Kestrel into
explicit-endpoints mode, which SUPPRESSES the URLs from ASPNETCORE_URLS/--urls
entirely — it is not additive, contrary to the comment T1A.2 shipped. Central's whole
HTTP/1 surface lives on that URL (http://+:5000 on the rig, a different port in prod),
so binding only the gRPC port silently deleted it. The site branch has the same shape
but no ASPNETCORE_URLS surface to lose — it binds every port it needs explicitly.

Fix: parse the port(s) from the configured URLs and re-declare them (Http1AndHttp2)
alongside the gRPC port (Http2) in the one ConfigureKestrel call. New
Program.ParseHttpBindPorts + CentralHttpBindPortsTests (14 cases: wildcard/ipv6/
hostname hosts, multi-URL, de-dupe, scheme-default, null/blank, unparseable-skipped).

Why no test caught it originally: unit/E2E tests use TestServer, which never binds
real Kestrel. Only a live node exposes the missing listener — which is exactly what
the rig proof is for.
docs(grpc): Phase 1A live gate — PASS (3 RPCs + restart-reconcile + coexistence); records the Kestrel-drop defect f7c7811940
dohertj2 commented 2026-07-22 20:02:29 -04:00
Author
Owner
Copy Link
Copy Source

Merged into main via merge commit aa60f438 (branch had diverged from main by docs-tracking commits, so a non-ff merge). Merged tree builds 0/0; Host.Tests 405 + Communication.Tests 371 green. Closing.

Merged into `main` via merge commit `aa60f438` (branch had diverged from main by docs-tracking commits, so a non-ff merge). Merged tree builds 0/0; Host.Tests 405 + Communication.Tests 371 green. Closing.
dohertj2 closed this pull request 2026-07-22 20:02:29 -04:00

Pull request closed

Please reopen this pull request to perform a merge.
Sign in to join this conversation.
Reviewers
No Reviewers
Labels
No items
No Label
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
dohertj2 (dohertj2)
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: dohertj2/ScadaBridge#26
Delete Branch "feat/grpc-central-control"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?