feat(grpc): Phase 1B — central→site command plane over gRPC (default Akka) #27

2026-07-22T22:39:02-04:00

dohertj2 commented

2026-07-22 22:39:02 -04:00

Phase 1B of the ClusterClient→gRPC migration: the central→site command plane now has a gRPC implementation behind a flag. Default is still Akka — nothing moves off ClusterClient by default. Rollback = flip ScadaBridge:Communication:SiteTransport back. Rebased onto 1A (PR #26).

What changed

site_command.proto (scadabridge.sitecommand.v1, 6 oneof RPCs / 28 commands) + SiteCommandDtoMapper with round-trip goldens for all 28 commands + 22 replies + 18 nested types, guarded by reflection so a new command with no golden fails the build. Checked-in codegen (no active <Protobuf> item).
SiteCommandDispatcher — one routing table shared by the Akka SiteCommunicationActor and the new gRPC SiteCommandGrpcService, so the two transports can't drift. Parked commands stay node-local (replicated store); TriggerSiteFailover acks before it leaves (dry-run resolve + deferred CommitLeave), pinned by two ordering tests. SiteCommandService added to the shared interceptor's DefaultGatedPrefixes (no second constructor — the Phase-0 defect stays fixed).
Central-side ISiteCommandTransport in CentralCommunicationActor — AkkaSiteTransport (today's per-site ClusterClient path, extracted) and GrpcSiteTransport (new: SitePairChannelProvider with sticky failover/failback, per-site PSK via ISitePskProvider, per-command deadlines matched to today's Ask timeouts, no cross-node retry on DeadlineExceeded). Rides the existing DB-refresh loop for gRPC channels — one poll loop, not two. CommunicationService and SiteCallAuditActor untouched.

Live gate — PASS (proportionate)

docs/plans/2026-07-22-clusterclient-to-grpc-live-gate.md. With central flipped to SiteTransport=Grpc: central→site rides authenticated gRPC SiteCommandService (ExecuteQuery, ExecuteParked → 200) for all three sites, each under its own SB-GRPC-PSK-{site}, 0 auth failures; central's :5000 + :8083 both intact (1A's Kestrel fix carried through).

Plan-vs-code findings (recorded)

SiteTransport is a central-wide flag, so "flip for site-a only" isn't achievable and command-plane per-site coexistence can't be shown (the site→central plane in 1A is per-site). The gate instead proves all three sites over gRPC with distinct keys.
Plan's deadline table was wrong twice: DeploymentStateQuery and TriggerSiteFailover use QueryTimeout, not LifecycleTimeout — matched to the real Ask sites.
A third SiteEnvelope producer exists (DebugStreamBridgeActor), so the reply plumbing routes to a real actor sender, not only Ask temp actors.

Deferred (honest)

Tag commands / lifecycle / standby parked-retry (no deployed instance) and TriggerSiteFailover (destructive, UI-only) — unit-proven, live-exercised in Phase 3's full UI command matrix.

Verification

Rebased onto 1A-merged main; conflicts (both tracks restructured SiteCommunicationActor, AkkaHostedService, CommunicationOptions, Program.cs) resolved keeping both seams. dotnet build 0/0. Communication.Tests 662, Host.Tests 420 — full runs, green.

Phase 1B of the ClusterClient→gRPC migration: the **central→site command plane** now has a gRPC implementation behind a flag. Default is still `Akka` — nothing moves off ClusterClient by default. Rollback = flip `ScadaBridge:Communication:SiteTransport` back. Rebased onto 1A (PR #26). ## What changed - **`site_command.proto`** (`scadabridge.sitecommand.v1`, 6 oneof RPCs / **28 commands**) + `SiteCommandDtoMapper` with **round-trip goldens for all 28 commands + 22 replies + 18 nested types**, guarded by reflection so a new command with no golden fails the build. Checked-in codegen (no active `<Protobuf>` item). - **`SiteCommandDispatcher`** — one routing table shared by the Akka `SiteCommunicationActor` and the new gRPC `SiteCommandGrpcService`, so the two transports can't drift. Parked commands stay **node-local** (replicated store); `TriggerSiteFailover` acks **before** it leaves (dry-run resolve + deferred `CommitLeave`), pinned by two ordering tests. `SiteCommandService` added to the shared interceptor's `DefaultGatedPrefixes` (no second constructor — the Phase-0 defect stays fixed). - **Central-side `ISiteCommandTransport`** in `CentralCommunicationActor` — `AkkaSiteTransport` (today's per-site ClusterClient path, extracted) and `GrpcSiteTransport` (new: `SitePairChannelProvider` with sticky failover/failback, per-site PSK via `ISitePskProvider`, per-command deadlines matched to today's Ask timeouts, no cross-node retry on `DeadlineExceeded`). Rides the existing DB-refresh loop for gRPC channels — one poll loop, not two. `CommunicationService` and `SiteCallAuditActor` untouched. ## Live gate — PASS (proportionate) `docs/plans/2026-07-22-clusterclient-to-grpc-live-gate.md`. With central flipped to `SiteTransport=Grpc`: central→site rides authenticated gRPC `SiteCommandService` (`ExecuteQuery`, `ExecuteParked` → 200) for **all three sites**, each under its own `SB-GRPC-PSK-{site}`, **0 auth failures**; central's `:5000` + `:8083` both intact (1A's Kestrel fix carried through). ## Plan-vs-code findings (recorded) - `SiteTransport` is a **central-wide** flag, so "flip for site-a only" isn't achievable and command-plane per-site coexistence can't be shown (the site→central plane in 1A *is* per-site). The gate instead proves all three sites over gRPC with distinct keys. - Plan's deadline table was wrong twice: `DeploymentStateQuery` and `TriggerSiteFailover` use `QueryTimeout`, not `LifecycleTimeout` — matched to the real Ask sites. - A **third** `SiteEnvelope` producer exists (`DebugStreamBridgeActor`), so the reply plumbing routes to a real actor sender, not only Ask temp actors. ## Deferred (honest) Tag commands / lifecycle / standby parked-retry (no deployed instance) and `TriggerSiteFailover` (destructive, UI-only) — unit-proven, live-exercised in Phase 3's full UI command matrix. ## Verification Rebased onto 1A-merged main; conflicts (both tracks restructured `SiteCommunicationActor`, `AkkaHostedService`, `CommunicationOptions`, `Program.cs`) resolved keeping both seams. `dotnet build` 0/0. Communication.Tests 662, Host.Tests 420 — full runs, green.

dohertj2 added 5 commits 2026-07-22 22:39:03 -04:00

feat(grpc): T1B.1 — site_command.proto + SiteCommandDtoMapper + round-trip goldens 59b13d317b

Phase 1B's contract slice: the wire shape and the canonical translation for the
28 central→site commands that leave ClusterClient. No behaviour changes yet —
SiteCommunicationActor and CentralCommunicationActor are untouched; the
dispatcher refactor (T1B.2) and the central transport seam (T1B.3) consume this.

Protos/site_command.proto (package scadabridge.sitecommand.v1, service
SiteCommandService): six domain RPCs, each with a `oneof` request/reply
envelope. The grouping is what carries deadline policy — every command inside a
group shares a CommunicationOptions timeout class today, so one RPC per group
keeps the deadline choice in one place on the client and one dispatch switch on
the server, while the oneof keeps each command individually typed:
ExecuteLifecycle(6) · ExecuteOpcUa(8) · ExecuteQuery(4) · ExecuteParked(5) ·
ExecuteRoute(4) · TriggerFailover(1). IntegrationCallRequest — the 29th entry on
SiteCommunicationActor's receive table — is deliberately excluded as dead code
(2026-07-22-integration-call-routing-is-dead-code.md).

Contract decisions worth knowing:

- Nullable COLLECTIONS ride in per-collection wrapper messages
  (DeployArtifactsCommand's six artifact lists, CertTrustResult.Certs,
  RouteToCallRequest.Parameters). proto3 repeated/map collapses null into empty,
  and that distinction is live at the site — the same silent-data-loss class the
  transport round-trip guard exposed in PLAN-05 T8. Goldens cover null, empty
  and populated for each.
- Nullable strings use the empty-string-means-null convention already set by
  AuditEventDtoMapper, with ONE exception: RouteToWaitForAttributeRequest's
  TargetValueEncoded, where "wait for the empty string" is a real target, so it
  carries a StringValue wrapper. Both behaviours are asserted, not assumed.
- Nullable enums ride in one-field messages (proto3 enums have no presence and
  no stock wrapper). Enum translation is an explicit switch in both directions —
  never by ordinal — so reordering a C# enum cannot re-map the wire; every wire
  enum reserves 0 for _UNSPECIFIED and decodes to a documented safe default
  rather than faulting a command from a version-skewed peer.
- New LooseValueCodec carries the surviving `object?` members (script params and
  return values, attribute values, tag read/write values) as a type-tagged union
  so a boxed value keeps its runtime CLR type, as it does today under Akka's
  type-preserving JSON serializer. Dates ride as invariant round-trip strings,
  not Timestamp, which would silently normalise away DateTime.Kind and
  DateTimeOffset.Offset. Lists/maps recurse; anything outside the tagged set
  falls back to JSON and is documented as CLR-type-lossy.
- DebugViewSnapshot gets its own full-fidelity alarm/attribute messages rather
  than reusing sitestream's AlarmStateUpdate, which flattens values to display
  strings — right for a live stream, lossy for a snapshot the UI treats as
  authoritative. The encoder omits an AlarmStateChanged.Condition that already
  equals the record's derived default, so computed alarms round-trip exactly
  (record equality compares the nullable backing field, not the property).

Tests are reflection-driven so the coverage cannot drift: the round-trip theory
enumerates the mapper's own ToProto overloads, the envelope guards enumerate the
generated oneof descriptors, and a missing golden fails the build. 216 new tests
green (Communication 532 total, Commons 684 total, solution build 0/0).

Codegen is checked in under SiteCommandGrpc/ per the sitestream recipe; the
<Protobuf> ItemGroup stays commented out (an active one segfaults protoc in the
linux_arm64 Docker image). docker/regen-proto.sh now handles every proto in that
ItemGroup instead of just sitestream, and re-comments idempotently.

feat(comm): extract SiteCommandDispatcher; site serves commands over gRPC too (T1B.2) 518c699b90

Refactor SiteCommunicationActor's central→site routing table into one
SiteCommandDispatcher — the single routing truth for the 28 migrated commands
(IntegrationCallRequest, the dead 29th, stays on the actor and out of the
dispatcher). The Akka actor and the new SiteCommandGrpcService both route through
one dispatcher instance so the two transports can never drift on where a command
goes. Server-side only: nothing central flips to gRPC yet (that is T1B.3);
ClusterClient remains the live path.

Decisions worth recording:

- Targets preserved byte-for-byte. Lifecycle/OPC UA/query/route → the Deployment
  Manager singleton proxy; DeployArtifacts/EventLog/parked → their null-guarded
  handlers with the exact same "handler not available" replies; the parked
  handler stays NODE-LOCAL (per-node replicated-store owner), never the singleton
  proxy — pinned by a dispatcher test that asserts the target is the parked probe
  and NOT the dm proxy.

- Sender preservation intact. The actor's command handlers became thin
  DispatchCommand delegations that still Forward (central Ask → reply routes
  straight back); the existing SiteCommunicationActorTests pass unchanged, which
  is the regression guard for that plumbing. UnsubscribeDebugView keeps its
  fire-and-forget shape: the actor Forwards, the gRPC service Tells + returns the
  synthetic UnsubscribeDebugViewAck so a unary RPC still answers.

- Ack-before-Leave on failover. The dispatcher's PrepareFailover resolves the
  standby with a DRY-RUN (no leave) to build the ack, and hands back a deferred
  CommitLeave; the gRPC service returns the ack, then schedules the real
  Cluster.Leave — so a caller reaching the very node about to leave still gets its
  ack instead of a broken stream. The actor path keeps today's coupled
  resolve-and-leave (over ClusterClient the ack Tell only enqueues, so order is
  immaterial). Proven at both levels: a dispatcher test asserts the ack is built
  before CommitLeave runs, and a TestServer test asserts the recorded seam order
  is resolve-then-leave.

- ControlPlaneAuthInterceptor gates SiteCommandService by EXTENDING
  DefaultGatedPrefixes (descriptor-derived), not by adding a constructor — the
  one-public-ctor invariant and its test stay green.

Tests: SiteCommandDispatcherTests (28-command routing incl. parked node-locality
and both failover paths) and SiteCommandGrpcService TestServer tests (auth,
readiness→Unavailable, one command per oneof group, failover ordering). Full
solution build 0/0; Communication.Tests 574 and Host.Tests 377 green. No active
<Protobuf> item.

feat(comm): T1B.3 — central-side gRPC site-command transport seam (default Akka) 86ad4d5c8e

Extract the central→site send path in CentralCommunicationActor behind a new
ISiteCommandTransport, selected by ScadaBridge:Communication:SiteTransport
(Akka | Grpc, default Akka — rollback = flip the flag). CommunicationService's
27 commands, SiteCallAuditActor's 2 parked relays and DebugStreamBridgeActor's
subscribe/unsubscribe are untouched; the seam sits below SiteEnvelope.

- AkkaSiteTransport: today's per-site ClusterClient path extracted verbatim
  (the _siteClients lookup + ClusterClient.Send with the reply-to sender
  preserved, and the "no client ⇒ warn + drop, caller's Ask times out" path).
- GrpcSiteTransport: dials the site SiteCommandService (T1B.1 proto client) via
  SiteCommandDtoMapper, PSK + x-scadabridge-site on the channel through
  ControlPlaneCredentials, per-command deadlines set EQUAL to today's
  CommunicationService Ask timeouts (per-command, not per-group: DeploymentState
  query and TriggerSiteFailover use QueryTimeout; the two parked relays map to
  QueryTimeout so SiteCallAudit's inner RelayTimeout 10s < 30s ordering holds;
  WaitForAttribute keeps its dynamic Timeout + IntegrationTimeout).
- SitePairChannelProvider: per-site A/B channel pair with sticky failover
  (flip only on Unavailable — NEVER on DeadlineExceeded, a write/deploy/failover
  may have run), background failback probe to the preferred node with 1s→60s
  doubling backoff, PSK invalidation on site removal. Fed by the SAME DB refresh
  loop (extended to carry GrpcNodeA/GrpcNodeBAddress) — no second poll.

Tests: actor-with-substitute-transport (routing, Ask-reply plumbing, per-site
lifecycle across refreshes), ResolveDeadline pinned to each command's current
Ask timeout, and GrpcSiteTransport/SitePairChannelProvider over dual in-process
TestServers (PSK+header+deadline attached, Unavailable failover + stickiness,
failback to preferred, no-retry-on-DeadlineExceeded). Proto csproj untouched
(no active <Protobuf> item). Full solution builds 0 warnings; Communication.Tests
607 green with Akka default.

docs(grpc): Phase 1B live gate — PASS (proportionate); central→site rides authenticated gRPC for 3 sites; records the central-wide SiteTransport finding 01693b13db

docs(plans): tick 1B DoD (proportionate rig gate PASS) 2fa5e93c73

dohertj2 commented

2026-07-22 22:39:51 -04:00

Merged into main by fast-forward at 2fa5e93c (branch was rebased on current main, so a clean ff — no merge commit). Merged tree builds 0/0; Communication.Tests 662 + Host.Tests 420 green. Closing.

Merged into `main` by fast-forward at `2fa5e93c` (branch was rebased on current main, so a clean ff — no merge commit). Merged tree builds 0/0; Communication.Tests 662 + Host.Tests 420 green. Closing.

dohertj2 closed this pull request

2026-07-22 22:39:51 -04:00

Pull request closed

Please reopen this pull request to perform a merge.

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: dohertj2/ScadaBridge#27