ScadaBridge/docs/plans/2026-07-22-clusterclient-to-grpc-live-gate.md

ClusterClient → gRPC migration — live gate results

Rig: docker/ (2 central + 3×2 site + traefik), rebuilt from the branch under test via bash docker/deploy.sh. Recorded check-by-check in the family's live-gate format. Phase 5's full eight-check gate is recorded further down as those phases land; this file starts with Phase 0, whose DoD has its own smaller gate.

---

Phase 0 — PSK auth + dead-code removal — PASS (2026-07-22)

Branch feat/grpc-phase0-psk @ 228ff8b4. Image rebuilt, all 9 containers recreated.

Baseline (pre-change build, same rig)

An unauthenticated call from the host to a site's audit-pull RPC was accepted:

$ grpcurl -plaintext -d '{"batch_size":1}' localhost:9023 sitestream.SiteStreamService/PullAuditEvents
{}

That is the gap Phase 0 closes, reproduced rather than assumed.

Checks

#	Check	Result
1	All 8 nodes boot with keys configured (new StartupValidator rule)	PASS — all recreated and reached ready
2	Unauthenticated PullAuditEvents ⇒ PermissionDenied, all 3 sites	PASS
3	Wrong key (site-b's key presented to site-a) ⇒ PermissionDenied	PASS — per-site scoping is real, not decorative
4	Correct key ⇒ success, all 3 sites	PASS
5	Central's own authenticated paths still work	PASS — 14 successful PullAuditEvents from central to site-a; 0 auth failures in either central's log
6	LocalDb sync unaffected by the new interceptor	PASS — 0 control-plane rejections and 0 sync auth failures on the passive peer; session connected after one boot-order retry
7	No interceptor-activation errors	PASS — 0 (see the defect below)

Evidence for 2–4:

=== NO CREDENTIALS ===
  :9023 -> ERROR: Code: PermissionDenied  Message: Control plane authentication failed.
  :9033 -> ERROR: Code: PermissionDenied  Message: Control plane authentication failed.
  :9043 -> ERROR: Code: PermissionDenied  Message: Control plane authentication failed.

=== WRONG KEY (site-b's key against site-a) ===
  :9023 -> ERROR: Code: PermissionDenied  Message: Control plane authentication failed.

=== CORRECT KEY ===
  site-a :9023 -> {}
  site-b :9033 -> {}
  site-c :9043 -> {}

Site-a rejected exactly 2 calls — the two deliberate probes above — and nothing else.

Defect the gate caught that the test suite did not

First run of this gate FAILED, and is worth recording because the failure mode is deceptive.

Grpc.AspNetCore activates a type-registered interceptor through InterceptorRegistration.GetFactory(), which throws when more than one public constructor is applicable. ControlPlaneAuthInterceptor shipped with two — the DI one and a prefix-set overload intended for later phases.

The throw happens inside the pipeline, per call, so:

• nothing failed at startup; the node booted, joined its pair and reported healthy;
• every gated call died with Unknown / "Exception was thrown by handler", which reads as a handler bug rather than an auth bug;
• correct key, wrong key and no key produced identical errors — the tell. A gate that cannot distinguish those is not authenticating anything.

Site-a's log at the time: three PullAuditEvents calls, three identical System.InvalidOperationException: Multiple constructors accepting all given argument types have been found in type 'ControlPlaneAuthInterceptor'.

The full suite was green when this shipped — 29 suites, 6,872 tests, 0 failures. The in-process end-to-end test missed it because it registered the interceptor with AddSingleton alongside AddGrpc, so DI returned the instance and gRPC's activation path never ran.

Fixed in 228ff8b4: the prefix-set constructor is internal; the end-to-end harness now registers exactly as Program.cs does (by type, not in DI); and a reflection assertion pins "exactly one public constructor", since that is the actual invariant.

Lesson for phases 1A/1B, which both add services to this interceptor: extend DefaultGatedPrefixes; do not add a second public constructor. And any in-process harness for a DI-activated component must mirror the production registration shape or it proves less than it appears to.

Test suite alongside the gate

Non-Playwright: 29 suites, 6,872 tests, 0 failures.

Playwright (against this rig): 170 passed, 2 failed, 1 skipped of 173. Both failures were run down to root cause and both are pre-existing on main, unrelated to Phase 0 — this branch touches no EF, CentralUI, Transport or ManagementService file (git diff --stat main...HEAD -- src/ is 15 files, all Communication/Host/AuditLog gRPC plumbing).

An earlier run of this suite reported 44 failures. That run is void: a docker/deploy.sh was recreating the cluster underneath it, so the fast LoginTests/NavigationTests failures were "app unreachable", not defects.

1. TransportImportTests.ImportSyntheticBundle_AppliesAndShowsAuditDrillIn — a real production bug, not a test defect. Central's log during the failure:

[ERR] An exception occurred while iterating over the results of a query ...
System.InvalidOperationException: The configured execution strategy
'SqlServerRetryingExecutionStrategy' does not support user-initiated transactions.
  at Microsoft.EntityFrameworkCore.Query.Internal.SplitQueryingEnumerable`1.AsyncEnumerator.MoveNextAsync()

BundleImporter.cs:1298 opens a user-initiated transaction; the central context is configured with EnableRetryOnFailure (ConfigurationDatabase/ServiceCollectionExtensions.cs:33). SQL Server's retrying strategy refuses to run a split query inside a caller's transaction, so bundle import fails against real MS SQL. The fix is the one the exception names: wrap the transaction in Database.CreateExecutionStrategy().ExecuteAsync(...).

Why the whole unit/integration suite is green on it: those tests use the in-memory EF provider, which has no retrying execution strategy — and BeginTransactionAsync is a no-op there. The comment directly above line 1298 documents that divergence without drawing the conclusion. Only a rig-backed test can see this.

2. SmsNotificationE2ETests.SmsConfigPage_CreateOrRender_NeverLeaksAuthToken — a stale test fixture. No server-side error at all: the page renders 200, and no INSERT INTO SmsConfigurations is ever issued. The test's fixture SID is ACtest123 (d6ead8ae, 2026-06-19). SmsConfiguration.razor:231 rejects anything not matching ^AC[0-9a-fA-F]{32}$, added by 40088a21 (2026-07-10) to close an un-escaped URI-interpolation hole. Save() sets _formError and returns — no toast, exactly as observed. The fixture was never updated.

This has been failing since 2026-07-10, and it matters more than a red line: everything after the toast assertion — including the secret-non-leak assertion that the Auth Token value never reaches the page HTML — has not executed since. Fix is a valid 32-hex SID in the fixture.

Not covered by this gate

• Streaming subscriptions were exercised in-process (TestServer), not over the rig. The interceptor is path-scoped, not method-scoped, so the rig's PullAuditEvents evidence covers the same code path — but a live SubscribeInstance under load is untested here.
• Key rotation on a live pair.
• docker-env2 was updated with its own key but not redeployed or gated.

---

Phase 1A — central control plane (site→central over gRPC) — PASS (2026-07-22)

Branch feat/grpc-central-control @ 0e162cb2. Rig rebuilt; site-a flipped to CentralTransport=Grpc with CentralGrpcEndpoints=[central-a:8083, central-b:8083], site-b/c left on Akka (default) to prove coexistence. The site-a flag flip was a DoD-test-only rig edit — reverted from the branch, never committed (the plan keeps the default Akka until Phase 4).

The defect this gate caught (T1A.2 shipped it; fixed in 0e162cb2)

First rebuild: central's entire HTTP surface was gone. central-a logged only Now listening on: http://[::]:8083 — no :5000. Central UI, the Management + Inbound API, and every /health/* endpoint (Traefik routing + IActiveNodeGate both depend on them) were dead. The node booted, joined the cluster and served gRPC fine; no startup error.

Cause: builder.WebHost.ConfigureKestrel(o => o.ListenAnyIP(8083, Http2)) puts Kestrel into explicit-endpoints mode, which suppresses the URLs from ASPNETCORE_URLS/--urls — it is not additive, contrary to the comment T1A.2 shipped. Central's whole HTTP/1 surface lives on that URL (http://+:5000 on the rig; a different port in production). The site branch has the same ConfigureKestrel shape but nothing on ASPNETCORE_URLS to lose — it binds every port it needs explicitly — which is why the pattern looked safe.

Every unit + E2E test uses TestServer, which never binds real Kestrel, so the whole suite (6,872) stayed green. Only a live node exposes a missing listener. Fix: parse the port(s) from the configured URLs and re-declare them (Http1AndHttp2) alongside the gRPC port (Http2) in the one ConfigureKestrel call — Program.ParseHttpBindPorts + CentralHttpBindPortsTests (14 cases). After the fix: Now listening on: http://[::]:5000 and :8083; 9001 ready 200/active Healthy, 9002 standby, Traefik LB 200, CLI over the LB works.

Checks (post-fix rebuild)

#	Check	Result
1	site-a rides authenticated gRPC to CentralControlService	PASS — Heartbeat, ReportSiteHealth, ReconcileSite all HTTP/2 → 200; 0 auth failures on either central
2	Heartbeat drives the active flag	PASS — 144+ heartbeats over gRPC; site-a online=True at central
3	Health page live	PASS — ReportSiteHealth lands; central shows site-a online=True, sequence advancing, alongside Akka site-b/c
4	Reconcile works after site restart	PASS — restarted site-a-a; it logged Site→central transport: gRPC to 2 central endpoint(s), then Reconcile pass … complete: 0 fetched, 0 failed, 0 orphan(s)
5	Coexistence	PASS — site-b/c log Created ClusterClient to central; both online=True — Akka and gRPC sites side by side
6	Central HTTP surface intact under the new gRPC listener	PASS — after the fix (see above)

Not independently exercised on this gate

• Notification e2e (SubmitNotification/QueryNotificationStatus) and audit ingest (IngestAuditEvents/IngestCachedTelemetry) were NOT driven live: the rig has templates but no deployed instance, so nothing emits site→central notifications or audit rows on its own. These four RPCs traverse the identical CentralControlGrpcService → CentralCommunicationActor Ask path that checks 1–3 proved live under real auth, and their payload mappers carry 32 round-trip goldens — but the payloads themselves were not put over the wire here. Phase 2's central-kill S&F soak is where they get their live workout; flag for a fuller 1A proof if a deployed-instance rig is set up before then.
• Cross-node failover/failback of the site→central channel under a central-node kill (unit-proven via TestServer; not exercised on the rig at 1A).

---

Phase 1B — site command plane (central→site over gRPC) — PASS (proportionate) (2026-07-23)

Branch feat/grpc-sitecommand (rebased onto 1A-merged main). Rig rebuilt with central flipped to SiteTransport=Grpc — a DoD-test-only edit to both central appsettings, reverted from the branch (default stays Akka).

SiteTransport is a central-wide flag (CentralCommunicationActor.SelectTransport picks one transport for all sites), so the plan's "flip for site-a only" is not achievable — the flip routes central→site commands for all three sites to gRPC. Command-plane per-site coexistence therefore cannot be shown (unlike the site→central plane in 1A, which is per-site). This is a plan-vs-code finding, recorded rather than worked around.

Checks

#	Check	Result
1	central→site commands ride authenticated gRPC SiteCommandService	PASS — ExecuteQuery (event-log) and ExecuteParked (parked query) HTTP/2 → 200
2	Per-site PSK resolution across all sites	PASS — site-a, site-b, site-c each answered ExecuteQuery → 200 under its own SB-GRPC-PSK-{site}; 0 auth failures on any site node
3	Central HTTP surface intact under the central-wide gRPC flip	PASS — central :5000 and :8083 both listening; 9001 ready 200, LB 200 (the 1A Kestrel fix carried through the merge)
4	Query round-trips return correct data	PASS — every health event-log/parked-messages returned success:true with the right siteId and empty result sets (bare rig)

Not driven on this proportionate gate

• TriggerSiteFailover — unit-proven (two ordering tests pin ack-before-Leave via the dispatcher's dry-run resolve + deferred CommitLeave), but not live-driven here: it is destructive (forces the active node to leave) and has no CLI verb (UI/management-only).
• Tag commands (BrowseNode/ReadTagValues/WriteTag) and the lifecycle enable/disable matrix — need a deployed instance + data connection the bare rig lacks (same blocker as 1A).
• Parked retry against the STANDBY node — needs a parked operation to exist, which needs a deployed instance.
• Command-plane coexistence (site-b/c on Akka while site-a on gRPC) — not expressible; the flag is central-wide (check-1/2 instead prove all three sites over gRPC with distinct keys).

The instance-dependent matrix (tag ops, lifecycle, standby parked retry) and TriggerSiteFailover get their live exercise in Phase 3's full UI command matrix; the transport itself is proven here.

---

Phase 2 — full site→central cutover + S&F soak — PASS (2026-07-23)

All three sites (all 6 nodes) flipped to CentralTransport=Grpc (edit to docker/site-*/appsettings.Site.json, reverted in git after the gate — defaults stay Akka), rebuilt from main + --force-recreate. Central left on SiteTransport=Akka (Phase 3 owns that direction). The gate was driven by a live S&F workload, closing the notification/audit path that 1A/1B could only unit-prove.

S&F driver

A minimal dependency-free template SoakNotify (one 5 s Interval script: Notify.To("Engineering Alerts").Send(...)), 3 instances deployed+enabled on site-a → a steady 3 notifications per 5 s bucket (the pre-existing Motor Controller "soak-motor" instances need 30 OPC UA bindings and were unusable). The central dbo.Notifications table (one row per NotificationId, insert-if-not-exists) is the no-loss/no-dupes source of truth.

Checks

#	Check	Result
1	All 6 site nodes on CentralTransport=Grpc	PASS — each logs Site→central transport: gRPC to 2 central endpoint(s); 0 PermissionDenied across all six for the whole run
2	Full control plane rides gRPC CentralControlService	PASS — central sees Heartbeat, ReportSiteHealth, SubmitNotification (the S&F path), IngestAuditEvents — 196 RPCs / 90 s, 0 non-200
3	Health/heartbeat cadence unchanged, no sequence regressions	PASS — CentralHealthAggregator logged 0 sequence-regression/out-of-order lines; heartbeat steady at ~144/30 s across 3 sites
4	Single-node failover — hard-kill the active central (central-a)	PASS — CentralChannelProvider logged sticky failover central-a:8083 → central-b:8083 at the instant of kill; central-b active in 29 s (auto-down); notif count froze at 72 during the gap
5	Buffer drains, no loss/dupes (single-node)	PASS — count resumed 72→101; every 5 s bucket through the outage = exactly 3, no gap; 101 total == 101 distinct
6	Failback — restart central-a	PASS — rejoined ready in ~5 s as standby (active=503); central-b retained active (oldest-Up, no role flap); traffic uninterrupted (uniform 3/bucket across failback); central-a singletons → Younger
7	Full central outage — hard-kill both central (~59 s)	PASS — count frozen at 155 for the entire outage; sites buffered continuously
8	Cold re-form + drain, no loss/dupes (both-down)	PASS — cold cluster re-formed, central-b active in ~14 s; ~42 buffered notifications drained; every 5 s bucket across the whole ~59 s both-dead window = exactly 3, no gap; 216 total == 216 distinct

Checks 4–8 also stand as a live preview of Phase 5 checks 4 (failover/failback) and 5 (mid-drain kill, zero loss/zero dupes).

Notes

• Audit telemetry also rides the new gRPC CentralControlService (IngestAuditEvents) — the SiteAuditTelemetryActor "ClusterClientSiteAuditClient" label is legacy naming, not the wire path. So CentralTransport=Grpc moves heartbeat, health, notification S&F and audit off ClusterClient in one flip.
• Sites settled on central-a as the gRPC endpoint after the cold both-restart while central-b held the active/singleton role — the gRPC endpoint node and the singleton host can differ; central-a receives the forward and Akka-routes to the NotificationOutboxActor singleton on central-b. Both are correct and independent.
• Some notifications land Parked at central (no SMTP config on the bare rig) — irrelevant to the transport proof: the Notifications row is written on forward regardless of downstream SMTP delivery, so the count is a faithful no-loss/no-dupes measure.
• Rig left running on the gRPC build; git config reverted to Akka default (a redeploy from main resets to all-Akka).