diff --git a/docs/plans/2026-07-22-clusterclient-to-grpc-live-gate.md b/docs/plans/2026-07-22-clusterclient-to-grpc-live-gate.md index 329e24ea..43505171 100644 --- a/docs/plans/2026-07-22-clusterclient-to-grpc-live-gate.md +++ b/docs/plans/2026-07-22-clusterclient-to-grpc-live-gate.md @@ -335,3 +335,145 @@ call recovered immediately via site-a-b. Rig left running with **both** transports on gRPC; git config reverted to Akka default (a redeploy from `main` resets to all-Akka). + +--- + +## Phase 5 — full eight-check gate on the deletion build — **PASS** (2026-07-23) + +The migration's terminal gate, run on `main` **after Phase 4** (`7fd5cb2b`) — the build where the +Akka `ClusterClient`/`ClusterClientReceptionist` transport is **physically deleted**, the +`CentralTransport`/`SiteTransport` flags are gone, and gRPC is the *only* site↔central transport +(no flags to flip). Rig rebuilt from `main` via `bash docker/deploy.sh`, all 9 containers recreated +from the new image; central MS SQL and the S&F driver (`SoakNotify` template #2147 / instances +#95–97 / list "Engineering Alerts" #28) persisted from prior phases. + +**Provenance guard:** the pre-existing rig image was built `11:17`, *before* the Phase 4 commit +(`12:54`); it still carried ClusterClient code and would have invalidated checks 7–8. Confirmed +by rebuild timestamp, then by **0** ClusterClient/receptionist log lines across all 8 nodes on the +fresh boot. (The lone `ClusterClientReceptionist` symbol still present is inside +`Akka.Cluster.Tools.dll` — the library we deliberately keep for `ClusterSingleton` — not our code.) + +Header names for the negative probes: `authorization: Bearer ` + `x-scadabridge-site: ` +(`ISitePskProvider.cs`). Docker rig PSKs: `dev-grpc-psk-docker-site-{a,b,c}` +(`ScadaBridge__Communication__SitePsks__site-*`, central override; sites read their own +`GrpcPsk` from mounted `appsettings.Site.json`). + +### Checks + +| # | Check | Result | +|---|---|---| +| 1 | **PSK negatives** | **PASS (with a contract clarification)** — see below | +| 2 | **Site→central matrix** | **PASS** — notifications, both audit paths, health, heartbeat, reconcile all live over gRPC | +| 3 | **Central→site matrix** | **PASS (proportionate)** — `ExecuteQuery`/`ExecuteParked`/`ExecuteLifecycle` live; instance-dependent RPCs carried forward (see below) | +| 4 | **Failover / failback** | **PASS** — active-central kill → sticky flip in 1 s, central-b active in 26 s; failback kept central-b active (oldest-Up, no flap) | +| 5 | **Mid-drain kill, zero loss/dupes** | **PASS** — buffered notifications flushed; **total == distinct** across the outage | +| 6 | **Frame-class retirement (>128 KB)** | **PASS** — a single **297,574-byte** gRPC reply succeeded (Akka's 128 KB frame would have dropped it) | +| 7 | **No cross-boundary Akka association** | **PASS** — each Akka cluster's membership is strictly its own pair; **0** cross-boundary association lines both directions | +| 8 | **Full-rig restart discipline** | **PASS** — all 9 restarted together came up clean; **0** receptionist/ClusterClient lines on any node; sites reconnected over gRPC; S&F resumed with no dupes | + +### Check 1 — PSK negatives (and the contract clarification) + +Against the two gated services on site-a (`:9023`), using the local proto descriptors +(`-proto sitestream.proto` / `site_command.proto`; server reflection is off in this build): + +``` +SiteStreamService/PullAuditEvents: + no credentials -> PermissionDenied "Control plane authentication failed." + wrong key (site-b's key) -> PermissionDenied + correct key -> success (audit events returned) +SiteCommandService/ExecuteQuery: + no credentials -> PermissionDenied + wrong key (site-b's key) -> PermissionDenied +``` + +site-a-a's log recorded **exactly 4** control-plane rejections — the four deliberate negative +probes above, nothing else. + +**Clarification the gate produced:** the plan listed "missing `x-scadabridge-site` ⇒ +`PermissionDenied`" as a negative case. It does **not** hold on the *site* side, and that is +correct by design. `ControlPlaneAuthInterceptor.Authorize` compares the presented bearer against +the node's **own single** `GrpcPsk` (`_options.Value.GrpcPsk`) and never reads +`x-scadabridge-site`. That header is a **central-side routing hint** — `SitePskProvider` uses it to +pick *which* site's key to expect, because central holds many. A single-key site node needs no such +hint. Per-site key isolation — the actual security property — is proven by the **wrong-key** +rejection (site-b's key refused at site-a), not by the header. Recorded rather than "fixed": adding +a header requirement to the site interceptor would be theatre. + +**LocalDb sync unaffected:** the replicated node (site-a) logged one boot-order +`faulted; reconnecting` (peer node-b not yet up), the passive peer accepted the +`/localdb_sync.v1.LocalDbSync/Sync` POST 1 s later, and the health report settled to +`localDbReplicationConnected:true, localDbOplogBacklog:0`. **0** LocalDb sync auth failures — the +control-plane interceptor and the separate `LocalDbSyncAuthInterceptor` don't interfere. site-b/c +report `localDbReplicationConnected:false` — the intended rig posture (only site-a replicated). + +### Check 2 — site→central matrix + +- **Notifications e2e:** `dbo.Notifications` climbed continuously at the driver's ~3 rows / 5 s, + **total == distinct** at every sample (insert-if-not-exists no-loss/no-dupe truth). +- **Both audit paths live over gRPC:** in a 3-minute window, `dbo.AuditLog` carried `node-a`/`node-b` + rows (105 each — **site-originated**, forwarded over the gRPC `IngestAuditEvents` RPC) *and* + `central-a` rows (420 — central-direct-write from the outbox dispatcher). Row count climbing live. +- **Health live per site:** `health summary` shows all three sites `isOnline:true`, sequence numbers + advancing, fresh heartbeats; site-a `enabledInstanceCount:3` with a live `Notification` S&F buffer. +- **Heartbeat → active flag:** central-a `active=200`, central-b `active=503`; each site's report + drives its online flag. +- **Reconcile self-heal:** restarting `site-a-a` produced a fresh (17:11:24) `Site→central transport: + gRPC to 2 central endpoint(s)` then `Reconcile pass … complete: 0 fetched, 0 failed, 0 orphan(s)`; + S&F never dropped across the restart (active node kept emitting), no dupes. +- **Not driven (carry-forward, bare rig):** cached-call telemetry (`SoakNotify` only calls + `Notify.Send`, no `CachedCall`/`CachedWrite`); a notification reaching **Delivered** rather than + `Parked` (no SMTP on the rig — the transport truth is the row-count, per Phase 2). + +### Check 3 — central→site matrix + +`ExecuteQuery` (event-log, all 3 sites → correlationIds + entries), `ExecuteParked` (parked-messages, +all 3 sites → empty, no parked ops on a bare rig), and `ExecuteLifecycle` (disable→enable #95 → +`success:true` both) all rode `SiteCommandService` gRPC; site active nodes logged them +(site-a-a: 8 `ExecuteLifecycle` + 4 `ExecuteParked` + 4 `ExecuteQuery`; site-b-a/c-a: +`ExecuteParked` + `ExecuteQuery`). **Carried forward, unchanged from Phase 3** (needs a deployed +OPC-bound instance / inbound method / a parked op / the destructive UI-only failover verb the bare +rig lacks): `ExecuteOpcUa`, `ExecuteRoute`, standby parked retry/discard, `TriggerFailover` +(the hard-kill in Check 4 is its live equivalent). All four are unit-proven (dispatcher routing ×28, +ack-before-`Leave` ordering). + +### Checks 4 & 5 — failover / failback / mid-drain kill + +Hard-killed the **active** central (central-a) mid-drain: site-a-b logged the sticky endpoint flip +`central-a:8083 → central-b:8083` **1 s** after the kill; central-b reached active in **26 s** +(auto-down). The notification count froze during the gap, then **flushed the buffered rows and +resumed** at the steady 3 / 5 s. Post-drain **total == distinct (4395 == 4395)** — zero loss, zero +duplicates. Failback: restarting central-a, it rejoined and central-b **retained** active +(`a=503` standby / `b=200` active) — oldest-Up, no role flap. + +### Check 6 — frame-class retirement + +`PullAuditEvents{since_utc: 2020-01-01, batch_size: 5000}` against the active site node returned a +single **297,574-byte** reply successfully. Over Akka's default 128 KB frame +(`log-frame-size-exceeding` off) this class of payload was silently dropped and the caller's Ask +timed out (`docs/known-issues/2026-06-26-…`). On gRPC (4 MB default cap) it is a normal reply. + +### Check 7 — no cross-boundary Akka association + +Each Akka cluster's membership is strictly its own pair: central `[central-a, central-b]`, site-a +`[site-a-a, site-a-b]`, site-c `[site-c-a, site-c-b]` — **no site node ever appears in central's +membership and vice versa**. Central's log holds **0** `Association with remote system …site-*` +lines and **0** references to any site address; site-a's log holds **0** references to any central +address. The two boundaries are wired only by per-pair Akka remoting (8082, internal) and the gRPC +control/data planes (8083); there is no Akka association across the site↔central line. + +### Check 8 — full-rig restart discipline + +Restarted all 9 rig containers together (pairs together, honouring the LocalDb-replication constraint). +Central re-formed with central-a active; **0** receptionist/ClusterClient lines across all 8 nodes; +all three sites logged `Site→central transport: gRPC to 2 central endpoint(s)`; S&F resumed +(4407 → 4483, **distinct == total**, no dupes). + +### End state + +The migration is complete and proven on the deletion build. The rig runs the committed all-gRPC +default (Phase 4 flipped the shipped appsettings to `CentralGrpcEndpoints` — unlike Phases 2/3, a +redeploy from `main` now yields all-gRPC, not all-Akka). No ClusterClient remains anywhere in the +running system or the source tree. Deferred, unchanged from earlier phases: the instance-dependent +central→site RPC matrix (`ExecuteOpcUa`/`ExecuteRoute`/standby parked retry) and live +`TriggerFailover`, all unit-proven; PSK **rotation** on a live pair; `docker-env2` was updated with +its own key but not gated here. diff --git a/docs/plans/2026-07-22-clusterclient-to-grpc-plan.md b/docs/plans/2026-07-22-clusterclient-to-grpc-plan.md index c874cd2f..16e0d4e7 100644 --- a/docs/plans/2026-07-22-clusterclient-to-grpc-plan.md +++ b/docs/plans/2026-07-22-clusterclient-to-grpc-plan.md @@ -331,8 +331,9 @@ Critical path ≈ 1B: **~4–6 weeks total**, matching the design estimate. - [x] Grep-gates pass (`CentralContactPoints` → only "replaces the former" doc refs + plan trackers; deleted symbols → 0 live refs; remaining `clusterclient` in src = the misleadingly-named `ClusterClientSiteAuditClient` [transport-agnostic, works unchanged] + stale inline doc-comments, noted as follow-up) - [x] Docs updated: `Component-Communication.md`, `components/Communication.md`, `Component-Host.md`, `Component-StoreAndForward.md`, `topology-guide.md`, `grpc_streams.md`, known-issues frame-size amendment, **CLAUDE.md** transport decisions -**Phase 5 — live gate** (record in `2026-07-22-clusterclient-to-grpc-live-gate.md`) -- [ ] 1 PSK negatives · [ ] 2 site→central matrix · [ ] 3 central→site matrix · [ ] 4 failover/failback both directions · [ ] 5 mid-drain kill · [ ] 6 >128 KB frame-class proof · [ ] 7 no cross-boundary Akka association · [ ] 8 full-rig restart clean +**Phase 5 — live gate** (record in `2026-07-22-clusterclient-to-grpc-live-gate.md`) — **PASS 2026-07-23** (deletion build, `main` @ `7fd5cb2b`) +- [x] 1 PSK negatives · [x] 2 site→central matrix · [x] 3 central→site matrix · [x] 4 failover/failback both directions · [x] 5 mid-drain kill · [x] 6 >128 KB frame-class proof · [x] 7 no cross-boundary Akka association · [x] 8 full-rig restart clean +- All 8 PASS. Notables: check 1 clarified the site-side gate uses the node's single `GrpcPsk` and ignores `x-scadabridge-site` (central-side routing hint) — per-site isolation proven by wrong-key reject, not the header; check 6 = a **297,574-byte** single gRPC reply (a payload Akka's 128 KB frame dropped); check 7 = Akka cluster membership strictly pair-only, 0 cross-boundary association; check 8 = full-rig restart, 0 receptionist/ClusterClient lines on any of the 8 nodes. Instance-dependent central→site RPCs (`ExecuteOpcUa`/`ExecuteRoute`/standby parked retry/`TriggerFailover`) carried forward unit-proven — bare rig has no OPC-bound instance / inbound method / parked op / CLI failover verb. ## Gotchas for the executor (will bite; read twice) diff --git a/docs/plans/2026-07-22-clusterclient-to-grpc-plan.md.tasks.json b/docs/plans/2026-07-22-clusterclient-to-grpc-plan.md.tasks.json index a23709ce..1ba8dda8 100644 --- a/docs/plans/2026-07-22-clusterclient-to-grpc-plan.md.tasks.json +++ b/docs/plans/2026-07-22-clusterclient-to-grpc-plan.md.tasks.json @@ -224,7 +224,7 @@ "P2", "P3" ], - "notes": "Branch feat/grpc-phase4-deletion (2026-07-23). Flags DELETED rather than flipped — end state is gRPC-only, identical. Deleted: AkkaCentralTransport, AkkaSiteTransport, ISiteClientFactory+DefaultSiteClientFactory, CentralCommunicationActor legacy ctor + SelectTransport, ClusterClient creation + both ClusterClientReceptionist.RegisterService in AkkaHostedService, CommunicationOptions.CentralContactPoints + CentralTransport/SiteTransport flags + CentralTransportMode/SiteTransportKind enums, RegisterCentralClient message + receive block, AkkaCentralTransportTests + DefaultSiteClientFactoryTests. Added NoOpCentralTransport (fail-loud null-default so command-dispatch TestKit suites keep constructing without a wired transport; production always injects GrpcCentralTransport). CommunicationOptionsValidator: CentralGrpcEndpoints now unconditional (no-blank-entries, role-agnostic); StartupValidator: Site-only requires ≥1 CentralGrpcEndpoint (fail-fast, mirrors GrpcPsk). Host builds GrpcSiteTransport (central) + GrpcCentralTransport (site) unconditionally. Kept Akka.Cluster.Tools (ClusterSingleton). Test rework: deleted the ClusterClient.Send per-site-routing tests (covered by CentralCommunicationActorTransportTests + GrpcSiteTransport suites), swapped ISiteClientFactory→substitute ISiteCommandTransport across 5 files, converted SiteAuditPushFlow's ClusterClientRelay→BridgeCentralTransport, converted Heartbeat_StampsIsActive to a substitute ICentralTransport, repurposed HealthReportAck no-client test to the NoOp fail-loud path, removed SiteActors_CentralClusterClient_Exists. Rig: docker ×6 + docker-env2 ×2 + Host appsettings.Site.json + deploy/wonder-app-vd03 moved CentralContactPoints→CentralGrpcEndpoints. Full solution build 0/0; Communication.Tests 640, StartupValidator 59, SiteActorPath 5, audit-push integration 1 all green. DID NOT fold in the dead IntegrationCallRequest deletion (#32) — SiteEnvelope routing is transport-agnostic so it still compiles; user-owned behavioral decision, left OUT of this PR." + "notes": "Branch feat/grpc-phase4-deletion (2026-07-23). Flags DELETED rather than flipped \u2014 end state is gRPC-only, identical. Deleted: AkkaCentralTransport, AkkaSiteTransport, ISiteClientFactory+DefaultSiteClientFactory, CentralCommunicationActor legacy ctor + SelectTransport, ClusterClient creation + both ClusterClientReceptionist.RegisterService in AkkaHostedService, CommunicationOptions.CentralContactPoints + CentralTransport/SiteTransport flags + CentralTransportMode/SiteTransportKind enums, RegisterCentralClient message + receive block, AkkaCentralTransportTests + DefaultSiteClientFactoryTests. Added NoOpCentralTransport (fail-loud null-default so command-dispatch TestKit suites keep constructing without a wired transport; production always injects GrpcCentralTransport). CommunicationOptionsValidator: CentralGrpcEndpoints now unconditional (no-blank-entries, role-agnostic); StartupValidator: Site-only requires \u22651 CentralGrpcEndpoint (fail-fast, mirrors GrpcPsk). Host builds GrpcSiteTransport (central) + GrpcCentralTransport (site) unconditionally. Kept Akka.Cluster.Tools (ClusterSingleton). Test rework: deleted the ClusterClient.Send per-site-routing tests (covered by CentralCommunicationActorTransportTests + GrpcSiteTransport suites), swapped ISiteClientFactory\u2192substitute ISiteCommandTransport across 5 files, converted SiteAuditPushFlow's ClusterClientRelay\u2192BridgeCentralTransport, converted Heartbeat_StampsIsActive to a substitute ICentralTransport, repurposed HealthReportAck no-client test to the NoOp fail-loud path, removed SiteActors_CentralClusterClient_Exists. Rig: docker \u00d76 + docker-env2 \u00d72 + Host appsettings.Site.json + deploy/wonder-app-vd03 moved CentralContactPoints\u2192CentralGrpcEndpoints. Full solution build 0/0; Communication.Tests 640, StartupValidator 59, SiteActorPath 5, audit-push integration 1 all green. DID NOT fold in the dead IntegrationCallRequest deletion (#32) \u2014 SiteEnvelope routing is transport-agnostic so it still compiles; user-owned behavioral decision, left OUT of this PR." }, { "id": "P4.2", @@ -235,17 +235,18 @@ "blockedBy": [ "P4.1" ], - "notes": "Grep-gates: CentralContactPoints → only intentional 'replaces the former' doc refs + plan trackers + stale bin/ artifacts (regenerate on build); deleted symbols (DefaultSiteClientFactory/ISiteClientFactory/RegisterCentralClient/AkkaCentralTransport/AkkaSiteTransport/CentralTransportMode/SiteTransportKind/the flags) → 0 live refs. Residual 'clusterclient' in src = ClusterClientSiteAuditClient (misleadingly named but transport-agnostic — it Asks SiteCommunicationActor, holds no ClusterClient; works unchanged) + ~25 stale inline XML-doc comments across SiteRuntime/Commons/SiteCallAudit — NOT scrubbed (out of plan scope, no behavior impact); FOLLOW-UP noted. Docs updated (subagent + me): Component-Communication.md, components/Communication.md, Component-Host.md, Component-StoreAndForward.md, deployment/topology-guide.md, plans/grpc_streams.md (SUPERSEDED note + LoadSiteAddressesFromDb correction), known-issues/2026-06-26-deploy-config-exceeds-akka-frame-size.md (2026-07-23 frame-size-retired amendment), and CLAUDE.md (2 transport-decision passages). Phase 5 (live gate) is next and requires a rig redeploy." + "notes": "Grep-gates: CentralContactPoints \u2192 only intentional 'replaces the former' doc refs + plan trackers + stale bin/ artifacts (regenerate on build); deleted symbols (DefaultSiteClientFactory/ISiteClientFactory/RegisterCentralClient/AkkaCentralTransport/AkkaSiteTransport/CentralTransportMode/SiteTransportKind/the flags) \u2192 0 live refs. Residual 'clusterclient' in src = ClusterClientSiteAuditClient (misleadingly named but transport-agnostic \u2014 it Asks SiteCommunicationActor, holds no ClusterClient; works unchanged) + ~25 stale inline XML-doc comments across SiteRuntime/Commons/SiteCallAudit \u2014 NOT scrubbed (out of plan scope, no behavior impact); FOLLOW-UP noted. Docs updated (subagent + me): Component-Communication.md, components/Communication.md, Component-Host.md, Component-StoreAndForward.md, deployment/topology-guide.md, plans/grpc_streams.md (SUPERSEDED note + LoadSiteAddressesFromDb correction), known-issues/2026-06-26-deploy-config-exceeds-akka-frame-size.md (2026-07-23 frame-size-retired amendment), and CLAUDE.md (2 transport-decision passages). Phase 5 (live gate) is next and requires a rig redeploy." }, { "id": "P5", "phase": "5", "subject": "Live gate, 8 checks, recorded in docs/plans/2026-07-22-clusterclient-to-grpc-live-gate.md", - "status": "pending", + "status": "completed", "activeForm": "Running the live gate", "blockedBy": [ "P4.2" - ] + ], + "notes": "Live gate PASS 2026-07-23 on Phase 4 deletion build (main @ 7fd5cb2b). All 8 checks PASS: PSK negatives (site-header contract clarified \u2014 site node has one key, header is central-side routing hint; per-site isolation proven by wrong-key reject); site->central matrix (notif no-loss/dupe, both audit paths node-a/b via IngestAuditEvents + central-a direct, health/heartbeat/active, reconcile self-heal over gRPC); central->site matrix ExecuteQuery/Parked/Lifecycle (OpcUa/Route/standby-parked/TriggerFailover carried forward, unit-proven); active-central kill sticky-flip 1s + central-b active 26s; mid-drain total==distinct zero loss/dupes; 297574-byte gRPC reply (frame-class retired); 0 cross-boundary Akka association (cluster membership pair-only); full-rig restart 0 clusterclient lines all 8 nodes. Recorded in docs/plans/2026-07-22-clusterclient-to-grpc-live-gate.md." } ] } \ No newline at end of file