docs(grpc): Phase 5 live gate — PASS (8/8), migration complete
Full eight-check gate on the Phase 4 deletion build (main @ 7fd5cb2b),
rig rebuilt from main. All 8 checks PASS: PSK negatives; site->central
matrix (notif no-loss/dupe, both audit paths, reconcile self-heal);
central->site matrix (Query/Parked/Lifecycle); active-central kill ->
sticky flip 1s + central-b active 26s; mid-drain total==distinct;
297,574-byte gRPC reply (128KB frame-class retired); cluster membership
pair-only (no cross-boundary Akka association); full-rig restart 0
receptionist/ClusterClient lines on any of 8 nodes.
Check-1 clarification recorded: site-side interceptor uses the node's
single GrpcPsk and ignores x-scadabridge-site (a central-side routing
hint) -- per-site isolation proven by wrong-key reject. Instance-
dependent central->site RPCs (OpcUa/Route/standby-parked/TriggerFailover)
carried forward, unit-proven.
This commit is contained in:
Joseph Doherty
@@ -335,3 +335,145 @@ call recovered immediately via site-a-b.
|
||||
|
||||
Rig left running with **both** transports on gRPC; git config reverted to Akka default (a redeploy
|
||||
from `main` resets to all-Akka).
|
||||
|
||||
---
|
||||
|
||||
## Phase 5 — full eight-check gate on the deletion build — **PASS** (2026-07-23)
|
||||
|
||||
The migration's terminal gate, run on `main` **after Phase 4** (`7fd5cb2b`) — the build where the
|
||||
Akka `ClusterClient`/`ClusterClientReceptionist` transport is **physically deleted**, the
|
||||
`CentralTransport`/`SiteTransport` flags are gone, and gRPC is the *only* site↔central transport
|
||||
(no flags to flip). Rig rebuilt from `main` via `bash docker/deploy.sh`, all 9 containers recreated
|
||||
from the new image; central MS SQL and the S&F driver (`SoakNotify` template #2147 / instances
|
||||
#95–97 / list "Engineering Alerts" #28) persisted from prior phases.
|
||||
|
||||
**Provenance guard:** the pre-existing rig image was built `11:17`, *before* the Phase 4 commit
|
||||
(`12:54`); it still carried ClusterClient code and would have invalidated checks 7–8. Confirmed
|
||||
by rebuild timestamp, then by **0** ClusterClient/receptionist log lines across all 8 nodes on the
|
||||
fresh boot. (The lone `ClusterClientReceptionist` symbol still present is inside
|
||||
`Akka.Cluster.Tools.dll` — the library we deliberately keep for `ClusterSingleton` — not our code.)
|
||||
|
||||
Header names for the negative probes: `authorization: Bearer <psk>` + `x-scadabridge-site: <siteId>`
|
||||
(`ISitePskProvider.cs`). Docker rig PSKs: `dev-grpc-psk-docker-site-{a,b,c}`
|
||||
(`ScadaBridge__Communication__SitePsks__site-*`, central override; sites read their own
|
||||
`GrpcPsk` from mounted `appsettings.Site.json`).
|
||||
|
||||
### Checks
|
||||
|
||||
| # | Check | Result |
|
||||
|---|---|---|
|
||||
| 1 | **PSK negatives** | **PASS (with a contract clarification)** — see below |
|
||||
| 2 | **Site→central matrix** | **PASS** — notifications, both audit paths, health, heartbeat, reconcile all live over gRPC |
|
||||
| 3 | **Central→site matrix** | **PASS (proportionate)** — `ExecuteQuery`/`ExecuteParked`/`ExecuteLifecycle` live; instance-dependent RPCs carried forward (see below) |
|
||||
| 4 | **Failover / failback** | **PASS** — active-central kill → sticky flip in 1 s, central-b active in 26 s; failback kept central-b active (oldest-Up, no flap) |
|
||||
| 5 | **Mid-drain kill, zero loss/dupes** | **PASS** — buffered notifications flushed; **total == distinct** across the outage |
|
||||
| 6 | **Frame-class retirement (>128 KB)** | **PASS** — a single **297,574-byte** gRPC reply succeeded (Akka's 128 KB frame would have dropped it) |
|
||||
| 7 | **No cross-boundary Akka association** | **PASS** — each Akka cluster's membership is strictly its own pair; **0** cross-boundary association lines both directions |
|
||||
| 8 | **Full-rig restart discipline** | **PASS** — all 9 restarted together came up clean; **0** receptionist/ClusterClient lines on any node; sites reconnected over gRPC; S&F resumed with no dupes |
|
||||
|
||||
### Check 1 — PSK negatives (and the contract clarification)
|
||||
|
||||
Against the two gated services on site-a (`:9023`), using the local proto descriptors
|
||||
(`-proto sitestream.proto` / `site_command.proto`; server reflection is off in this build):
|
||||
|
||||
```
|
||||
SiteStreamService/PullAuditEvents:
|
||||
no credentials -> PermissionDenied "Control plane authentication failed."
|
||||
wrong key (site-b's key) -> PermissionDenied
|
||||
correct key -> success (audit events returned)
|
||||
SiteCommandService/ExecuteQuery:
|
||||
no credentials -> PermissionDenied
|
||||
wrong key (site-b's key) -> PermissionDenied
|
||||
```
|
||||
|
||||
site-a-a's log recorded **exactly 4** control-plane rejections — the four deliberate negative
|
||||
probes above, nothing else.
|
||||
|
||||
**Clarification the gate produced:** the plan listed "missing `x-scadabridge-site` ⇒
|
||||
`PermissionDenied`" as a negative case. It does **not** hold on the *site* side, and that is
|
||||
correct by design. `ControlPlaneAuthInterceptor.Authorize` compares the presented bearer against
|
||||
the node's **own single** `GrpcPsk` (`_options.Value.GrpcPsk`) and never reads
|
||||
`x-scadabridge-site`. That header is a **central-side routing hint** — `SitePskProvider` uses it to
|
||||
pick *which* site's key to expect, because central holds many. A single-key site node needs no such
|
||||
hint. Per-site key isolation — the actual security property — is proven by the **wrong-key**
|
||||
rejection (site-b's key refused at site-a), not by the header. Recorded rather than "fixed": adding
|
||||
a header requirement to the site interceptor would be theatre.
|
||||
|
||||
**LocalDb sync unaffected:** the replicated node (site-a) logged one boot-order
|
||||
`faulted; reconnecting` (peer node-b not yet up), the passive peer accepted the
|
||||
`/localdb_sync.v1.LocalDbSync/Sync` POST 1 s later, and the health report settled to
|
||||
`localDbReplicationConnected:true, localDbOplogBacklog:0`. **0** LocalDb sync auth failures — the
|
||||
control-plane interceptor and the separate `LocalDbSyncAuthInterceptor` don't interfere. site-b/c
|
||||
report `localDbReplicationConnected:false` — the intended rig posture (only site-a replicated).
|
||||
|
||||
### Check 2 — site→central matrix
|
||||
|
||||
- **Notifications e2e:** `dbo.Notifications` climbed continuously at the driver's ~3 rows / 5 s,
|
||||
**total == distinct** at every sample (insert-if-not-exists no-loss/no-dupe truth).
|
||||
- **Both audit paths live over gRPC:** in a 3-minute window, `dbo.AuditLog` carried `node-a`/`node-b`
|
||||
rows (105 each — **site-originated**, forwarded over the gRPC `IngestAuditEvents` RPC) *and*
|
||||
`central-a` rows (420 — central-direct-write from the outbox dispatcher). Row count climbing live.
|
||||
- **Health live per site:** `health summary` shows all three sites `isOnline:true`, sequence numbers
|
||||
advancing, fresh heartbeats; site-a `enabledInstanceCount:3` with a live `Notification` S&F buffer.
|
||||
- **Heartbeat → active flag:** central-a `active=200`, central-b `active=503`; each site's report
|
||||
drives its online flag.
|
||||
- **Reconcile self-heal:** restarting `site-a-a` produced a fresh (17:11:24) `Site→central transport:
|
||||
gRPC to 2 central endpoint(s)` then `Reconcile pass … complete: 0 fetched, 0 failed, 0 orphan(s)`;
|
||||
S&F never dropped across the restart (active node kept emitting), no dupes.
|
||||
- **Not driven (carry-forward, bare rig):** cached-call telemetry (`SoakNotify` only calls
|
||||
`Notify.Send`, no `CachedCall`/`CachedWrite`); a notification reaching **Delivered** rather than
|
||||
`Parked` (no SMTP on the rig — the transport truth is the row-count, per Phase 2).
|
||||
|
||||
### Check 3 — central→site matrix
|
||||
|
||||
`ExecuteQuery` (event-log, all 3 sites → correlationIds + entries), `ExecuteParked` (parked-messages,
|
||||
all 3 sites → empty, no parked ops on a bare rig), and `ExecuteLifecycle` (disable→enable #95 →
|
||||
`success:true` both) all rode `SiteCommandService` gRPC; site active nodes logged them
|
||||
(site-a-a: 8 `ExecuteLifecycle` + 4 `ExecuteParked` + 4 `ExecuteQuery`; site-b-a/c-a:
|
||||
`ExecuteParked` + `ExecuteQuery`). **Carried forward, unchanged from Phase 3** (needs a deployed
|
||||
OPC-bound instance / inbound method / a parked op / the destructive UI-only failover verb the bare
|
||||
rig lacks): `ExecuteOpcUa`, `ExecuteRoute`, standby parked retry/discard, `TriggerFailover`
|
||||
(the hard-kill in Check 4 is its live equivalent). All four are unit-proven (dispatcher routing ×28,
|
||||
ack-before-`Leave` ordering).
|
||||
|
||||
### Checks 4 & 5 — failover / failback / mid-drain kill
|
||||
|
||||
Hard-killed the **active** central (central-a) mid-drain: site-a-b logged the sticky endpoint flip
|
||||
`central-a:8083 → central-b:8083` **1 s** after the kill; central-b reached active in **26 s**
|
||||
(auto-down). The notification count froze during the gap, then **flushed the buffered rows and
|
||||
resumed** at the steady 3 / 5 s. Post-drain **total == distinct (4395 == 4395)** — zero loss, zero
|
||||
duplicates. Failback: restarting central-a, it rejoined and central-b **retained** active
|
||||
(`a=503` standby / `b=200` active) — oldest-Up, no role flap.
|
||||
|
||||
### Check 6 — frame-class retirement
|
||||
|
||||
`PullAuditEvents{since_utc: 2020-01-01, batch_size: 5000}` against the active site node returned a
|
||||
single **297,574-byte** reply successfully. Over Akka's default 128 KB frame
|
||||
(`log-frame-size-exceeding` off) this class of payload was silently dropped and the caller's Ask
|
||||
timed out (`docs/known-issues/2026-06-26-…`). On gRPC (4 MB default cap) it is a normal reply.
|
||||
|
||||
### Check 7 — no cross-boundary Akka association
|
||||
|
||||
Each Akka cluster's membership is strictly its own pair: central `[central-a, central-b]`, site-a
|
||||
`[site-a-a, site-a-b]`, site-c `[site-c-a, site-c-b]` — **no site node ever appears in central's
|
||||
membership and vice versa**. Central's log holds **0** `Association with remote system …site-*`
|
||||
lines and **0** references to any site address; site-a's log holds **0** references to any central
|
||||
address. The two boundaries are wired only by per-pair Akka remoting (8082, internal) and the gRPC
|
||||
control/data planes (8083); there is no Akka association across the site↔central line.
|
||||
|
||||
### Check 8 — full-rig restart discipline
|
||||
|
||||
Restarted all 9 rig containers together (pairs together, honouring the LocalDb-replication constraint).
|
||||
Central re-formed with central-a active; **0** receptionist/ClusterClient lines across all 8 nodes;
|
||||
all three sites logged `Site→central transport: gRPC to 2 central endpoint(s)`; S&F resumed
|
||||
(4407 → 4483, **distinct == total**, no dupes).
|
||||
|
||||
### End state
|
||||
|
||||
The migration is complete and proven on the deletion build. The rig runs the committed all-gRPC
|
||||
default (Phase 4 flipped the shipped appsettings to `CentralGrpcEndpoints` — unlike Phases 2/3, a
|
||||
redeploy from `main` now yields all-gRPC, not all-Akka). No ClusterClient remains anywhere in the
|
||||
running system or the source tree. Deferred, unchanged from earlier phases: the instance-dependent
|
||||
central→site RPC matrix (`ExecuteOpcUa`/`ExecuteRoute`/standby parked retry) and live
|
||||
`TriggerFailover`, all unit-proven; PSK **rotation** on a live pair; `docker-env2` was updated with
|
||||
its own key but not gated here.
|
||||
|
||||
@@ -331,8 +331,9 @@ Critical path ≈ 1B: **~4–6 weeks total**, matching the design estimate.
|
||||
- [x] Grep-gates pass (`CentralContactPoints` → only "replaces the former" doc refs + plan trackers; deleted symbols → 0 live refs; remaining `clusterclient` in src = the misleadingly-named `ClusterClientSiteAuditClient` [transport-agnostic, works unchanged] + stale inline doc-comments, noted as follow-up)
|
||||
- [x] Docs updated: `Component-Communication.md`, `components/Communication.md`, `Component-Host.md`, `Component-StoreAndForward.md`, `topology-guide.md`, `grpc_streams.md`, known-issues frame-size amendment, **CLAUDE.md** transport decisions
|
||||
|
||||
**Phase 5 — live gate** (record in `2026-07-22-clusterclient-to-grpc-live-gate.md`)
|
||||
- [ ] 1 PSK negatives · [ ] 2 site→central matrix · [ ] 3 central→site matrix · [ ] 4 failover/failback both directions · [ ] 5 mid-drain kill · [ ] 6 >128 KB frame-class proof · [ ] 7 no cross-boundary Akka association · [ ] 8 full-rig restart clean
|
||||
**Phase 5 — live gate** (record in `2026-07-22-clusterclient-to-grpc-live-gate.md`) — **PASS 2026-07-23** (deletion build, `main` @ `7fd5cb2b`)
|
||||
- [x] 1 PSK negatives · [x] 2 site→central matrix · [x] 3 central→site matrix · [x] 4 failover/failback both directions · [x] 5 mid-drain kill · [x] 6 >128 KB frame-class proof · [x] 7 no cross-boundary Akka association · [x] 8 full-rig restart clean
|
||||
- All 8 PASS. Notables: check 1 clarified the site-side gate uses the node's single `GrpcPsk` and ignores `x-scadabridge-site` (central-side routing hint) — per-site isolation proven by wrong-key reject, not the header; check 6 = a **297,574-byte** single gRPC reply (a payload Akka's 128 KB frame dropped); check 7 = Akka cluster membership strictly pair-only, 0 cross-boundary association; check 8 = full-rig restart, 0 receptionist/ClusterClient lines on any of the 8 nodes. Instance-dependent central→site RPCs (`ExecuteOpcUa`/`ExecuteRoute`/standby parked retry/`TriggerFailover`) carried forward unit-proven — bare rig has no OPC-bound instance / inbound method / parked op / CLI failover verb.
|
||||
|
||||
## Gotchas for the executor (will bite; read twice)
|
||||
|
||||
|
||||
@@ -224,7 +224,7 @@
|
||||
"P2",
|
||||
"P3"
|
||||
],
|
||||
"notes": "Branch feat/grpc-phase4-deletion (2026-07-23). Flags DELETED rather than flipped — end state is gRPC-only, identical. Deleted: AkkaCentralTransport, AkkaSiteTransport, ISiteClientFactory+DefaultSiteClientFactory, CentralCommunicationActor legacy ctor + SelectTransport, ClusterClient creation + both ClusterClientReceptionist.RegisterService in AkkaHostedService, CommunicationOptions.CentralContactPoints + CentralTransport/SiteTransport flags + CentralTransportMode/SiteTransportKind enums, RegisterCentralClient message + receive block, AkkaCentralTransportTests + DefaultSiteClientFactoryTests. Added NoOpCentralTransport (fail-loud null-default so command-dispatch TestKit suites keep constructing without a wired transport; production always injects GrpcCentralTransport). CommunicationOptionsValidator: CentralGrpcEndpoints now unconditional (no-blank-entries, role-agnostic); StartupValidator: Site-only requires ≥1 CentralGrpcEndpoint (fail-fast, mirrors GrpcPsk). Host builds GrpcSiteTransport (central) + GrpcCentralTransport (site) unconditionally. Kept Akka.Cluster.Tools (ClusterSingleton). Test rework: deleted the ClusterClient.Send per-site-routing tests (covered by CentralCommunicationActorTransportTests + GrpcSiteTransport suites), swapped ISiteClientFactory→substitute ISiteCommandTransport across 5 files, converted SiteAuditPushFlow's ClusterClientRelay→BridgeCentralTransport, converted Heartbeat_StampsIsActive to a substitute ICentralTransport, repurposed HealthReportAck no-client test to the NoOp fail-loud path, removed SiteActors_CentralClusterClient_Exists. Rig: docker ×6 + docker-env2 ×2 + Host appsettings.Site.json + deploy/wonder-app-vd03 moved CentralContactPoints→CentralGrpcEndpoints. Full solution build 0/0; Communication.Tests 640, StartupValidator 59, SiteActorPath 5, audit-push integration 1 all green. DID NOT fold in the dead IntegrationCallRequest deletion (#32) — SiteEnvelope routing is transport-agnostic so it still compiles; user-owned behavioral decision, left OUT of this PR."
|
||||
"notes": "Branch feat/grpc-phase4-deletion (2026-07-23). Flags DELETED rather than flipped \u2014 end state is gRPC-only, identical. Deleted: AkkaCentralTransport, AkkaSiteTransport, ISiteClientFactory+DefaultSiteClientFactory, CentralCommunicationActor legacy ctor + SelectTransport, ClusterClient creation + both ClusterClientReceptionist.RegisterService in AkkaHostedService, CommunicationOptions.CentralContactPoints + CentralTransport/SiteTransport flags + CentralTransportMode/SiteTransportKind enums, RegisterCentralClient message + receive block, AkkaCentralTransportTests + DefaultSiteClientFactoryTests. Added NoOpCentralTransport (fail-loud null-default so command-dispatch TestKit suites keep constructing without a wired transport; production always injects GrpcCentralTransport). CommunicationOptionsValidator: CentralGrpcEndpoints now unconditional (no-blank-entries, role-agnostic); StartupValidator: Site-only requires \u22651 CentralGrpcEndpoint (fail-fast, mirrors GrpcPsk). Host builds GrpcSiteTransport (central) + GrpcCentralTransport (site) unconditionally. Kept Akka.Cluster.Tools (ClusterSingleton). Test rework: deleted the ClusterClient.Send per-site-routing tests (covered by CentralCommunicationActorTransportTests + GrpcSiteTransport suites), swapped ISiteClientFactory\u2192substitute ISiteCommandTransport across 5 files, converted SiteAuditPushFlow's ClusterClientRelay\u2192BridgeCentralTransport, converted Heartbeat_StampsIsActive to a substitute ICentralTransport, repurposed HealthReportAck no-client test to the NoOp fail-loud path, removed SiteActors_CentralClusterClient_Exists. Rig: docker \u00d76 + docker-env2 \u00d72 + Host appsettings.Site.json + deploy/wonder-app-vd03 moved CentralContactPoints\u2192CentralGrpcEndpoints. Full solution build 0/0; Communication.Tests 640, StartupValidator 59, SiteActorPath 5, audit-push integration 1 all green. DID NOT fold in the dead IntegrationCallRequest deletion (#32) \u2014 SiteEnvelope routing is transport-agnostic so it still compiles; user-owned behavioral decision, left OUT of this PR."
|
||||
},
|
||||
{
|
||||
"id": "P4.2",
|
||||
@@ -235,17 +235,18 @@
|
||||
"blockedBy": [
|
||||
"P4.1"
|
||||
],
|
||||
"notes": "Grep-gates: CentralContactPoints → only intentional 'replaces the former' doc refs + plan trackers + stale bin/ artifacts (regenerate on build); deleted symbols (DefaultSiteClientFactory/ISiteClientFactory/RegisterCentralClient/AkkaCentralTransport/AkkaSiteTransport/CentralTransportMode/SiteTransportKind/the flags) → 0 live refs. Residual 'clusterclient' in src = ClusterClientSiteAuditClient (misleadingly named but transport-agnostic — it Asks SiteCommunicationActor, holds no ClusterClient; works unchanged) + ~25 stale inline XML-doc comments across SiteRuntime/Commons/SiteCallAudit — NOT scrubbed (out of plan scope, no behavior impact); FOLLOW-UP noted. Docs updated (subagent + me): Component-Communication.md, components/Communication.md, Component-Host.md, Component-StoreAndForward.md, deployment/topology-guide.md, plans/grpc_streams.md (SUPERSEDED note + LoadSiteAddressesFromDb correction), known-issues/2026-06-26-deploy-config-exceeds-akka-frame-size.md (2026-07-23 frame-size-retired amendment), and CLAUDE.md (2 transport-decision passages). Phase 5 (live gate) is next and requires a rig redeploy."
|
||||
"notes": "Grep-gates: CentralContactPoints \u2192 only intentional 'replaces the former' doc refs + plan trackers + stale bin/ artifacts (regenerate on build); deleted symbols (DefaultSiteClientFactory/ISiteClientFactory/RegisterCentralClient/AkkaCentralTransport/AkkaSiteTransport/CentralTransportMode/SiteTransportKind/the flags) \u2192 0 live refs. Residual 'clusterclient' in src = ClusterClientSiteAuditClient (misleadingly named but transport-agnostic \u2014 it Asks SiteCommunicationActor, holds no ClusterClient; works unchanged) + ~25 stale inline XML-doc comments across SiteRuntime/Commons/SiteCallAudit \u2014 NOT scrubbed (out of plan scope, no behavior impact); FOLLOW-UP noted. Docs updated (subagent + me): Component-Communication.md, components/Communication.md, Component-Host.md, Component-StoreAndForward.md, deployment/topology-guide.md, plans/grpc_streams.md (SUPERSEDED note + LoadSiteAddressesFromDb correction), known-issues/2026-06-26-deploy-config-exceeds-akka-frame-size.md (2026-07-23 frame-size-retired amendment), and CLAUDE.md (2 transport-decision passages). Phase 5 (live gate) is next and requires a rig redeploy."
|
||||
},
|
||||
{
|
||||
"id": "P5",
|
||||
"phase": "5",
|
||||
"subject": "Live gate, 8 checks, recorded in docs/plans/2026-07-22-clusterclient-to-grpc-live-gate.md",
|
||||
"status": "pending",
|
||||
"status": "completed",
|
||||
"activeForm": "Running the live gate",
|
||||
"blockedBy": [
|
||||
"P4.2"
|
||||
]
|
||||
],
|
||||
"notes": "Live gate PASS 2026-07-23 on Phase 4 deletion build (main @ 7fd5cb2b). All 8 checks PASS: PSK negatives (site-header contract clarified \u2014 site node has one key, header is central-side routing hint; per-site isolation proven by wrong-key reject); site->central matrix (notif no-loss/dupe, both audit paths node-a/b via IngestAuditEvents + central-a direct, health/heartbeat/active, reconcile self-heal over gRPC); central->site matrix ExecuteQuery/Parked/Lifecycle (OpcUa/Route/standby-parked/TriggerFailover carried forward, unit-proven); active-central kill sticky-flip 1s + central-b active 26s; mid-drain total==distinct zero loss/dupes; 297574-byte gRPC reply (frame-class retired); 0 cross-boundary Akka association (cluster membership pair-only); full-rig restart 0 clusterclient lines all 8 nodes. Recorded in docs/plans/2026-07-22-clusterclient-to-grpc-live-gate.md."
|
||||
}
|
||||
]
|
||||
}
|
||||
Reference in New Issue
Block a user