docs(grpc): Phase 5 live gate — PASS (8/8), migration complete
Full eight-check gate on the Phase 4 deletion build (main @ 7fd5cb2b),
rig rebuilt from main. All 8 checks PASS: PSK negatives; site->central
matrix (notif no-loss/dupe, both audit paths, reconcile self-heal);
central->site matrix (Query/Parked/Lifecycle); active-central kill ->
sticky flip 1s + central-b active 26s; mid-drain total==distinct;
297,574-byte gRPC reply (128KB frame-class retired); cluster membership
pair-only (no cross-boundary Akka association); full-rig restart 0
receptionist/ClusterClient lines on any of 8 nodes.
Check-1 clarification recorded: site-side interceptor uses the node's
single GrpcPsk and ignores x-scadabridge-site (a central-side routing
hint) -- per-site isolation proven by wrong-key reject. Instance-
dependent central->site RPCs (OpcUa/Route/standby-parked/TriggerFailover)
carried forward, unit-proven.
This commit is contained in:
Joseph Doherty
@@ -335,3 +335,145 @@ call recovered immediately via site-a-b.
|
||||
|
||||
Rig left running with **both** transports on gRPC; git config reverted to Akka default (a redeploy
|
||||
from `main` resets to all-Akka).
|
||||
|
||||
---
|
||||
|
||||
## Phase 5 — full eight-check gate on the deletion build — **PASS** (2026-07-23)
|
||||
|
||||
The migration's terminal gate, run on `main` **after Phase 4** (`7fd5cb2b`) — the build where the
|
||||
Akka `ClusterClient`/`ClusterClientReceptionist` transport is **physically deleted**, the
|
||||
`CentralTransport`/`SiteTransport` flags are gone, and gRPC is the *only* site↔central transport
|
||||
(no flags to flip). Rig rebuilt from `main` via `bash docker/deploy.sh`, all 9 containers recreated
|
||||
from the new image; central MS SQL and the S&F driver (`SoakNotify` template #2147 / instances
|
||||
#95–97 / list "Engineering Alerts" #28) persisted from prior phases.
|
||||
|
||||
**Provenance guard:** the pre-existing rig image was built `11:17`, *before* the Phase 4 commit
|
||||
(`12:54`); it still carried ClusterClient code and would have invalidated checks 7–8. Confirmed
|
||||
by rebuild timestamp, then by **0** ClusterClient/receptionist log lines across all 8 nodes on the
|
||||
fresh boot. (The lone `ClusterClientReceptionist` symbol still present is inside
|
||||
`Akka.Cluster.Tools.dll` — the library we deliberately keep for `ClusterSingleton` — not our code.)
|
||||
|
||||
Header names for the negative probes: `authorization: Bearer <psk>` + `x-scadabridge-site: <siteId>`
|
||||
(`ISitePskProvider.cs`). Docker rig PSKs: `dev-grpc-psk-docker-site-{a,b,c}`
|
||||
(`ScadaBridge__Communication__SitePsks__site-*`, central override; sites read their own
|
||||
`GrpcPsk` from mounted `appsettings.Site.json`).
|
||||
|
||||
### Checks
|
||||
|
||||
| # | Check | Result |
|
||||
|---|---|---|
|
||||
| 1 | **PSK negatives** | **PASS (with a contract clarification)** — see below |
|
||||
| 2 | **Site→central matrix** | **PASS** — notifications, both audit paths, health, heartbeat, reconcile all live over gRPC |
|
||||
| 3 | **Central→site matrix** | **PASS (proportionate)** — `ExecuteQuery`/`ExecuteParked`/`ExecuteLifecycle` live; instance-dependent RPCs carried forward (see below) |
|
||||
| 4 | **Failover / failback** | **PASS** — active-central kill → sticky flip in 1 s, central-b active in 26 s; failback kept central-b active (oldest-Up, no flap) |
|
||||
| 5 | **Mid-drain kill, zero loss/dupes** | **PASS** — buffered notifications flushed; **total == distinct** across the outage |
|
||||
| 6 | **Frame-class retirement (>128 KB)** | **PASS** — a single **297,574-byte** gRPC reply succeeded (Akka's 128 KB frame would have dropped it) |
|
||||
| 7 | **No cross-boundary Akka association** | **PASS** — each Akka cluster's membership is strictly its own pair; **0** cross-boundary association lines both directions |
|
||||
| 8 | **Full-rig restart discipline** | **PASS** — all 9 restarted together came up clean; **0** receptionist/ClusterClient lines on any node; sites reconnected over gRPC; S&F resumed with no dupes |
|
||||
|
||||
### Check 1 — PSK negatives (and the contract clarification)
|
||||
|
||||
Against the two gated services on site-a (`:9023`), using the local proto descriptors
|
||||
(`-proto sitestream.proto` / `site_command.proto`; server reflection is off in this build):
|
||||
|
||||
```
|
||||
SiteStreamService/PullAuditEvents:
|
||||
no credentials -> PermissionDenied "Control plane authentication failed."
|
||||
wrong key (site-b's key) -> PermissionDenied
|
||||
correct key -> success (audit events returned)
|
||||
SiteCommandService/ExecuteQuery:
|
||||
no credentials -> PermissionDenied
|
||||
wrong key (site-b's key) -> PermissionDenied
|
||||
```
|
||||
|
||||
site-a-a's log recorded **exactly 4** control-plane rejections — the four deliberate negative
|
||||
probes above, nothing else.
|
||||
|
||||
**Clarification the gate produced:** the plan listed "missing `x-scadabridge-site` ⇒
|
||||
`PermissionDenied`" as a negative case. It does **not** hold on the *site* side, and that is
|
||||
correct by design. `ControlPlaneAuthInterceptor.Authorize` compares the presented bearer against
|
||||
the node's **own single** `GrpcPsk` (`_options.Value.GrpcPsk`) and never reads
|
||||
`x-scadabridge-site`. That header is a **central-side routing hint** — `SitePskProvider` uses it to
|
||||
pick *which* site's key to expect, because central holds many. A single-key site node needs no such
|
||||
hint. Per-site key isolation — the actual security property — is proven by the **wrong-key**
|
||||
rejection (site-b's key refused at site-a), not by the header. Recorded rather than "fixed": adding
|
||||
a header requirement to the site interceptor would be theatre.
|
||||
|
||||
**LocalDb sync unaffected:** the replicated node (site-a) logged one boot-order
|
||||
`faulted; reconnecting` (peer node-b not yet up), the passive peer accepted the
|
||||
`/localdb_sync.v1.LocalDbSync/Sync` POST 1 s later, and the health report settled to
|
||||
`localDbReplicationConnected:true, localDbOplogBacklog:0`. **0** LocalDb sync auth failures — the
|
||||
control-plane interceptor and the separate `LocalDbSyncAuthInterceptor` don't interfere. site-b/c
|
||||
report `localDbReplicationConnected:false` — the intended rig posture (only site-a replicated).
|
||||
|
||||
### Check 2 — site→central matrix
|
||||
|
||||
- **Notifications e2e:** `dbo.Notifications` climbed continuously at the driver's ~3 rows / 5 s,
|
||||
**total == distinct** at every sample (insert-if-not-exists no-loss/no-dupe truth).
|
||||
- **Both audit paths live over gRPC:** in a 3-minute window, `dbo.AuditLog` carried `node-a`/`node-b`
|
||||
rows (105 each — **site-originated**, forwarded over the gRPC `IngestAuditEvents` RPC) *and*
|
||||
`central-a` rows (420 — central-direct-write from the outbox dispatcher). Row count climbing live.
|
||||
- **Health live per site:** `health summary` shows all three sites `isOnline:true`, sequence numbers
|
||||
advancing, fresh heartbeats; site-a `enabledInstanceCount:3` with a live `Notification` S&F buffer.
|
||||
- **Heartbeat → active flag:** central-a `active=200`, central-b `active=503`; each site's report
|
||||
drives its online flag.
|
||||
- **Reconcile self-heal:** restarting `site-a-a` produced a fresh (17:11:24) `Site→central transport:
|
||||
gRPC to 2 central endpoint(s)` then `Reconcile pass … complete: 0 fetched, 0 failed, 0 orphan(s)`;
|
||||
S&F never dropped across the restart (active node kept emitting), no dupes.
|
||||
- **Not driven (carry-forward, bare rig):** cached-call telemetry (`SoakNotify` only calls
|
||||
`Notify.Send`, no `CachedCall`/`CachedWrite`); a notification reaching **Delivered** rather than
|
||||
`Parked` (no SMTP on the rig — the transport truth is the row-count, per Phase 2).
|
||||
|
||||
### Check 3 — central→site matrix
|
||||
|
||||
`ExecuteQuery` (event-log, all 3 sites → correlationIds + entries), `ExecuteParked` (parked-messages,
|
||||
all 3 sites → empty, no parked ops on a bare rig), and `ExecuteLifecycle` (disable→enable #95 →
|
||||
`success:true` both) all rode `SiteCommandService` gRPC; site active nodes logged them
|
||||
(site-a-a: 8 `ExecuteLifecycle` + 4 `ExecuteParked` + 4 `ExecuteQuery`; site-b-a/c-a:
|
||||
`ExecuteParked` + `ExecuteQuery`). **Carried forward, unchanged from Phase 3** (needs a deployed
|
||||
OPC-bound instance / inbound method / a parked op / the destructive UI-only failover verb the bare
|
||||
rig lacks): `ExecuteOpcUa`, `ExecuteRoute`, standby parked retry/discard, `TriggerFailover`
|
||||
(the hard-kill in Check 4 is its live equivalent). All four are unit-proven (dispatcher routing ×28,
|
||||
ack-before-`Leave` ordering).
|
||||
|
||||
### Checks 4 & 5 — failover / failback / mid-drain kill
|
||||
|
||||
Hard-killed the **active** central (central-a) mid-drain: site-a-b logged the sticky endpoint flip
|
||||
`central-a:8083 → central-b:8083` **1 s** after the kill; central-b reached active in **26 s**
|
||||
(auto-down). The notification count froze during the gap, then **flushed the buffered rows and
|
||||
resumed** at the steady 3 / 5 s. Post-drain **total == distinct (4395 == 4395)** — zero loss, zero
|
||||
duplicates. Failback: restarting central-a, it rejoined and central-b **retained** active
|
||||
(`a=503` standby / `b=200` active) — oldest-Up, no role flap.
|
||||
|
||||
### Check 6 — frame-class retirement
|
||||
|
||||
`PullAuditEvents{since_utc: 2020-01-01, batch_size: 5000}` against the active site node returned a
|
||||
single **297,574-byte** reply successfully. Over Akka's default 128 KB frame
|
||||
(`log-frame-size-exceeding` off) this class of payload was silently dropped and the caller's Ask
|
||||
timed out (`docs/known-issues/2026-06-26-…`). On gRPC (4 MB default cap) it is a normal reply.
|
||||
|
||||
### Check 7 — no cross-boundary Akka association
|
||||
|
||||
Each Akka cluster's membership is strictly its own pair: central `[central-a, central-b]`, site-a
|
||||
`[site-a-a, site-a-b]`, site-c `[site-c-a, site-c-b]` — **no site node ever appears in central's
|
||||
membership and vice versa**. Central's log holds **0** `Association with remote system …site-*`
|
||||
lines and **0** references to any site address; site-a's log holds **0** references to any central
|
||||
address. The two boundaries are wired only by per-pair Akka remoting (8082, internal) and the gRPC
|
||||
control/data planes (8083); there is no Akka association across the site↔central line.
|
||||
|
||||
### Check 8 — full-rig restart discipline
|
||||
|
||||
Restarted all 9 rig containers together (pairs together, honouring the LocalDb-replication constraint).
|
||||
Central re-formed with central-a active; **0** receptionist/ClusterClient lines across all 8 nodes;
|
||||
all three sites logged `Site→central transport: gRPC to 2 central endpoint(s)`; S&F resumed
|
||||
(4407 → 4483, **distinct == total**, no dupes).
|
||||
|
||||
### End state
|
||||
|
||||
The migration is complete and proven on the deletion build. The rig runs the committed all-gRPC
|
||||
default (Phase 4 flipped the shipped appsettings to `CentralGrpcEndpoints` — unlike Phases 2/3, a
|
||||
redeploy from `main` now yields all-gRPC, not all-Akka). No ClusterClient remains anywhere in the
|
||||
running system or the source tree. Deferred, unchanged from earlier phases: the instance-dependent
|
||||
central→site RPC matrix (`ExecuteOpcUa`/`ExecuteRoute`/standby parked retry) and live
|
||||
`TriggerFailover`, all unit-proven; PSK **rotation** on a live pair; `docker-env2` was updated with
|
||||
its own key but not gated here.
|
||||
|
||||
Reference in New Issue
Block a user