Joseph Doherty
8.8 KiB
ClusterClient → gRPC migration — live gate record
Rig: docker-dev/ (central-1/-2 = admin,driver,cluster-MAIN; site-a-1/-2, site-b-1/-2;
sql + traefik + migrator + cluster-seed), rebuilt from the branch under test via
docker compose -f docker-dev/docker-compose.yml up -d --build. Recorded check-by-check in
the family's live-gate format. Later phases' sections are appended as those phases land; every
check row gets a PASS/FAIL verdict plus one quantitative justification clause, with
fenced evidence (real commands, real output). A run invalidated by a concurrent rig rebuild is
void — note it and re-run (the ScadaBridge void-run rule).
Provenance discipline: every phase section opens with branch + commit SHA + "image rebuilt, all N containers recreated", and Phase 4 additionally verifies the image build timestamp post-dates the deletion merge (a stale image invalidates the boundary checks).
Plan: 2026-08-06-clusterclient-to-grpc.md. Tracking:
2026-08-06-clusterclient-to-grpc-tracking.md.
Baseline (run on the PRE-change build, before the Phase 2 flip)
The two gaps this program closes, reproduced rather than assumed:
(a) Unauthenticated Akka remoting crosses the WAN boundary. From the host:
# A live TCP association from a site container to central's Akka port:
docker exec otopcua-dev-site-a-1-1 bash -c 'exec 3<>/dev/tcp/central-1/4053 && echo OPEN'
# And the receptionist actually accepting cross-mesh traffic (central log after any AdminUI
# driver restart targeting site-a):
docker logs otopcua-dev-central-1-1 2>&1 | grep -i "clusterclient\|receptionist" | tail -5
Expected: OPEN; receptionist/client lines present. Nothing on this path presents any
credential.
(b) Silent drop on an unreachable cluster. Stop the site-b pair, dispatch a deployment
(POST http://localhost:9200/api/deployments, X-Api-Key: docker-dev-deploy-key), observe:
no per-node dispatch error surfaces anywhere; site-b's NodeDeploymentState rows sit
Applying until the 2-minute deadline marks the deployment TimedOut/partially failed. The
deadline is the only witness. Restart the pair.
- Baseline (a) recorded with evidence — PENDING
- Baseline (b) recorded with evidence — PENDING
Phase 0 — contract + auth + config groundwork — PENDING
Branch feat/grpc-phase0-contract-auth @ ________. Image rebuilt, all containers recreated.
| # | Check | Result |
|---|---|---|
| 1 | All six nodes boot on ClusterClient with the dark Grpc config present (PSKs, endpoints, ClusterPsks map); no validator fires |
PENDING |
| 2 | Deliberate misconfig: a scratch container with MeshTransport__Mode=Grpc and no GrpcPsk refuses to boot, error names MeshTransport:GrpcPsk + the env form |
PENDING |
| 3 | Same with empty CentralGrpcEndpoints — refuses, names the key |
PENDING |
| 4 | Full suite green, 0 warnings, on the branch | PENDING |
Note: the new gRPC services are not yet mapped in Phase 0, so no wire probes here — the interceptor negative/positive matrix is unit-tested and wire-probed in 1A/1B.
Phase 1A — central→node command plane — PENDING
Branch feat/grpc-node-command @ ________. Rig-only flip: central-1/-2 →
MeshTransport__Mode=Grpc; site nodes remain ClusterClient (coexistence is the point).
Flip reverted after the gate, never committed.
| # | Check | Result |
|---|---|---|
| 1 | Deploy dispatch from AdminUI reaches all six driver nodes over gRPC (per-node Execute log lines on every node; zero SendToAll lines on central) |
PENDING |
| 2 | Acks still return over ClusterClient (site mode unchanged) and the deployment seals — mixed-transport deploy proven end to end | PENDING |
| 3 | RestartDriver + ReconnectDriver from AdminUI land on the target site node (EventStream delivery observed via driver restart in logs) |
PENDING |
| 4 | AlarmCommand (ack a scripted alarm from AdminUI) lands and the condition acknowledges |
PENDING |
| 5 | Unauthenticated grpcurl to a site node :4056 mesh_transport.v1.NodeCommandService/Execute ⇒ PermissionDenied; correct dev PSK ⇒ accepted |
PENDING |
| 6 | Wrong-key isolation: site-b's PSK presented to site-a ⇒ PermissionDenied (per-cluster scoping is real, not decorative) |
PENDING |
| 7 | A node with NULL GrpcPort is skipped with the warn-once, fan continues to the others |
PENDING |
| 8 | Telemetry + config-serve + LocalDb sync unaffected: 0 auth failures on their interceptors across the run | PENDING |
Phase 1B — node→central ack plane — PENDING
Branch feat/grpc-central-ack @ ________ (rebased on merged 1A). Rig-only flip:
site-a-1/-2 → Grpc; site-b stays ClusterClient; central back on ClusterClient mode
(its command direction) — acks are governed by the node's mode, so this isolates 1B.
| # | Check | Result |
|---|---|---|
| 1 | Deploy e2e: site-a acks arrive via ReportApply (central CentralCommandService log lines carry site-a node ids), NodeDeploymentState rows land, deployment seals |
PENDING |
| 2 | Site-b acks still arrive via ClusterClient in the same deployment — per-pair coexistence proven | PENDING |
| 3 | Exactly one ack row per site-a node per dispatch (no double-count from endpoint failover) | PENDING |
| 4 | Kill central-1 before the apply completes: site-a acks fail over to central-2's endpoint; deployment still seals | PENDING |
| 5 | Unauthenticated / wrong-cluster-key / missing x-otopcua-cluster probes against central :4055 CentralCommandService/ReportApply ⇒ PermissionDenied (missing-header rejection is central-side by design) |
PENDING |
| 6 | ConfigServe (DeploymentArtifactService) on the same listener unaffected: fetches succeed, 0 auth failures |
PENDING |
Phase 2 — full-rig cutover + soak — PENDING
Rig default flipped and committed: MeshTransport__Mode: "${OTOPCUA_MESH_MODE:-Grpc}".
Branch/SHA ________. Run the Baseline section first on the pre-flip build.
| # | Check | Result |
|---|---|---|
| 1 | Full rebuild all-Grpc: six nodes ready, deploy e2e seals, all acks via gRPC |
PENDING |
| 2 | Command matrix: Restart/Reconnect/AlarmCommand against site-a and site-b nodes | PENDING |
| 3 | Kill site-a-1 mid-dispatch: per-node Execute failure logged with node id + status (the drop is visible now); no retry storm (exactly one attempt per node per dispatch); deployment outcome honest (deadline path) |
PENDING |
| 4 | Kill central-1: site acks fail over to central-2; restart central-1: rig converges, no manual action | PENDING |
| 5 | Stop the site-b pair entirely, dispatch: per-node dispatch failures logged immediately (contrast with Baseline (b)'s silence); deadline backstop still marks the deployment; no duplicate applies after the pair returns (boot-replay path only) | PENDING |
| 6 | Zero ClusterClient activity under Grpc: docker logs <every node> | grep -ci "clusterclient" = 0 new lines post-boot |
PENDING |
| 7 | Soak ≥ 2 h under periodic deploys + command traffic: 0 PermissionDenied/Unauthenticated across all six nodes (PSK drift detector), 0 unexplained Unavailable |
PENDING |
| 8 | Full suite green on the cutover SHA | PENDING |
Phase 4 — terminal gate on the post-deletion build — PENDING
master @ ________ (after the Phase 3 deletion merge). Provenance guard: image build
timestamp verified LATER than the merge commit time — a pre-deletion image would invalidate
checks 6–7.
| # | Check | Result |
|---|---|---|
| 1 | PSK negatives on both services (no key / wrong key / wrong cluster's key); unset server key ⇒ all calls rejected (fail-closed proven live) | PENDING |
| 2 | Command matrix all six nodes (deploy dispatch + Restart + Reconnect + AlarmCommand) | PENDING |
| 3 | Ack matrix: every node acks exactly once, deployment seals | PENDING |
| 4 | Failover drills: central-kill ack failover; site-node-kill dispatch honesty (one logged failure, no retry) | PENDING |
| 5 | Mid-deploy central kill + restart: deadline/boot-replay backstops hold, ack counts exact (no loss, no dupes across the restart) | PENDING |
| 6 | Boundary check: no Akka association between any site container and central — the Baseline (a) probe now shows no cross-mesh :4053 traffic (pair-internal remoting only) |
PENDING |
| 7 | Full-rig restart (pairs together): clean convergence; grep -ri "receptionist|clusterclient" over all node logs = 0 lines |
PENDING |
| 8 | MeshGrpcBoundaryTests + full suite green on the merge SHA |
PENDING |
Carry-forwards
Anything a phase gate could not drive gets named here with why + what proves it instead, and is re-checked in the next gate (the ScadaBridge proportionate-gate rule). Currently: none.
Defects the gates caught that the suite did not
Recorded per phase as they occur, with root cause + the pinned regression test.