Joseph Doherty
143 lines
8.8 KiB
Markdown
143 lines
8.8 KiB
Markdown
# ClusterClient → gRPC migration — live gate record
|
||
|
||
Rig: `docker-dev/` (central-1/-2 = `admin,driver,cluster-MAIN`; site-a-1/-2, site-b-1/-2;
|
||
sql + traefik + migrator + cluster-seed), rebuilt from the branch under test via
|
||
`docker compose -f docker-dev/docker-compose.yml up -d --build`. Recorded check-by-check in
|
||
the family's live-gate format. Later phases' sections are appended as those phases land; every
|
||
check row gets a **PASS**/**FAIL** verdict plus one quantitative justification clause, with
|
||
fenced evidence (real commands, real output). A run invalidated by a concurrent rig rebuild is
|
||
**void** — note it and re-run (the ScadaBridge void-run rule).
|
||
|
||
Provenance discipline: every phase section opens with branch + commit SHA + "image rebuilt,
|
||
all N containers recreated", and Phase 4 additionally verifies the image build timestamp
|
||
post-dates the deletion merge (a stale image invalidates the boundary checks).
|
||
|
||
Plan: `2026-08-06-clusterclient-to-grpc.md`. Tracking:
|
||
`2026-08-06-clusterclient-to-grpc-tracking.md`.
|
||
|
||
---
|
||
|
||
## Baseline (run on the PRE-change build, before the Phase 2 flip)
|
||
|
||
The two gaps this program closes, **reproduced rather than assumed**:
|
||
|
||
**(a) Unauthenticated Akka remoting crosses the WAN boundary.** From the host:
|
||
|
||
```bash
|
||
# A live TCP association from a site container to central's Akka port:
|
||
docker exec otopcua-dev-site-a-1-1 bash -c 'exec 3<>/dev/tcp/central-1/4053 && echo OPEN'
|
||
# And the receptionist actually accepting cross-mesh traffic (central log after any AdminUI
|
||
# driver restart targeting site-a):
|
||
docker logs otopcua-dev-central-1-1 2>&1 | grep -i "clusterclient\|receptionist" | tail -5
|
||
```
|
||
|
||
Expected: `OPEN`; receptionist/client lines present. Nothing on this path presents any
|
||
credential.
|
||
|
||
**(b) Silent drop on an unreachable cluster.** Stop the site-b pair, dispatch a deployment
|
||
(`POST http://localhost:9200/api/deployments`, `X-Api-Key: docker-dev-deploy-key`), observe:
|
||
no per-node dispatch error surfaces anywhere; site-b's `NodeDeploymentState` rows sit
|
||
`Applying` until the 2-minute deadline marks the deployment `TimedOut`/partially failed. The
|
||
deadline is the only witness. Restart the pair.
|
||
|
||
- [ ] Baseline (a) recorded with evidence — **PENDING**
|
||
- [ ] Baseline (b) recorded with evidence — **PENDING**
|
||
|
||
---
|
||
|
||
## Phase 0 — contract + auth + config groundwork — **PENDING**
|
||
|
||
Branch `feat/grpc-phase0-contract-auth` @ `________`. Image rebuilt, all containers recreated.
|
||
|
||
| # | Check | Result |
|
||
|---|---|---|
|
||
| 1 | All six nodes boot on `ClusterClient` with the dark `Grpc` config present (PSKs, endpoints, ClusterPsks map); no validator fires | PENDING |
|
||
| 2 | Deliberate misconfig: a scratch container with `MeshTransport__Mode=Grpc` and **no** `GrpcPsk` refuses to boot, error names `MeshTransport:GrpcPsk` + the env form | PENDING |
|
||
| 3 | Same with empty `CentralGrpcEndpoints` — refuses, names the key | PENDING |
|
||
| 4 | Full suite green, 0 warnings, on the branch | PENDING |
|
||
|
||
Note: the new gRPC services are not yet mapped in Phase 0, so no wire probes here — the
|
||
interceptor negative/positive matrix is unit-tested and wire-probed in 1A/1B.
|
||
|
||
---
|
||
|
||
## Phase 1A — central→node command plane — **PENDING**
|
||
|
||
Branch `feat/grpc-node-command` @ `________`. Rig-only flip: central-1/-2 →
|
||
`MeshTransport__Mode=Grpc`; **site nodes remain `ClusterClient`** (coexistence is the point).
|
||
Flip reverted after the gate, never committed.
|
||
|
||
| # | Check | Result |
|
||
|---|---|---|
|
||
| 1 | Deploy dispatch from AdminUI reaches **all six** driver nodes over gRPC (per-node `Execute` log lines on every node; zero `SendToAll` lines on central) | PENDING |
|
||
| 2 | Acks still return over ClusterClient (site mode unchanged) and the deployment **seals** — mixed-transport deploy proven end to end | PENDING |
|
||
| 3 | `RestartDriver` + `ReconnectDriver` from AdminUI land on the target site node (EventStream delivery observed via driver restart in logs) | PENDING |
|
||
| 4 | `AlarmCommand` (ack a scripted alarm from AdminUI) lands and the condition acknowledges | PENDING |
|
||
| 5 | Unauthenticated `grpcurl` to a site node `:4056` `mesh_transport.v1.NodeCommandService/Execute` ⇒ `PermissionDenied`; correct dev PSK ⇒ accepted | PENDING |
|
||
| 6 | **Wrong-key isolation:** site-b's PSK presented to site-a ⇒ `PermissionDenied` (per-cluster scoping is real, not decorative) | PENDING |
|
||
| 7 | A node with NULL `GrpcPort` is skipped with the warn-once, fan continues to the others | PENDING |
|
||
| 8 | Telemetry + config-serve + LocalDb sync unaffected: 0 auth failures on their interceptors across the run | PENDING |
|
||
|
||
---
|
||
|
||
## Phase 1B — node→central ack plane — **PENDING**
|
||
|
||
Branch `feat/grpc-central-ack` @ `________` (rebased on merged 1A). Rig-only flip:
|
||
site-a-1/-2 → `Grpc`; site-b stays `ClusterClient`; central back on `ClusterClient` mode
|
||
(its command direction) — acks are governed by the **node's** mode, so this isolates 1B.
|
||
|
||
| # | Check | Result |
|
||
|---|---|---|
|
||
| 1 | Deploy e2e: site-a acks arrive via `ReportApply` (central `CentralCommandService` log lines carry site-a node ids), `NodeDeploymentState` rows land, deployment seals | PENDING |
|
||
| 2 | Site-b acks still arrive via ClusterClient in the same deployment — per-pair coexistence proven | PENDING |
|
||
| 3 | Exactly **one** ack row per site-a node per dispatch (no double-count from endpoint failover) | PENDING |
|
||
| 4 | Kill central-1 before the apply completes: site-a acks fail over to central-2's endpoint; deployment still seals | PENDING |
|
||
| 5 | Unauthenticated / wrong-cluster-key / missing `x-otopcua-cluster` probes against central `:4055` `CentralCommandService/ReportApply` ⇒ `PermissionDenied` (missing-header rejection is central-side by design) | PENDING |
|
||
| 6 | ConfigServe (`DeploymentArtifactService`) on the same listener unaffected: fetches succeed, 0 auth failures | PENDING |
|
||
|
||
---
|
||
|
||
## Phase 2 — full-rig cutover + soak — **PENDING**
|
||
|
||
Rig default flipped and committed: `MeshTransport__Mode: "${OTOPCUA_MESH_MODE:-Grpc}"`.
|
||
Branch/SHA `________`. **Run the Baseline section first on the pre-flip build.**
|
||
|
||
| # | Check | Result |
|
||
|---|---|---|
|
||
| 1 | Full rebuild all-`Grpc`: six nodes ready, deploy e2e seals, all acks via gRPC | PENDING |
|
||
| 2 | Command matrix: Restart/Reconnect/AlarmCommand against site-a and site-b nodes | PENDING |
|
||
| 3 | Kill site-a-1 mid-dispatch: per-node `Execute` failure **logged with node id + status** (the drop is visible now); no retry storm (exactly one attempt per node per dispatch); deployment outcome honest (deadline path) | PENDING |
|
||
| 4 | Kill central-1: site acks fail over to central-2; restart central-1: rig converges, no manual action | PENDING |
|
||
| 5 | Stop the site-b pair entirely, dispatch: per-node dispatch failures logged immediately (contrast with Baseline (b)'s silence); deadline backstop still marks the deployment; **no duplicate applies** after the pair returns (boot-replay path only) | PENDING |
|
||
| 6 | **Zero ClusterClient activity under `Grpc`:** `docker logs <every node> \| grep -ci "clusterclient"` = 0 new lines post-boot | PENDING |
|
||
| 7 | Soak ≥ 2 h under periodic deploys + command traffic: 0 `PermissionDenied`/`Unauthenticated` across all six nodes (PSK drift detector), 0 unexplained `Unavailable` | PENDING |
|
||
| 8 | Full suite green on the cutover SHA | PENDING |
|
||
|
||
---
|
||
|
||
## Phase 4 — terminal gate on the post-deletion build — **PENDING**
|
||
|
||
`master` @ `________` (after the Phase 3 deletion merge). **Provenance guard:** image build
|
||
timestamp verified LATER than the merge commit time — a pre-deletion image would invalidate
|
||
checks 6–7.
|
||
|
||
| # | Check | Result |
|
||
|---|---|---|
|
||
| 1 | PSK negatives on both services (no key / wrong key / wrong cluster's key); **unset server key ⇒ all calls rejected** (fail-closed proven live) | PENDING |
|
||
| 2 | Command matrix all six nodes (deploy dispatch + Restart + Reconnect + AlarmCommand) | PENDING |
|
||
| 3 | Ack matrix: every node acks exactly once, deployment seals | PENDING |
|
||
| 4 | Failover drills: central-kill ack failover; site-node-kill dispatch honesty (one logged failure, no retry) | PENDING |
|
||
| 5 | Mid-deploy central kill + restart: deadline/boot-replay backstops hold, ack counts exact (no loss, no dupes across the restart) | PENDING |
|
||
| 6 | **Boundary check: no Akka association between any site container and central** — the Baseline (a) probe now shows no cross-mesh `:4053` traffic (pair-internal remoting only) | PENDING |
|
||
| 7 | Full-rig restart (pairs together): clean convergence; `grep -ri "receptionist\|clusterclient"` over all node logs = 0 lines | PENDING |
|
||
| 8 | `MeshGrpcBoundaryTests` + full suite green on the merge SHA | PENDING |
|
||
|
||
### Carry-forwards
|
||
|
||
Anything a phase gate could not drive gets named here with why + what proves it instead, and
|
||
is re-checked in the next gate (the ScadaBridge proportionate-gate rule). Currently: none.
|
||
|
||
### Defects the gates caught that the suite did not
|
||
|
||
Recorded per phase as they occur, with root cause + the pinned regression test.
|