# ClusterClient → gRPC migration — CENTRAL TRACKING DOCUMENT **This is the single place to see program state.** Update it at every phase boundary, merge, defect, and decision. Everything else (design, plan, live gate) is referenced from here. **Program goal:** all central↔site mesh traffic on gRPC with per-cluster PSK auth; `ClusterClient`/`ClusterClientReceptionist` deleted; Akka remoting never crosses the mesh boundary. Mirrors ScadaBridge's completed 2026-07-22 migration (same cluster boxes, same end posture). ## Document map | Doc | Role | |---|---| | [`2026-08-06-clusterclient-to-grpc-design.md`](2026-08-06-clusterclient-to-grpc-design.md) | Design: inventory, rationale, decisions (§3), plan-vs-code corrections (§4), steelman (§5) | | [`2026-08-06-clusterclient-to-grpc.md`](2026-08-06-clusterclient-to-grpc.md) | Implementation plan: phases 0/1A/1B/2/3/4, task metadata, Opus-subagent execution rules, gotchas | | [`2026-08-06-clusterclient-to-grpc.md.tasks.json`](2026-08-06-clusterclient-to-grpc.md.tasks.json) | Machine task tracker (executing-plans resume point) | | [`2026-08-06-clusterclient-to-grpc-live-gate.md`](2026-08-06-clusterclient-to-grpc-live-gate.md) | Live gate record: baseline reproduction + per-phase check tables | | ScadaBridge reference | `~/Desktop/scadaproj/scadabridge_clusterclient_to_grpc.md` + `~/Desktop/ScadaBridge/docs/plans/2026-07-22-clusterclient-to-grpc-{plan,live-gate}.md` | ## Phase status board | Phase | Scope | Branch | Status | PR / merge SHA | Live gate | |---|---|---|---|---|---| | 0 | Proto + mappers + goldens; both auth interceptors; `IMeshPskProvider`; options/validators (+`Grpc` value, SplitTopology relaxation); rig dark config; stale-comment fixes; FleetStatus issue filed | `feat/grpc-phase0-contract-auth` | **MERGED to `master`** @ `10b6d209` (2026-08-06) — all 6 tasks complete @ `23555ed0`; every task spec ✅ + code ✅ (Batch B on Opus with a closed fix loop); issues #542 + #543 filed | `10b6d209` | **PASS 4/4** | | 1A | Central→node command plane (`NodeCommandService` on the Telemetry listener; `MeshGrpcCommandDialer`; third mode branch in `CentralCommunicationActor`) — **worktree A** | `feat/grpc-node-command` | **COMPLETE @ `e54883a8` — gate re-run PASSED 2026-08-06.** Run 1 @ `fbf7b708` FAILED on defect 4 (no admin node could boot, mode-independent); fixed by `c0475f2d` + `e54883a8` and re-gated. 1A.1 ✅ `64774652` + 1A.2 ✅ `6a862011` (+fixes `48dc2120`/`86633f14`) all reviewed spec ✅ + code ✅. **MERGED to `master`** @ `b360239e` (2026-08-06) | `b360239e` | **PASS 8/8** (check 3's AdminUI half closed by composition — the control is structurally unreachable, see carry-forwards) | | 1B | Node→central ack plane (`CentralCommandService` on the ConfigServe listener; `GrpcApplyAckSender`; third branch in `NodeCommunicationActor`) — **worktree B**, rebased on merged 1A | `feat/grpc-central-ack` | **MERGED to `master`** @ `7203d135` (2026-08-06, branch tip `cf229a95` incl. the phases 2-4 plan update); 1B.1 ✅ + 1B.2 ✅ (Opus: deadline Critical found+fixed) + gate **PASS 7/7** incl. the 4b hang drill (~10 s failover = exactly one AckTimeoutSeconds — live proof of the fix); at-least-once duplicate absorbed idempotently exactly as traced | `7203d135` | **PASS 7/7** | | 2 | **Opens with Task 2.0 (MUST FIX): honest PreStart-race logging** (both gates: fires ~every admin boot, two false Errors — soak error-grep impossible without it). Then baseline reproduction on pre-flip build, full-rig cutover (`OTOPCUA_MESH_MODE` default → `Grpc`) + soak + kill drills. **Targeted-test policy in force (plan Phase 2 header): per-task unit runs targeted; ONE full-suite run per phase at the DoD** | (rig commit; 2.0 on master) | NOT STARTED | — | PENDING | | 3 | Delete ClusterClient (code/HOCON/config/tests); SplitTopology requires `Grpc`; grep-gates; docs sweep incl. scadaproj index | `feat/grpc-phase3-deletion` | NOT STARTED | — | — | | 4 | Terminal live gate on the post-deletion image (8 checks, provenance-guarded) | (on `master`) | NOT STARTED | — | PENDING | **Execution rules recap:** phase order 0 → (1A ∥ 1B) → 2 → 3 → 4; 1A∥1B only in separate git worktrees, one Opus subagent per worktree, sequential tasks within a worktree; 1A merges first, 1B rebases (union-resolve `Host/Program.cs` map block, compose env, `docs/Configuration.md`); rig mode flips before Phase 2 are never committed; **STOP at every phase DoD — PR + merge are user-owned.** ## Decision log | Date | Decision | Where recorded | |---|---|---| | 2026-08-06 | Reuse existing listeners (Telemetry `:4056` node-side, ConfigServe `:4055` central-side); rejected dedicated port + `ClusterNode` column | design §3.1 | | 2026-08-06 | Per-cluster PSK `OT-MESH-PSK-` from Secrets; rejected fleet-wide shared key (blast radius); `MeshTransport:ClusterPsks` config map as rig/test override | design §3.4, plan T0.3/T0.5 | | 2026-08-06 | Fire-and-forget semantics preserved exactly — no retries/queues; failures become visible logs, backstops (2-min deadline, boot replay) unchanged | design §3.3 | | 2026-08-06 | `Dps` mode survives for single-mesh topologies; only `ClusterClient` is deleted; the three stale "Phase 6 deletes DPS" comments get corrected, not obeyed | design §3.6/§4.2 | | 2026-08-06 | No transport interface — third mode branch inside the two choke-point actors (ScadaBridge §7.1 economy) | design §4.3 | | 2026-08-06 | Coexistence granularity: central's mode governs commands, each node's mode governs its acks (per-pair flips possible — better than ScadaBridge's central-wide flag) | design §3.6 | | 2026-08-06 | `AlarmCommandMessage.operation` is a proto `string`, NOT an enum — the CLR `AlarmCommand.Operation` is an unconstrained string, and an enum would silently decode unknown operations to `UNSPECIFIED` (the drop class this migration removes). Plan T0.1's "AlarmCommand verb" enum wording was a plan defect; only `ApplyOutcome` carries the enum-0 rule. T0.2's goldens must pin pass-through of an unrecognized operation string | proto comment + this log (T0.1 review) | | 2026-08-06 | **Gate follows the endpoint, not the mode** (1A.2 I-2): known-cluster set published on every contact load in all three modes, from ALL `ClusterNode` rows; fan set unchanged. Survives dark-switch rollback with nothing for 1B to remember | tracking §register refinement + actor docs | | 2026-08-06 | `CentralCommandService` mapped by PORT, not mode-gated (1B.1): a mode flip on a running fleet must not need a central restart before acks flow; matches every sibling service | `Program.cs` comment + this log | | 2026-08-06 | Ack deadline is PER ENDPOINT, not per walk; the walk is at-least-once on the deadline path, absorbed idempotently by the coordinator's NodeId-keyed upsert | `MeshTransportOptions` docs + defect log #1 | | 2026-08-06 | `correlation_id` lives on the envelope AND as inner fields on the three commands whose CLR record carries one — each command round-trips independently in the T0.2 mapper; the envelope covers `AlarmCommand` (no CLR correlation member). For the three, the inner field is authoritative on decode | proto comment + this log (T0.1 review) | ## Phase 1A security register (from the Batch B Opus code review — HARD GATE: all four close BEFORE `CentralCommandService` is mapped) 1. **CLOSED (1A.2 @ `48dc2120`, refined — see below).** Known-cluster-set membership gate in `TryGetPsk` — central rebuilds its cluster set from `ClusterNode` rows every `ContactRefreshSeconds`; refuse a cluster id outside that set in O(1) with no store touch. Closes BOTH open review findings at once: (a) the timing oracle that currently defeats the uniform-denial anti-enumeration text (unknown cluster pays a store read on every probe because negatives are deliberately never cached; known cluster is cached — a stable remotely-measurable distinguisher), and (b) the pre-auth resource exhaustion / audit-flood vector (each distinct forged `x-otopcua-cluster` header costs a secret-store read + a `secret.resolve.not-found` audit row into `ConfigAuditLog` via the audit singleton, attacker-chosen `Target` values, before any credential check). Namespace confinement (`OT-MESH-PSK-` prefix + `SecretName` allow-list) means no disclosure risk — this is flood/oracle only. 2. **CLOSED (1A.2 @ `14d21496`).** `bool Matches(string clusterId, string presented)` on `IMeshPskProvider` — keep `FixedTimeEquals` inside the provider so a future verify-side caller cannot write `==` and have it compile. Do before 1B adds the dial-side caller. 3. **DI-shape pin test** — `IMeshPskProvider` is `hasAdmin`-only and its pairing with the central interceptor is by convention; a driver-only node injecting it throws on first call, not at boot. Add a Host.IntegrationTests check (admin host resolves it; driver-only host does NOT) when Phase 1 maps the services. **Refinement on item 1 (decision, 2026-08-06):** the gate follows the ENDPOINT, not the mode — published on every contact load in every transport mode (the interceptor and service are port-mapped, not mode-gated, so a dark-switch rollback must not deactivate the gate), and built from ALL `ClusterNode` rows (gate answers "does central know this cluster"; the fan set stays `Enabled && !MaintenanceMode`, so gate ⊇ fan — this also dissolved the maintenance/ack-auth coupling). Pinned by the every-mode Theory + gate-⊋-fan fact + the Dps no-ClusterClient guard pin (`86633f14`). Residuals accepted + visible: gate is fail-open until the first successful contact load (and stays stale if central boots with SQL down — warned every tick); gate derives from the node table, so a `ServerCluster` with zero rows is outside it. **Follow-ups from the 1A.2/1B.2 Opus reviews (tracked, not blocking):** per-tick `InvalidateAll` puts a cold-cache 3 s store read on the inbound auth path — MEASURE on the 1B.3 gate (ack burst inside ~3 s after a refresh tick), then consider diff-based invalidation; the command plane is h2c-only so the per-cluster PSK crosses the network in cleartext on a write surface — precedent-consistent (all sibling gRPC planes), record as accepted risk + TLS-option issue (queued below); dialer minors (Lazy channel wrapper, retired-endpoint eviction, gRPC deadline vs token cancellation, pre-existing `LoadContactsFromDb` Task.Run(ct) pattern). 4. **Integration gate over real Kestrel h2c** — both new interceptors have unit coverage only; the one-public-ctor reflection test approximates but cannot prove the interceptor is actually invoked. Wire-probe in the 1A/1B live gates (already in the live-gate tables). Non-gating notes from the same review: central-side unseeded-PSK is invisible until a deploy stalls (consider a "PSK seeded for every known cluster?" `/hosts`/health surface — the #521/#541 shape); `SecretName` lowercases and allows only `[a-z0-9._/-]` (underscore IS allowed — the review's `Site_A` example was wrong, caught + corrected by the T0.6 agent against the pinned 0.2.3 DLL), so a cluster id with any character outside that set fails closed at the store and must use `ClusterPsks` — documented in `docs/Configuration.md` (rig ids `MAIN`/`SITE-A`/`SITE-B` are fine); sibling interceptors still use `IsNullOrEmpty` (whitespace key opens them) — family-wide backport is out of this program's scope, file alongside the FleetStatus issue in T0.6; consider a non-throwing counter metric if the poison-guard silent catch ever fires in production. ## Phase 1 dispatch notes (from Task 0.2 reviews — fold into the 1A/1B implementer prompts) - The alarm arm of `FromEnvelope` necessarily discards the envelope correlation (`AlarmCommand` has no CLR member for it) — the 1A router must read `envelope.CorrelationId` itself for logging. - `ApplyAck` is the ONE record of the five that never rides `CommandEnvelope`: `GrpcApplyAckSender` (1B) calls `MeshTransportDtoMapper.ToProto(ApplyAck)` directly on `ReportApply` — do not expect (or add) a sixth `ToEnvelope` overload. - Consider a thin try/catch at the `Execute`/`ReportApply` service boundary wrapping `FromEnvelope`/`FromProto` failures with the envelope's correlation_id (readable as a plain string even when inner ids fail to parse) so an operator can correlate a rejected call. - Revisit the untyped `ToEnvelope(object, CorrelationId alarmCorrelationId)` signature when the 1A router lands — if the router prefers one always-honoured transport correlation, change the signature then, before it has more callers. ## Defect log Defects found during execution (live-gate or review), with root cause + pinned test. | # | Phase | Defect | Root cause | Fix / pin | |---|---|---|---|---| | 1 | 1B.2 | **Failover silently defeated on a hanging central** (CRITICAL, caught by Opus code review): the ack deadline was one absolute `DateTime` shared across the endpoint walk — a hung (not refused) central-1 burned the whole budget and central-2 was dialed with an expired deadline | deadline hoisted outside the `foreach`; a kill-style drill can't see it (refused = fast-fail), only a hang discriminates | `e54711a2`; pinned by `Each_endpoint_gets_a_FRESH_deadline` (mutation-verified); 1B.3 gains a `docker pause` hang drill | | 2 | 1A.2 | **Dark-switch rollback deactivated the known-cluster gate** (Opus review, I-2): gate published only under `Mode=Grpc` while the interceptor + service are port-mapped in every mode — a rollback to ClusterClient/Dps would silently restore the timing oracle + pre-auth flood | gate activation coupled to transport mode instead of the endpoint it protects | `48dc2120` — gate follows the endpoint; pinned by the every-mode Theory + Dps guard pin `86633f14` | | 5 | 1A.3 (live gate, run 2) | **The `PreStart` resolve loses its race on real boots** — 2 of 3 observed admin boots logged `Failed to resolve IMeshCommandDialer` / `IMeshPskProvider`, same `AuditWriterActorKey` cause. NOT a defect: it is exactly the case the review-prescribed hardening was written for | `PreStart` runs on a dispatcher thread concurrently with the builder's remaining starters, so it can still precede starter #4 | **Closed by `e54883a8`, and the heal is PROVEN LIVE** — the coordinator singleton was moved onto the node that lost the race (stop central-2) and it fanned a dispatch to all five live nodes with no dialer error. Residual is cosmetic (Error text reads permanent; will trip a soak error-grep on a healthy node) → Phase 2 | | 4 | 1A.3 (live gate, run 1) | **An admin node cannot boot** (BLOCKED 1A): both central nodes die at startup with `MissingActorRegistryEntryException: ZB.MOM.WW.OtOpcUa.ControlPlane.AuditWriterActorKey`, taking `AkkaHostedService` and the whole host with them. Mode-INDEPENDENT — reproduced on the same image with `MeshTransport__Mode=ClusterClient`, so it is not the gate's flip. Driver-only nodes are unaffected (they register neither `IMeshPskProvider` nor `ActorAuditWriter`) | `WithOtOpcUaControlPlaneSingletons`'s first `WithActors` starter eagerly resolves `IMeshCommandDialer` + `IMeshPskProvider` to pass into `CentralCommunicationActor.Props`. That graph is dialer → `IMeshPskProvider` → `ISecretResolver` → (`AddZbSecrets`) → `IAuditWriter` → `ActorAuditWriter`, whose ctor calls `registry.Get()` — and `WithSingleton` is registered ~60 lines later in the same builder. Same class as scadaproj#1 and mesh Phase 6's `IClusterRoleInfo` crash: only visible with the DI graph built in a real host, which is why the container-activation test (no `AddZbSecrets` in its collection) and `TwoNodeClusterHarness` (no provider at all) both miss it. Aggravating: the failed process does not exit, so `docker ps` reports `Up restarts=0` while the node serves no HTTP | **OPEN.** Fix + red-first pin belong to 1A (the gate made no product change). Suggested shape: hand the actor lazy accessors, mirroring the `Func<>` the same block already uses for the coordinator proxy. Pin must build the graph in starter order (a Host.IntegrationTests boot of an admin-role host), not merely activate the dialer | | 3 | 1A.2 | **Fan parked the actor mailbox** (self-caught by the implementer): dialing from the receive ran `TryGetPsk`'s cold-cache 3 s block pre-await (cold after EVERY refresh due to `InvalidateAll`), stalling deployment acks behind the fan | async methods run synchronously to the first await | `Task.Run` hop (token None — a canceled scheduling would read as a DB failure via PipeTo); pinned red-first by `A_slow_dial_does_not_park_the_actors_mailbox` | ## Carry-forward register Checks a gate could not drive, named + carried to the next gate (proportionate-gate rule). | Item | Deferred from | Proven instead by | Closed in | |---|---|---|---| | Hang-style central failover drill (`docker pause`, not `docker stop` — only a hang discriminates the per-endpoint-deadline fix) | 1B.2 review | unit fact `Each_endpoint_gets_a_FRESH_deadline` | 1B.3 gate (MUST include) | | I-4 measurement: ack burst within ~3 s after a contact-refresh tick (cold PSK cache on the inbound auth path) | 1A.2 review | bounded 3 s store read + `Lazy` stampede suppression | 1B.3 gate (observe) | | `TwoNodeClusterHarness` logs the I-1 null-provider Error at every spawn (harness registers no `IMeshPskProvider`) — EXPECTED, not a rig fault | 1A.2 re-review | production pairing is `hasAdmin`-unconditional in one block | note for 1B.3/Phase 2 readers | | Register item 3 (DI role-scoping pin: admin resolves provider+dialer, driver-only does NOT) — container-activation test exists, role-scoping pin outstanding | 1A.2 | `It_activates_from_the_container_despite_the_optional_client_factory_parameter`; starter-order half now covered by `ControlPlaneStarterOrderTests` | Phase 2 (or 1B.3) — note defect 4 shows the missing pin is not merely role-scoping but **starter-order**: the container-activation test resolves from a collection without `AddZbSecrets` | | `Resolve`-failure Error wording says "continuing without it" (reads permanent — the memoize heals it at first use) and the line fires on real boots (2/3 in the gate) — reword or whitelist it before the Phase 2 soak error-grep, else a healthy central scores as failing the soak | 1A.3 re-run | heal proven live: singleton moved onto the race-losing node, fan clean, no dialer-missing error | Phase 2 (soak grep design) | | A repaired `GrpcPort` takes effect only at the next contact refresh (≤60 s): a deploy 5 s post-restore was still skipped and honestly TimedOut — by design, operator-visible; document if it surprises in Phase 2 | 1A.3 re-run | check 7 evidence | Phase 2 (observe) | | I-4 (cold PSK cache on the inbound auth path after each refresh tick) is structurally unmeasurable on docker-dev — `ClusterPsks` config map answers ahead of the store, so the cache is always empty and `InvalidateAll` clears nothing | 1B.3 gate | every ack landed within the posting second | Phase 2+ (needs a store-backed rig) | | After a central restart, a node's cached `GrpcChannel` can sit in transient failure ~2 min (client backoff; TCP open, sibling reaches the same endpoint) — no ack lost, the walk absorbs it loudly. Expect a failover-line burst after any central restart in the Phase 2 soak | 1B.3 gate | 3 dispatches: 2 failed over, 3rd direct | Phase 2 (soak expectations) | | **The PreStart race fires on essentially EVERY admin boot** (4/4 in the 1B gate, 2/3 in 1A) and logs two FALSE Errors ("continuing without it", "gate will NEVER be published") that the memoize immediately heals — the wording fix is now a MUST before the Phase 2 soak error-grep, not a nicety | 1B.3 gate obs (b) | gate published + acks accepted on every one of those boots | Phase 2 opening task | | `RevisionHash` moved on byte-identical config after a direct SQL `UPDATE` bumped `Script.RowVersion` (hash covers a column content-identity does not); determinism itself holds (3 no-change dispatches → identical hash) | 1A.3 re-run | incidental, out of program scope | note only | | 1A checks 1/2/7 (deploy fan over gRPC, mixed-transport seal, NULL-`GrpcPort` skip) and the central half of 4 | 1A.3 gate run 1 | — | **CLOSED in 1A.3 run 2** — all driven live and PASS | | Check 8 is vacuous for telemetry + config-serve — central never booted, so neither plane carried a call | 1A.3 gate run 1 | — | **CLOSED in 1A.3 run 2** — both planes carried authenticated traffic (28 KB artifact fetches from `:4055`, telemetry streams open on all six) with 0 denials | | **`DriverStatusPanel.razor` is referenced by no page or component** — the Reconnect/Restart buttons are unreachable in the running AdminUI (pre-existing on `master`, not caused by this program; re-verified in run 2: 0 `