lmxopcua/docs/plans/2026-08-06-clusterclient-to-grpc-tracking.md

# ClusterClient → gRPC migration — CENTRAL TRACKING DOCUMENT

**This is the single place to see program state.** Update it at every phase boundary, merge,
defect, and decision. Everything else (design, plan, live gate) is referenced from here.

**Program goal:** all central↔site mesh traffic on gRPC with per-cluster PSK auth;
`ClusterClient`/`ClusterClientReceptionist` deleted; Akka remoting never crosses the mesh
boundary. Mirrors ScadaBridge's completed 2026-07-22 migration (same cluster boxes, same end
posture).

## Document map

| Doc | Role |
|---|---|
| [`2026-08-06-clusterclient-to-grpc-design.md`](2026-08-06-clusterclient-to-grpc-design.md) | Design: inventory, rationale, decisions (§3), plan-vs-code corrections (§4), steelman (§5) |
| [`2026-08-06-clusterclient-to-grpc.md`](2026-08-06-clusterclient-to-grpc.md) | Implementation plan: phases 0/1A/1B/2/3/4, task metadata, Opus-subagent execution rules, gotchas |
| [`2026-08-06-clusterclient-to-grpc.md.tasks.json`](2026-08-06-clusterclient-to-grpc.md.tasks.json) | Machine task tracker (executing-plans resume point) |
| [`2026-08-06-clusterclient-to-grpc-live-gate.md`](2026-08-06-clusterclient-to-grpc-live-gate.md) | Live gate record: baseline reproduction + per-phase check tables |
| ScadaBridge reference | `~/Desktop/scadaproj/scadabridge_clusterclient_to_grpc.md` + `~/Desktop/ScadaBridge/docs/plans/2026-07-22-clusterclient-to-grpc-{plan,live-gate}.md` |

## Phase status board

| Phase | Scope | Branch | Status | PR / merge SHA | Live gate |
|---|---|---|---|---|---|
| 0 | Proto + mappers + goldens; both auth interceptors; `IMeshPskProvider`; options/validators (+`Grpc` value, SplitTopology relaxation); rig dark config; stale-comment fixes; FleetStatus issue filed | `feat/grpc-phase0-contract-auth` | **MERGED to `master`** @ `10b6d209` (2026-08-06) — all 6 tasks complete @ `23555ed0`; every task spec ✅ + code ✅ (Batch B on Opus with a closed fix loop); issues #542 + #543 filed | `10b6d209` | **PASS 4/4** |
| 1A | Central→node command plane (`NodeCommandService` on the Telemetry listener; `MeshGrpcCommandDialer`; third mode branch in `CentralCommunicationActor`) — **worktree A** | `feat/grpc-node-command` | **COMPLETE @ `e54883a8` — gate re-run PASSED 2026-08-06.** Run 1 @ `fbf7b708` FAILED on defect 4 (no admin node could boot, mode-independent); fixed by `c0475f2d` + `e54883a8` and re-gated. 1A.1 ✅ `64774652` + 1A.2 ✅ `6a862011` (+fixes `48dc2120`/`86633f14`) all reviewed spec ✅ + code ✅. **MERGED to `master`** @ `b360239e` (2026-08-06) | `b360239e` | **PASS 8/8** (check 3's AdminUI half closed by composition — the control is structurally unreachable, see carry-forwards) |
| 1B | Node→central ack plane (`CentralCommandService` on the ConfigServe listener; `GrpcApplyAckSender`; third branch in `NodeCommunicationActor`) — **worktree B**, rebased on merged 1A | `feat/grpc-central-ack` | **MERGED to `master`** @ `7203d135` (2026-08-06, branch tip `cf229a95` incl. the phases 2-4 plan update); 1B.1 ✅ + 1B.2 ✅ (Opus: deadline Critical found+fixed) + gate **PASS 7/7** incl. the 4b hang drill (~10 s failover = exactly one AckTimeoutSeconds — live proof of the fix); at-least-once duplicate absorbed idempotently exactly as traced | `7203d135` | **PASS 7/7** |
| 2 | **Opens with Task 2.0 (MUST FIX): honest PreStart-race logging** (both gates: fires ~every admin boot, two false Errors — soak error-grep impossible without it). Then baseline reproduction on pre-flip build, full-rig cutover (`OTOPCUA_MESH_MODE` default → `Grpc`) + soak + kill drills. **Targeted-test policy in force (plan Phase 2 header): per-task unit runs targeted; ONE full-suite run per phase at the DoD** | (rig commit; 2.0 on master) | NOT STARTED | — | PENDING |
| 3 | Delete ClusterClient (code/HOCON/config/tests); SplitTopology requires `Grpc`; grep-gates; docs sweep incl. scadaproj index | `feat/grpc-phase3-deletion` | NOT STARTED | — | — |
| 4 | Terminal live gate on the post-deletion image (8 checks, provenance-guarded) | (on `master`) | NOT STARTED | — | PENDING |

**Execution rules recap:** phase order 0 → (1A ∥ 1B) → 2 → 3 → 4; 1A∥1B only in separate git
worktrees, one Opus subagent per worktree, sequential tasks within a worktree; 1A merges
first, 1B rebases (union-resolve `Host/Program.cs` map block, compose env,
`docs/Configuration.md`); rig mode flips before Phase 2 are never committed; **STOP at every
phase DoD — PR + merge are user-owned.**

## Decision log

| Date | Decision | Where recorded |
|---|---|---|
| 2026-08-06 | Reuse existing listeners (Telemetry `:4056` node-side, ConfigServe `:4055` central-side); rejected dedicated port + `ClusterNode` column | design §3.1 |
| 2026-08-06 | Per-cluster PSK `OT-MESH-PSK-<clusterId>` from Secrets; rejected fleet-wide shared key (blast radius); `MeshTransport:ClusterPsks` config map as rig/test override | design §3.4, plan T0.3/T0.5 |
| 2026-08-06 | Fire-and-forget semantics preserved exactly — no retries/queues; failures become visible logs, backstops (2-min deadline, boot replay) unchanged | design §3.3 |
| 2026-08-06 | `Dps` mode survives for single-mesh topologies; only `ClusterClient` is deleted; the three stale "Phase 6 deletes DPS" comments get corrected, not obeyed | design §3.6/§4.2 |
| 2026-08-06 | No transport interface — third mode branch inside the two choke-point actors (ScadaBridge §7.1 economy) | design §4.3 |
| 2026-08-06 | Coexistence granularity: central's mode governs commands, each node's mode governs its acks (per-pair flips possible — better than ScadaBridge's central-wide flag) | design §3.6 |
| 2026-08-06 | `AlarmCommandMessage.operation` is a proto `string`, NOT an enum — the CLR `AlarmCommand.Operation` is an unconstrained string, and an enum would silently decode unknown operations to `UNSPECIFIED` (the drop class this migration removes). Plan T0.1's "AlarmCommand verb" enum wording was a plan defect; only `ApplyOutcome` carries the enum-0 rule. T0.2's goldens must pin pass-through of an unrecognized operation string | proto comment + this log (T0.1 review) |
| 2026-08-06 | **Gate follows the endpoint, not the mode** (1A.2 I-2): known-cluster set published on every contact load in all three modes, from ALL `ClusterNode` rows; fan set unchanged. Survives dark-switch rollback with nothing for 1B to remember | tracking §register refinement + actor docs |
| 2026-08-06 | `CentralCommandService` mapped by PORT, not mode-gated (1B.1): a mode flip on a running fleet must not need a central restart before acks flow; matches every sibling service | `Program.cs` comment + this log |
| 2026-08-06 | Ack deadline is PER ENDPOINT, not per walk; the walk is at-least-once on the deadline path, absorbed idempotently by the coordinator's NodeId-keyed upsert | `MeshTransportOptions` docs + defect log #1 |
| 2026-08-06 | `correlation_id` lives on the envelope AND as inner fields on the three commands whose CLR record carries one — each command round-trips independently in the T0.2 mapper; the envelope covers `AlarmCommand` (no CLR correlation member). For the three, the inner field is authoritative on decode | proto comment + this log (T0.1 review) |

## Phase 1A security register (from the Batch B Opus code review — HARD GATE: all four close BEFORE `CentralCommandService` is mapped)

1. **CLOSED (1A.2 @ `48dc2120`, refined — see below).** Known-cluster-set membership gate in `TryGetPsk` — central rebuilds its cluster set from
   `ClusterNode` rows every `ContactRefreshSeconds`; refuse a cluster id outside that set in
   O(1) with no store touch. Closes BOTH open review findings at once: (a) the timing oracle
   that currently defeats the uniform-denial anti-enumeration text (unknown cluster pays a
   store read on every probe because negatives are deliberately never cached; known cluster is
   cached — a stable remotely-measurable distinguisher), and (b) the pre-auth resource
   exhaustion / audit-flood vector (each distinct forged `x-otopcua-cluster` header costs a
   secret-store read + a `secret.resolve.not-found` audit row into `ConfigAuditLog` via the
   audit singleton, attacker-chosen `Target` values, before any credential check). Namespace
   confinement (`OT-MESH-PSK-` prefix + `SecretName` allow-list) means no disclosure risk —
   this is flood/oracle only.
2. **CLOSED (1A.2 @ `14d21496`).** `bool Matches(string clusterId, string presented)` on `IMeshPskProvider` — keep
   `FixedTimeEquals` inside the provider so a future verify-side caller cannot write `==` and
   have it compile. Do before 1B adds the dial-side caller.
3. **DI-shape pin test** — `IMeshPskProvider` is `hasAdmin`-only and its pairing with the
   central interceptor is by convention; a driver-only node injecting it throws on first call,
   not at boot. Add a Host.IntegrationTests check (admin host resolves it; driver-only host
   does NOT) when Phase 1 maps the services.
**Refinement on item 1 (decision, 2026-08-06):** the gate follows the ENDPOINT, not the mode —
published on every contact load in every transport mode (the interceptor and service are
port-mapped, not mode-gated, so a dark-switch rollback must not deactivate the gate), and
built from ALL `ClusterNode` rows (gate answers "does central know this cluster"; the fan set
stays `Enabled && !MaintenanceMode`, so gate ⊇ fan — this also dissolved the
maintenance/ack-auth coupling). Pinned by the every-mode Theory + gate-⊋-fan fact + the Dps
no-ClusterClient guard pin (`86633f14`). Residuals accepted + visible: gate is fail-open until
the first successful contact load (and stays stale if central boots with SQL down — warned
every tick); gate derives from the node table, so a `ServerCluster` with zero rows is outside
it.

**Follow-ups from the 1A.2/1B.2 Opus reviews (tracked, not blocking):** per-tick
`InvalidateAll` puts a cold-cache 3 s store read on the inbound auth path — MEASURE on the
1B.3 gate (ack burst inside ~3 s after a refresh tick), then consider diff-based
invalidation; the command plane is h2c-only so the per-cluster PSK crosses the network in
cleartext on a write surface — precedent-consistent (all sibling gRPC planes), record as
accepted risk + TLS-option issue (queued below); dialer minors (Lazy channel wrapper,
retired-endpoint eviction, gRPC deadline vs token cancellation, pre-existing
`LoadContactsFromDb` Task.Run(ct) pattern).

4. **Integration gate over real Kestrel h2c** — both new interceptors have unit coverage only;
   the one-public-ctor reflection test approximates but cannot prove the interceptor is
   actually invoked. Wire-probe in the 1A/1B live gates (already in the live-gate tables).

Non-gating notes from the same review: central-side unseeded-PSK is invisible until a deploy
stalls (consider a "PSK seeded for every known cluster?" `/hosts`/health surface — the
#521/#541 shape); `SecretName` lowercases and allows only `[a-z0-9._/-]` (underscore IS
allowed — the review's `Site_A` example was wrong, caught + corrected by the T0.6 agent
against the pinned 0.2.3 DLL), so a cluster id with any character outside that set fails
closed at the store and must use `ClusterPsks` — documented in `docs/Configuration.md`
(rig ids `MAIN`/`SITE-A`/`SITE-B` are fine); sibling interceptors
still use `IsNullOrEmpty` (whitespace key opens them) — family-wide backport is out of this
program's scope, file alongside the FleetStatus issue in T0.6; consider a non-throwing counter
metric if the poison-guard silent catch ever fires in production.

## Phase 1 dispatch notes (from Task 0.2 reviews — fold into the 1A/1B implementer prompts)

- The alarm arm of `FromEnvelope` necessarily discards the envelope correlation
  (`AlarmCommand` has no CLR member for it) — the 1A router must read
  `envelope.CorrelationId` itself for logging.
- `ApplyAck` is the ONE record of the five that never rides `CommandEnvelope`:
  `GrpcApplyAckSender` (1B) calls `MeshTransportDtoMapper.ToProto(ApplyAck)` directly on
  `ReportApply` — do not expect (or add) a sixth `ToEnvelope` overload.
- Consider a thin try/catch at the `Execute`/`ReportApply` service boundary wrapping
  `FromEnvelope`/`FromProto` failures with the envelope's correlation_id (readable as a plain
  string even when inner ids fail to parse) so an operator can correlate a rejected call.
- Revisit the untyped `ToEnvelope(object, CorrelationId alarmCorrelationId)` signature when
  the 1A router lands — if the router prefers one always-honoured transport correlation,
  change the signature then, before it has more callers.

## Defect log

Defects found during execution (live-gate or review), with root cause + pinned test.

| # | Phase | Defect | Root cause | Fix / pin |
|---|---|---|---|---|
| 1 | 1B.2 | **Failover silently defeated on a hanging central** (CRITICAL, caught by Opus code review): the ack deadline was one absolute `DateTime` shared across the endpoint walk — a hung (not refused) central-1 burned the whole budget and central-2 was dialed with an expired deadline | deadline hoisted outside the `foreach`; a kill-style drill can't see it (refused = fast-fail), only a hang discriminates | `e54711a2`; pinned by `Each_endpoint_gets_a_FRESH_deadline` (mutation-verified); 1B.3 gains a `docker pause` hang drill |
| 2 | 1A.2 | **Dark-switch rollback deactivated the known-cluster gate** (Opus review, I-2): gate published only under `Mode=Grpc` while the interceptor + service are port-mapped in every mode — a rollback to ClusterClient/Dps would silently restore the timing oracle + pre-auth flood | gate activation coupled to transport mode instead of the endpoint it protects | `48dc2120` — gate follows the endpoint; pinned by the every-mode Theory + Dps guard pin `86633f14` |
| 5 | 1A.3 (live gate, run 2) | **The `PreStart` resolve loses its race on real boots** — 2 of 3 observed admin boots logged `Failed to resolve IMeshCommandDialer` / `IMeshPskProvider`, same `AuditWriterActorKey` cause. NOT a defect: it is exactly the case the review-prescribed hardening was written for | `PreStart` runs on a dispatcher thread concurrently with the builder's remaining starters, so it can still precede starter #4 | **Closed by `e54883a8`, and the heal is PROVEN LIVE** — the coordinator singleton was moved onto the node that lost the race (stop central-2) and it fanned a dispatch to all five live nodes with no dialer error. Residual is cosmetic (Error text reads permanent; will trip a soak error-grep on a healthy node) → Phase 2 |
| 4 | 1A.3 (live gate, run 1) | **An admin node cannot boot** (BLOCKED 1A): both central nodes die at startup with `MissingActorRegistryEntryException: ZB.MOM.WW.OtOpcUa.ControlPlane.AuditWriterActorKey`, taking `AkkaHostedService` and the whole host with them. Mode-INDEPENDENT — reproduced on the same image with `MeshTransport__Mode=ClusterClient`, so it is not the gate's flip. Driver-only nodes are unaffected (they register neither `IMeshPskProvider` nor `ActorAuditWriter`) | `WithOtOpcUaControlPlaneSingletons`'s first `WithActors` starter eagerly resolves `IMeshCommandDialer` + `IMeshPskProvider` to pass into `CentralCommunicationActor.Props`. That graph is dialer → `IMeshPskProvider` → `ISecretResolver` → (`AddZbSecrets`) → `IAuditWriter` → `ActorAuditWriter`, whose ctor calls `registry.Get<AuditWriterActorKey>()` — and `WithSingleton<AuditWriterActorKey>` is registered ~60 lines later in the same builder. Same class as scadaproj#1 and mesh Phase 6's `IClusterRoleInfo` crash: only visible with the DI graph built in a real host, which is why the container-activation test (no `AddZbSecrets` in its collection) and `TwoNodeClusterHarness` (no provider at all) both miss it. Aggravating: the failed process does not exit, so `docker ps` reports `Up restarts=0` while the node serves no HTTP | **OPEN.** Fix + red-first pin belong to 1A (the gate made no product change). Suggested shape: hand the actor lazy accessors, mirroring the `Func<>` the same block already uses for the coordinator proxy. Pin must build the graph in starter order (a Host.IntegrationTests boot of an admin-role host), not merely activate the dialer |
| 3 | 1A.2 | **Fan parked the actor mailbox** (self-caught by the implementer): dialing from the receive ran `TryGetPsk`'s cold-cache 3 s block pre-await (cold after EVERY refresh due to `InvalidateAll`), stalling deployment acks behind the fan | async methods run synchronously to the first await | `Task.Run` hop (token None — a canceled scheduling would read as a DB failure via PipeTo); pinned red-first by `A_slow_dial_does_not_park_the_actors_mailbox` |

## Carry-forward register

Checks a gate could not drive, named + carried to the next gate (proportionate-gate rule).

| Item | Deferred from | Proven instead by | Closed in |
|---|---|---|---|
| Hang-style central failover drill (`docker pause`, not `docker stop` — only a hang discriminates the per-endpoint-deadline fix) | 1B.2 review | unit fact `Each_endpoint_gets_a_FRESH_deadline` | 1B.3 gate (MUST include) |
| I-4 measurement: ack burst within ~3 s after a contact-refresh tick (cold PSK cache on the inbound auth path) | 1A.2 review | bounded 3 s store read + `Lazy` stampede suppression | 1B.3 gate (observe) |
| `TwoNodeClusterHarness` logs the I-1 null-provider Error at every spawn (harness registers no `IMeshPskProvider`) — EXPECTED, not a rig fault | 1A.2 re-review | production pairing is `hasAdmin`-unconditional in one block | note for 1B.3/Phase 2 readers |
| Register item 3 (DI role-scoping pin: admin resolves provider+dialer, driver-only does NOT) — container-activation test exists, role-scoping pin outstanding | 1A.2 | `It_activates_from_the_container_despite_the_optional_client_factory_parameter`; starter-order half now covered by `ControlPlaneStarterOrderTests` | Phase 2 (or 1B.3) — note defect 4 shows the missing pin is not merely role-scoping but **starter-order**: the container-activation test resolves from a collection without `AddZbSecrets` |
| `Resolve`-failure Error wording says "continuing without it" (reads permanent — the memoize heals it at first use) and the line fires on real boots (2/3 in the gate) — reword or whitelist it before the Phase 2 soak error-grep, else a healthy central scores as failing the soak | 1A.3 re-run | heal proven live: singleton moved onto the race-losing node, fan clean, no dialer-missing error | Phase 2 (soak grep design) |
| A repaired `GrpcPort` takes effect only at the next contact refresh (≤60 s): a deploy 5 s post-restore was still skipped and honestly TimedOut — by design, operator-visible; document if it surprises in Phase 2 | 1A.3 re-run | check 7 evidence | Phase 2 (observe) |
| I-4 (cold PSK cache on the inbound auth path after each refresh tick) is structurally unmeasurable on docker-dev — `ClusterPsks` config map answers ahead of the store, so the cache is always empty and `InvalidateAll` clears nothing | 1B.3 gate | every ack landed within the posting second | Phase 2+ (needs a store-backed rig) |
| After a central restart, a node's cached `GrpcChannel` can sit in transient failure ~2 min (client backoff; TCP open, sibling reaches the same endpoint) — no ack lost, the walk absorbs it loudly. Expect a failover-line burst after any central restart in the Phase 2 soak | 1B.3 gate | 3 dispatches: 2 failed over, 3rd direct | Phase 2 (soak expectations) |
| **The PreStart race fires on essentially EVERY admin boot** (4/4 in the 1B gate, 2/3 in 1A) and logs two FALSE Errors ("continuing without it", "gate will NEVER be published") that the memoize immediately heals — the wording fix is now a MUST before the Phase 2 soak error-grep, not a nicety | 1B.3 gate obs (b) | gate published + acks accepted on every one of those boots | Phase 2 opening task |
| `RevisionHash` moved on byte-identical config after a direct SQL `UPDATE` bumped `Script.RowVersion` (hash covers a column content-identity does not); determinism itself holds (3 no-change dispatches → identical hash) | 1A.3 re-run | incidental, out of program scope | note only |
| 1A checks 1/2/7 (deploy fan over gRPC, mixed-transport seal, NULL-`GrpcPort` skip) and the central half of 4 | 1A.3 gate run 1 | — | **CLOSED in 1A.3 run 2** — all driven live and PASS |
| Check 8 is vacuous for telemetry + config-serve — central never booted, so neither plane carried a call | 1A.3 gate run 1 | — | **CLOSED in 1A.3 run 2** — both planes carried authenticated traffic (28 KB artifact fetches from `:4055`, telemetry streams open on all six) with 0 denials |
| **`DriverStatusPanel.razor` is referenced by no page or component** — the Reconnect/Restart buttons are unreachable in the running AdminUI (pre-existing on `master`, not caused by this program; re-verified in run 2: 0 `<DriverStatusPanel` usages, no Restart/Reconnect control on any rendered page, no such HTTP endpoint) | 1A.3 gate | Check 3 **closed by composition**: run 1's live node-side leg for both commands + run 2's check 4 driving the identical `AdminOperationsActor.Dispatch` → `DispatchOverGrpc` origin path from the AdminUI | Out of program scope — file separately if the control is meant to exist |
| `Failed to resolve` Error text reads as permanent though first use retries, and will trip a soak error-grep on a healthy node | 1A.3 gate run 2 (defect 5) | Self-heal proven live | Phase 2 |

## Issues filed by this program

| Issue | What | Status |
|---|---|---|
| [lmxopcua#542](https://gitea.dohertylan.com/dohertj2/lmxopcua/issues/542) | `FleetStatusBroadcaster` derives "fleet" from central's own gossip ring — silently central-pair-blind since mesh Phase 6; pre-existing, transport-independent, out of this program's scope | FILED 2026-08-06 |
| [lmxopcua#544](https://gitea.dohertylan.com/dohertj2/lmxopcua/issues/544) | `ConfigPublishCoordinatorTimeoutTests.DeadlineElapsed_for_current_deployment_marks_TimedOut` is racy on its own terms (150 ms wall-clock deadline; row inserted after spawn) — flaked 2/~11 full runs once 1A.2 grew the suite; green at base. Fix the test (inject clock / raise deadline) | FILED 2026-08-06 (1A.3) |
| [lmxopcua#545](https://gitea.dohertylan.com/dohertj2/lmxopcua/issues/545) | Mesh command/ack planes are h2c-only — the per-cluster PSK crosses the network in cleartext on a write surface (precedent: every sibling gRPC plane; trusted-site-link assumption). Decide: accepted risk, or a `UseTls`/scheme option | FILED 2026-08-06 (1A.3) |
| [lmxopcua#546](https://gitea.dohertylan.com/dohertj2/lmxopcua/issues/546) | AdminUI `DriverStatusPanel` (the only Restart/Reconnect control) is referenced by no page — the commands have no reachable UI; pre-existing, found by the 1A gate | FILED 2026-08-06 |
| [lmxopcua#543](https://gitea.dohertylan.com/dohertj2/lmxopcua/issues/543) | The three pre-existing auth interceptors (`TelemetryStream`/`ConfigServe`/`LocalDbSync`) treat a whitespace-only key as SET (`IsNullOrEmpty`), so a whitespace `Telemetry:ApiKey` etc. opens the endpoint to a whitespace bearer; the two new mesh interceptors use `IsNullOrWhiteSpace`. Backport is a behaviour change to shipped endpoints — own issue, out of this program's scope | FILED 2026-08-06 |

## Known constraints carried from the research (do not rediscover)

- No Ask crosses the boundary — 4+1 one-way Tells is the whole surface (design §1).
- `ConfigPublishCoordinator`'s unconditional `deployment-acks` DPS subscription is the
  `Dps`-mode ingress — keep it (design §4.4).
- Interceptors: exactly one public constructor, reflection-pinned (gotcha 1).
- Proto enum 0 = `UNSPECIFIED`; mapper goldens must cover every member incl. null variants
  (gotcha 2, plan T0.2).
- Kestrel block untouchable — no new listeners in this program (gotcha 3).
- `x-otopcua-cluster` is a central-side routing hint; isolation is proven by wrong-key
  rejection (gotcha 9).