lmxopcua/docs/plans/2026-08-06-clusterclient-to-grpc-tracking.md

# ClusterClient → gRPC migration — CENTRAL TRACKING DOCUMENT

**This is the single place to see program state.** Update it at every phase boundary, merge,
defect, and decision. Everything else (design, plan, live gate) is referenced from here.

**Program goal:** all central↔site mesh traffic on gRPC with per-cluster PSK auth;
`ClusterClient`/`ClusterClientReceptionist` deleted; Akka remoting never crosses the mesh
boundary. Mirrors ScadaBridge's completed 2026-07-22 migration (same cluster boxes, same end
posture).

## Document map

| Doc | Role |
|---|---|
| [`2026-08-06-clusterclient-to-grpc-design.md`](2026-08-06-clusterclient-to-grpc-design.md) | Design: inventory, rationale, decisions (§3), plan-vs-code corrections (§4), steelman (§5) |
| [`2026-08-06-clusterclient-to-grpc.md`](2026-08-06-clusterclient-to-grpc.md) | Implementation plan: phases 0/1A/1B/2/3/4, task metadata, Opus-subagent execution rules, gotchas |
| [`2026-08-06-clusterclient-to-grpc.md.tasks.json`](2026-08-06-clusterclient-to-grpc.md.tasks.json) | Machine task tracker (executing-plans resume point) |
| [`2026-08-06-clusterclient-to-grpc-live-gate.md`](2026-08-06-clusterclient-to-grpc-live-gate.md) | Live gate record: baseline reproduction + per-phase check tables |
| ScadaBridge reference | `~/Desktop/scadaproj/scadabridge_clusterclient_to_grpc.md` + `~/Desktop/ScadaBridge/docs/plans/2026-07-22-clusterclient-to-grpc-{plan,live-gate}.md` |

## Phase status board

| Phase | Scope | Branch | Status | PR / merge SHA | Live gate |
|---|---|---|---|---|---|
| 0 | Proto + mappers + goldens; both auth interceptors; `IMeshPskProvider`; options/validators (+`Grpc` value, SplitTopology relaxation); rig dark config; stale-comment fixes; FleetStatus issue filed | `feat/grpc-phase0-contract-auth` | **IN PROGRESS** — Batch A (0.1 + 0.3) @ `aa21f61f`; 0.2 (mapper + 27 wire-level goldens) @ `545413a8`; Batch B (0.4 + 0.5) @ `fbda015c`/`68a416e4` + review fixes `24e55d33`/`a4631410`; all reviews spec ✅ + code ✅ (Batch B code review on Opus, fix loop closed); only 0.6 remains | — | PENDING |
| 1A | Central→node command plane (`NodeCommandService` on the Telemetry listener; `MeshGrpcCommandDialer`; third mode branch in `CentralCommunicationActor`) — **worktree A** | `feat/grpc-node-command` | NOT STARTED | — | PENDING |
| 1B | Node→central ack plane (`CentralCommandService` on the ConfigServe listener; `GrpcApplyAckSender`; third branch in `NodeCommunicationActor`) — **worktree B**, rebases on 1A | `feat/grpc-central-ack` | NOT STARTED | — | PENDING |
| 2 | Baseline reproduction on pre-flip build, then full-rig cutover (`OTOPCUA_MESH_MODE` default → `Grpc`) + soak + kill drills | (rig commit) | NOT STARTED | — | PENDING |
| 3 | Delete ClusterClient (code/HOCON/config/tests); SplitTopology requires `Grpc`; grep-gates; docs sweep incl. scadaproj index | `feat/grpc-phase3-deletion` | NOT STARTED | — | — |
| 4 | Terminal live gate on the post-deletion image (8 checks, provenance-guarded) | (on `master`) | NOT STARTED | — | PENDING |

**Execution rules recap:** phase order 0 → (1A ∥ 1B) → 2 → 3 → 4; 1A∥1B only in separate git
worktrees, one Opus subagent per worktree, sequential tasks within a worktree; 1A merges
first, 1B rebases (union-resolve `Host/Program.cs` map block, compose env,
`docs/Configuration.md`); rig mode flips before Phase 2 are never committed; **STOP at every
phase DoD — PR + merge are user-owned.**

## Decision log

| Date | Decision | Where recorded |
|---|---|---|
| 2026-08-06 | Reuse existing listeners (Telemetry `:4056` node-side, ConfigServe `:4055` central-side); rejected dedicated port + `ClusterNode` column | design §3.1 |
| 2026-08-06 | Per-cluster PSK `OT-MESH-PSK-<clusterId>` from Secrets; rejected fleet-wide shared key (blast radius); `MeshTransport:ClusterPsks` config map as rig/test override | design §3.4, plan T0.3/T0.5 |
| 2026-08-06 | Fire-and-forget semantics preserved exactly — no retries/queues; failures become visible logs, backstops (2-min deadline, boot replay) unchanged | design §3.3 |
| 2026-08-06 | `Dps` mode survives for single-mesh topologies; only `ClusterClient` is deleted; the three stale "Phase 6 deletes DPS" comments get corrected, not obeyed | design §3.6/§4.2 |
| 2026-08-06 | No transport interface — third mode branch inside the two choke-point actors (ScadaBridge §7.1 economy) | design §4.3 |
| 2026-08-06 | Coexistence granularity: central's mode governs commands, each node's mode governs its acks (per-pair flips possible — better than ScadaBridge's central-wide flag) | design §3.6 |
| 2026-08-06 | `AlarmCommandMessage.operation` is a proto `string`, NOT an enum — the CLR `AlarmCommand.Operation` is an unconstrained string, and an enum would silently decode unknown operations to `UNSPECIFIED` (the drop class this migration removes). Plan T0.1's "AlarmCommand verb" enum wording was a plan defect; only `ApplyOutcome` carries the enum-0 rule. T0.2's goldens must pin pass-through of an unrecognized operation string | proto comment + this log (T0.1 review) |
| 2026-08-06 | `correlation_id` lives on the envelope AND as inner fields on the three commands whose CLR record carries one — each command round-trips independently in the T0.2 mapper; the envelope covers `AlarmCommand` (no CLR correlation member). For the three, the inner field is authoritative on decode | proto comment + this log (T0.1 review) |

## Phase 1A security register (from the Batch B Opus code review — HARD GATE: all four close BEFORE `CentralCommandService` is mapped)

1. **Known-cluster-set membership gate in `TryGetPsk`** — central rebuilds its cluster set from
   `ClusterNode` rows every `ContactRefreshSeconds`; refuse a cluster id outside that set in
   O(1) with no store touch. Closes BOTH open review findings at once: (a) the timing oracle
   that currently defeats the uniform-denial anti-enumeration text (unknown cluster pays a
   store read on every probe because negatives are deliberately never cached; known cluster is
   cached — a stable remotely-measurable distinguisher), and (b) the pre-auth resource
   exhaustion / audit-flood vector (each distinct forged `x-otopcua-cluster` header costs a
   secret-store read + a `secret.resolve.not-found` audit row into `ConfigAuditLog` via the
   audit singleton, attacker-chosen `Target` values, before any credential check). Namespace
   confinement (`OT-MESH-PSK-` prefix + `SecretName` allow-list) means no disclosure risk —
   this is flood/oracle only.
2. **`bool Matches(string clusterId, string presented)` on `IMeshPskProvider`** — keep
   `FixedTimeEquals` inside the provider so a future verify-side caller cannot write `==` and
   have it compile. Do before 1B adds the dial-side caller.
3. **DI-shape pin test** — `IMeshPskProvider` is `hasAdmin`-only and its pairing with the
   central interceptor is by convention; a driver-only node injecting it throws on first call,
   not at boot. Add a Host.IntegrationTests check (admin host resolves it; driver-only host
   does NOT) when Phase 1 maps the services.
4. **Integration gate over real Kestrel h2c** — both new interceptors have unit coverage only;
   the one-public-ctor reflection test approximates but cannot prove the interceptor is
   actually invoked. Wire-probe in the 1A/1B live gates (already in the live-gate tables).

Non-gating notes from the same review: central-side unseeded-PSK is invisible until a deploy
stalls (consider a "PSK seeded for every known cluster?" `/hosts`/health surface — the
#521/#541 shape); `SecretName` lowercases + bans `_`, so a cluster id like `Site_A` can never
resolve from the store and must use `ClusterPsks` — MUST be documented in T0.6's
`docs/Configuration.md` stub (rig ids `MAIN`/`SITE-A`/`SITE-B` are fine); sibling interceptors
still use `IsNullOrEmpty` (whitespace key opens them) — family-wide backport is out of this
program's scope, file alongside the FleetStatus issue in T0.6; consider a non-throwing counter
metric if the poison-guard silent catch ever fires in production.

## Phase 1 dispatch notes (from Task 0.2 reviews — fold into the 1A/1B implementer prompts)

- The alarm arm of `FromEnvelope` necessarily discards the envelope correlation
  (`AlarmCommand` has no CLR member for it) — the 1A router must read
  `envelope.CorrelationId` itself for logging.
- `ApplyAck` is the ONE record of the five that never rides `CommandEnvelope`:
  `GrpcApplyAckSender` (1B) calls `MeshTransportDtoMapper.ToProto(ApplyAck)` directly on
  `ReportApply` — do not expect (or add) a sixth `ToEnvelope` overload.
- Consider a thin try/catch at the `Execute`/`ReportApply` service boundary wrapping
  `FromEnvelope`/`FromProto` failures with the envelope's correlation_id (readable as a plain
  string even when inner ids fail to parse) so an operator can correlate a rejected call.
- Revisit the untyped `ToEnvelope(object, CorrelationId alarmCorrelationId)` signature when
  the 1A router lands — if the router prefers one always-honoured transport correlation,
  change the signature then, before it has more callers.

## Defect log

Defects found during execution (live-gate or review), with root cause + pinned test.

| # | Phase | Defect | Root cause | Fix / pin |
|---|---|---|---|---|
| — | — | *(none yet)* | | |

## Carry-forward register

Checks a gate could not drive, named + carried to the next gate (proportionate-gate rule).

| Item | Deferred from | Proven instead by | Closed in |
|---|---|---|---|
| — | — | *(none yet)* | |

## Issues filed by this program

| Issue | What | Status |
|---|---|---|
| *(Gitea # TBD in Phase 0, T0.6)* | `FleetStatusBroadcaster` derives "fleet" from central's own gossip ring — silently central-pair-blind since mesh Phase 6; pre-existing, transport-independent, out of this program's scope | to file |
| *(Gitea # TBD in Phase 0, T0.6)* | The three pre-existing auth interceptors (`TelemetryStream`/`ConfigServe`/`LocalDbSync`) treat a whitespace-only key as SET (`IsNullOrEmpty`), so a whitespace `Telemetry:ApiKey` etc. opens the endpoint to a whitespace bearer; the two new mesh interceptors use `IsNullOrWhiteSpace`. Backport is a behaviour change to shipped endpoints — own issue, out of this program's scope | to file |

## Known constraints carried from the research (do not rediscover)

- No Ask crosses the boundary — 4+1 one-way Tells is the whole surface (design §1).
- `ConfigPublishCoordinator`'s unconditional `deployment-acks` DPS subscription is the
  `Dps`-mode ingress — keep it (design §4.4).
- Interceptors: exactly one public constructor, reflection-pinned (gotcha 1).
- Proto enum 0 = `UNSPECIFIED`; mapper goldens must cover every member incl. null variants
  (gotcha 2, plan T0.2).
- Kestrel block untouchable — no new listeners in this program (gotcha 3).
- `x-otopcua-cluster` is a central-side routing hint; isolation is proven by wrong-key
  rejection (gotcha 9).