lmxopcua/docs/plans/2026-08-06-clusterclient-to-grpc-tracking.md

# ClusterClient → gRPC migration — CENTRAL TRACKING DOCUMENT

**This is the single place to see program state.** Update it at every phase boundary, merge,
defect, and decision. Everything else (design, plan, live gate) is referenced from here.

> **PROGRAM CLOSED 2026-08-06** — all phases 0/1A/1B/2/3/4 complete and merged; the Phase 4
> terminal live gate PASSED 8/8 on `master @ 0b194003` (see the phase board and the live-gate
> doc). Sole item leaving the program open-ended: the I-4 cold-PSK-cache latency measurement,
> re-carried to the first store-backed deployment (structurally unmeasurable on docker-dev).

**Program goal:** all central↔site mesh traffic on gRPC with per-cluster PSK auth;
`ClusterClient`/`ClusterClientReceptionist` deleted; Akka remoting never crosses the mesh
boundary. Mirrors ScadaBridge's completed 2026-07-22 migration (same cluster boxes, same end
posture).

## Document map

| Doc | Role |
|---|---|
| [`2026-08-06-clusterclient-to-grpc-design.md`](2026-08-06-clusterclient-to-grpc-design.md) | Design: inventory, rationale, decisions (§3), plan-vs-code corrections (§4), steelman (§5) |
| [`2026-08-06-clusterclient-to-grpc.md`](2026-08-06-clusterclient-to-grpc.md) | Implementation plan: phases 0/1A/1B/2/3/4, task metadata, Opus-subagent execution rules, gotchas |
| [`2026-08-06-clusterclient-to-grpc.md.tasks.json`](2026-08-06-clusterclient-to-grpc.md.tasks.json) | Machine task tracker (executing-plans resume point) |
| [`2026-08-06-clusterclient-to-grpc-live-gate.md`](2026-08-06-clusterclient-to-grpc-live-gate.md) | Live gate record: baseline reproduction + per-phase check tables |
| ScadaBridge reference | `~/Desktop/scadaproj/scadabridge_clusterclient_to_grpc.md` + `~/Desktop/ScadaBridge/docs/plans/2026-07-22-clusterclient-to-grpc-{plan,live-gate}.md` |

## Phase status board

| Phase | Scope | Branch | Status | PR / merge SHA | Live gate |
|---|---|---|---|---|---|
| 0 | Proto + mappers + goldens; both auth interceptors; `IMeshPskProvider`; options/validators (+`Grpc` value, SplitTopology relaxation); rig dark config; stale-comment fixes; FleetStatus issue filed | `feat/grpc-phase0-contract-auth` | **MERGED to `master`** @ `10b6d209` (2026-08-06) — all 6 tasks complete @ `23555ed0`; every task spec ✅ + code ✅ (Batch B on Opus with a closed fix loop); issues #542 + #543 filed | `10b6d209` | **PASS 4/4** |
| 1A | Central→node command plane (`NodeCommandService` on the Telemetry listener; `MeshGrpcCommandDialer`; third mode branch in `CentralCommunicationActor`) — **worktree A** | `feat/grpc-node-command` | **COMPLETE @ `e54883a8` — gate re-run PASSED 2026-08-06.** Run 1 @ `fbf7b708` FAILED on defect 4 (no admin node could boot, mode-independent); fixed by `c0475f2d` + `e54883a8` and re-gated. 1A.1 ✅ `64774652` + 1A.2 ✅ `6a862011` (+fixes `48dc2120`/`86633f14`) all reviewed spec ✅ + code ✅. **MERGED to `master`** @ `b360239e` (2026-08-06) | `b360239e` | **PASS 8/8** (check 3's AdminUI half closed by composition — the control is structurally unreachable, see carry-forwards) |
| 1B | Node→central ack plane (`CentralCommandService` on the ConfigServe listener; `GrpcApplyAckSender`; third branch in `NodeCommunicationActor`) — **worktree B**, rebased on merged 1A | `feat/grpc-central-ack` | **MERGED to `master`** @ `7203d135` (2026-08-06, branch tip `cf229a95` incl. the phases 2-4 plan update); 1B.1 ✅ + 1B.2 ✅ (Opus: deadline Critical found+fixed) + gate **PASS 7/7** incl. the 4b hang drill (~10 s failover = exactly one AckTimeoutSeconds — live proof of the fix); at-least-once duplicate absorbed idempotently exactly as traced | `7203d135` | **PASS 7/7** |
| 2 | **COMPLETE — gate PASS 8/8 + baseline + PSK negatives** (2026-08-06, all on `master`, all review chains closed). Task 2.0 ✅ (`a94eb8a3`+`d7c9ead6`, small chain: one Important fixed — non-vacuous once-per-incarnation pin — then APPROVED; healthy racing boot = ZERO Warn/Error, proven live on both centrals 4/4 boots). Register-item-3 role-scoping pin ✅ (`2e06dfe5`+`027f0403`, review found+fixed a Critical phantom-block false pass in the scan — braceless `if (hasAdmin)` at Program.cs:146). Baseline REPRODUCED pre-flip (`1bb3a937`): cross-mesh `:4053` open unauthenticated; the site-b drop is actively suppressed into SAMPLED deadLetters INF (456/467 unlogged) with the deploy deadline as sole witness. Cutover `caf2eec6` (rig default → `Grpc`, committed). Gate: deploy e2e all-gRPC sealed 6/6; command matrix live incl. real driver restart; kill-site-a-1 → per-node WRN naming node+status, exactly 1 attempt, honest deadline, boot-replay-only recovery; kill-central-1 → ack ordered-walk failover logged per node, reconvergence 6/6; site-b-pair drill → BOTH failures logged in the dispatch second (the baseline contrast); clusterclient/receptionist lines = 0 on all six through the whole gate; 135-min soak 14/14 cycles, denied=0/unauth=0, review items explained (historian-sidecar-absent DBG noise; 14×2 known per-deploy authoring WRNs); **full suite exit 0 on the cutover SHA, run against the live rig** | `caf2eec6` (cutover) + gate records `1bb3a937`/`bcc91268`/`eaf14e01`+final | **PASS 8/8** |
| 3 | Delete ClusterClient (code/HOCON/config/tests); SplitTopology requires `Grpc`; grep-gates; docs sweep incl. scadaproj index | `feat/grpc-phase3-deletion` | **MERGED to `master` @ `2fa880b6` and pushed** (2026-08-06, user-approved; branch tip `f8a43dfe`, 11 commits off `master @ 55104526`). Phase 4 ran later the same day on the user's go-ahead — see its row. Task 3.1 ✅ high-risk chain: 7 ordered steps `0db40a5e`/`ab5b958d`/`5d7ecdf7`/`d67a7e5d`/`85ca86e9`/`af3b0c31`/`4d2dfb5b`; spec review ✅ (all KEEPs verified: `LoadContactsFromDb`, DPS+frame-size canary, `deployment-acks` DPS ingress, Kestrel block, one-ctor interceptors, no retries, proto untouched; `MeshGrpcBoundaryTests`' no-association assert verified falsifiable); code review ✅ Approved, one Important fixed (`cd713557` — `MeshPaths` full-path constants were dead-in-production + doc claimed a wire-contract role they no longer have; collapsed to `*Name` constants). Retired-mode fail-closed: `Mode=ClusterClient` now FAILS validation naming `MeshTransport:Mode` + allowed values (pinned); `SplitTopologyTransportValidator` requires exactly `Grpc`. Task 3.2 ✅ docs sweep `406b1ae5` + review-fix `8e25ed55` (Configuration/Redundancy/Telemetry/config-db-schema/CLAUDE.md; deferment.md had no mention); docs-accuracy review ✅ Approved — every config key/port/behavior claim verified against source, two pre-existing Minor stalenesses fixed (Telemetry.md Phase 5 gate status; config-db-schema Phase 6 gossip premise). scadaproj index updated on its `main` and PUSHED (`0e416ca`, merge-state-accurate: Phase 3 recorded as pending merge). Grep-gates: `CentralContactPoints` in src/tests/docker-dev = 0; `clusterclient\|receptionist` = historical/fail-closed/retirement mentions only (classified by both reviewers). **DoD full-suite: run 3 = `dotnet test` exit 0, 50/50 suites, 9,326 passed / 0 failed / 57 skipped, on `8e25ed55` (docs-only commits followed).** Runs 1–2 each had ONE unrelated single-test failure (three distinct tests across the two runs: `RouteNativeAlarmAck_on_non_primary_is_dropped`, `Rediscovery_flag_persists_on_subsequent_health_publishes`, `LoggerLifecycleTests.ConfigureLogging_DisposesPreviousLogger…`) — all in files the 55-file branch diff never touches, all 3/3 green in isolation (9/9 total): pre-existing under-load timing flakes, not Phase 3 regressions. ⚠️ Process find: run 1's "exit 0" captured `tail`'s exit code, not `dotnet test`'s (`cmd \| tail; $?` trap) — fixed for runs 2–3; **Phase 2's check-8 "full suite exit 0" used the same pipeline shape and its log did not survive, so that exit-code claim is unverifiable in hindsight** (its per-suite pass lines were observed at the time; Phase 4's terminal run on the merge SHA, with correct capture, supersedes it) | `2fa880b6` | n/a (Phase 4 is the terminal live gate) |
| 4 | Terminal live gate on the post-deletion image (8 checks, provenance-guarded) | (on `master`) | **COMPLETE — gate PASS 8/8, 2026-08-06, PROGRAM CLOSED.** Ran on `master @ 0b194003` (Phase 3 merge `2fa880b6` + the #542–#546 fix merges + docs); provenance guard PASS (image `d2a290fc5711` built 17:55:10 EDT > deletion merge 16:57:05 > last code merge `9420f8ef` 17:28:48). Full record in the live-gate doc; headline results: PSK matrix 7/7 incl. unset-key boot refusal naming `MeshTransport:GrpcPsk`; command matrix with **Restart+Reconnect driven from the real `/hosts` panel for the first time** (#546 closed the 1A composition carry-forward by direct evidence — fan to six under one correlation id, both SITE-A nodes acted, operator identity carried); ack matrix exactly-once ×6 sealed in ~30 ms; central-kill ack failover (ordered walk to central-2, one INF per node) + site-kill dispatch honesty (exactly 1 WRN, +2:00 deadline, boot-replay, no retro-ack); **mid-deploy coordinator kill: central-2 killed in the dispatch second, central-1's recovered coordinator fired the deadline at exactly +2:00 with all 5 landed acks preserved (incl. the dead node's own), zero dupes across both restarts, reconverge sealed 6/6**; boundary: exactly 3 established `:4053` associations rig-wide, all pair-internal, zero site↔central; full-rig restart → transport grep `receptionist\|clusterclient` = **0 on all six**, post-restart deploy sealed 6/6 (one non-mesh observation: the CENTRAL pair reproduced the documented guard-off simultaneous-cold-start split-brain — Phase 7 territory, opt-in `Cluster:BootstrapGuard` exists, healed by single-node rejoin; both SITE pairs re-formed clean); `MeshGrpcBoundaryTests` 3/3; **terminal full suite RC=0 direct capture, 50/50 suites, 9,333 passed / 0 failed** (supersedes Phase 2's unverifiable check-8 exit code). Whole-gate PSK-drift sweep 0 denials ×6 nodes ×4 interceptors. I-4 cold-PSK-cache re-carried out of the program (docker-dev structurally cannot measure it; first store-backed deployment measures it). No new defects. Debug override reverted; rig left on the committed compose | n/a (gate only) | **PASS 8/8** |

**Execution rules recap:** phase order 0 → (1A ∥ 1B) → 2 → 3 → 4; 1A∥1B only in separate git
worktrees, one Opus subagent per worktree, sequential tasks within a worktree; 1A merges
first, 1B rebases (union-resolve `Host/Program.cs` map block, compose env,
`docs/Configuration.md`); rig mode flips before Phase 2 are never committed; **STOP at every
phase DoD — PR + merge are user-owned.**

## Decision log

| Date | Decision | Where recorded |
|---|---|---|
| 2026-08-06 | Reuse existing listeners (Telemetry `:4056` node-side, ConfigServe `:4055` central-side); rejected dedicated port + `ClusterNode` column | design §3.1 |
| 2026-08-06 | Per-cluster PSK `OT-MESH-PSK-<clusterId>` from Secrets; rejected fleet-wide shared key (blast radius); `MeshTransport:ClusterPsks` config map as rig/test override | design §3.4, plan T0.3/T0.5 |
| 2026-08-06 | Fire-and-forget semantics preserved exactly — no retries/queues; failures become visible logs, backstops (2-min deadline, boot replay) unchanged | design §3.3 |
| 2026-08-06 | `Dps` mode survives for single-mesh topologies; only `ClusterClient` is deleted; the three stale "Phase 6 deletes DPS" comments get corrected, not obeyed | design §3.6/§4.2 |
| 2026-08-06 | No transport interface — third mode branch inside the two choke-point actors (ScadaBridge §7.1 economy) | design §4.3 |
| 2026-08-06 | Coexistence granularity: central's mode governs commands, each node's mode governs its acks (per-pair flips possible — better than ScadaBridge's central-wide flag) | design §3.6 |
| 2026-08-06 | `AlarmCommandMessage.operation` is a proto `string`, NOT an enum — the CLR `AlarmCommand.Operation` is an unconstrained string, and an enum would silently decode unknown operations to `UNSPECIFIED` (the drop class this migration removes). Plan T0.1's "AlarmCommand verb" enum wording was a plan defect; only `ApplyOutcome` carries the enum-0 rule. T0.2's goldens must pin pass-through of an unrecognized operation string | proto comment + this log (T0.1 review) |
| 2026-08-06 | **Gate follows the endpoint, not the mode** (1A.2 I-2): known-cluster set published on every contact load in all three modes, from ALL `ClusterNode` rows; fan set unchanged. Survives dark-switch rollback with nothing for 1B to remember | tracking §register refinement + actor docs |
| 2026-08-06 | `CentralCommandService` mapped by PORT, not mode-gated (1B.1): a mode flip on a running fleet must not need a central restart before acks flow; matches every sibling service | `Program.cs` comment + this log |
| 2026-08-06 | Ack deadline is PER ENDPOINT, not per walk; the walk is at-least-once on the deadline path, absorbed idempotently by the coordinator's NodeId-keyed upsert | `MeshTransportOptions` docs + defect log #1 |
| 2026-08-06 | `correlation_id` lives on the envelope AND as inner fields on the three commands whose CLR record carries one — each command round-trips independently in the T0.2 mapper; the envelope covers `AlarmCommand` (no CLR correlation member). For the three, the inner field is authoritative on decode | proto comment + this log (T0.1 review) |

## Phase 1A security register (from the Batch B Opus code review — HARD GATE: all four close BEFORE `CentralCommandService` is mapped)

1. **CLOSED (1A.2 @ `48dc2120`, refined — see below).** Known-cluster-set membership gate in `TryGetPsk` — central rebuilds its cluster set from
   `ClusterNode` rows every `ContactRefreshSeconds`; refuse a cluster id outside that set in
   O(1) with no store touch. Closes BOTH open review findings at once: (a) the timing oracle
   that currently defeats the uniform-denial anti-enumeration text (unknown cluster pays a
   store read on every probe because negatives are deliberately never cached; known cluster is
   cached — a stable remotely-measurable distinguisher), and (b) the pre-auth resource
   exhaustion / audit-flood vector (each distinct forged `x-otopcua-cluster` header costs a
   secret-store read + a `secret.resolve.not-found` audit row into `ConfigAuditLog` via the
   audit singleton, attacker-chosen `Target` values, before any credential check). Namespace
   confinement (`OT-MESH-PSK-` prefix + `SecretName` allow-list) means no disclosure risk —
   this is flood/oracle only.
2. **CLOSED (1A.2 @ `14d21496`).** `bool Matches(string clusterId, string presented)` on `IMeshPskProvider` — keep
   `FixedTimeEquals` inside the provider so a future verify-side caller cannot write `==` and
   have it compile. Do before 1B adds the dial-side caller.
3. **DI-shape pin test** — `IMeshPskProvider` is `hasAdmin`-only and its pairing with the
   central interceptor is by convention; a driver-only node injecting it throws on first call,
   not at boot. Add a Host.IntegrationTests check (admin host resolves it; driver-only host
   does NOT) when Phase 1 maps the services.
**Refinement on item 1 (decision, 2026-08-06):** the gate follows the ENDPOINT, not the mode —
published on every contact load in every transport mode (the interceptor and service are
port-mapped, not mode-gated, so a dark-switch rollback must not deactivate the gate), and
built from ALL `ClusterNode` rows (gate answers "does central know this cluster"; the fan set
stays `Enabled && !MaintenanceMode`, so gate ⊇ fan — this also dissolved the
maintenance/ack-auth coupling). Pinned by the every-mode Theory + gate-⊋-fan fact + the Dps
no-ClusterClient guard pin (`86633f14`). Residuals accepted + visible: gate is fail-open until
the first successful contact load (and stays stale if central boots with SQL down — warned
every tick); gate derives from the node table, so a `ServerCluster` with zero rows is outside
it.

**Follow-ups from the 1A.2/1B.2 Opus reviews (tracked, not blocking):** per-tick
`InvalidateAll` puts a cold-cache 3 s store read on the inbound auth path — MEASURE on the
1B.3 gate (ack burst inside ~3 s after a refresh tick), then consider diff-based
invalidation; the command plane is h2c-only so the per-cluster PSK crosses the network in
cleartext on a write surface — precedent-consistent (all sibling gRPC planes), record as
accepted risk + TLS-option issue (queued below); dialer minors (Lazy channel wrapper,
retired-endpoint eviction, gRPC deadline vs token cancellation, pre-existing
`LoadContactsFromDb` Task.Run(ct) pattern).

4. **Integration gate over real Kestrel h2c** — both new interceptors have unit coverage only;
   the one-public-ctor reflection test approximates but cannot prove the interceptor is
   actually invoked. Wire-probe in the 1A/1B live gates (already in the live-gate tables).

Non-gating notes from the same review: central-side unseeded-PSK is invisible until a deploy
stalls (consider a "PSK seeded for every known cluster?" `/hosts`/health surface — the
#521/#541 shape); `SecretName` lowercases and allows only `[a-z0-9._/-]` (underscore IS
allowed — the review's `Site_A` example was wrong, caught + corrected by the T0.6 agent
against the pinned 0.2.3 DLL), so a cluster id with any character outside that set fails
closed at the store and must use `ClusterPsks` — documented in `docs/Configuration.md`
(rig ids `MAIN`/`SITE-A`/`SITE-B` are fine); sibling interceptors
still use `IsNullOrEmpty` (whitespace key opens them) — family-wide backport is out of this
program's scope, file alongside the FleetStatus issue in T0.6; consider a non-throwing counter
metric if the poison-guard silent catch ever fires in production.

## Phase 1 dispatch notes (from Task 0.2 reviews — fold into the 1A/1B implementer prompts)

- The alarm arm of `FromEnvelope` necessarily discards the envelope correlation
  (`AlarmCommand` has no CLR member for it) — the 1A router must read
  `envelope.CorrelationId` itself for logging.
- `ApplyAck` is the ONE record of the five that never rides `CommandEnvelope`:
  `GrpcApplyAckSender` (1B) calls `MeshTransportDtoMapper.ToProto(ApplyAck)` directly on
  `ReportApply` — do not expect (or add) a sixth `ToEnvelope` overload.
- Consider a thin try/catch at the `Execute`/`ReportApply` service boundary wrapping
  `FromEnvelope`/`FromProto` failures with the envelope's correlation_id (readable as a plain
  string even when inner ids fail to parse) so an operator can correlate a rejected call.
- Revisit the untyped `ToEnvelope(object, CorrelationId alarmCorrelationId)` signature when
  the 1A router lands — if the router prefers one always-honoured transport correlation,
  change the signature then, before it has more callers.

## Defect log

Defects found during execution (live-gate or review), with root cause + pinned test.

| # | Phase | Defect | Root cause | Fix / pin |
|---|---|---|---|---|
| 1 | 1B.2 | **Failover silently defeated on a hanging central** (CRITICAL, caught by Opus code review): the ack deadline was one absolute `DateTime` shared across the endpoint walk — a hung (not refused) central-1 burned the whole budget and central-2 was dialed with an expired deadline | deadline hoisted outside the `foreach`; a kill-style drill can't see it (refused = fast-fail), only a hang discriminates | `e54711a2`; pinned by `Each_endpoint_gets_a_FRESH_deadline` (mutation-verified); 1B.3 gains a `docker pause` hang drill |
| 2 | 1A.2 | **Dark-switch rollback deactivated the known-cluster gate** (Opus review, I-2): gate published only under `Mode=Grpc` while the interceptor + service are port-mapped in every mode — a rollback to ClusterClient/Dps would silently restore the timing oracle + pre-auth flood | gate activation coupled to transport mode instead of the endpoint it protects | `48dc2120` — gate follows the endpoint; pinned by the every-mode Theory + Dps guard pin `86633f14` |
| 5 | 1A.3 (live gate, run 2) | **The `PreStart` resolve loses its race on real boots** — 2 of 3 observed admin boots logged `Failed to resolve IMeshCommandDialer` / `IMeshPskProvider`, same `AuditWriterActorKey` cause. NOT a defect: it is exactly the case the review-prescribed hardening was written for | `PreStart` runs on a dispatcher thread concurrently with the builder's remaining starters, so it can still precede starter #4 | **Closed by `e54883a8`, and the heal is PROVEN LIVE** — the coordinator singleton was moved onto the node that lost the race (stop central-2) and it fanned a dispatch to all five live nodes with no dialer error. Residual is cosmetic (Error text reads permanent; will trip a soak error-grep on a healthy node) → Phase 2 |
| 4 | 1A.3 (live gate, run 1) | **An admin node cannot boot** (BLOCKED 1A): both central nodes die at startup with `MissingActorRegistryEntryException: ZB.MOM.WW.OtOpcUa.ControlPlane.AuditWriterActorKey`, taking `AkkaHostedService` and the whole host with them. Mode-INDEPENDENT — reproduced on the same image with `MeshTransport__Mode=ClusterClient`, so it is not the gate's flip. Driver-only nodes are unaffected (they register neither `IMeshPskProvider` nor `ActorAuditWriter`) | `WithOtOpcUaControlPlaneSingletons`'s first `WithActors` starter eagerly resolves `IMeshCommandDialer` + `IMeshPskProvider` to pass into `CentralCommunicationActor.Props`. That graph is dialer → `IMeshPskProvider` → `ISecretResolver` → (`AddZbSecrets`) → `IAuditWriter` → `ActorAuditWriter`, whose ctor calls `registry.Get<AuditWriterActorKey>()` — and `WithSingleton<AuditWriterActorKey>` is registered ~60 lines later in the same builder. Same class as scadaproj#1 and mesh Phase 6's `IClusterRoleInfo` crash: only visible with the DI graph built in a real host, which is why the container-activation test (no `AddZbSecrets` in its collection) and `TwoNodeClusterHarness` (no provider at all) both miss it. Aggravating: the failed process does not exit, so `docker ps` reports `Up restarts=0` while the node serves no HTTP | **FIXED** — closed in 1A by `c0475f2d` (lazy resolve of dialer + PSK provider — central boots again) + `e54883a8` (memoize on first use; a lost `PreStart` race self-heals at the first refresh, review-prescribed), merged in Phase 1A @ `b360239e`; gate re-run PASSED 8/8 (matches the Phase 1A board row above). Original suggested shape: hand the actor lazy accessors, mirroring the `Func<>` the same block already uses for the coordinator proxy |
| 3 | 1A.2 | **Fan parked the actor mailbox** (self-caught by the implementer): dialing from the receive ran `TryGetPsk`'s cold-cache 3 s block pre-await (cold after EVERY refresh due to `InvalidateAll`), stalling deployment acks behind the fan | async methods run synchronously to the first await | `Task.Run` hop (token None — a canceled scheduling would read as a DB failure via PipeTo); pinned red-first by `A_slow_dial_does_not_park_the_actors_mailbox` |

## Carry-forward register

Checks a gate could not drive, named + carried to the next gate (proportionate-gate rule).

| Item | Deferred from | Proven instead by | Closed in |
|---|---|---|---|
| Hang-style central failover drill (`docker pause`, not `docker stop` — only a hang discriminates the per-endpoint-deadline fix) | 1B.2 review | unit fact `Each_endpoint_gets_a_FRESH_deadline` | 1B.3 gate (MUST include) |
| I-4 measurement: ack burst within ~3 s after a contact-refresh tick (cold PSK cache on the inbound auth path) | 1A.2 review | bounded 3 s store read + `Lazy` stampede suppression | 1B.3 gate (observe) |
| `TwoNodeClusterHarness` logs the I-1 null-provider Error at every spawn (harness registers no `IMeshPskProvider`) — EXPECTED, not a rig fault | 1A.2 re-review | production pairing is `hasAdmin`-unconditional in one block | note for 1B.3/Phase 2 readers |
| Register item 3 (DI role-scoping pin: admin resolves provider+dialer, driver-only does NOT) — container-activation test exists, role-scoping pin outstanding | 1A.2 | `It_activates_from_the_container_despite_the_optional_client_factory_parameter`; starter-order half now covered by `ControlPlaneStarterOrderTests` | **CLOSED in Phase 2** (`2e06dfe5`+`027f0403`): source-scan pin in Host.Tests, review-hardened — a Critical phantom-block false pass (braceless `if (hasAdmin)` at Program.cs:146 made the scanner adopt the `if (hasDriver)` block's brace) was found and fixed, both bite-checks proven |
| `Resolve`-failure Error wording says "continuing without it" (reads permanent — the memoize heals it at first use) and the line fires on real boots (2/3 in the gate) — reword or whitelist it before the Phase 2 soak error-grep, else a healthy central scores as failing the soak | 1A.3 re-run | heal proven live: singleton moved onto the race-losing node, fan clean, no dialer-missing error | **CLOSED by Task 2.0** (`a94eb8a3`+`d7c9ead6`): PreStart resolve logs Debug-only; Error moved to first-use sites (per-dispatch for the dialer, once-per-incarnation for the PSK provider); healthy racing boot = 0 Warn/Error, pinned + proven live (soak grep ran clean with NO whitelist entry for this path) |
| A repaired `GrpcPort` takes effect only at the next contact refresh (≤60 s): a deploy 5 s post-restore was still skipped and honestly TimedOut — by design, operator-visible; document if it surprises in Phase 2 | 1A.3 re-run | check 7 evidence | Phase 2: not exercised (no GrpcPort repair drill was in the check table) and nothing surprised; behaviour unchanged by design — note stands for operators |
| I-4 (cold PSK cache on the inbound auth path after each refresh tick) is structurally unmeasurable on docker-dev — `ClusterPsks` config map answers ahead of the store, so the cache is always empty and `InvalidateAll` clears nothing | 1B.3 gate | every ack landed within the posting second | **Re-carried OUT of the program at Phase 4** (2026-08-06): docker-dev still configures `ClusterPsks`, so the cache never holds a store-resolved value and the latency cannot exist on this rig. Every ack in every gate landed within its posting second. To be measured once on the first store-backed deployment; cold-cache fail-closed behavior is pinned by unit tests |
| After a central restart, a node's cached `GrpcChannel` can sit in transient failure ~2 min (client backoff; TCP open, sibling reaches the same endpoint) — no ack lost, the walk absorbs it loudly. Expect a failover-line burst after any central restart in the Phase 2 soak | 1B.3 gate | 3 dispatches: 2 failed over, 3rd direct | **Phase 2 OBSERVED, on the command dial**: after the check-3 site-a-1 kill, central-2's cached channel dropped the check-4 dispatch to the already-healthy site-a-1 (loud, single-attempt, honest deadline) ~3 min post-kill; healed by the next deploy which sealed 6/6. **Phase 4 confirmed three more occurrences** (post-central-1-restart re-deploy; central-2→site-a-1 skip after the drill-B kill) — always loud, never lossy beyond the honest deadline, healed by the next dispatch after the ~2-min window. Standing operator note: expect one TimedOut deploy within ~2 min of any node kill/restart |
| **The PreStart race fires on essentially EVERY admin boot** (4/4 in the 1B gate, 2/3 in 1A) and logs two FALSE Errors ("continuing without it", "gate will NEVER be published") that the memoize immediately heals — the wording fix is now a MUST before the Phase 2 soak error-grep, not a nicety | 1B.3 gate obs (b) | gate published + acks accepted on every one of those boots | **CLOSED — Task 2.0 shipped as the Phase 2 opening task** (`a94eb8a3`+`d7c9ead6`) and verified live: both centrals booted with zero resolve-path Warn/Error on the cutover rig |
| `RevisionHash` moved on byte-identical config after a direct SQL `UPDATE` bumped `Script.RowVersion` (hash covers a column content-identity does not); determinism itself holds (3 no-change dispatches → identical hash) | 1A.3 re-run | incidental, out of program scope | note only |
| 1A checks 1/2/7 (deploy fan over gRPC, mixed-transport seal, NULL-`GrpcPort` skip) and the central half of 4 | 1A.3 gate run 1 | — | **CLOSED in 1A.3 run 2** — all driven live and PASS |
| Check 8 is vacuous for telemetry + config-serve — central never booted, so neither plane carried a call | 1A.3 gate run 1 | — | **CLOSED in 1A.3 run 2** — both planes carried authenticated traffic (28 KB artifact fetches from `:4055`, telemetry streams open on all six) with 0 denials |
| **`DriverStatusPanel.razor` is referenced by no page or component** — the Reconnect/Restart buttons are unreachable in the running AdminUI (pre-existing on `master`, not caused by this program; re-verified in run 2: 0 `<DriverStatusPanel` usages, no Restart/Reconnect control on any rendered page, no such HTTP endpoint) | 1A.3 gate | Check 3 **closed by composition**: run 1's live node-side leg for both commands + run 2's check 4 driving the identical `AdminOperationsActor.Dispatch` → `DispatchOverGrpc` origin path from the AdminUI | Filed as **#546**, FIXED + merged (`9420f8ef`) — panel wired into `/hosts`. **Phase 4 then drove Restart + Reconnect from the real page live** (fan to six, both SITE-A nodes acted, operator identity carried), converting the composition closure into direct evidence |
| `Failed to resolve` Error text reads as permanent though first use retries, and will trip a soak error-grep on a healthy node | 1A.3 gate run 2 (defect 5) | Self-heal proven live | Phase 2 |

## Issues filed by this program

| Issue | What | Status |
|---|---|---|
| [lmxopcua#542](https://gitea.dohertylan.com/dohertj2/lmxopcua/issues/542) | `FleetStatusBroadcaster` derives "fleet" from central's own gossip ring — silently central-pair-blind since mesh Phase 6; pre-existing, transport-independent, out of this program's scope | **CLOSED 2026-08-06** — resolved by DELETION (merge `d01b717e`): scoping found the whole chain dead end-to-end (hub had zero clients; heartbeat had no production sender), so it was deleted per the #522/#507 precedent rather than rewired; `/fleet` + Phase 5 telemetry are the real cross-mesh-correct sources |
| [lmxopcua#544](https://gitea.dohertylan.com/dohertj2/lmxopcua/issues/544) | `ConfigPublishCoordinatorTimeoutTests.DeadlineElapsed_for_current_deployment_marks_TimedOut` is racy on its own terms (150 ms wall-clock deadline; row inserted after spawn) — flaked 2/~11 full runs once 1A.2 grew the suite; green at base. Fix the test (inject clock / raise deadline) | **CLOSED 2026-08-06** — fixed test-only (merge `1df84703`, commit `722a222d`): row seeded before spawn + 5-min un-hittable deadline; 5× consecutive green |
| [lmxopcua#545](https://gitea.dohertylan.com/dohertj2/lmxopcua/issues/545) | Mesh command/ack planes are h2c-only — the per-cluster PSK crosses the network in cleartext on a write surface (precedent: every sibling gRPC plane; trusted-site-link assumption). Decide: accepted risk, or a `UseTls`/scheme option | **CLOSED 2026-08-06** — option 1, accepted + recorded once for all five planes (merge `1df84703`, commit `17a822b7`): new `docs/security.md` §"Internal gRPC planes (h2c)"; UseTls remains recorded future work, all-five-together |
| [lmxopcua#546](https://gitea.dohertylan.com/dohertj2/lmxopcua/issues/546) | AdminUI `DriverStatusPanel` (the only Restart/Reconnect control) is referenced by no page — the commands have no reachable UI; pre-existing, found by the 1A gate | **CLOSED 2026-08-06** — wired into `/hosts` as a collapsed per-driver Controls expander (merge `9420f8ef`); source-scan pin against re-orphaning; live-verified (browser Reconnect → both SITE-A nodes reconnected `SITEA-modbus`) |
| [lmxopcua#543](https://gitea.dohertylan.com/dohertj2/lmxopcua/issues/543) | The three pre-existing auth interceptors (`TelemetryStream`/`ConfigServe`/`LocalDbSync`) treat a whitespace-only key as SET (`IsNullOrEmpty`), so a whitespace `Telemetry:ApiKey` etc. opens the endpoint to a whitespace bearer; the two new mesh interceptors use `IsNullOrWhiteSpace`. Backport is a behaviour change to shipped endpoints — own issue, out of this program's scope | **CLOSED 2026-08-06** — backported (merge `1df84703`, commit `f10f4c37`), red-first verified, plus the two missing one-public-ctor pins; live-gated on docker-dev: whitespace-configured key denies a whitespace bearer AND a real key (fail-closed), real-key positive path restored after revert |

All five issues above were fixed, review-chained, merged to `master` @ `9420f8ef` and pushed,
live-gated on docker-dev, and closed on 2026-08-06 (full suite on the merge SHA: exit 0, 50/50
suites, 9,333 passed / 0 failed). Side-find filed as
[lmxopcua#547](https://gitea.dohertylan.com/dohertj2/lmxopcua/issues/547) (stale pre-#520 authz
claims in `docs/security.md`'s summary).

## Known constraints carried from the research (do not rediscover)

- No Ask crosses the boundary — 4+1 one-way Tells is the whole surface (design §1).
- `ConfigPublishCoordinator`'s unconditional `deployment-acks` DPS subscription is the
  `Dps`-mode ingress — keep it (design §4.4).
- Interceptors: exactly one public constructor, reflection-pinned (gotcha 1).
- Proto enum 0 = `UNSPECIFIED`; mapper goldens must cover every member incl. null variants
  (gotcha 2, plan T0.2).
- Kestrel block untouchable — no new listeners in this program (gotcha 3).
- `x-otopcua-cluster` is a central-side routing hint; isolation is proven by wrong-key
  rejection (gotcha 9).