lmxopcua/docs/plans/2026-08-06-clusterclient-to-grpc-tracking.md

ClusterClient → gRPC migration — CENTRAL TRACKING DOCUMENT

This is the single place to see program state. Update it at every phase boundary, merge, defect, and decision. Everything else (design, plan, live gate) is referenced from here.

PROGRAM CLOSED 2026-08-06 — all phases 0/1A/1B/2/3/4 complete and merged; the Phase 4 terminal live gate PASSED 8/8 on master @ 0b194003 (see the phase board and the live-gate doc). Sole item leaving the program open-ended: the I-4 cold-PSK-cache latency measurement, re-carried to the first store-backed deployment (structurally unmeasurable on docker-dev).

Program goal: all central↔site mesh traffic on gRPC with per-cluster PSK auth; ClusterClient/ClusterClientReceptionist deleted; Akka remoting never crosses the mesh boundary. Mirrors ScadaBridge's completed 2026-07-22 migration (same cluster boxes, same end posture).

Document map

Doc	Role
2026-08-06-clusterclient-to-grpc-design.md	Design: inventory, rationale, decisions (§3), plan-vs-code corrections (§4), steelman (§5)
2026-08-06-clusterclient-to-grpc.md	Implementation plan: phases 0/1A/1B/2/3/4, task metadata, Opus-subagent execution rules, gotchas
2026-08-06-clusterclient-to-grpc.md.tasks.json	Machine task tracker (executing-plans resume point)
2026-08-06-clusterclient-to-grpc-live-gate.md	Live gate record: baseline reproduction + per-phase check tables
ScadaBridge reference	~/Desktop/scadaproj/scadabridge_clusterclient_to_grpc.md + ~/Desktop/ScadaBridge/docs/plans/2026-07-22-clusterclient-to-grpc-{plan,live-gate}.md

Phase status board

Phase	Scope	Branch	Status	PR / merge SHA	Live gate
0	Proto + mappers + goldens; both auth interceptors; IMeshPskProvider; options/validators (+Grpc value, SplitTopology relaxation); rig dark config; stale-comment fixes; FleetStatus issue filed	feat/grpc-phase0-contract-auth	MERGED to master @ 10b6d209 (2026-08-06) — all 6 tasks complete @ 23555ed0; every task spec ✅ + code ✅ (Batch B on Opus with a closed fix loop); issues #542 + #543 filed	10b6d209	PASS 4/4
1A	Central→node command plane (NodeCommandService on the Telemetry listener; MeshGrpcCommandDialer; third mode branch in CentralCommunicationActor) — worktree A	feat/grpc-node-command	COMPLETE @ e54883a8 — gate re-run PASSED 2026-08-06. Run 1 @ fbf7b708 FAILED on defect 4 (no admin node could boot, mode-independent); fixed by c0475f2d + e54883a8 and re-gated. 1A.1 ✅ 64774652 + 1A.2 ✅ 6a862011 (+fixes 48dc2120/86633f14) all reviewed spec ✅ + code ✅. MERGED to master @ b360239e (2026-08-06)	b360239e	PASS 8/8 (check 3's AdminUI half closed by composition — the control is structurally unreachable, see carry-forwards)
1B	Node→central ack plane (CentralCommandService on the ConfigServe listener; GrpcApplyAckSender; third branch in NodeCommunicationActor) — worktree B, rebased on merged 1A	feat/grpc-central-ack	MERGED to master @ 7203d135 (2026-08-06, branch tip cf229a95 incl. the phases 2-4 plan update); 1B.1 ✅ + 1B.2 ✅ (Opus: deadline Critical found+fixed) + gate PASS 7/7 incl. the 4b hang drill (~10 s failover = exactly one AckTimeoutSeconds — live proof of the fix); at-least-once duplicate absorbed idempotently exactly as traced	7203d135	PASS 7/7
2	COMPLETE — gate PASS 8/8 + baseline + PSK negatives (2026-08-06, all on master, all review chains closed). Task 2.0 ✅ (a94eb8a3+d7c9ead6, small chain: one Important fixed — non-vacuous once-per-incarnation pin — then APPROVED; healthy racing boot = ZERO Warn/Error, proven live on both centrals 4/4 boots). Register-item-3 role-scoping pin ✅ (2e06dfe5+027f0403, review found+fixed a Critical phantom-block false pass in the scan — braceless if (hasAdmin) at Program.cs:146). Baseline REPRODUCED pre-flip (1bb3a937): cross-mesh :4053 open unauthenticated; the site-b drop is actively suppressed into SAMPLED deadLetters INF (456/467 unlogged) with the deploy deadline as sole witness. Cutover caf2eec6 (rig default → Grpc, committed). Gate: deploy e2e all-gRPC sealed 6/6; command matrix live incl. real driver restart; kill-site-a-1 → per-node WRN naming node+status, exactly 1 attempt, honest deadline, boot-replay-only recovery; kill-central-1 → ack ordered-walk failover logged per node, reconvergence 6/6; site-b-pair drill → BOTH failures logged in the dispatch second (the baseline contrast); clusterclient/receptionist lines = 0 on all six through the whole gate; 135-min soak 14/14 cycles, denied=0/unauth=0, review items explained (historian-sidecar-absent DBG noise; 14×2 known per-deploy authoring WRNs); full suite exit 0 on the cutover SHA, run against the live rig	caf2eec6 (cutover) + gate records 1bb3a937/bcc91268/eaf14e01+final	PASS 8/8
3	Delete ClusterClient (code/HOCON/config/tests); SplitTopology requires Grpc; grep-gates; docs sweep incl. scadaproj index	feat/grpc-phase3-deletion	MERGED to master @ 2fa880b6 and pushed (2026-08-06, user-approved; branch tip f8a43dfe, 11 commits off master @ 55104526). Phase 4 ran later the same day on the user's go-ahead — see its row. Task 3.1 ✅ high-risk chain: 7 ordered steps 0db40a5e/ab5b958d/5d7ecdf7/d67a7e5d/85ca86e9/af3b0c31/4d2dfb5b; spec review ✅ (all KEEPs verified: LoadContactsFromDb, DPS+frame-size canary, deployment-acks DPS ingress, Kestrel block, one-ctor interceptors, no retries, proto untouched; MeshGrpcBoundaryTests' no-association assert verified falsifiable); code review ✅ Approved, one Important fixed (cd713557 — MeshPaths full-path constants were dead-in-production + doc claimed a wire-contract role they no longer have; collapsed to *Name constants). Retired-mode fail-closed: Mode=ClusterClient now FAILS validation naming MeshTransport:Mode + allowed values (pinned); SplitTopologyTransportValidator requires exactly Grpc. Task 3.2 ✅ docs sweep 406b1ae5 + review-fix 8e25ed55 (Configuration/Redundancy/Telemetry/config-db-schema/CLAUDE.md; deferment.md had no mention); docs-accuracy review ✅ Approved — every config key/port/behavior claim verified against source, two pre-existing Minor stalenesses fixed (Telemetry.md Phase 5 gate status; config-db-schema Phase 6 gossip premise). scadaproj index updated on its main and PUSHED (0e416ca, merge-state-accurate: Phase 3 recorded as pending merge). Grep-gates: CentralContactPoints in src/tests/docker-dev = 0; clusterclient|receptionist = historical/fail-closed/retirement mentions only (classified by both reviewers). DoD full-suite: run 3 = dotnet test exit 0, 50/50 suites, 9,326 passed / 0 failed / 57 skipped, on 8e25ed55 (docs-only commits followed). Runs 1–2 each had ONE unrelated single-test failure (three distinct tests across the two runs: RouteNativeAlarmAck_on_non_primary_is_dropped, Rediscovery_flag_persists_on_subsequent_health_publishes, LoggerLifecycleTests.ConfigureLogging_DisposesPreviousLogger…) — all in files the 55-file branch diff never touches, all 3/3 green in isolation (9/9 total): pre-existing under-load timing flakes, not Phase 3 regressions. ⚠️ Process find: run 1's "exit 0" captured tail's exit code, not dotnet test's (cmd | tail; $? trap) — fixed for runs 2–3; Phase 2's check-8 "full suite exit 0" used the same pipeline shape and its log did not survive, so that exit-code claim is unverifiable in hindsight (its per-suite pass lines were observed at the time; Phase 4's terminal run on the merge SHA, with correct capture, supersedes it)	2fa880b6	n/a (Phase 4 is the terminal live gate)
4	Terminal live gate on the post-deletion image (8 checks, provenance-guarded)	(on master)	COMPLETE — gate PASS 8/8, 2026-08-06, PROGRAM CLOSED. Ran on master @ 0b194003 (Phase 3 merge 2fa880b6 + the #542–#546 fix merges + docs); provenance guard PASS (image d2a290fc5711 built 17:55:10 EDT > deletion merge 16:57:05 > last code merge 9420f8ef 17:28:48). Full record in the live-gate doc; headline results: PSK matrix 7/7 incl. unset-key boot refusal naming MeshTransport:GrpcPsk; command matrix with Restart+Reconnect driven from the real /hosts panel for the first time (#546 closed the 1A composition carry-forward by direct evidence — fan to six under one correlation id, both SITE-A nodes acted, operator identity carried); ack matrix exactly-once ×6 sealed in ~30 ms; central-kill ack failover (ordered walk to central-2, one INF per node) + site-kill dispatch honesty (exactly 1 WRN, +2:00 deadline, boot-replay, no retro-ack); mid-deploy coordinator kill: central-2 killed in the dispatch second, central-1's recovered coordinator fired the deadline at exactly +2:00 with all 5 landed acks preserved (incl. the dead node's own), zero dupes across both restarts, reconverge sealed 6/6; boundary: exactly 3 established :4053 associations rig-wide, all pair-internal, zero site↔central; full-rig restart → transport grep receptionist|clusterclient = 0 on all six, post-restart deploy sealed 6/6 (one non-mesh observation: the CENTRAL pair reproduced the documented guard-off simultaneous-cold-start split-brain — Phase 7 territory, opt-in Cluster:BootstrapGuard exists, healed by single-node rejoin; both SITE pairs re-formed clean); MeshGrpcBoundaryTests 3/3; terminal full suite RC=0 direct capture, 50/50 suites, 9,333 passed / 0 failed (supersedes Phase 2's unverifiable check-8 exit code). Whole-gate PSK-drift sweep 0 denials ×6 nodes ×4 interceptors. I-4 cold-PSK-cache re-carried out of the program (docker-dev structurally cannot measure it; first store-backed deployment measures it). No new defects. Debug override reverted; rig left on the committed compose	n/a (gate only)	PASS 8/8

Execution rules recap: phase order 0 → (1A ∥ 1B) → 2 → 3 → 4; 1A∥1B only in separate git worktrees, one Opus subagent per worktree, sequential tasks within a worktree; 1A merges first, 1B rebases (union-resolve Host/Program.cs map block, compose env, docs/Configuration.md); rig mode flips before Phase 2 are never committed; STOP at every phase DoD — PR + merge are user-owned.

Decision log

Date	Decision	Where recorded
2026-08-06	Reuse existing listeners (Telemetry :4056 node-side, ConfigServe :4055 central-side); rejected dedicated port + ClusterNode column	design §3.1
2026-08-06	Per-cluster PSK OT-MESH-PSK-<clusterId> from Secrets; rejected fleet-wide shared key (blast radius); MeshTransport:ClusterPsks config map as rig/test override	design §3.4, plan T0.3/T0.5
2026-08-06	Fire-and-forget semantics preserved exactly — no retries/queues; failures become visible logs, backstops (2-min deadline, boot replay) unchanged	design §3.3
2026-08-06	Dps mode survives for single-mesh topologies; only ClusterClient is deleted; the three stale "Phase 6 deletes DPS" comments get corrected, not obeyed	design §3.6/§4.2
2026-08-06	No transport interface — third mode branch inside the two choke-point actors (ScadaBridge §7.1 economy)	design §4.3
2026-08-06	Coexistence granularity: central's mode governs commands, each node's mode governs its acks (per-pair flips possible — better than ScadaBridge's central-wide flag)	design §3.6
2026-08-06	AlarmCommandMessage.operation is a proto string, NOT an enum — the CLR AlarmCommand.Operation is an unconstrained string, and an enum would silently decode unknown operations to UNSPECIFIED (the drop class this migration removes). Plan T0.1's "AlarmCommand verb" enum wording was a plan defect; only ApplyOutcome carries the enum-0 rule. T0.2's goldens must pin pass-through of an unrecognized operation string	proto comment + this log (T0.1 review)
2026-08-06	Gate follows the endpoint, not the mode (1A.2 I-2): known-cluster set published on every contact load in all three modes, from ALL ClusterNode rows; fan set unchanged. Survives dark-switch rollback with nothing for 1B to remember	tracking §register refinement + actor docs
2026-08-06	CentralCommandService mapped by PORT, not mode-gated (1B.1): a mode flip on a running fleet must not need a central restart before acks flow; matches every sibling service	Program.cs comment + this log
2026-08-06	Ack deadline is PER ENDPOINT, not per walk; the walk is at-least-once on the deadline path, absorbed idempotently by the coordinator's NodeId-keyed upsert	MeshTransportOptions docs + defect log #1
2026-08-06	correlation_id lives on the envelope AND as inner fields on the three commands whose CLR record carries one — each command round-trips independently in the T0.2 mapper; the envelope covers AlarmCommand (no CLR correlation member). For the three, the inner field is authoritative on decode	proto comment + this log (T0.1 review)

Phase 1A security register (from the Batch B Opus code review — HARD GATE: all four close BEFORE CentralCommandService is mapped)

1. CLOSED (1A.2 @ 48dc2120, refined — see below). Known-cluster-set membership gate in TryGetPsk — central rebuilds its cluster set from ClusterNode rows every ContactRefreshSeconds; refuse a cluster id outside that set in O(1) with no store touch. Closes BOTH open review findings at once: (a) the timing oracle that currently defeats the uniform-denial anti-enumeration text (unknown cluster pays a store read on every probe because negatives are deliberately never cached; known cluster is cached — a stable remotely-measurable distinguisher), and (b) the pre-auth resource exhaustion / audit-flood vector (each distinct forged x-otopcua-cluster header costs a secret-store read + a secret.resolve.not-found audit row into ConfigAuditLog via the audit singleton, attacker-chosen Target values, before any credential check). Namespace confinement (OT-MESH-PSK- prefix + SecretName allow-list) means no disclosure risk — this is flood/oracle only.
2. CLOSED (1A.2 @ 14d21496). bool Matches(string clusterId, string presented) on IMeshPskProvider — keep FixedTimeEquals inside the provider so a future verify-side caller cannot write == and have it compile. Do before 1B adds the dial-side caller.
3. DI-shape pin test — IMeshPskProvider is hasAdmin-only and its pairing with the central interceptor is by convention; a driver-only node injecting it throws on first call, not at boot. Add a Host.IntegrationTests check (admin host resolves it; driver-only host does NOT) when Phase 1 maps the services. Refinement on item 1 (decision, 2026-08-06): the gate follows the ENDPOINT, not the mode — published on every contact load in every transport mode (the interceptor and service are port-mapped, not mode-gated, so a dark-switch rollback must not deactivate the gate), and built from ALL ClusterNode rows (gate answers "does central know this cluster"; the fan set stays Enabled && !MaintenanceMode, so gate ⊇ fan — this also dissolved the maintenance/ack-auth coupling). Pinned by the every-mode Theory + gate-⊋-fan fact + the Dps no-ClusterClient guard pin (86633f14). Residuals accepted + visible: gate is fail-open until the first successful contact load (and stays stale if central boots with SQL down — warned every tick); gate derives from the node table, so a ServerCluster with zero rows is outside it.

Follow-ups from the 1A.2/1B.2 Opus reviews (tracked, not blocking): per-tick InvalidateAll puts a cold-cache 3 s store read on the inbound auth path — MEASURE on the 1B.3 gate (ack burst inside ~3 s after a refresh tick), then consider diff-based invalidation; the command plane is h2c-only so the per-cluster PSK crosses the network in cleartext on a write surface — precedent-consistent (all sibling gRPC planes), record as accepted risk + TLS-option issue (queued below); dialer minors (Lazy channel wrapper, retired-endpoint eviction, gRPC deadline vs token cancellation, pre-existing LoadContactsFromDb Task.Run(ct) pattern).

4. Integration gate over real Kestrel h2c — both new interceptors have unit coverage only; the one-public-ctor reflection test approximates but cannot prove the interceptor is actually invoked. Wire-probe in the 1A/1B live gates (already in the live-gate tables).

Non-gating notes from the same review: central-side unseeded-PSK is invisible until a deploy stalls (consider a "PSK seeded for every known cluster?" /hosts/health surface — the #521/#541 shape); SecretName lowercases and allows only [a-z0-9._/-] (underscore IS allowed — the review's Site_A example was wrong, caught + corrected by the T0.6 agent against the pinned 0.2.3 DLL), so a cluster id with any character outside that set fails closed at the store and must use ClusterPsks — documented in docs/Configuration.md (rig ids MAIN/SITE-A/SITE-B are fine); sibling interceptors still use IsNullOrEmpty (whitespace key opens them) — family-wide backport is out of this program's scope, file alongside the FleetStatus issue in T0.6; consider a non-throwing counter metric if the poison-guard silent catch ever fires in production.

Phase 1 dispatch notes (from Task 0.2 reviews — fold into the 1A/1B implementer prompts)

• The alarm arm of FromEnvelope necessarily discards the envelope correlation (AlarmCommand has no CLR member for it) — the 1A router must read envelope.CorrelationId itself for logging.
• ApplyAck is the ONE record of the five that never rides CommandEnvelope: GrpcApplyAckSender (1B) calls MeshTransportDtoMapper.ToProto(ApplyAck) directly on ReportApply — do not expect (or add) a sixth ToEnvelope overload.
• Consider a thin try/catch at the Execute/ReportApply service boundary wrapping FromEnvelope/FromProto failures with the envelope's correlation_id (readable as a plain string even when inner ids fail to parse) so an operator can correlate a rejected call.
• Revisit the untyped ToEnvelope(object, CorrelationId alarmCorrelationId) signature when the 1A router lands — if the router prefers one always-honoured transport correlation, change the signature then, before it has more callers.

Defect log

Defects found during execution (live-gate or review), with root cause + pinned test.

#	Phase	Defect	Root cause	Fix / pin
1	1B.2	Failover silently defeated on a hanging central (CRITICAL, caught by Opus code review): the ack deadline was one absolute DateTime shared across the endpoint walk — a hung (not refused) central-1 burned the whole budget and central-2 was dialed with an expired deadline	deadline hoisted outside the foreach; a kill-style drill can't see it (refused = fast-fail), only a hang discriminates	e54711a2; pinned by Each_endpoint_gets_a_FRESH_deadline (mutation-verified); 1B.3 gains a docker pause hang drill
2	1A.2	Dark-switch rollback deactivated the known-cluster gate (Opus review, I-2): gate published only under Mode=Grpc while the interceptor + service are port-mapped in every mode — a rollback to ClusterClient/Dps would silently restore the timing oracle + pre-auth flood	gate activation coupled to transport mode instead of the endpoint it protects	48dc2120 — gate follows the endpoint; pinned by the every-mode Theory + Dps guard pin 86633f14
5	1A.3 (live gate, run 2)	The PreStart resolve loses its race on real boots — 2 of 3 observed admin boots logged Failed to resolve IMeshCommandDialer / IMeshPskProvider, same AuditWriterActorKey cause. NOT a defect: it is exactly the case the review-prescribed hardening was written for	PreStart runs on a dispatcher thread concurrently with the builder's remaining starters, so it can still precede starter #4	Closed by e54883a8, and the heal is PROVEN LIVE — the coordinator singleton was moved onto the node that lost the race (stop central-2) and it fanned a dispatch to all five live nodes with no dialer error. Residual is cosmetic (Error text reads permanent; will trip a soak error-grep on a healthy node) → Phase 2
4	1A.3 (live gate, run 1)	An admin node cannot boot (BLOCKED 1A): both central nodes die at startup with MissingActorRegistryEntryException: ZB.MOM.WW.OtOpcUa.ControlPlane.AuditWriterActorKey, taking AkkaHostedService and the whole host with them. Mode-INDEPENDENT — reproduced on the same image with MeshTransport__Mode=ClusterClient, so it is not the gate's flip. Driver-only nodes are unaffected (they register neither IMeshPskProvider nor ActorAuditWriter)	WithOtOpcUaControlPlaneSingletons's first WithActors starter eagerly resolves IMeshCommandDialer + IMeshPskProvider to pass into CentralCommunicationActor.Props. That graph is dialer → IMeshPskProvider → ISecretResolver → (AddZbSecrets) → IAuditWriter → ActorAuditWriter, whose ctor calls registry.Get<AuditWriterActorKey>() — and WithSingleton<AuditWriterActorKey> is registered ~60 lines later in the same builder. Same class as scadaproj#1 and mesh Phase 6's IClusterRoleInfo crash: only visible with the DI graph built in a real host, which is why the container-activation test (no AddZbSecrets in its collection) and TwoNodeClusterHarness (no provider at all) both miss it. Aggravating: the failed process does not exit, so docker ps reports Up restarts=0 while the node serves no HTTP	FIXED — closed in 1A by c0475f2d (lazy resolve of dialer + PSK provider — central boots again) + e54883a8 (memoize on first use; a lost PreStart race self-heals at the first refresh, review-prescribed), merged in Phase 1A @ b360239e; gate re-run PASSED 8/8 (matches the Phase 1A board row above). Original suggested shape: hand the actor lazy accessors, mirroring the Func<> the same block already uses for the coordinator proxy
3	1A.2	Fan parked the actor mailbox (self-caught by the implementer): dialing from the receive ran TryGetPsk's cold-cache 3 s block pre-await (cold after EVERY refresh due to InvalidateAll), stalling deployment acks behind the fan	async methods run synchronously to the first await	Task.Run hop (token None — a canceled scheduling would read as a DB failure via PipeTo); pinned red-first by A_slow_dial_does_not_park_the_actors_mailbox

Carry-forward register

Checks a gate could not drive, named + carried to the next gate (proportionate-gate rule).

Item	Deferred from	Proven instead by	Closed in
Hang-style central failover drill (docker pause, not docker stop — only a hang discriminates the per-endpoint-deadline fix)	1B.2 review	unit fact Each_endpoint_gets_a_FRESH_deadline	1B.3 gate (MUST include)
I-4 measurement: ack burst within ~3 s after a contact-refresh tick (cold PSK cache on the inbound auth path)	1A.2 review	bounded 3 s store read + Lazy stampede suppression	1B.3 gate (observe)
TwoNodeClusterHarness logs the I-1 null-provider Error at every spawn (harness registers no IMeshPskProvider) — EXPECTED, not a rig fault	1A.2 re-review	production pairing is hasAdmin-unconditional in one block	note for 1B.3/Phase 2 readers
Register item 3 (DI role-scoping pin: admin resolves provider+dialer, driver-only does NOT) — container-activation test exists, role-scoping pin outstanding	1A.2	It_activates_from_the_container_despite_the_optional_client_factory_parameter; starter-order half now covered by ControlPlaneStarterOrderTests	CLOSED in Phase 2 (2e06dfe5+027f0403): source-scan pin in Host.Tests, review-hardened — a Critical phantom-block false pass (braceless if (hasAdmin) at Program.cs:146 made the scanner adopt the if (hasDriver) block's brace) was found and fixed, both bite-checks proven
Resolve-failure Error wording says "continuing without it" (reads permanent — the memoize heals it at first use) and the line fires on real boots (2/3 in the gate) — reword or whitelist it before the Phase 2 soak error-grep, else a healthy central scores as failing the soak	1A.3 re-run	heal proven live: singleton moved onto the race-losing node, fan clean, no dialer-missing error	CLOSED by Task 2.0 (a94eb8a3+d7c9ead6): PreStart resolve logs Debug-only; Error moved to first-use sites (per-dispatch for the dialer, once-per-incarnation for the PSK provider); healthy racing boot = 0 Warn/Error, pinned + proven live (soak grep ran clean with NO whitelist entry for this path)
A repaired GrpcPort takes effect only at the next contact refresh (≤60 s): a deploy 5 s post-restore was still skipped and honestly TimedOut — by design, operator-visible; document if it surprises in Phase 2	1A.3 re-run	check 7 evidence	Phase 2: not exercised (no GrpcPort repair drill was in the check table) and nothing surprised; behaviour unchanged by design — note stands for operators
I-4 (cold PSK cache on the inbound auth path after each refresh tick) is structurally unmeasurable on docker-dev — ClusterPsks config map answers ahead of the store, so the cache is always empty and InvalidateAll clears nothing	1B.3 gate	every ack landed within the posting second	Re-carried OUT of the program at Phase 4 (2026-08-06): docker-dev still configures ClusterPsks, so the cache never holds a store-resolved value and the latency cannot exist on this rig. Every ack in every gate landed within its posting second. To be measured once on the first store-backed deployment; cold-cache fail-closed behavior is pinned by unit tests
After a central restart, a node's cached GrpcChannel can sit in transient failure ~2 min (client backoff; TCP open, sibling reaches the same endpoint) — no ack lost, the walk absorbs it loudly. Expect a failover-line burst after any central restart in the Phase 2 soak	1B.3 gate	3 dispatches: 2 failed over, 3rd direct	Phase 2 OBSERVED, on the command dial: after the check-3 site-a-1 kill, central-2's cached channel dropped the check-4 dispatch to the already-healthy site-a-1 (loud, single-attempt, honest deadline) ~3 min post-kill; healed by the next deploy which sealed 6/6. Phase 4 confirmed three more occurrences (post-central-1-restart re-deploy; central-2→site-a-1 skip after the drill-B kill) — always loud, never lossy beyond the honest deadline, healed by the next dispatch after the ~2-min window. Standing operator note: expect one TimedOut deploy within ~2 min of any node kill/restart
The PreStart race fires on essentially EVERY admin boot (4/4 in the 1B gate, 2/3 in 1A) and logs two FALSE Errors ("continuing without it", "gate will NEVER be published") that the memoize immediately heals — the wording fix is now a MUST before the Phase 2 soak error-grep, not a nicety	1B.3 gate obs (b)	gate published + acks accepted on every one of those boots	CLOSED — Task 2.0 shipped as the Phase 2 opening task (a94eb8a3+d7c9ead6) and verified live: both centrals booted with zero resolve-path Warn/Error on the cutover rig
RevisionHash moved on byte-identical config after a direct SQL UPDATE bumped Script.RowVersion (hash covers a column content-identity does not); determinism itself holds (3 no-change dispatches → identical hash)	1A.3 re-run	incidental, out of program scope	note only
1A checks 1/2/7 (deploy fan over gRPC, mixed-transport seal, NULL-GrpcPort skip) and the central half of 4	1A.3 gate run 1	—	CLOSED in 1A.3 run 2 — all driven live and PASS
Check 8 is vacuous for telemetry + config-serve — central never booted, so neither plane carried a call	1A.3 gate run 1	—	CLOSED in 1A.3 run 2 — both planes carried authenticated traffic (28 KB artifact fetches from :4055, telemetry streams open on all six) with 0 denials
DriverStatusPanel.razor is referenced by no page or component — the Reconnect/Restart buttons are unreachable in the running AdminUI (pre-existing on master, not caused by this program; re-verified in run 2: 0 <DriverStatusPanel usages, no Restart/Reconnect control on any rendered page, no such HTTP endpoint)	1A.3 gate	Check 3 closed by composition: run 1's live node-side leg for both commands + run 2's check 4 driving the identical AdminOperationsActor.Dispatch → DispatchOverGrpc origin path from the AdminUI	Filed as #546, FIXED + merged (9420f8ef) — panel wired into /hosts. Phase 4 then drove Restart + Reconnect from the real page live (fan to six, both SITE-A nodes acted, operator identity carried), converting the composition closure into direct evidence
Failed to resolve Error text reads as permanent though first use retries, and will trip a soak error-grep on a healthy node	1A.3 gate run 2 (defect 5)	Self-heal proven live	Phase 2

Issues filed by this program

Issue	What	Status
lmxopcua#542	FleetStatusBroadcaster derives "fleet" from central's own gossip ring — silently central-pair-blind since mesh Phase 6; pre-existing, transport-independent, out of this program's scope	CLOSED 2026-08-06 — resolved by DELETION (merge d01b717e): scoping found the whole chain dead end-to-end (hub had zero clients; heartbeat had no production sender), so it was deleted per the #522/#507 precedent rather than rewired; /fleet + Phase 5 telemetry are the real cross-mesh-correct sources
lmxopcua#544	ConfigPublishCoordinatorTimeoutTests.DeadlineElapsed_for_current_deployment_marks_TimedOut is racy on its own terms (150 ms wall-clock deadline; row inserted after spawn) — flaked 2/~11 full runs once 1A.2 grew the suite; green at base. Fix the test (inject clock / raise deadline)	CLOSED 2026-08-06 — fixed test-only (merge 1df84703, commit 722a222d): row seeded before spawn + 5-min un-hittable deadline; 5× consecutive green
lmxopcua#545	Mesh command/ack planes are h2c-only — the per-cluster PSK crosses the network in cleartext on a write surface (precedent: every sibling gRPC plane; trusted-site-link assumption). Decide: accepted risk, or a UseTls/scheme option	CLOSED 2026-08-06 — option 1, accepted + recorded once for all five planes (merge 1df84703, commit 17a822b7): new docs/security.md §"Internal gRPC planes (h2c)"; UseTls remains recorded future work, all-five-together
lmxopcua#546	AdminUI DriverStatusPanel (the only Restart/Reconnect control) is referenced by no page — the commands have no reachable UI; pre-existing, found by the 1A gate	CLOSED 2026-08-06 — wired into /hosts as a collapsed per-driver Controls expander (merge 9420f8ef); source-scan pin against re-orphaning; live-verified (browser Reconnect → both SITE-A nodes reconnected SITEA-modbus)
lmxopcua#543	The three pre-existing auth interceptors (TelemetryStream/ConfigServe/LocalDbSync) treat a whitespace-only key as SET (IsNullOrEmpty), so a whitespace Telemetry:ApiKey etc. opens the endpoint to a whitespace bearer; the two new mesh interceptors use IsNullOrWhiteSpace. Backport is a behaviour change to shipped endpoints — own issue, out of this program's scope	CLOSED 2026-08-06 — backported (merge 1df84703, commit f10f4c37), red-first verified, plus the two missing one-public-ctor pins; live-gated on docker-dev: whitespace-configured key denies a whitespace bearer AND a real key (fail-closed), real-key positive path restored after revert

All five issues above were fixed, review-chained, merged to master @ 9420f8ef and pushed, live-gated on docker-dev, and closed on 2026-08-06 (full suite on the merge SHA: exit 0, 50/50 suites, 9,333 passed / 0 failed). Side-find filed as lmxopcua#547 (stale pre-#520 authz claims in docs/security.md's summary).

Known constraints carried from the research (do not rediscover)

• No Ask crosses the boundary — 4+1 one-way Tells is the whole surface (design §1).
• ConfigPublishCoordinator's unconditional deployment-acks DPS subscription is the Dps-mode ingress — keep it (design §4.4).
• Interceptors: exactly one public constructor, reflection-pinned (gotcha 1).
• Proto enum 0 = UNSPECIFIED; mapper goldens must cover every member incl. null variants (gotcha 2, plan T0.2).
• Kestrel block untouchable — no new listeners in this program (gotcha 3).
• x-otopcua-cluster is a central-side routing hint; isolation is proven by wrong-key rejection (gotcha 9).