# ClusterClient → gRPC migration — CENTRAL TRACKING DOCUMENT **This is the single place to see program state.** Update it at every phase boundary, merge, defect, and decision. Everything else (design, plan, live gate) is referenced from here. > **PROGRAM CLOSED 2026-08-06** — all phases 0/1A/1B/2/3/4 complete and merged; the Phase 4 > terminal live gate PASSED 8/8 on `master @ 0b194003` (see the phase board and the live-gate > doc). Sole item leaving the program open-ended: the I-4 cold-PSK-cache latency measurement, > re-carried to the first store-backed deployment (structurally unmeasurable on docker-dev). **Program goal:** all central↔site mesh traffic on gRPC with per-cluster PSK auth; `ClusterClient`/`ClusterClientReceptionist` deleted; Akka remoting never crosses the mesh boundary. Mirrors ScadaBridge's completed 2026-07-22 migration (same cluster boxes, same end posture). ## Document map | Doc | Role | |---|---| | [`2026-08-06-clusterclient-to-grpc-design.md`](2026-08-06-clusterclient-to-grpc-design.md) | Design: inventory, rationale, decisions (§3), plan-vs-code corrections (§4), steelman (§5) | | [`2026-08-06-clusterclient-to-grpc.md`](2026-08-06-clusterclient-to-grpc.md) | Implementation plan: phases 0/1A/1B/2/3/4, task metadata, Opus-subagent execution rules, gotchas | | [`2026-08-06-clusterclient-to-grpc.md.tasks.json`](2026-08-06-clusterclient-to-grpc.md.tasks.json) | Machine task tracker (executing-plans resume point) | | [`2026-08-06-clusterclient-to-grpc-live-gate.md`](2026-08-06-clusterclient-to-grpc-live-gate.md) | Live gate record: baseline reproduction + per-phase check tables | | ScadaBridge reference | `~/Desktop/scadaproj/scadabridge_clusterclient_to_grpc.md` + `~/Desktop/ScadaBridge/docs/plans/2026-07-22-clusterclient-to-grpc-{plan,live-gate}.md` | ## Phase status board | Phase | Scope | Branch | Status | PR / merge SHA | Live gate | |---|---|---|---|---|---| | 0 | Proto + mappers + goldens; both auth interceptors; `IMeshPskProvider`; options/validators (+`Grpc` value, SplitTopology relaxation); rig dark config; stale-comment fixes; FleetStatus issue filed | `feat/grpc-phase0-contract-auth` | **MERGED to `master`** @ `10b6d209` (2026-08-06) — all 6 tasks complete @ `23555ed0`; every task spec ✅ + code ✅ (Batch B on Opus with a closed fix loop); issues #542 + #543 filed | `10b6d209` | **PASS 4/4** | | 1A | Central→node command plane (`NodeCommandService` on the Telemetry listener; `MeshGrpcCommandDialer`; third mode branch in `CentralCommunicationActor`) — **worktree A** | `feat/grpc-node-command` | **COMPLETE @ `e54883a8` — gate re-run PASSED 2026-08-06.** Run 1 @ `fbf7b708` FAILED on defect 4 (no admin node could boot, mode-independent); fixed by `c0475f2d` + `e54883a8` and re-gated. 1A.1 ✅ `64774652` + 1A.2 ✅ `6a862011` (+fixes `48dc2120`/`86633f14`) all reviewed spec ✅ + code ✅. **MERGED to `master`** @ `b360239e` (2026-08-06) | `b360239e` | **PASS 8/8** (check 3's AdminUI half closed by composition — the control is structurally unreachable, see carry-forwards) | | 1B | Node→central ack plane (`CentralCommandService` on the ConfigServe listener; `GrpcApplyAckSender`; third branch in `NodeCommunicationActor`) — **worktree B**, rebased on merged 1A | `feat/grpc-central-ack` | **MERGED to `master`** @ `7203d135` (2026-08-06, branch tip `cf229a95` incl. the phases 2-4 plan update); 1B.1 ✅ + 1B.2 ✅ (Opus: deadline Critical found+fixed) + gate **PASS 7/7** incl. the 4b hang drill (~10 s failover = exactly one AckTimeoutSeconds — live proof of the fix); at-least-once duplicate absorbed idempotently exactly as traced | `7203d135` | **PASS 7/7** | | 2 | **COMPLETE — gate PASS 8/8 + baseline + PSK negatives** (2026-08-06, all on `master`, all review chains closed). Task 2.0 ✅ (`a94eb8a3`+`d7c9ead6`, small chain: one Important fixed — non-vacuous once-per-incarnation pin — then APPROVED; healthy racing boot = ZERO Warn/Error, proven live on both centrals 4/4 boots). Register-item-3 role-scoping pin ✅ (`2e06dfe5`+`027f0403`, review found+fixed a Critical phantom-block false pass in the scan — braceless `if (hasAdmin)` at Program.cs:146). Baseline REPRODUCED pre-flip (`1bb3a937`): cross-mesh `:4053` open unauthenticated; the site-b drop is actively suppressed into SAMPLED deadLetters INF (456/467 unlogged) with the deploy deadline as sole witness. Cutover `caf2eec6` (rig default → `Grpc`, committed). Gate: deploy e2e all-gRPC sealed 6/6; command matrix live incl. real driver restart; kill-site-a-1 → per-node WRN naming node+status, exactly 1 attempt, honest deadline, boot-replay-only recovery; kill-central-1 → ack ordered-walk failover logged per node, reconvergence 6/6; site-b-pair drill → BOTH failures logged in the dispatch second (the baseline contrast); clusterclient/receptionist lines = 0 on all six through the whole gate; 135-min soak 14/14 cycles, denied=0/unauth=0, review items explained (historian-sidecar-absent DBG noise; 14×2 known per-deploy authoring WRNs); **full suite exit 0 on the cutover SHA, run against the live rig** | `caf2eec6` (cutover) + gate records `1bb3a937`/`bcc91268`/`eaf14e01`+final | **PASS 8/8** | | 3 | Delete ClusterClient (code/HOCON/config/tests); SplitTopology requires `Grpc`; grep-gates; docs sweep incl. scadaproj index | `feat/grpc-phase3-deletion` | **MERGED to `master` @ `2fa880b6` and pushed** (2026-08-06, user-approved; branch tip `f8a43dfe`, 11 commits off `master @ 55104526`). Phase 4 ran later the same day on the user's go-ahead — see its row. Task 3.1 ✅ high-risk chain: 7 ordered steps `0db40a5e`/`ab5b958d`/`5d7ecdf7`/`d67a7e5d`/`85ca86e9`/`af3b0c31`/`4d2dfb5b`; spec review ✅ (all KEEPs verified: `LoadContactsFromDb`, DPS+frame-size canary, `deployment-acks` DPS ingress, Kestrel block, one-ctor interceptors, no retries, proto untouched; `MeshGrpcBoundaryTests`' no-association assert verified falsifiable); code review ✅ Approved, one Important fixed (`cd713557` — `MeshPaths` full-path constants were dead-in-production + doc claimed a wire-contract role they no longer have; collapsed to `*Name` constants). Retired-mode fail-closed: `Mode=ClusterClient` now FAILS validation naming `MeshTransport:Mode` + allowed values (pinned); `SplitTopologyTransportValidator` requires exactly `Grpc`. Task 3.2 ✅ docs sweep `406b1ae5` + review-fix `8e25ed55` (Configuration/Redundancy/Telemetry/config-db-schema/CLAUDE.md; deferment.md had no mention); docs-accuracy review ✅ Approved — every config key/port/behavior claim verified against source, two pre-existing Minor stalenesses fixed (Telemetry.md Phase 5 gate status; config-db-schema Phase 6 gossip premise). scadaproj index updated on its `main` and PUSHED (`0e416ca`, merge-state-accurate: Phase 3 recorded as pending merge). Grep-gates: `CentralContactPoints` in src/tests/docker-dev = 0; `clusterclient\|receptionist` = historical/fail-closed/retirement mentions only (classified by both reviewers). **DoD full-suite: run 3 = `dotnet test` exit 0, 50/50 suites, 9,326 passed / 0 failed / 57 skipped, on `8e25ed55` (docs-only commits followed).** Runs 1–2 each had ONE unrelated single-test failure (three distinct tests across the two runs: `RouteNativeAlarmAck_on_non_primary_is_dropped`, `Rediscovery_flag_persists_on_subsequent_health_publishes`, `LoggerLifecycleTests.ConfigureLogging_DisposesPreviousLogger…`) — all in files the 55-file branch diff never touches, all 3/3 green in isolation (9/9 total): pre-existing under-load timing flakes, not Phase 3 regressions. ⚠️ Process find: run 1's "exit 0" captured `tail`'s exit code, not `dotnet test`'s (`cmd \| tail; $?` trap) — fixed for runs 2–3; **Phase 2's check-8 "full suite exit 0" used the same pipeline shape and its log did not survive, so that exit-code claim is unverifiable in hindsight** (its per-suite pass lines were observed at the time; Phase 4's terminal run on the merge SHA, with correct capture, supersedes it) | `2fa880b6` | n/a (Phase 4 is the terminal live gate) | | 4 | Terminal live gate on the post-deletion image (8 checks, provenance-guarded) | (on `master`) | **COMPLETE — gate PASS 8/8, 2026-08-06, PROGRAM CLOSED.** Ran on `master @ 0b194003` (Phase 3 merge `2fa880b6` + the #542–#546 fix merges + docs); provenance guard PASS (image `d2a290fc5711` built 17:55:10 EDT > deletion merge 16:57:05 > last code merge `9420f8ef` 17:28:48). Full record in the live-gate doc; headline results: PSK matrix 7/7 incl. unset-key boot refusal naming `MeshTransport:GrpcPsk`; command matrix with **Restart+Reconnect driven from the real `/hosts` panel for the first time** (#546 closed the 1A composition carry-forward by direct evidence — fan to six under one correlation id, both SITE-A nodes acted, operator identity carried); ack matrix exactly-once ×6 sealed in ~30 ms; central-kill ack failover (ordered walk to central-2, one INF per node) + site-kill dispatch honesty (exactly 1 WRN, +2:00 deadline, boot-replay, no retro-ack); **mid-deploy coordinator kill: central-2 killed in the dispatch second, central-1's recovered coordinator fired the deadline at exactly +2:00 with all 5 landed acks preserved (incl. the dead node's own), zero dupes across both restarts, reconverge sealed 6/6**; boundary: exactly 3 established `:4053` associations rig-wide, all pair-internal, zero site↔central; full-rig restart → transport grep `receptionist\|clusterclient` = **0 on all six**, post-restart deploy sealed 6/6 (one non-mesh observation: the CENTRAL pair reproduced the documented guard-off simultaneous-cold-start split-brain — Phase 7 territory, opt-in `Cluster:BootstrapGuard` exists, healed by single-node rejoin; both SITE pairs re-formed clean); `MeshGrpcBoundaryTests` 3/3; **terminal full suite RC=0 direct capture, 50/50 suites, 9,333 passed / 0 failed** (supersedes Phase 2's unverifiable check-8 exit code). Whole-gate PSK-drift sweep 0 denials ×6 nodes ×4 interceptors. I-4 cold-PSK-cache re-carried out of the program (docker-dev structurally cannot measure it; first store-backed deployment measures it). No new defects. Debug override reverted; rig left on the committed compose | n/a (gate only) | **PASS 8/8** | **Execution rules recap:** phase order 0 → (1A ∥ 1B) → 2 → 3 → 4; 1A∥1B only in separate git worktrees, one Opus subagent per worktree, sequential tasks within a worktree; 1A merges first, 1B rebases (union-resolve `Host/Program.cs` map block, compose env, `docs/Configuration.md`); rig mode flips before Phase 2 are never committed; **STOP at every phase DoD — PR + merge are user-owned.** ## Decision log | Date | Decision | Where recorded | |---|---|---| | 2026-08-06 | Reuse existing listeners (Telemetry `:4056` node-side, ConfigServe `:4055` central-side); rejected dedicated port + `ClusterNode` column | design §3.1 | | 2026-08-06 | Per-cluster PSK `OT-MESH-PSK-` from Secrets; rejected fleet-wide shared key (blast radius); `MeshTransport:ClusterPsks` config map as rig/test override | design §3.4, plan T0.3/T0.5 | | 2026-08-06 | Fire-and-forget semantics preserved exactly — no retries/queues; failures become visible logs, backstops (2-min deadline, boot replay) unchanged | design §3.3 | | 2026-08-06 | `Dps` mode survives for single-mesh topologies; only `ClusterClient` is deleted; the three stale "Phase 6 deletes DPS" comments get corrected, not obeyed | design §3.6/§4.2 | | 2026-08-06 | No transport interface — third mode branch inside the two choke-point actors (ScadaBridge §7.1 economy) | design §4.3 | | 2026-08-06 | Coexistence granularity: central's mode governs commands, each node's mode governs its acks (per-pair flips possible — better than ScadaBridge's central-wide flag) | design §3.6 | | 2026-08-06 | `AlarmCommandMessage.operation` is a proto `string`, NOT an enum — the CLR `AlarmCommand.Operation` is an unconstrained string, and an enum would silently decode unknown operations to `UNSPECIFIED` (the drop class this migration removes). Plan T0.1's "AlarmCommand verb" enum wording was a plan defect; only `ApplyOutcome` carries the enum-0 rule. T0.2's goldens must pin pass-through of an unrecognized operation string | proto comment + this log (T0.1 review) | | 2026-08-06 | **Gate follows the endpoint, not the mode** (1A.2 I-2): known-cluster set published on every contact load in all three modes, from ALL `ClusterNode` rows; fan set unchanged. Survives dark-switch rollback with nothing for 1B to remember | tracking §register refinement + actor docs | | 2026-08-06 | `CentralCommandService` mapped by PORT, not mode-gated (1B.1): a mode flip on a running fleet must not need a central restart before acks flow; matches every sibling service | `Program.cs` comment + this log | | 2026-08-06 | Ack deadline is PER ENDPOINT, not per walk; the walk is at-least-once on the deadline path, absorbed idempotently by the coordinator's NodeId-keyed upsert | `MeshTransportOptions` docs + defect log #1 | | 2026-08-06 | `correlation_id` lives on the envelope AND as inner fields on the three commands whose CLR record carries one — each command round-trips independently in the T0.2 mapper; the envelope covers `AlarmCommand` (no CLR correlation member). For the three, the inner field is authoritative on decode | proto comment + this log (T0.1 review) | ## Phase 1A security register (from the Batch B Opus code review — HARD GATE: all four close BEFORE `CentralCommandService` is mapped) 1. **CLOSED (1A.2 @ `48dc2120`, refined — see below).** Known-cluster-set membership gate in `TryGetPsk` — central rebuilds its cluster set from `ClusterNode` rows every `ContactRefreshSeconds`; refuse a cluster id outside that set in O(1) with no store touch. Closes BOTH open review findings at once: (a) the timing oracle that currently defeats the uniform-denial anti-enumeration text (unknown cluster pays a store read on every probe because negatives are deliberately never cached; known cluster is cached — a stable remotely-measurable distinguisher), and (b) the pre-auth resource exhaustion / audit-flood vector (each distinct forged `x-otopcua-cluster` header costs a secret-store read + a `secret.resolve.not-found` audit row into `ConfigAuditLog` via the audit singleton, attacker-chosen `Target` values, before any credential check). Namespace confinement (`OT-MESH-PSK-` prefix + `SecretName` allow-list) means no disclosure risk — this is flood/oracle only. 2. **CLOSED (1A.2 @ `14d21496`).** `bool Matches(string clusterId, string presented)` on `IMeshPskProvider` — keep `FixedTimeEquals` inside the provider so a future verify-side caller cannot write `==` and have it compile. Do before 1B adds the dial-side caller. 3. **DI-shape pin test** — `IMeshPskProvider` is `hasAdmin`-only and its pairing with the central interceptor is by convention; a driver-only node injecting it throws on first call, not at boot. Add a Host.IntegrationTests check (admin host resolves it; driver-only host does NOT) when Phase 1 maps the services. **Refinement on item 1 (decision, 2026-08-06):** the gate follows the ENDPOINT, not the mode — published on every contact load in every transport mode (the interceptor and service are port-mapped, not mode-gated, so a dark-switch rollback must not deactivate the gate), and built from ALL `ClusterNode` rows (gate answers "does central know this cluster"; the fan set stays `Enabled && !MaintenanceMode`, so gate ⊇ fan — this also dissolved the maintenance/ack-auth coupling). Pinned by the every-mode Theory + gate-⊋-fan fact + the Dps no-ClusterClient guard pin (`86633f14`). Residuals accepted + visible: gate is fail-open until the first successful contact load (and stays stale if central boots with SQL down — warned every tick); gate derives from the node table, so a `ServerCluster` with zero rows is outside it. **Follow-ups from the 1A.2/1B.2 Opus reviews (tracked, not blocking):** per-tick `InvalidateAll` puts a cold-cache 3 s store read on the inbound auth path — MEASURE on the 1B.3 gate (ack burst inside ~3 s after a refresh tick), then consider diff-based invalidation; the command plane is h2c-only so the per-cluster PSK crosses the network in cleartext on a write surface — precedent-consistent (all sibling gRPC planes), record as accepted risk + TLS-option issue (queued below); dialer minors (Lazy channel wrapper, retired-endpoint eviction, gRPC deadline vs token cancellation, pre-existing `LoadContactsFromDb` Task.Run(ct) pattern). 4. **Integration gate over real Kestrel h2c** — both new interceptors have unit coverage only; the one-public-ctor reflection test approximates but cannot prove the interceptor is actually invoked. Wire-probe in the 1A/1B live gates (already in the live-gate tables). Non-gating notes from the same review: central-side unseeded-PSK is invisible until a deploy stalls (consider a "PSK seeded for every known cluster?" `/hosts`/health surface — the #521/#541 shape); `SecretName` lowercases and allows only `[a-z0-9._/-]` (underscore IS allowed — the review's `Site_A` example was wrong, caught + corrected by the T0.6 agent against the pinned 0.2.3 DLL), so a cluster id with any character outside that set fails closed at the store and must use `ClusterPsks` — documented in `docs/Configuration.md` (rig ids `MAIN`/`SITE-A`/`SITE-B` are fine); sibling interceptors still use `IsNullOrEmpty` (whitespace key opens them) — family-wide backport is out of this program's scope, file alongside the FleetStatus issue in T0.6; consider a non-throwing counter metric if the poison-guard silent catch ever fires in production. ## Phase 1 dispatch notes (from Task 0.2 reviews — fold into the 1A/1B implementer prompts) - The alarm arm of `FromEnvelope` necessarily discards the envelope correlation (`AlarmCommand` has no CLR member for it) — the 1A router must read `envelope.CorrelationId` itself for logging. - `ApplyAck` is the ONE record of the five that never rides `CommandEnvelope`: `GrpcApplyAckSender` (1B) calls `MeshTransportDtoMapper.ToProto(ApplyAck)` directly on `ReportApply` — do not expect (or add) a sixth `ToEnvelope` overload. - Consider a thin try/catch at the `Execute`/`ReportApply` service boundary wrapping `FromEnvelope`/`FromProto` failures with the envelope's correlation_id (readable as a plain string even when inner ids fail to parse) so an operator can correlate a rejected call. - Revisit the untyped `ToEnvelope(object, CorrelationId alarmCorrelationId)` signature when the 1A router lands — if the router prefers one always-honoured transport correlation, change the signature then, before it has more callers. ## Defect log Defects found during execution (live-gate or review), with root cause + pinned test. | # | Phase | Defect | Root cause | Fix / pin | |---|---|---|---|---| | 1 | 1B.2 | **Failover silently defeated on a hanging central** (CRITICAL, caught by Opus code review): the ack deadline was one absolute `DateTime` shared across the endpoint walk — a hung (not refused) central-1 burned the whole budget and central-2 was dialed with an expired deadline | deadline hoisted outside the `foreach`; a kill-style drill can't see it (refused = fast-fail), only a hang discriminates | `e54711a2`; pinned by `Each_endpoint_gets_a_FRESH_deadline` (mutation-verified); 1B.3 gains a `docker pause` hang drill | | 2 | 1A.2 | **Dark-switch rollback deactivated the known-cluster gate** (Opus review, I-2): gate published only under `Mode=Grpc` while the interceptor + service are port-mapped in every mode — a rollback to ClusterClient/Dps would silently restore the timing oracle + pre-auth flood | gate activation coupled to transport mode instead of the endpoint it protects | `48dc2120` — gate follows the endpoint; pinned by the every-mode Theory + Dps guard pin `86633f14` | | 5 | 1A.3 (live gate, run 2) | **The `PreStart` resolve loses its race on real boots** — 2 of 3 observed admin boots logged `Failed to resolve IMeshCommandDialer` / `IMeshPskProvider`, same `AuditWriterActorKey` cause. NOT a defect: it is exactly the case the review-prescribed hardening was written for | `PreStart` runs on a dispatcher thread concurrently with the builder's remaining starters, so it can still precede starter #4 | **Closed by `e54883a8`, and the heal is PROVEN LIVE** — the coordinator singleton was moved onto the node that lost the race (stop central-2) and it fanned a dispatch to all five live nodes with no dialer error. Residual is cosmetic (Error text reads permanent; will trip a soak error-grep on a healthy node) → Phase 2 | | 4 | 1A.3 (live gate, run 1) | **An admin node cannot boot** (BLOCKED 1A): both central nodes die at startup with `MissingActorRegistryEntryException: ZB.MOM.WW.OtOpcUa.ControlPlane.AuditWriterActorKey`, taking `AkkaHostedService` and the whole host with them. Mode-INDEPENDENT — reproduced on the same image with `MeshTransport__Mode=ClusterClient`, so it is not the gate's flip. Driver-only nodes are unaffected (they register neither `IMeshPskProvider` nor `ActorAuditWriter`) | `WithOtOpcUaControlPlaneSingletons`'s first `WithActors` starter eagerly resolves `IMeshCommandDialer` + `IMeshPskProvider` to pass into `CentralCommunicationActor.Props`. That graph is dialer → `IMeshPskProvider` → `ISecretResolver` → (`AddZbSecrets`) → `IAuditWriter` → `ActorAuditWriter`, whose ctor calls `registry.Get()` — and `WithSingleton` is registered ~60 lines later in the same builder. Same class as scadaproj#1 and mesh Phase 6's `IClusterRoleInfo` crash: only visible with the DI graph built in a real host, which is why the container-activation test (no `AddZbSecrets` in its collection) and `TwoNodeClusterHarness` (no provider at all) both miss it. Aggravating: the failed process does not exit, so `docker ps` reports `Up restarts=0` while the node serves no HTTP | **FIXED** — closed in 1A by `c0475f2d` (lazy resolve of dialer + PSK provider — central boots again) + `e54883a8` (memoize on first use; a lost `PreStart` race self-heals at the first refresh, review-prescribed), merged in Phase 1A @ `b360239e`; gate re-run PASSED 8/8 (matches the Phase 1A board row above). Original suggested shape: hand the actor lazy accessors, mirroring the `Func<>` the same block already uses for the coordinator proxy | | 3 | 1A.2 | **Fan parked the actor mailbox** (self-caught by the implementer): dialing from the receive ran `TryGetPsk`'s cold-cache 3 s block pre-await (cold after EVERY refresh due to `InvalidateAll`), stalling deployment acks behind the fan | async methods run synchronously to the first await | `Task.Run` hop (token None — a canceled scheduling would read as a DB failure via PipeTo); pinned red-first by `A_slow_dial_does_not_park_the_actors_mailbox` | ## Carry-forward register Checks a gate could not drive, named + carried to the next gate (proportionate-gate rule). | Item | Deferred from | Proven instead by | Closed in | |---|---|---|---| | Hang-style central failover drill (`docker pause`, not `docker stop` — only a hang discriminates the per-endpoint-deadline fix) | 1B.2 review | unit fact `Each_endpoint_gets_a_FRESH_deadline` | 1B.3 gate (MUST include) | | I-4 measurement: ack burst within ~3 s after a contact-refresh tick (cold PSK cache on the inbound auth path) | 1A.2 review | bounded 3 s store read + `Lazy` stampede suppression | 1B.3 gate (observe) | | `TwoNodeClusterHarness` logs the I-1 null-provider Error at every spawn (harness registers no `IMeshPskProvider`) — EXPECTED, not a rig fault | 1A.2 re-review | production pairing is `hasAdmin`-unconditional in one block | note for 1B.3/Phase 2 readers | | Register item 3 (DI role-scoping pin: admin resolves provider+dialer, driver-only does NOT) — container-activation test exists, role-scoping pin outstanding | 1A.2 | `It_activates_from_the_container_despite_the_optional_client_factory_parameter`; starter-order half now covered by `ControlPlaneStarterOrderTests` | **CLOSED in Phase 2** (`2e06dfe5`+`027f0403`): source-scan pin in Host.Tests, review-hardened — a Critical phantom-block false pass (braceless `if (hasAdmin)` at Program.cs:146 made the scanner adopt the `if (hasDriver)` block's brace) was found and fixed, both bite-checks proven | | `Resolve`-failure Error wording says "continuing without it" (reads permanent — the memoize heals it at first use) and the line fires on real boots (2/3 in the gate) — reword or whitelist it before the Phase 2 soak error-grep, else a healthy central scores as failing the soak | 1A.3 re-run | heal proven live: singleton moved onto the race-losing node, fan clean, no dialer-missing error | **CLOSED by Task 2.0** (`a94eb8a3`+`d7c9ead6`): PreStart resolve logs Debug-only; Error moved to first-use sites (per-dispatch for the dialer, once-per-incarnation for the PSK provider); healthy racing boot = 0 Warn/Error, pinned + proven live (soak grep ran clean with NO whitelist entry for this path) | | A repaired `GrpcPort` takes effect only at the next contact refresh (≤60 s): a deploy 5 s post-restore was still skipped and honestly TimedOut — by design, operator-visible; document if it surprises in Phase 2 | 1A.3 re-run | check 7 evidence | Phase 2: not exercised (no GrpcPort repair drill was in the check table) and nothing surprised; behaviour unchanged by design — note stands for operators | | I-4 (cold PSK cache on the inbound auth path after each refresh tick) is structurally unmeasurable on docker-dev — `ClusterPsks` config map answers ahead of the store, so the cache is always empty and `InvalidateAll` clears nothing | 1B.3 gate | every ack landed within the posting second | **Re-carried OUT of the program at Phase 4** (2026-08-06): docker-dev still configures `ClusterPsks`, so the cache never holds a store-resolved value and the latency cannot exist on this rig. Every ack in every gate landed within its posting second. To be measured once on the first store-backed deployment; cold-cache fail-closed behavior is pinned by unit tests | | After a central restart, a node's cached `GrpcChannel` can sit in transient failure ~2 min (client backoff; TCP open, sibling reaches the same endpoint) — no ack lost, the walk absorbs it loudly. Expect a failover-line burst after any central restart in the Phase 2 soak | 1B.3 gate | 3 dispatches: 2 failed over, 3rd direct | **Phase 2 OBSERVED, on the command dial**: after the check-3 site-a-1 kill, central-2's cached channel dropped the check-4 dispatch to the already-healthy site-a-1 (loud, single-attempt, honest deadline) ~3 min post-kill; healed by the next deploy which sealed 6/6. **Phase 4 confirmed three more occurrences** (post-central-1-restart re-deploy; central-2→site-a-1 skip after the drill-B kill) — always loud, never lossy beyond the honest deadline, healed by the next dispatch after the ~2-min window. Standing operator note: expect one TimedOut deploy within ~2 min of any node kill/restart | | **The PreStart race fires on essentially EVERY admin boot** (4/4 in the 1B gate, 2/3 in 1A) and logs two FALSE Errors ("continuing without it", "gate will NEVER be published") that the memoize immediately heals — the wording fix is now a MUST before the Phase 2 soak error-grep, not a nicety | 1B.3 gate obs (b) | gate published + acks accepted on every one of those boots | **CLOSED — Task 2.0 shipped as the Phase 2 opening task** (`a94eb8a3`+`d7c9ead6`) and verified live: both centrals booted with zero resolve-path Warn/Error on the cutover rig | | `RevisionHash` moved on byte-identical config after a direct SQL `UPDATE` bumped `Script.RowVersion` (hash covers a column content-identity does not); determinism itself holds (3 no-change dispatches → identical hash) | 1A.3 re-run | incidental, out of program scope | note only | | 1A checks 1/2/7 (deploy fan over gRPC, mixed-transport seal, NULL-`GrpcPort` skip) and the central half of 4 | 1A.3 gate run 1 | — | **CLOSED in 1A.3 run 2** — all driven live and PASS | | Check 8 is vacuous for telemetry + config-serve — central never booted, so neither plane carried a call | 1A.3 gate run 1 | — | **CLOSED in 1A.3 run 2** — both planes carried authenticated traffic (28 KB artifact fetches from `:4055`, telemetry streams open on all six) with 0 denials | | **`DriverStatusPanel.razor` is referenced by no page or component** — the Reconnect/Restart buttons are unreachable in the running AdminUI (pre-existing on `master`, not caused by this program; re-verified in run 2: 0 `