docs(grpc): Phase 3 live gate — PASS; central→site command cutover

Central flipped to SiteTransport=Grpc (both nodes, central-wide flag), sites
kept on gRPC from Phase 2 → both directions on gRPC at once. Central logs
'central→site command transport: gRPC (SiteCommandService)'; 0 ClusterClient-to-
site on either central.

Command matrix over SiteCommandService (all 200):
- ExecuteQuery (event-log) + ExecuteParked (parked) — all 3 sites
- ExecuteLifecycle disable/enable #95 on site-a — NEW live coverage vs 1B/P2
  (SoakNotify instances survived the recreate as Enabled)

Resilience:
- hard-kill the ACTIVE site-a node mid-query-loop → in-flight call returned
  TIMEOUT at exactly the 30s QueryTimeout deadline (bounded, no hang; deadline≠
  retry — an in-flight call can't be safely re-sent)
- next + all subsequent queries recovered automatically via site-a-b
  (SitePairChannelProvider NodeA→NodeB); site→central S&F never stopped
  (notif 619→715, still no dupes)
- 0 PermissionDenied across all 8 nodes

ExecuteOpcUa/ExecuteRoute/TriggerFailover deferred (no OPC-bound instance / no
CLI verb; unit-proven). Rig left both-gRPC; git reverted to Akka default.

docs/plans/2026-07-22-clusterclient-to-grpc-live-gate.md

@@ -234,3 +234,104 @@ plan-vs-code finding, recorded rather than worked around.
 
 The instance-dependent matrix (tag ops, lifecycle, standby parked retry) and `TriggerSiteFailover`
 get their live exercise in Phase 3's full UI command matrix; the transport itself is proven here.
+
+---
+
+## Phase 2 — full site→central cutover + S&F soak — **PASS** (2026-07-23)
+
+All three sites (**all 6 nodes**) flipped to `CentralTransport=Grpc` (edit to
+`docker/site-*/appsettings.Site.json`, reverted in git after the gate — defaults stay Akka),
+rebuilt from `main` + `--force-recreate`. Central left on `SiteTransport=Akka` (Phase 3 owns
+that direction). The gate was driven by a **live S&F workload**, closing the notification/audit
+path that 1A/1B could only unit-prove.
+
+### S&F driver
+
+A minimal dependency-free template `SoakNotify` (one 5 s `Interval` script:
+`Notify.To("Engineering Alerts").Send(...)`), 3 instances deployed+enabled on site-a → a steady
+**3 notifications per 5 s bucket** (the pre-existing `Motor Controller` "soak-motor" instances
+need 30 OPC UA bindings and were unusable). The central `dbo.Notifications` table (one row per
+`NotificationId`, insert-if-not-exists) is the no-loss/no-dupes source of truth.
+
+### Checks
+
+| # | Check | Result |
+|---|---|---|
+| 1 | All 6 site nodes on `CentralTransport=Grpc` | **PASS** — each logs `Site→central transport: gRPC to 2 central endpoint(s)`; **0** `PermissionDenied` across all six for the whole run |
+| 2 | Full control plane rides gRPC `CentralControlService` | **PASS** — central sees `Heartbeat`, `ReportSiteHealth`, `SubmitNotification` (the S&F path), `IngestAuditEvents` — **196 RPCs / 90 s, 0 non-200** |
+| 3 | Health/heartbeat cadence unchanged, no sequence regressions | **PASS** — `CentralHealthAggregator` logged **0** sequence-regression/out-of-order lines; heartbeat steady at ~144/30 s across 3 sites |
+| 4 | **Single-node failover** — hard-kill the **active** central (central-a) | **PASS** — `CentralChannelProvider` logged sticky failover `central-a:8083 → central-b:8083` at the instant of kill; central-b active in **29 s** (auto-down); notif count froze at 72 during the gap |
+| 5 | Buffer drains, no loss/dupes (single-node) | **PASS** — count resumed 72→101; **every 5 s bucket through the outage = exactly 3**, no gap; 101 total == 101 distinct |
+| 6 | **Failback** — restart central-a | **PASS** — rejoined **ready in ~5 s as standby** (`active=503`); central-b **retained active** (oldest-Up, no role flap); traffic uninterrupted (uniform 3/bucket across failback); central-a singletons → `Younger` |
+| 7 | **Full central outage** — hard-kill **both** central (~59 s) | **PASS** — count frozen at 155 for the entire outage; sites buffered continuously |
+| 8 | Cold re-form + drain, no loss/dupes (both-down) | **PASS** — cold cluster re-formed, central-b active in **~14 s**; ~42 buffered notifications drained; **every 5 s bucket across the whole ~59 s both-dead window = exactly 3**, no gap; **216 total == 216 distinct** |
+
+Checks 4–8 also stand as a live preview of Phase 5 checks 4 (failover/failback) and 5 (mid-drain
+kill, zero loss/zero dupes).
+
+### Notes
+
+- **Audit telemetry also rides the new gRPC `CentralControlService`** (`IngestAuditEvents`) — the
+  `SiteAuditTelemetryActor` "ClusterClientSiteAuditClient" label is legacy naming, not the wire
+  path. So `CentralTransport=Grpc` moves heartbeat, health, notification S&F **and** audit off
+  ClusterClient in one flip.
+- Sites settled on **central-a** as the gRPC endpoint after the cold both-restart while **central-b**
+  held the active/singleton role — the gRPC endpoint node and the singleton host can differ; central-a
+  receives the forward and Akka-routes to the `NotificationOutboxActor` singleton on central-b. Both
+  are correct and independent.
+- Some notifications land `Parked` at central (no SMTP config on the bare rig) — irrelevant to the
+  transport proof: the `Notifications` **row** is written on forward regardless of downstream SMTP
+  delivery, so the count is a faithful no-loss/no-dupes measure.
+- Rig left running on the gRPC build; git config reverted to Akka default (a redeploy from `main`
+  resets to all-Akka).
+
+---
+
+## Phase 3 — full central→site cutover + command matrix — **PASS** (2026-07-23)
+
+Central flipped to `SiteTransport=Grpc` (both central nodes; the flag is **central-wide** —
+`CentralCommunicationActor.SelectTransport`). Sites kept on `CentralTransport=Grpc` from Phase 2,
+so the rig ran **both directions on gRPC simultaneously** — the eventual all-gRPC end state.
+Config bind-mounted (`:ro`), so a container `--force-recreate` (no image rebuild — code unchanged
+since Phase 2) applied it. Central logged the selection at startup:
+`central→site command transport: gRPC (SiteCommandService)`.
+
+The Phase 2 `SoakNotify` instances (#95–97) **survived the recreate as Enabled** (site volume
+persisted `deployed_configurations` this time), so the notification workload kept flowing —
+and lifecycle commands became drivable, closing the 1B/Phase-2 gap.
+
+### Checks
+
+| # | Check | Result |
+|---|---|---|
+| 1 | Central selects the gRPC command transport | **PASS** — `central→site command transport: gRPC (SiteCommandService)`; **0** `Created ClusterClient to` site lines on either central |
+| 2 | `ExecuteQuery` (event-log) over gRPC, all 3 sites | **PASS** — all 3 returned correlationIds; site logs `SiteCommandService/ExecuteQuery - 200` (site-a-a/b-a/c-a each served the calls; standby nodes 0 — work lands on the active site node via the singleton proxy) |
+| 3 | `ExecuteParked` (parked query) over gRPC, all 3 sites | **PASS** — all 3 returned `success` payloads over `SiteCommandService/ExecuteParked` |
+| 4 | `ExecuteLifecycle` (disable→enable) over gRPC, site-a | **PASS** — `instance disable`/`enable #95` → `success:true`; site served `SiteCommandService/ExecuteLifecycle` (×4), instance state toggled — **NEW live coverage vs 1B/Phase-2 (needed a deployed instance)** |
+| 5 | **Site-node kill mid-command → clean error, no hang** | **PASS** — killed the **active** site-a node (site-a-a) during a 1 s query loop: the in-flight call returned `TIMEOUT` at **dur=30.1 s = the `QueryTimeout` deadline** — bounded, not an indefinite hang (see note) |
+| 6 | **Site-pair failover mid-stream** | **PASS** — the very next query (~32 s after kill) and all subsequent ones succeeded automatically via **site-a-b**; `SitePairChannelProvider` failed the gRPC channel NodeA→NodeB and the site singleton migrated; site→central S&F never stopped (notif count climbed 619→715 through the kill, still no dupes) |
+| 7 | No PSK drift | **PASS** — **0** `PermissionDenied`/`Unauthenticated` across all 8 nodes for the whole run |
+| 8 | Zero ClusterClient activity on the flipped path | **PASS** — central built no site ClusterClients; command routing is entirely `SiteCommandService` gRPC |
+
+### Note on check 5 (deadline vs fast-fail)
+
+The command in flight when site-a-a was **hard-killed** (SIGKILL) waited the full 30 s
+`QueryTimeout` rather than failing fast on connect-refused: an already-dispatched gRPC call on a
+dropped connection isn't observed as unsent, so it correctly cannot be auto-retried on the peer
+node (it might have executed) and returns the deadline error to the caller — exactly the plan's
+"deadline ≠ retry" rule. The deadline is the backstop; "no hang beyond deadline" is satisfied
+(30.1 s). Only **provably-unsent** connect failures fail over fast, which is why every *subsequent*
+call recovered immediately via site-a-b.
+
+### Not driven on this gate (unchanged from 1B/Phase 2)
+
+- **`ExecuteOpcUa`** (BrowseNode/ReadTagValues/WriteTag) — needs an OPC-bound deployed instance;
+  the bare rig's only deployable template (`SoakNotify`) has no data connection, and `Motor
+  Controller` needs 30 OPC UA bindings. Unit-proven (dispatcher routing ×28).
+- **`ExecuteRoute`** (inbound-API → routed site script) — needs an inbound method + routing target
+  the rig lacks.
+- **`TriggerFailover`** — no CLI verb (UI/management-only), destructive; unit-proven (ack-before-
+  `Leave` ordering tests). The hard-kill in check 6 is the live equivalent of a site-pair failover.
+
+Rig left running with **both** transports on gRPC; git config reverted to Akka default (a redeploy
+from `main` resets to all-Akka).

docs/plans/2026-07-22-clusterclient-to-grpc-plan.md

@@ -323,8 +323,8 @@ Critical path ≈ 1B: **~4–6 weeks total**, matching the design estimate.
 - [x] 1B DoD (proportionate): rig central on `SiteTransport=Grpc` proves central→site rides authenticated gRPC `SiteCommandService` for all 3 sites (`ExecuteQuery`/`ExecuteParked` → 200, per-site PSK, 0 auth failures); rebased on 1A. Instance-dependent commands (tag ops/lifecycle/standby parked retry) + `TriggerSiteFailover` deferred to Phase 3 (no deployed instance / destructive UI-only); command-plane coexistence not expressible (`SiteTransport` is central-wide). Gate: `2026-07-22-clusterclient-to-grpc-live-gate.md`
 
 **Phase 2 ∥ 3 — cutover + soak**
-- [ ] P2 All sites `CentralTransport=Grpc`; central-kill S&F soak (no loss/dupes), failback observed, health sequences clean
+- [x] P2 All sites `CentralTransport=Grpc`; central-kill S&F soak (no loss/dupes), failback observed, health sequences clean — **PASS 2026-07-23** (live gate: single-node active-kill failover 29s + full-outage ~59s both-down drain; every 5s bucket = exactly 3 through both outages, 216 total == 216 distinct; 0 auth failures / 0 seq regressions; whole control plane — heartbeat/health/notification/audit — on gRPC `CentralControlService`)
-- [ ] P3 Central `SiteTransport=Grpc` all sites; full UI command matrix per site; site-kill mid-command clean; no PSK noise; zero ClusterClient log activity on flipped paths
+- [x] P3 Central `SiteTransport=Grpc` all sites; full UI command matrix per site; site-kill mid-command clean; no PSK noise; zero ClusterClient log activity on flipped paths — **PASS 2026-07-23** (live gate: `ExecuteQuery`/`ExecuteParked` all 3 sites + `ExecuteLifecycle` disable/enable on site-a, all 200 over `SiteCommandService`; active-site-node hard-kill mid-command → clean `TIMEOUT` at the 30s deadline, next call failed over to site-a-b automatically; 0 PermissionDenied / 0 ClusterClient-to-site on both central. `ExecuteOpcUa`/`ExecuteRoute`/`TriggerFailover` deferred — no OPC-bound instance / no CLI verb, unit-proven)
 
 **Phase 4 — deletion**
 - [ ] Defaults flip to `Grpc` + soak; then delete Akka transports, ClusterClient creation, `DefaultSiteClientFactory`, receptionist registrations, `CentralContactPoints`, then the flags

docs/plans/2026-07-22-clusterclient-to-grpc-plan.md.tasks.json

@@ -198,7 +198,7 @@
       "id": "P2",
       "phase": "2",
       "subject": "All sites CentralTransport=Grpc; central-kill S&F soak, failback observed, health sequences clean",
-      "status": "pending",
+      "status": "completed",
       "activeForm": "Running the site->central cutover soak",
       "blockedBy": [
         "P1A.DoD"
@@ -208,7 +208,7 @@
       "id": "P3",
       "phase": "3",
       "subject": "Central SiteTransport=Grpc all sites; full UI command matrix; site-kill mid-command clean; zero ClusterClient activity",
-      "status": "pending",
+      "status": "completed",
       "activeForm": "Running the central->site cutover soak",
       "blockedBy": [
         "P1B.DoD"