Central flipped to SiteTransport=Grpc (both nodes, central-wide flag), sites
kept on gRPC from Phase 2 → both directions on gRPC at once. Central logs
'central→site command transport: gRPC (SiteCommandService)'; 0 ClusterClient-to-
site on either central.
Command matrix over SiteCommandService (all 200):
- ExecuteQuery (event-log) + ExecuteParked (parked) — all 3 sites
- ExecuteLifecycle disable/enable #95 on site-a — NEW live coverage vs 1B/P2
(SoakNotify instances survived the recreate as Enabled)
Resilience:
- hard-kill the ACTIVE site-a node mid-query-loop → in-flight call returned
TIMEOUT at exactly the 30s QueryTimeout deadline (bounded, no hang; deadline≠
retry — an in-flight call can't be safely re-sent)
- next + all subsequent queries recovered automatically via site-a-b
(SitePairChannelProvider NodeA→NodeB); site→central S&F never stopped
(notif 619→715, still no dupes)
- 0 PermissionDenied across all 8 nodes
ExecuteOpcUa/ExecuteRoute/TriggerFailover deferred (no OPC-bound instance / no
CLI verb; unit-proven). Rig left both-gRPC; git reverted to Akka default.
All 6 site nodes flipped to CentralTransport=Grpc; whole control plane
(heartbeat/health/notification S&F/audit) rides gRPC CentralControlService,
196 RPCs/90s 0 non-200, 0 PermissionDenied, 0 health-sequence regressions.
Soak driven by a live SoakNotify workload (3 instances, 3 notifs/5s):
- single-node active-kill (central-a): sticky failover central-a:8083→b:8083
logged instantly, central-b active in 29s, buffer drained 72→101, every 5s
bucket = exactly 3 through the gap, 101==101 distinct
- failback: central-a rejoined ready ~5s as standby, central-b kept active
(oldest-Up, no flap), traffic uninterrupted
- full outage (both central down ~59s): count frozen, cold re-form central-b
active ~14s, ~42 buffered drained, every 5s bucket = exactly 3 across the
whole dead window, 216==216 distinct — zero loss, zero dupes
Also previews Phase 5 checks 4 (failover/failback) and 5 (mid-drain kill).
Rig config reverted to Akka default (defaults stay Akka until Phase 4).
2026-07-23 11:31:09 -04:00
3 changed files with 105 additions and 4 deletions
All three sites (**all 6 nodes**) flipped to `CentralTransport=Grpc` (edit to
`docker/site-*/appsettings.Site.json`, reverted in git after the gate — defaults stay Akka),
rebuilt from `main` + `--force-recreate`. Central left on `SiteTransport=Akka` (Phase 3 owns
that direction). The gate was driven by a **live S&F workload**, closing the notification/audit
path that 1A/1B could only unit-prove.
### S&F driver
A minimal dependency-free template `SoakNotify` (one 5 s `Interval` script:
`Notify.To("Engineering Alerts").Send(...)`), 3 instances deployed+enabled on site-a → a steady
**3 notifications per 5 s bucket** (the pre-existing `Motor Controller` "soak-motor" instances
need 30 OPC UA bindings and were unusable). The central `dbo.Notifications` table (one row per
`NotificationId`, insert-if-not-exists) is the no-loss/no-dupes source of truth.
### Checks
| # | Check | Result |
|---|---|---|
| 1 | All 6 site nodes on `CentralTransport=Grpc` | **PASS** — each logs `Site→central transport: gRPC to 2 central endpoint(s)`; **0**`PermissionDenied` across all six for the whole run |
| 2 | Full control plane rides gRPC `CentralControlService` | **PASS** — central sees `Heartbeat`, `ReportSiteHealth`, `SubmitNotification` (the S&F path), `IngestAuditEvents` — **196 RPCs / 90 s, 0 non-200** |
| 3 | Health/heartbeat cadence unchanged, no sequence regressions | **PASS** — `CentralHealthAggregator` logged **0** sequence-regression/out-of-order lines; heartbeat steady at ~144/30 s across 3 sites |
| 4 | **Single-node failover** — hard-kill the **active** central (central-a) | **PASS** — `CentralChannelProvider` logged sticky failover `central-a:8083 → central-b:8083` at the instant of kill; central-b active in **29 s** (auto-down); notif count froze at 72 during the gap |
| 5 | Buffer drains, no loss/dupes (single-node) | **PASS** — count resumed 72→101; **every 5 s bucket through the outage = exactly 3**, no gap; 101 total == 101 distinct |
| 6 | **Failback** — restart central-a | **PASS** — rejoined **ready in ~5 s as standby** (`active=503`); central-b **retained active** (oldest-Up, no role flap); traffic uninterrupted (uniform 3/bucket across failback); central-a singletons → `Younger` |
| 7 | **Full central outage** — hard-kill **both** central (~59 s) | **PASS** — count frozen at 155 for the entire outage; sites buffered continuously |
| 8 | Cold re-form + drain, no loss/dupes (both-down) | **PASS** — cold cluster re-formed, central-b active in **~14 s**; ~42 buffered notifications drained; **every 5 s bucket across the whole ~59 s both-dead window = exactly 3**, no gap; **216 total == 216 distinct** |
Checks 4–8 also stand as a live preview of Phase 5 checks 4 (failover/failback) and 5 (mid-drain
kill, zero loss/zero dupes).
### Notes
- **Audit telemetry also rides the new gRPC `CentralControlService`** (`IngestAuditEvents`) — the
`SiteAuditTelemetryActor` "ClusterClientSiteAuditClient" label is legacy naming, not the wire
path. So `CentralTransport=Grpc` moves heartbeat, health, notification S&F **and** audit off
ClusterClient in one flip.
- Sites settled on **central-a** as the gRPC endpoint after the cold both-restart while **central-b**
held the active/singleton role — the gRPC endpoint node and the singleton host can differ; central-a
receives the forward and Akka-routes to the `NotificationOutboxActor` singleton on central-b. Both
are correct and independent.
- Some notifications land `Parked` at central (no SMTP config on the bare rig) — irrelevant to the
transport proof: the `Notifications`**row** is written on forward regardless of downstream SMTP
delivery, so the count is a faithful no-loss/no-dupes measure.
- Rig left running on the gRPC build; git config reverted to Akka default (a redeploy from `main`
The Phase 2 `SoakNotify` instances (#95–97) **survived the recreate as Enabled** (site volume
persisted `deployed_configurations` this time), so the notification workload kept flowing —
and lifecycle commands became drivable, closing the 1B/Phase-2 gap.
### Checks
| # | Check | Result |
|---|---|---|
| 1 | Central selects the gRPC command transport | **PASS** — `central→site command transport: gRPC (SiteCommandService)`; **0**`Created ClusterClient to` site lines on either central |
| 2 | `ExecuteQuery` (event-log) over gRPC, all 3 sites | **PASS** — all 3 returned correlationIds; site logs `SiteCommandService/ExecuteQuery - 200` (site-a-a/b-a/c-a each served the calls; standby nodes 0 — work lands on the active site node via the singleton proxy) |
| 3 | `ExecuteParked` (parked query) over gRPC, all 3 sites | **PASS** — all 3 returned `success` payloads over `SiteCommandService/ExecuteParked` |
| 4 | `ExecuteLifecycle` (disable→enable) over gRPC, site-a | **PASS** — `instance disable`/`enable #95` → `success:true`; site served `SiteCommandService/ExecuteLifecycle` (×4), instance state toggled — **NEW live coverage vs 1B/Phase-2 (needed a deployed instance)** |
| 5 | **Site-node kill mid-command → clean error, no hang** | **PASS** — killed the **active** site-a node (site-a-a) during a 1 s query loop: the in-flight call returned `TIMEOUT` at **dur=30.1 s = the `QueryTimeout` deadline** — bounded, not an indefinite hang (see note) |
| 6 | **Site-pair failover mid-stream** | **PASS** — the very next query (~32 s after kill) and all subsequent ones succeeded automatically via **site-a-b**; `SitePairChannelProvider` failed the gRPC channel NodeA→NodeB and the site singleton migrated; site→central S&F never stopped (notif count climbed 619→715 through the kill, still no dupes) |
| 7 | No PSK drift | **PASS** — **0**`PermissionDenied`/`Unauthenticated` across all 8 nodes for the whole run |
| 8 | Zero ClusterClient activity on the flipped path | **PASS** — central built no site ClusterClients; command routing is entirely `SiteCommandService` gRPC |
### Note on check 5 (deadline vs fast-fail)
The command in flight when site-a-a was **hard-killed** (SIGKILL) waited the full 30 s
`QueryTimeout` rather than failing fast on connect-refused: an already-dispatched gRPC call on a
dropped connection isn't observed as unsent, so it correctly cannot be auto-retried on the peer
node (it might have executed) and returns the deadline error to the caller — exactly the plan's
"deadline ≠ retry" rule. The deadline is the backstop; "no hang beyond deadline" is satisfied
(30.1 s). Only **provably-unsent** connect failures fail over fast, which is why every *subsequent*
call recovered immediately via site-a-b.
### Not driven on this gate (unchanged from 1B/Phase 2)
- **`ExecuteOpcUa`** (BrowseNode/ReadTagValues/WriteTag) — needs an OPC-bound deployed instance;
the bare rig's only deployable template (`SoakNotify`) has no data connection, and `Motor
Controller` needs 30 OPC UA bindings. Unit-proven (dispatcher routing ×28).
- **`ExecuteRoute`** (inbound-API → routed site script) — needs an inbound method + routing target
the rig lacks.
- **`TriggerFailover`** — no CLI verb (UI/management-only), destructive; unit-proven (ack-before-
`Leave` ordering tests). The hard-kill in check 6 is the live equivalent of a site-pair failover.
Rig left running with **both** transports on gRPC; git config reverted to Akka default (a redeploy
- [x] 1B DoD (proportionate): rig central on `SiteTransport=Grpc` proves central→site rides authenticated gRPC `SiteCommandService` for all 3 sites (`ExecuteQuery`/`ExecuteParked` → 200, per-site PSK, 0 auth failures); rebased on 1A. Instance-dependent commands (tag ops/lifecycle/standby parked retry) + `TriggerSiteFailover` deferred to Phase 3 (no deployed instance / destructive UI-only); command-plane coexistence not expressible (`SiteTransport` is central-wide). Gate: `2026-07-22-clusterclient-to-grpc-live-gate.md`
**Phase 2 ∥ 3 — cutover + soak**
- [] P2 All sites `CentralTransport=Grpc`; central-kill S&F soak (no loss/dupes), failback observed, health sequences clean
- [] P3 Central `SiteTransport=Grpc` all sites; full UI command matrix per site; site-kill mid-command clean; no PSK noise; zero ClusterClient log activity on flipped paths
- [x] P2 All sites `CentralTransport=Grpc`; central-kill S&F soak (no loss/dupes), failback observed, health sequences clean — **PASS 2026-07-23** (live gate: single-node active-kill failover 29s + full-outage ~59s both-down drain; every 5s bucket = exactly 3 through both outages, 216 total == 216 distinct; 0 auth failures / 0 seq regressions; whole control plane — heartbeat/health/notification/audit — on gRPC `CentralControlService`)
- [x] P3 Central `SiteTransport=Grpc` all sites; full UI command matrix per site; site-kill mid-command clean; no PSK noise; zero ClusterClient log activity on flipped paths — **PASS 2026-07-23** (live gate: `ExecuteQuery`/`ExecuteParked` all 3 sites + `ExecuteLifecycle` disable/enable on site-a, all 200 over `SiteCommandService`; active-site-node hard-kill mid-command → clean `TIMEOUT` at the 30s deadline, next call failed over to site-a-b automatically; 0 PermissionDenied / 0 ClusterClient-to-site on both central. `ExecuteOpcUa`/`ExecuteRoute`/`TriggerFailover` deferred — no OPC-bound instance / no CLI verb, unit-proven)
**Phase 4 — deletion**
- [ ] Defaults flip to `Grpc` + soak; then delete Akka transports, ClusterClient creation, `DefaultSiteClientFactory`, receptionist registrations, `CentralContactPoints`, then the flags
"activeForm":"Running the site->central cutover soak",
"blockedBy":[
"P1A.DoD"
@@ -208,7 +208,7 @@
"id":"P3",
"phase":"3",
"subject":"Central SiteTransport=Grpc all sites; full UI command matrix; site-kill mid-command clean; zero ClusterClient activity",
"status":"pending",
"status":"completed",
"activeForm":"Running the central->site cutover soak",
"blockedBy":[
"P1B.DoD"
Reference in New Issue
Block a user
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
Joseph Doherty