ScadaBridge/docs/plans/2026-07-22-clusterclient-to-grpc-live-gate.md

# ClusterClient → gRPC migration — live gate results

Rig: `docker/` (2 central + 3×2 site + traefik), rebuilt from the branch under test via
`bash docker/deploy.sh`. Recorded check-by-check in the family's live-gate format. Phase 5's
full eight-check gate is recorded further down as those phases land; this file starts with
Phase 0, whose DoD has its own smaller gate.

---

## Phase 0 — PSK auth + dead-code removal — **PASS** (2026-07-22)

Branch `feat/grpc-phase0-psk` @ `228ff8b4`. Image rebuilt, all 9 containers recreated.

### Baseline (pre-change build, same rig)

An unauthenticated call from the host to a site's audit-pull RPC was **accepted**:

```
$ grpcurl -plaintext -d '{"batch_size":1}' localhost:9023 sitestream.SiteStreamService/PullAuditEvents
{}
```

That is the gap Phase 0 closes, reproduced rather than assumed.

### Checks

| # | Check | Result |
|---|---|---|
| 1 | All 8 nodes boot with keys configured (new `StartupValidator` rule) | **PASS** — all recreated and reached ready |
| 2 | Unauthenticated `PullAuditEvents` ⇒ `PermissionDenied`, all 3 sites | **PASS** |
| 3 | Wrong key (site-b's key presented to site-a) ⇒ `PermissionDenied` | **PASS** — per-site scoping is real, not decorative |
| 4 | Correct key ⇒ success, all 3 sites | **PASS** |
| 5 | Central's own authenticated paths still work | **PASS** — 14 successful `PullAuditEvents` from central to site-a; **0** auth failures in either central's log |
| 6 | LocalDb sync unaffected by the new interceptor | **PASS** — **0** control-plane rejections and **0** sync auth failures on the passive peer; session connected after one boot-order retry |
| 7 | No interceptor-activation errors | **PASS** — 0 (see the defect below) |

Evidence for 2–4:

```
=== NO CREDENTIALS ===
  :9023 -> ERROR: Code: PermissionDenied  Message: Control plane authentication failed.
  :9033 -> ERROR: Code: PermissionDenied  Message: Control plane authentication failed.
  :9043 -> ERROR: Code: PermissionDenied  Message: Control plane authentication failed.

=== WRONG KEY (site-b's key against site-a) ===
  :9023 -> ERROR: Code: PermissionDenied  Message: Control plane authentication failed.

=== CORRECT KEY ===
  site-a :9023 -> {}
  site-b :9033 -> {}
  site-c :9043 -> {}
```

Site-a rejected exactly **2** calls — the two deliberate probes above — and nothing else.

### Defect the gate caught that the test suite did not

**First run of this gate FAILED**, and is worth recording because the failure mode is
deceptive.

`Grpc.AspNetCore` activates a type-registered interceptor through
`InterceptorRegistration.GetFactory()`, which throws when more than one public constructor is
applicable. `ControlPlaneAuthInterceptor` shipped with two — the DI one and a prefix-set
overload intended for later phases.

The throw happens **inside the pipeline, per call**, so:

- nothing failed at startup; the node booted, joined its pair and reported healthy;
- every gated call died with `Unknown / "Exception was thrown by handler"`, which reads as a
  handler bug rather than an auth bug;
- **correct key, wrong key and no key produced identical errors** — the tell. A gate that
  cannot distinguish those is not authenticating anything.

Site-a's log at the time: three `PullAuditEvents` calls, three identical
`System.InvalidOperationException: Multiple constructors accepting all given argument types
have been found in type 'ControlPlaneAuthInterceptor'`.

The full suite was green when this shipped — **29 suites, 6,872 tests, 0 failures**. The
in-process end-to-end test missed it because it registered the interceptor with
`AddSingleton` alongside `AddGrpc`, so DI returned the instance and gRPC's activation path
never ran.

Fixed in `228ff8b4`: the prefix-set constructor is `internal`; the end-to-end harness now
registers exactly as `Program.cs` does (by type, not in DI); and a reflection assertion pins
"exactly one public constructor", since that is the actual invariant.

**Lesson for phases 1A/1B, which both add services to this interceptor:** extend
`DefaultGatedPrefixes`; do not add a second public constructor. And any in-process harness for
a DI-activated component must mirror the production registration shape or it proves less than
it appears to.

### Test suite alongside the gate

Non-Playwright: **29 suites, 6,872 tests, 0 failures**.

Playwright (against this rig): **170 passed, 2 failed, 1 skipped** of 173. Both failures were
run down to root cause and **both are pre-existing on `main`, unrelated to Phase 0** — this
branch touches no EF, CentralUI, Transport or ManagementService file (`git diff --stat
main...HEAD -- src/` is 15 files, all Communication/Host/AuditLog gRPC plumbing).

An earlier run of this suite reported 44 failures. That run is **void**: a `docker/deploy.sh`
was recreating the cluster underneath it, so the fast `LoginTests`/`NavigationTests` failures
were "app unreachable", not defects.

**1. `TransportImportTests.ImportSyntheticBundle_AppliesAndShowsAuditDrillIn` — a real
production bug, not a test defect.** Central's log during the failure:

```
[ERR] An exception occurred while iterating over the results of a query ...
System.InvalidOperationException: The configured execution strategy
'SqlServerRetryingExecutionStrategy' does not support user-initiated transactions.
  at Microsoft.EntityFrameworkCore.Query.Internal.SplitQueryingEnumerable`1.AsyncEnumerator.MoveNextAsync()
```

`BundleImporter.cs:1298` opens a user-initiated transaction; the central context is configured
with `EnableRetryOnFailure` (`ConfigurationDatabase/ServiceCollectionExtensions.cs:33`). SQL
Server's retrying strategy refuses to run a split query inside a caller's transaction, so
**bundle import fails against real MS SQL**. The fix is the one the exception names: wrap the
transaction in `Database.CreateExecutionStrategy().ExecuteAsync(...)`.

Why the whole unit/integration suite is green on it: those tests use the in-memory EF provider,
which has no retrying execution strategy — and `BeginTransactionAsync` is a no-op there. The
comment directly above line 1298 documents that divergence without drawing the conclusion. Only
a rig-backed test can see this.

**2. `SmsNotificationE2ETests.SmsConfigPage_CreateOrRender_NeverLeaksAuthToken` — a stale test
fixture.** No server-side error at all: the page renders 200, and no `INSERT INTO
SmsConfigurations` is ever issued. The test's fixture SID is `ACtest123` (`d6ead8ae`,
2026-06-19). `SmsConfiguration.razor:231` rejects anything not matching `^AC[0-9a-fA-F]{32}$`,
added by `40088a21` (2026-07-10) to close an un-escaped URI-interpolation hole. `Save()` sets
`_formError` and returns — no toast, exactly as observed. The fixture was never updated.

This has been failing since 2026-07-10, and it matters more than a red line: everything after
the toast assertion — including **the secret-non-leak assertion that the Auth Token value never
reaches the page HTML** — has not executed since. Fix is a valid 32-hex SID in the fixture.

### Not covered by this gate

- Streaming subscriptions were exercised in-process (TestServer), not over the rig. The
  interceptor is path-scoped, not method-scoped, so the rig's `PullAuditEvents` evidence covers
  the same code path — but a live `SubscribeInstance` under load is untested here.
- Key **rotation** on a live pair.
- `docker-env2` was updated with its own key but not redeployed or gated.

---

## Phase 1A — central control plane (site→central over gRPC) — **PASS** (2026-07-22)

Branch `feat/grpc-central-control` @ `0e162cb2`. Rig rebuilt; **site-a flipped to
`CentralTransport=Grpc`** with `CentralGrpcEndpoints=[central-a:8083, central-b:8083]`,
**site-b/c left on Akka** (default) to prove coexistence. The site-a flag flip was a
DoD-test-only rig edit — reverted from the branch, never committed (the plan keeps the default
`Akka` until Phase 4).

### The defect this gate caught (T1A.2 shipped it; fixed in `0e162cb2`)

**First rebuild: central's entire HTTP surface was gone.** central-a logged only
`Now listening on: http://[::]:8083` — no `:5000`. Central UI, the Management + Inbound API,
and every `/health/*` endpoint (Traefik routing + `IActiveNodeGate` both depend on them) were
dead. The node booted, joined the cluster and served gRPC fine; **no startup error.**

Cause: `builder.WebHost.ConfigureKestrel(o => o.ListenAnyIP(8083, Http2))` puts Kestrel into
explicit-endpoints mode, which **suppresses the URLs from `ASPNETCORE_URLS`/`--urls`** — it is
not additive, contrary to the comment T1A.2 shipped. Central's whole HTTP/1 surface lives on
that URL (`http://+:5000` on the rig; a different port in production). The site branch has the
same `ConfigureKestrel` shape but nothing on `ASPNETCORE_URLS` to lose — it binds every port it
needs explicitly — which is why the pattern looked safe.

Every unit + E2E test uses `TestServer`, which never binds real Kestrel, so the whole suite
(6,872) stayed green. Only a live node exposes a missing listener. Fix: parse the port(s) from
the configured URLs and re-declare them (`Http1AndHttp2`) alongside the gRPC port (`Http2`) in
the one `ConfigureKestrel` call — `Program.ParseHttpBindPorts` + `CentralHttpBindPortsTests`
(14 cases). After the fix: `Now listening on: http://[::]:5000` **and** `:8083`; 9001 ready
`200`/active Healthy, 9002 standby, Traefik LB `200`, CLI over the LB works.

### Checks (post-fix rebuild)

| # | Check | Result |
|---|---|---|
| 1 | site-a rides authenticated gRPC to `CentralControlService` | **PASS** — Heartbeat, `ReportSiteHealth`, `ReconcileSite` all HTTP/2 → 200; **0** auth failures on either central |
| 2 | Heartbeat drives the active flag | **PASS** — 144+ heartbeats over gRPC; site-a `online=True` at central |
| 3 | Health page live | **PASS** — `ReportSiteHealth` lands; central shows site-a `online=True`, sequence advancing, alongside Akka site-b/c |
| 4 | Reconcile works after site restart | **PASS** — restarted `site-a-a`; it logged `Site→central transport: gRPC to 2 central endpoint(s)`, then `Reconcile pass … complete: 0 fetched, 0 failed, 0 orphan(s)` |
| 5 | Coexistence | **PASS** — site-b/c log `Created ClusterClient to central`; both `online=True` — Akka and gRPC sites side by side |
| 6 | Central HTTP surface intact under the new gRPC listener | **PASS** — after the fix (see above) |

### Not independently exercised on this gate

- **Notification e2e (`SubmitNotification`/`QueryNotificationStatus`) and audit ingest
  (`IngestAuditEvents`/`IngestCachedTelemetry`)** were NOT driven live: the rig has templates
  but **no deployed instance**, so nothing emits site→central notifications or audit rows on its
  own. These four RPCs traverse the identical `CentralControlGrpcService` →
  `CentralCommunicationActor` Ask path that checks 1–3 proved live under real auth, and their
  payload mappers carry 32 round-trip goldens — but the payloads themselves were not put over
  the wire here. **Phase 2's central-kill S&F soak is where they get their live workout;** flag
  for a fuller 1A proof if a deployed-instance rig is set up before then.
- Cross-node failover/failback of the site→central channel under a central-node kill (unit-proven
  via TestServer; not exercised on the rig at 1A).

---

## Phase 1B — site command plane (central→site over gRPC) — **PASS (proportionate)** (2026-07-23)

Branch `feat/grpc-sitecommand` (rebased onto 1A-merged main). Rig rebuilt with **central
flipped to `SiteTransport=Grpc`** — a DoD-test-only edit to both central appsettings, reverted
from the branch (default stays `Akka`).

`SiteTransport` is a **central-wide** flag (`CentralCommunicationActor.SelectTransport` picks one
transport for all sites), so the plan's "flip for site-a only" is not achievable — the flip
routes central→site commands for **all three sites** to gRPC. Command-plane per-site coexistence
therefore cannot be shown (unlike the site→central plane in 1A, which is per-site). This is a
plan-vs-code finding, recorded rather than worked around.

### Checks

| # | Check | Result |
|---|---|---|
| 1 | central→site commands ride authenticated gRPC `SiteCommandService` | **PASS** — `ExecuteQuery` (event-log) and `ExecuteParked` (parked query) HTTP/2 → 200 |
| 2 | Per-site PSK resolution across all sites | **PASS** — site-a, site-b, site-c each answered `ExecuteQuery` → 200 under its own `SB-GRPC-PSK-{site}`; **0** auth failures on any site node |
| 3 | Central HTTP surface intact under the central-wide gRPC flip | **PASS** — central `:5000` **and** `:8083` both listening; 9001 ready `200`, LB `200` (the 1A Kestrel fix carried through the merge) |
| 4 | Query round-trips return correct data | **PASS** — every `health event-log`/`parked-messages` returned `success:true` with the right `siteId` and empty result sets (bare rig) |

### Not driven on this proportionate gate

- **`TriggerSiteFailover`** — unit-proven (two ordering tests pin ack-before-`Leave` via the
  dispatcher's dry-run resolve + deferred `CommitLeave`), but not live-driven here: it is
  destructive (forces the active node to leave) and has no CLI verb (UI/management-only).
- **Tag commands (`BrowseNode`/`ReadTagValues`/`WriteTag`) and the lifecycle enable/disable
  matrix** — need a deployed instance + data connection the bare rig lacks (same blocker as 1A).
- **Parked retry against the STANDBY node** — needs a parked operation to exist, which needs a
  deployed instance.
- **Command-plane coexistence (site-b/c on Akka while site-a on gRPC)** — not expressible; the
  flag is central-wide (check-1/2 instead prove all three sites over gRPC with distinct keys).

The instance-dependent matrix (tag ops, lifecycle, standby parked retry) and `TriggerSiteFailover`
get their live exercise in Phase 3's full UI command matrix; the transport itself is proven here.

---

## Phase 2 — full site→central cutover + S&F soak — **PASS** (2026-07-23)

All three sites (**all 6 nodes**) flipped to `CentralTransport=Grpc` (edit to
`docker/site-*/appsettings.Site.json`, reverted in git after the gate — defaults stay Akka),
rebuilt from `main` + `--force-recreate`. Central left on `SiteTransport=Akka` (Phase 3 owns
that direction). The gate was driven by a **live S&F workload**, closing the notification/audit
path that 1A/1B could only unit-prove.

### S&F driver

A minimal dependency-free template `SoakNotify` (one 5 s `Interval` script:
`Notify.To("Engineering Alerts").Send(...)`), 3 instances deployed+enabled on site-a → a steady
**3 notifications per 5 s bucket** (the pre-existing `Motor Controller` "soak-motor" instances
need 30 OPC UA bindings and were unusable). The central `dbo.Notifications` table (one row per
`NotificationId`, insert-if-not-exists) is the no-loss/no-dupes source of truth.

### Checks

| # | Check | Result |
|---|---|---|
| 1 | All 6 site nodes on `CentralTransport=Grpc` | **PASS** — each logs `Site→central transport: gRPC to 2 central endpoint(s)`; **0** `PermissionDenied` across all six for the whole run |
| 2 | Full control plane rides gRPC `CentralControlService` | **PASS** — central sees `Heartbeat`, `ReportSiteHealth`, `SubmitNotification` (the S&F path), `IngestAuditEvents` — **196 RPCs / 90 s, 0 non-200** |
| 3 | Health/heartbeat cadence unchanged, no sequence regressions | **PASS** — `CentralHealthAggregator` logged **0** sequence-regression/out-of-order lines; heartbeat steady at ~144/30 s across 3 sites |
| 4 | **Single-node failover** — hard-kill the **active** central (central-a) | **PASS** — `CentralChannelProvider` logged sticky failover `central-a:8083 → central-b:8083` at the instant of kill; central-b active in **29 s** (auto-down); notif count froze at 72 during the gap |
| 5 | Buffer drains, no loss/dupes (single-node) | **PASS** — count resumed 72→101; **every 5 s bucket through the outage = exactly 3**, no gap; 101 total == 101 distinct |
| 6 | **Failback** — restart central-a | **PASS** — rejoined **ready in ~5 s as standby** (`active=503`); central-b **retained active** (oldest-Up, no role flap); traffic uninterrupted (uniform 3/bucket across failback); central-a singletons → `Younger` |
| 7 | **Full central outage** — hard-kill **both** central (~59 s) | **PASS** — count frozen at 155 for the entire outage; sites buffered continuously |
| 8 | Cold re-form + drain, no loss/dupes (both-down) | **PASS** — cold cluster re-formed, central-b active in **~14 s**; ~42 buffered notifications drained; **every 5 s bucket across the whole ~59 s both-dead window = exactly 3**, no gap; **216 total == 216 distinct** |

Checks 4–8 also stand as a live preview of Phase 5 checks 4 (failover/failback) and 5 (mid-drain
kill, zero loss/zero dupes).

### Notes

- **Audit telemetry also rides the new gRPC `CentralControlService`** (`IngestAuditEvents`) — the
  `SiteAuditTelemetryActor` "ClusterClientSiteAuditClient" label is legacy naming, not the wire
  path. So `CentralTransport=Grpc` moves heartbeat, health, notification S&F **and** audit off
  ClusterClient in one flip.
- Sites settled on **central-a** as the gRPC endpoint after the cold both-restart while **central-b**
  held the active/singleton role — the gRPC endpoint node and the singleton host can differ; central-a
  receives the forward and Akka-routes to the `NotificationOutboxActor` singleton on central-b. Both
  are correct and independent.
- Some notifications land `Parked` at central (no SMTP config on the bare rig) — irrelevant to the
  transport proof: the `Notifications` **row** is written on forward regardless of downstream SMTP
  delivery, so the count is a faithful no-loss/no-dupes measure.
- Rig left running on the gRPC build; git config reverted to Akka default (a redeploy from `main`
  resets to all-Akka).

---

## Phase 3 — full central→site cutover + command matrix — **PASS** (2026-07-23)

Central flipped to `SiteTransport=Grpc` (both central nodes; the flag is **central-wide** —
`CentralCommunicationActor.SelectTransport`). Sites kept on `CentralTransport=Grpc` from Phase 2,
so the rig ran **both directions on gRPC simultaneously** — the eventual all-gRPC end state.
Config bind-mounted (`:ro`), so a container `--force-recreate` (no image rebuild — code unchanged
since Phase 2) applied it. Central logged the selection at startup:
`central→site command transport: gRPC (SiteCommandService)`.

The Phase 2 `SoakNotify` instances (#95–97) **survived the recreate as Enabled** (site volume
persisted `deployed_configurations` this time), so the notification workload kept flowing —
and lifecycle commands became drivable, closing the 1B/Phase-2 gap.

### Checks

| # | Check | Result |
|---|---|---|
| 1 | Central selects the gRPC command transport | **PASS** — `central→site command transport: gRPC (SiteCommandService)`; **0** `Created ClusterClient to` site lines on either central |
| 2 | `ExecuteQuery` (event-log) over gRPC, all 3 sites | **PASS** — all 3 returned correlationIds; site logs `SiteCommandService/ExecuteQuery - 200` (site-a-a/b-a/c-a each served the calls; standby nodes 0 — work lands on the active site node via the singleton proxy) |
| 3 | `ExecuteParked` (parked query) over gRPC, all 3 sites | **PASS** — all 3 returned `success` payloads over `SiteCommandService/ExecuteParked` |
| 4 | `ExecuteLifecycle` (disable→enable) over gRPC, site-a | **PASS** — `instance disable`/`enable #95` → `success:true`; site served `SiteCommandService/ExecuteLifecycle` (×4), instance state toggled — **NEW live coverage vs 1B/Phase-2 (needed a deployed instance)** |
| 5 | **Site-node kill mid-command → clean error, no hang** | **PASS** — killed the **active** site-a node (site-a-a) during a 1 s query loop: the in-flight call returned `TIMEOUT` at **dur=30.1 s = the `QueryTimeout` deadline** — bounded, not an indefinite hang (see note) |
| 6 | **Site-pair failover mid-stream** | **PASS** — the very next query (~32 s after kill) and all subsequent ones succeeded automatically via **site-a-b**; `SitePairChannelProvider` failed the gRPC channel NodeA→NodeB and the site singleton migrated; site→central S&F never stopped (notif count climbed 619→715 through the kill, still no dupes) |
| 7 | No PSK drift | **PASS** — **0** `PermissionDenied`/`Unauthenticated` across all 8 nodes for the whole run |
| 8 | Zero ClusterClient activity on the flipped path | **PASS** — central built no site ClusterClients; command routing is entirely `SiteCommandService` gRPC |

### Note on check 5 (deadline vs fast-fail)

The command in flight when site-a-a was **hard-killed** (SIGKILL) waited the full 30 s
`QueryTimeout` rather than failing fast on connect-refused: an already-dispatched gRPC call on a
dropped connection isn't observed as unsent, so it correctly cannot be auto-retried on the peer
node (it might have executed) and returns the deadline error to the caller — exactly the plan's
"deadline ≠ retry" rule. The deadline is the backstop; "no hang beyond deadline" is satisfied
(30.1 s). Only **provably-unsent** connect failures fail over fast, which is why every *subsequent*
call recovered immediately via site-a-b.

### Not driven on this gate (unchanged from 1B/Phase 2)

- **`ExecuteOpcUa`** (BrowseNode/ReadTagValues/WriteTag) — needs an OPC-bound deployed instance;
  the bare rig's only deployable template (`SoakNotify`) has no data connection, and `Motor
  Controller` needs 30 OPC UA bindings. Unit-proven (dispatcher routing ×28).
- **`ExecuteRoute`** (inbound-API → routed site script) — needs an inbound method + routing target
  the rig lacks.
- **`TriggerFailover`** — no CLI verb (UI/management-only), destructive; unit-proven (ack-before-
  `Leave` ordering tests). The hard-kill in check 6 is the live equivalent of a site-pair failover.

Rig left running with **both** transports on gRPC; git config reverted to Akka default (a redeploy
from `main` resets to all-Akka).

---

## Phase 5 — full eight-check gate on the deletion build — **PASS** (2026-07-23)

The migration's terminal gate, run on `main` **after Phase 4** (`7fd5cb2b`) — the build where the
Akka `ClusterClient`/`ClusterClientReceptionist` transport is **physically deleted**, the
`CentralTransport`/`SiteTransport` flags are gone, and gRPC is the *only* site↔central transport
(no flags to flip). Rig rebuilt from `main` via `bash docker/deploy.sh`, all 9 containers recreated
from the new image; central MS SQL and the S&F driver (`SoakNotify` template #2147 / instances
#95–97 / list "Engineering Alerts" #28) persisted from prior phases.

**Provenance guard:** the pre-existing rig image was built `11:17`, *before* the Phase 4 commit
(`12:54`); it still carried ClusterClient code and would have invalidated checks 7–8. Confirmed
by rebuild timestamp, then by **0** ClusterClient/receptionist log lines across all 8 nodes on the
fresh boot. (The lone `ClusterClientReceptionist` symbol still present is inside
`Akka.Cluster.Tools.dll` — the library we deliberately keep for `ClusterSingleton` — not our code.)

Header names for the negative probes: `authorization: Bearer <psk>` + `x-scadabridge-site: <siteId>`
(`ISitePskProvider.cs`). Docker rig PSKs: `dev-grpc-psk-docker-site-{a,b,c}`
(`ScadaBridge__Communication__SitePsks__site-*`, central override; sites read their own
`GrpcPsk` from mounted `appsettings.Site.json`).

### Checks

| # | Check | Result |
|---|---|---|
| 1 | **PSK negatives** | **PASS (with a contract clarification)** — see below |
| 2 | **Site→central matrix** | **PASS** — notifications, both audit paths, health, heartbeat, reconcile all live over gRPC |
| 3 | **Central→site matrix** | **PASS (proportionate)** — `ExecuteQuery`/`ExecuteParked`/`ExecuteLifecycle` live; instance-dependent RPCs carried forward (see below) |
| 4 | **Failover / failback** | **PASS** — active-central kill → sticky flip in 1 s, central-b active in 26 s; failback kept central-b active (oldest-Up, no flap) |
| 5 | **Mid-drain kill, zero loss/dupes** | **PASS** — buffered notifications flushed; **total == distinct** across the outage |
| 6 | **Frame-class retirement (>128 KB)** | **PASS** — a single **297,574-byte** gRPC reply succeeded (Akka's 128 KB frame would have dropped it) |
| 7 | **No cross-boundary Akka association** | **PASS** — each Akka cluster's membership is strictly its own pair; **0** cross-boundary association lines both directions |
| 8 | **Full-rig restart discipline** | **PASS** — all 9 restarted together came up clean; **0** receptionist/ClusterClient lines on any node; sites reconnected over gRPC; S&F resumed with no dupes |

### Check 1 — PSK negatives (and the contract clarification)

Against the two gated services on site-a (`:9023`), using the local proto descriptors
(`-proto sitestream.proto` / `site_command.proto`; server reflection is off in this build):

```
SiteStreamService/PullAuditEvents:
  no credentials           -> PermissionDenied "Control plane authentication failed."
  wrong key (site-b's key) -> PermissionDenied
  correct key              -> success (audit events returned)
SiteCommandService/ExecuteQuery:
  no credentials           -> PermissionDenied
  wrong key (site-b's key) -> PermissionDenied
```

site-a-a's log recorded **exactly 4** control-plane rejections — the four deliberate negative
probes above, nothing else.

**Clarification the gate produced:** the plan listed "missing `x-scadabridge-site` ⇒
`PermissionDenied`" as a negative case. It does **not** hold on the *site* side, and that is
correct by design. `ControlPlaneAuthInterceptor.Authorize` compares the presented bearer against
the node's **own single** `GrpcPsk` (`_options.Value.GrpcPsk`) and never reads
`x-scadabridge-site`. That header is a **central-side routing hint** — `SitePskProvider` uses it to
pick *which* site's key to expect, because central holds many. A single-key site node needs no such
hint. Per-site key isolation — the actual security property — is proven by the **wrong-key**
rejection (site-b's key refused at site-a), not by the header. Recorded rather than "fixed": adding
a header requirement to the site interceptor would be theatre.

**LocalDb sync unaffected:** the replicated node (site-a) logged one boot-order
`faulted; reconnecting` (peer node-b not yet up), the passive peer accepted the
`/localdb_sync.v1.LocalDbSync/Sync` POST 1 s later, and the health report settled to
`localDbReplicationConnected:true, localDbOplogBacklog:0`. **0** LocalDb sync auth failures — the
control-plane interceptor and the separate `LocalDbSyncAuthInterceptor` don't interfere. site-b/c
report `localDbReplicationConnected:false` — the intended rig posture (only site-a replicated).

### Check 2 — site→central matrix

- **Notifications e2e:** `dbo.Notifications` climbed continuously at the driver's ~3 rows / 5 s,
  **total == distinct** at every sample (insert-if-not-exists no-loss/no-dupe truth).
- **Both audit paths live over gRPC:** in a 3-minute window, `dbo.AuditLog` carried `node-a`/`node-b`
  rows (105 each — **site-originated**, forwarded over the gRPC `IngestAuditEvents` RPC) *and*
  `central-a` rows (420 — central-direct-write from the outbox dispatcher). Row count climbing live.
- **Health live per site:** `health summary` shows all three sites `isOnline:true`, sequence numbers
  advancing, fresh heartbeats; site-a `enabledInstanceCount:3` with a live `Notification` S&F buffer.
- **Heartbeat → active flag:** central-a `active=200`, central-b `active=503`; each site's report
  drives its online flag.
- **Reconcile self-heal:** restarting `site-a-a` produced a fresh (17:11:24) `Site→central transport:
  gRPC to 2 central endpoint(s)` then `Reconcile pass … complete: 0 fetched, 0 failed, 0 orphan(s)`;
  S&F never dropped across the restart (active node kept emitting), no dupes.
- **Not driven (carry-forward, bare rig):** cached-call telemetry (`SoakNotify` only calls
  `Notify.Send`, no `CachedCall`/`CachedWrite`); a notification reaching **Delivered** rather than
  `Parked` (no SMTP on the rig — the transport truth is the row-count, per Phase 2).

### Check 3 — central→site matrix

`ExecuteQuery` (event-log, all 3 sites → correlationIds + entries), `ExecuteParked` (parked-messages,
all 3 sites → empty, no parked ops on a bare rig), and `ExecuteLifecycle` (disable→enable #95 →
`success:true` both) all rode `SiteCommandService` gRPC; site active nodes logged them
(site-a-a: 8 `ExecuteLifecycle` + 4 `ExecuteParked` + 4 `ExecuteQuery`; site-b-a/c-a:
`ExecuteParked` + `ExecuteQuery`). **Carried forward, unchanged from Phase 3** (needs a deployed
OPC-bound instance / inbound method / a parked op / the destructive UI-only failover verb the bare
rig lacks): `ExecuteOpcUa`, `ExecuteRoute`, standby parked retry/discard, `TriggerFailover`
(the hard-kill in Check 4 is its live equivalent). All four are unit-proven (dispatcher routing ×28,
ack-before-`Leave` ordering).

### Checks 4 & 5 — failover / failback / mid-drain kill

Hard-killed the **active** central (central-a) mid-drain: site-a-b logged the sticky endpoint flip
`central-a:8083 → central-b:8083` **1 s** after the kill; central-b reached active in **26 s**
(auto-down). The notification count froze during the gap, then **flushed the buffered rows and
resumed** at the steady 3 / 5 s. Post-drain **total == distinct (4395 == 4395)** — zero loss, zero
duplicates. Failback: restarting central-a, it rejoined and central-b **retained** active
(`a=503` standby / `b=200` active) — oldest-Up, no role flap.

### Check 6 — frame-class retirement

`PullAuditEvents{since_utc: 2020-01-01, batch_size: 5000}` against the active site node returned a
single **297,574-byte** reply successfully. Over Akka's default 128 KB frame
(`log-frame-size-exceeding` off) this class of payload was silently dropped and the caller's Ask
timed out (`docs/known-issues/2026-06-26-…`). On gRPC (4 MB default cap) it is a normal reply.

### Check 7 — no cross-boundary Akka association

Each Akka cluster's membership is strictly its own pair: central `[central-a, central-b]`, site-a
`[site-a-a, site-a-b]`, site-c `[site-c-a, site-c-b]` — **no site node ever appears in central's
membership and vice versa**. Central's log holds **0** `Association with remote system …site-*`
lines and **0** references to any site address; site-a's log holds **0** references to any central
address. The two boundaries are wired only by per-pair Akka remoting (8082, internal) and the gRPC
control/data planes (8083); there is no Akka association across the site↔central line.

### Check 8 — full-rig restart discipline

Restarted all 9 rig containers together (pairs together, honouring the LocalDb-replication constraint).
Central re-formed with central-a active; **0** receptionist/ClusterClient lines across all 8 nodes;
all three sites logged `Site→central transport: gRPC to 2 central endpoint(s)`; S&F resumed
(4407 → 4483, **distinct == total**, no dupes).

### End state

The migration is complete and proven on the deletion build. The rig runs the committed all-gRPC
default (Phase 4 flipped the shipped appsettings to `CentralGrpcEndpoints` — unlike Phases 2/3, a
redeploy from `main` now yields all-gRPC, not all-Akka). No ClusterClient remains anywhere in the
running system or the source tree. Deferred, unchanged from earlier phases: the instance-dependent
central→site RPC matrix (`ExecuteOpcUa`/`ExecuteRoute`/standby parked retry) and live
`TriggerFailover`, all unit-proven; PSK **rotation** on a live pair; `docker-env2` was updated with
its own key but not gated here.