docs(grpc): Phase 1A live gate — PASS (3 RPCs + restart-reconcile + coexistence); records the Kestrel-drop defect

2026-07-22 19:55:09 -04:00
parent 0e162cb250
commit f7c7811940
1 changed files with 55 additions and 0 deletions
@@ -141,3 +141,58 @@ reaches the page HTML** — has not executed since. Fix is a valid 32-hex SID in
  the same code path — but a live `SubscribeInstance` under load is untested here.
 - Key **rotation** on a live pair.
 - `docker-env2` was updated with its own key but not redeployed or gated.
+
+---
+
+## Phase 1A — central control plane (site→central over gRPC) — **PASS** (2026-07-22)
+
+Branch `feat/grpc-central-control` @ `0e162cb2`. Rig rebuilt; **site-a flipped to
+`CentralTransport=Grpc`** with `CentralGrpcEndpoints=[central-a:8083, central-b:8083]`,
+**site-b/c left on Akka** (default) to prove coexistence. The site-a flag flip was a
+DoD-test-only rig edit — reverted from the branch, never committed (the plan keeps the default
+`Akka` until Phase 4).
+
+### The defect this gate caught (T1A.2 shipped it; fixed in `0e162cb2`)
+
+**First rebuild: central's entire HTTP surface was gone.** central-a logged only
+`Now listening on: http://[::]:8083` — no `:5000`. Central UI, the Management + Inbound API,
+and every `/health/*` endpoint (Traefik routing + `IActiveNodeGate` both depend on them) were
+dead. The node booted, joined the cluster and served gRPC fine; **no startup error.**
+
+Cause: `builder.WebHost.ConfigureKestrel(o => o.ListenAnyIP(8083, Http2))` puts Kestrel into
+explicit-endpoints mode, which **suppresses the URLs from `ASPNETCORE_URLS`/`--urls`** — it is
+not additive, contrary to the comment T1A.2 shipped. Central's whole HTTP/1 surface lives on
+that URL (`http://+:5000` on the rig; a different port in production). The site branch has the
+same `ConfigureKestrel` shape but nothing on `ASPNETCORE_URLS` to lose — it binds every port it
+needs explicitly — which is why the pattern looked safe.
+
+Every unit + E2E test uses `TestServer`, which never binds real Kestrel, so the whole suite
+(6,872) stayed green. Only a live node exposes a missing listener. Fix: parse the port(s) from
+the configured URLs and re-declare them (`Http1AndHttp2`) alongside the gRPC port (`Http2`) in
+the one `ConfigureKestrel` call — `Program.ParseHttpBindPorts` + `CentralHttpBindPortsTests`
+(14 cases). After the fix: `Now listening on: http://[::]:5000` **and** `:8083`; 9001 ready
+`200`/active Healthy, 9002 standby, Traefik LB `200`, CLI over the LB works.
+
+### Checks (post-fix rebuild)
+
+| # | Check | Result |
+|---|---|---|
+| 1 | site-a rides authenticated gRPC to `CentralControlService` | **PASS** — Heartbeat, `ReportSiteHealth`, `ReconcileSite` all HTTP/2 → 200; **0** auth failures on either central |
+| 2 | Heartbeat drives the active flag | **PASS** — 144+ heartbeats over gRPC; site-a `online=True` at central |
+| 3 | Health page live | **PASS** — `ReportSiteHealth` lands; central shows site-a `online=True`, sequence advancing, alongside Akka site-b/c |
+| 4 | Reconcile works after site restart | **PASS** — restarted `site-a-a`; it logged `Site→central transport: gRPC to 2 central endpoint(s)`, then `Reconcile pass … complete: 0 fetched, 0 failed, 0 orphan(s)` |
+| 5 | Coexistence | **PASS** — site-b/c log `Created ClusterClient to central`; both `online=True` — Akka and gRPC sites side by side |
+| 6 | Central HTTP surface intact under the new gRPC listener | **PASS** — after the fix (see above) |
+
+### Not independently exercised on this gate
+
+- **Notification e2e (`SubmitNotification`/`QueryNotificationStatus`) and audit ingest
+  (`IngestAuditEvents`/`IngestCachedTelemetry`)** were NOT driven live: the rig has templates
+  but **no deployed instance**, so nothing emits site→central notifications or audit rows on its
+  own. These four RPCs traverse the identical `CentralControlGrpcService` →
+  `CentralCommunicationActor` Ask path that checks 1–3 proved live under real auth, and their
+  payload mappers carry 32 round-trip goldens — but the payloads themselves were not put over
+  the wire here. **Phase 2's central-kill S&F soak is where they get their live workout;** flag
+  for a fuller 1A proof if a deployed-instance rig is set up before then.
+- Cross-node failover/failback of the site→central channel under a central-node kill (unit-proven
+  via TestServer; not exercised on the rig at 1A).