docs(grpc): Phase 0 live gate PASS — record results, the inert-gate defect, and the trap for phases 1A/1B
The gate's first run failed on a defect the green suite could not see: two public constructors on ControlPlaneAuthInterceptor made Grpc.AspNetCore's activation throw per call, so correct key, wrong key and no key all produced identical errors. Recorded in full because the symptom (Unknown / "Exception was thrown by handler") points at the handler, not at auth, and because phases 1A/1B both add services to this same interceptor — they must extend DefaultGatedPrefixes rather than add a second public constructor. Also records what the gate does NOT cover: live streaming under load, key rotation on a running pair, and docker-env2 (keyed but neither redeployed nor gated).
This commit is contained in:
@@ -0,0 +1,98 @@
|
|||||||
|
# ClusterClient → gRPC migration — live gate results
|
||||||
|
|
||||||
|
Rig: `docker/` (2 central + 3×2 site + traefik), rebuilt from the branch under test via
|
||||||
|
`bash docker/deploy.sh`. Recorded check-by-check in the family's live-gate format. Phase 5's
|
||||||
|
full eight-check gate is recorded further down as those phases land; this file starts with
|
||||||
|
Phase 0, whose DoD has its own smaller gate.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Phase 0 — PSK auth + dead-code removal — **PASS** (2026-07-22)
|
||||||
|
|
||||||
|
Branch `feat/grpc-phase0-psk` @ `228ff8b4`. Image rebuilt, all 9 containers recreated.
|
||||||
|
|
||||||
|
### Baseline (pre-change build, same rig)
|
||||||
|
|
||||||
|
An unauthenticated call from the host to a site's audit-pull RPC was **accepted**:
|
||||||
|
|
||||||
|
```
|
||||||
|
$ grpcurl -plaintext -d '{"batch_size":1}' localhost:9023 sitestream.SiteStreamService/PullAuditEvents
|
||||||
|
{}
|
||||||
|
```
|
||||||
|
|
||||||
|
That is the gap Phase 0 closes, reproduced rather than assumed.
|
||||||
|
|
||||||
|
### Checks
|
||||||
|
|
||||||
|
| # | Check | Result |
|
||||||
|
|---|---|---|
|
||||||
|
| 1 | All 8 nodes boot with keys configured (new `StartupValidator` rule) | **PASS** — all recreated and reached ready |
|
||||||
|
| 2 | Unauthenticated `PullAuditEvents` ⇒ `PermissionDenied`, all 3 sites | **PASS** |
|
||||||
|
| 3 | Wrong key (site-b's key presented to site-a) ⇒ `PermissionDenied` | **PASS** — per-site scoping is real, not decorative |
|
||||||
|
| 4 | Correct key ⇒ success, all 3 sites | **PASS** |
|
||||||
|
| 5 | Central's own authenticated paths still work | **PASS** — 14 successful `PullAuditEvents` from central to site-a; **0** auth failures in either central's log |
|
||||||
|
| 6 | LocalDb sync unaffected by the new interceptor | **PASS** — **0** control-plane rejections and **0** sync auth failures on the passive peer; session connected after one boot-order retry |
|
||||||
|
| 7 | No interceptor-activation errors | **PASS** — 0 (see the defect below) |
|
||||||
|
|
||||||
|
Evidence for 2–4:
|
||||||
|
|
||||||
|
```
|
||||||
|
=== NO CREDENTIALS ===
|
||||||
|
:9023 -> ERROR: Code: PermissionDenied Message: Control plane authentication failed.
|
||||||
|
:9033 -> ERROR: Code: PermissionDenied Message: Control plane authentication failed.
|
||||||
|
:9043 -> ERROR: Code: PermissionDenied Message: Control plane authentication failed.
|
||||||
|
|
||||||
|
=== WRONG KEY (site-b's key against site-a) ===
|
||||||
|
:9023 -> ERROR: Code: PermissionDenied Message: Control plane authentication failed.
|
||||||
|
|
||||||
|
=== CORRECT KEY ===
|
||||||
|
site-a :9023 -> {}
|
||||||
|
site-b :9033 -> {}
|
||||||
|
site-c :9043 -> {}
|
||||||
|
```
|
||||||
|
|
||||||
|
Site-a rejected exactly **2** calls — the two deliberate probes above — and nothing else.
|
||||||
|
|
||||||
|
### Defect the gate caught that the test suite did not
|
||||||
|
|
||||||
|
**First run of this gate FAILED**, and is worth recording because the failure mode is
|
||||||
|
deceptive.
|
||||||
|
|
||||||
|
`Grpc.AspNetCore` activates a type-registered interceptor through
|
||||||
|
`InterceptorRegistration.GetFactory()`, which throws when more than one public constructor is
|
||||||
|
applicable. `ControlPlaneAuthInterceptor` shipped with two — the DI one and a prefix-set
|
||||||
|
overload intended for later phases.
|
||||||
|
|
||||||
|
The throw happens **inside the pipeline, per call**, so:
|
||||||
|
|
||||||
|
- nothing failed at startup; the node booted, joined its pair and reported healthy;
|
||||||
|
- every gated call died with `Unknown / "Exception was thrown by handler"`, which reads as a
|
||||||
|
handler bug rather than an auth bug;
|
||||||
|
- **correct key, wrong key and no key produced identical errors** — the tell. A gate that
|
||||||
|
cannot distinguish those is not authenticating anything.
|
||||||
|
|
||||||
|
Site-a's log at the time: three `PullAuditEvents` calls, three identical
|
||||||
|
`System.InvalidOperationException: Multiple constructors accepting all given argument types
|
||||||
|
have been found in type 'ControlPlaneAuthInterceptor'`.
|
||||||
|
|
||||||
|
The full suite was green when this shipped — **29 suites, 6,872 tests, 0 failures**. The
|
||||||
|
in-process end-to-end test missed it because it registered the interceptor with
|
||||||
|
`AddSingleton` alongside `AddGrpc`, so DI returned the instance and gRPC's activation path
|
||||||
|
never ran.
|
||||||
|
|
||||||
|
Fixed in `228ff8b4`: the prefix-set constructor is `internal`; the end-to-end harness now
|
||||||
|
registers exactly as `Program.cs` does (by type, not in DI); and a reflection assertion pins
|
||||||
|
"exactly one public constructor", since that is the actual invariant.
|
||||||
|
|
||||||
|
**Lesson for phases 1A/1B, which both add services to this interceptor:** extend
|
||||||
|
`DefaultGatedPrefixes`; do not add a second public constructor. And any in-process harness for
|
||||||
|
a DI-activated component must mirror the production registration shape or it proves less than
|
||||||
|
it appears to.
|
||||||
|
|
||||||
|
### Not covered by this gate
|
||||||
|
|
||||||
|
- Streaming subscriptions were exercised in-process (TestServer), not over the rig. The
|
||||||
|
interceptor is path-scoped, not method-scoped, so the rig's `PullAuditEvents` evidence covers
|
||||||
|
the same code path — but a live `SubscribeInstance` under load is untested here.
|
||||||
|
- Key **rotation** on a live pair.
|
||||||
|
- `docker-env2` was updated with its own key but not redeployed or gated.
|
||||||
@@ -81,7 +81,7 @@
|
|||||||
"T0.3",
|
"T0.3",
|
||||||
"T0.4"
|
"T0.4"
|
||||||
],
|
],
|
||||||
"notes": "Rig half needs user approval (docker/deploy.sh recreates 8 containers)."
|
"notes": "Live gate PASS 2026-07-22 (docs/plans/2026-07-22-clusterclient-to-grpc-live-gate.md). Suite: 29 non-Playwright suites / 6872 tests / 0 failures; Host.Tests 366. Playwright re-run against the stable rig was still in flight at compaction \u2014 the earlier 44 failures were contaminated by a concurrent rig rebuild and are VOID. REMAINING: confirm Playwright, then PR + merge (held for the user per the plan's stop-at-DoD rule)."
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"id": "T1A.1",
|
"id": "T1A.1",
|
||||||
|
|||||||
Reference in New Issue
Block a user