Joseph Doherty
6ef8c7d70a
The gate's first run failed on a defect the green suite could not see: two public constructors on ControlPlaneAuthInterceptor made Grpc.AspNetCore's activation throw per call, so correct key, wrong key and no key all produced identical errors. Recorded in full because the symptom (Unknown / "Exception was thrown by handler") points at the handler, not at auth, and because phases 1A/1B both add services to this same interceptor — they must extend DefaultGatedPrefixes rather than add a second public constructor. Also records what the gate does NOT cover: live streaming under load, key rotation on a running pair, and docker-env2 (keyed but neither redeployed nor gated).
4.6 KiB
ClusterClient → gRPC migration — live gate results
Rig: docker/ (2 central + 3×2 site + traefik), rebuilt from the branch under test via
bash docker/deploy.sh. Recorded check-by-check in the family's live-gate format. Phase 5's
full eight-check gate is recorded further down as those phases land; this file starts with
Phase 0, whose DoD has its own smaller gate.
Phase 0 — PSK auth + dead-code removal — PASS (2026-07-22)
Branch feat/grpc-phase0-psk @ 228ff8b4. Image rebuilt, all 9 containers recreated.
Baseline (pre-change build, same rig)
An unauthenticated call from the host to a site's audit-pull RPC was accepted:
$ grpcurl -plaintext -d '{"batch_size":1}' localhost:9023 sitestream.SiteStreamService/PullAuditEvents
{}
That is the gap Phase 0 closes, reproduced rather than assumed.
Checks
| # | Check | Result |
|---|---|---|
| 1 | All 8 nodes boot with keys configured (new StartupValidator rule) |
PASS — all recreated and reached ready |
| 2 | Unauthenticated PullAuditEvents ⇒ PermissionDenied, all 3 sites |
PASS |
| 3 | Wrong key (site-b's key presented to site-a) ⇒ PermissionDenied |
PASS — per-site scoping is real, not decorative |
| 4 | Correct key ⇒ success, all 3 sites | PASS |
| 5 | Central's own authenticated paths still work | PASS — 14 successful PullAuditEvents from central to site-a; 0 auth failures in either central's log |
| 6 | LocalDb sync unaffected by the new interceptor | PASS — 0 control-plane rejections and 0 sync auth failures on the passive peer; session connected after one boot-order retry |
| 7 | No interceptor-activation errors | PASS — 0 (see the defect below) |
Evidence for 2–4:
=== NO CREDENTIALS ===
:9023 -> ERROR: Code: PermissionDenied Message: Control plane authentication failed.
:9033 -> ERROR: Code: PermissionDenied Message: Control plane authentication failed.
:9043 -> ERROR: Code: PermissionDenied Message: Control plane authentication failed.
=== WRONG KEY (site-b's key against site-a) ===
:9023 -> ERROR: Code: PermissionDenied Message: Control plane authentication failed.
=== CORRECT KEY ===
site-a :9023 -> {}
site-b :9033 -> {}
site-c :9043 -> {}
Site-a rejected exactly 2 calls — the two deliberate probes above — and nothing else.
Defect the gate caught that the test suite did not
First run of this gate FAILED, and is worth recording because the failure mode is deceptive.
Grpc.AspNetCore activates a type-registered interceptor through
InterceptorRegistration.GetFactory(), which throws when more than one public constructor is
applicable. ControlPlaneAuthInterceptor shipped with two — the DI one and a prefix-set
overload intended for later phases.
The throw happens inside the pipeline, per call, so:
- nothing failed at startup; the node booted, joined its pair and reported healthy;
- every gated call died with
Unknown / "Exception was thrown by handler", which reads as a handler bug rather than an auth bug; - correct key, wrong key and no key produced identical errors — the tell. A gate that cannot distinguish those is not authenticating anything.
Site-a's log at the time: three PullAuditEvents calls, three identical
System.InvalidOperationException: Multiple constructors accepting all given argument types have been found in type 'ControlPlaneAuthInterceptor'.
The full suite was green when this shipped — 29 suites, 6,872 tests, 0 failures. The
in-process end-to-end test missed it because it registered the interceptor with
AddSingleton alongside AddGrpc, so DI returned the instance and gRPC's activation path
never ran.
Fixed in 228ff8b4: the prefix-set constructor is internal; the end-to-end harness now
registers exactly as Program.cs does (by type, not in DI); and a reflection assertion pins
"exactly one public constructor", since that is the actual invariant.
Lesson for phases 1A/1B, which both add services to this interceptor: extend
DefaultGatedPrefixes; do not add a second public constructor. And any in-process harness for
a DI-activated component must mirror the production registration shape or it proves less than
it appears to.
Not covered by this gate
- Streaming subscriptions were exercised in-process (TestServer), not over the rig. The
interceptor is path-scoped, not method-scoped, so the rig's
PullAuditEventsevidence covers the same code path — but a liveSubscribeInstanceunder load is untested here. - Key rotation on a live pair.
docker-env2was updated with its own key but not redeployed or gated.