BoundedApiKeyStore bounded ConfigDbApiKeyStore with a
CancellationTokenSource(TimeSpan, TimeProvider) linked into the inner call. That
bounds the call only if the callee chooses to observe the token, and
Microsoft.Data.SqlClient does not while a connect/login is in flight on a
black-holed socket. The 2026-08-20 live gate paused the SQL container and watched
a keyed activation hang 42.9 s past its 2000 ms bound — the "did not answer
within 2000 ms" Warning fired only at unpause — and because this runs inside the
SDK's synchronous SessionManager-lock-holding impersonation callback, an
unrelated already-open subscription stalled 57 s behind it. That is precisely the
hazard the bound exists to prevent, and the TTL cache cannot help a miss.
BoundAsync now races the inner call against Task.Delay on the injected clock with
Task.WhenAny. Inner wins: stop the timer, await it, faults and genuine caller
cancellation propagate unchanged. Delay wins: cancel the linked token anyway (a
cooperative inner store should still stop), hand the abandoned task to
ObserveAbandoned, meter outcome=timeout, log the now-honest Warning, and throw
OperationCanceledException — the same exception the authenticator's catch-all
already maps to the standard opaque denial. Fail-closed is unchanged: nothing
here can admit.
ObserveAbandoned disposes the CTS and TOUCHES finished.Exception (the observation
itself), so a late SqlClient fault can never become an UnobservedTaskException.
Debug, not Warning — the timeout is already reported at Warning, and an outage
must not produce two lines per activation for one event.
An abandoned answer is NEVER cached: it was taken before the deadline and would
be served after it, so a key revoked or expired in the meantime could be admitted
from a lookup nobody was still waiting for. StoreTimeoutMs <= 0 still
short-circuits to the unbounded call byte-for-byte (the documented rollback).
RED BEFORE GREEN. Every pre-existing deadline test drives GatedApiKeyStore, whose
waits take the token, so a cooperative CancelAfter unblocks them and all 15
passed against the defective build — the suite could not see the gate's failure
at all. The new UncooperativeApiKeyStore ignores ct entirely and completes only
when the test releases it. Against the unmodified store the three new tests fail
3/18 with "should throw OperationCanceledException but threw TimeoutException"
under a test-side WaitAsync(2 s), so the red is a failure and not a hang — and
that exception swap IS the finding restated as an assertion.
Host.Tests 137/137.
options.Roles is unconditionally overwritten by AddOtOpcUaCluster's PostConfigure to equal
EffectiveRoles.Gossiped (T2's own structural-identity fix), so comparing options.Roles against
Gossiped could never observe a divergence — an overlay setting Cluster:Roles was silently
ignored, not fail-start. The validator now injects IConfiguration and re-reads the raw
Cluster:Roles section LIVE (reflecting any overlay applied after EffectiveRoles was first
resolved), normalized through EffectiveRoles.Resolve for reuse of its own validation.
AddOtOpcUaCluster now TryAddSingleton(configuration) so a bare ServiceCollection test harness
(not just WebApplicationBuilder) can supply IConfiguration to the validator.
Both guard tests rewritten to go through the real AddOtOpcUaCluster + PostConfigure pipeline
over a live ConfigurationManager, simulating an overlay as a second provider added AFTER
EffectiveRoles was resolved off the same object — exactly Program.cs's own sequencing. Confirmed
red beforehand: the "changes" test failed to throw against the pre-fix validator.
The default decider restarts a faulting child immediately and without limit, and
the hook makes every restart do ShutdownAsync+InitializeAsync+SubscribeAsync, so a
driver faulting on every value became a reconnect storm. Per-driver last-reprovision
stamp + DefaultReprovisionBackoff (10s, matched to DefaultReconnectInterval);
a held request is re-delivered by a single per-driver timer carrying the original
incarnation ref, so the F2 identity check still applies.
Seen RED with the backoff disabled: InitializeCount == 3 (second applied
immediately) where the test requires 2-then-3.
Also: the _children.Keys union-arm test (seeder gained taglessDrivers) and the
_desiredByDriver xmldoc note that it lingers on last-known-good after a failed apply.
A ReprovisionRequested in flight from an old incarnation could reach the NEW child
after HandleRestartDriver swapped it in. In Connecting that child DOES handle
InitializeRequested, so it started a second concurrent InitializeAsync, racing the
#527 dispose-before-overwrite.
Seen RED: InitializeCount == 2 on a child held in Connecting by a gated init
(an already-connected child swallows InitializeRequested and passes vacuously).
MeshGrpcMapConventionTests - source-scan of Program.cs (solution-root walk,
following FlushLoggingConventionTests.cs's idiom): zero direct
MapGrpcService</MapZbLocalDbSync( occurrences, exactly one
MapOtOpcUaMeshGrpcServices( call.
Complements the endpoint-metadata census (T1-T3): the census catches a map
added to MeshGrpcEndpoints without the AllowAnonymous convention, but cannot
see a map that bypasses the helper and goes straight into Program.cs - that
shape never reaches MapOtOpcUaMeshGrpcServices at all. Verified red against
the pre-T1 Program.cs (five inline map sites, zero mapper calls) and green
against the current extracted form.
The alarm subscribe gated on 'a handle exists', which is correct for an idempotent
re-push and wrong for a deploy that widens/narrows the set. Combined with the
spawn-time desired set, a respawning deploy could latch the PREVIOUS revision's
alarm refs for the life of the connection; sourceNodeIds-filtering drivers
(OpcUaClient/AbCip/Focas) would never deliver the new alarm.
Seen RED: LastSubscribedAlarmRefs stayed [alarmX] after a deploy widening to
[alarmX, alarmY].
MeshGrpcDriverOnlyPipelineTests - the driver-role shape (no AddOtOpcUaAuth, no
UseAuthentication/UseAuthorization at all) behaves identically to the fused
shape's T4 positive/negative tests for NodeCommandService and
TelemetryStreamService: a correctly-keyed call succeeds, a no-key call gets
the interceptor's PermissionDenied + detail string.
Also asserts the STRUCTURAL reason it is unaffected - no
IAuthorizationPolicyProvider registered at all - so a future "register auth
everywhere for consistency" change fails this test before it ever reaches a
live site outage. Confirmed (by temporarily swapping in the pre-T2/T3 mapper)
that this class passes identically on both builds: driver-only nodes were
never exposed to 03/S14 in the first place, with or without
.AllowAnonymous().
Extends MeshGrpcAuthStackTests (T5): a no-bearer or wrong-bearer call to
DeploymentArtifactService.Fetch, TelemetryStreamService.Subscribe and
NodeCommandService.Execute fails with the INTERCEPTOR's detail string
("Config-serve authentication failed.", "Telemetry stream authentication
failed.", "Node command authentication failed."), asserted alongside the
status code rather than in place of it - the vacuity trap the plan calls out
(ConfigServeAuthInterceptor throws Unauthenticated, and Grpc.Net.Client can
map an HTTP 401 to the same status, so status alone would not discriminate a
broken build).
Verified red-then-green against the T1 (pre-AllowAnonymous) mapper: all 6 new
theory cases fail there with StatusCode.Internal ("HTTP/2 server reset the
stream, PROTOCOL_ERROR") instead of Unauthenticated/PermissionDenied - the
cookie challenge breaking the h2c stream before any gRPC trailers are
written, never a clean auth denial. That is the exact signature the live
gate (T10) should grep for; recorded in the class remarks.
MeshGrpcAuthStackTests - the CI-shaped guard 03/S14 asked for. A real
AddOtOpcUaAuth(config) + UseAuthentication/UseAuthorization pipeline (cookie
scheme, DisableLogin=false, EnvironmentName=Production) with the production
interceptor set, driving DeploymentArtifactService.Fetch,
CentralCommandService.ReportApply and NodeCommandService.Execute through
MapOtOpcUaMeshGrpcServices.
Verified red-then-green by hand: with the T1 (pre-AllowAnonymous) mapper
swapped in, all three positive calls fail with
"HTTP/2 server reset the stream, PROTOCOL_ERROR" (RpcException/Internal) -
the cookie challenge breaking prior-knowledge h2c, not a clean auth denial.
With the current mapper (T2/T3) all three succeed. That asymmetry is the only
thing distinguishing the two builds; every other assertion here (cookie
handler identity, the falsifiability control) is unchanged by the fix and
proves nothing on its own.
Falsifiability control: LocalDbSyncService is mapped directly (bypassing
MapOtOpcUaMeshGrpcServices) with no .AllowAnonymous(), and a correctly-shaped
call to it is refused - proving the fixture's auth stack is genuinely live,
not accidentally permissive.
Crash seam: the planned null-Snapshot OnDataChange (no SupervisorStrategy fallback needed).
Seen RED with T6 reverted: InitializeCount 1, not 2 — with a preceding ShutdownCount>=1
assertion proving the restart really happened, so the red is not ambiguous with 'never crashed'.
Seen RED against a T3-reverted build: all 4 failed, Created[1].SubscribeCount == 0
on the headline case. 20x isolation run green after the fix was restored.
MapZbLocalDbSync returns IEndpointRouteBuilder, not an IEndpointConventionBuilder,
so .AllowAnonymous() cannot be chained onto it the way the other four mesh maps
are (T2). app.MapGroup(string.Empty).AllowAnonymous().MapZbLocalDbSync() carries
the same convention while leaving the gRPC route pattern byte-identical -
verified directly (LocalDbSync_AllowAnonymousGroup_LeavesTheRoutePatternByteIdentical).
The census's RealMethodEndpoints filter had to move off a "route pattern starts
with /" heuristic to an exact-route-pattern-set match: inside the empty-prefix
group, grpc-dotnet's own "Unimplemented service"/"Unimplemented method" fallback
endpoints also gain a leading "/", so the heuristic let them back in and
over-counted (caught by DriverOnly_WithASyncPort_AlsoMapsTheLocalDbSyncEndpoint
and LocalDbSync_AllowAnonymousGroup_LeavesTheRoutePatternByteIdentical going red
against the naive filter before this fix).
All five mesh gRPC endpoints now carry IAllowAnonymous (archreview 03/S14, T3).
.AllowAnonymous() on DeploymentArtifactGrpcService, CentralCommandGrpcService,
TelemetryStreamGrpcService and NodeCommandGrpcService — the four MapGrpcService
maps that are chainable. On a login-enabled admin node the global cookie
FallbackPolicy (RequireAuthenticatedUser) applies to every endpoint without
explicit authorization metadata, gRPC included; the PSK bearer interceptors run
INSIDE the endpoint invocation, strictly after that middleware, so a
correctly-keyed node call was challenged before the interceptor's verdict could
matter. Safe because all four interceptors fail closed on an unset key.
The census now asserts IAllowAnonymous on all four (red before this change,
green after) plus a scope guard proving the opt-out never leaks onto an
unrelated endpoint. MapZbLocalDbSync (T3) is not chainable the same way and is
asserted absent here on purpose, so this test cannot start passing for the
wrong reason once T3 lands.
Joseph Doherty