The R3-05 gate's own recipe correction, actioned. Security__Acl__Enabled:
"${OTOPCUA_ACL_ENABLED:-false}" was wired to site-a-1/site-a-2 only, so
`OTOPCUA_ACL_ENABLED=true docker compose up -d central-1 central-2` was a silent
no-op ("Running", no recreate, no env) — on the one node kind that carries
BoundedApiKeyStore at all, since only a fused admin+driver node wraps
ConfigDbApiKeyStore. The gate had to proceed via a scratch compose override.
The key is now on central-1 and central-2 too, same env var and same default
false, so today's rig behaviour is byte-identical and the plan's step 1 does what
it says. site-b stays hardcoded OFF as the A/B control.
Plan gains a "## S18 hard-deadline follow-up (2026-08-20)" section: the
cooperative-vs-hard root cause, the red-before-green observation (3/18 failing
with TimeoutException, and why the 15 existing deadline tests could not see it),
the fix, and the still-owed live re-gate table (expected vs. what the gate saw).
BoundedApiKeyStore bounded ConfigDbApiKeyStore with a
CancellationTokenSource(TimeSpan, TimeProvider) linked into the inner call. That
bounds the call only if the callee chooses to observe the token, and
Microsoft.Data.SqlClient does not while a connect/login is in flight on a
black-holed socket. The 2026-08-20 live gate paused the SQL container and watched
a keyed activation hang 42.9 s past its 2000 ms bound — the "did not answer
within 2000 ms" Warning fired only at unpause — and because this runs inside the
SDK's synchronous SessionManager-lock-holding impersonation callback, an
unrelated already-open subscription stalled 57 s behind it. That is precisely the
hazard the bound exists to prevent, and the TTL cache cannot help a miss.
BoundAsync now races the inner call against Task.Delay on the injected clock with
Task.WhenAny. Inner wins: stop the timer, await it, faults and genuine caller
cancellation propagate unchanged. Delay wins: cancel the linked token anyway (a
cooperative inner store should still stop), hand the abandoned task to
ObserveAbandoned, meter outcome=timeout, log the now-honest Warning, and throw
OperationCanceledException — the same exception the authenticator's catch-all
already maps to the standard opaque denial. Fail-closed is unchanged: nothing
here can admit.
ObserveAbandoned disposes the CTS and TOUCHES finished.Exception (the observation
itself), so a late SqlClient fault can never become an UnobservedTaskException.
Debug, not Warning — the timeout is already reported at Warning, and an outage
must not produce two lines per activation for one event.
An abandoned answer is NEVER cached: it was taken before the deadline and would
be served after it, so a key revoked or expired in the meantime could be admitted
from a lookup nobody was still waiting for. StoreTimeoutMs <= 0 still
short-circuits to the unbounded call byte-for-byte (the documented rollback).
RED BEFORE GREEN. Every pre-existing deadline test drives GatedApiKeyStore, whose
waits take the token, so a cooperative CancelAfter unblocks them and all 15
passed against the defective build — the suite could not see the gate's failure
at all. The new UncooperativeApiKeyStore ignores ct entirely and completes only
when the test releases it. Against the unmodified store the three new tests fail
3/18 with "should throw OperationCanceledException but threw TimeoutException"
under a test-side WaitAsync(2 s), so the red is a failure and not a hang — and
that exception swap IS the finding restated as an assertion.
Host.Tests 137/137.
options.Roles is unconditionally overwritten by AddOtOpcUaCluster's PostConfigure to equal
EffectiveRoles.Gossiped (T2's own structural-identity fix), so comparing options.Roles against
Gossiped could never observe a divergence — an overlay setting Cluster:Roles was silently
ignored, not fail-start. The validator now injects IConfiguration and re-reads the raw
Cluster:Roles section LIVE (reflecting any overlay applied after EffectiveRoles was first
resolved), normalized through EffectiveRoles.Resolve for reuse of its own validation.
AddOtOpcUaCluster now TryAddSingleton(configuration) so a bare ServiceCollection test harness
(not just WebApplicationBuilder) can supply IConfiguration to the validator.
Both guard tests rewritten to go through the real AddOtOpcUaCluster + PostConfigure pipeline
over a live ConfigurationManager, simulating an overlay as a second provider added AFTER
EffectiveRoles was resolved off the same object — exactly Program.cs's own sequencing. Confirmed
red beforehand: the "changes" test failed to throw against the pre-fix validator.
The default decider restarts a faulting child immediately and without limit, and
the hook makes every restart do ShutdownAsync+InitializeAsync+SubscribeAsync, so a
driver faulting on every value became a reconnect storm. Per-driver last-reprovision
stamp + DefaultReprovisionBackoff (10s, matched to DefaultReconnectInterval);
a held request is re-delivered by a single per-driver timer carrying the original
incarnation ref, so the F2 identity check still applies.
Seen RED with the backoff disabled: InitializeCount == 3 (second applied
immediately) where the test requires 2-then-3.
Also: the _children.Keys union-arm test (seeder gained taglessDrivers) and the
_desiredByDriver xmldoc note that it lingers on last-known-good after a failed apply.
T11: site-b-2 drops OTOPCUA_ROLES entirely — Cluster:Roles is its sole role source, the rig's
only node that can exhibit the two-role-authority split (01/S-12). Expected to fail every
deploy on a pre-R3-04 build.
T12: site-a-2 pins Cluster__Roles__0="Driver" (capital D) as the standing 01/C-9 case-mismatch
pin, keeping cluster-SITE-A verbatim. Pre-fix this un-elects the SITE-A Primary.
Combined into one commit (deviation from the plan's two separate T11/T12 commits): the header
comment block documenting all six nodes' role sources interleaves both nodes on adjacent lines,
so a clean split was not practical. Recorded in the plan's Execution notes.
Compose edits only — the rig is NOT brought up as part of this change.
New docs/security.md section "Mesh gRPC ingress is AllowAnonymous by design"
(archreview 03/S14): what the cookie FallbackPolicy covers, why it can block a
correctly-PSK'd call before the interceptor's verdict matters, why the
AllowAnonymous opt-out is safe (all five interceptors fail-closed), the
affected-endpoint table (fused-vs-driver-only distinction), and the two
standing guards (MeshGrpcAnonymousMetadataTests, MeshGrpcMapConventionTests)
plus the auth-stack/driver-only boundary tests.
One cross-reference line each in docs/Configuration.md (§MeshTransport,
§ConfigSource/ConfigServe) and docs/Telemetry.md, so an operator reading
about the listener ports meets the constraint where they'd look for it.
A ReprovisionRequested in flight from an old incarnation could reach the NEW child
after HandleRestartDriver swapped it in. In Connecting that child DOES handle
InitializeRequested, so it started a second concurrent InitializeAsync, racing the
#527 dispose-before-overwrite.
Seen RED: InitializeCount == 2 on a child held in Connecting by a gated init
(an already-connected child swallows InitializeRequested and passes vacuously).
MeshGrpcMapConventionTests - source-scan of Program.cs (solution-root walk,
following FlushLoggingConventionTests.cs's idiom): zero direct
MapGrpcService</MapZbLocalDbSync( occurrences, exactly one
MapOtOpcUaMeshGrpcServices( call.
Complements the endpoint-metadata census (T1-T3): the census catches a map
added to MeshGrpcEndpoints without the AllowAnonymous convention, but cannot
see a map that bypasses the helper and goes straight into Program.cs - that
shape never reaches MapOtOpcUaMeshGrpcServices at all. Verified red against
the pre-T1 Program.cs (five inline map sites, zero mapper calls) and green
against the current extracted form.
The alarm subscribe gated on 'a handle exists', which is correct for an idempotent
re-push and wrong for a deploy that widens/narrows the set. Combined with the
spawn-time desired set, a respawning deploy could latch the PREVIOUS revision's
alarm refs for the life of the connection; sourceNodeIds-filtering drivers
(OpcUaClient/AbCip/Focas) would never deliver the new alarm.
Seen RED: LastSubscribedAlarmRefs stayed [alarmX] after a deploy widening to
[alarmX, alarmY].
Security__Auth__DisableLogin: "${OTOPCUA_DISABLE_LOGIN:-true}" on central-1
and central-2 - byte-identical default behaviour, verified via `docker
compose config` both unset (resolves "true") and with
OTOPCUA_DISABLE_LOGIN=false (resolves "false").
DisableLogin's always-succeeding handler, registered under the cookie scheme
name, masks the AdminUI cookie FallbackPolicy entirely - which is exactly how
03/S14 (the FallbackPolicy blocking every mesh gRPC call on a login-enabled
admin node) survived seven phases of prior live gates that all ran with the
default. The production-faithful mode is now a one-variable flip instead of
an edit-and-remember-to-revert; header comment names the flip and the LDAP
users it needs (archreview 03/S14, R3-01 T8).
MeshGrpcDriverOnlyPipelineTests - the driver-role shape (no AddOtOpcUaAuth, no
UseAuthentication/UseAuthorization at all) behaves identically to the fused
shape's T4 positive/negative tests for NodeCommandService and
TelemetryStreamService: a correctly-keyed call succeeds, a no-key call gets
the interceptor's PermissionDenied + detail string.
Also asserts the STRUCTURAL reason it is unaffected - no
IAuthorizationPolicyProvider registered at all - so a future "register auth
everywhere for consistency" change fails this test before it ever reaches a
live site outage. Confirmed (by temporarily swapping in the pre-T2/T3 mapper)
that this class passes identically on both builds: driver-only nodes were
never exposed to 03/S14 in the first place, with or without
.AllowAnonymous().
Joseph Doherty