BoundedApiKeyStore bounded ConfigDbApiKeyStore with a
CancellationTokenSource(TimeSpan, TimeProvider) linked into the inner call. That
bounds the call only if the callee chooses to observe the token, and
Microsoft.Data.SqlClient does not while a connect/login is in flight on a
black-holed socket. The 2026-08-20 live gate paused the SQL container and watched
a keyed activation hang 42.9 s past its 2000 ms bound — the "did not answer
within 2000 ms" Warning fired only at unpause — and because this runs inside the
SDK's synchronous SessionManager-lock-holding impersonation callback, an
unrelated already-open subscription stalled 57 s behind it. That is precisely the
hazard the bound exists to prevent, and the TTL cache cannot help a miss.
BoundAsync now races the inner call against Task.Delay on the injected clock with
Task.WhenAny. Inner wins: stop the timer, await it, faults and genuine caller
cancellation propagate unchanged. Delay wins: cancel the linked token anyway (a
cooperative inner store should still stop), hand the abandoned task to
ObserveAbandoned, meter outcome=timeout, log the now-honest Warning, and throw
OperationCanceledException — the same exception the authenticator's catch-all
already maps to the standard opaque denial. Fail-closed is unchanged: nothing
here can admit.
ObserveAbandoned disposes the CTS and TOUCHES finished.Exception (the observation
itself), so a late SqlClient fault can never become an UnobservedTaskException.
Debug, not Warning — the timeout is already reported at Warning, and an outage
must not produce two lines per activation for one event.
An abandoned answer is NEVER cached: it was taken before the deadline and would
be served after it, so a key revoked or expired in the meantime could be admitted
from a lookup nobody was still waiting for. StoreTimeoutMs <= 0 still
short-circuits to the unbounded call byte-for-byte (the documented rollback).
RED BEFORE GREEN. Every pre-existing deadline test drives GatedApiKeyStore, whose
waits take the token, so a cooperative CancelAfter unblocks them and all 15
passed against the defective build — the suite could not see the gate's failure
at all. The new UncooperativeApiKeyStore ignores ct entirely and completes only
when the test releases it. Against the unmodified store the three new tests fail
3/18 with "should throw OperationCanceledException but threw TimeoutException"
under a test-side WaitAsync(2 s), so the red is a failure and not a hang — and
that exception swap IS the finding restated as an assertion.
Host.Tests 137/137.
The default decider restarts a faulting child immediately and without limit, and
the hook makes every restart do ShutdownAsync+InitializeAsync+SubscribeAsync, so a
driver faulting on every value became a reconnect storm. Per-driver last-reprovision
stamp + DefaultReprovisionBackoff (10s, matched to DefaultReconnectInterval);
a held request is re-delivered by a single per-driver timer carrying the original
incarnation ref, so the F2 identity check still applies.
Seen RED with the backoff disabled: InitializeCount == 3 (second applied
immediately) where the test requires 2-then-3.
Also: the _children.Keys union-arm test (seeder gained taglessDrivers) and the
_desiredByDriver xmldoc note that it lingers on last-known-good after a failed apply.
A ReprovisionRequested in flight from an old incarnation could reach the NEW child
after HandleRestartDriver swapped it in. In Connecting that child DOES handle
InitializeRequested, so it started a second concurrent InitializeAsync, racing the
#527 dispose-before-overwrite.
Seen RED: InitializeCount == 2 on a child held in Connecting by a gated init
(an already-connected child swallows InitializeRequested and passes vacuously).
The alarm subscribe gated on 'a handle exists', which is correct for an idempotent
re-push and wrong for a deploy that widens/narrows the set. Combined with the
spawn-time desired set, a respawning deploy could latch the PREVIOUS revision's
alarm refs for the life of the connection; sourceNodeIds-filtering drivers
(OpcUaClient/AbCip/Focas) would never deliver the new alarm.
Seen RED: LastSubscribedAlarmRefs stayed [alarmX] after a deploy widening to
[alarmX, alarmY].
Seen RED against a T3-reverted build: all 4 failed, Created[1].SubscribeCount == 0
on the headline case. 20x isolation run green after the fix was restored.
MapZbLocalDbSync returns IEndpointRouteBuilder, not an IEndpointConventionBuilder,
so .AllowAnonymous() cannot be chained onto it the way the other four mesh maps
are (T2). app.MapGroup(string.Empty).AllowAnonymous().MapZbLocalDbSync() carries
the same convention while leaving the gRPC route pattern byte-identical -
verified directly (LocalDbSync_AllowAnonymousGroup_LeavesTheRoutePatternByteIdentical).
The census's RealMethodEndpoints filter had to move off a "route pattern starts
with /" heuristic to an exact-route-pattern-set match: inside the empty-prefix
group, grpc-dotnet's own "Unimplemented service"/"Unimplemented method" fallback
endpoints also gain a leading "/", so the heuristic let them back in and
over-counted (caught by DriverOnly_WithASyncPort_AlsoMapsTheLocalDbSyncEndpoint
and LocalDbSync_AllowAnonymousGroup_LeavesTheRoutePatternByteIdentical going red
against the naive filter before this fix).
All five mesh gRPC endpoints now carry IAllowAnonymous (archreview 03/S14, T3).
.AllowAnonymous() on DeploymentArtifactGrpcService, CentralCommandGrpcService,
TelemetryStreamGrpcService and NodeCommandGrpcService — the four MapGrpcService
maps that are chainable. On a login-enabled admin node the global cookie
FallbackPolicy (RequireAuthenticatedUser) applies to every endpoint without
explicit authorization metadata, gRPC included; the PSK bearer interceptors run
INSIDE the endpoint invocation, strictly after that middleware, so a
correctly-keyed node call was challenged before the interceptor's verdict could
matter. Safe because all four interceptors fail closed on an unset key.
The census now asserts IAllowAnonymous on all four (red before this change,
green after) plus a scope guard proving the opt-out never leaks onto an
unrelated endpoint. MapZbLocalDbSync (T3) is not chainable the same way and is
asserted absent here on purpose, so this test cannot start passing for the
wrong reason once T3 lands.
Behaviour-preserving: Program.cs:743-784 collapses to a single call, same gates,
same order, every existing per-map rationale comment carried over. A new
endpoint-metadata census (MeshGrpcAnonymousMetadataTests) drives the SAME
production mapper across all three role shapes (admin-only / driver-only /
fused), so the guard can never assert a shape production no longer has -
archreview 03/S14, T1.
The XML remark above OtOpcUaNodeManager.Browse still carried the exact claim the
security.md correction in 22ae1836 removed: there is ONE shared root folder
(V3NodeIds.SharedRootFolderId, raw namespace) fronting both realm trees, not one
per namespace. Rewritten to name the anchor, the ns=0 listing behaviour, and the
zero-segment realm-root mapping that now admits transit holders.
Also completes the BuildScope collision remark (code-review Minor): the degenerate
raw folder literally named OtOpcUa loses the accidental grant leak onto the
anchor's Read/Write that the old one-segment scope allowed — a narrowing,
fail-closed side effect, now recorded.
Comment-only change; addresses the code review's sole Important + Minor.
With Security:Acl:Enabled=true, a session holding any grant narrower than
Cluster could not browse the served address space at all — confirmed by the
live gate.
Mechanism. Both realm trees hang under ONE anchor folder minted in
CreateAddressSpace: NodeId("OtOpcUa", <raw ns>). ResolveParentFolder parents
every top-level raw folder AND every top-level UNS area under it, and the
Browse override throws BadUserAccessDenied for the WHOLE call when its source
node is denied. DataPlaneAuthorizer.BuildScope split that identifier like a
path, producing {Kind: Raw, Segments: ["OtOpcUa"]} — but trie grant paths are
["raw", <RawPath segments…>] and "OtOpcUa" is never a path segment, so
PermissionTrie.HasBrowseTransit failed at the realm-child lookup and
CollectMatches yielded only Cluster-scoped grants. Deny, at the front door.
The trie already handled a zero-segment scope as "the realm root, itself an
ancestor of everything granted in that realm" (PermissionTrie.cs:101-118) —
no production path had ever produced that scope. This wires it up.
- V3NodeIds.SharedRootFolderId names the anchor identity; CreateAddressSpace
now mints the NodeId/BrowseName/DisplayName from it (behaviour-identical,
makes the coupling explicit).
- BuildScope maps that identifier (realm-qualified to Raw — in the UNS
namespace s=OtOpcUa is an ordinary Area) onto the zero-segment Raw scope.
- Evaluate adds a second-realm OR-leg for Browse on the anchor only: it fronts
both trees, so a principal whose only Browse grant lives in the UNS tree
still enters through this raw node. Browse only; transit confers navigation,
never read/write/subscribe.
- No trie, evaluator or builder change. No reordering above the decision cache
— key revocation is still re-checked before the lookup — and the cache key
is unchanged (the combined verdict caches under the anchor's own
identifier/realm; a generation advance clears it).
Falsifications actually run (implement-then-revert-the-line, plus one
genuine red-first):
- DataPlaneAuthorizerTests: the 7 new facts were written BEFORE the fix and 4
went red (the 3 deny-only ones passed vacuously, as a deny-only test must).
- Deleting the UNS OR-leg reddens exactly one fact,
The_shared_root_folder_is_browsable_on_a_uns_only_browse_grant (26/27 pass).
- Dropping the realm qualification from IsSharedRootFolder reddens exactly
A_uns_area_named_like_the_anchor_gets_no_anchor_semantics. Its first draft
did NOT catch this — a raw-only grant is refused either way; the
discriminating grant had to be an unrelated UNS one, whose own transit marks
the zero-segment UNS scope the un-qualified mapping would land on.
- Disabling the BuildScope anchor branch reddens the new end-to-end fact
NodeManagerStage6EnforcementTests.The_shared_root_folder_is_browsable_on_a_narrow_grant_and_still_filters
(real booted server, real Browse dispatch: BadUserAccessDenied on the whole
call), which also pins that the reference filter still withholds the
non-granted sibling.
- Core PermissionTrieBrowseTransitTests gains 3 characterization facts pinning
the previously-dead realm-root branch (transit true for the granting realm's
root, false for the other realm's, false without the Browse bit). These were
green on arrival — the trie was always correct; nothing called it.
Tests: Core 351 -> 354 passed; OpcUaServer 482 -> 490 passed. Build clean.
Review follow-ups to 34829ea0.
IMPORTANT - the leak filter had its own flake vector. This test saturates the
thread pool by design, and TimedScriptEvaluator starts its 5s clock at WaitAsync
rather than when the inner Task.Run gets a thread, so a legitimately-enqueued
evaluation can blow the budget while still queued and surface "script timed out
after 5.0s". The reason != exhaustion filter classified that as a raw disposed
escape and hard-failed on it - handing the test back the load-fragility the
reshape removed. Timeouts are now their own bucket, tolerated and rate-bounded
exactly as exhaustions are; a raw disposed escape still fails on first occurrence.
Doc-comment names this second residual vector alongside the exhaustion calibration.
MINOR (scope extension authorised by review) - "evaluator disposed during
evaluation" existed as three unshared literals. Promoted to
ScriptEvalFailureReasons alongside VirtualTagEvalResult in Commons.Engines (the
assembly all three already reference) and referenced from both evaluators and the
test. The timeout prefix joins it, since the test now classifies on it too. String
values are byte-identical - no behaviour change.
Falsified: with a 12ms script budget the race produced real timeouts (3/2695).
Classified as leaks (the pre-review filter) they hard-failed the test 6/6 items;
bucketed as starvation they sit 0.11% under the 10% bound and the test is green.
Review round on 3e62f161. Two coverage gaps and three wording/comment nits.
IMPORTANT 1 — deregistration is now proven DATA-PRESERVING. The self-heal test
inserted a deployment_pointer row and rebooted unconfigured but never looked at
the row again, so a library regression that touched base-table data during
cleanup would have gone unnoticed — on the very tables that ARE the
boot-from-cache backstop, and on the default-OFF nodes least likely to notice.
The new DisablingReplication_PrunesBookkeepingButNeverTouchesTheCachedDataItself
asserts the row's CONTENT survives (not just its count — a count passes against
a cleanup that blanks columns) while __localdb_row_version and __localdb_oplog,
the bookkeeping deregistration does own, go to zero.
IMPORTANT 2 — mid-loop partial failure is documented and pinned. Both loops flip
four tables with no try/catch, so a throw on the third leaves two flipped and
propagates out of the AddZbLocalDb singleton factory, failing host start. That
is correct — a node that came up anyway would be silently capturing on an
arbitrary subset — and both library calls are idempotent, so the retried boot
completes the flip. Written into OnReady's remarks and pinned by
AFailureMidwayThroughRegistration_FailsBootLoudly_AndTheRetriedBootConverges,
which drives OnReady through a delegating ILocalDb that throws on the Nth
RegisterReplicated, asserts the throw propagates, asserts the wrapper actually
reached that call, then re-runs OnReady unwrapped and asserts all four converge.
MINOR 3 — the Boot helper now records that OnReady's null-logger fallback lands
on Serilog's static Log.Logger, which this assembly never assigns; assertions
about logging pass an explicit logger.
MINOR 5 — log wording: "are both unset" -> "are both absent — unset or blank",
matching IsNullOrWhiteSpace.
MINOR 4 — LocalDbDeploymentArtifactCacheTests said its setup "mirrors
LocalDbSetup.OnReady"; it reproduces the two tables only and registers them
unconditionally without baselining. Softened to say so and to name the
divergence as intentional and inert there.
Falsified both new pins. Wrapper set to failOnCall: 99 (never throws) => the
partial-flip pin is the SOLE failure, so it genuinely exercises the Nth-call
throw path rather than some incidental exception. Predicate forced constant-true
=> five pins red including the new data-preservation one. Both mutations
reverted.
LocalDbSetupTests 15/15. Host.IntegrationTests 226 passed / 0 failed / 5
skipped. Runtime.Tests LocalDbDeploymentArtifactCacheTests 12/12.
LocalDbSetup.OnReady registered all four replicated tables unconditionally on
every driver node, replicating or not. That was not merely wasteful (an
unreplicated node paid two INSERTs plus a json_object of the whole row inside
every write transaction, for an oplog nothing ever drained) — it made
baselineExistingRows unusable, so enabling replication on a long-running
default-OFF node converged on SUBSEQUENT WRITES ONLY and the rows already in the
file stayed invisible to the peer, silently and forever.
Registration now follows one predicate, LocalDbSetup.ReplicationIsConfigured:
LocalDb:Replication:PeerAddress OR :ApiKey non-blank. The OR is load-bearing —
only the dialling half of a pair sets PeerAddress, so keying on it alone would
strip capture from every passive node and the pair would converge in one
direction only. SyncListenPort deliberately does not join it: the passive
endpoint is fail-closed on the key, so a port with no ApiKey syncs with nobody.
configured => RegisterReplicated(table, baselineExistingRows: true), per
table, in the unchanged load-bearing DDL->register order
unconfigured => DeregisterReplicated(table), an idempotent boot self-heal that
drops the capture triggers an earlier build left behind and
prunes that table's oplog/row-version rows. One Information
line, only when it actually cleaned something.
PAIR SYMMETRY: the sync handshake compares registered-table digests fail-closed,
so flipping either key is a both-nodes-together change — set it on both and
restart both. Documented in CLAUDE.md, docs/Configuration.md and the replication
runbook (which claimed "this repo never calls DeregisterReplicated").
Verified against the pinned LocalDb 0.2.1: RegisterReplicated(string, bool) and
DeregisterReplicated(string) -> bool both exist; baselining seeds
__localdb_row_version at HLC 0 and the deregister return value distinguishes
"cleaned" from "nothing to clean". No pin bump needed.
Tests: LocalDbSetupTests gains pins for both branches (configured via
PeerAddress alone and via ApiKey alone; unconfigured => no tables, zero
triggers; late-enable => an HLC-0 baseline ledger entry; the log line fires once
and only when something was cleaned) — four went red against the old
unconditional code, and all four go red again under a constant-true predicate.
LocalDbWiringTests splits its exact-set pin into a replicating graph and a
default-OFF one. LocalDbPairHarness and AlarmSfLegacyMigratorTests built their
database from LocalDb:Path alone while configuring the replication engine from a
second config — production reads one configuration, so the two are now aligned.
Host.IntegrationTests: 224 passed, 0 failed, 5 skipped.
Joseph Doherty