fix(acl): make the API-key store deadline a HARD race (R3-05 S18 live-gate failure)

BoundedApiKeyStore bounded ConfigDbApiKeyStore with a
CancellationTokenSource(TimeSpan, TimeProvider) linked into the inner call. That
bounds the call only if the callee chooses to observe the token, and
Microsoft.Data.SqlClient does not while a connect/login is in flight on a
black-holed socket. The 2026-08-20 live gate paused the SQL container and watched
a keyed activation hang 42.9 s past its 2000 ms bound — the "did not answer
within 2000 ms" Warning fired only at unpause — and because this runs inside the
SDK's synchronous SessionManager-lock-holding impersonation callback, an
unrelated already-open subscription stalled 57 s behind it. That is precisely the
hazard the bound exists to prevent, and the TTL cache cannot help a miss.

BoundAsync now races the inner call against Task.Delay on the injected clock with
Task.WhenAny. Inner wins: stop the timer, await it, faults and genuine caller
cancellation propagate unchanged. Delay wins: cancel the linked token anyway (a
cooperative inner store should still stop), hand the abandoned task to
ObserveAbandoned, meter outcome=timeout, log the now-honest Warning, and throw
OperationCanceledException — the same exception the authenticator's catch-all
already maps to the standard opaque denial. Fail-closed is unchanged: nothing
here can admit.

ObserveAbandoned disposes the CTS and TOUCHES finished.Exception (the observation
itself), so a late SqlClient fault can never become an UnobservedTaskException.
Debug, not Warning — the timeout is already reported at Warning, and an outage
must not produce two lines per activation for one event.

An abandoned answer is NEVER cached: it was taken before the deadline and would
be served after it, so a key revoked or expired in the meantime could be admitted
from a lookup nobody was still waiting for. StoreTimeoutMs <= 0 still
short-circuits to the unbounded call byte-for-byte (the documented rollback).

RED BEFORE GREEN. Every pre-existing deadline test drives GatedApiKeyStore, whose
waits take the token, so a cooperative CancelAfter unblocks them and all 15
passed against the defective build — the suite could not see the gate's failure
at all. The new UncooperativeApiKeyStore ignores ct entirely and completes only
when the test releases it. Against the unmodified store the three new tests fail
3/18 with "should throw OperationCanceledException but threw TimeoutException"
under a test-side WaitAsync(2 s), so the red is a failure and not a hang — and
that exception swap IS the finding restated as an assertion.

Host.Tests 137/137.
This commit is contained in:
Joseph Doherty
2026-08-20 13:29:21 -04:00
parent 9242630c98
commit c657b2e03d
3 changed files with 337 additions and 23 deletions
+15
View File
@@ -214,6 +214,21 @@ with two `Security:ApiKeys` knobs:
| `StoreTimeoutMs` | `2000` | Hard per-call deadline on the store. Exceeding it **denies** the activation. `≤ 0` disables the deadline (the pre-R3-05 unbounded behaviour — do not do this on an admin node). |
| `StoreCacheSeconds` | `5` | TTL of a read-through cache in front of the store, so a burst of activations costs one round trip. `0` disables it (a read per activation). |
- ⚠️ **The deadline is a HARD race, not a cooperative token — and the difference was a live-gate
failure.** It was originally a `CancellationTokenSource(TimeSpan, TimeProvider)` linked into the inner
call, which bounds the call only if the callee chooses to observe the token.
`Microsoft.Data.SqlClient` does not, while a connect or login is in flight on a black-holed socket: the
2026-08-20 R3-05 gate paused the SQL container and watched a keyed activation hang **42.9 s** past its
2000 ms bound (the Warning below fired only at unpause), stalling an unrelated already-open
subscription **57 s** behind the `SessionManager` lock. `BoundedApiKeyStore` now races the inner call
against a `Task.Delay` on the injected clock and returns control the moment the delay wins.
- **An abandoned lookup runs on, and its answer is discarded.** Nothing can cancel a SqlClient call that
is not listening, so the query may still complete (or fault) in the background long after the
activation was denied. Its result is **never** cached — it was taken before the deadline and would be
served after it, so a key revoked or expired in the meantime could be admitted from a lookup nobody
was still waiting for. The token is still cancelled, so a *cooperative* inner store does stop, and the
abandoned task's outcome is observed and logged at Debug (*"an abandoned key store lookup finished /
faulted after its deadline; discarded"*) so it can never surface as an `UnobservedTaskException`.
- **A timeout denies; it never admits.** The deadline surfaces as an `OperationCanceledException` into
`ApiKeyOpcUaUserAuthenticator`'s existing catch-all, which returns the same opaque denial as a bad
credential. There is no second denial path to get wrong, and the decorator can only ever return what