fix(acl): make the API-key store deadline a HARD race (R3-05 S18 live-gate failure)
BoundedApiKeyStore bounded ConfigDbApiKeyStore with a CancellationTokenSource(TimeSpan, TimeProvider) linked into the inner call. That bounds the call only if the callee chooses to observe the token, and Microsoft.Data.SqlClient does not while a connect/login is in flight on a black-holed socket. The 2026-08-20 live gate paused the SQL container and watched a keyed activation hang 42.9 s past its 2000 ms bound — the "did not answer within 2000 ms" Warning fired only at unpause — and because this runs inside the SDK's synchronous SessionManager-lock-holding impersonation callback, an unrelated already-open subscription stalled 57 s behind it. That is precisely the hazard the bound exists to prevent, and the TTL cache cannot help a miss. BoundAsync now races the inner call against Task.Delay on the injected clock with Task.WhenAny. Inner wins: stop the timer, await it, faults and genuine caller cancellation propagate unchanged. Delay wins: cancel the linked token anyway (a cooperative inner store should still stop), hand the abandoned task to ObserveAbandoned, meter outcome=timeout, log the now-honest Warning, and throw OperationCanceledException — the same exception the authenticator's catch-all already maps to the standard opaque denial. Fail-closed is unchanged: nothing here can admit. ObserveAbandoned disposes the CTS and TOUCHES finished.Exception (the observation itself), so a late SqlClient fault can never become an UnobservedTaskException. Debug, not Warning — the timeout is already reported at Warning, and an outage must not produce two lines per activation for one event. An abandoned answer is NEVER cached: it was taken before the deadline and would be served after it, so a key revoked or expired in the meantime could be admitted from a lookup nobody was still waiting for. StoreTimeoutMs <= 0 still short-circuits to the unbounded call byte-for-byte (the documented rollback). RED BEFORE GREEN. Every pre-existing deadline test drives GatedApiKeyStore, whose waits take the token, so a cooperative CancelAfter unblocks them and all 15 passed against the defective build — the suite could not see the gate's failure at all. The new UncooperativeApiKeyStore ignores ct entirely and completes only when the test releases it. Against the unmodified store the three new tests fail 3/18 with "should throw OperationCanceledException but threw TimeoutException" under a test-side WaitAsync(2 s), so the red is a failure and not a hang — and that exception swap IS the finding restated as an assertion. Host.Tests 137/137.
This commit is contained in:
@@ -214,6 +214,21 @@ with two `Security:ApiKeys` knobs:
|
||||
| `StoreTimeoutMs` | `2000` | Hard per-call deadline on the store. Exceeding it **denies** the activation. `≤ 0` disables the deadline (the pre-R3-05 unbounded behaviour — do not do this on an admin node). |
|
||||
| `StoreCacheSeconds` | `5` | TTL of a read-through cache in front of the store, so a burst of activations costs one round trip. `0` disables it (a read per activation). |
|
||||
|
||||
- ⚠️ **The deadline is a HARD race, not a cooperative token — and the difference was a live-gate
|
||||
failure.** It was originally a `CancellationTokenSource(TimeSpan, TimeProvider)` linked into the inner
|
||||
call, which bounds the call only if the callee chooses to observe the token.
|
||||
`Microsoft.Data.SqlClient` does not, while a connect or login is in flight on a black-holed socket: the
|
||||
2026-08-20 R3-05 gate paused the SQL container and watched a keyed activation hang **42.9 s** past its
|
||||
2000 ms bound (the Warning below fired only at unpause), stalling an unrelated already-open
|
||||
subscription **57 s** behind the `SessionManager` lock. `BoundedApiKeyStore` now races the inner call
|
||||
against a `Task.Delay` on the injected clock and returns control the moment the delay wins.
|
||||
- **An abandoned lookup runs on, and its answer is discarded.** Nothing can cancel a SqlClient call that
|
||||
is not listening, so the query may still complete (or fault) in the background long after the
|
||||
activation was denied. Its result is **never** cached — it was taken before the deadline and would be
|
||||
served after it, so a key revoked or expired in the meantime could be admitted from a lookup nobody
|
||||
was still waiting for. The token is still cancelled, so a *cooperative* inner store does stop, and the
|
||||
abandoned task's outcome is observed and logged at Debug (*"an abandoned key store lookup finished /
|
||||
faulted after its deadline; discarded"*) so it can never surface as an `UnobservedTaskException`.
|
||||
- **A timeout denies; it never admits.** The deadline surfaces as an `OperationCanceledException` into
|
||||
`ApiKeyOpcUaUserAuthenticator`'s existing catch-all, which returns the same opaque denial as a bad
|
||||
credential. There is no second denial path to get wrong, and the decorator can only ever return what
|
||||
|
||||
Reference in New Issue
Block a user