dohertj2/lmxopcua
1
0
Fork 0
Code Issues 32 Pull Requests Actions Packages Projects Releases Wiki Activity

fix(acl): make the API-key store deadline a HARD race (R3-05 S18 live-gate failure)

Browse Source 
BoundedApiKeyStore bounded ConfigDbApiKeyStore with a
CancellationTokenSource(TimeSpan, TimeProvider) linked into the inner call. That
bounds the call only if the callee chooses to observe the token, and
Microsoft.Data.SqlClient does not while a connect/login is in flight on a
black-holed socket. The 2026-08-20 live gate paused the SQL container and watched
a keyed activation hang 42.9 s past its 2000 ms bound — the "did not answer
within 2000 ms" Warning fired only at unpause — and because this runs inside the
SDK's synchronous SessionManager-lock-holding impersonation callback, an
unrelated already-open subscription stalled 57 s behind it. That is precisely the
hazard the bound exists to prevent, and the TTL cache cannot help a miss.

BoundAsync now races the inner call against Task.Delay on the injected clock with
Task.WhenAny. Inner wins: stop the timer, await it, faults and genuine caller
cancellation propagate unchanged. Delay wins: cancel the linked token anyway (a
cooperative inner store should still stop), hand the abandoned task to
ObserveAbandoned, meter outcome=timeout, log the now-honest Warning, and throw
OperationCanceledException — the same exception the authenticator's catch-all
already maps to the standard opaque denial. Fail-closed is unchanged: nothing
here can admit.

ObserveAbandoned disposes the CTS and TOUCHES finished.Exception (the observation
itself), so a late SqlClient fault can never become an UnobservedTaskException.
Debug, not Warning — the timeout is already reported at Warning, and an outage
must not produce two lines per activation for one event.

An abandoned answer is NEVER cached: it was taken before the deadline and would
be served after it, so a key revoked or expired in the meantime could be admitted
from a lookup nobody was still waiting for. StoreTimeoutMs <= 0 still
short-circuits to the unbounded call byte-for-byte (the documented rollback).

RED BEFORE GREEN. Every pre-existing deadline test drives GatedApiKeyStore, whose
waits take the token, so a cooperative CancelAfter unblocks them and all 15
passed against the defective build — the suite could not see the gate's failure
at all. The new UncooperativeApiKeyStore ignores ct entirely and completes only
when the test releases it. Against the unmodified store the three new tests fail
3/18 with "should throw OperationCanceledException but threw TimeoutException"
under a test-side WaitAsync(2 s), so the red is a failure and not a hang — and
that exception swap IS the finding restated as an assertion.

Host.Tests 137/137.
This commit is contained in:
Joseph Doherty
2026-08-20 13:29:21 -04:00
parent 9242630c98
commit c657b2e03d
3 changed files with 337 additions and 23 deletions
Download Patch File Download Diff File Expand all files Collapse all files
src/Server/ZB.MOM.WW.OtOpcUa.Host/OpcUa/BoundedApiKeyStore.cs
+96 -21
View File
@@ -27,10 +27,12 @@ namespace ZB.MOM.WW.OtOpcUa.Host.OpcUa;
/// decorator can only return what the inner store returned, or throw.
/// </para>
/// <para>
/// ⚠️ <b>Faults and timeouts are NEVER cached.</b> Caching a deny would be fail-closed and is
/// still wrong: it would extend a ConfigDb outage past the outage, denying valid keys for a
/// further TTL after the database came back. Only a COMPLETED lookup — a hit or an authoritative
/// miss — is stored.
/// ⚠️ <b>Faults and timeouts are NEVER cached</b>, and neither is the late answer of a lookup the
/// deadline abandoned. Caching a deny would be fail-closed and is still wrong: it would extend a
/// ConfigDb outage past the outage, denying valid keys for a further TTL after the database came
/// back. Caching an abandoned answer would be worse than that — it was taken before the deadline
/// and served after it. Only a lookup that completed BEFORE its deadline — a hit or an
/// authoritative miss — is stored.
/// </para>
/// </remarks>
public sealed class BoundedApiKeyStore : IApiKeyStore
@@ -202,11 +204,33 @@ public sealed class BoundedApiKeyStore : IApiKeyStore
/// sees in production is always uncancellable.
/// </para>
/// <para>
/// ⚠️ <b>It re-throws rather than returning <see langword="null"/>.</b> <c>null</c> means
/// ⚠️ <b>The deadline is a HARD race, not a cooperative token — and that distinction was a live
/// gate failure, not a theoretical one.</b> This used to be a
/// <c>CancellationTokenSource(TimeSpan, TimeProvider)</c> linked into the inner call, which
/// bounds the call only if the callee chooses to observe the token.
/// <c>Microsoft.Data.SqlClient</c> does not, while a connect or login is in flight on a
/// black-holed socket: the 2026-08-20 R3-05 gate paused the SQL container and watched a keyed
/// activation hang <b>42.9 s</b> past its 2000 ms bound (the Warning below fired only at
/// unpause), stalling an unrelated already-open subscription for <b>57 s</b> behind the
/// <c>SessionManager</c> lock — the exact hazard the bound exists to prevent. So the inner call
/// is now raced against a <see cref="Task.Delay(TimeSpan, TimeProvider, CancellationToken)"/>
/// on the injected clock and <b>abandoned</b> when the delay wins.
/// </para>
/// <para>
/// ⚠️ <b>An abandoned lookup's answer is DISCARDED, never cached.</b> It was taken before the
/// deadline and would be served after it, so a key revoked or expired in the meantime could be
/// admitted from a lookup nobody was still waiting for. The token is still cancelled — a
/// cooperative inner store should stop — and the abandoned task's outcome is observed by
/// <see cref="ObserveAbandoned{T}"/> so it can never surface as an
/// <c>UnobservedTaskException</c>.
/// </para>
/// <para>
/// ⚠️ <b>It throws rather than returning <see langword="null"/>.</b> <c>null</c> means
/// "no such usable key", which the authenticator logs as <i>denied — no enabled, unexpired key
/// with id X</i>. A timeout is not that, and an operator reading that line during a ConfigDb
/// outage would go hunting for a key problem that does not exist. The distinct Warning below is
/// the honest signal; the denial itself is unchanged either way.
/// the honest signal; the denial itself is unchanged either way — the authenticator's catch-all
/// turns any throw into the same opaque denial.
/// </para>
/// </remarks>
private async Task<T> BoundAsync<T>(Func<CancellationToken, Task<T>> call, CancellationToken ct)
@@ -214,27 +238,78 @@ public sealed class BoundedApiKeyStore : IApiKeyStore
var timeoutMs = _options.Value.StoreTimeoutMs;
if (timeoutMs <= 0) return await call(ct).ConfigureAwait(false);
// The deadline runs on the INJECTED clock (the TimeProvider ctor overload), not on a plain
// CancelAfter, so it is the same TimeProvider the authorizer and the revalidation timer read.
// Linked with the caller's token so a genuine caller cancellation is still honoured — and so the
// catch below can tell the two apart.
using var deadline = new CancellationTokenSource(TimeSpan.FromMilliseconds(timeoutMs), _time);
using var cts = CancellationTokenSource.CreateLinkedTokenSource(ct, deadline.Token);
// winner check below can tell the two apart. NOT linked to the deadline: the deadline is the race,
// and this token is only the courtesy signal to an inner store that happens to be cooperative.
var innerCts = CancellationTokenSource.CreateLinkedTokenSource(ct);
Task<T> inner;
try
{
return await call(cts.Token).ConfigureAwait(false);
inner = call(innerCts.Token);
}
catch (OperationCanceledException) when (!ct.IsCancellationRequested)
catch
{
OtOpcUaTelemetry.ApiKeyStoreBounded.Add(
1, new KeyValuePair<string, object?>("outcome", "timeout"));
_log.LogWarning(
"ApiKeyStore: the key store did not answer within {TimeoutMs} ms — denying this activation. "
+ "Every OPC UA session activation on this server is serialized behind this call; "
+ "check ConfigDb health.",
timeoutMs);
innerCts.Dispose();
throw;
}
// The delay runs on the INJECTED clock, so it is the same TimeProvider the authorizer and the
// revalidation timer read. Its own CTS exists only to stop the timer when the inner call wins.
using var delayCts = new CancellationTokenSource();
var deadline = Task.Delay(TimeSpan.FromMilliseconds(timeoutMs), _time, delayCts.Token);
if (await Task.WhenAny(inner, deadline).ConfigureAwait(false) == inner)
{
delayCts.Cancel();
innerCts.Dispose();
return await inner.ConfigureAwait(false);
}
innerCts.Cancel();
ObserveAbandoned(inner, innerCts);
OtOpcUaTelemetry.ApiKeyStoreBounded.Add(
1, new KeyValuePair<string, object?>("outcome", "timeout"));
_log.LogWarning(
"ApiKeyStore: the key store did not answer within {TimeoutMs} ms — denying this activation. "
+ "The lookup is abandoned and its result, if it ever arrives, is discarded. Every OPC UA "
+ "session activation on this server is serialized behind this call; check ConfigDb health.",
timeoutMs);
throw new OperationCanceledException(
$"The API key store did not answer within {timeoutMs} ms; the lookup was abandoned.");
}
/// <summary>
/// Watches an abandoned inner lookup to its end so its outcome is observed and its resources are
/// released, and discards whatever it produced.
/// </summary>
/// <typeparam name="T">The abandoned call's result type.</typeparam>
/// <param name="abandoned">The lookup the deadline gave up on.</param>
/// <param name="cts">The cancellation source that call was handed; disposed once it finishes.</param>
/// <remarks>
/// Nobody awaits <paramref name="abandoned"/>, so without this its fault would be unobserved.
/// Debug rather than Warning: the timeout has already been reported at Warning, and a ConfigDb
/// outage would otherwise produce two lines per activation for one event.
/// </remarks>
private void ObserveAbandoned<T>(Task<T> abandoned, CancellationTokenSource cts) =>
_ = abandoned.ContinueWith(
finished =>
{
cts.Dispose();
var error = finished.Exception; // The observation itself — do not remove.
if (error is not null)
{
_log.LogDebug(
error,
"ApiKeyStore: an abandoned key store lookup faulted after its deadline; discarded.");
}
else
{
_log.LogDebug(
"ApiKeyStore: an abandoned key store lookup finished after its deadline; discarded.");
}
},
CancellationToken.None,
TaskContinuationOptions.ExecuteSynchronously,
TaskScheduler.Default);
}