fix(docker-dev): wire OTOPCUA_ACL_ENABLED to the centrals; record the S18 follow-up

The R3-05 gate's own recipe correction, actioned. Security__Acl__Enabled:
"${OTOPCUA_ACL_ENABLED:-false}" was wired to site-a-1/site-a-2 only, so
`OTOPCUA_ACL_ENABLED=true docker compose up -d central-1 central-2` was a silent
no-op ("Running", no recreate, no env) — on the one node kind that carries
BoundedApiKeyStore at all, since only a fused admin+driver node wraps
ConfigDbApiKeyStore. The gate had to proceed via a scratch compose override.

The key is now on central-1 and central-2 too, same env var and same default
false, so today's rig behaviour is byte-identical and the plan's step 1 does what
it says. site-b stays hardcoded OFF as the A/B control.

Plan gains a "## S18 hard-deadline follow-up (2026-08-20)" section: the
cooperative-vs-hard root cause, the red-before-green observation (3/18 failing
with TimeoutException, and why the 15 existing deadline tests could not see it),
the fix, and the still-owed live re-gate table (expected vs. what the gate saw).
This commit is contained in:
Joseph Doherty
2026-08-20 13:29:30 -04:00
parent c657b2e03d
commit cde5cdfd9f
2 changed files with 99 additions and 0 deletions
@@ -910,3 +910,91 @@ Gate rows deleted (`NodeAcl` row, the `expiry-probe` key, nothing else had been
recreated **without** the override (`Security__Acl__Enabled` absent from both env lists, i.e. default `false`),
parity `b344565cad3d` ×6 re-asserted, readiness back, final deploy `1c666be3…` Sealed, anonymous read of `HR100`
answers a value again. Rig left UP on the master image with `DisableLogin=true`.
## S18 hard-deadline follow-up (2026-08-20)
The gate's S18 finding is **fixed on `fix/r3-05-s18-hard-deadline`** (branched off master `9242630c`), plus the
compose recipe defect the same leg exposed. Not yet live re-gated — the paused-SQL leg has to be re-run.
### What was wrong
`BoundedApiKeyStore.BoundAsync` bounded the inner store with a
`CancellationTokenSource(TimeSpan, TimeProvider)` linked into the call. **That bounds the call only if the
callee chooses to observe the token**, and `Microsoft.Data.SqlClient` does not while a connect or login is
in flight on a black-holed socket. So the deadline never returned control: the gate's keyed activation ran
**42.9 s** against a 2000 ms bound, the *"did not answer within 2000 ms"* Warning was emitted only at
unpause, and because the whole thing sits inside the SDK's synchronous `SessionManager`-lock-holding
impersonation callback, an unrelated already-open subscription stalled **57 s** behind it. The T3 TTL cache
cannot help: a miss is exactly what the outage produces.
### Red before green — the observation, not just the claim
Every pre-existing deadline test drives `GatedApiKeyStore`, whose waits take the token
(`Task.Delay(Timeout.Infinite, ct)`, `gate.WaitAsync(ct)`). A cooperative `CancelAfter` unblocks those, so
**all 15 of them passed against the defective build** — the suite could not see the gate's failure at all.
The new fake, `UncooperativeApiKeyStore`, ignores `ct` entirely and completes only when the test releases
it, modelling SqlClient. Against the unmodified `BoundedApiKeyStore` the three new tests fail — with a
test-side `WaitAsync(2 s)`, so the red is a **failure, not a hang**:
```
Failed! - Failed: 3, Passed: 15, Skipped: 0, Total: 18
An_uncancellable_store_is_abandoned_at_the_deadline [FAIL]
An_abandoned_lookup_that_completes_later_is_not_cached [FAIL]
An_abandoned_lookup_that_faults_later_is_observed_and_swallowed [FAIL]
Shouldly.ShouldAssertException : … should throw System.OperationCanceledException but threw System.TimeoutException
```
`TimeoutException` rather than `OperationCanceledException` **is** the finding, restated as an assertion:
with a 200 ms bound the decorator had still not returned 2 s later.
### The fix
`BoundAsync` now races the inner call against `Task.Delay(timeout, _time, …)` — the injected clock, so the
deadline stays on the same `TimeProvider` as the authorizer and the revalidation timer — with
`Task.WhenAny`:
- **Inner wins** ⇒ stop the delay timer, dispose, `await inner` (faults and a genuine *caller* cancellation
propagate exactly as before).
- **Delay wins** ⇒ cancel the linked token anyway (a *cooperative* inner store should still stop), hand the
abandoned task to `ObserveAbandoned`, meter `outcome=timeout`, log the (now honest) Warning, and throw
`OperationCanceledException` — the same exception the authenticator's catch-all already maps to the
standard opaque denial. **Fail-closed, unchanged: nothing here can admit.**
- `ObserveAbandoned` attaches a continuation that disposes the CTS, **touches `finished.Exception`** (the
observation itself, so a late SqlClient fault can never become an `UnobservedTaskException`), and logs at
Debug. Debug rather than Warning because the timeout is already reported at Warning — a ConfigDb outage
should not produce two lines per activation for one event.
- **An abandoned answer is never cached.** It was taken before the deadline and would be served after it,
so a key revoked or expired in the meantime could be admitted from a lookup nobody was still waiting for.
`Store(...)` is simply unreachable on the timeout path; the test asserts it by call count (a cached late
answer would make the next lookup a hit and leave `Calls` at 1).
- `StoreTimeoutMs ≤ 0` still short-circuits to the unbounded call, byte-for-byte — the documented rollback.
Files: `src/Server/ZB.MOM.WW.OtOpcUa.Host/OpcUa/BoundedApiKeyStore.cs`,
`tests/Server/ZB.MOM.WW.OtOpcUa.Host.Tests/OpcUa/BoundedApiKeyStoreTests.cs`, `docs/security.md`
§"Bounding the key store". Host.Tests **137/137 green** (was 134; +3).
### Recipe defect fixed: `OTOPCUA_ACL_ENABLED` never reached the centrals
The gate's own correction note, actioned. `Security__Acl__Enabled: "${OTOPCUA_ACL_ENABLED:-false}"` was
wired to **site-a-1 / site-a-2 only**, so
`OTOPCUA_ACL_ENABLED=true docker compose up -d central-1 central-2` was a silent no-op — on the one node
kind that carries `BoundedApiKeyStore` at all (only a fused admin+driver node wraps `ConfigDbApiKeyStore`).
The key is now on **central-1 and central-2** too, same env var, **same default-`false`**, so today's rig
behaviour is unchanged and the plan's step 1 does what it says. site-b stays hardcoded OFF as the A/B
control.
### Live re-gate — still owed
Re-run **step 8's paused-SQL leg only** (setup steps 2-4 unchanged, and step 1 now needs no scratch
override). Expected, versus what the gate saw:
| Observation | Gate (defective) | Expected now |
|---|---|---|
| Keyed `read` connect with SQL paused | 42.9 s, client `BadRequestTimeout` | denies in ≈ `StoreTimeoutMs` (2 s) |
| *"did not answer within 2000 ms"* Warning | at unpause, ~40 s late | at the deadline, while SQL is still paused |
| Already-open subscription on the same node | 57 s gap | no gap beyond the ~2 s of the activation |
| `apikey_store_bounded{outcome="timeout"}` | 1, ~40 s late | 1, promptly |
| Falsification control (pause with **no** activation) | streamed clean | must still stream clean |
⚠️ The Debug abandonment line is not visible at the rig's default log level — it is not part of the gate's
pass criteria; the Warning and the restored subscription are.