lmxopcua/archreview/plans/R2-08-ldap-historyread-async-plan.md

# Design + Implementation Plan — R2-08: Async LDAP authentication + HistoryRead bridging

- **Source report:** `archreview/03-server-runtime.md` (2026-07-12 re-review, working-tree version)
- **Review commit:** `f6eaa267` · **Plan verified against tree at:** `f6eaa267` (master, clean)
- **Scope:** OVERALL prioritized action item **#8** — "Async LDAP + configurable timeout; channel-ize
  HistoryRead bridging" — covering findings **03/S2** (High — LDAP authentication block-bridges OPC UA
  session activation with an unconfigurable ~10 s timeout, no pooling, no fail-fast) and **03/S3**
  (High — HistoryRead block-bridges the gateway per node, sequentially, on SDK request threads,
  bounded only by the 30 s gateway `CallTimeout`). Both were carried unfixed from round 1
  (`archreview/plans/03-server-runtime-plan.md` §S2/§S3 — designs verified + refreshed below).
- **Touches:** `Security/Ldap/LdapOptions.cs`, `Security/Ldap/OtOpcUaLdapAuthService.cs`,
  `Security/Ldap/LdapAuthResult.cs`, `Host/OpcUa/LdapOpcUaUserAuthenticator.cs`,
  `Host/Configuration/LdapOptionsValidator.cs`, `OpcUaServer/OpcUaApplicationHost.cs` (doc only),
  `OpcUaServer/OtOpcUaNodeManager.cs`, `Runtime/Historian/ServerHistorianOptions.cs`,
  `Host/OpcUa/OtOpcUaServerHostedService.cs`, plus tests in `Host.IntegrationTests`,
  `Security.Tests`, `Runtime.Tests`, `OpcUaServer.Tests`. **No shared-lib (`ZB.MOM.WW.Auth`) change
  required** — see Verification.

## Verification summary

Every anchor in both findings was opened against the working tree at `f6eaa267`; the round-1 plan's
S2/S3 designs were re-verified against the code and the actual SDK + shared-library sources. **Both
findings CONFIRMED — and S2 is materially *worse* than the report states.** Details:

- **S2 anchors — confirmed, one path drift.** The report cites
  `src/Server/ZB.MOM.WW.OtOpcUa.Host/OpcUa/OpcUaApplicationHost.cs:271-274`; the file actually lives at
  `src/Server/ZB.MOM.WW.OtOpcUa.OpcUaServer/OpcUaApplicationHost.cs` (the authenticator *is* under
  `Host/OpcUa/`). The block-bridge is exact: `HandleImpersonation` calls
  `authenticator.AuthenticateUserNameAsync(…, CancellationToken.None).GetAwaiter().GetResult()`
  (`OpcUaApplicationHost.cs:270-273`). `LdapOpcUaUserAuthenticator.AuthenticateUserNameAsync`
  (`Host/OpcUa/LdapOpcUaUserAuthenticator.cs:30-48`) adds no timeout, no concurrency bound, no
  fail-fast. `LdapOptions.ToLibraryOptions()` (`Security/Ldap/LdapOptions.cs:94-107`) projects no
  timeout field and `LdapOptions` has none.
- **NEW (verified from SDK source, worse than reported): the impersonation callback runs inside a
  global event lock.** Verified against the pinned `OPCFoundation.NetStandard.Opc.Ua.Server`
  **1.5.378.106** sources (`Libraries/Opc.Ua.Server/Session/SessionManager.cs`):
  `ActivateSessionAsync` raises the handler as `lock (m_eventLock) { … m_ImpersonateUser(session, args); … }`
  (lines 352-360; `m_eventLock` is a single `SessionManager`-wide `Lock`, line 698). The handler
  signature is a synchronous `void` delegate; there is **no async variant** of the impersonation
  callback (the surrounding `ActivateSessionAsync` is `ValueTask`-returning, but the callback itself
  must complete inline). Consequence: one stalled LDAP bind does not just hold *one* SDK thread — it
  **serializes every session activation server-wide** (Anonymous and X509 included, since even the
  early-return path must acquire `m_eventLock` to invoke the handler). The SDK contract is therefore
  irreducibly sync → the fix is a hard wall-clock bound + bounded concurrency + outage fail-fast at
  the authenticator boundary, exactly the shape the report recommends. No "fake async" is attempted.
- **NEW (resolves a round-1 open choice): the shared library already exposes the timeout.**
  `ZB.MOM.WW.Auth.Abstractions.Ldap.LdapOptions.ConnectionTimeoutMs` exists (default `10_000`,
  `LdapContracts.cs:20` in `~/Desktop/scadaproj/ZB.MOM.WW.Auth`), and the Novell adapter applies it to
  both the socket connect (`_conn.ConnectionTimeout`) and the per-operation search constraint
  (`constraints.TimeLimit`) (`NovellLdapConnection.ApplyTimeout`, lines 131-146). The round-1 plan's
  step 1 said "do **not** project into `ToLibraryOptions()` … unless the library later exposes a
  matching field" — **the field exists; this plan projects it.** Also verified: the library's
  `AuthenticateAsync` is `Task.FromResult(Authenticate(...))` — synchronous-in-Task, token observed at
  entry only (`LdapAuthService.cs:59-66`), and each call is a fresh connect + service-bind + search +
  user-bind (no pooling; pooling stays a shared-lib follow-up, out of scope here). The library is
  fully fail-closed (never throws; system-side faults map to `LdapAuthFailure.ServiceAccountBindFailed`).
- **S3 anchors — all four bridge sites line-exact** at `f6eaa267`:
  `OtOpcUaNodeManager.cs:1911` (Events), `:2059` (`ServeNode` — serves Processed + AtTime), `:2171`
  (Raw page read), `:2203` (Raw tie-cluster over-fetch) — each `.GetAwaiter().GetResult()` with
  `CancellationToken.None`. The four arms iterate `nodesToProcess` in a sequential `foreach`
  (`:1784` RawModified, `:1811` Processed, `:1851` AtTime, `:1873` Events). SDK thread budget
  confirmed: `MinRequestThreadCount = 5, MaxRequestThreadCount = 100, MaxQueuedRequestCount = 200`
  (`OpcUaServer/OpcUaApplicationHost.cs:333-335`). The overrides run **outside** the node-manager
  `Lock` (documented at `:1777-1780`) — this is a request-pool-exhaustion risk, not a lock-freeze risk.
- **S3 thread-safety audit (round-1 plan asked for it — done):** parallel per-node serving is safe.
  `results[handle.Index]` / `errors[handle.Index]` writes are index-disjoint into pre-sized lists (the
  base dispatcher seeds every slot). All shared node-manager maps the read path touches are
  `ConcurrentDictionary` (`_variables:41`, `_historizedTagnames:55`, `_eventNotifierSources:69`).
  `IHistoryContinuationStore` is explicitly thread-safe (SDK per-session store — "the SDK session
  locks internally", `HistoryContinuationStore.cs`; the test store is a lock-protected dictionary).
  `ServeRawPaged` state is all per-handle locals. `IHistorianDataSource` (the gateway client) is a
  gRPC channel wrapper — safe for concurrent calls; each call already carries the 30 s `CallTimeout`
  deadline (`ServerHistorianOptions.CallTimeout`, `ServerHistorianOptions.cs:58`).
- **Round-1 plan §S3 design re-verified:** bounded per-node parallelism + per-request deadline +
  server-side limiter all still fit the tree; refined below (OCE→`BadTimeout` per node, limiter wait
  budget, lazy gate creation, options names) and the "fully async HistoryRead" alternative re-confirmed
  impossible — `CustomNodeManager2`'s `HistoryRead*` overrides are synchronous `void` fillers.
- **Existing test harnesses located** (reused, not invented): `Host.IntegrationTests/
  LdapOpcUaUserAuthenticatorTests.cs` (offline `FakeLdap : ILdapAuthService` + `FakeMapper` — the exact
  seam the prompt's stall-repro needs), `LdapOptionsBindingTests.cs`, `LdapOptionsValidatorTests.cs`,
  `Security.Tests/OtOpcUaLdapAuthServiceTests.cs`, `OpcUaServer.Tests/NodeManagerHistoryReadTests.cs`
  (boots the real SDK server in-process, invokes the public `HistoryRead` → real base dispatch → our
  arms, with a `RecordingHistorianDataSource` fake) and `NodeManagerHistoryReadPagingTests.cs`,
  `Runtime.Tests/Historian/AddServerHistorianTests.cs`. Dev/test LDAP facts confirmed: shared GLAuth
  `10.100.0.35:3893` (bind `cn=serviceaccount,dc=zb,dc=local`) for manual verification; the
  `Host.IntegrationTests` harness uses ephemeral bitnami/openldap on `:3894` for automated runs — the
  outage sim below needs **neither** (an unroutable IP reproduces the outage shape offline).

---

## S2 — HIGH — LDAP authentication block-bridges session activation with a non-cancellable, non-configurable timeout

**Restatement:** The SDK invokes the impersonation callback synchronously (inside a `SessionManager`-wide
event lock — see Verification); `HandleImpersonation` block-bridges the authenticator with
`CancellationToken.None`; no layer enforces a timeout; the shared library's 10 s connect timeout is a
non-projected default; every authentication is a fresh connect + service-bind + search + user-bind.
Under a directory outage, session activations stall **serially, server-wide**, ~10-20 s each.

**Verification:** Confirmed (all anchors above). The event-lock serialization makes this strictly worse
than the report's "stalls SDK threads serially": Anonymous/X509 activations queue behind a stalled
UserName bind too.

**Root cause:** `HandleImpersonation`'s doc comment delegates timeout responsibility to the
authenticator, but `LdapOpcUaUserAuthenticator` never took it up; and OtOpcUa's `LdapOptions` never
projected the library's existing `ConnectionTimeoutMs`, so the 10 s library default is both invisible
and unconfigurable. Nothing bounds concurrent in-flight binds or fails fast during a sustained outage.

**Proposed design — hard wall-clock bound + bounded concurrency + outage circuit, all inside
`LdapOpcUaUserAuthenticator`; project the library timeout.** The SDK callback is irreducibly sync
(verified), so the fix is entirely at the OtOpcUa authenticator boundary:

1. **Project the library timeout** — add `ConnectionTimeoutMs` (default `10000`, matching the library
   default so behavior is unchanged unless configured) to OtOpcUa's `LdapOptions` and project it in
   `ToLibraryOptions()`. This bounds the socket connect and each search op inside the library.
2. **Hard whole-flow timeout at the boundary** — `AuthenticateUserNameAsync` wraps its existing body
   (bind **and** role-map — one budget for everything between SDK-callback entry and result) in
   `core.WaitAsync(TimeSpan.FromMilliseconds(AuthTimeoutMs), ct)`; `TimeoutException` ⇒
   `Deny("Authentication timed out")` (fail-closed) + Warning log. Default `AuthTimeoutMs = 15000` —
   deliberately > `ConnectionTimeoutMs` so a healthy-but-slow full flow (connect + 2 binds + search,
   each individually under 10 s) isn't cut off by the boundary, while an outage (connect stall) is
   bounded by the library timeout first and the boundary is the backstop. Note the abandoned inner
   task runs to completion in the background — the library is documented never-throw, so no unobserved
   exceptions; the point is releasing the SDK thread (and `m_eventLock`).
3. **Bounded in-flight concurrency** — a `SemaphoreSlim(MaxConcurrentAuthentications)` acquired before
   starting the core task and released by a continuation **when the core task completes** (not in the
   caller's `finally`) — so abandoned/orphaned binds count against the cap and an outage can never
   accumulate unbounded orphan tasks. A caller that cannot acquire a slot within its own budget denies
   with `"Authentication backend busy"` (does not count toward the circuit — local backpressure is not
   directory evidence). With the SDK's event-lock serialization only one *SDK* thread is ever in the
   callback at a time, but the cap still bounds orphan accumulation across successive timed-out calls.
4. **Directory-outage circuit (fail-fast)** — after `OutageFailureThreshold` (default 3) *consecutive
   system-side* failures (boundary timeout, `IsSystemFailure` result, or unexpected throw), the
   authenticator denies instantly (`"Authentication backend unavailable"`) for
   `OutageCooldownSeconds` (default 15) without touching LDAP; after cooldown the next attempt probes
   (half-open — the counter stays at threshold so one more system failure re-opens immediately). Any
   success or *user-side* deny resets the counter. Distinguishing system-side from user-side needs a
   structured signal: add `bool IsSystemFailure { get; init; }` to the app `LdapAuthResult`
   (`Security/Ldap/LdapAuthResult.cs` — additive init property, no call-site churn), set by
   `OtOpcUaLdapAuthService.Adapt` for `LdapAuthFailure.ServiceAccountBindFailed` (the library's
   directory-unreachable bucket — its enum has no dedicated `DirectoryUnavailable`; documented
   limitation, `LdapAuthService.cs:94-96`).
5. **Negative cache — REJECTED** (resolves the round-1 "consider a short negative cache" and the
   security caution): a per-credential deny cache requires holding password digests in memory, creates
   a window where a just-corrected password is still rejected, and adds a cache-timing side channel —
   for near-zero benefit, because a bad-credential bind is *fast* when the directory is up; the
   expensive stall only exists when the directory is *down*, which the credential-blind outage circuit
   covers strictly better.

**Alternatives considered:**
- *Make the shared library cooperatively cancellable / pooled* — cross-repo change to
  `ZB.MOM.WW.Auth` (Novell calls are sync-blocking; pooling changes the security posture of bind
  reuse). Track as a shared-lib follow-up; not blocking — the boundary bound is sufficient here.
- *A `ResilientOpcUaUserAuthenticator` decorator over `IOpcUaUserAuthenticator`* — cleaner layering on
  paper, but it would need the system-vs-user failure signal widened onto `OpcUaUserAuthResult`
  (an OpcUaServer-project contract) and a second options plumbing path; the inline fix keeps the
  resilience next to the only LDAP-backed implementation and the signal on the app-owned
  `LdapAuthResult`. Rejected for scope.
- *Pass a real timeout-derived token from `HandleImpersonation`* — the library observes the token at
  entry only (verified), so it cannot shorten anything the boundary `WaitAsync` doesn't already bound.
  Not done; `HandleImpersonation` keeps `CancellationToken.None` and gains a doc comment stating the
  verified SDK contract (sync callback under `m_eventLock`) and where the bound lives.
- *Fully async activation* — impossible; no async impersonation callback exists in 1.5.378 (verified).

**Implementation steps** (task-level detail in the breakdown):
1. `src/Server/ZB.MOM.WW.OtOpcUa.Security/Ldap/LdapOptions.cs` — add five keys (defaults):
   `ConnectionTimeoutMs = 10000` (projected into `ToLibraryOptions()`), `AuthTimeoutMs = 15000`,
   `MaxConcurrentAuthentications = 8`, `OutageFailureThreshold = 3` (`0` disables the circuit),
   `OutageCooldownSeconds = 15`.
2. `src/Server/ZB.MOM.WW.OtOpcUa.Security/Ldap/LdapAuthResult.cs` — add
   `public bool IsSystemFailure { get; init; }`.
3. `src/Server/ZB.MOM.WW.OtOpcUa.Security/Ldap/OtOpcUaLdapAuthService.cs` — `Adapt` sets
   `IsSystemFailure = result.Failure is LdapAuthFailure.ServiceAccountBindFailed` on the failure path.
4. `src/Server/ZB.MOM.WW.OtOpcUa.Host/OpcUa/LdapOpcUaUserAuthenticator.cs` — take
   `IOptions<LdapOptions>`; move the existing body to a private `AuthenticateCoreAsync`; add the
   circuit check → semaphore acquire (release-on-core-completion continuation) → `WaitAsync` timeout →
   circuit bookkeeping, per the design. DI registration (`Host/Program.cs:249`) is unchanged.
5. `src/Server/ZB.MOM.WW.OtOpcUa.Host/Configuration/LdapOptionsValidator.cs` — when
   `Enabled && !DevStubMode`: `ConnectionTimeoutMs > 0`, `AuthTimeoutMs > 0`,
   `MaxConcurrentAuthentications > 0`, and `OutageCooldownSeconds > 0` when
   `OutageFailureThreshold > 0`.
6. `src/Server/ZB.MOM.WW.OtOpcUa.OpcUaServer/OpcUaApplicationHost.cs` — doc-comment only on
   `HandleImpersonation`: record the verified SDK contract (sync `void` handler raised under
   `SessionManager.m_eventLock` in 1.5.378 — activations serialize) and that the wall-clock bound is
   the authenticator's responsibility (now actually enforced).
7. `docs/security.md` — document the five new `Security:Ldap` keys + fail-fast semantics.

**Tests.**
- **Stall-repro (RED first, no real LDAP):** `Host.IntegrationTests` — new
  `LdapAuthResilienceTests` beside the existing suite, reusing its `FakeLdap : ILdapAuthService`
  pattern with a delaying handler. `Authenticate_slow_backend_denies_within_AuthTimeout`
  (fake delays 5 s, `AuthTimeoutMs = 200` → assert Deny + elapsed < ~1.5 s) — **fails on current
  code** (returns Allow after the full 5 s).
- **Unit:** fast success unaffected; timeout denial is fail-closed + logged; circuit opens after N
  consecutive system failures and denies without invoking the fake (call-count pinned); user-side deny
  does *not* trip the circuit; half-open probe after cooldown; success resets; semaphore denies
  "busy" when saturated; `OtOpcUaLdapAuthServiceTests` — `Adapt` maps `ServiceAccountBindFailed` →
  `IsSystemFailure = true`, `BadCredentials` → `false`; `LdapOptionsBindingTests` /
  `LdapOptionsValidatorTests` extended for the new keys. All six existing
  `LdapOpcUaUserAuthenticatorTests` stay green (ctor gains options — update construction).
- **Outage-sim integration (offline-safe):** real `OtOpcUaLdapAuthService` + real library pointed at an
  unroutable host (`10.255.255.1:3893` — TCP SYN blackhole, the exact shape of "stop the GLAuth
  container"), small `ConnectionTimeoutMs`/`AuthTimeoutMs`, driven through the *real*
  `HandleImpersonation` (it is `internal static` — invokable without booting the SDK, precedent
  `OpcUaApplicationHostImpersonationTests`): assert bounded fail + `BadIdentityTokenRejected`.
- **Live/manual (post-merge, not gating):** docker-dev rig — point `Security:Ldap:Server` at an
  unreachable host, drive `Client.CLI connect -u opc.tcp://localhost:4840` with a UserName token
  (e.g. the `multi-role` GLAuth user), confirm fast bounded denial and that a burst of activations
  doesn't serialize for 10 s each; then restore `10.100.0.35:3893` and confirm normal login.

**Effort:** M. **Risk/blast-radius:** Low-Medium. All defaults chosen so behavior is unchanged on a
healthy directory (`ConnectionTimeoutMs` 10 s = current library default; `AuthTimeoutMs` 15 s only cuts
flows that today stall past 15 s; the circuit requires 3 *consecutive system* failures). Fail-closed
posture preserved end-to-end (every new path denies). The AdminUI login path
(`OtOpcUaLdapAuthService` direct) is untouched except the additive `IsSystemFailure` flag.

---

## S3 — HIGH — HistoryRead block-bridges the gateway per node, sequentially, on SDK request threads

**Restatement:** All four HistoryRead arms serve nodes sequentially, each node a
`.GetAwaiter().GetResult()` gateway call with `CancellationToken.None`, bounded only by the 30 s
per-call gateway `CallTimeout`. One request naming N historized nodes against a slow historian holds
one SDK request thread up to N × 30 s against `MaxRequestThreadCount = 100` /
`MaxQueuedRequestCount = 200`; a handful of misbehaving history clients can exhaust the pool and
degrade every OPC UA service.

**Verification:** Confirmed — bridge sites `OtOpcUaNodeManager.cs:1911, 2059, 2171, 2203`; sequential
`foreach` in all four arms; thread-safety audit for parallelization passed (see Verification summary).
`GatewayHistorianDataSource` is already fully async underneath with a per-call deadline — **the
bridging pattern is the problem, not the data source.**

**Root cause:** Per-node reads are serialized with no per-request deadline and no server-side cap on
concurrent HistoryRead work; the block-bridge is structural (the `CustomNodeManager2.HistoryRead*`
override surface is synchronous `void` — verified), but nothing says it must be *per-node* and
*unbounded*.

**Proposed design — bounded per-node fan-out + per-request deadline + process-wide batch limiter,
one block-bridge per arm.** Three composable bounds, each configurable under `ServerHistorian`:

1. **Bounded per-node parallelism within a batch.** Async-ify the per-node serving internals
   (`ServeNode` → `ServeNodeAsync`, `ServeRawPaged` → `ServeRawPagedAsync`, the Events body →
   `ServeEventsAsync`) — each stays fully self-contained try/catch (per-node error isolation
   unchanged). Each arm collects one `Func<Task>` per handle and runs them through a shared
   `RunBounded(work, degree)` helper: a `SemaphoreSlim(degree)`-gated `Task.WhenAll`, block-bridged
   **once** at the arm boundary (`Task.WhenAll(...).GetAwaiter().GetResult()` — safe: bodies never
   throw). Worst case drops from N × 30 s to ~⌈N/P⌉ × 30 s. Degree:
   `HistoryReadBatchParallelism`, default **4** (bounds the gateway-load multiplication a single
   client can command — see Risk).
2. **Per-request deadline.** Each arm creates `using var cts = new CancellationTokenSource(HistoryReadDeadline)`
   (default **60 s**) and threads `cts.Token` into every data-source call (replacing
   `CancellationToken.None` at all four sites). Per-node handlers gain a
   `catch (OperationCanceledException)` arm mapping to **`BadTimeout`** for that node (before the
   generic catch, which keeps mapping other failures to `BadHistoryOperationUnsupported` as today).
   A single request can no longer hold a thread longer than the deadline regardless of node count.
3. **Process-wide concurrent-batch limiter.** A node-manager `SemaphoreSlim(MaxConcurrentHistoryReadBatches)`
   (default **16**, created lazily on first HistoryRead so the post-boot property set is respected —
   same benign wiring race as the existing `MaxTieClusterOverfetch` property) gates each arm's body.
   Entry waits up to `min(HistoryReadDeadline, 5 s)`; on expiry every handle in the batch gets
   **`BadTooManyOperations`** (fail-fast under flood, brief waits under mild contention — resolves the
   round-1 "wait *or* fail" open choice with both, bounded). 16 batches × degree 4 also caps total
   in-flight gateway calls at 64.

**Alternatives considered:**
- *Fully async HistoryRead* — the SDK override surface is synchronous `void` (verified); structural.
  Rejected; parallelize within the sync boundary.
- *`Parallel.ForEachAsync`* for the fan-out — throws `OperationCanceledException` at the *scheduling*
  level on token cancellation, which would bypass per-node error isolation and throw out of the arm.
  The `SemaphoreSlim` + `Task.WhenAll` shape keeps "bodies never throw" airtight. Rejected.
- *Rejecting multi-node history requests* — breaks legitimate clients. Rejected (round-1 concurrence).
- *A channel/queue + dedicated reader pool* — heavier machinery for the same bound; the semaphore
  composition is sufficient and keeps the diff reviewable. Rejected.
- *Per-node continuation-point-aware sub-paging changes* — none needed: paging state
  (`ServeRawPaged`/continuation store) is per-handle and thread-safe (audit above);
  `TimestampsToReturn` handling is untouched (the arms already receive it and the projection helpers
  are pure).

**Implementation steps** (task-level detail in the breakdown):
1. `src/Server/ZB.MOM.WW.OtOpcUa.Runtime/Historian/ServerHistorianOptions.cs` — add
   `HistoryReadBatchParallelism = 4`, `MaxConcurrentHistoryReadBatches = 16`,
   `HistoryReadDeadline = TimeSpan.FromSeconds(60)`; `Validate()` warns on non-positive values
   (Enabled-gated, same pattern as `MaxTieClusterOverfetch`).
2. `src/Server/ZB.MOM.WW.OtOpcUa.OpcUaServer/OtOpcUaNodeManager.cs` — three settable properties
   mirroring the `MaxTieClusterOverfetch` precedent (`HistoryReadBatchParallelism`,
   `MaxConcurrentHistoryReadBatches`, `HistoryReadDeadline`); async-ified per-node internals with
   `ct` parameters; `RunBounded` helper; lazily-created batch-limiter gate; the four arms rewritten to
   limiter → deadline CTS → work-item collection → bounded fan-out → single bridge.
3. `src/Server/ZB.MOM.WW.OtOpcUa.Host/OpcUa/OtOpcUaServerHostedService.cs` — set the three node-manager
   properties beside the existing `MaxTieClusterOverfetch` assignment (`StartAsync`).
4. `docs/Historian.md` + the CLAUDE.md `ServerHistorian` key table — document the three new keys.

**Tests.**
- **Stall-repro (RED first, no real gateway):** `OpcUaServer.Tests` — new
  `NodeManagerHistoryReadConcurrencyTests` reusing the `NodeManagerHistoryReadTests` boot harness with
  a delaying fake `IHistorianDataSource` that tracks max observed concurrent reads (interlocked).
  `Raw_batch_is_served_with_bounded_parallelism`: 8 historized nodes, 150 ms delay per read → assert
  `maxConcurrent >= 2` (deterministic — **fails on current sequential code** where it is pinned at 1)
  + a loose wall-clock secondary bound (< ~⌈8/4⌉ × 150 ms + slack).
- **Unit:** per-node isolation regression under parallelism (one node's fake throws → that node Bad,
  others Good — the existing guarantee, re-proven concurrent); parallelism also bounded above
  (`maxConcurrent <= degree`); deadline — a fake whose task only completes on token cancellation →
  every node `BadTimeout` within deadline + slack, call returns (fails-by-hanging is prevented by
  wrapping the invoke in `Task.Run(...).WaitAsync(10 s)`); limiter — `MaxConcurrentHistoryReadBatches=1`,
  first batch parked in the fake, second concurrent `HistoryRead` (small `HistoryReadDeadline` so the
  wait budget is short) → all its handles `BadTooManyOperations`; existing
  `NodeManagerHistoryReadTests` + `NodeManagerHistoryReadPagingTests` (incl. tie-cluster paths) stay
  green — the paging suite is the continuation-point-correctness regression net;
  `AddServerHistorianTests` / options tests extended for the new keys.
- **Integration/live (post-merge, not gating):** the env-gated `Category=LiveIntegration` gateway suite
  (VPN + real HistorianGateway) re-run; optionally a manual soak — one client issuing a 50-node Raw
  HistoryRead against a slow historian while a second client browses/subscribes, confirming the second
  client stays responsive (the pool-exhaustion scenario).

**Effort:** M. **Risk/blast-radius:** Medium. Parallelizing multiplies peak gateway load by up to
`HistoryReadBatchParallelism` per request (default 4, total in-flight capped at 64 by the limiter —
the knobs exist precisely to tune this against a weak historian). Result/error index writes verified
disjoint; all touched shared state verified concurrent-safe. Client-visible behavior changes only in
failure shapes (`BadTimeout` on deadline, `BadTooManyOperations` under flood — both standard,
spec-conformant codes) and latency (better). Continuation-point semantics untouched.

---

## Task breakdown

Branch: `fix/archreview-r2-08-ldap-historyread-async` off `master@f6eaa267`. Tasks are ≤5 min each;
TDD-ordered (each RED repro precedes its implementation). Test commands run from the repo root; all
automated tests are offline-safe (no Docker, no VPN, no real LDAP/gateway).

---

### R2-08-T1 — S2 stall-repro test (RED — must fail on current code)

**Classification:** test-first repro (deterministic unit, fake `ILdapAuthService` with a delay)
**Estimated implement time:** 5 min
**Parallelizable with:** T7, T8 (different files/subsystem).
**Files:** `tests/Server/ZB.MOM.WW.OtOpcUa.Host.IntegrationTests/LdapAuthResilienceTests.cs` (new)

1. New test class beside `LdapOpcUaUserAuthenticatorTests`, cloning its `FakeLdap`/`FakeMapper`
   pattern but with an async handler seam: `DelayingFakeLdap(TimeSpan delay, LdapAuthResult then)`
   (`await Task.Delay(delay, ct)` before returning) and an `Interlocked` call counter.
2. Add `Authenticate_slow_backend_denies_within_AuthTimeout`: construct `LdapOpcUaUserAuthenticator`
   exactly as the existing tests do (this is RED partly *because* there is no way to pass a timeout
   yet — write the test against the intended ctor shape
   `new LdapOpcUaUserAuthenticator(ldap, scopeFactory, Options.Create(new LdapOptions { AuthTimeoutMs = 200 }), logger)`;
   until T4 lands the file won't compile — acceptable for a same-branch RED, or temporarily pin with
   the current ctor + assert elapsed, which fails by timing). Fake delays 5 s; assert
   `result.Success == false`, `result.Error` contains "timed out", and a `Stopwatch` elapsed < 1.5 s.
   Wrap the await in `.WaitAsync(TimeSpan.FromSeconds(10))` so a regression fails rather than hangs.
3. `dotnet test tests/Server/ZB.MOM.WW.OtOpcUa.Host.IntegrationTests --filter "FullyQualifiedName~LdapAuthResilienceTests.Authenticate_slow_backend_denies_within_AuthTimeout"`
   → **expected: FAIL** (compile-fail against intended ctor, or Allow-after-5s timing fail).

Commit: `test(host): red repro — LDAP authenticator has no wall-clock bound on session activation (03/S2)`

---

### R2-08-T2 — LdapOptions keys + library projection + validator

**Classification:** implementation (options + validation, additive)
**Estimated implement time:** 5 min
**Parallelizable with:** T3, T7, T8.
**Files:** `src/Server/ZB.MOM.WW.OtOpcUa.Security/Ldap/LdapOptions.cs`,
`src/Server/ZB.MOM.WW.OtOpcUa.Host/Configuration/LdapOptionsValidator.cs`,
`tests/Server/ZB.MOM.WW.OtOpcUa.Host.IntegrationTests/LdapOptionsBindingTests.cs`,
`tests/Server/ZB.MOM.WW.OtOpcUa.Host.IntegrationTests/LdapOptionsValidatorTests.cs`

1. RED: extend `LdapOptionsBindingTests` — a config section carrying the five new keys binds them;
   defaults hold when absent; `ToLibraryOptions().ConnectionTimeoutMs` mirrors the app value. Extend
   `LdapOptionsValidatorTests` — non-positive `ConnectionTimeoutMs`/`AuthTimeoutMs`/
   `MaxConcurrentAuthentications` fail validation when `Enabled && !DevStubMode`;
   `OutageFailureThreshold = 3` with `OutageCooldownSeconds = 0` fails; `OutageFailureThreshold = 0`
   with any cooldown passes; disabled/DevStub configs stay exempt.
   `dotnet test tests/Server/ZB.MOM.WW.OtOpcUa.Host.IntegrationTests --filter "FullyQualifiedName~LdapOptionsBindingTests|FullyQualifiedName~LdapOptionsValidatorTests"`
   → **expected: FAIL** (properties don't exist).
2. GREEN: add `ConnectionTimeoutMs = 10000`, `AuthTimeoutMs = 15000`,
   `MaxConcurrentAuthentications = 8`, `OutageFailureThreshold = 3`, `OutageCooldownSeconds = 15`
   (XML-doc each: what it bounds, why the default, `0` disables the circuit); project
   `ConnectionTimeoutMs` in `ToLibraryOptions()`; add the validator rules. Re-run the filter → **PASS**.

Commit: `feat(security): LDAP auth timeout/concurrency/outage-circuit options; project ConnectionTimeoutMs into the shared library (03/S2)`

---

### R2-08-T3 — `LdapAuthResult.IsSystemFailure` seam

**Classification:** implementation (additive contract field + adapter mapping)
**Estimated implement time:** 5 min
**Parallelizable with:** T2, T7, T8.
**Files:** `src/Server/ZB.MOM.WW.OtOpcUa.Security/Ldap/LdapAuthResult.cs`,
`src/Server/ZB.MOM.WW.OtOpcUa.Security/Ldap/OtOpcUaLdapAuthService.cs`,
`tests/Server/ZB.MOM.WW.OtOpcUa.Security.Tests/OtOpcUaLdapAuthServiceTests.cs`

1. RED: extend `OtOpcUaLdapAuthServiceTests` — library `ServiceAccountBindFailed` adapts to
   `IsSystemFailure == true`; `BadCredentials`/`UserNotFound`/success adapt to `false`.
   `dotnet test tests/Server/ZB.MOM.WW.OtOpcUa.Security.Tests --filter "FullyQualifiedName~OtOpcUaLdapAuthServiceTests"`
   → **expected: FAIL**.
2. GREEN: add `public bool IsSystemFailure { get; init; }` (XML-doc: "system-side directory failure —
   unreachable/misconfigured — as opposed to a user-credential deny; feeds the OPC UA authenticator's
   outage circuit"); set it in `Adapt`'s failure branch for `ServiceAccountBindFailed`. Re-run →
   **PASS** (whole class — existing adapt tests must stay green).

Commit: `feat(security): LdapAuthResult.IsSystemFailure distinguishes directory outage from credential deny (03/S2)`

---

### R2-08-T4 — Authenticator wall-clock bound + concurrency cap (turns T1 green)

**Classification:** implementation (the core S2 fix)
**Estimated implement time:** 5 min
**Parallelizable with:** T7-T13 (S3 files). Blocked by T1, T2.
**Files:** `src/Server/ZB.MOM.WW.OtOpcUa.Host/OpcUa/LdapOpcUaUserAuthenticator.cs`,
`tests/Server/ZB.MOM.WW.OtOpcUa.Host.IntegrationTests/LdapAuthResilienceTests.cs`,
`tests/Server/ZB.MOM.WW.OtOpcUa.Host.IntegrationTests/LdapOpcUaUserAuthenticatorTests.cs`

1. Add `IOptions<LdapOptions>` to the ctor; move the existing method body to
   `private async Task<OpcUaUserAuthResult> AuthenticateCoreAsync(...)` unchanged; new outer body:
   semaphore `WaitAsync(AuthTimeoutMs, ct)` → deny `"Authentication backend busy"` on false; start
   `core = AuthenticateCoreAsync(...)` with a `core.ContinueWith(_ => _gate.Release(), TaskScheduler.Default)`
   release-on-completion; `await core.WaitAsync(TimeSpan.FromMilliseconds(AuthTimeoutMs), ct)` in
   try/catch — `TimeoutException` ⇒ Warning log + deny `"Authentication timed out"`. Update the class
   XML-doc (why: SDK sync callback under `m_eventLock`).
2. Update the six existing `LdapOpcUaUserAuthenticatorTests` constructions to pass
   `Options.Create(new LdapOptions())`.
3. Add `Authenticate_fast_success_unaffected` and
   `Authenticate_saturated_concurrency_denies_busy` (`MaxConcurrentAuthentications = 1`, first call
   parked in the fake, second denies "busy" within bound) to `LdapAuthResilienceTests`.
4. `dotnet test tests/Server/ZB.MOM.WW.OtOpcUa.Host.IntegrationTests --filter "FullyQualifiedName~LdapAuthResilienceTests|FullyQualifiedName~LdapOpcUaUserAuthenticatorTests"`
   → **expected: PASS** (T1's repro now green; all six pre-existing tests green).

Commit: `feat(host): hard wall-clock bound + bounded concurrency on OPC UA LDAP authentication (03/S2)`

---

### R2-08-T5 — Directory-outage circuit (fail-fast during sustained outage)

**Classification:** implementation (resilience state machine)
**Estimated implement time:** 5 min
**Parallelizable with:** T7-T13. Blocked by T3, T4.
**Files:** `src/Server/ZB.MOM.WW.OtOpcUa.Host/OpcUa/LdapOpcUaUserAuthenticator.cs`,
`tests/Server/ZB.MOM.WW.OtOpcUa.Host.IntegrationTests/LdapAuthResilienceTests.cs`

1. RED: add circuit tests — three consecutive timeouts (or `IsSystemFailure` denies) open the circuit:
   the 4th call denies `"Authentication backend unavailable"` **without** invoking the fake (call
   counter pinned at 3) and returns in ~0 ms; a user-side deny (`Invalid username or password`,
   `IsSystemFailure=false`) between failures resets the streak; after `OutageCooldownSeconds` (use 1 s
   in test) the next call probes the fake (half-open) and a success closes the circuit;
   `OutageFailureThreshold = 0` never opens.
   `dotnet test tests/Server/ZB.MOM.WW.OtOpcUa.Host.IntegrationTests --filter "FullyQualifiedName~LdapAuthResilienceTests"`
   → **expected: FAIL** (new tests only).
2. GREEN: small lock-guarded state (`_consecutiveSystemFailures`, `_circuitOpenUntil` via
   `Environment.TickCount64`); entry check denies while open (Debug log; single Warning when opening,
   Information when closing); system failure = boundary timeout ∨ `result.IsSystemFailure` ∨ unexpected
   throw from the core; success or user-side deny resets. Re-run → **PASS**.

Commit: `feat(host): LDAP directory-outage circuit — fail-fast denial during sustained outage (03/S2)`

---

### R2-08-T6 — Outage-sim integration test + `HandleImpersonation` contract doc

**Classification:** integration test (offline-safe) + doc
**Estimated implement time:** 5 min
**Parallelizable with:** T7-T13. Blocked by T4.
**Files:** `tests/Server/ZB.MOM.WW.OtOpcUa.Host.IntegrationTests/LdapAuthResilienceTests.cs`,
`src/Server/ZB.MOM.WW.OtOpcUa.OpcUaServer/OpcUaApplicationHost.cs`, `docs/security.md`

1. Add `Activation_with_unreachable_directory_fails_within_bound`: real `OtOpcUaLdapAuthService`
   (real `ZB.MOM.WW.Auth` library, no fake) with `Server = "10.255.255.1"`, `Port = 3893`,
   `Transport = None`, `AllowInsecure = true`, `ConnectionTimeoutMs = 500`, `AuthTimeoutMs = 1000`;
   wrap in `LdapOpcUaUserAuthenticator`; drive the real `internal static`
   `OpcUaApplicationHost.HandleImpersonation` with a `UserNameIdentityToken` (precedent:
   `OpcUaApplicationHostImpersonationTests`); assert `args.IdentityValidationError` is
   `BadIdentityTokenRejected` and total elapsed < ~3 s. (This is the automated stand-in for
   "stop the GLAuth container" — a TCP blackhole is the same outage shape; the bitnami/openldap `:3894`
   harness is not needed for the outage direction.)
2. Rewrite `HandleImpersonation`'s doc comment: verified 1.5.378 contract (synchronous `void` handler
   raised under `SessionManager.m_eventLock` — a stalled handler serializes *all* activations), why the
   block-bridge is irreducible, and where the wall-clock bound + circuit live.
3. Document the new `Security:Ldap` keys + semantics in `docs/security.md`.
4. `dotnet test tests/Server/ZB.MOM.WW.OtOpcUa.Host.IntegrationTests --filter "FullyQualifiedName~LdapAuthResilienceTests"`
   → **expected: PASS**.

Commit: `test(host): unreachable-directory activation fails bounded; document the SDK impersonation threading contract (03/S2)`

---

### R2-08-T7 — S3 sequential-serving repro test (RED — must fail on current code)

**Classification:** test-first repro (deterministic — concurrency counter, not raw timing)
**Estimated implement time:** 5 min
**Parallelizable with:** T1-T6, T8.
**Files:** `tests/Server/ZB.MOM.WW.OtOpcUa.OpcUaServer.Tests/NodeManagerHistoryReadConcurrencyTests.cs` (new)

1. New class reusing the `NodeManagerHistoryReadTests` boot pattern (`BootAsync` + `EnsureVariable` +
   public `HistoryRead` invoke). Fake `IHistorianDataSource` whose `ReadRawAsync` does
   `Interlocked.Increment` a live-counter, records the max, `await Task.Delay(150ms, ct)`, decrements,
   returns one sample.
2. Add `Raw_batch_is_served_with_bounded_parallelism`: materialise 8 historized variables; issue one
   Raw `HistoryRead` naming all 8; assert `fake.MaxObservedConcurrency >= 2` (primary — deterministic)
   and elapsed < ~1.0 s (secondary, loose). Wrap the invoke in `Task.Run(...).WaitAsync(15 s)`.
3. `dotnet test tests/Server/ZB.MOM.WW.OtOpcUa.OpcUaServer.Tests --filter "FullyQualifiedName~NodeManagerHistoryReadConcurrencyTests.Raw_batch_is_served_with_bounded_parallelism"`
   → **expected: FAIL** (`MaxObservedConcurrency == 1` on the sequential `foreach`).

Commit: `test(opcuaserver): red repro — HistoryRead serves a batch strictly sequentially (03/S3)`

---

### R2-08-T8 — `ServerHistorian` keys + hosted-service wiring

**Classification:** implementation (options + wiring, additive)
**Estimated implement time:** 5 min
**Parallelizable with:** T1-T6, T7.
**Files:** `src/Server/ZB.MOM.WW.OtOpcUa.Runtime/Historian/ServerHistorianOptions.cs`,
`src/Server/ZB.MOM.WW.OtOpcUa.OpcUaServer/OtOpcUaNodeManager.cs` (properties only),
`src/Server/ZB.MOM.WW.OtOpcUa.Host/OpcUa/OtOpcUaServerHostedService.cs`,
`tests/Server/ZB.MOM.WW.OtOpcUa.Runtime.Tests/Historian/AddServerHistorianTests.cs` (or the options
test file beside it)

1. RED: options tests — defaults (`4` / `16` / `60 s`); `Validate()` warns on non-positive
   parallelism/batches/deadline when `Enabled` (and stays silent when disabled).
   `dotnet test tests/Server/ZB.MOM.WW.OtOpcUa.Runtime.Tests --filter "FullyQualifiedName~ServerHistorianOptions|FullyQualifiedName~AddServerHistorianTests"`
   → **expected: FAIL**.
2. GREEN: add `HistoryReadBatchParallelism = 4`, `MaxConcurrentHistoryReadBatches = 16`,
   `HistoryReadDeadline = TimeSpan.FromSeconds(60)` (+ XML-doc each: what it bounds, the
   ⌈N/P⌉ × CallTimeout worst case, 16 × 4 = 64 in-flight cap) + `Validate()` warnings; add the three
   settable node-manager properties (defaults matching — `MaxTieClusterOverfetch` precedent, unused
   until T9-T13); assign all three in `OtOpcUaServerHostedService.StartAsync` beside the existing
   `MaxTieClusterOverfetch` line. Re-run → **PASS**.

Commit: `feat(runtime): ServerHistorian HistoryRead parallelism/limiter/deadline knobs + node-manager wiring (03/S3)`

---

### R2-08-T9 — Async-ify `ServeNode` + Processed/AtTime arms (still sequential)

**Classification:** implementation (mechanical async refactor — behavior-neutral)
**Estimated implement time:** 5 min
**Parallelizable with:** T1-T6. Blocked by T7 (repro must be red first), T8 (ct/property names).
**Files:** `src/Server/ZB.MOM.WW.OtOpcUa.OpcUaServer/OtOpcUaNodeManager.cs`

1. `ServeNode` → `private async Task ServeNodeAsync(handle, results, errors, read, CancellationToken ct)`;
   the `read` callback gains the token (`Func<IHistorianDataSource, string, CancellationToken, Task<HistorianRead>>`);
   `await` instead of bridging; add `catch (OperationCanceledException)` → `errors/results = BadTimeout`
   ahead of the generic catch. Processed/AtTime arms: pass `ct` into the source calls
   (`:1841`/`:1865` today `CancellationToken.None`), collect the per-handle tasks but for now run them
   sequentially with a single arm-boundary bridge (`foreach … ServeNodeAsync(...).GetAwaiter().GetResult()`
   is acceptable interim — the fan-out lands in T11). Thread a `ct` parameter down from the arm (pass
   `CancellationToken.None` at the arm until T12).
2. `dotnet test tests/Server/ZB.MOM.WW.OtOpcUa.OpcUaServer.Tests --filter "FullyQualifiedName~NodeManagerHistoryReadTests"`
   → **expected: PASS** (behavior-neutral; T7's repro stays red).

Commit: `refactor(opcuaserver): async-ify ServeNode + Processed/AtTime HistoryRead internals, thread cancellation (03/S3)`

---

### R2-08-T10 — Async-ify `ServeRawPaged` + Events body

**Classification:** implementation (mechanical async refactor — behavior-neutral)
**Estimated implement time:** 5 min
**Parallelizable with:** T1-T6. Blocked by T9 (shared `BadTimeout` pattern/signature conventions).
**Files:** `src/Server/ZB.MOM.WW.OtOpcUa.OpcUaServer/OtOpcUaNodeManager.cs`

1. `ServeRawPaged` → `ServeRawPagedAsync(..., CancellationToken ct)`: `await` the page read (`:2171`)
   and the tie-cluster over-fetch (`:2203`) with `ct`; add the OCE → `BadTimeout` arm; continuation
   store calls unchanged (thread-safe, audited).
2. Extract the `HistoryReadEvents` per-handle body into `ServeEventsAsync(handle, sourceName, details,
   selectClauses, results, errors, ct)` — `await` the `:1911` read with `ct`; OCE → `BadTimeout`;
   existing catch preserved. Both arms keep the interim sequential arm-boundary bridge.
3. `dotnet test tests/Server/ZB.MOM.WW.OtOpcUa.OpcUaServer.Tests --filter "FullyQualifiedName~NodeManagerHistoryReadTests|FullyQualifiedName~NodeManagerHistoryReadPagingTests"`
   → **expected: PASS** (paging + tie-cluster suites are the regression net).

Commit: `refactor(opcuaserver): async-ify ServeRawPaged + HistoryReadEvents internals (03/S3)`

---

### R2-08-T11 — Bounded per-node fan-out in all four arms (turns T7 green)

**Classification:** implementation (the core S3 parallelism fix)
**Estimated implement time:** 5 min
**Parallelizable with:** T1-T6. Blocked by T9, T10.
**Files:** `src/Server/ZB.MOM.WW.OtOpcUa.OpcUaServer/OtOpcUaNodeManager.cs`,
`tests/Server/ZB.MOM.WW.OtOpcUa.OpcUaServer.Tests/NodeManagerHistoryReadConcurrencyTests.cs`

1. Add `private void RunBounded(IReadOnlyList<Func<Task>> work, int degree)`: `degree <= 1` or single
   item ⇒ run sequentially; else `SemaphoreSlim(degree)`-gated `Task.WhenAll`, bridged once
   (`GetAwaiter().GetResult()`; safe — every body is self-contained try/catch and never throws;
   document that invariant on the helper).
2. Rewrite the four arms: pre-filter the cheap synchronous rejections inline (IsReadModified,
   non-notifier Events handles), collect one `Func<Task>` per remaining handle, call
   `RunBounded(work, HistoryReadBatchParallelism)`.
3. Extend the concurrency test class: `Parallelism_is_bounded_above` (12 nodes, property set to 3 →
   `MaxObservedConcurrency <= 3`) and `One_node_failure_does_not_poison_a_parallel_batch` (one handle's
   fake throws → that node `BadHistoryOperationUnsupported`, the other 7 Good).
4. `dotnet test tests/Server/ZB.MOM.WW.OtOpcUa.OpcUaServer.Tests --filter "FullyQualifiedName~NodeManagerHistoryRead"`
   → **expected: PASS** (T7 repro green; paging + base suites green).

Commit: `feat(opcuaserver): bounded per-node parallel HistoryRead within a batch (03/S3)`

---

### R2-08-T12 — Per-request deadline

**Classification:** implementation (deadline CTS + BadTimeout surfacing)
**Estimated implement time:** 5 min
**Parallelizable with:** T1-T6. Blocked by T11.
**Files:** `src/Server/ZB.MOM.WW.OtOpcUa.OpcUaServer/OtOpcUaNodeManager.cs`,
`tests/Server/ZB.MOM.WW.OtOpcUa.OpcUaServer.Tests/NodeManagerHistoryReadConcurrencyTests.cs`

1. RED: `Deadline_bounds_a_hung_backend` — fake whose reads complete only on token cancellation
   (`await Task.Delay(Timeout.Infinite, ct)`); `HistoryReadDeadline = 500 ms`; 4 nodes → every node
   `BadTimeout`, call returns within ~2 s (invoke wrapped in `.WaitAsync(10 s)`).
   `dotnet test tests/Server/ZB.MOM.WW.OtOpcUa.OpcUaServer.Tests --filter "FullyQualifiedName~NodeManagerHistoryReadConcurrencyTests.Deadline_bounds_a_hung_backend"`
   → **expected: FAIL** (hangs 30 s+ / `WaitAsync` trips — nothing cancels today).
2. GREEN: each arm creates `using var cts = HistoryReadDeadline > TimeSpan.Zero ? new CancellationTokenSource(HistoryReadDeadline) : null`
   and passes `cts?.Token ?? CancellationToken.None` down (defensive: non-positive = unbounded,
   matching the options warning). Re-run → **PASS**.

Commit: `feat(opcuaserver): per-request HistoryRead deadline — hung backends surface BadTimeout (03/S3)`

---

### R2-08-T13 — Process-wide concurrent-batch limiter

**Classification:** implementation (server-side admission control)
**Estimated implement time:** 5 min
**Parallelizable with:** T1-T6. Blocked by T12.
**Files:** `src/Server/ZB.MOM.WW.OtOpcUa.OpcUaServer/OtOpcUaNodeManager.cs`,
`tests/Server/ZB.MOM.WW.OtOpcUa.OpcUaServer.Tests/NodeManagerHistoryReadConcurrencyTests.cs`

1. RED: `Saturated_limiter_rejects_with_BadTooManyOperations` — `MaxConcurrentHistoryReadBatches = 1`,
   `HistoryReadDeadline = 200 ms`; first batch parked in a gated fake; a second concurrent
   `HistoryRead` → all its handles `BadTooManyOperations`, returns within ~1 s; release the gate; a
   third read succeeds (slot freed).
   `dotnet test tests/Server/ZB.MOM.WW.OtOpcUa.OpcUaServer.Tests --filter "FullyQualifiedName~NodeManagerHistoryReadConcurrencyTests.Saturated_limiter_rejects_with_BadTooManyOperations"`
   → **expected: FAIL** (no limiter exists).
2. GREEN: lazily-created `SemaphoreSlim` (double-checked, sized from `MaxConcurrentHistoryReadBatches`
   at first use — respects the post-boot property set, same benign race as `MaxTieClusterOverfetch`);
   arm entry `Wait(min(HistoryReadDeadline, 5 s))` — on false, set `BadTooManyOperations` on every
   handle and return; `try/finally` release. Re-run → **PASS**.

Commit: `feat(opcuaserver): process-wide HistoryRead batch limiter — flood degrades to BadTooManyOperations, not pool exhaustion (03/S3)`

---

### R2-08-T14 — Docs + full regression sweep

**Classification:** docs + verification
**Estimated implement time:** 5 min
**Parallelizable with:** —. Blocked by T5, T6, T13.
**Files:** `docs/Historian.md`, `docs/security.md` (if residue from T6), `CLAUDE.md`
(`ServerHistorian` key table), `archreview/plans/STATUS.md`

1. Add the three `ServerHistorian` keys to `docs/Historian.md` + the CLAUDE.md table (defaults +
   one-line semantics); confirm `docs/security.md` covers the five `Security:Ldap` keys.
2. Full touched-suite sweep:
   `dotnet test tests/Server/ZB.MOM.WW.OtOpcUa.Host.IntegrationTests tests/Server/ZB.MOM.WW.OtOpcUa.Security.Tests tests/Server/ZB.MOM.WW.OtOpcUa.OpcUaServer.Tests tests/Server/ZB.MOM.WW.OtOpcUa.Runtime.Tests`
   (or per-project) → all green; then whole-solution `dotnet test ZB.MOM.WW.OtOpcUa.slnx` (the CI
   fail-on-skip gate applies).
3. Record R2-08 outcome in `archreview/plans/STATUS.md` (branch, commits, test counts).

Commit: `docs: ServerHistorian HistoryRead knobs + Security:Ldap resilience keys; R2-08 status (03/S2, 03/S3)`

---

## Post-merge (not gating, operator-run)

- **S2 live check:** docker-dev rig — unreachable `Security:Ldap:Server` → Client.CLI UserName connect
  denies fast; restore shared GLAuth `10.100.0.35:3893` → normal login. Optionally stop/start the
  GLAuth container on the docker host for a true outage/recovery cycle (circuit opens, then closes).
- **S3 live check:** env-gated `Category=LiveIntegration` gateway suite on the VPN; a 50-node Raw
  HistoryRead soak against the real gateway while a second client stays interactive.

## Overall effort

**M (≈1.5-2 dev-days):** 6 tasks S2 + 7 tasks S3 + 1 docs/sweep, each ≤5 min of implementation but with
the usual test-run + review cadence. Risk concentrated in T11 (parallel fan-out — mitigated by the
thread-safety audit + the paging regression suite) and T4/T5 (auth-path semantics — mitigated by the
fail-closed default posture and behavior-neutral defaults).