lmxopcua/archreview/plans/R2-08-ldap-historyread-async-plan.md

Design + Implementation Plan — R2-08: Async LDAP authentication + HistoryRead bridging

• Source report: archreview/03-server-runtime.md (2026-07-12 re-review, working-tree version)
• Review commit: f6eaa267 · Plan verified against tree at: f6eaa267 (master, clean)
• Scope: OVERALL prioritized action item #8 — "Async LDAP + configurable timeout; channel-ize HistoryRead bridging" — covering findings 03/S2 (High — LDAP authentication block-bridges OPC UA session activation with an unconfigurable ~10 s timeout, no pooling, no fail-fast) and 03/S3 (High — HistoryRead block-bridges the gateway per node, sequentially, on SDK request threads, bounded only by the 30 s gateway CallTimeout). Both were carried unfixed from round 1 (archreview/plans/03-server-runtime-plan.md §S2/§S3 — designs verified + refreshed below).
• Touches: Security/Ldap/LdapOptions.cs, Security/Ldap/OtOpcUaLdapAuthService.cs, Security/Ldap/LdapAuthResult.cs, Host/OpcUa/LdapOpcUaUserAuthenticator.cs, Host/Configuration/LdapOptionsValidator.cs, OpcUaServer/OpcUaApplicationHost.cs (doc only), OpcUaServer/OtOpcUaNodeManager.cs, Runtime/Historian/ServerHistorianOptions.cs, Host/OpcUa/OtOpcUaServerHostedService.cs, plus tests in Host.IntegrationTests, Security.Tests, Runtime.Tests, OpcUaServer.Tests. No shared-lib (ZB.MOM.WW.Auth) change required — see Verification.

Verification summary

Every anchor in both findings was opened against the working tree at f6eaa267; the round-1 plan's S2/S3 designs were re-verified against the code and the actual SDK + shared-library sources. Both findings CONFIRMED — and S2 is materially worse than the report states. Details:

• S2 anchors — confirmed, one path drift. The report cites src/Server/ZB.MOM.WW.OtOpcUa.Host/OpcUa/OpcUaApplicationHost.cs:271-274; the file actually lives at src/Server/ZB.MOM.WW.OtOpcUa.OpcUaServer/OpcUaApplicationHost.cs (the authenticator is under Host/OpcUa/). The block-bridge is exact: HandleImpersonation calls authenticator.AuthenticateUserNameAsync(…, CancellationToken.None).GetAwaiter().GetResult() (OpcUaApplicationHost.cs:270-273). LdapOpcUaUserAuthenticator.AuthenticateUserNameAsync (Host/OpcUa/LdapOpcUaUserAuthenticator.cs:30-48) adds no timeout, no concurrency bound, no fail-fast. LdapOptions.ToLibraryOptions() (Security/Ldap/LdapOptions.cs:94-107) projects no timeout field and LdapOptions has none.
• NEW (verified from SDK source, worse than reported): the impersonation callback runs inside a global event lock. Verified against the pinned OPCFoundation.NetStandard.Opc.Ua.Server 1.5.378.106 sources (Libraries/Opc.Ua.Server/Session/SessionManager.cs): ActivateSessionAsync raises the handler as lock (m_eventLock) { … m_ImpersonateUser(session, args); … } (lines 352-360; m_eventLock is a single SessionManager-wide Lock, line 698). The handler signature is a synchronous void delegate; there is no async variant of the impersonation callback (the surrounding ActivateSessionAsync is ValueTask-returning, but the callback itself must complete inline). Consequence: one stalled LDAP bind does not just hold one SDK thread — it serializes every session activation server-wide (Anonymous and X509 included, since even the early-return path must acquire m_eventLock to invoke the handler). The SDK contract is therefore irreducibly sync → the fix is a hard wall-clock bound + bounded concurrency + outage fail-fast at the authenticator boundary, exactly the shape the report recommends. No "fake async" is attempted.
• NEW (resolves a round-1 open choice): the shared library already exposes the timeout. ZB.MOM.WW.Auth.Abstractions.Ldap.LdapOptions.ConnectionTimeoutMs exists (default 10_000, LdapContracts.cs:20 in ~/Desktop/scadaproj/ZB.MOM.WW.Auth), and the Novell adapter applies it to both the socket connect (_conn.ConnectionTimeout) and the per-operation search constraint (constraints.TimeLimit) (NovellLdapConnection.ApplyTimeout, lines 131-146). The round-1 plan's step 1 said "do not project into ToLibraryOptions() … unless the library later exposes a matching field" — the field exists; this plan projects it. Also verified: the library's AuthenticateAsync is Task.FromResult(Authenticate(...)) — synchronous-in-Task, token observed at entry only (LdapAuthService.cs:59-66), and each call is a fresh connect + service-bind + search + user-bind (no pooling; pooling stays a shared-lib follow-up, out of scope here). The library is fully fail-closed (never throws; system-side faults map to LdapAuthFailure.ServiceAccountBindFailed).
• S3 anchors — all four bridge sites line-exact at f6eaa267: OtOpcUaNodeManager.cs:1911 (Events), :2059 (ServeNode — serves Processed + AtTime), :2171 (Raw page read), :2203 (Raw tie-cluster over-fetch) — each .GetAwaiter().GetResult() with CancellationToken.None. The four arms iterate nodesToProcess in a sequential foreach (:1784 RawModified, :1811 Processed, :1851 AtTime, :1873 Events). SDK thread budget confirmed: MinRequestThreadCount = 5, MaxRequestThreadCount = 100, MaxQueuedRequestCount = 200 (OpcUaServer/OpcUaApplicationHost.cs:333-335). The overrides run outside the node-manager Lock (documented at :1777-1780) — this is a request-pool-exhaustion risk, not a lock-freeze risk.
• S3 thread-safety audit (round-1 plan asked for it — done): parallel per-node serving is safe. results[handle.Index] / errors[handle.Index] writes are index-disjoint into pre-sized lists (the base dispatcher seeds every slot). All shared node-manager maps the read path touches are ConcurrentDictionary (_variables:41, _historizedTagnames:55, _eventNotifierSources:69). IHistoryContinuationStore is explicitly thread-safe (SDK per-session store — "the SDK session locks internally", HistoryContinuationStore.cs; the test store is a lock-protected dictionary). ServeRawPaged state is all per-handle locals. IHistorianDataSource (the gateway client) is a gRPC channel wrapper — safe for concurrent calls; each call already carries the 30 s CallTimeout deadline (ServerHistorianOptions.CallTimeout, ServerHistorianOptions.cs:58).
• Round-1 plan §S3 design re-verified: bounded per-node parallelism + per-request deadline + server-side limiter all still fit the tree; refined below (OCE→BadTimeout per node, limiter wait budget, lazy gate creation, options names) and the "fully async HistoryRead" alternative re-confirmed impossible — CustomNodeManager2's HistoryRead* overrides are synchronous void fillers.
• Existing test harnesses located (reused, not invented): Host.IntegrationTests/ LdapOpcUaUserAuthenticatorTests.cs (offline FakeLdap : ILdapAuthService + FakeMapper — the exact seam the prompt's stall-repro needs), LdapOptionsBindingTests.cs, LdapOptionsValidatorTests.cs, Security.Tests/OtOpcUaLdapAuthServiceTests.cs, OpcUaServer.Tests/NodeManagerHistoryReadTests.cs (boots the real SDK server in-process, invokes the public HistoryRead → real base dispatch → our arms, with a RecordingHistorianDataSource fake) and NodeManagerHistoryReadPagingTests.cs, Runtime.Tests/Historian/AddServerHistorianTests.cs. Dev/test LDAP facts confirmed: shared GLAuth 10.100.0.35:3893 (bind cn=serviceaccount,dc=zb,dc=local) for manual verification; the Host.IntegrationTests harness uses ephemeral bitnami/openldap on :3894 for automated runs — the outage sim below needs neither (an unroutable IP reproduces the outage shape offline).

---

S2 — HIGH — LDAP authentication block-bridges session activation with a non-cancellable, non-configurable timeout

Restatement: The SDK invokes the impersonation callback synchronously (inside a SessionManager-wide event lock — see Verification); HandleImpersonation block-bridges the authenticator with CancellationToken.None; no layer enforces a timeout; the shared library's 10 s connect timeout is a non-projected default; every authentication is a fresh connect + service-bind + search + user-bind. Under a directory outage, session activations stall serially, server-wide, ~10-20 s each.

Verification: Confirmed (all anchors above). The event-lock serialization makes this strictly worse than the report's "stalls SDK threads serially": Anonymous/X509 activations queue behind a stalled UserName bind too.

Root cause: HandleImpersonation's doc comment delegates timeout responsibility to the authenticator, but LdapOpcUaUserAuthenticator never took it up; and OtOpcUa's LdapOptions never projected the library's existing ConnectionTimeoutMs, so the 10 s library default is both invisible and unconfigurable. Nothing bounds concurrent in-flight binds or fails fast during a sustained outage.

Proposed design — hard wall-clock bound + bounded concurrency + outage circuit, all inside LdapOpcUaUserAuthenticator; project the library timeout. The SDK callback is irreducibly sync (verified), so the fix is entirely at the OtOpcUa authenticator boundary:

1. Project the library timeout — add ConnectionTimeoutMs (default 10000, matching the library default so behavior is unchanged unless configured) to OtOpcUa's LdapOptions and project it in ToLibraryOptions(). This bounds the socket connect and each search op inside the library.
2. Hard whole-flow timeout at the boundary — AuthenticateUserNameAsync wraps its existing body (bind and role-map — one budget for everything between SDK-callback entry and result) in core.WaitAsync(TimeSpan.FromMilliseconds(AuthTimeoutMs), ct); TimeoutException ⇒ Deny("Authentication timed out") (fail-closed) + Warning log. Default AuthTimeoutMs = 15000 — deliberately > ConnectionTimeoutMs so a healthy-but-slow full flow (connect + 2 binds + search, each individually under 10 s) isn't cut off by the boundary, while an outage (connect stall) is bounded by the library timeout first and the boundary is the backstop. Note the abandoned inner task runs to completion in the background — the library is documented never-throw, so no unobserved exceptions; the point is releasing the SDK thread (and m_eventLock).
3. Bounded in-flight concurrency — a SemaphoreSlim(MaxConcurrentAuthentications) acquired before starting the core task and released by a continuation when the core task completes (not in the caller's finally) — so abandoned/orphaned binds count against the cap and an outage can never accumulate unbounded orphan tasks. A caller that cannot acquire a slot within its own budget denies with "Authentication backend busy" (does not count toward the circuit — local backpressure is not directory evidence). With the SDK's event-lock serialization only one SDK thread is ever in the callback at a time, but the cap still bounds orphan accumulation across successive timed-out calls.
4. Directory-outage circuit (fail-fast) — after OutageFailureThreshold (default 3) consecutive system-side failures (boundary timeout, IsSystemFailure result, or unexpected throw), the authenticator denies instantly ("Authentication backend unavailable") for OutageCooldownSeconds (default 15) without touching LDAP; after cooldown the next attempt probes (half-open — the counter stays at threshold so one more system failure re-opens immediately). Any success or user-side deny resets the counter. Distinguishing system-side from user-side needs a structured signal: add bool IsSystemFailure { get; init; } to the app LdapAuthResult (Security/Ldap/LdapAuthResult.cs — additive init property, no call-site churn), set by OtOpcUaLdapAuthService.Adapt for LdapAuthFailure.ServiceAccountBindFailed (the library's directory-unreachable bucket — its enum has no dedicated DirectoryUnavailable; documented limitation, LdapAuthService.cs:94-96).
5. Negative cache — REJECTED (resolves the round-1 "consider a short negative cache" and the security caution): a per-credential deny cache requires holding password digests in memory, creates a window where a just-corrected password is still rejected, and adds a cache-timing side channel — for near-zero benefit, because a bad-credential bind is fast when the directory is up; the expensive stall only exists when the directory is down, which the credential-blind outage circuit covers strictly better.

Alternatives considered:

• Make the shared library cooperatively cancellable / pooled — cross-repo change to ZB.MOM.WW.Auth (Novell calls are sync-blocking; pooling changes the security posture of bind reuse). Track as a shared-lib follow-up; not blocking — the boundary bound is sufficient here.
• A ResilientOpcUaUserAuthenticator decorator over IOpcUaUserAuthenticator — cleaner layering on paper, but it would need the system-vs-user failure signal widened onto OpcUaUserAuthResult (an OpcUaServer-project contract) and a second options plumbing path; the inline fix keeps the resilience next to the only LDAP-backed implementation and the signal on the app-owned LdapAuthResult. Rejected for scope.
• Pass a real timeout-derived token from HandleImpersonation — the library observes the token at entry only (verified), so it cannot shorten anything the boundary WaitAsync doesn't already bound. Not done; HandleImpersonation keeps CancellationToken.None and gains a doc comment stating the verified SDK contract (sync callback under m_eventLock) and where the bound lives.
• Fully async activation — impossible; no async impersonation callback exists in 1.5.378 (verified).

Implementation steps (task-level detail in the breakdown):

1. src/Server/ZB.MOM.WW.OtOpcUa.Security/Ldap/LdapOptions.cs — add five keys (defaults): ConnectionTimeoutMs = 10000 (projected into ToLibraryOptions()), AuthTimeoutMs = 15000, MaxConcurrentAuthentications = 8, OutageFailureThreshold = 3 (0 disables the circuit), OutageCooldownSeconds = 15.
2. src/Server/ZB.MOM.WW.OtOpcUa.Security/Ldap/LdapAuthResult.cs — add public bool IsSystemFailure { get; init; }.
3. src/Server/ZB.MOM.WW.OtOpcUa.Security/Ldap/OtOpcUaLdapAuthService.cs — Adapt sets IsSystemFailure = result.Failure is LdapAuthFailure.ServiceAccountBindFailed on the failure path.
4. src/Server/ZB.MOM.WW.OtOpcUa.Host/OpcUa/LdapOpcUaUserAuthenticator.cs — take IOptions<LdapOptions>; move the existing body to a private AuthenticateCoreAsync; add the circuit check → semaphore acquire (release-on-core-completion continuation) → WaitAsync timeout → circuit bookkeeping, per the design. DI registration (Host/Program.cs:249) is unchanged.
5. src/Server/ZB.MOM.WW.OtOpcUa.Host/Configuration/LdapOptionsValidator.cs — when Enabled && !DevStubMode: ConnectionTimeoutMs > 0, AuthTimeoutMs > 0, MaxConcurrentAuthentications > 0, and OutageCooldownSeconds > 0 when OutageFailureThreshold > 0.
6. src/Server/ZB.MOM.WW.OtOpcUa.OpcUaServer/OpcUaApplicationHost.cs — doc-comment only on HandleImpersonation: record the verified SDK contract (sync void handler raised under SessionManager.m_eventLock in 1.5.378 — activations serialize) and that the wall-clock bound is the authenticator's responsibility (now actually enforced).
7. docs/security.md — document the five new Security:Ldap keys + fail-fast semantics.

Tests.

• Stall-repro (RED first, no real LDAP): Host.IntegrationTests — new LdapAuthResilienceTests beside the existing suite, reusing its FakeLdap : ILdapAuthService pattern with a delaying handler. Authenticate_slow_backend_denies_within_AuthTimeout (fake delays 5 s, AuthTimeoutMs = 200 → assert Deny + elapsed < ~1.5 s) — fails on current code (returns Allow after the full 5 s).
• Unit: fast success unaffected; timeout denial is fail-closed + logged; circuit opens after N consecutive system failures and denies without invoking the fake (call-count pinned); user-side deny does not trip the circuit; half-open probe after cooldown; success resets; semaphore denies "busy" when saturated; OtOpcUaLdapAuthServiceTests — Adapt maps ServiceAccountBindFailed → IsSystemFailure = true, BadCredentials → false; LdapOptionsBindingTests / LdapOptionsValidatorTests extended for the new keys. All six existing LdapOpcUaUserAuthenticatorTests stay green (ctor gains options — update construction).
• Outage-sim integration (offline-safe): real OtOpcUaLdapAuthService + real library pointed at an unroutable host (10.255.255.1:3893 — TCP SYN blackhole, the exact shape of "stop the GLAuth container"), small ConnectionTimeoutMs/AuthTimeoutMs, driven through the real HandleImpersonation (it is internal static — invokable without booting the SDK, precedent OpcUaApplicationHostImpersonationTests): assert bounded fail + BadIdentityTokenRejected.
• Live/manual (post-merge, not gating): docker-dev rig — point Security:Ldap:Server at an unreachable host, drive Client.CLI connect -u opc.tcp://localhost:4840 with a UserName token (e.g. the multi-role GLAuth user), confirm fast bounded denial and that a burst of activations doesn't serialize for 10 s each; then restore 10.100.0.35:3893 and confirm normal login.

Effort: M. Risk/blast-radius: Low-Medium. All defaults chosen so behavior is unchanged on a healthy directory (ConnectionTimeoutMs 10 s = current library default; AuthTimeoutMs 15 s only cuts flows that today stall past 15 s; the circuit requires 3 consecutive system failures). Fail-closed posture preserved end-to-end (every new path denies). The AdminUI login path (OtOpcUaLdapAuthService direct) is untouched except the additive IsSystemFailure flag.

---

S3 — HIGH — HistoryRead block-bridges the gateway per node, sequentially, on SDK request threads

Restatement: All four HistoryRead arms serve nodes sequentially, each node a .GetAwaiter().GetResult() gateway call with CancellationToken.None, bounded only by the 30 s per-call gateway CallTimeout. One request naming N historized nodes against a slow historian holds one SDK request thread up to N × 30 s against MaxRequestThreadCount = 100 / MaxQueuedRequestCount = 200; a handful of misbehaving history clients can exhaust the pool and degrade every OPC UA service.

Verification: Confirmed — bridge sites OtOpcUaNodeManager.cs:1911, 2059, 2171, 2203; sequential foreach in all four arms; thread-safety audit for parallelization passed (see Verification summary). GatewayHistorianDataSource is already fully async underneath with a per-call deadline — the bridging pattern is the problem, not the data source.

Root cause: Per-node reads are serialized with no per-request deadline and no server-side cap on concurrent HistoryRead work; the block-bridge is structural (the CustomNodeManager2.HistoryRead* override surface is synchronous void — verified), but nothing says it must be per-node and unbounded.

Proposed design — bounded per-node fan-out + per-request deadline + process-wide batch limiter, one block-bridge per arm. Three composable bounds, each configurable under ServerHistorian:

1. Bounded per-node parallelism within a batch. Async-ify the per-node serving internals (ServeNode → ServeNodeAsync, ServeRawPaged → ServeRawPagedAsync, the Events body → ServeEventsAsync) — each stays fully self-contained try/catch (per-node error isolation unchanged). Each arm collects one Func<Task> per handle and runs them through a shared RunBounded(work, degree) helper: a SemaphoreSlim(degree)-gated Task.WhenAll, block-bridged once at the arm boundary (Task.WhenAll(...).GetAwaiter().GetResult() — safe: bodies never throw). Worst case drops from N × 30 s to ~⌈N/P⌉ × 30 s. Degree: HistoryReadBatchParallelism, default 4 (bounds the gateway-load multiplication a single client can command — see Risk).
2. Per-request deadline. Each arm creates using var cts = new CancellationTokenSource(HistoryReadDeadline) (default 60 s) and threads cts.Token into every data-source call (replacing CancellationToken.None at all four sites). Per-node handlers gain a catch (OperationCanceledException) arm mapping to BadTimeout for that node (before the generic catch, which keeps mapping other failures to BadHistoryOperationUnsupported as today). A single request can no longer hold a thread longer than the deadline regardless of node count.
3. Process-wide concurrent-batch limiter. A node-manager SemaphoreSlim(MaxConcurrentHistoryReadBatches) (default 16, created lazily on first HistoryRead so the post-boot property set is respected — same benign wiring race as the existing MaxTieClusterOverfetch property) gates each arm's body. Entry waits up to min(HistoryReadDeadline, 5 s); on expiry every handle in the batch gets BadTooManyOperations (fail-fast under flood, brief waits under mild contention — resolves the round-1 "wait or fail" open choice with both, bounded). 16 batches × degree 4 also caps total in-flight gateway calls at 64.

Alternatives considered:

• Fully async HistoryRead — the SDK override surface is synchronous void (verified); structural. Rejected; parallelize within the sync boundary.
• Parallel.ForEachAsync for the fan-out — throws OperationCanceledException at the scheduling level on token cancellation, which would bypass per-node error isolation and throw out of the arm. The SemaphoreSlim + Task.WhenAll shape keeps "bodies never throw" airtight. Rejected.
• Rejecting multi-node history requests — breaks legitimate clients. Rejected (round-1 concurrence).
• A channel/queue + dedicated reader pool — heavier machinery for the same bound; the semaphore composition is sufficient and keeps the diff reviewable. Rejected.
• Per-node continuation-point-aware sub-paging changes — none needed: paging state (ServeRawPaged/continuation store) is per-handle and thread-safe (audit above); TimestampsToReturn handling is untouched (the arms already receive it and the projection helpers are pure).

Implementation steps (task-level detail in the breakdown):

1. src/Server/ZB.MOM.WW.OtOpcUa.Runtime/Historian/ServerHistorianOptions.cs — add HistoryReadBatchParallelism = 4, MaxConcurrentHistoryReadBatches = 16, HistoryReadDeadline = TimeSpan.FromSeconds(60); Validate() warns on non-positive values (Enabled-gated, same pattern as MaxTieClusterOverfetch).
2. src/Server/ZB.MOM.WW.OtOpcUa.OpcUaServer/OtOpcUaNodeManager.cs — three settable properties mirroring the MaxTieClusterOverfetch precedent (HistoryReadBatchParallelism, MaxConcurrentHistoryReadBatches, HistoryReadDeadline); async-ified per-node internals with ct parameters; RunBounded helper; lazily-created batch-limiter gate; the four arms rewritten to limiter → deadline CTS → work-item collection → bounded fan-out → single bridge.
3. src/Server/ZB.MOM.WW.OtOpcUa.Host/OpcUa/OtOpcUaServerHostedService.cs — set the three node-manager properties beside the existing MaxTieClusterOverfetch assignment (StartAsync).
4. docs/Historian.md + the CLAUDE.md ServerHistorian key table — document the three new keys.

Tests.

• Stall-repro (RED first, no real gateway): OpcUaServer.Tests — new NodeManagerHistoryReadConcurrencyTests reusing the NodeManagerHistoryReadTests boot harness with a delaying fake IHistorianDataSource that tracks max observed concurrent reads (interlocked). Raw_batch_is_served_with_bounded_parallelism: 8 historized nodes, 150 ms delay per read → assert maxConcurrent >= 2 (deterministic — fails on current sequential code where it is pinned at 1)
  • a loose wall-clock secondary bound (< ~⌈8/4⌉ × 150 ms + slack).
• Unit: per-node isolation regression under parallelism (one node's fake throws → that node Bad, others Good — the existing guarantee, re-proven concurrent); parallelism also bounded above (maxConcurrent <= degree); deadline — a fake whose task only completes on token cancellation → every node BadTimeout within deadline + slack, call returns (fails-by-hanging is prevented by wrapping the invoke in Task.Run(...).WaitAsync(10 s)); limiter — MaxConcurrentHistoryReadBatches=1, first batch parked in the fake, second concurrent HistoryRead (small HistoryReadDeadline so the wait budget is short) → all its handles BadTooManyOperations; existing NodeManagerHistoryReadTests + NodeManagerHistoryReadPagingTests (incl. tie-cluster paths) stay green — the paging suite is the continuation-point-correctness regression net; AddServerHistorianTests / options tests extended for the new keys.
• Integration/live (post-merge, not gating): the env-gated Category=LiveIntegration gateway suite (VPN + real HistorianGateway) re-run; optionally a manual soak — one client issuing a 50-node Raw HistoryRead against a slow historian while a second client browses/subscribes, confirming the second client stays responsive (the pool-exhaustion scenario).

Effort: M. Risk/blast-radius: Medium. Parallelizing multiplies peak gateway load by up to HistoryReadBatchParallelism per request (default 4, total in-flight capped at 64 by the limiter — the knobs exist precisely to tune this against a weak historian). Result/error index writes verified disjoint; all touched shared state verified concurrent-safe. Client-visible behavior changes only in failure shapes (BadTimeout on deadline, BadTooManyOperations under flood — both standard, spec-conformant codes) and latency (better). Continuation-point semantics untouched.

---

Task breakdown

Branch: fix/archreview-r2-08-ldap-historyread-async off master@f6eaa267. Tasks are ≤5 min each; TDD-ordered (each RED repro precedes its implementation). Test commands run from the repo root; all automated tests are offline-safe (no Docker, no VPN, no real LDAP/gateway).

---

R2-08-T1 — S2 stall-repro test (RED — must fail on current code)

Classification: test-first repro (deterministic unit, fake ILdapAuthService with a delay) Estimated implement time: 5 min Parallelizable with: T7, T8 (different files/subsystem). Files: tests/Server/ZB.MOM.WW.OtOpcUa.Host.IntegrationTests/LdapAuthResilienceTests.cs (new)

1. New test class beside LdapOpcUaUserAuthenticatorTests, cloning its FakeLdap/FakeMapper pattern but with an async handler seam: DelayingFakeLdap(TimeSpan delay, LdapAuthResult then) (await Task.Delay(delay, ct) before returning) and an Interlocked call counter.
2. Add Authenticate_slow_backend_denies_within_AuthTimeout: construct LdapOpcUaUserAuthenticator exactly as the existing tests do (this is RED partly because there is no way to pass a timeout yet — write the test against the intended ctor shape new LdapOpcUaUserAuthenticator(ldap, scopeFactory, Options.Create(new LdapOptions { AuthTimeoutMs = 200 }), logger); until T4 lands the file won't compile — acceptable for a same-branch RED, or temporarily pin with the current ctor + assert elapsed, which fails by timing). Fake delays 5 s; assert result.Success == false, result.Error contains "timed out", and a Stopwatch elapsed < 1.5 s. Wrap the await in .WaitAsync(TimeSpan.FromSeconds(10)) so a regression fails rather than hangs.
3. dotnet test tests/Server/ZB.MOM.WW.OtOpcUa.Host.IntegrationTests --filter "FullyQualifiedName~LdapAuthResilienceTests.Authenticate_slow_backend_denies_within_AuthTimeout" → expected: FAIL (compile-fail against intended ctor, or Allow-after-5s timing fail).

Commit: test(host): red repro — LDAP authenticator has no wall-clock bound on session activation (03/S2)

---

R2-08-T2 — LdapOptions keys + library projection + validator

Classification: implementation (options + validation, additive) Estimated implement time: 5 min Parallelizable with: T3, T7, T8. Files: src/Server/ZB.MOM.WW.OtOpcUa.Security/Ldap/LdapOptions.cs, src/Server/ZB.MOM.WW.OtOpcUa.Host/Configuration/LdapOptionsValidator.cs, tests/Server/ZB.MOM.WW.OtOpcUa.Host.IntegrationTests/LdapOptionsBindingTests.cs, tests/Server/ZB.MOM.WW.OtOpcUa.Host.IntegrationTests/LdapOptionsValidatorTests.cs

1. RED: extend LdapOptionsBindingTests — a config section carrying the five new keys binds them; defaults hold when absent; ToLibraryOptions().ConnectionTimeoutMs mirrors the app value. Extend LdapOptionsValidatorTests — non-positive ConnectionTimeoutMs/AuthTimeoutMs/ MaxConcurrentAuthentications fail validation when Enabled && !DevStubMode; OutageFailureThreshold = 3 with OutageCooldownSeconds = 0 fails; OutageFailureThreshold = 0 with any cooldown passes; disabled/DevStub configs stay exempt. dotnet test tests/Server/ZB.MOM.WW.OtOpcUa.Host.IntegrationTests --filter "FullyQualifiedName~LdapOptionsBindingTests|FullyQualifiedName~LdapOptionsValidatorTests" → expected: FAIL (properties don't exist).
2. GREEN: add ConnectionTimeoutMs = 10000, AuthTimeoutMs = 15000, MaxConcurrentAuthentications = 8, OutageFailureThreshold = 3, OutageCooldownSeconds = 15 (XML-doc each: what it bounds, why the default, 0 disables the circuit); project ConnectionTimeoutMs in ToLibraryOptions(); add the validator rules. Re-run the filter → PASS.

Commit: feat(security): LDAP auth timeout/concurrency/outage-circuit options; project ConnectionTimeoutMs into the shared library (03/S2)

---

R2-08-T3 — LdapAuthResult.IsSystemFailure seam

Classification: implementation (additive contract field + adapter mapping) Estimated implement time: 5 min Parallelizable with: T2, T7, T8. Files: src/Server/ZB.MOM.WW.OtOpcUa.Security/Ldap/LdapAuthResult.cs, src/Server/ZB.MOM.WW.OtOpcUa.Security/Ldap/OtOpcUaLdapAuthService.cs, tests/Server/ZB.MOM.WW.OtOpcUa.Security.Tests/OtOpcUaLdapAuthServiceTests.cs

1. RED: extend OtOpcUaLdapAuthServiceTests — library ServiceAccountBindFailed adapts to IsSystemFailure == true; BadCredentials/UserNotFound/success adapt to false. dotnet test tests/Server/ZB.MOM.WW.OtOpcUa.Security.Tests --filter "FullyQualifiedName~OtOpcUaLdapAuthServiceTests" → expected: FAIL.
2. GREEN: add public bool IsSystemFailure { get; init; } (XML-doc: "system-side directory failure — unreachable/misconfigured — as opposed to a user-credential deny; feeds the OPC UA authenticator's outage circuit"); set it in Adapt's failure branch for ServiceAccountBindFailed. Re-run → PASS (whole class — existing adapt tests must stay green).

Commit: feat(security): LdapAuthResult.IsSystemFailure distinguishes directory outage from credential deny (03/S2)

---

R2-08-T4 — Authenticator wall-clock bound + concurrency cap (turns T1 green)

Classification: implementation (the core S2 fix) Estimated implement time: 5 min Parallelizable with: T7-T13 (S3 files). Blocked by T1, T2. Files: src/Server/ZB.MOM.WW.OtOpcUa.Host/OpcUa/LdapOpcUaUserAuthenticator.cs, tests/Server/ZB.MOM.WW.OtOpcUa.Host.IntegrationTests/LdapAuthResilienceTests.cs, tests/Server/ZB.MOM.WW.OtOpcUa.Host.IntegrationTests/LdapOpcUaUserAuthenticatorTests.cs

1. Add IOptions<LdapOptions> to the ctor; move the existing method body to private async Task<OpcUaUserAuthResult> AuthenticateCoreAsync(...) unchanged; new outer body: semaphore WaitAsync(AuthTimeoutMs, ct) → deny "Authentication backend busy" on false; start core = AuthenticateCoreAsync(...) with a core.ContinueWith(_ => _gate.Release(), TaskScheduler.Default) release-on-completion; await core.WaitAsync(TimeSpan.FromMilliseconds(AuthTimeoutMs), ct) in try/catch — TimeoutException ⇒ Warning log + deny "Authentication timed out". Update the class XML-doc (why: SDK sync callback under m_eventLock).
2. Update the six existing LdapOpcUaUserAuthenticatorTests constructions to pass Options.Create(new LdapOptions()).
3. Add Authenticate_fast_success_unaffected and Authenticate_saturated_concurrency_denies_busy (MaxConcurrentAuthentications = 1, first call parked in the fake, second denies "busy" within bound) to LdapAuthResilienceTests.
4. dotnet test tests/Server/ZB.MOM.WW.OtOpcUa.Host.IntegrationTests --filter "FullyQualifiedName~LdapAuthResilienceTests|FullyQualifiedName~LdapOpcUaUserAuthenticatorTests" → expected: PASS (T1's repro now green; all six pre-existing tests green).

Commit: feat(host): hard wall-clock bound + bounded concurrency on OPC UA LDAP authentication (03/S2)

---

R2-08-T5 — Directory-outage circuit (fail-fast during sustained outage)

Classification: implementation (resilience state machine) Estimated implement time: 5 min Parallelizable with: T7-T13. Blocked by T3, T4. Files: src/Server/ZB.MOM.WW.OtOpcUa.Host/OpcUa/LdapOpcUaUserAuthenticator.cs, tests/Server/ZB.MOM.WW.OtOpcUa.Host.IntegrationTests/LdapAuthResilienceTests.cs

1. RED: add circuit tests — three consecutive timeouts (or IsSystemFailure denies) open the circuit: the 4th call denies "Authentication backend unavailable" without invoking the fake (call counter pinned at 3) and returns in ~0 ms; a user-side deny (Invalid username or password, IsSystemFailure=false) between failures resets the streak; after OutageCooldownSeconds (use 1 s in test) the next call probes the fake (half-open) and a success closes the circuit; OutageFailureThreshold = 0 never opens. dotnet test tests/Server/ZB.MOM.WW.OtOpcUa.Host.IntegrationTests --filter "FullyQualifiedName~LdapAuthResilienceTests" → expected: FAIL (new tests only).
2. GREEN: small lock-guarded state (_consecutiveSystemFailures, _circuitOpenUntil via Environment.TickCount64); entry check denies while open (Debug log; single Warning when opening, Information when closing); system failure = boundary timeout ∨ result.IsSystemFailure ∨ unexpected throw from the core; success or user-side deny resets. Re-run → PASS.

Commit: feat(host): LDAP directory-outage circuit — fail-fast denial during sustained outage (03/S2)

---

R2-08-T6 — Outage-sim integration test + HandleImpersonation contract doc

Classification: integration test (offline-safe) + doc Estimated implement time: 5 min Parallelizable with: T7-T13. Blocked by T4. Files: tests/Server/ZB.MOM.WW.OtOpcUa.Host.IntegrationTests/LdapAuthResilienceTests.cs, src/Server/ZB.MOM.WW.OtOpcUa.OpcUaServer/OpcUaApplicationHost.cs, docs/security.md

1. Add Activation_with_unreachable_directory_fails_within_bound: real OtOpcUaLdapAuthService (real ZB.MOM.WW.Auth library, no fake) with Server = "10.255.255.1", Port = 3893, Transport = None, AllowInsecure = true, ConnectionTimeoutMs = 500, AuthTimeoutMs = 1000; wrap in LdapOpcUaUserAuthenticator; drive the real internal static OpcUaApplicationHost.HandleImpersonation with a UserNameIdentityToken (precedent: OpcUaApplicationHostImpersonationTests); assert args.IdentityValidationError is BadIdentityTokenRejected and total elapsed < ~3 s. (This is the automated stand-in for "stop the GLAuth container" — a TCP blackhole is the same outage shape; the bitnami/openldap :3894 harness is not needed for the outage direction.)
2. Rewrite HandleImpersonation's doc comment: verified 1.5.378 contract (synchronous void handler raised under SessionManager.m_eventLock — a stalled handler serializes all activations), why the block-bridge is irreducible, and where the wall-clock bound + circuit live.
3. Document the new Security:Ldap keys + semantics in docs/security.md.
4. dotnet test tests/Server/ZB.MOM.WW.OtOpcUa.Host.IntegrationTests --filter "FullyQualifiedName~LdapAuthResilienceTests" → expected: PASS.

Commit: test(host): unreachable-directory activation fails bounded; document the SDK impersonation threading contract (03/S2)

---

R2-08-T7 — S3 sequential-serving repro test (RED — must fail on current code)

Classification: test-first repro (deterministic — concurrency counter, not raw timing) Estimated implement time: 5 min Parallelizable with: T1-T6, T8. Files: tests/Server/ZB.MOM.WW.OtOpcUa.OpcUaServer.Tests/NodeManagerHistoryReadConcurrencyTests.cs (new)

1. New class reusing the NodeManagerHistoryReadTests boot pattern (BootAsync + EnsureVariable + public HistoryRead invoke). Fake IHistorianDataSource whose ReadRawAsync does Interlocked.Increment a live-counter, records the max, await Task.Delay(150ms, ct), decrements, returns one sample.
2. Add Raw_batch_is_served_with_bounded_parallelism: materialise 8 historized variables; issue one Raw HistoryRead naming all 8; assert fake.MaxObservedConcurrency >= 2 (primary — deterministic) and elapsed < ~1.0 s (secondary, loose). Wrap the invoke in Task.Run(...).WaitAsync(15 s).
3. dotnet test tests/Server/ZB.MOM.WW.OtOpcUa.OpcUaServer.Tests --filter "FullyQualifiedName~NodeManagerHistoryReadConcurrencyTests.Raw_batch_is_served_with_bounded_parallelism" → expected: FAIL (MaxObservedConcurrency == 1 on the sequential foreach).

Commit: test(opcuaserver): red repro — HistoryRead serves a batch strictly sequentially (03/S3)

---

R2-08-T8 — ServerHistorian keys + hosted-service wiring

Classification: implementation (options + wiring, additive) Estimated implement time: 5 min Parallelizable with: T1-T6, T7. Files: src/Server/ZB.MOM.WW.OtOpcUa.Runtime/Historian/ServerHistorianOptions.cs, src/Server/ZB.MOM.WW.OtOpcUa.OpcUaServer/OtOpcUaNodeManager.cs (properties only), src/Server/ZB.MOM.WW.OtOpcUa.Host/OpcUa/OtOpcUaServerHostedService.cs, tests/Server/ZB.MOM.WW.OtOpcUa.Runtime.Tests/Historian/AddServerHistorianTests.cs (or the options test file beside it)

1. RED: options tests — defaults (4 / 16 / 60 s); Validate() warns on non-positive parallelism/batches/deadline when Enabled (and stays silent when disabled). dotnet test tests/Server/ZB.MOM.WW.OtOpcUa.Runtime.Tests --filter "FullyQualifiedName~ServerHistorianOptions|FullyQualifiedName~AddServerHistorianTests" → expected: FAIL.
2. GREEN: add HistoryReadBatchParallelism = 4, MaxConcurrentHistoryReadBatches = 16, HistoryReadDeadline = TimeSpan.FromSeconds(60) (+ XML-doc each: what it bounds, the ⌈N/P⌉ × CallTimeout worst case, 16 × 4 = 64 in-flight cap) + Validate() warnings; add the three settable node-manager properties (defaults matching — MaxTieClusterOverfetch precedent, unused until T9-T13); assign all three in OtOpcUaServerHostedService.StartAsync beside the existing MaxTieClusterOverfetch line. Re-run → PASS.

Commit: feat(runtime): ServerHistorian HistoryRead parallelism/limiter/deadline knobs + node-manager wiring (03/S3)

---

R2-08-T9 — Async-ify ServeNode + Processed/AtTime arms (still sequential)

Classification: implementation (mechanical async refactor — behavior-neutral) Estimated implement time: 5 min Parallelizable with: T1-T6. Blocked by T7 (repro must be red first), T8 (ct/property names). Files: src/Server/ZB.MOM.WW.OtOpcUa.OpcUaServer/OtOpcUaNodeManager.cs

1. ServeNode → private async Task ServeNodeAsync(handle, results, errors, read, CancellationToken ct); the read callback gains the token (Func<IHistorianDataSource, string, CancellationToken, Task<HistorianRead>>); await instead of bridging; add catch (OperationCanceledException) → errors/results = BadTimeout ahead of the generic catch. Processed/AtTime arms: pass ct into the source calls (:1841/:1865 today CancellationToken.None), collect the per-handle tasks but for now run them sequentially with a single arm-boundary bridge (foreach … ServeNodeAsync(...).GetAwaiter().GetResult() is acceptable interim — the fan-out lands in T11). Thread a ct parameter down from the arm (pass CancellationToken.None at the arm until T12).
2. dotnet test tests/Server/ZB.MOM.WW.OtOpcUa.OpcUaServer.Tests --filter "FullyQualifiedName~NodeManagerHistoryReadTests" → expected: PASS (behavior-neutral; T7's repro stays red).

Commit: refactor(opcuaserver): async-ify ServeNode + Processed/AtTime HistoryRead internals, thread cancellation (03/S3)

---

R2-08-T10 — Async-ify ServeRawPaged + Events body

Classification: implementation (mechanical async refactor — behavior-neutral) Estimated implement time: 5 min Parallelizable with: T1-T6. Blocked by T9 (shared BadTimeout pattern/signature conventions). Files: src/Server/ZB.MOM.WW.OtOpcUa.OpcUaServer/OtOpcUaNodeManager.cs

1. ServeRawPaged → ServeRawPagedAsync(..., CancellationToken ct): await the page read (:2171) and the tie-cluster over-fetch (:2203) with ct; add the OCE → BadTimeout arm; continuation store calls unchanged (thread-safe, audited).
2. Extract the HistoryReadEvents per-handle body into ServeEventsAsync(handle, sourceName, details, selectClauses, results, errors, ct) — await the :1911 read with ct; OCE → BadTimeout; existing catch preserved. Both arms keep the interim sequential arm-boundary bridge.
3. dotnet test tests/Server/ZB.MOM.WW.OtOpcUa.OpcUaServer.Tests --filter "FullyQualifiedName~NodeManagerHistoryReadTests|FullyQualifiedName~NodeManagerHistoryReadPagingTests" → expected: PASS (paging + tie-cluster suites are the regression net).

Commit: refactor(opcuaserver): async-ify ServeRawPaged + HistoryReadEvents internals (03/S3)

---

R2-08-T11 — Bounded per-node fan-out in all four arms (turns T7 green)

Classification: implementation (the core S3 parallelism fix) Estimated implement time: 5 min Parallelizable with: T1-T6. Blocked by T9, T10. Files: src/Server/ZB.MOM.WW.OtOpcUa.OpcUaServer/OtOpcUaNodeManager.cs, tests/Server/ZB.MOM.WW.OtOpcUa.OpcUaServer.Tests/NodeManagerHistoryReadConcurrencyTests.cs

1. Add private void RunBounded(IReadOnlyList<Func<Task>> work, int degree): degree <= 1 or single item ⇒ run sequentially; else SemaphoreSlim(degree)-gated Task.WhenAll, bridged once (GetAwaiter().GetResult(); safe — every body is self-contained try/catch and never throws; document that invariant on the helper).
2. Rewrite the four arms: pre-filter the cheap synchronous rejections inline (IsReadModified, non-notifier Events handles), collect one Func<Task> per remaining handle, call RunBounded(work, HistoryReadBatchParallelism).
3. Extend the concurrency test class: Parallelism_is_bounded_above (12 nodes, property set to 3 → MaxObservedConcurrency <= 3) and One_node_failure_does_not_poison_a_parallel_batch (one handle's fake throws → that node BadHistoryOperationUnsupported, the other 7 Good).
4. dotnet test tests/Server/ZB.MOM.WW.OtOpcUa.OpcUaServer.Tests --filter "FullyQualifiedName~NodeManagerHistoryRead" → expected: PASS (T7 repro green; paging + base suites green).

Commit: feat(opcuaserver): bounded per-node parallel HistoryRead within a batch (03/S3)

---

R2-08-T12 — Per-request deadline

Classification: implementation (deadline CTS + BadTimeout surfacing) Estimated implement time: 5 min Parallelizable with: T1-T6. Blocked by T11. Files: src/Server/ZB.MOM.WW.OtOpcUa.OpcUaServer/OtOpcUaNodeManager.cs, tests/Server/ZB.MOM.WW.OtOpcUa.OpcUaServer.Tests/NodeManagerHistoryReadConcurrencyTests.cs

1. RED: Deadline_bounds_a_hung_backend — fake whose reads complete only on token cancellation (await Task.Delay(Timeout.Infinite, ct)); HistoryReadDeadline = 500 ms; 4 nodes → every node BadTimeout, call returns within ~2 s (invoke wrapped in .WaitAsync(10 s)). dotnet test tests/Server/ZB.MOM.WW.OtOpcUa.OpcUaServer.Tests --filter "FullyQualifiedName~NodeManagerHistoryReadConcurrencyTests.Deadline_bounds_a_hung_backend" → expected: FAIL (hangs 30 s+ / WaitAsync trips — nothing cancels today).
2. GREEN: each arm creates using var cts = HistoryReadDeadline > TimeSpan.Zero ? new CancellationTokenSource(HistoryReadDeadline) : null and passes cts?.Token ?? CancellationToken.None down (defensive: non-positive = unbounded, matching the options warning). Re-run → PASS.

Commit: feat(opcuaserver): per-request HistoryRead deadline — hung backends surface BadTimeout (03/S3)

---

R2-08-T13 — Process-wide concurrent-batch limiter

Classification: implementation (server-side admission control) Estimated implement time: 5 min Parallelizable with: T1-T6. Blocked by T12. Files: src/Server/ZB.MOM.WW.OtOpcUa.OpcUaServer/OtOpcUaNodeManager.cs, tests/Server/ZB.MOM.WW.OtOpcUa.OpcUaServer.Tests/NodeManagerHistoryReadConcurrencyTests.cs

1. RED: Saturated_limiter_rejects_with_BadTooManyOperations — MaxConcurrentHistoryReadBatches = 1, HistoryReadDeadline = 200 ms; first batch parked in a gated fake; a second concurrent HistoryRead → all its handles BadTooManyOperations, returns within ~1 s; release the gate; a third read succeeds (slot freed). dotnet test tests/Server/ZB.MOM.WW.OtOpcUa.OpcUaServer.Tests --filter "FullyQualifiedName~NodeManagerHistoryReadConcurrencyTests.Saturated_limiter_rejects_with_BadTooManyOperations" → expected: FAIL (no limiter exists).
2. GREEN: lazily-created SemaphoreSlim (double-checked, sized from MaxConcurrentHistoryReadBatches at first use — respects the post-boot property set, same benign race as MaxTieClusterOverfetch); arm entry Wait(min(HistoryReadDeadline, 5 s)) — on false, set BadTooManyOperations on every handle and return; try/finally release. Re-run → PASS.

Commit: feat(opcuaserver): process-wide HistoryRead batch limiter — flood degrades to BadTooManyOperations, not pool exhaustion (03/S3)

---

R2-08-T14 — Docs + full regression sweep

Classification: docs + verification Estimated implement time: 5 min Parallelizable with: —. Blocked by T5, T6, T13. Files: docs/Historian.md, docs/security.md (if residue from T6), CLAUDE.md (ServerHistorian key table), archreview/plans/STATUS.md

1. Add the three ServerHistorian keys to docs/Historian.md + the CLAUDE.md table (defaults + one-line semantics); confirm docs/security.md covers the five Security:Ldap keys.
2. Full touched-suite sweep: dotnet test tests/Server/ZB.MOM.WW.OtOpcUa.Host.IntegrationTests tests/Server/ZB.MOM.WW.OtOpcUa.Security.Tests tests/Server/ZB.MOM.WW.OtOpcUa.OpcUaServer.Tests tests/Server/ZB.MOM.WW.OtOpcUa.Runtime.Tests (or per-project) → all green; then whole-solution dotnet test ZB.MOM.WW.OtOpcUa.slnx (the CI fail-on-skip gate applies).
3. Record R2-08 outcome in archreview/plans/STATUS.md (branch, commits, test counts).

Commit: docs: ServerHistorian HistoryRead knobs + Security:Ldap resilience keys; R2-08 status (03/S2, 03/S3)

---

Post-merge (not gating, operator-run)

• S2 live check: docker-dev rig — unreachable Security:Ldap:Server → Client.CLI UserName connect denies fast; restore shared GLAuth 10.100.0.35:3893 → normal login. Optionally stop/start the GLAuth container on the docker host for a true outage/recovery cycle (circuit opens, then closes).
• S3 live check: env-gated Category=LiveIntegration gateway suite on the VPN; a 50-node Raw HistoryRead soak against the real gateway while a second client stays interactive.

Overall effort

M (≈1.5-2 dev-days): 6 tasks S2 + 7 tasks S3 + 1 docs/sweep, each ≤5 min of implementation but with the usual test-run + review cadence. Risk concentrated in T11 (parallel fan-out — mitigated by the thread-safety audit + the paging regression suite) and T4/T5 (auth-path semantics — mitigated by the fail-closed default posture and behavior-neutral defaults).