Merge R2-08 Async LDAP + HistoryRead (arch-review round 2) [PR #435]
v2-ci / build (push) Successful in 4m13s
v2-ci / unit-tests (push) Failing after 8m29s

Findings 03/S2 (LDAP bind no longer block-bridges the SDK session-activation
lock — AuthTimeout/concurrency-semaphore/directory-outage circuit, fail-closed),
03/S3 (HistoryRead async-ified + bounded per-node fan-out + deadline CTS + batch
limiter). All 14 tasks; full-sweep + live legs deferred. Clean merge, build clean.
This commit is contained in:
Joseph Doherty
2026-07-13 12:23:21 -04:00
23 changed files with 1814 additions and 131 deletions
+3
View File
@@ -265,6 +265,9 @@ authorization uses the standard `AccessLevels.HistoryRead` bit set at materializ
| `CaCertificatePath` | `null` | PEM CA file pinning the gateway TLS chain; null/empty uses the OS trust store |
| `CallTimeout` | `00:00:30` | Per-call deadline for each unary gateway read |
| `MaxTieClusterOverfetch` | `65536` | Bounded over-fetch the HistoryRead-Raw paging uses to page within an oversized same-timestamp tie cluster (retained from the prior backend) |
| `HistoryReadBatchParallelism` | `4` | Max nodes served concurrently within one HistoryRead batch (03/S3); `⌈N/P⌉ × CallTimeout` worst case instead of sequential |
| `MaxConcurrentHistoryReadBatches` | `16` | Process-wide concurrent HistoryRead batch cap (03/S3); flood degrades to `BadTooManyOperations`. Total in-flight reads capped at 16 × 4 = 64 |
| `HistoryReadDeadline` | `00:01:00` | Per-request HistoryRead wall-clock deadline (03/S3); a node still in flight surfaces `BadTimeout`. Non-positive = unbounded |
### Alarm-history path (`AlarmHistorian` section)
@@ -674,3 +674,27 @@ Commit: `docs: ServerHistorian HistoryRead knobs + Security:Ldap resilience keys
the usual test-run + review cadence. Risk concentrated in T11 (parallel fan-out — mitigated by the
thread-safety audit + the paging regression suite) and T4/T5 (auth-path semantics — mitigated by the
fail-closed default posture and behavior-neutral defaults).
## Execution deviations (R2-08)
- **T1 RED shape.** Written against the *current* 3-arg ctor with a `Stopwatch` timing assertion (the
plan's explicitly-allowed alternative), not the intended-ctor compile-fail — so every commit compiles.
T4 then re-pointed it at the new `Options.Create(...)` ctor to turn it green.
- **Async parking fakes.** The concurrency-cap repro (`GatedFakeLdap`) and the limiter repro
(`GatedRawFake`) had to park **asynchronously** (await a `TaskCompletionSource`), not block a thread
synchronously — a synchronous block before the first `await` deadlocks the caller (the core task never
yields its `Task`). Documented inline in both fakes.
- **T6 InternalsVisibleTo.** Driving the real `internal static OpcUaApplicationHost.HandleImpersonation`
from `Host.IntegrationTests` required adding an `InternalsVisibleTo` for that test assembly to the
OpcUaServer csproj (the plan called for the internal drive but not the grant). Additive + reversible.
- **T13 fake ignores the deadline token.** `GatedRawFake.ReadRawAsync` deliberately does **not** observe
the per-request deadline `ct`: with a small `HistoryReadDeadline`, the parked first batch would
otherwise cancel its own read and free the limiter permit before the second batch checks (race). By
parking on the release gate only, the first batch holds the permit deterministically, so the second
batch reliably hits the saturated limiter → `BadTooManyOperations`.
- **T14 sweep scope.** Per the executor's memory constraint (heavy `*.IntegrationTests` / Playwright
suites leak ~16 GB and the LDAP integration harness spins an ephemeral openldap on :3894), the
whole-solution `dotnet test` was **not** run. Instead: a full-solution `dotnet build` (clean) + the
impacted **filtered** unit suites (Host.IntegrationTests LDAP-classes only, Security.Tests,
Runtime.Tests options, OpcUaServer.Tests HistoryRead). The whole-solution + live legs are deferred to a
serial heavy pass / VPN run.
@@ -1,20 +1,20 @@
{
"planPath": "archreview/plans/R2-08-ldap-historyread-async-plan.md",
"tasks": [
{ "id": "R2-08-T1", "subject": "S2 stall-repro test (RED): slow LDAP backend must deny within AuthTimeout — fails on current code", "status": "pending", "blockedBy": [] },
{ "id": "R2-08-T2", "subject": "LdapOptions: ConnectionTimeoutMs/AuthTimeoutMs/MaxConcurrentAuthentications/OutageFailureThreshold/OutageCooldownSeconds + ToLibraryOptions projection + validator rules", "status": "pending", "blockedBy": [] },
{ "id": "R2-08-T3", "subject": "LdapAuthResult.IsSystemFailure seam + OtOpcUaLdapAuthService.Adapt mapping (ServiceAccountBindFailed => system-side)", "status": "pending", "blockedBy": [] },
{ "id": "R2-08-T4", "subject": "LdapOpcUaUserAuthenticator: WaitAsync wall-clock bound + SemaphoreSlim concurrency cap with release-on-core-completion (turns T1 green)", "status": "pending", "blockedBy": ["R2-08-T1", "R2-08-T2"] },
{ "id": "R2-08-T5", "subject": "Directory-outage circuit: fail-fast deny after N consecutive system-side failures, half-open after cooldown", "status": "pending", "blockedBy": ["R2-08-T3", "R2-08-T4"] },
{ "id": "R2-08-T6", "subject": "Outage-sim integration test (real library, unroutable host, via internal HandleImpersonation) + SDK m_eventLock contract doc + docs/security.md", "status": "pending", "blockedBy": ["R2-08-T4"] },
{ "id": "R2-08-T7", "subject": "S3 sequential-serving repro test (RED): MaxObservedConcurrency >= 2 on an 8-node Raw batch — fails on current code", "status": "pending", "blockedBy": [] },
{ "id": "R2-08-T8", "subject": "ServerHistorianOptions: HistoryReadBatchParallelism/MaxConcurrentHistoryReadBatches/HistoryReadDeadline + Validate warnings + node-manager properties + hosted-service wiring", "status": "pending", "blockedBy": [] },
{ "id": "R2-08-T9", "subject": "Async-ify ServeNode + Processed/AtTime arms (ct threading, OCE=>BadTimeout, behavior-neutral)", "status": "pending", "blockedBy": ["R2-08-T7", "R2-08-T8"] },
{ "id": "R2-08-T10", "subject": "Async-ify ServeRawPaged (page read + tie-cluster over-fetch) + extract ServeEventsAsync", "status": "pending", "blockedBy": ["R2-08-T9"] },
{ "id": "R2-08-T11", "subject": "RunBounded fan-out in all four HistoryRead arms, single bridge per arm (turns T7 green) + upper-bound + isolation tests", "status": "pending", "blockedBy": ["R2-08-T9", "R2-08-T10"] },
{ "id": "R2-08-T12", "subject": "Per-request HistoryRead deadline CTS — hung backend surfaces BadTimeout per node", "status": "pending", "blockedBy": ["R2-08-T11"] },
{ "id": "R2-08-T13", "subject": "Process-wide concurrent-batch limiter — saturation degrades to BadTooManyOperations (lazy gate, bounded wait)", "status": "pending", "blockedBy": ["R2-08-T12"] },
{ "id": "R2-08-T14", "subject": "Docs (Historian.md + CLAUDE.md ServerHistorian table + security.md) + full touched-suite and whole-solution regression sweep + STATUS.md entry", "status": "pending", "blockedBy": ["R2-08-T5", "R2-08-T6", "R2-08-T13"] }
{ "id": "R2-08-T1", "subject": "S2 stall-repro test (RED): slow LDAP backend must deny within AuthTimeout — fails on current code", "status": "completed", "blockedBy": [] },
{ "id": "R2-08-T2", "subject": "LdapOptions: ConnectionTimeoutMs/AuthTimeoutMs/MaxConcurrentAuthentications/OutageFailureThreshold/OutageCooldownSeconds + ToLibraryOptions projection + validator rules", "status": "completed", "blockedBy": [] },
{ "id": "R2-08-T3", "subject": "LdapAuthResult.IsSystemFailure seam + OtOpcUaLdapAuthService.Adapt mapping (ServiceAccountBindFailed => system-side)", "status": "completed", "blockedBy": [] },
{ "id": "R2-08-T4", "subject": "LdapOpcUaUserAuthenticator: WaitAsync wall-clock bound + SemaphoreSlim concurrency cap with release-on-core-completion (turns T1 green)", "status": "completed", "blockedBy": ["R2-08-T1", "R2-08-T2"] },
{ "id": "R2-08-T5", "subject": "Directory-outage circuit: fail-fast deny after N consecutive system-side failures, half-open after cooldown", "status": "completed", "blockedBy": ["R2-08-T3", "R2-08-T4"] },
{ "id": "R2-08-T6", "subject": "Outage-sim integration test (real library, unroutable host, via internal HandleImpersonation) + SDK m_eventLock contract doc + docs/security.md", "status": "completed", "blockedBy": ["R2-08-T4"] },
{ "id": "R2-08-T7", "subject": "S3 sequential-serving repro test (RED): MaxObservedConcurrency >= 2 on an 8-node Raw batch — fails on current code", "status": "completed", "blockedBy": [] },
{ "id": "R2-08-T8", "subject": "ServerHistorianOptions: HistoryReadBatchParallelism/MaxConcurrentHistoryReadBatches/HistoryReadDeadline + Validate warnings + node-manager properties + hosted-service wiring", "status": "completed", "blockedBy": [] },
{ "id": "R2-08-T9", "subject": "Async-ify ServeNode + Processed/AtTime arms (ct threading, OCE=>BadTimeout, behavior-neutral)", "status": "completed", "blockedBy": ["R2-08-T7", "R2-08-T8"] },
{ "id": "R2-08-T10", "subject": "Async-ify ServeRawPaged (page read + tie-cluster over-fetch) + extract ServeEventsAsync", "status": "completed", "blockedBy": ["R2-08-T9"] },
{ "id": "R2-08-T11", "subject": "RunBounded fan-out in all four HistoryRead arms, single bridge per arm (turns T7 green) + upper-bound + isolation tests", "status": "completed", "blockedBy": ["R2-08-T9", "R2-08-T10"] },
{ "id": "R2-08-T12", "subject": "Per-request HistoryRead deadline CTS — hung backend surfaces BadTimeout per node", "status": "completed", "blockedBy": ["R2-08-T11"] },
{ "id": "R2-08-T13", "subject": "Process-wide concurrent-batch limiter — saturation degrades to BadTooManyOperations (lazy gate, bounded wait)", "status": "completed", "blockedBy": ["R2-08-T12"] },
{ "id": "R2-08-T14", "subject": "Docs (Historian.md + CLAUDE.md ServerHistorian table + security.md) + full touched-suite and whole-solution regression sweep + STATUS.md entry", "status": "completed", "blockedBy": ["R2-08-T5", "R2-08-T6", "R2-08-T13"] }
],
"lastUpdated": "2026-07-12"
}
+1
View File
@@ -176,6 +176,7 @@ what the unit leg cannot; (c) migrate the 3 xunit v2 holdouts once Akka ships an
| **R2-02** | 01/S-6 (High, rank 2), 01/U-6, 01/S-8=03/S12 (rank 4), 04/C-7, 01/S-7=03/S13 | `r2/02-resilience-config-hardening` (off `1676c8f4`) | `4be13429``dd82e740` (17 task commits) | **Code-complete, offline-verified.** **S-6:** parser clamps timeout/retry/breaker overrides (timeout≤0→tier default, breaker==1→2, caps) with diagnostics + belt-and-braces `Build` clamp so pipeline construction can never throw Polly `ValidationException` (new `ResiliencePolicyRanges` in Core.Abstractions); brick-repro + `Build_NeverThrows_ForHostileDirectOptions` + factory-seam wiring proof; AdminUI `Validate()` range warnings. **S-8:** parser forces Write/AlarmAck retry→0 (spec invariant) **and** dispatch flips to `isIdempotent:false` (RED-first on the Runtime wiring test); non-idempotent arm caches the no-retry snapshot once/invoker (01/P-4) + tracker in-flight accounting; `WriteIdempotentAttribute` false claim corrected; retry cells disabled. **U-6:** bulkhead knob deleted end-to-end (migration-free; stored keys ignored) + `Options_properties_are_exactly_the_pipeline_wired_set` inertness guard. **C-7:** `ResilienceFormModel` JsonObject-bag round-trip preserves unknown keys; malformed JSON → `ParseFailed` lock + verbatim `ToJson` + discard button. **S-7:** per-`Create` `Interlocked` options-generation in the pipeline-cache key kills the respawn stale-pipeline race. Core resilience 121/121, AdminUI form 10/10, Analyzers 32/32, Runtime 364/364; 0 production OTOPCUA0001. **Deferred (T15/T18):** docker-dev `:9200` `/run` (warning banner, input lockout, unknown-key survival) + whole-solution sweep. |
| **R2-10** | 03/U10=04/U-5 (Medium/Low, action #10) | `r2/10-resilience-observability` (off `46fedda3`) | `205d3980``<docs>` (11 task commits) | **Code-complete, offline-verified.** The resilience status tracker now has a production reader, mirroring the driver-health flow. Core truthfulness: `RecordBreakerClose` + derived `IsBreakerOpen` (breaker-open state derivable), `OnClosed` records the close, and `RecordSuccess` wired onto the invoker success paths (`ConsecutiveFailures` is a true consecutive counter, not stuck-forever). Transport: `DriverResilienceStatusChanged` Commons contract → periodic 5 s full-snapshot publisher `DriverResilienceStatusPublisherService` (driver nodes, registered in `AddOtOpcUaDriverFactories` with wiring guard `ResilienceStatusReaderRegistrationTests` — closes the built-but-never-wired class) → `driver-resilience-status` DPS topic → per-admin-node `DriverResilienceStatusBridge``IDriverResilienceStatusStore` → resilience chips (breaker-OPEN / consecutive-failures / in-flight / staleness) on `DriverStatusPanel.razor`, read in-process (self-HubConnection ban honored). Core.Tests resilience 31/31 (incl. deterministic breaker-close via a controllable TimeProvider), Host.IntegrationTests publisher+guard 6/6, AdminUI.Tests store 5/5; full-solution build clean. **Deferred (T10):** docker-dev live-`/run` — S7 dead-endpoint breaker-open observed on BOTH centrals + recovery-clear + staleness-dim (also the first live sighting of the #10 `circuit-breaker OPENED` log line, closing FOLLOWUP-10's behavioral gate). |
| **R2-08** | 03/S2 (High, action #8), 03/S3 (High) | `r2/08-ldap-historyread-async` (off `1a698cbb`) | `ce6d9e21``<docs>` (14 task commits) | **Code-complete, offline-verified.** **S2 (LDAP async/timeout):** the SDK raises the impersonation callback synchronously under a `SessionManager`-wide `m_eventLock` (verified vs. 1.5.378 — a hung bind serializes ALL activations), so the fix is entirely at the `LdapOpcUaUserAuthenticator` boundary: five new `Security:Ldap` keys (`ConnectionTimeoutMs`=10000 projected into the shared lib, `AuthTimeoutMs`=15000 hard whole-flow `WaitAsync` bound → fail-closed "timed out", `MaxConcurrentAuthentications`=8 `SemaphoreSlim` with release-on-core-completion so orphaned binds still count, `OutageFailureThreshold`=3/`OutageCooldownSeconds`=15 directory-outage circuit → fast-deny "unavailable", half-open probe) + `LdapAuthResult.IsSystemFailure` seam (mapped from `ServiceAccountBindFailed`) distinguishing outage from credential-deny + `LdapOptionsValidator` rules + `HandleImpersonation` contract doc + `docs/security.md`. Stall-repro RED→GREEN; offline outage-sim through the real `HandleImpersonation` against an unroutable host. **S3 (HistoryRead fan-out):** all four HistoryRead arms async-ified (`ServeNodeAsync`/`ServeRawPagedAsync`/`ServeEventsAsync`, OCE→`BadTimeout`) then bounded per-node fan-out (`RunBounded` semaphore + single arm-boundary bridge) + per-request deadline CTS (`HistoryReadDeadline`=60s) + process-wide lazy batch limiter (`MaxConcurrentHistoryReadBatches`=16 → flood degrades to `BadTooManyOperations`); 16×4=64 in-flight cap; three new `ServerHistorian` keys + node-manager props + hosted-service wiring + `docs/Historian.md`/CLAUDE.md tables. Sequential-serving RED→GREEN. Host.IntegrationTests (LDAP resilience) 13/13, Security.Tests 11/11, Runtime.Tests options 11/11, OpcUaServer.Tests HistoryRead 30/30. **Deferred (live):** S2 docker-dev unreachable-LDAP + GLAuth outage/recovery cycle; S3 `Category=LiveIntegration` gateway soak. |
| **R2-11** | 01/C-1, 01/P-1 + 05/CONV-2, UNDER-1, UNDER-6 | `r2/11-tagconfig-consolidation` (off `46fedda3`) | first `2e14fe1f` … (24 task commits) | **Code-complete, offline-verified.** **Part 1 (C-1/P-1):** single `TagConfigIntent.Parse` + `DeviceConfigIntent` in Commons replace the four byte-parity TagConfig parse copies (composer/artifact/walker/DraftValidator) — composer now parses once per tag (P-1); a permanent `Runtime.Tests/TagConfigCorpusParityTests` (30-blob corpus) guards compose≡decode; `grep "MUST parse identically" src/` = 0. **Part 2 (CONV-2):** shared strict-capable `TagConfigJson` readers in Core.Abstractions replace the six copy-pasted `ReadEnum`s; all six driver parsers gain `Inspect()` warnings + honour the `writable` key (FOCAS **forced read-only**, 05/UNDER-1); FOCAS equipment tags run the capability-matrix pre-flight (05/UNDER-6); Modbus probe parses the factory DTO shape (`timeoutMs` no longer dropped). New `EquipmentTagConfigInspector` in ControlPlane + `AdminOperationsActor` deploy-gate wiring with `Deployment:TagConfigValidationMode` (**Warn default | Error opt-in**), proven through the actor (F10b consuming test). AdminUI: `writable` checkbox in the six typed editors. **Phase-C follow-up (deferred, recorded here):** flip the runtime parsers to strict (typo'd enum ⇒ `BadNodeIdUnknown` instead of wrong-width `Good`) once fleets have run `Error` mode clean — behavior change deliberately not shipped in this plan. **Deferred (T22/T24):** docker-dev `/run` of one typed editor's Writable checkbox + whole-solution serial sweep. |
**Report anchor drifts to fold into the next report refresh** (from the R2-02 verification pass): (1)
+7 -1
View File
@@ -69,7 +69,10 @@ and all HistoryRead calls on historized nodes return `GoodNoData` (empty, not an
"AllowUntrustedServerCertificate": false,
"CaCertificatePath": null,
"CallTimeout": "00:00:30",
"MaxTieClusterOverfetch": 65536
"MaxTieClusterOverfetch": 65536,
"HistoryReadBatchParallelism": 4,
"MaxConcurrentHistoryReadBatches": 16,
"HistoryReadDeadline": "00:01:00"
}
}
```
@@ -84,6 +87,9 @@ and all HistoryRead calls on historized nodes return `GoodNoData` (empty, not an
| `CaCertificatePath` | string\|null | `null` | PEM CA file pinning the gateway's TLS chain. Null/empty uses the OS trust store. |
| `CallTimeout` | TimeSpan | `00:00:30` | Per-call deadline applied to each unary gateway read. A read for a tag the historian does not yet know about — e.g. a freshly-historized tag that has **not** been provisioned (see [Tag auto-provisioning](#tag-auto-provisioning-ensuretags)) — can run to this **full** deadline before the gateway errors, so an unprovisioned-tag HistoryRead can block for ~30 s. Lower this to fail faster, at the cost of truncating legitimately long reads. |
| `MaxTieClusterOverfetch` | int | `65536` | Maximum samples the server will fetch in one shot to page through a tie cluster (multiple samples sharing one `SourceTimestamp`). A cluster larger than this ceiling fails `BadHistoryOperationUnsupported`. Raise to handle abnormally large tie clusters; the default covers all normal-data cases. |
| `HistoryReadBatchParallelism` | int | `4` | Max historized nodes served **concurrently** within one HistoryRead batch (arch-review 03/S3). The SDK's `HistoryRead*` override surface is synchronous, so each arm bridges once at its boundary; this turns per-node serving from sequential (`N × CallTimeout` worst case) into `⌈N / P⌉ × CallTimeout`. Also caps the gateway-load multiplication a single client can command. `≤ 1` serves sequentially. |
| `MaxConcurrentHistoryReadBatches` | int | `16` | Process-wide limit on HistoryRead batches served **concurrently** (arch-review 03/S3). A flood of history clients degrades to `BadTooManyOperations` rather than exhausting the SDK request-thread pool. With `HistoryReadBatchParallelism` this caps total in-flight gateway reads at `MaxConcurrentHistoryReadBatches × HistoryReadBatchParallelism` (default 16 × 4 = 64). A batch that cannot get a slot within `min(HistoryReadDeadline, 5 s)` fails every handle `BadTooManyOperations`. |
| `HistoryReadDeadline` | TimeSpan | `00:01:00` | Per-request wall-clock deadline for a HistoryRead batch (arch-review 03/S3). A node still in flight at the deadline surfaces `BadTimeout`, so one request can never hold an SDK request thread longer than this regardless of node count. Non-positive = unbounded (a `Validate()` warning). |
> **Do not commit `ApiKey` to `appsettings.json`.** Set it via the environment variable
> `ServerHistorian__ApiKey`, a secrets store, or a deployment-time overlay. The checked-in default is
+14
View File
@@ -162,6 +162,20 @@ LDAP is configured under the `Security:Ldap` section (bound to `LdapOptions`, `s
`GroupToRole` maps LDAP group names → roles (case-insensitive); a user gets every role whose source group is in their membership. The values are the canonical control-plane role strings (`Viewer` / `Designer` / `Administrator`, plus the appsettings-only `Operator` for the `DriverOperator` policy); the same map also supplies data-plane role strings (`ReadOnly`, `WriteOperate`, `WriteTune`, `WriteConfigure`, `AlarmAck`) — see [Role grant source (data-plane)](#role-grant-source-data-plane) below. `UserNameAttribute: "sAMAccountName"` is the critical AD override — the GLAuth dev default is `cn`, which is not how AD users are looked up; use `userPrincipalName` instead if operators log in with `user@corp.example.com` form. `LdapOptionsValidator` (`src/Server/ZB.MOM.WW.OtOpcUa.Host/Configuration/LdapOptionsValidator.cs`) fails startup when `Transport = None` and `AllowInsecure = false` on a real-LDAP (non-DevStub) config.
### LDAP resilience — timeout, concurrency, outage circuit (arch-review 03/S2)
The OPC UA SDK raises its impersonation (UserName) callback **synchronously**, inside a `SessionManager`-wide event lock (verified against `OPCFoundation.NetStandard.Opc.Ua.Server` 1.5.378). A slow or hung LDAP bind inside that callback therefore does not stall one SDK thread — it **serializes every session activation server-wide** (Anonymous and X509 included). Since the callback is irreducibly synchronous, `LdapOpcUaUserAuthenticator` bounds it at its own boundary with five `Security:Ldap` keys (all validated at startup by `LdapOptionsValidator` when `Enabled && !DevStubMode`):
| Key | Default | Bounds |
|---|---|---|
| `ConnectionTimeoutMs` | `10000` | Projected into the shared library's `ConnectionTimeoutMs` — bounds the socket connect **and** each per-operation search. First line of defence against an outage. |
| `AuthTimeoutMs` | `15000` | Hard whole-flow wall-clock bound (bind **and** role-map) enforced at the authenticator boundary; a slow/hung flow denies **"Authentication timed out"** (fail-closed) and releases the SDK thread. Deliberately `> ConnectionTimeoutMs` so a healthy-but-slow flow is not cut off. |
| `MaxConcurrentAuthentications` | `8` | Max in-flight authentications; excess denies **"Authentication backend busy"** (local backpressure — does **not** feed the circuit). Bounds accumulation of orphaned/timed-out binds during an outage. |
| `OutageFailureThreshold` | `3` | Consecutive **system-side** failures (boundary timeout, directory-unreachable, or unexpected throw) that open the outage circuit. Any success or user-credential deny resets the streak. `0` disables the circuit. |
| `OutageCooldownSeconds` | `15` | How long the open circuit fast-denies **"Authentication backend unavailable"** without touching LDAP; after cooldown the next call half-open-probes, and one more system failure re-opens immediately. |
Defaults are behaviour-neutral on a healthy directory. All new paths are **fail-closed** (they deny). A user-credential deny (wrong password) is fast when the directory is up and never trips the circuit; only a directory *outage* opens it.
---
## Data-Plane Authorization
@@ -47,5 +47,18 @@ public sealed class LdapOptionsValidator : OptionsValidatorBase<LdapOptions>
builder.RequireThat(
!(options.Transport == LdapTransport.None && !options.AllowInsecure),
"LDAP transport is None (plaintext) but AllowInsecure is false — set Transport to Ldaps/StartTls or set AllowInsecure for dev.");
// S2 resilience knobs — a non-positive timeout / concurrency bound would silently disable the
// wall-clock/concurrency protection the OPC UA data-plane authenticator relies on, so fail fast.
builder.RequireThat(options.ConnectionTimeoutMs > 0,
"Ldap:ConnectionTimeoutMs must be > 0.");
builder.RequireThat(options.AuthTimeoutMs > 0,
"Ldap:AuthTimeoutMs must be > 0.");
builder.RequireThat(options.MaxConcurrentAuthentications > 0,
"Ldap:MaxConcurrentAuthentications must be > 0.");
// An open circuit with no cooldown would re-probe LDAP on every call — pointless. Only enforce
// a positive cooldown when the circuit is actually enabled (OutageFailureThreshold > 0; 0 disables).
builder.RequireThat(options.OutageFailureThreshold <= 0 || options.OutageCooldownSeconds > 0,
"Ldap:OutageCooldownSeconds must be > 0 when Ldap:OutageFailureThreshold > 0 (0 disables the circuit).");
}
}
@@ -1,5 +1,6 @@
using Microsoft.Extensions.DependencyInjection;
using Microsoft.Extensions.Logging;
using Microsoft.Extensions.Options;
using ZB.MOM.WW.Auth.Abstractions.Roles;
using ZB.MOM.WW.OtOpcUa.OpcUaServer.Security;
using ZB.MOM.WW.OtOpcUa.Security.Ldap;
@@ -16,37 +17,215 @@ namespace ZB.MOM.WW.OtOpcUa.Host.OpcUa;
/// session identity for the downstream data-plane ACL evaluator.
/// </summary>
/// <remarks>
/// <para>
/// This authenticator is registered as a singleton, but <see cref="IGroupRoleMapper{TRole}"/>
/// (and its DbContext-backed mapping service) is scoped. A per-call DI scope is opened to
/// resolve the mapper so the singleton never captures a scoped dependency.
/// </para>
/// <para>
/// <b>Resilience (arch-review 03/S2).</b> The OPC UA SDK raises the impersonation callback that
/// reaches here <em>synchronously</em>, inside a <c>SessionManager</c>-wide event lock (verified
/// against <c>OPCFoundation.NetStandard.Opc.Ua.Server</c> 1.5.378 —
/// <c>SessionManager.ActivateSessionAsync</c> holds <c>m_eventLock</c> while invoking the
/// <c>void</c> impersonation delegate). A slow / hung LDAP bind inside that callback therefore does
/// not merely stall one SDK thread — it serializes <em>every</em> session activation server-wide
/// (Anonymous and X509 included). The callback contract is irreducibly synchronous (there is no
/// async impersonation callback), so the fix lives entirely at this boundary: a hard whole-flow
/// wall-clock bound (<see cref="LdapOptions.AuthTimeoutMs"/>) and a bounded in-flight concurrency cap
/// (<see cref="LdapOptions.MaxConcurrentAuthentications"/>), both fail-closed. All new paths deny.
/// </para>
/// </remarks>
public sealed class LdapOpcUaUserAuthenticator(
ILdapAuthService ldap,
IServiceScopeFactory scopeFactory,
ILogger<LdapOpcUaUserAuthenticator> logger)
: IOpcUaUserAuthenticator
public sealed class LdapOpcUaUserAuthenticator : IOpcUaUserAuthenticator
{
private readonly ILdapAuthService _ldap;
private readonly IServiceScopeFactory _scopeFactory;
private readonly ILogger<LdapOpcUaUserAuthenticator> _logger;
private readonly LdapOptions _options;
/// <summary>Bounds in-flight authentication flows. A boundary timeout abandons its core task, which
/// keeps holding its slot until it actually completes — so an outage cannot accumulate unbounded
/// orphan binds.</summary>
private readonly SemaphoreSlim _gate;
/// <summary>Guards the outage-circuit state (<see cref="_consecutiveSystemFailures"/> +
/// <see cref="_circuitOpenUntilTicks"/>).</summary>
private readonly object _circuitLock = new();
/// <summary>Count of CONSECUTIVE system-side failures; reset by any success / user-side deny.</summary>
private int _consecutiveSystemFailures;
/// <summary><see cref="Environment.TickCount64"/> the circuit stays open until; <c>0</c> = closed.</summary>
private long _circuitOpenUntilTicks;
/// <summary>Constructs the authenticator from its dependencies and the bound LDAP options.</summary>
/// <param name="ldap">The app LDAP auth service (shared with the Admin UI login flow).</param>
/// <param name="scopeFactory">Opens a per-call DI scope to resolve the scoped role mapper.</param>
/// <param name="options">The bound <see cref="LdapOptions"/> (timeout + concurrency knobs).</param>
/// <param name="logger">The logger.</param>
public LdapOpcUaUserAuthenticator(
ILdapAuthService ldap,
IServiceScopeFactory scopeFactory,
IOptions<LdapOptions> options,
ILogger<LdapOpcUaUserAuthenticator> logger)
{
_ldap = ldap;
_scopeFactory = scopeFactory;
_options = options.Value;
_logger = logger;
_gate = new SemaphoreSlim(Math.Max(1, _options.MaxConcurrentAuthentications));
}
/// <inheritdoc />
public async Task<OpcUaUserAuthResult> AuthenticateUserNameAsync(string username, string password, CancellationToken ct)
{
var authTimeout = TimeSpan.FromMilliseconds(Math.Max(1, _options.AuthTimeoutMs));
// Directory-outage circuit: after a run of consecutive system-side failures, fail fast WITHOUT
// touching LDAP for the cooldown window — so a sustained outage cannot serialize every session
// activation behind a per-call stall (the SDK raises this callback under a server-wide lock).
if (IsCircuitOpen())
{
_logger.LogDebug(
"LDAP outage circuit open — fast-denying OPC UA user {User} without a directory call", username);
return OpcUaUserAuthResult.Deny("Authentication backend unavailable");
}
// Bounded in-flight concurrency: acquire a slot within our own wall-clock budget. All slots
// busy is LOCAL backpressure, not directory evidence — deny "busy" (fail-closed) and do NOT
// feed the outage circuit.
if (!await _gate.WaitAsync(authTimeout, ct).ConfigureAwait(false))
{
_logger.LogWarning(
"LDAP authentication backend busy (concurrency cap {Cap}) — denying OPC UA user {User}",
_options.MaxConcurrentAuthentications, username);
return OpcUaUserAuthResult.Deny("Authentication backend busy");
}
// Release the slot when the CORE task completes — deliberately NOT in a finally here. A boundary
// timeout below abandons the core task (it runs to completion in the background; the underlying
// library is documented never-throw, so there are no unobserved exceptions), and it must keep
// counting against the cap until it finishes. The point of the boundary is to release the SDK
// thread (and m_eventLock), not to cancel the bind.
var core = AuthenticateCoreAsync(username, password, ct);
_ = core.ContinueWith(
static (_, state) => ((SemaphoreSlim)state!).Release(),
_gate,
CancellationToken.None,
TaskContinuationOptions.None,
TaskScheduler.Default);
try
{
var result = await ldap.AuthenticateAsync(username, password, ct).ConfigureAwait(false);
var outcome = await core.WaitAsync(authTimeout, ct).ConfigureAwait(false);
RecordOutcome(outcome.SystemFailure, username);
return outcome.Result;
}
catch (TimeoutException)
{
// A boundary timeout is directory evidence — it feeds the circuit.
RecordOutcome(systemFailure: true, username);
_logger.LogWarning(
"LDAP authentication timed out after {Timeout} for OPC UA user {User} — denying (fail-closed)",
authTimeout, username);
return OpcUaUserAuthResult.Deny("Authentication timed out");
}
}
/// <summary>Whether the outage circuit is currently open (fast-deny window active). Always false when
/// the circuit is disabled (<see cref="LdapOptions.OutageFailureThreshold"/> ≤ 0).</summary>
private bool IsCircuitOpen()
{
if (_options.OutageFailureThreshold <= 0) return false;
lock (_circuitLock)
{
return _circuitOpenUntilTicks > Environment.TickCount64;
}
}
/// <summary>
/// Updates the outage-circuit state from one authentication outcome. A system-side failure advances
/// the consecutive-failure streak and opens the circuit once it reaches
/// <see cref="LdapOptions.OutageFailureThreshold"/>; any non-system outcome (success or user-side
/// deny) resets the streak and closes the circuit. Half-open behaviour is implicit: once the cooldown
/// elapses <see cref="IsCircuitOpen"/> lets the next call through with the streak still at threshold,
/// so a single further system failure re-opens immediately.
/// </summary>
/// <param name="systemFailure">Whether this outcome was a system-side (directory) failure.</param>
/// <param name="username">The login name, for diagnostics.</param>
private void RecordOutcome(bool systemFailure, string username)
{
if (_options.OutageFailureThreshold <= 0) return; // circuit disabled
lock (_circuitLock)
{
if (systemFailure)
{
_consecutiveSystemFailures++;
if (_consecutiveSystemFailures >= _options.OutageFailureThreshold)
{
var wasOpen = _circuitOpenUntilTicks > Environment.TickCount64;
_circuitOpenUntilTicks =
Environment.TickCount64 + (long)Math.Max(1, _options.OutageCooldownSeconds) * 1000L;
if (!wasOpen)
{
_logger.LogWarning(
"LDAP outage circuit OPENED after {Failures} consecutive system-side failures — " +
"fast-denying for {Cooldown}s (last user {User})",
_consecutiveSystemFailures, _options.OutageCooldownSeconds, username);
}
}
}
else
{
if (_consecutiveSystemFailures >= _options.OutageFailureThreshold)
{
_logger.LogInformation(
"LDAP outage circuit CLOSED — directory recovered (probe succeeded for {User})", username);
}
_consecutiveSystemFailures = 0;
_circuitOpenUntilTicks = 0;
}
}
}
/// <summary>
/// The unbounded core authenticate flow (bind + role-map) that the outer
/// <see cref="AuthenticateUserNameAsync"/> wraps in a wall-clock bound + concurrency cap. Fail-closed:
/// any backend throw becomes a denial rather than escaping into the SDK. The returned
/// <see cref="CoreOutcome.SystemFailure"/> flags directory-side faults (a system-side
/// <see cref="LdapAuthResult.IsSystemFailure"/> deny, or an unexpected throw) so the caller can feed
/// the outage circuit; a user-credential deny is NOT a system failure.
/// </summary>
/// <param name="username">The username to authenticate.</param>
/// <param name="password">The cleartext password.</param>
/// <param name="ct">Cancellation token.</param>
private async Task<CoreOutcome> AuthenticateCoreAsync(string username, string password, CancellationToken ct)
{
try
{
var result = await _ldap.AuthenticateAsync(username, password, ct).ConfigureAwait(false);
if (!result.Success)
{
return OpcUaUserAuthResult.Deny(result.Error ?? "Invalid credentials");
return new CoreOutcome(
OpcUaUserAuthResult.Deny(result.Error ?? "Invalid credentials"), result.IsSystemFailure);
}
var roles = await ResolveRolesAsync(result.Groups, result.Roles, username, ct).ConfigureAwait(false);
return OpcUaUserAuthResult.Allow(result.DisplayName ?? username, roles);
return new CoreOutcome(OpcUaUserAuthResult.Allow(result.DisplayName ?? username, roles), SystemFailure: false);
}
catch (Exception ex) when (ex is not OperationCanceledException)
{
logger.LogWarning(ex, "LDAP authentication threw for OPC UA user {User}", username);
return OpcUaUserAuthResult.Deny("Authentication backend error");
_logger.LogWarning(ex, "LDAP authentication threw for OPC UA user {User}", username);
return new CoreOutcome(OpcUaUserAuthResult.Deny("Authentication backend error"), SystemFailure: true);
}
}
/// <summary>The result of the core authenticate flow plus whether it was a system-side (directory)
/// failure — the signal that drives the outage circuit.</summary>
/// <param name="Result">The auth result to return to the SDK.</param>
/// <param name="SystemFailure">Whether the outcome was a directory-side fault.</param>
private readonly record struct CoreOutcome(OpcUaUserAuthResult Result, bool SystemFailure);
/// <summary>
/// Resolves the user's roles from their LDAP groups via the scoped
/// <see cref="IGroupRoleMapper{TRole}"/>, unioned with any pre-resolved roles (the DevStub
@@ -62,7 +241,7 @@ public sealed class LdapOpcUaUserAuthenticator(
{
try
{
await using var scope = scopeFactory.CreateAsyncScope();
await using var scope = _scopeFactory.CreateAsyncScope();
var mapper = scope.ServiceProvider.GetRequiredService<IGroupRoleMapper<string>>();
var mapping = await mapper.MapAsync(groups, ct).ConfigureAwait(false);
@@ -73,7 +252,7 @@ public sealed class LdapOpcUaUserAuthenticator(
}
catch (Exception ex) when (ex is not OperationCanceledException)
{
logger.LogWarning(ex,
_logger.LogWarning(ex,
"Role-map lookup failed for OPC UA user {User}; using pre-resolved baseline roles", username);
return preResolved;
}
@@ -222,6 +222,13 @@ public sealed class OtOpcUaServerHostedService : IHostedService, IAsyncDisposabl
// section is absent; NodeManager is non-null here (guarded above).
_server.NodeManager.MaxTieClusterOverfetch = _serverHistorianOptions.MaxTieClusterOverfetch;
// Propagate the S3 HistoryRead concurrency bounds (arch-review 03/S3): bounded per-node fan-out
// within a batch, the process-wide concurrent-batch limiter, and the per-request deadline. Defaults
// (4 / 16 / 60 s) survive when the ServerHistorian section is absent.
_server.NodeManager.HistoryReadBatchParallelism = _serverHistorianOptions.HistoryReadBatchParallelism;
_server.NodeManager.MaxConcurrentHistoryReadBatches = _serverHistorianOptions.MaxConcurrentHistoryReadBatches;
_server.NodeManager.HistoryReadDeadline = _serverHistorianOptions.HistoryReadDeadline;
// ServiceLevel publisher needs IServerInternal — only available after Start.
if (_server.CurrentInstance is { } serverInternal)
{
@@ -235,6 +235,19 @@ public sealed class OpcUaApplicationHost : IAsyncDisposable
/// the full SDK. Side-effects are confined to mutating <see cref="ImpersonateEventArgs"/>
/// and logging.
/// </summary>
/// <remarks>
/// <b>SDK threading contract (verified against OPCFoundation.NetStandard.Opc.Ua.Server 1.5.378).</b>
/// The SDK raises this handler SYNCHRONOUSLY — it is a plain <c>void</c> delegate, and there is no
/// async impersonation callback in this SDK version. Worse, <c>SessionManager.ActivateSessionAsync</c>
/// invokes it while holding a single <c>SessionManager</c>-wide event lock (<c>m_eventLock</c>), so a
/// handler that blocks does not merely stall one SDK request thread — it SERIALIZES every session
/// activation server-wide (Anonymous and X509 included, since even those paths must take the lock to
/// raise the handler). Consequently the <see cref="IOpcUaUserAuthenticator"/> block-bridge below
/// (<c>GetAwaiter().GetResult()</c> under <see cref="CancellationToken.None"/>) is irreducible: the
/// callback must complete inline. Because the SDK gives no timeout, the hard wall-clock bound + bounded
/// concurrency + directory-outage circuit that keep a slow/hung directory from wedging the SDK live
/// inside the authenticator (<c>LdapOpcUaUserAuthenticator</c>, arch-review 03/S2) — NOT here.
/// </remarks>
/// <param name="authenticator">The user authenticator to validate credentials.</param>
/// <param name="args">The impersonation event arguments to process.</param>
/// <param name="logger">The logger for diagnostic output.</param>
@@ -197,6 +197,34 @@ public sealed class OtOpcUaNodeManager : CustomNodeManager2
/// </summary>
public int MaxTieClusterOverfetch { get; set; } = 65536;
/// <summary>
/// Maximum historized nodes served concurrently within one HistoryRead batch (arch-review 03/S3).
/// Mirrors <c>ServerHistorianOptions.HistoryReadBatchParallelism</c>; the Host sets it at
/// <c>StartAsync</c>. <c>≤ 1</c> falls back to sequential serving. The default (4) survives when the
/// ServerHistorian section is absent.
/// </summary>
public int HistoryReadBatchParallelism { get; set; } = 4;
/// <summary>
/// Process-wide cap on concurrently-served HistoryRead batches (arch-review 03/S3). A batch that
/// cannot acquire a slot within its (bounded) wait budget fast-fails every handle with
/// <c>BadTooManyOperations</c>. Mirrors <c>ServerHistorianOptions.MaxConcurrentHistoryReadBatches</c>;
/// the Host sets it at <c>StartAsync</c>. The default (16) survives when the section is absent.
/// </summary>
public int MaxConcurrentHistoryReadBatches { get; set; } = 16;
/// <summary>
/// Per-request wall-clock deadline for a HistoryRead batch (arch-review 03/S3). A node still in
/// flight at the deadline surfaces <c>BadTimeout</c>. Non-positive = unbounded. Mirrors
/// <c>ServerHistorianOptions.HistoryReadDeadline</c>; the Host sets it at <c>StartAsync</c>. The
/// default (60 s) survives when the section is absent.
/// </summary>
public TimeSpan HistoryReadDeadline { get; set; } = TimeSpan.FromSeconds(60);
/// <summary>Lazily-created process-wide HistoryRead-batch limiter (see <see cref="MaxConcurrentHistoryReadBatches"/>).</summary>
private volatile SemaphoreSlim? _historyReadBatchLimiter;
private readonly object _historyReadBatchLimiterLock = new();
private volatile IHistoryContinuationStore _historyContinuationStore = new SessionHistoryContinuationStore();
/// <summary>
@@ -1792,19 +1820,28 @@ public sealed class OtOpcUaNodeManager : CustomNodeManager2
IDictionary<NodeId, NodeState> cache)
{
var session = context.OperationContext?.Session;
foreach (var handle in nodesToProcess)
WithHistoryReadBatchLimiter(nodesToProcess, results, errors, () =>
{
if (details.IsReadModified)
using var deadline = CreateDeadline();
var ct = deadline?.Token ?? CancellationToken.None;
var work = new List<Func<Task>>(nodesToProcess.Count);
foreach (var handle in nodesToProcess)
{
// We never serve modified-value history; mark this node unsupported and move on.
errors[handle.Index] = StatusCodes.BadHistoryOperationUnsupported;
continue;
if (details.IsReadModified)
{
// We never serve modified-value history; mark this node unsupported and move on.
errors[handle.Index] = StatusCodes.BadHistoryOperationUnsupported;
continue;
}
work.Add(() => ServeRawPagedAsync(
handle, session, nodesToRead, results, errors,
details.StartTime, details.EndTime, details.NumValuesPerNode,
ct));
}
ServeRawPaged(
handle, session, nodesToRead, results, errors,
details.StartTime, details.EndTime, details.NumValuesPerNode);
}
RunBounded(work, HistoryReadBatchParallelism);
});
}
/// <inheritdoc />
@@ -1820,31 +1857,40 @@ public sealed class OtOpcUaNodeManager : CustomNodeManager2
{
// OPC UA ProcessingInterval is a Duration in milliseconds — convert once per batch.
var interval = TimeSpan.FromMilliseconds(details.ProcessingInterval);
foreach (var handle in nodesToProcess)
WithHistoryReadBatchLimiter(nodesToProcess, results, errors, () =>
{
// AggregateType is a per-node parallel collection (same length as nodesToRead, enforced by
// the base dispatcher). handle.Index is the node's position in that collection.
var aggregateNodeId = details.AggregateType[handle.Index];
var aggregate = MapAggregate(aggregateNodeId);
if (aggregate is null)
using var deadline = CreateDeadline();
var ct = deadline?.Token ?? CancellationToken.None;
var work = new List<Func<Task>>(nodesToProcess.Count);
foreach (var handle in nodesToProcess)
{
errors[handle.Index] = StatusCodes.BadAggregateNotSupported;
continue;
// AggregateType is a per-node parallel collection (same length as nodesToRead, enforced by
// the base dispatcher). handle.Index is the node's position in that collection.
var aggregateNodeId = details.AggregateType[handle.Index];
var aggregate = MapAggregate(aggregateNodeId);
if (aggregate is null)
{
errors[handle.Index] = StatusCodes.BadAggregateNotSupported;
continue;
}
// Processed is SINGLE-SHOT (no continuation point). Unlike Raw, ReadProcessedDetails carries
// NO client count cap (NumValuesPerNode) — the bucket count is deterministic (window / interval)
// and the single-shot backend returns every bucket in one read, so there is no "full page ⇒
// maybe more" signal to page on. Returning the complete aggregate result with a null CP is
// spec-conformant (OPC UA Part 11 lets a server return all available data in one response).
var agg = aggregate.Value;
work.Add(() => ServeNodeAsync(handle, results, errors, (source, tagname, token) => source.ReadProcessedAsync(
tagname,
details.StartTime,
details.EndTime,
interval,
agg,
token), ct));
}
// Processed is SINGLE-SHOT (no continuation point). Unlike Raw, ReadProcessedDetails carries
// NO client count cap (NumValuesPerNode) — the bucket count is deterministic (window / interval)
// and the single-shot backend returns every bucket in one read, so there is no "full page ⇒
// maybe more" signal to page on. Returning the complete aggregate result with a null CP is
// spec-conformant (OPC UA Part 11 lets a server return all available data in one response).
ServeNode(handle, results, errors, (source, tagname) => source.ReadProcessedAsync(
tagname,
details.StartTime,
details.EndTime,
interval,
aggregate.Value,
CancellationToken.None));
}
RunBounded(work, HistoryReadBatchParallelism);
});
}
/// <inheritdoc />
@@ -1860,13 +1906,21 @@ public sealed class OtOpcUaNodeManager : CustomNodeManager2
{
// Snapshot the requested timestamps once — the same list is read for every node.
var timestamps = details.ReqTimes?.ToList() ?? new List<DateTime>();
foreach (var handle in nodesToProcess)
WithHistoryReadBatchLimiter(nodesToProcess, results, errors, () =>
{
ServeNode(handle, results, errors, (source, tagname) => source.ReadAtTimeAsync(
tagname,
timestamps,
CancellationToken.None));
}
using var deadline = CreateDeadline();
var ct = deadline?.Token ?? CancellationToken.None;
var work = new List<Func<Task>>(nodesToProcess.Count);
foreach (var handle in nodesToProcess)
{
work.Add(() => ServeNodeAsync(handle, results, errors, (source, tagname, token) => source.ReadAtTimeAsync(
tagname,
timestamps,
token), ct));
}
RunBounded(work, HistoryReadBatchParallelism);
});
}
/// <inheritdoc />
@@ -1883,54 +1937,95 @@ public sealed class OtOpcUaNodeManager : CustomNodeManager2
// Snapshot the select clauses once — the same filter projects every node's events.
var selectClauses = details.Filter?.SelectClauses ?? new SimpleAttributeOperandCollection();
foreach (var handle in nodesToProcess)
WithHistoryReadBatchLimiter(nodesToProcess, results, errors, () =>
{
var idString = handle.NodeId.Identifier?.ToString();
if (idString is null || !_eventNotifierSources.TryGetValue(idString, out var sourceName))
using var deadline = CreateDeadline();
var ct = deadline?.Token ?? CancellationToken.None;
var work = new List<Func<Task>>(nodesToProcess.Count);
foreach (var handle in nodesToProcess)
{
// Not a registered event-history source (plain folder / Null-source promotion) ⇒ unsupported.
// Set both errors and results explicitly on every bad path — don't rely on the SDK base
// pre-seeding results[i], so every path is self-contained and the contract is obvious.
errors[handle.Index] = StatusCodes.BadHistoryOperationUnsupported;
results[handle.Index] = new SdkHistoryReadResult { StatusCode = StatusCodes.BadHistoryOperationUnsupported };
continue;
}
try
{
// NOT under the node-manager Lock — block-bridging the async source is safe here.
var sourceResult = _historianDataSource.ReadEventsAsync(
sourceName,
details.StartTime,
details.EndTime,
// NumValuesPerNode==0 means "no limit — return ALL values" per OPC UA
// Part 4/11, so translate it to UNBOUNDED (int.MaxValue) here. Passing the int<=0
// backend-default-cap sentinel instead would silently truncate a "give me everything"
// events read at the backend default. A positive cap passes through (clamped).
EventMaxEvents(details.NumValuesPerNode),
CancellationToken.None).GetAwaiter().GetResult();
var historyEvent = ProjectEvents(sourceResult.Events, selectClauses);
results[handle.Index] = new SdkHistoryReadResult
var idString = handle.NodeId.Identifier?.ToString();
if (idString is null || !_eventNotifierSources.TryGetValue(idString, out var sourceName))
{
// No events ⇒ GoodNoData (the notifier is historized, the window just held no events).
StatusCode = sourceResult.Events.Count == 0 ? StatusCodes.GoodNoData : StatusCodes.Good,
HistoryData = new ExtensionObject(historyEvent),
// We never issue continuation points — every read returns the full window in one shot.
ContinuationPoint = null,
};
errors[handle.Index] = ServiceResult.Good;
// Not a registered event-history source (plain folder / Null-source promotion) ⇒ unsupported.
// Set both errors and results explicitly on every bad path — don't rely on the SDK base
// pre-seeding results[i], so every path is self-contained and the contract is obvious.
errors[handle.Index] = StatusCodes.BadHistoryOperationUnsupported;
results[handle.Index] = new SdkHistoryReadResult { StatusCode = StatusCodes.BadHistoryOperationUnsupported };
continue;
}
var sn = sourceName;
work.Add(() => ServeEventsAsync(handle, sn, details, selectClauses, results, errors, ct));
}
catch (Exception ex)
RunBounded(work, HistoryReadBatchParallelism);
});
}
/// <summary>
/// Serve one registered event-history notifier handle: reads the source's events over the request
/// window, projects them onto the select clauses, and fills the service-level results/errors slots.
/// Per-node error isolation matches <see cref="ServeNodeAsync"/> — a backend throw becomes a Bad
/// status for THIS node only and never throws out of the batch; a deadline cancellation surfaces
/// <c>BadTimeout</c>.
/// </summary>
/// <param name="handle">The notifier node handle; <c>handle.Index</c> indexes results/errors.</param>
/// <param name="sourceName">The registered historian event-source name for this notifier.</param>
/// <param name="details">The event-read details (window + count cap).</param>
/// <param name="selectClauses">The event-filter select clauses (the fields to emit, in order).</param>
/// <param name="results">The service-level results list to fill at <c>handle.Index</c>.</param>
/// <param name="errors">The service-level errors list to fill at <c>handle.Index</c>.</param>
/// <param name="ct">Per-request deadline token.</param>
private async Task ServeEventsAsync(
NodeHandle handle,
string? sourceName,
ReadEventDetails details,
SimpleAttributeOperandCollection selectClauses,
IList<SdkHistoryReadResult> results,
IList<ServiceResult> errors,
CancellationToken ct)
{
try
{
// NOT under the node-manager Lock — awaiting the async source is safe here.
var sourceResult = await _historianDataSource.ReadEventsAsync(
sourceName,
details.StartTime,
details.EndTime,
// NumValuesPerNode==0 means "no limit — return ALL values" per OPC UA
// Part 4/11, so translate it to UNBOUNDED (int.MaxValue) here. Passing the int<=0
// backend-default-cap sentinel instead would silently truncate a "give me everything"
// events read at the backend default. A positive cap passes through (clamped).
EventMaxEvents(details.NumValuesPerNode),
ct).ConfigureAwait(false);
var historyEvent = ProjectEvents(sourceResult.Events, selectClauses);
results[handle.Index] = new SdkHistoryReadResult
{
// One node's backend failure must not throw out of the batch — surface Bad for THIS node
// only. This manager carries no ILogger, so log via the SDK's static trace (see ServeNode).
// No events ⇒ GoodNoData (the notifier is historized, the window just held no events).
StatusCode = sourceResult.Events.Count == 0 ? StatusCodes.GoodNoData : StatusCodes.Good,
HistoryData = new ExtensionObject(historyEvent),
// We never issue continuation points — every read returns the full window in one shot.
ContinuationPoint = null,
};
errors[handle.Index] = ServiceResult.Good;
}
catch (OperationCanceledException)
{
// The per-request deadline elapsed while reading this notifier — BadTimeout for THIS node only.
errors[handle.Index] = StatusCodes.BadTimeout;
results[handle.Index] = new SdkHistoryReadResult { StatusCode = StatusCodes.BadTimeout };
}
catch (Exception ex)
{
// One node's backend failure must not throw out of the batch — surface Bad for THIS node
// only. This manager carries no ILogger, so log via the SDK's static trace (see ServeNodeAsync).
#pragma warning disable CS0618 // Type or member is obsolete
Utils.LogError(ex, "OtOpcUaNodeManager: HistoryReadEvents failed for node {0}", handle.NodeId);
Utils.LogError(ex, "OtOpcUaNodeManager: HistoryReadEvents failed for node {0}", handle.NodeId);
#pragma warning restore CS0618
errors[handle.Index] = StatusCodes.BadHistoryOperationUnsupported;
results[handle.Index] = new SdkHistoryReadResult { StatusCode = StatusCodes.BadHistoryOperationUnsupported };
}
errors[handle.Index] = StatusCodes.BadHistoryOperationUnsupported;
results[handle.Index] = new SdkHistoryReadResult { StatusCode = StatusCodes.BadHistoryOperationUnsupported };
}
}
@@ -2037,11 +2132,131 @@ public sealed class OtOpcUaNodeManager : CustomNodeManager2
/// <param name="errors">The service-level errors list to fill at <c>handle.Index</c>.</param>
/// <param name="read">Invokes the resolved data-source read with the resolved tagname; only called
/// once the tagname is confirmed present.</param>
private void ServeNode(
/// <summary>
/// Run a batch of per-node HistoryRead work items with BOUNDED parallelism (arch-review 03/S3),
/// block-bridged ONCE at the arm boundary. <b>Contract:</b> every work item must be fully
/// self-contained — it fills its own results/errors slot and NEVER throws (all per-node handlers
/// catch every exception, including cancellation, and surface a Bad status for that node only). That
/// invariant is what makes the single <c>GetAwaiter().GetResult()</c> safe: <see cref="Task.WhenAll(Task[])"/>
/// can never fault, so it cannot throw out of the synchronous SDK override. A degree ≤ 1 (or a
/// single item) runs sequentially — no semaphore, no extra tasks.
/// </summary>
/// <param name="work">The per-node work items (index-disjoint writes into the shared results/errors).</param>
/// <param name="degree">Max concurrent work items (<see cref="HistoryReadBatchParallelism"/>).</param>
private static void RunBounded(IReadOnlyList<Func<Task>> work, int degree)
{
if (work.Count == 0) return;
if (degree <= 1 || work.Count == 1)
{
foreach (var item in work)
item().GetAwaiter().GetResult();
return;
}
using var gate = new SemaphoreSlim(degree);
var tasks = new Task[work.Count];
for (var i = 0; i < work.Count; i++)
tasks[i] = RunGatedAsync(gate, work[i]);
// Safe: every body is self-contained + never throws, so WhenAll never faults (see contract above).
Task.WhenAll(tasks).GetAwaiter().GetResult();
}
/// <summary>
/// Run one HistoryRead arm under the process-wide concurrent-batch limiter (arch-review 03/S3). Each
/// arm waits up to <c>min(HistoryReadDeadline, 5 s)</c> for a slot; on expiry every handle in the
/// batch gets <c>BadTooManyOperations</c> (fast-fail under flood, brief waits under mild contention)
/// and <paramref name="serve"/> is skipped. On acquisition the arm body runs and the slot is
/// released in a <c>finally</c>. Combined with <see cref="HistoryReadBatchParallelism"/> this caps
/// total in-flight gateway reads at
/// <c>MaxConcurrentHistoryReadBatches × HistoryReadBatchParallelism</c>.
/// </summary>
/// <param name="nodesToProcess">The batch's handles — every slot is failed on limiter saturation.</param>
/// <param name="results">The service-level results list.</param>
/// <param name="errors">The service-level errors list.</param>
/// <param name="serve">The arm body to run once a slot is acquired.</param>
private void WithHistoryReadBatchLimiter(
List<NodeHandle> nodesToProcess,
IList<SdkHistoryReadResult> results,
IList<ServiceResult> errors,
Action serve)
{
var gate = GetHistoryReadBatchLimiter();
// Bounded admission wait: cap the block at 5 s, and never exceed the per-request deadline.
var waitMs = HistoryReadDeadline > TimeSpan.Zero
? Math.Min(HistoryReadDeadline.TotalMilliseconds, 5000.0)
: 5000.0;
if (!gate.Wait(TimeSpan.FromMilliseconds(waitMs)))
{
foreach (var handle in nodesToProcess)
{
errors[handle.Index] = StatusCodes.BadTooManyOperations;
results[handle.Index] = new SdkHistoryReadResult { StatusCode = StatusCodes.BadTooManyOperations };
}
return;
}
try
{
serve();
}
finally
{
gate.Release();
}
}
/// <summary>
/// Lazily create (double-checked) the process-wide HistoryRead-batch limiter, sized from
/// <see cref="MaxConcurrentHistoryReadBatches"/> at first use so the Host's post-boot property set is
/// respected (the same benign wiring race as <see cref="MaxTieClusterOverfetch"/>).
/// </summary>
/// <returns>The shared batch limiter semaphore.</returns>
private SemaphoreSlim GetHistoryReadBatchLimiter()
{
var gate = _historyReadBatchLimiter;
if (gate is not null) return gate;
lock (_historyReadBatchLimiterLock)
{
return _historyReadBatchLimiter ??= new SemaphoreSlim(Math.Max(1, MaxConcurrentHistoryReadBatches));
}
}
/// <summary>
/// Create the per-request deadline source for a HistoryRead arm (arch-review 03/S3): a
/// <see cref="CancellationTokenSource"/> that fires after <see cref="HistoryReadDeadline"/>, so a
/// single request can never hold an SDK request thread longer than the deadline regardless of node
/// count. Returns <c>null</c> when the deadline is non-positive (unbounded — the arm then passes
/// <see cref="CancellationToken.None"/>), matching the <c>ServerHistorianOptions</c> warning.
/// </summary>
/// <returns>The deadline source, or <c>null</c> for an unbounded (non-positive) deadline.</returns>
private CancellationTokenSource? CreateDeadline() =>
HistoryReadDeadline > TimeSpan.Zero ? new CancellationTokenSource(HistoryReadDeadline) : null;
/// <summary>Awaits one work item under a semaphore permit, releasing it when the item completes.</summary>
/// <param name="gate">The concurrency-limiting semaphore.</param>
/// <param name="work">The per-node work item.</param>
private static async Task RunGatedAsync(SemaphoreSlim gate, Func<Task> work)
{
await gate.WaitAsync().ConfigureAwait(false);
try
{
await work().ConfigureAwait(false);
}
finally
{
gate.Release();
}
}
private async Task ServeNodeAsync(
NodeHandle handle,
IList<SdkHistoryReadResult> results,
IList<ServiceResult> errors,
Func<IHistorianDataSource, string, Task<HistorianRead>> read)
Func<IHistorianDataSource, string, CancellationToken, Task<HistorianRead>> read,
CancellationToken ct)
{
var idString = handle.NodeId.Identifier?.ToString();
if (idString is null || !TryGetHistorizedTagname(idString, out var tagname))
@@ -2054,9 +2269,9 @@ public sealed class OtOpcUaNodeManager : CustomNodeManager2
try
{
// HistoryRead is NOT invoked under the node-manager Lock (unlike OnWriteValue), so blocking
// on the async source here is safe and won't freeze the address space.
var sourceResult = read(HistorianDataSource, tagname!).GetAwaiter().GetResult();
// HistoryRead is NOT invoked under the node-manager Lock (unlike OnWriteValue), so awaiting
// the async source here is safe and won't freeze the address space.
var sourceResult = await read(HistorianDataSource, tagname!, ct).ConfigureAwait(false);
var historyData = ToHistoryData(sourceResult);
results[handle.Index] = new SdkHistoryReadResult
@@ -2071,13 +2286,20 @@ public sealed class OtOpcUaNodeManager : CustomNodeManager2
};
errors[handle.Index] = ServiceResult.Good;
}
catch (OperationCanceledException)
{
// The per-request deadline elapsed while this node was in flight — surface BadTimeout for
// THIS node only (a spec-conformant history-read status), ahead of the generic catch.
errors[handle.Index] = StatusCodes.BadTimeout;
results[handle.Index] = new SdkHistoryReadResult { StatusCode = StatusCodes.BadTimeout };
}
catch (Exception ex)
{
// One node's backend failure (throw / timeout / cancellation) must not throw out of the
// batch — surface a Bad status for THIS node only. This CustomNodeManager2 carries no
// ILogger (see ReportConditionEvent), so log through the SDK's static trace rather than
// swallowing silently. Utils.LogError is [Obsolete] in 1.5.378 (favours an ITelemetryContext
// this manager doesn't wire) — suppress the deprecation, matching the existing pattern.
// One node's backend failure (throw / timeout) must not throw out of the batch — surface a
// Bad status for THIS node only. This CustomNodeManager2 carries no ILogger (see
// ReportConditionEvent), so log through the SDK's static trace rather than swallowing silently.
// Utils.LogError is [Obsolete] in 1.5.378 (favours an ITelemetryContext this manager doesn't
// wire) — suppress the deprecation, matching the existing pattern.
#pragma warning disable CS0618 // Type or member is obsolete
Utils.LogError(ex, "OtOpcUaNodeManager: HistoryRead failed for node {0}", handle.NodeId);
#pragma warning restore CS0618
@@ -2115,7 +2337,7 @@ public sealed class OtOpcUaNodeManager : CustomNodeManager2
/// <param name="startTimeUtc">The request window's (inclusive) lower bound, used for a fresh read.</param>
/// <param name="endUtc">The (inclusive) upper bound of the read window; unchanged across pages.</param>
/// <param name="numValuesPerNode">The client's per-page cap; <c>0</c> means "all values, no paging".</param>
private void ServeRawPaged(
private async Task ServeRawPagedAsync(
NodeHandle handle,
ISession? session,
IList<HistoryReadValueId> nodesToRead,
@@ -2123,7 +2345,8 @@ public sealed class OtOpcUaNodeManager : CustomNodeManager2
IList<ServiceResult> errors,
DateTime startTimeUtc,
DateTime endUtc,
uint numValuesPerNode)
uint numValuesPerNode,
CancellationToken ct)
{
var inboundCp = nodesToRead[handle.Index].ContinuationPoint;
@@ -2165,10 +2388,10 @@ public sealed class OtOpcUaNodeManager : CustomNodeManager2
startUtc = startTimeUtc;
}
// HistoryRead is NOT under the node-manager Lock — block-bridging the async source is safe.
var sourceResult = HistorianDataSource
.ReadRawAsync(tagname, startUtc, endUtc, numValuesPerNode, CancellationToken.None)
.GetAwaiter().GetResult();
// HistoryRead is NOT under the node-manager Lock — awaiting the async source is safe.
var sourceResult = await HistorianDataSource
.ReadRawAsync(tagname, startUtc, endUtc, numValuesPerNode, ct)
.ConfigureAwait(false);
var backendFull = HistoryPaging.IsFullPage(sourceResult.Samples.Count, numValuesPerNode);
@@ -2198,9 +2421,9 @@ public sealed class OtOpcUaNodeManager : CustomNodeManager2
// Compute in uint so +1 can never wrap: Math.Max(1,…) also guards against a zero/negative
// config value slipping through (Validate() warns but does not block startup).
var overfetchCap = (uint)Math.Max(1, MaxTieClusterOverfetch) + 1u;
var cluster = HistorianDataSource
.ReadRawAsync(tagname, startUtc, startUtc, overfetchCap, CancellationToken.None)
.GetAwaiter().GetResult().Samples;
var cluster = (await HistorianDataSource
.ReadRawAsync(tagname, startUtc, startUtc, overfetchCap, ct)
.ConfigureAwait(false)).Samples;
// Absurd burst: more ties than we're willing to buffer in memory. Preserve today's loud-fail
// for that node rather than over-fetch an unbounded cluster; the operator's remedy is a
@@ -2272,6 +2495,12 @@ public sealed class OtOpcUaNodeManager : CustomNodeManager2
};
errors[handle.Index] = ServiceResult.Good;
}
catch (OperationCanceledException)
{
// The per-request deadline elapsed while paging this node — BadTimeout for THIS node only.
errors[handle.Index] = StatusCodes.BadTimeout;
results[handle.Index] = new SdkHistoryReadResult { StatusCode = StatusCodes.BadTimeout };
}
catch (Exception ex)
{
// One node's backend failure must not throw out of the batch — Bad for THIS node only.
@@ -23,6 +23,9 @@
<ItemGroup>
<InternalsVisibleTo Include="ZB.MOM.WW.OtOpcUa.OpcUaServer.Tests"/>
<!-- R2-08 (03/S2): the outage-sim integration test drives the internal static
OpcUaApplicationHost.HandleImpersonation against the real LDAP library. -->
<InternalsVisibleTo Include="ZB.MOM.WW.OtOpcUa.Host.IntegrationTests"/>
</ItemGroup>
</Project>
@@ -68,6 +68,32 @@ public sealed class ServerHistorianOptions
/// </summary>
public int MaxTieClusterOverfetch { get; init; } = 65536;
/// <summary>
/// Maximum number of historized nodes served CONCURRENTLY within one HistoryRead batch (arch-review
/// 03/S3). The SDK's <c>HistoryRead*</c> override surface is synchronous, so each arm block-bridges
/// once at its boundary; this bound turns the per-node fan-out inside that boundary from sequential
/// (worst case <c>N × CallTimeout</c>) into <c>⌈N / P⌉ × CallTimeout</c>. It also caps the gateway-load
/// multiplication a single client can command. Default 4.
/// </summary>
public int HistoryReadBatchParallelism { get; init; } = 4;
/// <summary>
/// Process-wide limit on the number of HistoryRead batches served CONCURRENTLY (arch-review 03/S3).
/// A flood of history clients degrades to <c>BadTooManyOperations</c> rather than exhausting the SDK
/// request-thread pool. Together with <see cref="HistoryReadBatchParallelism"/> this caps total
/// in-flight gateway reads at <c>MaxConcurrentHistoryReadBatches × HistoryReadBatchParallelism</c>
/// (default 16 × 4 = 64). Default 16.
/// </summary>
public int MaxConcurrentHistoryReadBatches { get; init; } = 16;
/// <summary>
/// Per-request wall-clock deadline for a HistoryRead batch (arch-review 03/S3). A single request can
/// no longer hold an SDK request thread longer than this regardless of node count; a node still
/// in flight at the deadline surfaces <c>BadTimeout</c>. Non-positive = unbounded (a
/// <see cref="Validate"/> warning). Default 60 seconds.
/// </summary>
public TimeSpan HistoryReadDeadline { get; init; } = TimeSpan.FromSeconds(60);
/// <summary>Returns operator-facing misconfiguration warnings for an <c>Enabled</c> historian
/// (empty when disabled or correctly configured). Pure — the registration logs each entry.
/// <para>
@@ -100,6 +126,14 @@ public sealed class ServerHistorianOptions
// so a zero/negative value is harmless (and noise-free) when the historian is disabled.
if (MaxTieClusterOverfetch <= 0)
warnings.Add($"ServerHistorian:MaxTieClusterOverfetch is {MaxTieClusterOverfetch} — must be > 0; HistoryRead-Raw cannot page within an oversized tie cluster and will surface BadHistoryOperationUnsupported for those reads.");
// S3 HistoryRead concurrency knobs — a non-positive value disables the corresponding bound, so warn
// (these gate the per-node fan-out / batch limiter that only run when a real data source is wired).
if (HistoryReadBatchParallelism <= 0)
warnings.Add($"ServerHistorian:HistoryReadBatchParallelism is {HistoryReadBatchParallelism} — must be > 0; HistoryRead falls back to serving each node sequentially.");
if (MaxConcurrentHistoryReadBatches <= 0)
warnings.Add($"ServerHistorian:MaxConcurrentHistoryReadBatches is {MaxConcurrentHistoryReadBatches} — must be > 0; the process-wide HistoryRead batch limiter is disabled.");
if (HistoryReadDeadline <= TimeSpan.Zero)
warnings.Add($"ServerHistorian:HistoryReadDeadline is {HistoryReadDeadline} — must be > 0; HistoryRead requests run unbounded (no per-request deadline).");
return warnings;
}
}
@@ -7,4 +7,15 @@ public sealed record LdapAuthResult(
string? Username,
IReadOnlyList<string> Groups,
IReadOnlyList<string> Roles,
string? Error);
string? Error)
{
/// <summary>
/// <c>true</c> when this failure is a SYSTEM-side directory fault (the directory is unreachable or
/// the service account is misconfigured) rather than a user-credential deny. Feeds the OPC UA
/// data-plane authenticator's outage circuit (03/S2): a run of consecutive system-side failures
/// opens the circuit and fails fast, whereas user-side denies never trip it. Always <c>false</c>
/// on success and on user-side denies; set only on the delegated real path for the library's
/// directory-unreachable bucket.
/// </summary>
public bool IsSystemFailure { get; init; }
}
@@ -83,6 +83,49 @@ public sealed class LdapOptions
/// </summary>
public Dictionary<string, string> GroupToRole { get; set; } = new(StringComparer.OrdinalIgnoreCase);
/// <summary>
/// Bounds (in milliseconds) the shared-library directory client's socket connect AND each
/// per-operation search inside a single bind flow. Projected into the library's
/// <see cref="LibLdapOptions.ConnectionTimeoutMs"/> via <see cref="ToLibraryOptions"/>. The
/// default (10000) matches the library default, so behaviour is unchanged unless configured.
/// This is the FIRST line of defence against a directory outage; the whole-flow
/// <see cref="AuthTimeoutMs"/> is the backstop (03/S2).
/// </summary>
public int ConnectionTimeoutMs { get; set; } = 10_000;
/// <summary>
/// Hard whole-flow wall-clock bound (in milliseconds) the OPC UA data-plane authenticator
/// (<c>LdapOpcUaUserAuthenticator</c>) applies across the ENTIRE authenticate flow — bind plus
/// role-map — so the OPC UA SDK's synchronous impersonation callback (raised under a
/// <c>SessionManager</c>-wide event lock) can never be wedged by a slow / hung directory.
/// Deliberately &gt; <see cref="ConnectionTimeoutMs"/> (default 15000) so a healthy-but-slow
/// full flow is not cut off while an outage is bounded by the library timeout first (03/S2).
/// </summary>
public int AuthTimeoutMs { get; set; } = 15_000;
/// <summary>
/// Maximum number of in-flight OPC UA authentication flows the data-plane authenticator admits
/// at once. A caller that cannot acquire a slot within its budget is denied "backend busy"
/// (local backpressure, not counted against the outage circuit). Bounds accumulation of
/// orphaned / timed-out bind tasks during an outage (03/S2). Default 8.
/// </summary>
public int MaxConcurrentAuthentications { get; set; } = 8;
/// <summary>
/// Number of CONSECUTIVE system-side failures (boundary timeout, directory-unreachable result,
/// or unexpected throw) after which the data-plane authenticator's outage circuit opens and
/// denies instantly for <see cref="OutageCooldownSeconds"/> without touching LDAP. Any success
/// or user-side (credential) deny resets the counter. <c>0</c> disables the circuit (03/S2).
/// Default 3.
/// </summary>
public int OutageFailureThreshold { get; set; } = 3;
/// <summary>
/// How long (in seconds) the outage circuit stays open before a half-open probe is allowed
/// through. Only meaningful when <see cref="OutageFailureThreshold"/> &gt; 0. Default 15 (03/S2).
/// </summary>
public int OutageCooldownSeconds { get; set; } = 15;
/// <summary>
/// Projects the wire fields onto the shared <c>ZB.MOM.WW.Auth.Ldap</c>
/// <see cref="LibLdapOptions"/> the directory client consumes. App-only concerns
@@ -104,5 +147,6 @@ public sealed class LdapOptions
UserNameAttribute = UserNameAttribute,
DisplayNameAttribute = DisplayNameAttribute,
GroupAttribute = GroupAttribute,
ConnectionTimeoutMs = ConnectionTimeoutMs,
};
}
@@ -113,7 +113,13 @@ public sealed class OtOpcUaLdapAuthService : ILdapAuthService
if (result.Succeeded)
return new(true, result.DisplayName, result.Username, result.Groups, [], null);
return new(false, null, username, [], [], FailureToError(result.Failure));
// ServiceAccountBindFailed is the library's directory-unreachable / service-account-misconfigured
// bucket (its enum has no dedicated DirectoryUnavailable) — flag it as system-side so the OPC UA
// authenticator's outage circuit can distinguish an outage from a user-credential deny (03/S2).
return new(false, null, username, [], [], FailureToError(result.Failure))
{
IsSystemFailure = result.Failure is LdapAuthFailure.ServiceAccountBindFailed,
};
}
/// <summary>Folds a structured library failure code into the app's user-facing error text.</summary>
@@ -0,0 +1,389 @@
using System.Diagnostics;
using System.Text;
using Microsoft.Extensions.DependencyInjection;
using Microsoft.Extensions.Logging.Abstractions;
using Microsoft.Extensions.Options;
using Opc.Ua;
using Opc.Ua.Server;
using Shouldly;
using Xunit;
using ZB.MOM.WW.Auth.Abstractions.Roles;
using ZB.MOM.WW.OtOpcUa.Host.OpcUa;
using ZB.MOM.WW.OtOpcUa.OpcUaServer;
using ZB.MOM.WW.OtOpcUa.Security.Ldap;
using LdapTransport = ZB.MOM.WW.Auth.Abstractions.Ldap.LdapTransport;
namespace ZB.MOM.WW.OtOpcUa.Host.IntegrationTests;
/// <summary>
/// Resilience guard for <see cref="LdapOpcUaUserAuthenticator"/> (arch-review 03/S2). The OPC UA SDK
/// raises the impersonation callback synchronously under a <c>SessionManager</c>-wide event lock, so a
/// slow / hung LDAP bind inside it does not merely stall one SDK thread — it serializes EVERY session
/// activation server-wide. These tests prove the authenticator enforces a hard wall-clock bound,
/// bounded in-flight concurrency, and a directory-outage circuit at its own boundary (the SDK callback
/// being irreducibly synchronous), all fail-closed. No real LDAP server is used — a delaying fake
/// reproduces the outage shape offline.
/// </summary>
public sealed class LdapAuthResilienceTests
{
/// <summary>
/// RED repro (03/S2): a backend that takes 5 s must NOT hold the caller for 5 s — the authenticator's
/// wall-clock bound denies (fail-closed) well before then. On current code (no bound) this fails: the
/// call returns Allow after the full 5 s.
/// </summary>
[Fact]
public async Task Authenticate_slow_backend_denies_within_AuthTimeout()
{
var ldap = new DelayingFakeLdap(
TimeSpan.FromSeconds(5),
new LdapAuthResult(true, "Alice", "alice", new[] { "configeditor" }, Array.Empty<string>(), null));
var mapper = new FakeMapper(g => g.ToArray());
var sut = Build(ldap, mapper, new LdapOptions { AuthTimeoutMs = 200 });
var sw = Stopwatch.StartNew();
var result = await sut
.AuthenticateUserNameAsync("alice", "secret", CancellationToken.None)
.WaitAsync(TimeSpan.FromSeconds(10), CancellationToken.None);
sw.Stop();
result.Success.ShouldBeFalse();
result.Error.ShouldNotBeNull();
result.Error!.ShouldContain("timed out");
sw.Elapsed.ShouldBeLessThan(TimeSpan.FromSeconds(1.5));
}
/// <summary>A fast, healthy backend is unaffected by the resilience wrapping — success + roles pass
/// through untouched (03/S2).</summary>
[Fact]
public async Task Authenticate_fast_success_unaffected()
{
var ldap = new DelayingFakeLdap(
TimeSpan.Zero,
new LdapAuthResult(true, "Alice", "alice", new[] { "configeditor" }, Array.Empty<string>(), null));
var mapper = new FakeMapper(g => g.Select(x => x == "configeditor" ? "Designer" : x).ToArray());
var sut = Build(ldap, mapper, new LdapOptions());
var result = await sut
.AuthenticateUserNameAsync("alice", "secret", CancellationToken.None)
.WaitAsync(TimeSpan.FromSeconds(10), CancellationToken.None);
result.Success.ShouldBeTrue();
result.DisplayName.ShouldBe("Alice");
result.Roles.ShouldBe(new[] { "Designer" });
}
/// <summary>With the concurrency cap saturated (a parked in-flight bind holding the only slot), a
/// second call is denied "backend busy" within its own budget — it never waits behind the parked
/// bind indefinitely (03/S2).</summary>
[Fact]
public async Task Authenticate_saturated_concurrency_denies_busy()
{
var ldap = new GatedFakeLdap(
new LdapAuthResult(true, "Alice", "alice", new[] { "configeditor" }, Array.Empty<string>(), null));
var mapper = new FakeMapper(g => g.ToArray());
// MaxConcurrentAuthentications = 1 → the second call cannot get a slot; small AuthTimeoutMs so the
// "busy" denial is quick and the first call's boundary also trips fast (its core stays parked,
// holding the slot).
var sut = Build(ldap, mapper, new LdapOptions { MaxConcurrentAuthentications = 1, AuthTimeoutMs = 300 });
// First call: acquires the only slot; its core parks in the gated fake.
var first = sut.AuthenticateUserNameAsync("alice", "secret", CancellationToken.None);
// Wait until the fake has actually been entered (slot acquired, core parked).
var spin = Stopwatch.StartNew();
while (ldap.Entered == 0 && spin.Elapsed < TimeSpan.FromSeconds(5))
await Task.Delay(10, TestContext.Current.CancellationToken);
ldap.Entered.ShouldBe(1);
// Second concurrent call: the slot is held by the parked core, so acquisition times out → busy.
var second = await sut
.AuthenticateUserNameAsync("bob", "secret", CancellationToken.None)
.WaitAsync(TimeSpan.FromSeconds(10), CancellationToken.None);
second.Success.ShouldBeFalse();
second.Error.ShouldNotBeNull();
second.Error!.ShouldContain("busy");
ldap.Entered.ShouldBe(1, "the second call must be denied without ever reaching the backend");
// Release the parked bind so the first call's core completes cleanly (no orphaned resources).
ldap.Release();
await first.WaitAsync(TimeSpan.FromSeconds(10), CancellationToken.None);
}
/// <summary>After <c>OutageFailureThreshold</c> consecutive system-side failures the circuit opens and
/// the next call is fast-denied "backend unavailable" WITHOUT touching LDAP (03/S2).</summary>
[Fact]
public async Task Consecutive_system_failures_open_circuit_and_fast_deny()
{
var ldap = new DelayingFakeLdap(TimeSpan.Zero, SystemFailure());
var mapper = new FakeMapper(g => g.ToArray());
var sut = Build(ldap, mapper,
new LdapOptions { OutageFailureThreshold = 3, OutageCooldownSeconds = 60, AuthTimeoutMs = 5000 });
for (var i = 0; i < 3; i++)
{
var r = await sut.AuthenticateUserNameAsync("u", "p", CancellationToken.None)
.WaitAsync(TimeSpan.FromSeconds(10), CancellationToken.None);
r.Success.ShouldBeFalse();
}
ldap.Calls.ShouldBe(3);
var sw = Stopwatch.StartNew();
var blocked = await sut.AuthenticateUserNameAsync("u", "p", CancellationToken.None)
.WaitAsync(TimeSpan.FromSeconds(10), CancellationToken.None);
sw.Stop();
blocked.Success.ShouldBeFalse();
blocked.Error.ShouldNotBeNull();
blocked.Error!.ShouldContain("unavailable");
ldap.Calls.ShouldBe(3, "an open circuit must fast-deny without reaching the backend");
sw.Elapsed.ShouldBeLessThan(TimeSpan.FromSeconds(1));
}
/// <summary>A user-side (credential) deny between system failures resets the streak, so the circuit
/// never opens — user denies are not directory evidence (03/S2).</summary>
[Fact]
public async Task User_side_deny_resets_the_system_failure_streak()
{
var ldap = new ScriptedFakeLdap(
SystemFailure(), SystemFailure(), UserDeny(), SystemFailure(), SystemFailure());
var mapper = new FakeMapper(g => g.ToArray());
var sut = Build(ldap, mapper,
new LdapOptions { OutageFailureThreshold = 3, OutageCooldownSeconds = 60, AuthTimeoutMs = 5000 });
// 5 scripted calls: sys, sys, user(reset), sys, sys → streak ends at 2 (< 3), circuit stays closed.
for (var i = 0; i < 5; i++)
{
await sut.AuthenticateUserNameAsync("u", "p", CancellationToken.None)
.WaitAsync(TimeSpan.FromSeconds(10), CancellationToken.None);
}
// A 6th call must still REACH the backend (circuit never opened).
await sut.AuthenticateUserNameAsync("u", "p", CancellationToken.None)
.WaitAsync(TimeSpan.FromSeconds(10), CancellationToken.None);
ldap.Calls.ShouldBe(6);
}
/// <summary>After the cooldown elapses the circuit half-opens: the next call probes the backend and a
/// success closes the circuit (03/S2).</summary>
[Fact]
public async Task Half_open_probe_after_cooldown_closes_on_success()
{
var ldap = new ScriptedFakeLdap(
SystemFailure(), SystemFailure(), SystemFailure(),
new LdapAuthResult(true, "Alice", "alice", new[] { "configeditor" }, Array.Empty<string>(), null));
var mapper = new FakeMapper(g => g.ToArray());
var sut = Build(ldap, mapper,
new LdapOptions { OutageFailureThreshold = 3, OutageCooldownSeconds = 1, AuthTimeoutMs = 5000 });
for (var i = 0; i < 3; i++)
await sut.AuthenticateUserNameAsync("u", "p", CancellationToken.None)
.WaitAsync(TimeSpan.FromSeconds(10), CancellationToken.None);
ldap.Calls.ShouldBe(3);
// Circuit open: this call is fast-denied without touching the backend.
var blocked = await sut.AuthenticateUserNameAsync("u", "p", CancellationToken.None)
.WaitAsync(TimeSpan.FromSeconds(10), CancellationToken.None);
blocked.Error!.ShouldContain("unavailable");
ldap.Calls.ShouldBe(3);
// Wait past the cooldown, then a probe reaches the backend and succeeds → circuit closes.
await Task.Delay(TimeSpan.FromMilliseconds(1200), TestContext.Current.CancellationToken);
var probe = await sut.AuthenticateUserNameAsync("alice", "secret", CancellationToken.None)
.WaitAsync(TimeSpan.FromSeconds(10), CancellationToken.None);
probe.Success.ShouldBeTrue();
ldap.Calls.ShouldBe(4, "the half-open probe reaches the backend");
}
/// <summary><c>OutageFailureThreshold = 0</c> disables the circuit — it never opens, every call reaches
/// the backend (03/S2).</summary>
[Fact]
public async Task Circuit_disabled_never_opens()
{
var ldap = new DelayingFakeLdap(TimeSpan.Zero, SystemFailure());
var mapper = new FakeMapper(g => g.ToArray());
var sut = Build(ldap, mapper,
new LdapOptions { OutageFailureThreshold = 0, OutageCooldownSeconds = 60, AuthTimeoutMs = 5000 });
for (var i = 0; i < 6; i++)
await sut.AuthenticateUserNameAsync("u", "p", CancellationToken.None)
.WaitAsync(TimeSpan.FromSeconds(10), CancellationToken.None);
ldap.Calls.ShouldBe(6, "a disabled circuit never fast-denies");
}
/// <summary>
/// Outage-sim (offline-safe): the REAL <see cref="OtOpcUaLdapAuthService"/> + real shared library
/// pointed at an UNROUTABLE host (a TCP blackhole — the same outage shape as "stop the GLAuth
/// container"), driven through the REAL internal <see cref="OpcUaApplicationHost.HandleImpersonation"/>
/// block-bridge. The activation must fail bounded (well under the SDK's default ~10-20 s stall) with
/// <see cref="StatusCodes.BadIdentityTokenRejected"/> and no granted identity (03/S2).
/// </summary>
[Fact]
public async Task Activation_with_unreachable_directory_fails_within_bound()
{
var options = new LdapOptions
{
Enabled = true,
DevStubMode = false,
Server = "10.255.255.1", // RFC-blackhole: SYN goes nowhere
Port = 3893,
Transport = LdapTransport.None,
AllowInsecure = true,
SearchBase = "dc=zb,dc=local",
ConnectionTimeoutMs = 500,
AuthTimeoutMs = 1000,
};
var ldap = new OtOpcUaLdapAuthService(Options.Create(options), NullLogger<OtOpcUaLdapAuthService>.Instance);
var mapper = new FakeMapper(g => g.ToArray());
var authenticator = new LdapOpcUaUserAuthenticator(
ldap, ScopeFactoryWith(mapper), Options.Create(options),
NullLogger<LdapOpcUaUserAuthenticator>.Instance);
var token = new UserNameIdentityToken { UserName = "alice", DecryptedPassword = Encoding.UTF8.GetBytes("secret") };
var policy = new UserTokenPolicy(UserTokenType.UserName) { PolicyId = "username_basic256sha256" };
var args = new ImpersonateEventArgs(token, policy, new EndpointDescription());
var sw = Stopwatch.StartNew();
await Task.Run(() => OpcUaApplicationHost.HandleImpersonation(
authenticator, args, NullLogger<object>.Instance), TestContext.Current.CancellationToken)
.WaitAsync(TimeSpan.FromSeconds(10), TestContext.Current.CancellationToken);
sw.Stop();
args.Identity.ShouldBeNull();
args.IdentityValidationError.ShouldNotBeNull();
args.IdentityValidationError.Code.ShouldBe(Opc.Ua.StatusCodes.BadIdentityTokenRejected);
sw.Elapsed.ShouldBeLessThan(TimeSpan.FromSeconds(3));
}
private static LdapAuthResult SystemFailure() =>
new(false, null, "u", Array.Empty<string>(), Array.Empty<string>(), "Unexpected authentication error")
{ IsSystemFailure = true };
private static LdapAuthResult UserDeny() =>
new(false, null, "u", Array.Empty<string>(), Array.Empty<string>(), "Invalid username or password")
{ IsSystemFailure = false };
private static LdapOpcUaUserAuthenticator Build(
ILdapAuthService ldap, IGroupRoleMapper<string> mapper, LdapOptions options) =>
new(ldap, ScopeFactoryWith(mapper), Options.Create(options),
NullLogger<LdapOpcUaUserAuthenticator>.Instance);
/// <summary>Builds an IServiceScopeFactory whose scopes resolve the supplied mapper.</summary>
private static IServiceScopeFactory ScopeFactoryWith(IGroupRoleMapper<string> mapper)
{
var services = new ServiceCollection();
services.AddScoped(_ => mapper);
return services.BuildServiceProvider().GetRequiredService<IServiceScopeFactory>();
}
/// <summary>
/// A fake <see cref="ILdapAuthService"/> that awaits a configurable delay (observing the token) before
/// returning a fixed result — the offline stand-in for a slow / hung directory. Counts invocations so
/// tests can assert the outage circuit denies WITHOUT reaching the backend.
/// </summary>
private sealed class DelayingFakeLdap : ILdapAuthService
{
private readonly TimeSpan _delay;
private readonly LdapAuthResult _then;
private int _calls;
/// <summary>Initializes the fake with a per-call delay and the result to return afterwards.</summary>
/// <param name="delay">How long each authenticate call blocks before returning.</param>
/// <param name="then">The result returned once the delay elapses.</param>
public DelayingFakeLdap(TimeSpan delay, LdapAuthResult then)
{
_delay = delay;
_then = then;
}
/// <summary>The number of times the backend was actually invoked.</summary>
public int Calls => Volatile.Read(ref _calls);
/// <summary>Awaits the configured delay, then returns the canned result.</summary>
/// <param name="username">The username (ignored).</param>
/// <param name="password">The password (ignored).</param>
/// <param name="ct">Cancellation token, observed by the delay.</param>
public async Task<LdapAuthResult> AuthenticateAsync(string username, string password, CancellationToken ct = default)
{
Interlocked.Increment(ref _calls);
await Task.Delay(_delay, ct).ConfigureAwait(false);
return _then;
}
}
/// <summary>
/// A fake <see cref="ILdapAuthService"/> that parks ASYNCHRONOUSLY (awaits a
/// <see cref="TaskCompletionSource"/>) until the test calls <see cref="Release"/> — used to hold an
/// in-flight bind so the concurrency cap is deterministically saturated WITHOUT blocking a thread.
/// Records how many calls have entered (past the slot acquire).
/// </summary>
private sealed class GatedFakeLdap : ILdapAuthService
{
private readonly TaskCompletionSource _park = new(TaskCreationOptions.RunContinuationsAsynchronously);
private readonly LdapAuthResult _then;
private int _entered;
/// <summary>Initializes the fake with the result to return once released.</summary>
/// <param name="then">The result returned once <see cref="Release"/> is called.</param>
public GatedFakeLdap(LdapAuthResult then) => _then = then;
/// <summary>Number of authenticate calls that reached the backend body.</summary>
public int Entered => Volatile.Read(ref _entered);
/// <summary>Releases the parked bind so its awaiting call can complete.</summary>
public void Release() => _park.TrySetResult();
/// <summary>Marks entry, awaits the parking gate, then returns the canned result.</summary>
/// <param name="username">The username (ignored).</param>
/// <param name="password">The password (ignored).</param>
/// <param name="ct">Cancellation token, observed while parked.</param>
public async Task<LdapAuthResult> AuthenticateAsync(string username, string password, CancellationToken ct = default)
{
Interlocked.Increment(ref _entered);
await _park.Task.WaitAsync(ct).ConfigureAwait(false);
return _then;
}
}
/// <summary>
/// A fake <see cref="ILdapAuthService"/> that returns a scripted sequence of results (one per call;
/// the last entry repeats once exhausted). Counts calls so the outage-circuit tests can assert the
/// backend was — or was NOT — reached on a given call.
/// </summary>
private sealed class ScriptedFakeLdap : ILdapAuthService
{
private readonly IReadOnlyList<LdapAuthResult> _script;
private int _calls;
/// <summary>Initializes the fake with the scripted results, returned in order.</summary>
/// <param name="script">The results to return, one per call; the last repeats once exhausted.</param>
public ScriptedFakeLdap(params LdapAuthResult[] script) => _script = script;
/// <summary>The number of times the backend was invoked.</summary>
public int Calls => Volatile.Read(ref _calls);
/// <summary>Returns the next scripted result (clamped to the last entry).</summary>
/// <param name="username">The username (ignored).</param>
/// <param name="password">The password (ignored).</param>
/// <param name="ct">The cancellation token (ignored).</param>
public Task<LdapAuthResult> AuthenticateAsync(string username, string password, CancellationToken ct = default)
{
var idx = Interlocked.Increment(ref _calls) - 1;
return Task.FromResult(_script[Math.Min(idx, _script.Count - 1)]);
}
}
/// <summary>Test group→role mapper driven by a delegate over the supplied groups.</summary>
private sealed class FakeMapper(Func<IReadOnlyList<string>, IReadOnlyList<string>> map) : IGroupRoleMapper<string>
{
/// <summary>Maps groups to roles via the configured delegate; Scope is always null.</summary>
/// <param name="groups">The LDAP groups to map.</param>
/// <param name="ct">The cancellation token.</param>
public Task<GroupRoleMapping<string>> MapAsync(IReadOnlyList<string> groups, CancellationToken ct)
=> Task.FromResult(new GroupRoleMapping<string>(map(groups), Scope: null));
}
}
@@ -1,5 +1,6 @@
using Microsoft.Extensions.DependencyInjection;
using Microsoft.Extensions.Logging.Abstractions;
using Microsoft.Extensions.Options;
using Shouldly;
using Xunit;
using ZB.MOM.WW.Auth.Abstractions.Roles;
@@ -26,7 +27,7 @@ public sealed class LdapOpcUaUserAuthenticatorTests
// group "configeditor" -> "Designer" (canonical, Task 1.7).
var ldap = new FakeLdap(new LdapAuthResult(true, "Alice", "alice", new[] { "configeditor" }, Array.Empty<string>(), null));
var mapper = new FakeMapper(g => g.Select(x => x == "configeditor" ? "Designer" : x).ToArray());
var sut = new LdapOpcUaUserAuthenticator(ldap, ScopeFactoryWith(mapper), NullLogger<LdapOpcUaUserAuthenticator>.Instance);
var sut = new LdapOpcUaUserAuthenticator(ldap, ScopeFactoryWith(mapper), Options.Create(new LdapOptions()), NullLogger<LdapOpcUaUserAuthenticator>.Instance);
var result = await sut.AuthenticateUserNameAsync("alice", "secret", CancellationToken.None);
@@ -44,7 +45,7 @@ public sealed class LdapOpcUaUserAuthenticatorTests
// nothing, so the union is exactly {Administrator}.
var ldap = new FakeLdap(new LdapAuthResult(true, "dev", "dev", new[] { "dev" }, new[] { "Administrator" }, null));
var mapper = new FakeMapper(_ => Array.Empty<string>());
var sut = new LdapOpcUaUserAuthenticator(ldap, ScopeFactoryWith(mapper), NullLogger<LdapOpcUaUserAuthenticator>.Instance);
var sut = new LdapOpcUaUserAuthenticator(ldap, ScopeFactoryWith(mapper), Options.Create(new LdapOptions()), NullLogger<LdapOpcUaUserAuthenticator>.Instance);
var result = await sut.AuthenticateUserNameAsync("dev", "anything", CancellationToken.None);
@@ -59,7 +60,7 @@ public sealed class LdapOpcUaUserAuthenticatorTests
{
var ldap = new FakeLdap(new LdapAuthResult(true, "dev", "dev", new[] { "dev" }, new[] { "Administrator" }, null));
var mapper = new FakeMapper(_ => throw new InvalidOperationException("DB down"));
var sut = new LdapOpcUaUserAuthenticator(ldap, ScopeFactoryWith(mapper), NullLogger<LdapOpcUaUserAuthenticator>.Instance);
var sut = new LdapOpcUaUserAuthenticator(ldap, ScopeFactoryWith(mapper), Options.Create(new LdapOptions()), NullLogger<LdapOpcUaUserAuthenticator>.Instance);
var result = await sut.AuthenticateUserNameAsync("dev", "anything", CancellationToken.None);
@@ -73,7 +74,7 @@ public sealed class LdapOpcUaUserAuthenticatorTests
{
var ldap = new FakeLdap(new LdapAuthResult(false, null, "mallory", Array.Empty<string>(), Array.Empty<string>(), "Invalid username or password"));
var mapper = new FakeMapper(g => g.ToArray());
var sut = new LdapOpcUaUserAuthenticator(ldap, ScopeFactoryWith(mapper), NullLogger<LdapOpcUaUserAuthenticator>.Instance);
var sut = new LdapOpcUaUserAuthenticator(ldap, ScopeFactoryWith(mapper), Options.Create(new LdapOptions()), NullLogger<LdapOpcUaUserAuthenticator>.Instance);
var result = await sut.AuthenticateUserNameAsync("mallory", "wrong", CancellationToken.None);
@@ -87,7 +88,7 @@ public sealed class LdapOpcUaUserAuthenticatorTests
{
var ldap = new FakeLdap(_ => throw new InvalidOperationException("LDAP unreachable"));
var mapper = new FakeMapper(g => g.ToArray());
var sut = new LdapOpcUaUserAuthenticator(ldap, ScopeFactoryWith(mapper), NullLogger<LdapOpcUaUserAuthenticator>.Instance);
var sut = new LdapOpcUaUserAuthenticator(ldap, ScopeFactoryWith(mapper), Options.Create(new LdapOptions()), NullLogger<LdapOpcUaUserAuthenticator>.Instance);
var result = await sut.AuthenticateUserNameAsync("anyone", "x", CancellationToken.None);
@@ -102,7 +103,7 @@ public sealed class LdapOpcUaUserAuthenticatorTests
{
var ldap = new FakeLdap(new LdapAuthResult(true, null, "alice", new[] { "ReadOnly" }, Array.Empty<string>(), null));
var mapper = new FakeMapper(g => g.ToArray());
var sut = new LdapOpcUaUserAuthenticator(ldap, ScopeFactoryWith(mapper), NullLogger<LdapOpcUaUserAuthenticator>.Instance);
var sut = new LdapOpcUaUserAuthenticator(ldap, ScopeFactoryWith(mapper), Options.Create(new LdapOptions()), NullLogger<LdapOpcUaUserAuthenticator>.Instance);
var result = await sut.AuthenticateUserNameAsync("alice", "x", CancellationToken.None);
@@ -61,6 +61,56 @@ public sealed class LdapOptionsBindingTests
options.DevStubMode.ShouldBeFalse();
}
/// <summary>
/// The five S2 resilience keys bind from the real <c>Security:Ldap</c> section (03/S2).
/// </summary>
[Fact]
public void Binding_reads_resilience_keys()
{
var configuration = new ConfigurationBuilder()
.AddInMemoryCollection(new Dictionary<string, string?>
{
["Security:Ldap:ConnectionTimeoutMs"] = "7000",
["Security:Ldap:AuthTimeoutMs"] = "12000",
["Security:Ldap:MaxConcurrentAuthentications"] = "5",
["Security:Ldap:OutageFailureThreshold"] = "4",
["Security:Ldap:OutageCooldownSeconds"] = "20",
})
.Build();
var options = configuration.GetSection(LdapOptions.SectionName).Get<LdapOptions>();
options.ShouldNotBeNull();
options.ConnectionTimeoutMs.ShouldBe(7000);
options.AuthTimeoutMs.ShouldBe(12000);
options.MaxConcurrentAuthentications.ShouldBe(5);
options.OutageFailureThreshold.ShouldBe(4);
options.OutageCooldownSeconds.ShouldBe(20);
}
/// <summary>The resilience keys carry the documented defaults when absent from config.</summary>
[Fact]
public void Resilience_key_defaults_hold_when_absent()
{
var options = new LdapOptions();
options.ConnectionTimeoutMs.ShouldBe(10000);
options.AuthTimeoutMs.ShouldBe(15000);
options.MaxConcurrentAuthentications.ShouldBe(8);
options.OutageFailureThreshold.ShouldBe(3);
options.OutageCooldownSeconds.ShouldBe(15);
}
/// <summary><c>ConnectionTimeoutMs</c> is projected onto the shared-library options so the
/// socket-connect + per-search timeout is bounded inside the directory client.</summary>
[Fact]
public void ToLibraryOptions_projects_ConnectionTimeoutMs()
{
var options = new LdapOptions { ConnectionTimeoutMs = 4321 };
options.ToLibraryOptions().ConnectionTimeoutMs.ShouldBe(4321);
}
}
/// <summary>
@@ -203,6 +203,95 @@ public sealed class LdapOptionsValidatorTests
result.Failures.ShouldContain("Ldap:SearchBase is required when LDAP login is enabled.");
}
/// <summary>Enabled with a non-positive <c>ConnectionTimeoutMs</c> fails validation (03/S2).</summary>
[Fact]
public void Enabled_with_nonpositive_ConnectionTimeoutMs_fails()
{
var options = new LdapOptions
{
Enabled = true,
Server = "ldap",
SearchBase = "dc=x",
Port = 389,
Transport = LdapTransport.Ldaps,
ConnectionTimeoutMs = 0,
};
Sut.Validate(null, options).Failed.ShouldBeTrue();
}
/// <summary>Enabled with a non-positive <c>AuthTimeoutMs</c> fails validation (03/S2).</summary>
[Fact]
public void Enabled_with_nonpositive_AuthTimeoutMs_fails()
{
var options = new LdapOptions
{
Enabled = true,
Server = "ldap",
SearchBase = "dc=x",
Port = 389,
Transport = LdapTransport.Ldaps,
AuthTimeoutMs = 0,
};
Sut.Validate(null, options).Failed.ShouldBeTrue();
}
/// <summary>Enabled with a non-positive <c>MaxConcurrentAuthentications</c> fails validation (03/S2).</summary>
[Fact]
public void Enabled_with_nonpositive_MaxConcurrentAuthentications_fails()
{
var options = new LdapOptions
{
Enabled = true,
Server = "ldap",
SearchBase = "dc=x",
Port = 389,
Transport = LdapTransport.Ldaps,
MaxConcurrentAuthentications = 0,
};
Sut.Validate(null, options).Failed.ShouldBeTrue();
}
/// <summary>A positive <c>OutageFailureThreshold</c> with a zero <c>OutageCooldownSeconds</c> fails —
/// an open circuit with no cooldown would probe on every call (03/S2).</summary>
[Fact]
public void OutageFailureThreshold_positive_with_zero_cooldown_fails()
{
var options = new LdapOptions
{
Enabled = true,
Server = "ldap",
SearchBase = "dc=x",
Port = 389,
Transport = LdapTransport.Ldaps,
OutageFailureThreshold = 3,
OutageCooldownSeconds = 0,
};
Sut.Validate(null, options).Failed.ShouldBeTrue();
}
/// <summary>A zero <c>OutageFailureThreshold</c> (circuit disabled) passes regardless of the
/// cooldown value (03/S2).</summary>
[Fact]
public void OutageFailureThreshold_zero_disables_circuit_and_passes()
{
var options = new LdapOptions
{
Enabled = true,
Server = "ldap",
SearchBase = "dc=x",
Port = 389,
Transport = LdapTransport.Ldaps,
OutageFailureThreshold = 0,
OutageCooldownSeconds = 0,
};
Sut.Validate(null, options).Succeeded.ShouldBeTrue();
}
/// <summary>Enabled with port 0 reports the port-range failure using the shared primitive wording.</summary>
[Fact]
public void Enabled_with_zero_port_fails()
@@ -0,0 +1,466 @@
using System.Diagnostics;
using Microsoft.Extensions.Logging.Abstractions;
using Opc.Ua;
using Opc.Ua.Server;
using Shouldly;
using Xunit;
using ZB.MOM.WW.OtOpcUa.Core.Abstractions;
using HistorianRead = ZB.MOM.WW.OtOpcUa.Core.Abstractions.HistoryReadResult;
using SdkHistoryReadResult = Opc.Ua.HistoryReadResult;
namespace ZB.MOM.WW.OtOpcUa.OpcUaServer.Tests;
/// <summary>
/// Arch-review 03/S3 — the node-manager's HistoryRead arms must serve a multi-node batch with BOUNDED
/// per-node parallelism, a per-request deadline, and a process-wide concurrent-batch limiter, rather
/// than block-bridging the gateway per node sequentially on an SDK request thread. Boots the same real
/// <see cref="OtOpcUaSdkServer"/> harness as <see cref="NodeManagerHistoryReadTests"/> and drives the
/// public <c>HistoryRead</c> with a delaying fake <see cref="IHistorianDataSource"/> that records the
/// maximum observed concurrent reads (deterministic — no raw-timing dependence for the primary
/// assertions).
/// </summary>
public sealed class NodeManagerHistoryReadConcurrencyTests : IDisposable
{
private static CancellationToken Ct => TestContext.Current.CancellationToken;
private readonly string _pkiRoot = Path.Combine(
Path.GetTempPath(),
$"otopcua-historyread-conc-{Guid.NewGuid():N}");
/// <summary>
/// RED repro (03/S3): a single Raw HistoryRead naming 8 historized nodes against a 150 ms-per-read
/// backend must serve at least two nodes concurrently. On the current sequential <c>foreach</c> the
/// max observed concurrency is pinned at 1, so this fails.
/// </summary>
[Fact]
public async Task Raw_batch_is_served_with_bounded_parallelism()
{
var (host, server) = await BootAsync();
var nm = server.NodeManager!;
var fake = new ConcurrencyTrackingHistorianDataSource(TimeSpan.FromMilliseconds(150));
nm.HistorianDataSource = fake;
var nodeIds = MaterializeHistorizedNodes(nm, count: 8);
var details = new ReadRawModifiedDetails
{
StartTime = DateTime.UtcNow.AddHours(-1),
EndTime = DateTime.UtcNow,
NumValuesPerNode = 10,
IsReadModified = false,
};
var sw = Stopwatch.StartNew();
await Task.Run(() => InvokeHistoryRead(server, nm, details, nodeIds), Ct)
.WaitAsync(TimeSpan.FromSeconds(15), Ct);
sw.Stop();
// Primary (deterministic): the batch was served with >1 node in flight at once.
fake.MaxObservedConcurrency.ShouldBeGreaterThanOrEqualTo(2);
await host.DisposeAsync();
}
/// <summary>
/// Parallelism is bounded ABOVE by <c>HistoryReadBatchParallelism</c>: with the property set to 3 and
/// 12 nodes in flight, no more than 3 reads are ever concurrent (03/S3).
/// </summary>
[Fact]
public async Task Parallelism_is_bounded_above()
{
var (host, server) = await BootAsync();
var nm = server.NodeManager!;
nm.HistoryReadBatchParallelism = 3;
var fake = new ConcurrencyTrackingHistorianDataSource(TimeSpan.FromMilliseconds(100));
nm.HistorianDataSource = fake;
var nodeIds = MaterializeHistorizedNodes(nm, count: 12);
var details = new ReadRawModifiedDetails
{
StartTime = DateTime.UtcNow.AddHours(-1),
EndTime = DateTime.UtcNow,
NumValuesPerNode = 10,
IsReadModified = false,
};
await Task.Run(() => InvokeHistoryRead(server, nm, details, nodeIds), Ct)
.WaitAsync(TimeSpan.FromSeconds(15), Ct);
fake.MaxObservedConcurrency.ShouldBeGreaterThanOrEqualTo(2);
fake.MaxObservedConcurrency.ShouldBeLessThanOrEqualTo(3);
await host.DisposeAsync();
}
/// <summary>
/// One node's backend throw does not poison a parallel batch: that node surfaces
/// <c>BadHistoryOperationUnsupported</c> while the other seven return Good (per-node isolation holds
/// under parallelism — 03/S3).
/// </summary>
[Fact]
public async Task One_node_failure_does_not_poison_a_parallel_batch()
{
var (host, server) = await BootAsync();
var nm = server.NodeManager!;
var src = DateTime.UtcNow.AddSeconds(-5);
var srv = DateTime.UtcNow;
// Node index 3's tagname throws; all others return a single good sample.
var fake = new PerTagnameFake(tagname => tagname == "WW.Tag3"
? throw new InvalidOperationException("backend boom for Tag3")
: new HistorianRead(new[] { new DataValueSnapshot(1.0f, StatusCodes.Good, src, srv) }, null));
nm.HistorianDataSource = fake;
var nodeIds = MaterializeHistorizedNodes(nm, count: 8);
var details = new ReadRawModifiedDetails
{
StartTime = DateTime.UtcNow.AddHours(-1),
EndTime = DateTime.UtcNow,
NumValuesPerNode = 10,
IsReadModified = false,
};
var (results, errors) = await Task.Run(() => InvokeHistoryRead(server, nm, details, nodeIds), Ct)
.WaitAsync(TimeSpan.FromSeconds(15), Ct);
for (var i = 0; i < 8; i++)
{
if (i == 3)
{
StatusCode.IsBad(errors[i].StatusCode).ShouldBeTrue();
errors[i].StatusCode.Code.ShouldBe(StatusCodes.BadHistoryOperationUnsupported);
}
else
{
errors[i].StatusCode.Code.ShouldBe(StatusCodes.Good);
results[i].StatusCode.Code.ShouldBe(StatusCodes.Good);
}
}
await host.DisposeAsync();
}
/// <summary>
/// A per-request deadline bounds a hung backend: reads that only complete on token cancellation
/// surface <c>BadTimeout</c> for every node and the call returns within the deadline (plus slack) —
/// it does not hang an SDK request thread indefinitely (03/S3).
/// </summary>
[Fact]
public async Task Deadline_bounds_a_hung_backend()
{
var (host, server) = await BootAsync();
var nm = server.NodeManager!;
nm.HistoryReadDeadline = TimeSpan.FromMilliseconds(500);
nm.HistorianDataSource = new HangingHistorianDataSource();
var nodeIds = MaterializeHistorizedNodes(nm, count: 4);
var details = new ReadRawModifiedDetails
{
StartTime = DateTime.UtcNow.AddHours(-1),
EndTime = DateTime.UtcNow,
NumValuesPerNode = 10,
IsReadModified = false,
};
var (_, errors) = await Task.Run(() => InvokeHistoryRead(server, nm, details, nodeIds), Ct)
.WaitAsync(TimeSpan.FromSeconds(10), Ct);
for (var i = 0; i < 4; i++)
errors[i].StatusCode.Code.ShouldBe(StatusCodes.BadTimeout);
await host.DisposeAsync();
}
/// <summary>
/// A saturated process-wide batch limiter degrades a second concurrent HistoryRead to
/// <c>BadTooManyOperations</c> (rather than exhausting the SDK request-thread pool); once the parked
/// batch completes the slot frees and a later read succeeds (03/S3).
/// </summary>
[Fact]
public async Task Saturated_limiter_rejects_with_BadTooManyOperations()
{
var (host, server) = await BootAsync();
var nm = server.NodeManager!;
nm.MaxConcurrentHistoryReadBatches = 1;
nm.HistoryReadDeadline = TimeSpan.FromMilliseconds(200);
var fake = new GatedRawFake();
nm.HistorianDataSource = fake;
var ids = MaterializeHistorizedNodes(nm, count: 2);
var details = new ReadRawModifiedDetails
{
StartTime = DateTime.UtcNow.AddHours(-1),
EndTime = DateTime.UtcNow,
NumValuesPerNode = 10,
IsReadModified = false,
};
// First batch parks in the gated fake, holding the single limiter permit.
var first = Task.Run(() => InvokeHistoryRead(server, nm, details, ids[0]), Ct);
var spin = Stopwatch.StartNew();
while (fake.Entered == 0 && spin.Elapsed < TimeSpan.FromSeconds(5))
await Task.Delay(10, Ct);
fake.Entered.ShouldBe(1);
// Second concurrent batch: the limiter is saturated → all handles BadTooManyOperations.
var (_, errors2) = await Task.Run(() => InvokeHistoryRead(server, nm, details, ids[1]), Ct)
.WaitAsync(TimeSpan.FromSeconds(5), Ct);
errors2[0].StatusCode.Code.ShouldBe(StatusCodes.BadTooManyOperations);
fake.Entered.ShouldBe(1, "the rejected batch must never reach the backend");
// Release the parked batch; the slot frees and a later read succeeds.
fake.Release();
await first.WaitAsync(TimeSpan.FromSeconds(10), Ct);
var (_, errors3) = await Task.Run(() => InvokeHistoryRead(server, nm, details, ids[1]), Ct)
.WaitAsync(TimeSpan.FromSeconds(10), Ct);
errors3[0].StatusCode.Code.ShouldBe(StatusCodes.Good);
await host.DisposeAsync();
}
/// <summary>Materialises <paramref name="count"/> historized Float variables and returns their NodeIds.</summary>
private static NodeId[] MaterializeHistorizedNodes(OtOpcUaNodeManager nm, int count)
{
var ids = new NodeId[count];
for (var i = 0; i < count; i++)
{
var key = $"eq/tag{i}";
nm.EnsureVariable(key, parentFolderNodeId: null, displayName: $"Tag{i}", dataType: "Float",
writable: false, historianTagname: $"WW.Tag{i}");
ids[i] = nm.TryGetVariable(key)!.NodeId;
}
return ids;
}
/// <summary>Issues one HistoryRead batch for all supplied node ids in order.</summary>
private static (IList<SdkHistoryReadResult> Results, IList<ServiceResult> Errors) InvokeHistoryRead(
OtOpcUaSdkServer server, OtOpcUaNodeManager nm, HistoryReadDetails details, params NodeId[] nodeIds)
{
var context = new OperationContext(
new RequestHeader(), secureChannelContext: null, RequestType.HistoryRead, identity: null);
var nodesToRead = nodeIds.Select(id => new HistoryReadValueId { NodeId = id }).ToList();
var results = Enumerable.Repeat<SdkHistoryReadResult>(null!, nodeIds.Length).ToList();
var errors = Enumerable.Repeat<ServiceResult>(null!, nodeIds.Length).ToList();
nm.HistoryRead(
context,
details,
TimestampsToReturn.Both,
releaseContinuationPoints: false,
nodesToRead,
results,
errors);
return (results, errors);
}
/// <summary>
/// A delaying fake historian source that tracks the maximum number of reads in flight at once
/// (via interlocked live-counter) so a test can prove per-node parallelism deterministically.
/// </summary>
private sealed class ConcurrencyTrackingHistorianDataSource(TimeSpan delay) : IHistorianDataSource
{
private int _current;
private int _max;
/// <summary>The highest number of concurrently in-flight reads observed.</summary>
public int MaxObservedConcurrency => Volatile.Read(ref _max);
private async Task<HistorianRead> TrackAsync(CancellationToken ct)
{
var now = Interlocked.Increment(ref _current);
UpdateMax(now);
try
{
await Task.Delay(delay, ct).ConfigureAwait(false);
}
finally
{
Interlocked.Decrement(ref _current);
}
return new HistorianRead(
new[] { new DataValueSnapshot(1.0f, StatusCodes.Good, DateTime.UtcNow, DateTime.UtcNow) }, null);
}
private void UpdateMax(int observed)
{
int cur;
while ((cur = Volatile.Read(ref _max)) < observed &&
Interlocked.CompareExchange(ref _max, observed, cur) != cur)
{
}
}
public Task<HistorianRead> ReadRawAsync(
string fullReference, DateTime startUtc, DateTime endUtc, uint maxValuesPerNode,
CancellationToken cancellationToken) => TrackAsync(cancellationToken);
public Task<HistorianRead> ReadProcessedAsync(
string fullReference, DateTime startUtc, DateTime endUtc, TimeSpan interval,
HistoryAggregateType aggregate, CancellationToken cancellationToken) => TrackAsync(cancellationToken);
public Task<HistorianRead> ReadAtTimeAsync(
string fullReference, IReadOnlyList<DateTime> timestampsUtc, CancellationToken cancellationToken) =>
TrackAsync(cancellationToken);
public Task<HistoricalEventsResult> ReadEventsAsync(
string? sourceName, DateTime startUtc, DateTime endUtc, int maxEvents,
CancellationToken cancellationToken) =>
Task.FromResult(new HistoricalEventsResult(Array.Empty<HistoricalEvent>(), null));
public HistorianHealthSnapshot GetHealthSnapshot() => NullHistorianDataSource.Instance.GetHealthSnapshot();
public void Dispose()
{
}
}
/// <summary>A fake whose reads complete ONLY on token cancellation — used to prove the per-request
/// deadline surfaces <c>BadTimeout</c> rather than hanging.</summary>
private sealed class HangingHistorianDataSource : IHistorianDataSource
{
private static async Task<HistorianRead> HangAsync(CancellationToken ct)
{
await Task.Delay(Timeout.Infinite, ct).ConfigureAwait(false);
return new HistorianRead(Array.Empty<DataValueSnapshot>(), null);
}
public Task<HistorianRead> ReadRawAsync(
string fullReference, DateTime startUtc, DateTime endUtc, uint maxValuesPerNode,
CancellationToken cancellationToken) => HangAsync(cancellationToken);
public Task<HistorianRead> ReadProcessedAsync(
string fullReference, DateTime startUtc, DateTime endUtc, TimeSpan interval,
HistoryAggregateType aggregate, CancellationToken cancellationToken) => HangAsync(cancellationToken);
public Task<HistorianRead> ReadAtTimeAsync(
string fullReference, IReadOnlyList<DateTime> timestampsUtc, CancellationToken cancellationToken) =>
HangAsync(cancellationToken);
public async Task<HistoricalEventsResult> ReadEventsAsync(
string? sourceName, DateTime startUtc, DateTime endUtc, int maxEvents,
CancellationToken cancellationToken)
{
await Task.Delay(Timeout.Infinite, cancellationToken).ConfigureAwait(false);
return new HistoricalEventsResult(Array.Empty<HistoricalEvent>(), null);
}
public HistorianHealthSnapshot GetHealthSnapshot() => NullHistorianDataSource.Instance.GetHealthSnapshot();
public void Dispose() { }
}
/// <summary>A fake whose raw read parks (awaits a <see cref="TaskCompletionSource"/>) until the test
/// releases it — used to hold the process-wide batch limiter permit while a second batch is issued.</summary>
private sealed class GatedRawFake : IHistorianDataSource
{
private readonly TaskCompletionSource _gate = new(TaskCreationOptions.RunContinuationsAsynchronously);
private int _entered;
/// <summary>Number of raw reads that reached the backend body.</summary>
public int Entered => Volatile.Read(ref _entered);
/// <summary>Releases the parked read so its awaiting batch can complete.</summary>
public void Release() => _gate.TrySetResult();
public async Task<HistorianRead> ReadRawAsync(
string fullReference, DateTime startUtc, DateTime endUtc, uint maxValuesPerNode,
CancellationToken cancellationToken)
{
Interlocked.Increment(ref _entered);
// Deliberately ignore the per-request deadline token: the first batch must keep HOLDING the
// single limiter permit until the test releases it, so the second batch deterministically hits
// a saturated limiter (rather than the first batch's own deadline freeing the slot early).
await _gate.Task.ConfigureAwait(false);
return new HistorianRead(
new[] { new DataValueSnapshot(1.0f, StatusCodes.Good, DateTime.UtcNow, DateTime.UtcNow) }, null);
}
public Task<HistorianRead> ReadProcessedAsync(
string fullReference, DateTime startUtc, DateTime endUtc, TimeSpan interval,
HistoryAggregateType aggregate, CancellationToken cancellationToken) =>
NullHistorianDataSource.Instance.ReadProcessedAsync(fullReference, startUtc, endUtc, interval, aggregate, cancellationToken);
public Task<HistorianRead> ReadAtTimeAsync(
string fullReference, IReadOnlyList<DateTime> timestampsUtc, CancellationToken cancellationToken) =>
NullHistorianDataSource.Instance.ReadAtTimeAsync(fullReference, timestampsUtc, cancellationToken);
public Task<HistoricalEventsResult> ReadEventsAsync(
string? sourceName, DateTime startUtc, DateTime endUtc, int maxEvents,
CancellationToken cancellationToken) =>
NullHistorianDataSource.Instance.ReadEventsAsync(sourceName, startUtc, endUtc, maxEvents, cancellationToken);
public HistorianHealthSnapshot GetHealthSnapshot() => NullHistorianDataSource.Instance.GetHealthSnapshot();
public void Dispose() { }
}
/// <summary>A per-tagname fake that delegates each raw read to a caller-supplied func (which may throw
/// to simulate a per-node backend failure). Processed/AtTime/Events delegate to the null source.</summary>
private sealed class PerTagnameFake(Func<string, HistorianRead> rawHandler) : IHistorianDataSource
{
public Task<HistorianRead> ReadRawAsync(
string fullReference, DateTime startUtc, DateTime endUtc, uint maxValuesPerNode,
CancellationToken cancellationToken) => Task.FromResult(rawHandler(fullReference));
public Task<HistorianRead> ReadProcessedAsync(
string fullReference, DateTime startUtc, DateTime endUtc, TimeSpan interval,
HistoryAggregateType aggregate, CancellationToken cancellationToken) =>
NullHistorianDataSource.Instance.ReadProcessedAsync(fullReference, startUtc, endUtc, interval, aggregate, cancellationToken);
public Task<HistorianRead> ReadAtTimeAsync(
string fullReference, IReadOnlyList<DateTime> timestampsUtc, CancellationToken cancellationToken) =>
NullHistorianDataSource.Instance.ReadAtTimeAsync(fullReference, timestampsUtc, cancellationToken);
public Task<HistoricalEventsResult> ReadEventsAsync(
string? sourceName, DateTime startUtc, DateTime endUtc, int maxEvents,
CancellationToken cancellationToken) =>
NullHistorianDataSource.Instance.ReadEventsAsync(sourceName, startUtc, endUtc, maxEvents, cancellationToken);
public HistorianHealthSnapshot GetHealthSnapshot() => NullHistorianDataSource.Instance.GetHealthSnapshot();
public void Dispose() { }
}
private async Task<(OpcUaApplicationHost Host, OtOpcUaSdkServer Server)> BootAsync()
{
var host = new OpcUaApplicationHost(
new OpcUaApplicationHostOptions
{
ApplicationName = "OtOpcUa.HistoryReadConcTest",
ApplicationUri = $"urn:OtOpcUa.HistoryReadConcTest:{Guid.NewGuid():N}",
OpcUaPort = AllocateFreePort(),
PublicHostname = "localhost",
PkiStoreRoot = _pkiRoot,
},
NullLogger<OpcUaApplicationHost>.Instance);
var server = new OtOpcUaSdkServer();
await host.StartAsync(server, Ct);
return (host, server);
}
private static int AllocateFreePort()
{
using var listener = new System.Net.Sockets.TcpListener(System.Net.IPAddress.Loopback, 0);
listener.Start();
var port = ((System.Net.IPEndPoint)listener.LocalEndpoint).Port;
listener.Stop();
return port;
}
/// <summary>Cleans up the PKI root directory.</summary>
public void Dispose()
{
if (Directory.Exists(_pkiRoot))
{
try { Directory.Delete(_pkiRoot, recursive: true); }
catch { /* best-effort cleanup */ }
}
}
}
@@ -50,4 +50,55 @@ public sealed class ServerHistorianOptionsTests
.Validate();
Assert.Contains(w, m => m.Contains("MaxTieClusterOverfetch"));
}
// ── S3 HistoryRead concurrency knobs (arch-review 03/S3) ────────────────────────────────────
[Fact]
public void HistoryRead_knob_defaults()
{
var o = new ServerHistorianOptions();
Assert.Equal(4, o.HistoryReadBatchParallelism);
Assert.Equal(16, o.MaxConcurrentHistoryReadBatches);
Assert.Equal(TimeSpan.FromSeconds(60), o.HistoryReadDeadline);
}
[Fact]
public void Enabled_with_nonpositive_HistoryReadBatchParallelism_warns()
{
var w = new ServerHistorianOptions
{ Enabled = true, Endpoint = "https://h:5222", ApiKey = "histgw_x_y", HistoryReadBatchParallelism = 0 }
.Validate();
Assert.Contains(w, m => m.Contains("HistoryReadBatchParallelism"));
}
[Fact]
public void Enabled_with_nonpositive_MaxConcurrentHistoryReadBatches_warns()
{
var w = new ServerHistorianOptions
{ Enabled = true, Endpoint = "https://h:5222", ApiKey = "histgw_x_y", MaxConcurrentHistoryReadBatches = 0 }
.Validate();
Assert.Contains(w, m => m.Contains("MaxConcurrentHistoryReadBatches"));
}
[Fact]
public void Enabled_with_nonpositive_HistoryReadDeadline_warns()
{
var w = new ServerHistorianOptions
{ Enabled = true, Endpoint = "https://h:5222", ApiKey = "histgw_x_y", HistoryReadDeadline = TimeSpan.Zero }
.Validate();
Assert.Contains(w, m => m.Contains("HistoryReadDeadline"));
}
[Fact]
public void Disabled_with_nonpositive_HistoryRead_knobs_is_silent()
{
var w = new ServerHistorianOptions
{
Enabled = false,
HistoryReadBatchParallelism = 0,
MaxConcurrentHistoryReadBatches = 0,
HistoryReadDeadline = TimeSpan.Zero,
}.Validate();
Assert.Empty(w);
}
}
@@ -121,6 +121,56 @@ public sealed class OtOpcUaLdapAuthServiceTests
inner.Called.ShouldBeFalse();
}
/// <summary>A system-side library failure (<c>ServiceAccountBindFailed</c> — the directory-unreachable
/// bucket) adapts to <c>IsSystemFailure == true</c> so the OPC UA authenticator's outage circuit can
/// distinguish it from a user-credential deny (03/S2).</summary>
[Fact]
public async Task ServiceAccountBindFailed_adapts_to_IsSystemFailure_true()
{
var inner = new RecordingLibService(LibLdapAuthResult.Fail(LdapAuthFailure.ServiceAccountBindFailed));
var sut = Build(
new LdapOptions { Enabled = true, DevStubMode = false, Transport = LdapTransport.Ldaps },
inner);
var result = await sut.AuthenticateAsync("alice", "secret", CancellationToken.None);
result.Success.ShouldBeFalse();
result.IsSystemFailure.ShouldBeTrue();
}
/// <summary>User-side library failures and success adapt to <c>IsSystemFailure == false</c> — they must
/// NOT trip the outage circuit (03/S2).</summary>
[Theory]
[InlineData(LdapAuthFailure.BadCredentials)]
[InlineData(LdapAuthFailure.UserNotFound)]
public async Task User_side_failures_adapt_to_IsSystemFailure_false(LdapAuthFailure failure)
{
var inner = new RecordingLibService(LibLdapAuthResult.Fail(failure));
var sut = Build(
new LdapOptions { Enabled = true, DevStubMode = false, Transport = LdapTransport.Ldaps },
inner);
var result = await sut.AuthenticateAsync("alice", "wrong", CancellationToken.None);
result.Success.ShouldBeFalse();
result.IsSystemFailure.ShouldBeFalse();
}
/// <summary>A successful bind is never a system failure.</summary>
[Fact]
public async Task Success_is_not_a_system_failure()
{
var inner = new RecordingLibService(LibLdapAuthResult.Success("alice", "Alice", new[] { "ReadOnly" }));
var sut = Build(
new LdapOptions { Enabled = true, DevStubMode = false, Transport = LdapTransport.Ldaps },
inner);
var result = await sut.AuthenticateAsync("alice", "secret", CancellationToken.None);
result.Success.ShouldBeTrue();
result.IsSystemFailure.ShouldBeFalse();
}
/// <summary>Records whether the library service was invoked and returns a canned result.</summary>
private sealed class RecordingLibService(LibLdapAuthResult result) : LibILdapAuthService
{