test(host): unreachable-directory activation fails bounded; document the SDK impersonation threading contract (03/S2)

archreview/plans/R2-08-ldap-historyread-async-plan.md.tasks.json

@@ -6,7 +6,7 @@
     { "id": "R2-08-T3", "subject": "LdapAuthResult.IsSystemFailure seam + OtOpcUaLdapAuthService.Adapt mapping (ServiceAccountBindFailed => system-side)", "status": "completed", "blockedBy": [] },
     { "id": "R2-08-T4", "subject": "LdapOpcUaUserAuthenticator: WaitAsync wall-clock bound + SemaphoreSlim concurrency cap with release-on-core-completion (turns T1 green)", "status": "completed", "blockedBy": ["R2-08-T1", "R2-08-T2"] },
     { "id": "R2-08-T5", "subject": "Directory-outage circuit: fail-fast deny after N consecutive system-side failures, half-open after cooldown", "status": "completed", "blockedBy": ["R2-08-T3", "R2-08-T4"] },
-    { "id": "R2-08-T6", "subject": "Outage-sim integration test (real library, unroutable host, via internal HandleImpersonation) + SDK m_eventLock contract doc + docs/security.md", "status": "pending", "blockedBy": ["R2-08-T4"] },
+    { "id": "R2-08-T6", "subject": "Outage-sim integration test (real library, unroutable host, via internal HandleImpersonation) + SDK m_eventLock contract doc + docs/security.md", "status": "completed", "blockedBy": ["R2-08-T4"] },
     { "id": "R2-08-T7", "subject": "S3 sequential-serving repro test (RED): MaxObservedConcurrency >= 2 on an 8-node Raw batch — fails on current code", "status": "pending", "blockedBy": [] },
     { "id": "R2-08-T8", "subject": "ServerHistorianOptions: HistoryReadBatchParallelism/MaxConcurrentHistoryReadBatches/HistoryReadDeadline + Validate warnings + node-manager properties + hosted-service wiring", "status": "pending", "blockedBy": [] },
     { "id": "R2-08-T9", "subject": "Async-ify ServeNode + Processed/AtTime arms (ct threading, OCE=>BadTimeout, behavior-neutral)", "status": "pending", "blockedBy": ["R2-08-T7", "R2-08-T8"] },

docs/security.md

@@ -162,6 +162,20 @@ LDAP is configured under the `Security:Ldap` section (bound to `LdapOptions`, `s
 
 `GroupToRole` maps LDAP group names → roles (case-insensitive); a user gets every role whose source group is in their membership. The values are the canonical control-plane role strings (`Viewer` / `Designer` / `Administrator`, plus the appsettings-only `Operator` for the `DriverOperator` policy); the same map also supplies data-plane role strings (`ReadOnly`, `WriteOperate`, `WriteTune`, `WriteConfigure`, `AlarmAck`) — see [Role grant source (data-plane)](#role-grant-source-data-plane) below. `UserNameAttribute: "sAMAccountName"` is the critical AD override — the GLAuth dev default is `cn`, which is not how AD users are looked up; use `userPrincipalName` instead if operators log in with `user@corp.example.com` form. `LdapOptionsValidator` (`src/Server/ZB.MOM.WW.OtOpcUa.Host/Configuration/LdapOptionsValidator.cs`) fails startup when `Transport = None` and `AllowInsecure = false` on a real-LDAP (non-DevStub) config.
 
+### LDAP resilience — timeout, concurrency, outage circuit (arch-review 03/S2)
+
+The OPC UA SDK raises its impersonation (UserName) callback **synchronously**, inside a `SessionManager`-wide event lock (verified against `OPCFoundation.NetStandard.Opc.Ua.Server` 1.5.378). A slow or hung LDAP bind inside that callback therefore does not stall one SDK thread — it **serializes every session activation server-wide** (Anonymous and X509 included). Since the callback is irreducibly synchronous, `LdapOpcUaUserAuthenticator` bounds it at its own boundary with five `Security:Ldap` keys (all validated at startup by `LdapOptionsValidator` when `Enabled && !DevStubMode`):
+
+| Key | Default | Bounds |
+|---|---|---|
+| `ConnectionTimeoutMs` | `10000` | Projected into the shared library's `ConnectionTimeoutMs` — bounds the socket connect **and** each per-operation search. First line of defence against an outage. |
+| `AuthTimeoutMs` | `15000` | Hard whole-flow wall-clock bound (bind **and** role-map) enforced at the authenticator boundary; a slow/hung flow denies **"Authentication timed out"** (fail-closed) and releases the SDK thread. Deliberately `> ConnectionTimeoutMs` so a healthy-but-slow flow is not cut off. |
+| `MaxConcurrentAuthentications` | `8` | Max in-flight authentications; excess denies **"Authentication backend busy"** (local backpressure — does **not** feed the circuit). Bounds accumulation of orphaned/timed-out binds during an outage. |
+| `OutageFailureThreshold` | `3` | Consecutive **system-side** failures (boundary timeout, directory-unreachable, or unexpected throw) that open the outage circuit. Any success or user-credential deny resets the streak. `0` disables the circuit. |
+| `OutageCooldownSeconds` | `15` | How long the open circuit fast-denies **"Authentication backend unavailable"** without touching LDAP; after cooldown the next call half-open-probes, and one more system failure re-opens immediately. |
+
+Defaults are behaviour-neutral on a healthy directory. All new paths are **fail-closed** (they deny). A user-credential deny (wrong password) is fast when the directory is up and never trips the circuit; only a directory *outage* opens it.
+
 ---
 
 ## Data-Plane Authorization

src/Server/ZB.MOM.WW.OtOpcUa.OpcUaServer/OpcUaApplicationHost.cs

@@ -235,6 +235,19 @@ public sealed class OpcUaApplicationHost : IAsyncDisposable
     /// the full SDK. Side-effects are confined to mutating <see cref="ImpersonateEventArgs"/>
     /// and logging.
     /// </summary>
+    /// <remarks>
+    /// <b>SDK threading contract (verified against OPCFoundation.NetStandard.Opc.Ua.Server 1.5.378).</b>
+    /// The SDK raises this handler SYNCHRONOUSLY — it is a plain <c>void</c> delegate, and there is no
+    /// async impersonation callback in this SDK version. Worse, <c>SessionManager.ActivateSessionAsync</c>
+    /// invokes it while holding a single <c>SessionManager</c>-wide event lock (<c>m_eventLock</c>), so a
+    /// handler that blocks does not merely stall one SDK request thread — it SERIALIZES every session
+    /// activation server-wide (Anonymous and X509 included, since even those paths must take the lock to
+    /// raise the handler). Consequently the <see cref="IOpcUaUserAuthenticator"/> block-bridge below
+    /// (<c>GetAwaiter().GetResult()</c> under <see cref="CancellationToken.None"/>) is irreducible: the
+    /// callback must complete inline. Because the SDK gives no timeout, the hard wall-clock bound + bounded
+    /// concurrency + directory-outage circuit that keep a slow/hung directory from wedging the SDK live
+    /// inside the authenticator (<c>LdapOpcUaUserAuthenticator</c>, arch-review 03/S2) — NOT here.
+    /// </remarks>
     /// <param name="authenticator">The user authenticator to validate credentials.</param>
     /// <param name="args">The impersonation event arguments to process.</param>
     /// <param name="logger">The logger for diagnostic output.</param>

src/Server/ZB.MOM.WW.OtOpcUa.OpcUaServer/ZB.MOM.WW.OtOpcUa.OpcUaServer.csproj

@@ -23,6 +23,9 @@
 
     <ItemGroup>
         <InternalsVisibleTo Include="ZB.MOM.WW.OtOpcUa.OpcUaServer.Tests"/>
+        <!-- R2-08 (03/S2): the outage-sim integration test drives the internal static
+             OpcUaApplicationHost.HandleImpersonation against the real LDAP library. -->
+        <InternalsVisibleTo Include="ZB.MOM.WW.OtOpcUa.Host.IntegrationTests"/>
     </ItemGroup>
 
 </Project>

tests/Server/ZB.MOM.WW.OtOpcUa.Host.IntegrationTests/LdapAuthResilienceTests.cs

@@ -1,12 +1,17 @@
 using System.Diagnostics;
+using System.Text;
 using Microsoft.Extensions.DependencyInjection;
 using Microsoft.Extensions.Logging.Abstractions;
 using Microsoft.Extensions.Options;
+using Opc.Ua;
+using Opc.Ua.Server;
 using Shouldly;
 using Xunit;
 using ZB.MOM.WW.Auth.Abstractions.Roles;
 using ZB.MOM.WW.OtOpcUa.Host.OpcUa;
+using ZB.MOM.WW.OtOpcUa.OpcUaServer;
 using ZB.MOM.WW.OtOpcUa.Security.Ldap;
+using LdapTransport = ZB.MOM.WW.Auth.Abstractions.Ldap.LdapTransport;
 
 namespace ZB.MOM.WW.OtOpcUa.Host.IntegrationTests;
 
@@ -209,6 +214,51 @@ public sealed class LdapAuthResilienceTests
         ldap.Calls.ShouldBe(6, "a disabled circuit never fast-denies");
     }
 
+    /// <summary>
+    /// Outage-sim (offline-safe): the REAL <see cref="OtOpcUaLdapAuthService"/> + real shared library
+    /// pointed at an UNROUTABLE host (a TCP blackhole — the same outage shape as "stop the GLAuth
+    /// container"), driven through the REAL internal <see cref="OpcUaApplicationHost.HandleImpersonation"/>
+    /// block-bridge. The activation must fail bounded (well under the SDK's default ~10-20 s stall) with
+    /// <see cref="StatusCodes.BadIdentityTokenRejected"/> and no granted identity (03/S2).
+    /// </summary>
+    [Fact]
+    public async Task Activation_with_unreachable_directory_fails_within_bound()
+    {
+        var options = new LdapOptions
+        {
+            Enabled = true,
+            DevStubMode = false,
+            Server = "10.255.255.1", // RFC-blackhole: SYN goes nowhere
+            Port = 3893,
+            Transport = LdapTransport.None,
+            AllowInsecure = true,
+            SearchBase = "dc=zb,dc=local",
+            ConnectionTimeoutMs = 500,
+            AuthTimeoutMs = 1000,
+        };
+
+        var ldap = new OtOpcUaLdapAuthService(Options.Create(options), NullLogger<OtOpcUaLdapAuthService>.Instance);
+        var mapper = new FakeMapper(g => g.ToArray());
+        var authenticator = new LdapOpcUaUserAuthenticator(
+            ldap, ScopeFactoryWith(mapper), Options.Create(options),
+            NullLogger<LdapOpcUaUserAuthenticator>.Instance);
+
+        var token = new UserNameIdentityToken { UserName = "alice", DecryptedPassword = Encoding.UTF8.GetBytes("secret") };
+        var policy = new UserTokenPolicy(UserTokenType.UserName) { PolicyId = "username_basic256sha256" };
+        var args = new ImpersonateEventArgs(token, policy, new EndpointDescription());
+
+        var sw = Stopwatch.StartNew();
+        await Task.Run(() => OpcUaApplicationHost.HandleImpersonation(
+                authenticator, args, NullLogger<object>.Instance), TestContext.Current.CancellationToken)
+            .WaitAsync(TimeSpan.FromSeconds(10), TestContext.Current.CancellationToken);
+        sw.Stop();
+
+        args.Identity.ShouldBeNull();
+        args.IdentityValidationError.ShouldNotBeNull();
+        args.IdentityValidationError.Code.ShouldBe(Opc.Ua.StatusCodes.BadIdentityTokenRejected);
+        sw.Elapsed.ShouldBeLessThan(TimeSpan.FromSeconds(3));
+    }
+
     private static LdapAuthResult SystemFailure() =>
         new(false, null, "u", Array.Empty<string>(), Array.Empty<string>(), "Unexpected authentication error")
         { IsSystemFailure = true };