test(integration): key expiry stops a live subscription over the wire, with no deploy

archreview/plans/R3-05-acl-hot-path-expiry-plan.md

@@ -710,3 +710,23 @@ for the orchestrator.
 - The node manager is resolved **per tick** (`() => _server?.NodeManager?.Revalidate…`) rather than
   captured, so a restart cannot leave the tick walking a dead manager's item map.
 - Host.Tests: 132 green.
+
+### T9 — over-the-wire expiry test (key-carrying session)
+
+- **RED observed (build):** `error CS1503: Argument 2: cannot convert from 'string' to 'byte[]'` — the
+  1.5.378 `UserNameIdentityToken.DecryptedPassword` is raw bytes, not a string (the host already decodes
+  UTF-8; the token had to be built the same way).
+- **FALSIFICATION (the plan's step 2, run and recorded):** deleting the single `rig.TickRevalidation()`
+  call from `An_expired_key_stops_a_live_subscription_with_no_deploy` makes it **`[FAIL]`** — the expired
+  key goes on streaming. So the denial is produced by the T8 tick and by nothing else. Restored, both
+  cases green.
+- **DEVIATION (fake choice):** the plan calls for "an `ApiKeyOpcUaUserAuthenticator` over a fake
+  `IApiKeyStore`". The rig instead uses a 6-line `KeyPresentingAuthenticator` returning
+  `OpcUaUserAuthResult.AllowKey(principal, [], keyId)`. The credential-verification path (parse, peppered
+  HMAC, store lookup, deny shapes) is already covered in the Host suite and changes nothing about what
+  this file asks — **whether the SDK hands a monitored item back an identity that still carries the key
+  id**. Grants come from the deployed snapshot, so the authenticator returns no roles.
+- The tick is driven through the **real `AclRevalidationTimer`**, which required a new `ProjectReference`
+  from the integration test project to `ZB.MOM.WW.OtOpcUa.Host` (the timer lives there). Without it the
+  test would have called the node manager's pass directly and could not have falsified T8.
+- OpcUaServer.IntegrationTests: 20 green (was 18).

archreview/plans/R3-05-acl-hot-path-expiry-plan.md.tasks.json

@@ -68,7 +68,7 @@
     {
       "id": "T9",
       "subject": "Over-the-wire AclKeyExpiryEnforcementTests: key-carrying session, clock advanced past expiry, item goes BadUserAccessDenied with no deploy (+ negative control)",
-      "status": "pending",
+      "status": "done",
       "blockedBy": [
         "T8"
       ]

tests/Server/ZB.MOM.WW.OtOpcUa.OpcUaServer.IntegrationTests/AclKeyExpiryEnforcementTests.cs

@@ -0,0 +1,370 @@
+using System.Net;
+using System.Net.Sockets;
+using System.Text;
+using Microsoft.Extensions.Logging.Abstractions;
+using Opc.Ua;
+using Opc.Ua.Client;
+using Shouldly;
+using Xunit;
+using ZB.MOM.WW.OtOpcUa.Commons.OpcUa;
+using ZB.MOM.WW.OtOpcUa.Configuration.Entities;
+using ZB.MOM.WW.OtOpcUa.Configuration.Enums;
+using ZB.MOM.WW.OtOpcUa.Core.Authorization;
+using ZB.MOM.WW.OtOpcUa.Host.OpcUa;
+using ZB.MOM.WW.OtOpcUa.OpcUaServer.Security;
+
+namespace ZB.MOM.WW.OtOpcUa.OpcUaServer.IntegrationTests;
+
+/// <summary>
+///     arch-review 03/S19 over the wire — a key that expires by the CLOCK stops a live subscription, with
+///     no deployment anywhere in the run.
+/// </summary>
+/// <remarks>
+///     <para>
+///         This cannot be reasoned into existence for the same reason
+///         <c>AclSubscriptionEnforcementTests</c> gives: whether a monitored item can recover the identity
+///         of the session that created it is an assumption about SDK behaviour. Here it must recover an
+///         identity carrying an <b>API key id</b> — the field the per-call expiry re-check keys on — and
+///         the existing rig has never exercised one, because it connects anonymously.
+///     </para>
+///     <para>
+///         The tick is driven through the real <see cref="AclRevalidationTimer"/> rather than by calling
+///         the node manager's pass directly, so what is asserted here is the whole T8 path: revert the
+///         host's timer and the denial below cannot be produced by anything except a deployment.
+///     </para>
+/// </remarks>
+public sealed class AclKeyExpiryEnforcementTests
+{
+    private const string Cluster = "SITE-A";
+    private const string Device = "Plant/Modbus/dev1";
+    private const string Tag = "Plant/Modbus/dev1/Speed";
+    private const string Principal = "expiry-probe";
+    private const string KeyId = "k-expiry";
+
+    private static readonly DateTime Start = new(2026, 8, 20, 12, 0, 0, DateTimeKind.Utc);
+
+    /// <summary>
+    ///     THE test this file exists for: the key's <c>ExpiresUtc</c> passes, a revalidation tick runs, and
+    ///     the client is told <c>BadUserAccessDenied</c> — with <c>DataPlaneAuthorityState.Apply</c> called
+    ///     exactly once, in setup.
+    /// </summary>
+    [Fact]
+    public async Task An_expired_key_stops_a_live_subscription_with_no_deploy()
+    {
+        var ct = TestContext.Current.CancellationToken;
+        var rig = await Rig.StartAsync("AclKeyExpiry", ct);
+        try
+        {
+            rig.Enforce(expiresUtc: Start.AddMinutes(10));
+            rig.Materialise();
+            rig.Write(1);
+
+            using var session = await rig.ConnectAsync(ct);
+            var (subscription, received) = await rig.SubscribeAsync(session, ct);
+
+            // Baseline: the key-carrying session's feed is live.
+            await WaitUntilAsync(() => received.Any(v => Equals(v.Value, 1f)), TimeSpan.FromSeconds(10));
+            received.Clear();
+
+            // Only the clock moves. No deployment, no new generation, no grant change.
+            rig.Advance(TimeSpan.FromMinutes(20));
+            rig.TickRevalidation();
+
+            await WaitUntilAsync(
+                () => received.Any(v => v.StatusCode.Code == StatusCodes.BadUserAccessDenied),
+                TimeSpan.FromSeconds(10));
+
+            // And the data really has stopped.
+            received.Clear();
+            rig.Write(2);
+            await Task.Delay(TimeSpan.FromSeconds(2), ct);
+            received.ShouldNotContain(v => Equals(v.Value, 2f));
+
+            await subscription.DeleteAsync(true, ct);
+        }
+        finally { await rig.DisposeAsync(); }
+    }
+
+    /// <summary>
+    ///     The negative control: identical run, clock advanced but not past <c>ExpiresUtc</c> ⇒ values keep
+    ///     arriving.
+    /// </summary>
+    /// <remarks>
+    ///     Without it the test above also passes on a build whose tick detaches every item it sees, which
+    ///     would silence every subscription in the fleet once a minute.
+    /// </remarks>
+    [Fact]
+    public async Task An_unexpired_key_keeps_streaming_across_a_tick()
+    {
+        var ct = TestContext.Current.CancellationToken;
+        var rig = await Rig.StartAsync("AclKeyLive", ct);
+        try
+        {
+            rig.Enforce(expiresUtc: Start.AddHours(4));
+            rig.Materialise();
+            rig.Write(1);
+
+            using var session = await rig.ConnectAsync(ct);
+            var (subscription, received) = await rig.SubscribeAsync(session, ct);
+            await WaitUntilAsync(() => received.Any(v => Equals(v.Value, 1f)), TimeSpan.FromSeconds(10));
+
+            received.Clear();
+            rig.Advance(TimeSpan.FromMinutes(20));
+            rig.TickRevalidation();
+
+            rig.Write(2);
+            await WaitUntilAsync(() => received.Any(v => Equals(v.Value, 2f)), TimeSpan.FromSeconds(10));
+            received.ShouldNotContain(v => v.StatusCode.Code == StatusCodes.BadUserAccessDenied);
+
+            await subscription.DeleteAsync(true, ct);
+        }
+        finally { await rig.DisposeAsync(); }
+    }
+
+    // ───────────────────────────── helpers ─────────────────────────────
+
+    private static readonly IReadOnlyDictionary<AclScopeRef, IReadOnlyList<NodeAclPath>> NoScopes =
+        new Dictionary<AclScopeRef, IReadOnlyList<NodeAclPath>>();
+
+    private static async Task WaitUntilAsync(Func<bool> condition, TimeSpan timeout)
+    {
+        var deadline = DateTime.UtcNow + timeout;
+        while (DateTime.UtcNow < deadline)
+        {
+            if (condition()) return;
+            await Task.Delay(50);
+        }
+        condition().ShouldBeTrue("condition not met within timeout");
+    }
+
+    /// <summary>A booted server, a key-carrying client session, and a hand-driven revalidation tick.</summary>
+    private sealed class Rig : IAsyncDisposable
+    {
+        private readonly string _pkiRoot;
+        private readonly FakeClock _clock = new(Start);
+        private OpcUaApplicationHost _host = null!;
+        private OtOpcUaSdkServer _server = null!;
+        private AclRevalidationTimer? _timer;
+        private int _port;
+
+        private Rig(string pkiRoot) => _pkiRoot = pkiRoot;
+
+        private OtOpcUaNodeManager Nm => _server.NodeManager!;
+
+        public static async Task<Rig> StartAsync(string name, CancellationToken ct)
+        {
+            var rig = new Rig(Path.Combine(Path.GetTempPath(), $"otopcua-{name}-{Guid.NewGuid():N}"));
+            rig._port = AllocateFreePort();
+            rig._server = new OtOpcUaSdkServer();
+            rig._host = new OpcUaApplicationHost(
+                new OpcUaApplicationHostOptions
+                {
+                    ApplicationName = $"OtOpcUa.{name}",
+                    ApplicationUri = $"urn:OtOpcUa.{name}.{Guid.NewGuid():N}",
+                    OpcUaPort = rig._port,
+                    PublicHostname = "127.0.0.1",
+                    PkiStoreRoot = rig._pkiRoot,
+                    EnabledSecurityProfiles = [OpcUaSecurityProfile.None],
+                    AutoAcceptUntrustedClientCertificates = true,
+                },
+                NullLogger<OpcUaApplicationHost>.Instance,
+                new KeyPresentingAuthenticator());
+
+            await rig._host.StartAsync(rig._server, ct);
+            return rig;
+        }
+
+        /// <summary>
+        ///     Switches enforcement on with one cluster-wide grant and one key expiring at
+        ///     <paramref name="expiresUtc"/>, and arms the revalidation timer over the same clock.
+        /// </summary>
+        /// <param name="expiresUtc">When the key stops being usable.</param>
+        public void Enforce(DateTime expiresUtc)
+        {
+            var key = new ApiKey
+            {
+                ApiKeyId = KeyId,
+                Name = KeyId,
+                SecretHash = "hash",
+                Principal = Principal,
+                Enabled = true,
+                CreatedUtc = Start.AddDays(-1),
+                ExpiresUtc = expiresUtc,
+            };
+            var grant = new NodeAcl
+            {
+                NodeAclId = "acl-cluster",
+                ClusterId = Cluster,
+                Principal = Principal,
+                ScopeKind = NodeAclScopeKind.Cluster,
+                PermissionFlags = NodePermissions.Read | NodePermissions.Browse | NodePermissions.Subscribe,
+            };
+
+            var state = new DataPlaneAuthorityState();
+
+            // The ONLY Apply in the whole test. Everything after this moves the clock.
+            state.Apply(Cluster, "rev-1", [key], [grant], NoScopes);
+
+            Nm.DataPlaneAuthority = state;
+            Nm.Authorizer = new DataPlaneAuthorizer(
+                state, new TriePermissionEvaluator(state.Tries), enabled: true, NullLogger.Instance, _clock);
+
+            // A long interval: the tick is driven by hand, so a real timer firing underneath would only
+            // make the test non-deterministic. What matters is that the pass is reached through the same
+            // component the hosted service arms.
+            _timer = new AclRevalidationTimer(
+                () => _server.NodeManager?.RevalidateMonitoredItems(),
+                TimeSpan.FromHours(1),
+                TimeProvider.System,
+                NullLogger.Instance);
+        }
+
+        /// <summary>Moves the authorizer's clock forward — the only thing that changes mid-test.</summary>
+        /// <param name="by">How far forward.</param>
+        public void Advance(TimeSpan by) => _clock.Advance(by);
+
+        /// <summary>Runs one revalidation tick through the host's own timer component.</summary>
+        public void TickRevalidation() => _timer!.Tick();
+
+        public void Materialise()
+        {
+            Nm.EnsureFolder("Plant", null, "Plant", AddressSpaceRealm.Raw);
+            Nm.EnsureFolder("Plant/Modbus", "Plant", "Modbus", AddressSpaceRealm.Raw);
+            Nm.EnsureFolder(Device, "Plant/Modbus", "dev1", AddressSpaceRealm.Raw);
+            Nm.EnsureVariable(Tag, Device, "Speed", "Float", writable: false, AddressSpaceRealm.Raw);
+        }
+
+        public void Write(float value) =>
+            Nm.WriteValue(Tag, value, OpcUaStatus.Good, DateTime.UtcNow, AddressSpaceRealm.Raw);
+
+        public ushort RawNamespace(ISession session)
+        {
+            var ns = (ushort)session.NamespaceUris.GetIndex(V3NodeIds.RawNamespaceUri);
+            ns.ShouldBeGreaterThan((ushort)0);
+            return ns;
+        }
+
+        public NodeId TagNodeId(ISession session) => new(Tag, RawNamespace(session));
+
+        public async Task<(Subscription Subscription, List<DataValue> Received)> SubscribeAsync(
+            ISession session, CancellationToken ct)
+        {
+            var received = new List<DataValue>();
+            var gate = new object();
+
+            var subscription = new Subscription(session.DefaultSubscription) { PublishingInterval = 100 };
+            session.AddSubscription(subscription);
+            await subscription.CreateAsync(ct);
+
+            var item = new MonitoredItem(subscription.DefaultItem)
+            {
+                StartNodeId = TagNodeId(session),
+                AttributeId = Attributes.Value,
+                SamplingInterval = 50,
+                QueueSize = 10,
+            };
+            item.Notification += (mi, _) =>
+            {
+                foreach (var v in mi.DequeueValues()) lock (gate) received.Add(v);
+            };
+            subscription.AddItem(item);
+            await subscription.ApplyChangesAsync(ct);
+
+            item.Status.Error.ShouldBeNull();
+            return (subscription, received);
+        }
+
+        /// <summary>Connects presenting the fixed <c>apikey</c> username with an <c>otua_</c> credential.</summary>
+        /// <param name="ct">The cancellation token.</param>
+        /// <returns>The connected session.</returns>
+        public async Task<ISession> ConnectAsync(CancellationToken ct)
+        {
+            var appConfig = new ApplicationConfiguration
+            {
+                ApplicationName = "OtOpcUa.AclKeyClient",
+                ApplicationUri = $"urn:OtOpcUa.AclKeyClient.{Guid.NewGuid():N}",
+                ApplicationType = ApplicationType.Client,
+                SecurityConfiguration = TestClientSecurity.Build(TestClientSecurity.AllocatePkiRoot()),
+                ClientConfiguration = new ClientConfiguration { DefaultSessionTimeout = 60_000 },
+            };
+            await appConfig.ValidateAsync(ApplicationType.Client, ct);
+            appConfig.CertificateValidator.CertificateValidation += (_, e) => e.Accept = true;
+
+            var endpoint = await CoreClientUtils.SelectEndpointAsync(
+                appConfig, $"opc.tcp://127.0.0.1:{_port}/OtOpcUa", false, DefaultTelemetry.Create(_ => { }), ct);
+
+            return await new DefaultSessionFactory(DefaultTelemetry.Create(_ => { })).CreateAsync(
+                appConfig,
+                new ConfiguredEndpoint(null, endpoint, EndpointConfiguration.Create(appConfig)),
+                updateBeforeConnect: false,
+                checkDomain: false,
+                sessionName: "AclKeyExpiryEnforcementTests",
+                sessionTimeout: 60_000,
+                identity: new UserIdentity(new UserNameIdentityToken
+                {
+                    UserName = "apikey",
+                    // 1.5.378 carries the UserName token password as raw bytes; the host decodes UTF-8.
+                    DecryptedPassword = Encoding.UTF8.GetBytes($"otua_{KeyId}_secret"),
+                }),
+                preferredLocales: null,
+                ct: ct);
+        }
+
+        public async ValueTask DisposeAsync()
+        {
+            _timer?.Dispose();
+            await _host.DisposeAsync();
+            if (Directory.Exists(_pkiRoot))
+            {
+                try { Directory.Delete(_pkiRoot, recursive: true); }
+                catch (IOException) { /* best effort */ }
+            }
+        }
+
+        private static int AllocateFreePort()
+        {
+            var listener = new TcpListener(IPAddress.Loopback, 0);
+            listener.Start();
+            var port = ((IPEndPoint)listener.LocalEndpoint).Port;
+            listener.Stop();
+            return port;
+        }
+
+        /// <summary>
+        ///     Admits the fixed <c>apikey</c> username as <see cref="Principal"/>, carrying
+        ///     <see cref="KeyId"/>.
+        /// </summary>
+        /// <remarks>
+        ///     Deliberately not the production <c>ApiKeyOpcUaUserAuthenticator</c> over a fake store: the
+        ///     credential-verification path (parse, peppered HMAC, store lookup, deny shapes) is covered by
+        ///     <c>ApiKeyAuthenticationTests</c> and <c>BoundedApiKeyStoreTests</c> in the Host suite, and
+        ///     none of it changes what this file is about — whether the SDK hands a monitored item back an
+        ///     identity that still carries the key id. Grants come from the deployed snapshot, so this
+        ///     returns no roles.
+        /// </remarks>
+        private sealed class KeyPresentingAuthenticator : IOpcUaUserAuthenticator
+        {
+            /// <inheritdoc />
+            public Task<OpcUaUserAuthResult> AuthenticateUserNameAsync(
+                string username, string password, CancellationToken ct) =>
+                Task.FromResult(
+                    string.Equals(username, "apikey", StringComparison.Ordinal)
+                    && password.StartsWith($"otua_{KeyId}_", StringComparison.Ordinal)
+                        ? OpcUaUserAuthResult.AllowKey(Principal, [], KeyId)
+                        : OpcUaUserAuthResult.Deny("Invalid API key."));
+        }
+
+        /// <summary>A hand-advanced clock; the authorizer reads it on every decision.</summary>
+        private sealed class FakeClock(DateTime utcNow) : TimeProvider
+        {
+            private DateTimeOffset _now = new(utcNow, TimeSpan.Zero);
+
+            /// <inheritdoc />
+            public override DateTimeOffset GetUtcNow() => _now;
+
+            /// <summary>Moves the clock forward.</summary>
+            /// <param name="by">How far forward.</param>
+            public void Advance(TimeSpan by) => _now = _now.Add(by);
+        }
+    }
+}

tests/Server/ZB.MOM.WW.OtOpcUa.OpcUaServer.IntegrationTests/ZB.MOM.WW.OtOpcUa.OpcUaServer.IntegrationTests.csproj

@@ -25,6 +25,10 @@
 
     <ItemGroup>
         <ProjectReference Include="..\..\..\src\Server\ZB.MOM.WW.OtOpcUa.OpcUaServer\ZB.MOM.WW.OtOpcUa.OpcUaServer.csproj"/>
+        <!-- AclKeyExpiryEnforcementTests drives the real AclRevalidationTimer (arch-review 03/S19), so the
+             over-the-wire denial is produced by the same component the hosted service arms rather than by
+             a direct call to the node manager's pass. -->
+        <ProjectReference Include="..\..\..\src\Server\ZB.MOM.WW.OtOpcUa.Host\ZB.MOM.WW.OtOpcUa.Host.csproj"/>
     </ItemGroup>
 
 </Project>