fix(host): leave the cluster before Kestrel drains and end open streams on stop (#906)

A planned stop of the Primary took ~30 s on QA: WebApplication stops the web
server before Akka.Hosting's service, and Kestrel's graceful drain waited the
whole ShutdownTimeout for the peer's open LocalDb sync stream (the library's
LocalDbSyncService only observes the call token, which fires at the END of the
drain). Only then did CoordinatedShutdown run, so the stopping node held the
Primary role and the peer sat at ServiceLevel 240 for the whole window.

- GracefulClusterLeaveService (IHostedLifecycleService): StoppingAsync runs
  before any hosted service's StopAsync; it raises ApplicationStopping, then
  runs and awaits CoordinatedShutdown with ClrExitReason (not Cluster.Leave,
  whose ClusterLeavingReason makes Akka.Hosting set ExitCode -1). The cluster
  is left while Kestrel is still up.
- ShutdownStreamCancellationInterceptor (innermost on the shared gRPC
  pipeline): every streaming call gets a context whose token is linked with
  ApplicationStopping; streams arriving once stopping are refused Unavailable.
  Host-side mitigation until LocalDb 0.3.8 is published and pinned
  (scadaproj 3ddb083, scadaproj#57); it also covers the mesh streams.

Test GracefulStopWithOpenSyncStreamTests (real WebApplication + Kestrel h2c +
MapZbLocalDbSync + single-node Akka cluster + real peer initiator, 8 s drain
budget): stop 895 ms, cluster left at 892 ms, drain began at 894 ms. Positive
control without the fix: stop 8863 ms, cluster left at 8841 ms after the drain
began at 0 ms. Mutations: no CoordinatedShutdown in StoppingAsync -> order
assertion fails (drain at 1 ms, removed at 893 ms); interceptor passes the
original context -> stop 8006 ms, fails.

Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi

src/Server/ZB.MOM.WW.OtOpcUa.Host/Configuration/ShutdownStreamCancellationInterceptor.cs

@@ -0,0 +1,180 @@
+using Grpc.Core;
+using Grpc.Core.Interceptors;
+using Microsoft.Extensions.Hosting;
+using Microsoft.Extensions.Logging;
+
+namespace ZB.MOM.WW.OtOpcUa.Host.Configuration;
+
+/// <summary>
+///     Ends every open streaming gRPC call on this host's shared gRPC pipeline when the host starts
+///     stopping, and refuses new ones from then on (lmxopcua#906).
+/// </summary>
+/// <remarks>
+///     <para>
+///         <b>Why.</b> A server-side streaming handler observes <see cref="ServerCallContext.CancellationToken"/>,
+///         which ASP.NET Core cancels only when the client goes away or Kestrel aborts the request — and
+///         Kestrel aborts only at the END of its graceful drain, i.e. after the host's whole
+///         <c>ShutdownTimeout</c> (30 s by default). The peer's LocalDb sync stream
+///         (<c>/localdb_sync.v1.LocalDbSync/Sync</c>) never ends on its own, so every graceful stop of
+///         the node it is connected to took the full 30 s. Through LocalDb 0.3.7 the library's
+///         <c>LocalDbSyncService</c> does not observe <c>ApplicationStopping</c> itself (fixed on scadaproj
+///         main <c>3ddb083</c>, publish tracked as scadaproj#57), so the host supplies it here: the handler is
+///         given a context whose token is the call's own token linked with <see cref="IHostApplicationLifetime.ApplicationStopping"/>.
+///         Handlers that already treat a cancelled call token as "client gone / server shutting down"
+///         — the LocalDb sync service does — then return cleanly, and Kestrel's drain has nothing left
+///         to wait for.
+///     </para>
+///     <para>
+///         <b>Scope.</b> Every client-, server- and duplex-streaming call, whatever its service: the
+///         mesh telemetry and command streams are long-lived too and would hold the drain the same
+///         way. Unary calls are left alone — they are short and finishing them is the point of a drain.
+///     </para>
+///     <para>
+///         A stream that arrives once stopping has begun (the peer redialling the moment its stream
+///         ended) is refused with <see cref="StatusCode.Unavailable"/> — the retryable code — rather
+///         than admitted into a node that is about to go away.
+///     </para>
+///     <para>
+///         <b>Exactly one public constructor</b> — <c>Grpc.AspNetCore</c> silently stops invoking an
+///         interceptor that has more than one.
+///     </para>
+/// </remarks>
+public sealed class ShutdownStreamCancellationInterceptor : Interceptor
+{
+    private readonly IHostApplicationLifetime _lifetime;
+    private readonly ILogger<ShutdownStreamCancellationInterceptor> _logger;
+
+    /// <summary>Creates the interceptor.</summary>
+    /// <param name="lifetime">The host lifetime whose <c>ApplicationStopping</c> ends open streams.</param>
+    /// <param name="logger">Logger for the shutdown diagnostics.</param>
+    public ShutdownStreamCancellationInterceptor(
+        IHostApplicationLifetime lifetime,
+        ILogger<ShutdownStreamCancellationInterceptor> logger)
+    {
+        ArgumentNullException.ThrowIfNull(lifetime);
+        ArgumentNullException.ThrowIfNull(logger);
+        _lifetime = lifetime;
+        _logger = logger;
+    }
+
+    /// <inheritdoc />
+    public override Task DuplexStreamingServerHandler<TRequest, TResponse>(
+        IAsyncStreamReader<TRequest> requestStream,
+        IServerStreamWriter<TResponse> responseStream,
+        ServerCallContext context,
+        DuplexStreamingServerMethod<TRequest, TResponse> continuation)
+        => RunAsync(context, linked => continuation(requestStream, responseStream, linked));
+
+    /// <inheritdoc />
+    public override Task ServerStreamingServerHandler<TRequest, TResponse>(
+        TRequest request,
+        IServerStreamWriter<TResponse> responseStream,
+        ServerCallContext context,
+        ServerStreamingServerMethod<TRequest, TResponse> continuation)
+        => RunAsync(context, linked => continuation(request, responseStream, linked));
+
+    /// <inheritdoc />
+    public override async Task<TResponse> ClientStreamingServerHandler<TRequest, TResponse>(
+        IAsyncStreamReader<TRequest> requestStream,
+        ServerCallContext context,
+        ClientStreamingServerMethod<TRequest, TResponse> continuation)
+    {
+        TResponse response = default!;
+        await RunAsync(context, async linked => response = await continuation(requestStream, linked).ConfigureAwait(false))
+            .ConfigureAwait(false);
+        return response;
+    }
+
+    private async Task RunAsync(ServerCallContext context, Func<ServerCallContext, Task> handler)
+    {
+        var stopping = _lifetime.ApplicationStopping;
+        if (stopping.IsCancellationRequested)
+        {
+            throw new RpcException(new Status(
+                StatusCode.Unavailable, "This node is shutting down; retry against its peer or after it restarts."));
+        }
+
+        // GetHttpContext() recognises only ASP.NET Core's own context type, falling back to a
+        // UserState entry for any other — so seed that entry before handing the handler a wrapper.
+        try
+        {
+            context.UserState.TryAdd(HttpContextKey, context.GetHttpContext());
+        }
+        catch (InvalidOperationException)
+        {
+            // Not hosted by ASP.NET Core (a unit test's fake context): nothing to carry over.
+        }
+
+        using var linked = CancellationTokenSource.CreateLinkedTokenSource(context.CancellationToken, stopping);
+        using var onStopping = stopping.Register(
+            static state =>
+            {
+                var (logger, method) = ((ILogger, string))state!;
+                logger.LogInformation("Host stopping: ending the open gRPC stream {Method}.", method);
+            },
+            ((ILogger)_logger, context.Method));
+
+        await handler(new LinkedTokenServerCallContext(context, linked.Token)).ConfigureAwait(false);
+    }
+
+    /// <summary>The key <c>GetHttpContext()</c> falls back to for a context that is not ASP.NET Core's own.</summary>
+    private const string HttpContextKey = "__HttpContext";
+
+    /// <summary>
+    ///     Delegates everything to the real call context except <see cref="ServerCallContext.CancellationToken"/>.
+    ///     <see cref="ServerCallContext.UserState"/> is delegated too and carries the <c>__HttpContext</c> entry
+    ///     seeded in <c>RunAsync</c>, so <c>GetHttpContext()</c> still resolves on the wrapper.
+    /// </summary>
+    internal sealed class LinkedTokenServerCallContext(ServerCallContext inner, CancellationToken token)
+        : ServerCallContext
+    {
+        /// <inheritdoc />
+        protected override string MethodCore => inner.Method;
+
+        /// <inheritdoc />
+        protected override string HostCore => inner.Host;
+
+        /// <inheritdoc />
+        protected override string PeerCore => inner.Peer;
+
+        /// <inheritdoc />
+        protected override DateTime DeadlineCore => inner.Deadline;
+
+        /// <inheritdoc />
+        protected override Metadata RequestHeadersCore => inner.RequestHeaders;
+
+        /// <inheritdoc />
+        protected override CancellationToken CancellationTokenCore => token;
+
+        /// <inheritdoc />
+        protected override Metadata ResponseTrailersCore => inner.ResponseTrailers;
+
+        /// <inheritdoc />
+        protected override Status StatusCore
+        {
+            get => inner.Status;
+            set => inner.Status = value;
+        }
+
+        /// <inheritdoc />
+        protected override WriteOptions? WriteOptionsCore
+        {
+            get => inner.WriteOptions;
+            set => inner.WriteOptions = value;
+        }
+
+        /// <inheritdoc />
+        protected override AuthContext AuthContextCore => inner.AuthContext;
+
+        /// <inheritdoc />
+        protected override IDictionary<object, object> UserStateCore => inner.UserState;
+
+        /// <inheritdoc />
+        protected override ContextPropagationToken CreatePropagationTokenCore(ContextPropagationOptions? options)
+            => inner.CreatePropagationToken(options);
+
+        /// <inheritdoc />
+        protected override Task WriteResponseHeadersAsyncCore(Metadata responseHeaders)
+            => inner.WriteResponseHeadersAsync(responseHeaders);
+    }
+}

src/Server/ZB.MOM.WW.OtOpcUa.Host/GracefulClusterLeaveService.cs

@@ -0,0 +1,156 @@
+using Akka.Actor;
+using Microsoft.Extensions.DependencyInjection;
+using Microsoft.Extensions.Hosting;
+using Microsoft.Extensions.Logging;
+
+namespace ZB.MOM.WW.OtOpcUa.Host;
+
+/// <summary>
+///     Makes a graceful host stop leave the Akka cluster FIRST — before Kestrel drains its open
+///     requests — so the peer of a redundant pair takes over the Primary role promptly (lmxopcua#906).
+/// </summary>
+/// <remarks>
+///     <para>
+///         <b>Why it is needed.</b> With <c>WebApplication</c> the web server's hosted service starts
+///         after every hosted service the app registers, so on stop it runs FIRST, and Akka.Hosting's
+///         own <c>StopAsync</c> (which runs <see cref="CoordinatedShutdown"/>) only runs once Kestrel
+///         has finished draining. Kestrel waits for every open request up to the host's
+///         <c>ShutdownTimeout</c> (30 s by default) — and the peer's LocalDb sync stream is a
+///         long-lived gRPC call that never ends on its own. On QA (2026-11-13) a planned stop of the
+///         Primary therefore held the Primary role for ~30 s while the Secondary sat at ServiceLevel 240.
+///     </para>
+///     <para>
+///         <b>How.</b> <see cref="IHostedLifecycleService.StoppingAsync"/> runs for every lifecycle service
+///         before ANY hosted service's <c>StopAsync</c>, Kestrel's included. Here it (1) raises
+///         <see cref="IHostApplicationLifetime.ApplicationStopping"/> — idempotent; in production
+///         <c>RunAsync</c> has already raised it, but a direct <c>IHost.StopAsync</c> raises it only
+///         after <c>StoppingAsync</c> — so the long-lived streams are cancelled by
+///         <see cref="Configuration.ShutdownStreamCancellationInterceptor"/> and the
+///         <see cref="ActorSystemTerminationWatchdog"/> sees an expected shutdown; then (2) runs
+///         <see cref="CoordinatedShutdown"/> and awaits it, bounded by the host's shutdown token. The
+///         cluster-leave and singleton hand-over phases therefore complete while Kestrel is still up
+///         (so <c>/health/active</c> keeps answering and flips to 503), and only then does Kestrel drain.
+///     </para>
+///     <para>
+///         <b>Why <see cref="CoordinatedShutdown.ClrExitReason"/>, not <c>Cluster.Leave</c>.</b> A plain
+///         leave ends in <c>CoordinatedShutdown</c> with <c>ClusterLeavingReason</c>, which Akka.Hosting's
+///         termination hook turns into <c>Environment.ExitCode = -1</c> — a service supervisor would read
+///         a planned stop as a crash. <c>ClrExitReason</c> is exactly the reason Akka.Hosting's own
+///         <c>StopAsync</c> uses; its later <c>Run</c> call returns this same, already completed task.
+///     </para>
+///     <para>
+///         Ordering relative to the hosted services that stop AFTER Akka today (the OPC UA endpoint, the
+///         LocalDb sync initiator, everything registered before <c>AddAkka</c>) is unchanged: they still
+///         stop after the ActorSystem has terminated. Only the web server and the few services registered
+///         after <c>AddAkka</c> now stop after it too.
+///     </para>
+/// </remarks>
+public sealed class GracefulClusterLeaveService : IHostedLifecycleService
+{
+    private readonly Func<ActorSystem> _actorSystemAccessor;
+    private readonly IHostApplicationLifetime _lifetime;
+    private readonly ILogger<GracefulClusterLeaveService> _logger;
+    private volatile bool _started;
+
+    /// <summary>Creates the service over a lazy <see cref="ActorSystem"/> accessor.</summary>
+    /// <param name="actorSystemAccessor">Lazy accessor; resolved only on stop, never at construction.</param>
+    /// <param name="lifetime">The host lifetime whose <c>ApplicationStopping</c> this raises.</param>
+    /// <param name="logger">Logger for the leave diagnostics.</param>
+    public GracefulClusterLeaveService(
+        Func<ActorSystem> actorSystemAccessor,
+        IHostApplicationLifetime lifetime,
+        ILogger<GracefulClusterLeaveService> logger)
+    {
+        ArgumentNullException.ThrowIfNull(actorSystemAccessor);
+        ArgumentNullException.ThrowIfNull(lifetime);
+        ArgumentNullException.ThrowIfNull(logger);
+        _actorSystemAccessor = actorSystemAccessor;
+        _lifetime = lifetime;
+        _logger = logger;
+    }
+
+    /// <inheritdoc />
+    public Task StartingAsync(CancellationToken cancellationToken) => Task.CompletedTask;
+
+    /// <inheritdoc />
+    public Task StartAsync(CancellationToken cancellationToken) => Task.CompletedTask;
+
+    /// <inheritdoc />
+    public Task StartedAsync(CancellationToken cancellationToken)
+    {
+        // Only a host that got this far has a running ActorSystem worth leaving; a host whose start
+        // failed earlier must not resolve (and thereby try to build) one on the way down.
+        _started = true;
+        return Task.CompletedTask;
+    }
+
+    /// <summary>Raises <c>ApplicationStopping</c>, then runs and awaits CoordinatedShutdown.</summary>
+    /// <param name="cancellationToken">The host's shutdown token (bounded by <c>ShutdownTimeout</c>).</param>
+    public async Task StoppingAsync(CancellationToken cancellationToken)
+    {
+        _lifetime.StopApplication();
+
+        if (!_started)
+            return;
+
+        ActorSystem system;
+        try
+        {
+            system = _actorSystemAccessor();
+        }
+        catch (Exception ex)
+        {
+            _logger.LogWarning(ex, "Graceful cluster leave skipped: the ActorSystem could not be resolved.");
+            return;
+        }
+
+        var started = TimeProvider.System.GetTimestamp();
+        _logger.LogInformation(
+            "Host stopping: leaving the Akka cluster before the web server drains its open requests (lmxopcua#906).");
+        try
+        {
+            await CoordinatedShutdown.Get(system)
+                .Run(CoordinatedShutdown.ClrExitReason.Instance)
+                .WaitAsync(cancellationToken)
+                .ConfigureAwait(false);
+            _logger.LogInformation(
+                "Akka CoordinatedShutdown completed in {ElapsedMs:0} ms; continuing host shutdown.",
+                TimeProvider.System.GetElapsedTime(started).TotalMilliseconds);
+        }
+        catch (OperationCanceledException) when (cancellationToken.IsCancellationRequested)
+        {
+            _logger.LogWarning(
+                "Akka CoordinatedShutdown did not finish within the host shutdown timeout; continuing host shutdown.");
+        }
+        catch (Exception ex)
+        {
+            _logger.LogWarning(ex, "Akka CoordinatedShutdown failed during the graceful leave; continuing host shutdown.");
+        }
+    }
+
+    /// <inheritdoc />
+    public Task StopAsync(CancellationToken cancellationToken) => Task.CompletedTask;
+
+    /// <inheritdoc />
+    public Task StoppedAsync(CancellationToken cancellationToken) => Task.CompletedTask;
+}
+
+/// <summary>Registration for <see cref="GracefulClusterLeaveService"/> — shared by <c>Program.cs</c> and its tests.</summary>
+public static class GracefulClusterLeaveRegistration
+{
+    /// <summary>
+    ///     Registers <see cref="GracefulClusterLeaveService"/>. Call it after <c>AddAkka</c>; the
+    ///     <see cref="ActorSystem"/> is resolved lazily, on stop, never at construction.
+    /// </summary>
+    /// <param name="services">The service collection.</param>
+    /// <returns>The same collection.</returns>
+    public static IServiceCollection AddOtOpcUaGracefulClusterLeave(this IServiceCollection services)
+    {
+        ArgumentNullException.ThrowIfNull(services);
+        services.AddHostedService(sp => new GracefulClusterLeaveService(
+            () => sp.GetRequiredService<ActorSystem>(),
+            sp.GetRequiredService<IHostApplicationLifetime>(),
+            sp.GetRequiredService<ILogger<GracefulClusterLeaveService>>()));
+        return services;
+    }
+}

src/Server/ZB.MOM.WW.OtOpcUa.Host/Program.cs

@@ -549,6 +549,14 @@ builder.Services.AddAkka("otopcua", (ab, sp) =>
     }
 });
 
+// lmxopcua#906: a graceful stop leaves the cluster FIRST. Kestrel's hosted service stops before Akka's
+// (WebApplication starts the server last), and Kestrel drains open requests for up to ShutdownTimeout,
+// so without this the peer's long-lived LocalDb sync stream held the Primary role for ~30 s on every
+// planned stop. Its StoppingAsync runs before ANY hosted service's StopAsync. Registered BEFORE the
+// watchdog below only for readability — it raises ApplicationStopping itself, which is what keeps the
+// watchdog quiet, so it does not depend on lifecycle-callback order.
+builder.Services.AddOtOpcUaGracefulClusterLeave();
+
 // Simultaneous-cold-start split-brain guard (dark switch Cluster:BootstrapGuard:Enabled, default off).
 // Registered AFTER AddAkka so its StartAsync runs once Akka's hosted service has built (and, when the
 // guard is off, already auto-joined) the ActorSystem. When on, the node started with no config seeds
@@ -651,6 +659,10 @@ if (hasDriver || hasAdmin)
             o.Interceptors.Add<ConfigServeAuthInterceptor>();
             o.Interceptors.Add<CentralCommandAuthInterceptor>();
         }
+        // Innermost (after every auth gate): ends open streaming calls — the peer's LocalDb sync stream
+        // above all — on ApplicationStopping instead of letting Kestrel's drain wait out ShutdownTimeout
+        // (lmxopcua#906), and refuses new streams once stopping has begun.
+        o.Interceptors.Add<ShutdownStreamCancellationInterceptor>();
     });
 }