fix(host): leave the cluster before Kestrel drains and end open streams on stop (#140)

A graceful stop of a site node whose peer holds an open LocalDb sync stream
left the Akka cluster ~30 s late (the ScadaBridge half of lmxopcua#906,
scadaproj#57). WebApplication stops the web server before AkkaHostedService,
whose StopAsync ran CoordinatedShutdown, and Kestrel's graceful drain waited
the whole ShutdownTimeout for /localdb_sync.v1.LocalDbSync/Sync: LocalDb 0.3.7
observes only the call token, which Kestrel cancels at the END of the drain.
CancelAllStreams only covered SiteStreamGrpcServer's own streams.

- AkkaHostedService is now an IHostedLifecycleService. StoppingAsync runs
  before any hosted service's StopAsync (Kestrel's included): marks the stop
  intentional, raises ApplicationStopping (so CancelAllStreams still precedes
  actor teardown), then runs and awaits CoordinatedShutdown(ClrExitReason),
  bounded by the host shutdown token. StopAsync returns the same completed
  shutdown. Exit codes unchanged: planned stop exits 0; only a termination
  outside host shutdown (self-down / #132 shut-out) exits 75. No Akka.Hosting
  here, so no ClusterLeavingReason/-1 exit-code path is involved.
- ShutdownStreamCancellationInterceptor, innermost on both the central and the
  site AddGrpc pipelines: every streaming call's token is linked with
  ApplicationStopping; new streams are refused Unavailable while stopping.
  Covers the LocalDb sync stream until LocalDb 0.3.8 is pinned.

Evidence (IntegrationTests/GracefulStopWithOpenSyncStreamTests: real
WebApplication + Kestrel h2c + MapZbLocalDbSync behind LocalDbSyncAuthInterceptor
+ a real LocalDb SyncBackgroundService peer + the production AkkaHostedService
forming a single-member cluster, 8 s drain budget, IServer wrapped to timestamp
the drain): before 8859 ms stop, member removed at 8834 ms, drain began at 0 ms;
after 960 ms stop, member removed at 950 ms, drain began at 959 ms.

Mutations (all killed): StoppingAsync skips CoordinatedShutdown; interceptor
passes the original context (stop 8003 ms); no Unavailable refusal; StoppingAsync
does not raise ApplicationStopping; lifecycle interface dropped; site pipeline
loses the interceptor.

Release build of the solution: 0 warnings. Host.Tests 733/733,
IntegrationTests 133/133, Communication.Tests 843/843.

Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi

src/ZB.MOM.WW.ScadaBridge.Host/Actors/AkkaHostedService.cs

@@ -26,7 +26,31 @@ namespace ZB.MOM.WW.ScadaBridge.Host.Actors;
 ///
 /// Transport heartbeat is explicitly configured in HOCON from CommunicationOptions.
 /// </summary>
-public class AkkaHostedService : IHostedService
+/// <remarks>
+/// <para>
+/// <b>Leave the cluster before the web server drains (ScadaBridge#140, lmxopcua#906).</b> With
+/// <c>WebApplication</c> the web server's hosted service starts after every hosted service the app
+/// registers, so on stop it runs FIRST — and Kestrel's graceful drain waits for every open request
+/// up to the host's <c>ShutdownTimeout</c> (30 s default). On a site pair the peer's LocalDb sync
+/// stream never ends on its own, so CoordinatedShutdown (cluster leave + singleton hand-over),
+/// which used to run only in <see cref="StopAsync"/>, ran ~30 s late and the peer took over that
+/// much later. This service is therefore an <see cref="IHostedLifecycleService"/>:
+/// <see cref="StoppingAsync"/> runs before ANY hosted service's <c>StopAsync</c>, Kestrel's
+/// included, and runs the shutdown there, while Kestrel is still up (so <c>/health/active</c>
+/// keeps answering and flips to 503 as the node leaves).
+/// </para>
+/// <para>
+/// <b>Exit codes are unchanged.</b> The shutdown still runs with
+/// <see cref="CoordinatedShutdown.ClrExitReason"/> — the same reason <see cref="StopAsync"/> always
+/// used — and marks the stop intentional first, so the <see cref="ActorSystem.WhenTerminated"/>
+/// watchdog stays quiet and a planned stop exits 0. Only a termination OUTSIDE host shutdown (a
+/// self-down or a ScadaBridge#132 shut-out restart) still produces <see cref="SelfDownExitCode"/>.
+/// It deliberately does not call <c>Cluster.Leave</c>: CoordinatedShutdown's own
+/// <c>cluster-leave</c> phase performs the leave, and running it through one reason keeps a single
+/// shutdown path.
+/// </para>
+/// </remarks>
+public class AkkaHostedService : IHostedLifecycleService
 {
     private readonly IServiceProvider _serviceProvider;
     private readonly NodeOptions _nodeOptions;
@@ -489,7 +513,72 @@ akka {{
     /// </summary>
     /// <param name="cancellationToken">A cancellation token that can be used to cancel the operation.</param>
     /// <returns>A task representing the asynchronous operation.</returns>
-    public async Task StopAsync(CancellationToken cancellationToken)
+    /// <remarks>
+    /// Under a real host, <see cref="StoppingAsync"/> has already run the shutdown and this returns
+    /// the same, completed CoordinatedShutdown task. Called on its own (tests, hosts that do not
+    /// honour <see cref="IHostedLifecycleService"/>) it performs the whole shutdown as before.
+    /// </remarks>
+    public Task StopAsync(CancellationToken cancellationToken) => ShutdownAsync();
+
+    /// <inheritdoc />
+    public Task StartingAsync(CancellationToken cancellationToken) => Task.CompletedTask;
+
+    /// <inheritdoc />
+    public Task StartedAsync(CancellationToken cancellationToken) => Task.CompletedTask;
+
+    /// <summary>
+    /// Runs before ANY hosted service's <c>StopAsync</c> — Kestrel's included — and leaves the Akka
+    /// cluster there (ScadaBridge#140): raises <c>ApplicationStopping</c> (idempotent; in production
+    /// the service/console lifetime has already raised it, but a direct <c>IHost.StopAsync</c> raises
+    /// it only after the lifecycle callbacks), then runs and awaits CoordinatedShutdown, bounded by
+    /// the host's shutdown token.
+    /// </summary>
+    /// <param name="cancellationToken">The host's shutdown token (bounded by <c>ShutdownTimeout</c>).</param>
+    /// <returns>A task that completes once the node has left (or the budget ran out).</returns>
+    public async Task StoppingAsync(CancellationToken cancellationToken)
+    {
+        // Intentional shutdown first, so the WhenTerminated watchdog stays quiet; then
+        // ApplicationStopping, so the site's stream cancellation (SiteStreamGrpcServer.CancelAllStreams,
+        // ShutdownStreamCancellationInterceptor) runs before the actors are torn down — the documented
+        // order — even when the host is stopped directly rather than through its lifetime.
+        _stopRequested = true;
+        _appLifetime?.StopApplication();
+
+        if (_actorSystem is null)
+        {
+            return;
+        }
+
+        var started = TimeProvider.System.GetTimestamp();
+        _logger.LogInformation(
+            "Host stopping: leaving the Akka cluster before the web server drains its open requests (ScadaBridge#140).");
+        try
+        {
+            await ShutdownAsync().WaitAsync(cancellationToken).ConfigureAwait(false);
+            _logger.LogInformation(
+                "Akka CoordinatedShutdown completed in {ElapsedMs:0} ms; continuing host shutdown.",
+                TimeProvider.System.GetElapsedTime(started).TotalMilliseconds);
+        }
+        catch (OperationCanceledException) when (cancellationToken.IsCancellationRequested)
+        {
+            _logger.LogWarning(
+                "Akka CoordinatedShutdown did not finish within the host shutdown timeout; continuing host shutdown.");
+        }
+        catch (Exception ex)
+        {
+            _logger.LogWarning(ex, "Akka CoordinatedShutdown failed during the graceful leave; continuing host shutdown.");
+        }
+    }
+
+    /// <inheritdoc />
+    public Task StoppedAsync(CancellationToken cancellationToken) => Task.CompletedTask;
+
+    /// <summary>
+    /// The shutdown proper, shared by <see cref="StoppingAsync"/> and <see cref="StopAsync"/>.
+    /// Idempotent: the tracked-disposable list is drained once and <c>CoordinatedShutdown.Run</c>
+    /// returns the same task on every call.
+    /// </summary>
+    private async Task ShutdownAsync()
     {
         // Mark this as an intentional shutdown FIRST, so the WhenTerminated
         // watchdog (wired in GetOrCreateActorSystem) does not mistake the
@@ -524,10 +613,13 @@ akka {{
 
         if (_actorSystem != null)
         {
-            _logger.LogInformation("Shutting down Akka.NET actor system via CoordinatedShutdown...");
             var shutdown = Akka.Actor.CoordinatedShutdown.Get(_actorSystem);
+            var first = shutdown.ShutdownReason is null;
+            if (first)
+                _logger.LogInformation("Shutting down Akka.NET actor system via CoordinatedShutdown...");
             await shutdown.Run(Akka.Actor.CoordinatedShutdown.ClrExitReason.Instance);
-            _logger.LogInformation("Akka.NET actor system shutdown complete.");
+            if (first)
+                _logger.LogInformation("Akka.NET actor system shutdown complete.");
         }
     }
 

src/ZB.MOM.WW.ScadaBridge.Host/Program.cs

@@ -297,6 +297,11 @@ try
             options.MaxReceiveMessageSize = ZB.MOM.WW.ScadaBridge.Communication.Grpc.GrpcMessageLimits.MaxMessageBytes;
             options.MaxSendMessageSize = ZB.MOM.WW.ScadaBridge.Communication.Grpc.GrpcMessageLimits.MaxMessageBytes;
             options.Interceptors.Add<CentralControlAuthInterceptor>();
+            // Innermost, after the auth gate (ScadaBridge#140): ends any open streaming call on
+            // ApplicationStopping instead of letting Kestrel's drain wait out ShutdownTimeout, and
+            // refuses new streams Unavailable once stopping. Central's control plane is all unary
+            // today, so this is a backstop for any stream added later.
+            options.Interceptors.Add<ShutdownStreamCancellationInterceptor>();
         });
         builder.Services.AddSingleton<
             ZB.MOM.WW.ScadaBridge.Communication.Grpc.CentralControlGrpcService>();
@@ -857,6 +862,12 @@ try
             options.MaxSendMessageSize = ZB.MOM.WW.ScadaBridge.Communication.Grpc.GrpcMessageLimits.MaxMessageBytes;
             options.Interceptors.Add<LocalDbSyncAuthInterceptor>();
             options.Interceptors.Add<ControlPlaneAuthInterceptor>();
+            // Innermost, after both auth gates (ScadaBridge#140, lmxopcua#906): ends every open
+            // streaming call — the peer's LocalDb sync stream above all — on ApplicationStopping
+            // instead of letting Kestrel's drain wait out ShutdownTimeout (30 s), and refuses new
+            // streams Unavailable once stopping. Host-side until LocalDb 0.3.8 (scadaproj#57) is
+            // pinned; it stays useful after that as the backstop for every other stream.
+            options.Interceptors.Add<ShutdownStreamCancellationInterceptor>();
         });
         builder.Services.AddSingleton<ZB.MOM.WW.ScadaBridge.Communication.Grpc.SiteStreamGrpcServer>();
         // Site command plane (central→site) — the gRPC peer of SiteStreamGrpcServer. Both share
@@ -917,7 +928,11 @@ try
         // THEN does AkkaHostedService run CoordinatedShutdown and tear down
         // actors. Without this hand-off, in-flight streams go silent and only
         // time out via gRPC keepalive (~25 s), violating the documented
-        // four-step sequence.
+        // four-step sequence. Since ScadaBridge#140 AkkaHostedService runs that
+        // CoordinatedShutdown from IHostedLifecycleService.StoppingAsync (raising
+        // ApplicationStopping first), i.e. BEFORE Kestrel starts draining, and
+        // ShutdownStreamCancellationInterceptor ends the remaining streams (the
+        // peer's LocalDb sync stream) so the drain itself is short.
         var siteLifetime = app.Services.GetRequiredService<Microsoft.Extensions.Hosting.IHostApplicationLifetime>();
         var siteGrpcServer = app.Services.GetRequiredService<ZB.MOM.WW.ScadaBridge.Communication.Grpc.SiteStreamGrpcServer>();
         siteLifetime.ApplicationStopping.Register(() => siteGrpcServer.CancelAllStreams());

src/ZB.MOM.WW.ScadaBridge.Host/ShutdownStreamCancellationInterceptor.cs

@@ -0,0 +1,181 @@
+using Grpc.Core;
+using Grpc.Core.Interceptors;
+
+namespace ZB.MOM.WW.ScadaBridge.Host;
+
+/// <summary>
+/// Ends every open streaming gRPC call on this host when the host starts stopping, and refuses
+/// new ones from then on (ScadaBridge#140, the ScadaBridge half of lmxopcua#906).
+/// </summary>
+/// <remarks>
+/// <para>
+/// <b>Why.</b> A server-streaming handler observes <see cref="ServerCallContext.CancellationToken"/>,
+/// which ASP.NET Core cancels only when the client goes away or Kestrel aborts the request — and
+/// Kestrel aborts only at the END of its graceful drain, after the host's whole
+/// <c>ShutdownTimeout</c> (30 s by default). On a site pair the peer's LocalDb sync stream
+/// (<c>/localdb_sync.v1.LocalDbSync/Sync</c>) never ends on its own, so every graceful stop of the
+/// node it is connected to held the drain for the full 30 s. Through LocalDb 0.3.7 the library's
+/// <c>LocalDbSyncService</c> does not observe <c>ApplicationStopping</c> itself (fixed on scadaproj
+/// main <c>3ddb083</c>, publish tracked as scadaproj#57), so the host supplies it here: the handler
+/// gets a context whose token is the call's own token linked with
+/// <see cref="IHostApplicationLifetime.ApplicationStopping"/>. Handlers that treat a cancelled call
+/// token as "client gone / server stopping" — the LocalDb sync service does — then return
+/// cleanly, and Kestrel's drain has nothing left to wait for.
+/// </para>
+/// <para>
+/// <b>Scope.</b> Every client-, server- and duplex-streaming call, whatever its service. The site's
+/// own <c>SiteStreamService</c> streams are also ended by <c>SiteStreamGrpcServer.CancelAllStreams</c>
+/// on <c>ApplicationStopping</c>; this is the generic backstop for every other stream, today and
+/// later. Unary calls pass through untouched — they are short, and finishing them is what a drain
+/// is for.
+/// </para>
+/// <para>
+/// A stream that arrives once stopping has begun (the peer redialling the moment its stream
+/// ended) is refused with <see cref="StatusCode.Unavailable"/> — the retryable code — rather than
+/// admitted into a node that is about to go away.
+/// </para>
+/// <para>
+/// <b>Exactly one public constructor</b> — <c>Grpc.AspNetCore</c> silently stops invoking an
+/// interceptor that has more than one. Register it LAST (innermost), after the auth gates, so an
+/// unauthenticated stream is still refused by its gate first.
+/// </para>
+/// </remarks>
+public sealed class ShutdownStreamCancellationInterceptor : Interceptor
+{
+    /// <summary>The key <c>GetHttpContext()</c> falls back to for a context that is not ASP.NET Core's own.</summary>
+    private const string HttpContextKey = "__HttpContext";
+
+    private readonly IHostApplicationLifetime _lifetime;
+    private readonly ILogger<ShutdownStreamCancellationInterceptor> _logger;
+
+    /// <summary>Creates the interceptor.</summary>
+    /// <param name="lifetime">The host lifetime whose <c>ApplicationStopping</c> ends open streams.</param>
+    /// <param name="logger">Logger for the shutdown diagnostics.</param>
+    public ShutdownStreamCancellationInterceptor(
+        IHostApplicationLifetime lifetime,
+        ILogger<ShutdownStreamCancellationInterceptor> logger)
+    {
+        ArgumentNullException.ThrowIfNull(lifetime);
+        ArgumentNullException.ThrowIfNull(logger);
+        _lifetime = lifetime;
+        _logger = logger;
+    }
+
+    /// <inheritdoc />
+    public override Task DuplexStreamingServerHandler<TRequest, TResponse>(
+        IAsyncStreamReader<TRequest> requestStream,
+        IServerStreamWriter<TResponse> responseStream,
+        ServerCallContext context,
+        DuplexStreamingServerMethod<TRequest, TResponse> continuation)
+        => RunAsync(context, linked => continuation(requestStream, responseStream, linked));
+
+    /// <inheritdoc />
+    public override Task ServerStreamingServerHandler<TRequest, TResponse>(
+        TRequest request,
+        IServerStreamWriter<TResponse> responseStream,
+        ServerCallContext context,
+        ServerStreamingServerMethod<TRequest, TResponse> continuation)
+        => RunAsync(context, linked => continuation(request, responseStream, linked));
+
+    /// <inheritdoc />
+    public override async Task<TResponse> ClientStreamingServerHandler<TRequest, TResponse>(
+        IAsyncStreamReader<TRequest> requestStream,
+        ServerCallContext context,
+        ClientStreamingServerMethod<TRequest, TResponse> continuation)
+    {
+        TResponse response = default!;
+        await RunAsync(context, async linked => response = await continuation(requestStream, linked).ConfigureAwait(false))
+            .ConfigureAwait(false);
+        return response;
+    }
+
+    private async Task RunAsync(ServerCallContext context, Func<ServerCallContext, Task> handler)
+    {
+        var stopping = _lifetime.ApplicationStopping;
+        if (stopping.IsCancellationRequested)
+        {
+            throw new RpcException(new Status(
+                StatusCode.Unavailable, "This node is shutting down; retry against its peer or after it restarts."));
+        }
+
+        // GetHttpContext() recognises only ASP.NET Core's own context type and otherwise falls back
+        // to a UserState entry — seed that entry so a handler given the wrapper can still reach it.
+        try
+        {
+            context.UserState.TryAdd(HttpContextKey, context.GetHttpContext());
+        }
+        catch (InvalidOperationException)
+        {
+            // Not hosted by ASP.NET Core (a unit test's fake context): nothing to carry over.
+        }
+
+        using var linked = CancellationTokenSource.CreateLinkedTokenSource(context.CancellationToken, stopping);
+        using var onStopping = stopping.Register(
+            static state =>
+            {
+                var (logger, method) = ((ILogger, string))state!;
+                logger.LogInformation("Host stopping: ending the open gRPC stream {Method}.", method);
+            },
+            ((ILogger)_logger, context.Method));
+
+        await handler(new LinkedTokenServerCallContext(context, linked.Token)).ConfigureAwait(false);
+    }
+
+    /// <summary>
+    /// Delegates everything to the real call context except <see cref="ServerCallContext.CancellationToken"/>.
+    /// <see cref="ServerCallContext.UserState"/> is delegated too and carries the <c>__HttpContext</c>
+    /// entry seeded in <c>RunAsync</c>, so <c>GetHttpContext()</c> still resolves on the wrapper.
+    /// </summary>
+    internal sealed class LinkedTokenServerCallContext(ServerCallContext inner, CancellationToken token)
+        : ServerCallContext
+    {
+        /// <inheritdoc />
+        protected override string MethodCore => inner.Method;
+
+        /// <inheritdoc />
+        protected override string HostCore => inner.Host;
+
+        /// <inheritdoc />
+        protected override string PeerCore => inner.Peer;
+
+        /// <inheritdoc />
+        protected override DateTime DeadlineCore => inner.Deadline;
+
+        /// <inheritdoc />
+        protected override Metadata RequestHeadersCore => inner.RequestHeaders;
+
+        /// <inheritdoc />
+        protected override CancellationToken CancellationTokenCore => token;
+
+        /// <inheritdoc />
+        protected override Metadata ResponseTrailersCore => inner.ResponseTrailers;
+
+        /// <inheritdoc />
+        protected override Status StatusCore
+        {
+            get => inner.Status;
+            set => inner.Status = value;
+        }
+
+        /// <inheritdoc />
+        protected override WriteOptions? WriteOptionsCore
+        {
+            get => inner.WriteOptions;
+            set => inner.WriteOptions = value;
+        }
+
+        /// <inheritdoc />
+        protected override AuthContext AuthContextCore => inner.AuthContext;
+
+        /// <inheritdoc />
+        protected override IDictionary<object, object> UserStateCore => inner.UserState;
+
+        /// <inheritdoc />
+        protected override ContextPropagationToken CreatePropagationTokenCore(ContextPropagationOptions? options)
+            => inner.CreatePropagationToken(options);
+
+        /// <inheritdoc />
+        protected override Task WriteResponseHeadersAsyncCore(Metadata responseHeaders)
+            => inner.WriteResponseHeadersAsync(responseHeaders);
+    }
+}