fix(host): leave the cluster before Kestrel drains and end open streams on stop (#906)
v2-ci / unit-tests (push) Waiting to run

A planned stop of the Primary took ~30 s on QA: WebApplication stops the web
server before Akka.Hosting's service, and Kestrel's graceful drain waited the
whole ShutdownTimeout for the peer's open LocalDb sync stream (the library's
LocalDbSyncService only observes the call token, which fires at the END of the
drain). Only then did CoordinatedShutdown run, so the stopping node held the
Primary role and the peer sat at ServiceLevel 240 for the whole window.

- GracefulClusterLeaveService (IHostedLifecycleService): StoppingAsync runs
  before any hosted service's StopAsync; it raises ApplicationStopping, then
  runs and awaits CoordinatedShutdown with ClrExitReason (not Cluster.Leave,
  whose ClusterLeavingReason makes Akka.Hosting set ExitCode -1). The cluster
  is left while Kestrel is still up.
- ShutdownStreamCancellationInterceptor (innermost on the shared gRPC
  pipeline): every streaming call gets a context whose token is linked with
  ApplicationStopping; streams arriving once stopping are refused Unavailable.
  Host-side mitigation until LocalDb 0.3.8 is published and pinned
  (scadaproj 3ddb083, scadaproj#57); it also covers the mesh streams.

Test GracefulStopWithOpenSyncStreamTests (real WebApplication + Kestrel h2c +
MapZbLocalDbSync + single-node Akka cluster + real peer initiator, 8 s drain
budget): stop 895 ms, cluster left at 892 ms, drain began at 894 ms. Positive
control without the fix: stop 8863 ms, cluster left at 8841 ms after the drain
began at 0 ms. Mutations: no CoordinatedShutdown in StoppingAsync -> order
assertion fails (drain at 1 ms, removed at 893 ms); interceptor passes the
original context -> stop 8006 ms, fails.

Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi
This commit is contained in:
Joseph Doherty
2026-11-13 03:32:13 -05:00
parent 2688bd5cfd
commit dcff5ae994
4 changed files with 629 additions and 0 deletions
@@ -0,0 +1,180 @@
using Grpc.Core;
using Grpc.Core.Interceptors;
using Microsoft.Extensions.Hosting;
using Microsoft.Extensions.Logging;
namespace ZB.MOM.WW.OtOpcUa.Host.Configuration;
/// <summary>
/// Ends every open streaming gRPC call on this host's shared gRPC pipeline when the host starts
/// stopping, and refuses new ones from then on (lmxopcua#906).
/// </summary>
/// <remarks>
/// <para>
/// <b>Why.</b> A server-side streaming handler observes <see cref="ServerCallContext.CancellationToken"/>,
/// which ASP.NET Core cancels only when the client goes away or Kestrel aborts the request — and
/// Kestrel aborts only at the END of its graceful drain, i.e. after the host's whole
/// <c>ShutdownTimeout</c> (30 s by default). The peer's LocalDb sync stream
/// (<c>/localdb_sync.v1.LocalDbSync/Sync</c>) never ends on its own, so every graceful stop of
/// the node it is connected to took the full 30 s. Through LocalDb 0.3.7 the library's
/// <c>LocalDbSyncService</c> does not observe <c>ApplicationStopping</c> itself (fixed on scadaproj
/// main <c>3ddb083</c>, publish tracked as scadaproj#57), so the host supplies it here: the handler is
/// given a context whose token is the call's own token linked with <see cref="IHostApplicationLifetime.ApplicationStopping"/>.
/// Handlers that already treat a cancelled call token as "client gone / server shutting down"
/// — the LocalDb sync service does — then return cleanly, and Kestrel's drain has nothing left
/// to wait for.
/// </para>
/// <para>
/// <b>Scope.</b> Every client-, server- and duplex-streaming call, whatever its service: the
/// mesh telemetry and command streams are long-lived too and would hold the drain the same
/// way. Unary calls are left alone — they are short and finishing them is the point of a drain.
/// </para>
/// <para>
/// A stream that arrives once stopping has begun (the peer redialling the moment its stream
/// ended) is refused with <see cref="StatusCode.Unavailable"/> — the retryable code — rather
/// than admitted into a node that is about to go away.
/// </para>
/// <para>
/// <b>Exactly one public constructor</b> — <c>Grpc.AspNetCore</c> silently stops invoking an
/// interceptor that has more than one.
/// </para>
/// </remarks>
public sealed class ShutdownStreamCancellationInterceptor : Interceptor
{
private readonly IHostApplicationLifetime _lifetime;
private readonly ILogger<ShutdownStreamCancellationInterceptor> _logger;
/// <summary>Creates the interceptor.</summary>
/// <param name="lifetime">The host lifetime whose <c>ApplicationStopping</c> ends open streams.</param>
/// <param name="logger">Logger for the shutdown diagnostics.</param>
public ShutdownStreamCancellationInterceptor(
IHostApplicationLifetime lifetime,
ILogger<ShutdownStreamCancellationInterceptor> logger)
{
ArgumentNullException.ThrowIfNull(lifetime);
ArgumentNullException.ThrowIfNull(logger);
_lifetime = lifetime;
_logger = logger;
}
/// <inheritdoc />
public override Task DuplexStreamingServerHandler<TRequest, TResponse>(
IAsyncStreamReader<TRequest> requestStream,
IServerStreamWriter<TResponse> responseStream,
ServerCallContext context,
DuplexStreamingServerMethod<TRequest, TResponse> continuation)
=> RunAsync(context, linked => continuation(requestStream, responseStream, linked));
/// <inheritdoc />
public override Task ServerStreamingServerHandler<TRequest, TResponse>(
TRequest request,
IServerStreamWriter<TResponse> responseStream,
ServerCallContext context,
ServerStreamingServerMethod<TRequest, TResponse> continuation)
=> RunAsync(context, linked => continuation(request, responseStream, linked));
/// <inheritdoc />
public override async Task<TResponse> ClientStreamingServerHandler<TRequest, TResponse>(
IAsyncStreamReader<TRequest> requestStream,
ServerCallContext context,
ClientStreamingServerMethod<TRequest, TResponse> continuation)
{
TResponse response = default!;
await RunAsync(context, async linked => response = await continuation(requestStream, linked).ConfigureAwait(false))
.ConfigureAwait(false);
return response;
}
private async Task RunAsync(ServerCallContext context, Func<ServerCallContext, Task> handler)
{
var stopping = _lifetime.ApplicationStopping;
if (stopping.IsCancellationRequested)
{
throw new RpcException(new Status(
StatusCode.Unavailable, "This node is shutting down; retry against its peer or after it restarts."));
}
// GetHttpContext() recognises only ASP.NET Core's own context type, falling back to a
// UserState entry for any other — so seed that entry before handing the handler a wrapper.
try
{
context.UserState.TryAdd(HttpContextKey, context.GetHttpContext());
}
catch (InvalidOperationException)
{
// Not hosted by ASP.NET Core (a unit test's fake context): nothing to carry over.
}
using var linked = CancellationTokenSource.CreateLinkedTokenSource(context.CancellationToken, stopping);
using var onStopping = stopping.Register(
static state =>
{
var (logger, method) = ((ILogger, string))state!;
logger.LogInformation("Host stopping: ending the open gRPC stream {Method}.", method);
},
((ILogger)_logger, context.Method));
await handler(new LinkedTokenServerCallContext(context, linked.Token)).ConfigureAwait(false);
}
/// <summary>The key <c>GetHttpContext()</c> falls back to for a context that is not ASP.NET Core's own.</summary>
private const string HttpContextKey = "__HttpContext";
/// <summary>
/// Delegates everything to the real call context except <see cref="ServerCallContext.CancellationToken"/>.
/// <see cref="ServerCallContext.UserState"/> is delegated too and carries the <c>__HttpContext</c> entry
/// seeded in <c>RunAsync</c>, so <c>GetHttpContext()</c> still resolves on the wrapper.
/// </summary>
internal sealed class LinkedTokenServerCallContext(ServerCallContext inner, CancellationToken token)
: ServerCallContext
{
/// <inheritdoc />
protected override string MethodCore => inner.Method;
/// <inheritdoc />
protected override string HostCore => inner.Host;
/// <inheritdoc />
protected override string PeerCore => inner.Peer;
/// <inheritdoc />
protected override DateTime DeadlineCore => inner.Deadline;
/// <inheritdoc />
protected override Metadata RequestHeadersCore => inner.RequestHeaders;
/// <inheritdoc />
protected override CancellationToken CancellationTokenCore => token;
/// <inheritdoc />
protected override Metadata ResponseTrailersCore => inner.ResponseTrailers;
/// <inheritdoc />
protected override Status StatusCore
{
get => inner.Status;
set => inner.Status = value;
}
/// <inheritdoc />
protected override WriteOptions? WriteOptionsCore
{
get => inner.WriteOptions;
set => inner.WriteOptions = value;
}
/// <inheritdoc />
protected override AuthContext AuthContextCore => inner.AuthContext;
/// <inheritdoc />
protected override IDictionary<object, object> UserStateCore => inner.UserState;
/// <inheritdoc />
protected override ContextPropagationToken CreatePropagationTokenCore(ContextPropagationOptions? options)
=> inner.CreatePropagationToken(options);
/// <inheritdoc />
protected override Task WriteResponseHeadersAsyncCore(Metadata responseHeaders)
=> inner.WriteResponseHeadersAsync(responseHeaders);
}
}