fix(host): leave the cluster before Kestrel drains and end open streams on stop (#906)
v2-ci / unit-tests (push) Waiting to run
v2-ci / unit-tests (push) Waiting to run
A planned stop of the Primary took ~30 s on QA: WebApplication stops the web server before Akka.Hosting's service, and Kestrel's graceful drain waited the whole ShutdownTimeout for the peer's open LocalDb sync stream (the library's LocalDbSyncService only observes the call token, which fires at the END of the drain). Only then did CoordinatedShutdown run, so the stopping node held the Primary role and the peer sat at ServiceLevel 240 for the whole window. - GracefulClusterLeaveService (IHostedLifecycleService): StoppingAsync runs before any hosted service's StopAsync; it raises ApplicationStopping, then runs and awaits CoordinatedShutdown with ClrExitReason (not Cluster.Leave, whose ClusterLeavingReason makes Akka.Hosting set ExitCode -1). The cluster is left while Kestrel is still up. - ShutdownStreamCancellationInterceptor (innermost on the shared gRPC pipeline): every streaming call gets a context whose token is linked with ApplicationStopping; streams arriving once stopping are refused Unavailable. Host-side mitigation until LocalDb 0.3.8 is published and pinned (scadaproj 3ddb083, scadaproj#57); it also covers the mesh streams. Test GracefulStopWithOpenSyncStreamTests (real WebApplication + Kestrel h2c + MapZbLocalDbSync + single-node Akka cluster + real peer initiator, 8 s drain budget): stop 895 ms, cluster left at 892 ms, drain began at 894 ms. Positive control without the fix: stop 8863 ms, cluster left at 8841 ms after the drain began at 0 ms. Mutations: no CoordinatedShutdown in StoppingAsync -> order assertion fails (drain at 1 ms, removed at 893 ms); interceptor passes the original context -> stop 8006 ms, fails. Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi
This commit is contained in:
+180
@@ -0,0 +1,180 @@
|
||||
using Grpc.Core;
|
||||
using Grpc.Core.Interceptors;
|
||||
using Microsoft.Extensions.Hosting;
|
||||
using Microsoft.Extensions.Logging;
|
||||
|
||||
namespace ZB.MOM.WW.OtOpcUa.Host.Configuration;
|
||||
|
||||
/// <summary>
|
||||
/// Ends every open streaming gRPC call on this host's shared gRPC pipeline when the host starts
|
||||
/// stopping, and refuses new ones from then on (lmxopcua#906).
|
||||
/// </summary>
|
||||
/// <remarks>
|
||||
/// <para>
|
||||
/// <b>Why.</b> A server-side streaming handler observes <see cref="ServerCallContext.CancellationToken"/>,
|
||||
/// which ASP.NET Core cancels only when the client goes away or Kestrel aborts the request — and
|
||||
/// Kestrel aborts only at the END of its graceful drain, i.e. after the host's whole
|
||||
/// <c>ShutdownTimeout</c> (30 s by default). The peer's LocalDb sync stream
|
||||
/// (<c>/localdb_sync.v1.LocalDbSync/Sync</c>) never ends on its own, so every graceful stop of
|
||||
/// the node it is connected to took the full 30 s. Through LocalDb 0.3.7 the library's
|
||||
/// <c>LocalDbSyncService</c> does not observe <c>ApplicationStopping</c> itself (fixed on scadaproj
|
||||
/// main <c>3ddb083</c>, publish tracked as scadaproj#57), so the host supplies it here: the handler is
|
||||
/// given a context whose token is the call's own token linked with <see cref="IHostApplicationLifetime.ApplicationStopping"/>.
|
||||
/// Handlers that already treat a cancelled call token as "client gone / server shutting down"
|
||||
/// — the LocalDb sync service does — then return cleanly, and Kestrel's drain has nothing left
|
||||
/// to wait for.
|
||||
/// </para>
|
||||
/// <para>
|
||||
/// <b>Scope.</b> Every client-, server- and duplex-streaming call, whatever its service: the
|
||||
/// mesh telemetry and command streams are long-lived too and would hold the drain the same
|
||||
/// way. Unary calls are left alone — they are short and finishing them is the point of a drain.
|
||||
/// </para>
|
||||
/// <para>
|
||||
/// A stream that arrives once stopping has begun (the peer redialling the moment its stream
|
||||
/// ended) is refused with <see cref="StatusCode.Unavailable"/> — the retryable code — rather
|
||||
/// than admitted into a node that is about to go away.
|
||||
/// </para>
|
||||
/// <para>
|
||||
/// <b>Exactly one public constructor</b> — <c>Grpc.AspNetCore</c> silently stops invoking an
|
||||
/// interceptor that has more than one.
|
||||
/// </para>
|
||||
/// </remarks>
|
||||
public sealed class ShutdownStreamCancellationInterceptor : Interceptor
|
||||
{
|
||||
private readonly IHostApplicationLifetime _lifetime;
|
||||
private readonly ILogger<ShutdownStreamCancellationInterceptor> _logger;
|
||||
|
||||
/// <summary>Creates the interceptor.</summary>
|
||||
/// <param name="lifetime">The host lifetime whose <c>ApplicationStopping</c> ends open streams.</param>
|
||||
/// <param name="logger">Logger for the shutdown diagnostics.</param>
|
||||
public ShutdownStreamCancellationInterceptor(
|
||||
IHostApplicationLifetime lifetime,
|
||||
ILogger<ShutdownStreamCancellationInterceptor> logger)
|
||||
{
|
||||
ArgumentNullException.ThrowIfNull(lifetime);
|
||||
ArgumentNullException.ThrowIfNull(logger);
|
||||
_lifetime = lifetime;
|
||||
_logger = logger;
|
||||
}
|
||||
|
||||
/// <inheritdoc />
|
||||
public override Task DuplexStreamingServerHandler<TRequest, TResponse>(
|
||||
IAsyncStreamReader<TRequest> requestStream,
|
||||
IServerStreamWriter<TResponse> responseStream,
|
||||
ServerCallContext context,
|
||||
DuplexStreamingServerMethod<TRequest, TResponse> continuation)
|
||||
=> RunAsync(context, linked => continuation(requestStream, responseStream, linked));
|
||||
|
||||
/// <inheritdoc />
|
||||
public override Task ServerStreamingServerHandler<TRequest, TResponse>(
|
||||
TRequest request,
|
||||
IServerStreamWriter<TResponse> responseStream,
|
||||
ServerCallContext context,
|
||||
ServerStreamingServerMethod<TRequest, TResponse> continuation)
|
||||
=> RunAsync(context, linked => continuation(request, responseStream, linked));
|
||||
|
||||
/// <inheritdoc />
|
||||
public override async Task<TResponse> ClientStreamingServerHandler<TRequest, TResponse>(
|
||||
IAsyncStreamReader<TRequest> requestStream,
|
||||
ServerCallContext context,
|
||||
ClientStreamingServerMethod<TRequest, TResponse> continuation)
|
||||
{
|
||||
TResponse response = default!;
|
||||
await RunAsync(context, async linked => response = await continuation(requestStream, linked).ConfigureAwait(false))
|
||||
.ConfigureAwait(false);
|
||||
return response;
|
||||
}
|
||||
|
||||
private async Task RunAsync(ServerCallContext context, Func<ServerCallContext, Task> handler)
|
||||
{
|
||||
var stopping = _lifetime.ApplicationStopping;
|
||||
if (stopping.IsCancellationRequested)
|
||||
{
|
||||
throw new RpcException(new Status(
|
||||
StatusCode.Unavailable, "This node is shutting down; retry against its peer or after it restarts."));
|
||||
}
|
||||
|
||||
// GetHttpContext() recognises only ASP.NET Core's own context type, falling back to a
|
||||
// UserState entry for any other — so seed that entry before handing the handler a wrapper.
|
||||
try
|
||||
{
|
||||
context.UserState.TryAdd(HttpContextKey, context.GetHttpContext());
|
||||
}
|
||||
catch (InvalidOperationException)
|
||||
{
|
||||
// Not hosted by ASP.NET Core (a unit test's fake context): nothing to carry over.
|
||||
}
|
||||
|
||||
using var linked = CancellationTokenSource.CreateLinkedTokenSource(context.CancellationToken, stopping);
|
||||
using var onStopping = stopping.Register(
|
||||
static state =>
|
||||
{
|
||||
var (logger, method) = ((ILogger, string))state!;
|
||||
logger.LogInformation("Host stopping: ending the open gRPC stream {Method}.", method);
|
||||
},
|
||||
((ILogger)_logger, context.Method));
|
||||
|
||||
await handler(new LinkedTokenServerCallContext(context, linked.Token)).ConfigureAwait(false);
|
||||
}
|
||||
|
||||
/// <summary>The key <c>GetHttpContext()</c> falls back to for a context that is not ASP.NET Core's own.</summary>
|
||||
private const string HttpContextKey = "__HttpContext";
|
||||
|
||||
/// <summary>
|
||||
/// Delegates everything to the real call context except <see cref="ServerCallContext.CancellationToken"/>.
|
||||
/// <see cref="ServerCallContext.UserState"/> is delegated too and carries the <c>__HttpContext</c> entry
|
||||
/// seeded in <c>RunAsync</c>, so <c>GetHttpContext()</c> still resolves on the wrapper.
|
||||
/// </summary>
|
||||
internal sealed class LinkedTokenServerCallContext(ServerCallContext inner, CancellationToken token)
|
||||
: ServerCallContext
|
||||
{
|
||||
/// <inheritdoc />
|
||||
protected override string MethodCore => inner.Method;
|
||||
|
||||
/// <inheritdoc />
|
||||
protected override string HostCore => inner.Host;
|
||||
|
||||
/// <inheritdoc />
|
||||
protected override string PeerCore => inner.Peer;
|
||||
|
||||
/// <inheritdoc />
|
||||
protected override DateTime DeadlineCore => inner.Deadline;
|
||||
|
||||
/// <inheritdoc />
|
||||
protected override Metadata RequestHeadersCore => inner.RequestHeaders;
|
||||
|
||||
/// <inheritdoc />
|
||||
protected override CancellationToken CancellationTokenCore => token;
|
||||
|
||||
/// <inheritdoc />
|
||||
protected override Metadata ResponseTrailersCore => inner.ResponseTrailers;
|
||||
|
||||
/// <inheritdoc />
|
||||
protected override Status StatusCore
|
||||
{
|
||||
get => inner.Status;
|
||||
set => inner.Status = value;
|
||||
}
|
||||
|
||||
/// <inheritdoc />
|
||||
protected override WriteOptions? WriteOptionsCore
|
||||
{
|
||||
get => inner.WriteOptions;
|
||||
set => inner.WriteOptions = value;
|
||||
}
|
||||
|
||||
/// <inheritdoc />
|
||||
protected override AuthContext AuthContextCore => inner.AuthContext;
|
||||
|
||||
/// <inheritdoc />
|
||||
protected override IDictionary<object, object> UserStateCore => inner.UserState;
|
||||
|
||||
/// <inheritdoc />
|
||||
protected override ContextPropagationToken CreatePropagationTokenCore(ContextPropagationOptions? options)
|
||||
=> inner.CreatePropagationToken(options);
|
||||
|
||||
/// <inheritdoc />
|
||||
protected override Task WriteResponseHeadersAsyncCore(Metadata responseHeaders)
|
||||
=> inner.WriteResponseHeadersAsync(responseHeaders);
|
||||
}
|
||||
}
|
||||
Reference in New Issue
Block a user