fix(host): leave the cluster before Kestrel drains and end open streams on stop (#140)

A graceful stop of a site node whose peer holds an open LocalDb sync stream
left the Akka cluster ~30 s late (the ScadaBridge half of lmxopcua#906,
scadaproj#57). WebApplication stops the web server before AkkaHostedService,
whose StopAsync ran CoordinatedShutdown, and Kestrel's graceful drain waited
the whole ShutdownTimeout for /localdb_sync.v1.LocalDbSync/Sync: LocalDb 0.3.7
observes only the call token, which Kestrel cancels at the END of the drain.
CancelAllStreams only covered SiteStreamGrpcServer's own streams.

- AkkaHostedService is now an IHostedLifecycleService. StoppingAsync runs
  before any hosted service's StopAsync (Kestrel's included): marks the stop
  intentional, raises ApplicationStopping (so CancelAllStreams still precedes
  actor teardown), then runs and awaits CoordinatedShutdown(ClrExitReason),
  bounded by the host shutdown token. StopAsync returns the same completed
  shutdown. Exit codes unchanged: planned stop exits 0; only a termination
  outside host shutdown (self-down / #132 shut-out) exits 75. No Akka.Hosting
  here, so no ClusterLeavingReason/-1 exit-code path is involved.
- ShutdownStreamCancellationInterceptor, innermost on both the central and the
  site AddGrpc pipelines: every streaming call's token is linked with
  ApplicationStopping; new streams are refused Unavailable while stopping.
  Covers the LocalDb sync stream until LocalDb 0.3.8 is pinned.

Evidence (IntegrationTests/GracefulStopWithOpenSyncStreamTests: real
WebApplication + Kestrel h2c + MapZbLocalDbSync behind LocalDbSyncAuthInterceptor
+ a real LocalDb SyncBackgroundService peer + the production AkkaHostedService
forming a single-member cluster, 8 s drain budget, IServer wrapped to timestamp
the drain): before 8859 ms stop, member removed at 8834 ms, drain began at 0 ms;
after 960 ms stop, member removed at 950 ms, drain began at 959 ms.

Mutations (all killed): StoppingAsync skips CoordinatedShutdown; interceptor
passes the original context (stop 8003 ms); no Unavailable refusal; StoppingAsync
does not raise ApplicationStopping; lifecycle interface dropped; site pipeline
loses the interceptor.

Release build of the solution: 0 warnings. Host.Tests 733/733,
IntegrationTests 133/133, Communication.Tests 843/843.

Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi
This commit is contained in:
Joseph Doherty
2026-11-13 04:16:42 -05:00
parent 3badd1f9fe
commit 1a6108c402
7 changed files with 886 additions and 5 deletions
+7
View File
@@ -111,6 +111,13 @@ Before branching on role, `AkkaHostedService.StartAsync` creates one actor uncon
Shutdown ordering for the site role is explicit: `IHostApplicationLifetime.ApplicationStopping` fires before `IHostedService.StopAsync`, so `SiteStreamGrpcServer.CancelAllStreams` is called first (clients observe a clean cancellation and reconnect), then `AkkaHostedService` runs `CoordinatedShutdown` and tears down actors.
Since ScadaBridge#140 (the ScadaBridge half of lmxopcua#906) a graceful stop on **both** roles leaves the Akka cluster **before** Kestrel drains. `WebApplication` stops its web server before every hosted service the app registers, and Kestrel's drain waits for open requests up to `HostOptions.ShutdownTimeout` (30 s default) — on a site pair the peer's LocalDb sync stream (`/localdb_sync.v1.LocalDbSync/Sync`) never ends on its own, so the leave used to run ~30 s late and the peer took over that much later. Two pieces fix it:
- `AkkaHostedService` is an `IHostedLifecycleService`. Its `StoppingAsync` runs before **any** hosted service's `StopAsync` (Kestrel's included): it marks the stop intentional, raises `ApplicationStopping` (idempotent — so `CancelAllStreams` still precedes actor teardown even on a direct `IHost.StopAsync`), then runs and awaits `CoordinatedShutdown` with `ClrExitReason`, bounded by the host's shutdown token. Cluster leave and singleton hand-over complete while Kestrel still serves `/health/active`. `StopAsync` then returns the same completed shutdown. Exit codes are unchanged: a planned stop exits 0; only a termination outside host shutdown (self-down, #132 shut-out restart) exits 75.
- `ShutdownStreamCancellationInterceptor`, registered innermost (after the auth gates) on both the central and site `AddGrpc` pipelines, gives every client-, server- and duplex-streaming call a token linked with `ApplicationStopping` and refuses new streams `Unavailable` once stopping. This ends the LocalDb sync stream (the library only does it itself from LocalDb 0.3.8, scadaproj#57). Unary calls pass through.
Measured by `IntegrationTests/GracefulStopWithOpenSyncStreamTests` (real `WebApplication` + Kestrel h2c + `MapZbLocalDbSync` + a real LocalDb sync peer + the production `AkkaHostedService`, 8 s drain budget): without the fix the stop took ~8.9 s with the member removed ~8.8 s after the drain began; with it the stop takes ~0.95 s and the member is removed before the drain begins.
```csharp
siteLifetime.ApplicationStopping.Register(() => siteGrpcServer.CancelAllStreams());
```