fix(host): leave the cluster before Kestrel drains and end open streams on stop (#140)
A graceful stop of a site node whose peer holds an open LocalDb sync stream left the Akka cluster ~30 s late (the ScadaBridge half of lmxopcua#906, scadaproj#57). WebApplication stops the web server before AkkaHostedService, whose StopAsync ran CoordinatedShutdown, and Kestrel's graceful drain waited the whole ShutdownTimeout for /localdb_sync.v1.LocalDbSync/Sync: LocalDb 0.3.7 observes only the call token, which Kestrel cancels at the END of the drain. CancelAllStreams only covered SiteStreamGrpcServer's own streams. - AkkaHostedService is now an IHostedLifecycleService. StoppingAsync runs before any hosted service's StopAsync (Kestrel's included): marks the stop intentional, raises ApplicationStopping (so CancelAllStreams still precedes actor teardown), then runs and awaits CoordinatedShutdown(ClrExitReason), bounded by the host shutdown token. StopAsync returns the same completed shutdown. Exit codes unchanged: planned stop exits 0; only a termination outside host shutdown (self-down / #132 shut-out) exits 75. No Akka.Hosting here, so no ClusterLeavingReason/-1 exit-code path is involved. - ShutdownStreamCancellationInterceptor, innermost on both the central and the site AddGrpc pipelines: every streaming call's token is linked with ApplicationStopping; new streams are refused Unavailable while stopping. Covers the LocalDb sync stream until LocalDb 0.3.8 is pinned. Evidence (IntegrationTests/GracefulStopWithOpenSyncStreamTests: real WebApplication + Kestrel h2c + MapZbLocalDbSync behind LocalDbSyncAuthInterceptor + a real LocalDb SyncBackgroundService peer + the production AkkaHostedService forming a single-member cluster, 8 s drain budget, IServer wrapped to timestamp the drain): before 8859 ms stop, member removed at 8834 ms, drain began at 0 ms; after 960 ms stop, member removed at 950 ms, drain began at 959 ms. Mutations (all killed): StoppingAsync skips CoordinatedShutdown; interceptor passes the original context (stop 8003 ms); no Unavailable refusal; StoppingAsync does not raise ApplicationStopping; lifecycle interface dropped; site pipeline loses the interceptor. Release build of the solution: 0 warnings. Host.Tests 733/733, IntegrationTests 133/133, Communication.Tests 843/843. Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi
This commit is contained in:
@@ -111,6 +111,13 @@ Before branching on role, `AkkaHostedService.StartAsync` creates one actor uncon
|
||||
|
||||
Shutdown ordering for the site role is explicit: `IHostApplicationLifetime.ApplicationStopping` fires before `IHostedService.StopAsync`, so `SiteStreamGrpcServer.CancelAllStreams` is called first (clients observe a clean cancellation and reconnect), then `AkkaHostedService` runs `CoordinatedShutdown` and tears down actors.
|
||||
|
||||
Since ScadaBridge#140 (the ScadaBridge half of lmxopcua#906) a graceful stop on **both** roles leaves the Akka cluster **before** Kestrel drains. `WebApplication` stops its web server before every hosted service the app registers, and Kestrel's drain waits for open requests up to `HostOptions.ShutdownTimeout` (30 s default) — on a site pair the peer's LocalDb sync stream (`/localdb_sync.v1.LocalDbSync/Sync`) never ends on its own, so the leave used to run ~30 s late and the peer took over that much later. Two pieces fix it:
|
||||
|
||||
- `AkkaHostedService` is an `IHostedLifecycleService`. Its `StoppingAsync` runs before **any** hosted service's `StopAsync` (Kestrel's included): it marks the stop intentional, raises `ApplicationStopping` (idempotent — so `CancelAllStreams` still precedes actor teardown even on a direct `IHost.StopAsync`), then runs and awaits `CoordinatedShutdown` with `ClrExitReason`, bounded by the host's shutdown token. Cluster leave and singleton hand-over complete while Kestrel still serves `/health/active`. `StopAsync` then returns the same completed shutdown. Exit codes are unchanged: a planned stop exits 0; only a termination outside host shutdown (self-down, #132 shut-out restart) exits 75.
|
||||
- `ShutdownStreamCancellationInterceptor`, registered innermost (after the auth gates) on both the central and site `AddGrpc` pipelines, gives every client-, server- and duplex-streaming call a token linked with `ApplicationStopping` and refuses new streams `Unavailable` once stopping. This ends the LocalDb sync stream (the library only does it itself from LocalDb 0.3.8, scadaproj#57). Unary calls pass through.
|
||||
|
||||
Measured by `IntegrationTests/GracefulStopWithOpenSyncStreamTests` (real `WebApplication` + Kestrel h2c + `MapZbLocalDbSync` + a real LocalDb sync peer + the production `AkkaHostedService`, 8 s drain budget): without the fix the stop took ~8.9 s with the member removed ~8.8 s after the drain began; with it the stop takes ~0.95 s and the member is removed before the drain begins.
|
||||
|
||||
```csharp
|
||||
siteLifetime.ApplicationStopping.Register(() => siteGrpcServer.CancelAllStreams());
|
||||
```
|
||||
|
||||
Reference in New Issue
Block a user