dohertj2/ScadaBridge
1
0
Fork 0
Code Issues 5 Pull Requests Actions Packages Projects Releases Wiki Activity

Graceful stop of a site node leaves the Akka cluster ~30 s late: Kestrel drains the peer's open LocalDb sync stream first (lmxopcua#906, scadaproj#57) #140

New Issue
Closed
opened 2026-11-13 03:55:13 -05:00 by dohertj2 · 2 comments
dohertj2 commented 2026-11-13 03:55:13 -05:00
Owner
Copy Link
Copy Source

Same exposure as dohertj2/lmxopcua#906, recorded in dohertj2/scadaproj#57.

Problem. On a graceful stop (Windows service stop / SIGTERM) WebApplication stops Kestrel before AkkaHostedService.StopAsync, which is where CoordinatedShutdown (cluster leave + singleton hand-over) runs. Kestrel's graceful drain waits up to HostOptions.ShutdownTimeout (30 s default) for open requests. On a site pair the peer's LocalDb sync stream (/localdb_sync.v1.LocalDbSync/Sync, served by MapZbLocalDbSync()) never ends on its own: ZB.MOM.WW.LocalDb.Replication 0.3.7 observes only the call token, which Kestrel cancels at the END of the drain. Program.cs cancels only SiteStreamGrpcServer's own streams on ApplicationStopping (CancelAllStreams). So the stopping site node leaves the cluster about 30 s late and failover to its peer is delayed by that much. Central has no streaming RPCs today but has the same ordering.

Fix plan.

  1. Leave the Akka cluster before Kestrel drains, on both Central and Site: run the existing CoordinatedShutdown (ClrExitReason, unchanged exit-code behaviour; the exit-75 self-down watchdog stays quiet) from an IHostedLifecycleService.StoppingAsync, which runs before any hosted service's StopAsync.
  2. Host-side gRPC interceptor that links ApplicationStopping into every streaming call's token and refuses new streams Unavailable while stopping. Covers the LocalDb sync stream until LocalDb 0.3.8 (scadaproj 3ddb083) is published and pinned.

No deploy.

Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi

Same exposure as dohertj2/lmxopcua#906, recorded in dohertj2/scadaproj#57. **Problem.** On a graceful stop (Windows service stop / SIGTERM) `WebApplication` stops Kestrel before `AkkaHostedService.StopAsync`, which is where `CoordinatedShutdown` (cluster leave + singleton hand-over) runs. Kestrel's graceful drain waits up to `HostOptions.ShutdownTimeout` (30 s default) for open requests. On a site pair the peer's LocalDb sync stream (`/localdb_sync.v1.LocalDbSync/Sync`, served by `MapZbLocalDbSync()`) never ends on its own: `ZB.MOM.WW.LocalDb.Replication` 0.3.7 observes only the call token, which Kestrel cancels at the END of the drain. `Program.cs` cancels only `SiteStreamGrpcServer`'s own streams on `ApplicationStopping` (`CancelAllStreams`). So the stopping site node leaves the cluster about 30 s late and failover to its peer is delayed by that much. Central has no streaming RPCs today but has the same ordering. **Fix plan.** 1. Leave the Akka cluster before Kestrel drains, on both Central and Site: run the existing CoordinatedShutdown (`ClrExitReason`, unchanged exit-code behaviour; the exit-75 self-down watchdog stays quiet) from an `IHostedLifecycleService.StoppingAsync`, which runs before any hosted service's `StopAsync`. 2. Host-side gRPC interceptor that links `ApplicationStopping` into every streaming call's token and refuses new streams `Unavailable` while stopping. Covers the LocalDb sync stream until LocalDb 0.3.8 (scadaproj 3ddb083) is published and pinned. No deploy. Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi
dohertj2 added the type:bugpriority:p3cross-repo labels 2026-11-13 03:55:13 -05:00
dohertj2 commented 2026-11-13 04:17:10 -05:00
Author
Owner
Copy Link
Copy Source

Fixed on main 1a6108c4. No deploy.

Root cause. WebApplication stops its web server before AkkaHostedService, whose StopAsync ran CoordinatedShutdown. Kestrel's graceful drain waits up to ShutdownTimeout (30 s) for open requests, and the peer's /localdb_sync.v1.LocalDbSync/Sync stream never ends on its own (LocalDb 0.3.7 observes only the call token, which Kestrel cancels at the end of the drain). CancelAllStreams covered only SiteStreamGrpcServer's streams.

Fix.

  • AkkaHostedService is now an IHostedLifecycleService. StoppingAsync runs before any hosted service's StopAsync, Kestrel's included. It marks the stop intentional, raises ApplicationStopping (so CancelAllStreams still runs before the actors are torn down), then runs and awaits CoordinatedShutdown(ClrExitReason), bounded by the host shutdown token. Exit codes are unchanged: a planned stop exits 0, and only a termination outside host shutdown exits 75. ScadaBridge does not use Akka.Hosting, so the ClusterLeavingReason → -1 path OtOpcUa had to avoid does not apply. The same ClrExitReason is kept anyway.
  • ShutdownStreamCancellationInterceptor sits innermost, after the auth gates, on both the central and the site AddGrpc pipelines. It links ApplicationStopping into every streaming call's token and refuses new streams with Unavailable while stopping. Central has no streaming RPCs today, so there it is only a backstop.

Evidence (GracefulStopWithOpenSyncStreamTests: real WebApplication, Kestrel h2c, MapZbLocalDbSync behind LocalDbSyncAuthInterceptor, a real LocalDb sync peer and the production AkkaHostedService in a single-member cluster; 8 s drain budget; the drain start is timestamped by a wrapped IServer):

stop member removed drain began
before (pre-fix wiring, positive control) 8859 ms 8834 ms 0 ms
after 960 ms 950 ms 959 ms

Mutations, all killed:

  • StoppingAsync skips CoordinatedShutdown → order assertion fails (drain at 1 ms, removed at 937 ms).
  • The interceptor passes the original context → stop takes 8003 ms and fails. The leave still happens first (947 ms), so failover is prompt; only the process stop waits.
  • No Unavailable refusal while stopping.
  • StoppingAsync does not raise ApplicationStopping.
  • The lifecycle interface is dropped.
  • The site pipeline loses the interceptor.

The last four are caught by unit tests or the wiring test.

Tests. Release build of the solution: 0 warnings. Host.Tests 733/733, IntegrationTests 133/133, Communication.Tests 843/843.

Pinning LocalDb 0.3.8 stays in scadaproj#57. The interceptor can stay after that.

Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi

**Fixed on `main` 1a6108c4.** No deploy. **Root cause.** `WebApplication` stops its web server before `AkkaHostedService`, whose `StopAsync` ran `CoordinatedShutdown`. Kestrel's graceful drain waits up to `ShutdownTimeout` (30 s) for open requests, and the peer's `/localdb_sync.v1.LocalDbSync/Sync` stream never ends on its own (LocalDb 0.3.7 observes only the call token, which Kestrel cancels at the end of the drain). `CancelAllStreams` covered only `SiteStreamGrpcServer`'s streams. **Fix.** - `AkkaHostedService` is now an `IHostedLifecycleService`. `StoppingAsync` runs before any hosted service's `StopAsync`, Kestrel's included. It marks the stop intentional, raises `ApplicationStopping` (so `CancelAllStreams` still runs before the actors are torn down), then runs and awaits `CoordinatedShutdown(ClrExitReason)`, bounded by the host shutdown token. Exit codes are unchanged: a planned stop exits 0, and only a termination outside host shutdown exits 75. ScadaBridge does not use Akka.Hosting, so the `ClusterLeavingReason` → -1 path OtOpcUa had to avoid does not apply. The same `ClrExitReason` is kept anyway. - `ShutdownStreamCancellationInterceptor` sits innermost, after the auth gates, on both the central and the site `AddGrpc` pipelines. It links `ApplicationStopping` into every streaming call's token and refuses new streams with `Unavailable` while stopping. Central has no streaming RPCs today, so there it is only a backstop. **Evidence** (`GracefulStopWithOpenSyncStreamTests`: real WebApplication, Kestrel h2c, `MapZbLocalDbSync` behind `LocalDbSyncAuthInterceptor`, a real LocalDb sync peer and the production `AkkaHostedService` in a single-member cluster; 8 s drain budget; the drain start is timestamped by a wrapped `IServer`): | | stop | member removed | drain began | |---|---|---|---| | before (pre-fix wiring, positive control) | 8859 ms | 8834 ms | 0 ms | | after | 960 ms | 950 ms | 959 ms | **Mutations, all killed:** - `StoppingAsync` skips `CoordinatedShutdown` → order assertion fails (drain at 1 ms, removed at 937 ms). - The interceptor passes the original context → stop takes 8003 ms and fails. The leave still happens first (947 ms), so failover is prompt; only the process stop waits. - No `Unavailable` refusal while stopping. - `StoppingAsync` does not raise `ApplicationStopping`. - The lifecycle interface is dropped. - The site pipeline loses the interceptor. The last four are caught by unit tests or the wiring test. **Tests.** Release build of the solution: 0 warnings. Host.Tests 733/733, IntegrationTests 133/133, Communication.Tests 843/843. Pinning LocalDb 0.3.8 stays in scadaproj#57. The interceptor can stay after that. Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi
dohertj2 commented 2026-11-13 08:14:53 -05:00
Author
Owner
Copy Link
Copy Source

Now on QA vq03/vq04 too (2026-11-13, 8e32f353, record 8ba5d5d7). The window's own stops ran the old 0e540291 binaries, so they still showed the bug: vq04's Site stop waited the full 30 s on the peer's LocalDb sync stream, then logged the old [FTL] (#76). Stops of the new build were not exercised in this window: there were no drills or restarts after it.

**Now on QA vq03/vq04 too (2026-11-13, `8e32f353`, record `8ba5d5d7`).** The window's own stops ran the old `0e540291` binaries, so they still showed the bug: vq04's Site stop waited the full 30 s on the peer's LocalDb sync stream, then logged the old `[FTL]` (#76). Stops of the new build were not exercised in this window: there were no drills or restarts after it.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
blocked
Blocked by dependency or external hold
 cross-repo
Needs coordinated change in another family repo (wire contract etc.)
 migrated
Migrated from a markdown backlog (2026-11 standardization)
 priority:p0
Must-have / blocking
 priority:p1
High priority
 priority:p2
Normal priority
 priority:p3
Low priority / nice to have
 type:bug
Defect in shipped behaviour
 type:debt
Technical debt / hygiene
 type:deploy-gate
Rollout/deploy/migration step not yet done on an environment
 type:docs
Documentation work
 type:feature
Feature implementation
 type:followup
Follow-up deferred from a plan, PR or gate
 type:infra
Infrastructure and scaffolding
 type:review-finding
Unfixed finding from an architecture/code review
 type:test
Test work
No Label cross-repo priority:p3 type:bug
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
dohertj2 (dohertj2)
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: dohertj2/ScadaBridge#140
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?