fix(host): unhandled boot exception now kills the process instead of wedging the container (#34)
Root cause: dotnet runs as container PID 1 and Linux ignores default-action signals sent to PID 1, so the runtime's unhandled-exception path (banner, then abort() -> SIGABRT) could never terminate the process — it printed the trace and spun the main thread at 100% CPU with the container `running`, so `restart: unless-stopped` never fired. Reproduced deterministically: same StartupValidator throw exits 134 under an init process and wedges without one. Two layers, each covering the other's gap: - Program.cs registers an AppDomain.UnhandledException handler before the first statement that can throw: prints the trace, best-effort flushes Serilog, Environment.Exit(134) — exit() is a syscall PID 1 CAN perform, 134 preserves the 128+SIGABRT crash code, and it covers every thread, not just the boot window. It cannot fire under WebApplicationFactory (the test host catches entry-point exceptions), so the designed boot-refusal exceptions still propagate to tests unchanged. - docker-compose: init: true on all 8 nodes for the crash paths that bypass the managed event (Environment.FailFast, runtime-internal aborts). The CoordinatedShutdown no-Environment.Exit guard gains a precise carve-out (exactly one call, only inside the handler); Environment.Exit still fires the CLR shutdown hook Akka binds via run-by-clr-shutdown-hook = on, so the crash path skips nothing abort() kept. New pin test keeps the handler ahead of the configuration build. Live-verified on the rig image: crash now yields Exited (134) + RestartCount climbing under `unless-stopped`, trace intact, with and without init; full 8-node rig redeployed healthy with docker-init as PID 1. Closes #34. Claude-Session: https://claude.ai/code/session_014WNM4vjoVksyyBraTXSZE1
This commit is contained in:
@@ -60,6 +60,13 @@ x-secrets-hub-site-env: &secrets-hub-site-env
|
||||
services:
|
||||
central-a:
|
||||
image: scadabridge:latest
|
||||
# An init process (tini) as PID 1, so a crashed dotnet process actually dies.
|
||||
# Without it dotnet IS PID 1, Linux ignores the SIGABRT the runtime's crash path
|
||||
# raises against PID 1, and any unhandled boot exception left the container
|
||||
# `running` with the main thread spinning at 100% CPU — restart policy never
|
||||
# fired (ScadaBridge#34). Belt to Program.cs's UnhandledException handler, which
|
||||
# covers managed exceptions but not FailFast/runtime-internal aborts.
|
||||
init: true
|
||||
# CoordinatedShutdown needs cluster-leave (15s budget) + cluster-exiting +
|
||||
# actor-system-terminate + Serilog flush; the 10s SIGTERM default SIGKILLed
|
||||
# mid-drain, turning every redeploy into the crash path (review 01 [Medium]).
|
||||
@@ -116,6 +123,7 @@ services:
|
||||
|
||||
central-b:
|
||||
image: scadabridge:latest
|
||||
init: true # PID-1 crash wedge, ScadaBridge#34 — see central-a
|
||||
# CoordinatedShutdown needs cluster-leave (15s budget) + cluster-exiting +
|
||||
# actor-system-terminate + Serilog flush; the 10s SIGTERM default SIGKILLed
|
||||
# mid-drain, turning every redeploy into the crash path (review 01 [Medium]).
|
||||
@@ -172,6 +180,7 @@ services:
|
||||
|
||||
site-a-a:
|
||||
image: scadabridge:latest
|
||||
init: true # PID-1 crash wedge, ScadaBridge#34 — see central-a
|
||||
# CoordinatedShutdown needs cluster-leave (15s budget) + cluster-exiting +
|
||||
# actor-system-terminate + Serilog flush; the 10s SIGTERM default SIGKILLed
|
||||
# mid-drain, turning every redeploy into the crash path (review 01 [Medium]).
|
||||
@@ -195,6 +204,7 @@ services:
|
||||
|
||||
site-a-b:
|
||||
image: scadabridge:latest
|
||||
init: true # PID-1 crash wedge, ScadaBridge#34 — see central-a
|
||||
# CoordinatedShutdown needs cluster-leave (15s budget) + cluster-exiting +
|
||||
# actor-system-terminate + Serilog flush; the 10s SIGTERM default SIGKILLed
|
||||
# mid-drain, turning every redeploy into the crash path (review 01 [Medium]).
|
||||
@@ -218,6 +228,7 @@ services:
|
||||
|
||||
site-b-a:
|
||||
image: scadabridge:latest
|
||||
init: true # PID-1 crash wedge, ScadaBridge#34 — see central-a
|
||||
# CoordinatedShutdown needs cluster-leave (15s budget) + cluster-exiting +
|
||||
# actor-system-terminate + Serilog flush; the 10s SIGTERM default SIGKILLed
|
||||
# mid-drain, turning every redeploy into the crash path (review 01 [Medium]).
|
||||
@@ -238,6 +249,7 @@ services:
|
||||
|
||||
site-b-b:
|
||||
image: scadabridge:latest
|
||||
init: true # PID-1 crash wedge, ScadaBridge#34 — see central-a
|
||||
# CoordinatedShutdown needs cluster-leave (15s budget) + cluster-exiting +
|
||||
# actor-system-terminate + Serilog flush; the 10s SIGTERM default SIGKILLed
|
||||
# mid-drain, turning every redeploy into the crash path (review 01 [Medium]).
|
||||
@@ -258,6 +270,7 @@ services:
|
||||
|
||||
site-c-a:
|
||||
image: scadabridge:latest
|
||||
init: true # PID-1 crash wedge, ScadaBridge#34 — see central-a
|
||||
# CoordinatedShutdown needs cluster-leave (15s budget) + cluster-exiting +
|
||||
# actor-system-terminate + Serilog flush; the 10s SIGTERM default SIGKILLed
|
||||
# mid-drain, turning every redeploy into the crash path (review 01 [Medium]).
|
||||
@@ -278,6 +291,7 @@ services:
|
||||
|
||||
site-c-b:
|
||||
image: scadabridge:latest
|
||||
init: true # PID-1 crash wedge, ScadaBridge#34 — see central-a
|
||||
# CoordinatedShutdown needs cluster-leave (15s budget) + cluster-exiting +
|
||||
# actor-system-terminate + Serilog flush; the 10s SIGTERM default SIGKILLed
|
||||
# mid-drain, turning every redeploy into the crash path (review 01 [Medium]).
|
||||
|
||||
Reference in New Issue
Block a user