fix(host): unhandled boot exception now kills the process instead of wedging the container (#34)
Root cause: dotnet runs as container PID 1 and Linux ignores default-action signals sent to PID 1, so the runtime's unhandled-exception path (banner, then abort() -> SIGABRT) could never terminate the process — it printed the trace and spun the main thread at 100% CPU with the container `running`, so `restart: unless-stopped` never fired. Reproduced deterministically: same StartupValidator throw exits 134 under an init process and wedges without one. Two layers, each covering the other's gap: - Program.cs registers an AppDomain.UnhandledException handler before the first statement that can throw: prints the trace, best-effort flushes Serilog, Environment.Exit(134) — exit() is a syscall PID 1 CAN perform, 134 preserves the 128+SIGABRT crash code, and it covers every thread, not just the boot window. It cannot fire under WebApplicationFactory (the test host catches entry-point exceptions), so the designed boot-refusal exceptions still propagate to tests unchanged. - docker-compose: init: true on all 8 nodes for the crash paths that bypass the managed event (Environment.FailFast, runtime-internal aborts). The CoordinatedShutdown no-Environment.Exit guard gains a precise carve-out (exactly one call, only inside the handler); Environment.Exit still fires the CLR shutdown hook Akka binds via run-by-clr-shutdown-hook = on, so the crash path skips nothing abort() kept. New pin test keeps the handler ahead of the configuration build. Live-verified on the rig image: crash now yields Exited (134) + RestartCount climbing under `unless-stopped`, trace intact, with and without init; full 8-node rig redeployed healthy with docker-init as PID 1. Closes #34. Claude-Session: https://claude.ai/code/session_014WNM4vjoVksyyBraTXSZE1
This commit is contained in:
@@ -39,6 +39,27 @@ using ZB.MOM.WW.Secrets.Ui;
|
||||
using ZB.MOM.WW.Telemetry;
|
||||
using Serilog;
|
||||
|
||||
// Terminate honestly on ANY unhandled exception, from any thread — registered before the
|
||||
// first statement that can throw. In a container this process runs as PID 1, and Linux
|
||||
// ignores default-action signals sent to PID 1, so the runtime's crash path (print the
|
||||
// banner, then abort() → SIGABRT) can never actually kill the process: observed live, it
|
||||
// printed the banner and then spun the main thread at 100% CPU with the container still
|
||||
// `running`, so `restart: unless-stopped` never fired and every fail-closed boot refusal
|
||||
// became a silent manual-restart outage (ScadaBridge#34, reproduced 2026-08-07).
|
||||
// Environment.Exit uses the exit() syscall, which PID 1 CAN perform, and 134 preserves the
|
||||
// 128+SIGABRT code the abort would have produced. This never fires under test hosts:
|
||||
// WebApplicationFactory catches entry-point exceptions itself, so they are never
|
||||
// "unhandled" there and the designed boot-refusal exceptions still propagate to tests.
|
||||
// The docker compose additionally sets `init: true` for the crash paths that bypass this
|
||||
// event entirely (Environment.FailFast, runtime-internal aborts).
|
||||
AppDomain.CurrentDomain.UnhandledException += (_, e) =>
|
||||
{
|
||||
// Print ourselves: exiting here preempts the runtime's own banner.
|
||||
Console.Error.WriteLine(e.ExceptionObject);
|
||||
try { Log.CloseAndFlush(); } catch { /* stderr above already has the exception */ }
|
||||
Environment.Exit(134);
|
||||
};
|
||||
|
||||
// SCADABRIDGE_CONFIG determines which role-specific config to load (Central or Site)
|
||||
// DOTNET_ENVIRONMENT/ASPNETCORE_ENVIRONMENT stay as "Development" for dev tooling (static assets, EF migrations, etc.)
|
||||
var scadabridgeConfig = Environment.GetEnvironmentVariable("SCADABRIDGE_CONFIG")
|
||||
@@ -68,7 +89,9 @@ var configuration = new ConfigurationBuilder()
|
||||
// Two consequences worth knowing. This block runs BEFORE Serilog exists and outside the
|
||||
// try/catch below, so a SQL Server outage at central boot exits with a bare stderr stack
|
||||
// trace, no structured log — honest and container-restart-retryable, just unenriched; the
|
||||
// SQLite path had the same window but its only failure mode was a local file. And the
|
||||
// SQLite path had the same window but its only failure mode was a local file. (That exit
|
||||
// is real only because of the UnhandledException handler at the top of this file — the
|
||||
// runtime's own crash path wedges as container PID 1, ScadaBridge#34.) And the
|
||||
// role check is deliberately NOT folded into SecretsRegistration: that class refuses to
|
||||
// read the node role from configuration (a config-read role could silently turn a site
|
||||
// into a hub — see SecretsNodeRole), while Program.cs must read it to branch the whole
|
||||
|
||||
Reference in New Issue
Block a user