fix(host): unhandled boot exception now kills the process instead of wedging the container (#34)

Root cause: dotnet runs as container PID 1 and Linux ignores default-action
signals sent to PID 1, so the runtime's unhandled-exception path (banner,
then abort() -> SIGABRT) could never terminate the process — it printed the
trace and spun the main thread at 100% CPU with the container `running`,
so `restart: unless-stopped` never fired. Reproduced deterministically:
same StartupValidator throw exits 134 under an init process and wedges
without one.

Two layers, each covering the other's gap:
- Program.cs registers an AppDomain.UnhandledException handler before the
  first statement that can throw: prints the trace, best-effort flushes
  Serilog, Environment.Exit(134) — exit() is a syscall PID 1 CAN perform,
  134 preserves the 128+SIGABRT crash code, and it covers every thread,
  not just the boot window. It cannot fire under WebApplicationFactory
  (the test host catches entry-point exceptions), so the designed
  boot-refusal exceptions still propagate to tests unchanged.
- docker-compose: init: true on all 8 nodes for the crash paths that
  bypass the managed event (Environment.FailFast, runtime-internal aborts).

The CoordinatedShutdown no-Environment.Exit guard gains a precise carve-out
(exactly one call, only inside the handler); Environment.Exit still fires
the CLR shutdown hook Akka binds via run-by-clr-shutdown-hook = on, so the
crash path skips nothing abort() kept. New pin test keeps the handler ahead
of the configuration build.

Live-verified on the rig image: crash now yields Exited (134) +
RestartCount climbing under `unless-stopped`, trace intact, with and
without init; full 8-node rig redeployed healthy with docker-init as PID 1.

Closes #34.

Claude-Session: https://claude.ai/code/session_014WNM4vjoVksyyBraTXSZE1
This commit is contained in:
Joseph Doherty
2026-08-08 05:23:26 -04:00
parent e697477c1f
commit e9c412e528
4 changed files with 89 additions and 1 deletions
+24 -1
View File
@@ -39,6 +39,27 @@ using ZB.MOM.WW.Secrets.Ui;
using ZB.MOM.WW.Telemetry;
using Serilog;
// Terminate honestly on ANY unhandled exception, from any thread — registered before the
// first statement that can throw. In a container this process runs as PID 1, and Linux
// ignores default-action signals sent to PID 1, so the runtime's crash path (print the
// banner, then abort() → SIGABRT) can never actually kill the process: observed live, it
// printed the banner and then spun the main thread at 100% CPU with the container still
// `running`, so `restart: unless-stopped` never fired and every fail-closed boot refusal
// became a silent manual-restart outage (ScadaBridge#34, reproduced 2026-08-07).
// Environment.Exit uses the exit() syscall, which PID 1 CAN perform, and 134 preserves the
// 128+SIGABRT code the abort would have produced. This never fires under test hosts:
// WebApplicationFactory catches entry-point exceptions itself, so they are never
// "unhandled" there and the designed boot-refusal exceptions still propagate to tests.
// The docker compose additionally sets `init: true` for the crash paths that bypass this
// event entirely (Environment.FailFast, runtime-internal aborts).
AppDomain.CurrentDomain.UnhandledException += (_, e) =>
{
// Print ourselves: exiting here preempts the runtime's own banner.
Console.Error.WriteLine(e.ExceptionObject);
try { Log.CloseAndFlush(); } catch { /* stderr above already has the exception */ }
Environment.Exit(134);
};
// SCADABRIDGE_CONFIG determines which role-specific config to load (Central or Site)
// DOTNET_ENVIRONMENT/ASPNETCORE_ENVIRONMENT stay as "Development" for dev tooling (static assets, EF migrations, etc.)
var scadabridgeConfig = Environment.GetEnvironmentVariable("SCADABRIDGE_CONFIG")
@@ -68,7 +89,9 @@ var configuration = new ConfigurationBuilder()
// Two consequences worth knowing. This block runs BEFORE Serilog exists and outside the
// try/catch below, so a SQL Server outage at central boot exits with a bare stderr stack
// trace, no structured log — honest and container-restart-retryable, just unenriched; the
// SQLite path had the same window but its only failure mode was a local file. And the
// SQLite path had the same window but its only failure mode was a local file. (That exit
// is real only because of the UnhandledException handler at the top of this file — the
// runtime's own crash path wedges as container PID 1, ScadaBridge#34.) And the
// role check is deliberately NOT folded into SecretsRegistration: that class refuses to
// read the node role from configuration (a config-read role could silently turn a site
// into a hub — see SecretsNodeRole), while Program.cs must read it to branch the whole