fix(host): unhandled boot exception now kills the process instead of wedging the container (#34)

Root cause: dotnet runs as container PID 1 and Linux ignores default-action
signals sent to PID 1, so the runtime's unhandled-exception path (banner,
then abort() -> SIGABRT) could never terminate the process — it printed the
trace and spun the main thread at 100% CPU with the container `running`,
so `restart: unless-stopped` never fired. Reproduced deterministically:
same StartupValidator throw exits 134 under an init process and wedges
without one.

Two layers, each covering the other's gap:
- Program.cs registers an AppDomain.UnhandledException handler before the
  first statement that can throw: prints the trace, best-effort flushes
  Serilog, Environment.Exit(134) — exit() is a syscall PID 1 CAN perform,
  134 preserves the 128+SIGABRT crash code, and it covers every thread,
  not just the boot window. It cannot fire under WebApplicationFactory
  (the test host catches entry-point exceptions), so the designed
  boot-refusal exceptions still propagate to tests unchanged.
- docker-compose: init: true on all 8 nodes for the crash paths that
  bypass the managed event (Environment.FailFast, runtime-internal aborts).

The CoordinatedShutdown no-Environment.Exit guard gains a precise carve-out
(exactly one call, only inside the handler); Environment.Exit still fires
the CLR shutdown hook Akka binds via run-by-clr-shutdown-hook = on, so the
crash path skips nothing abort() kept. New pin test keeps the handler ahead
of the configuration build.

Live-verified on the rig image: crash now yields Exited (134) +
RestartCount climbing under `unless-stopped`, trace intact, with and
without init; full 8-node rig redeployed healthy with docker-init as PID 1.

Closes #34.

Claude-Session: https://claude.ai/code/session_014WNM4vjoVksyyBraTXSZE1
This commit is contained in:
Joseph Doherty
2026-08-08 05:23:26 -04:00
parent e697477c1f
commit e9c412e528
4 changed files with 89 additions and 1 deletions
@@ -165,6 +165,34 @@ public class HostStartupTests : IDisposable
}
}
[Fact]
public void Program_RegistersUnhandledExceptionExitHandler()
{
// ScadaBridge#34: in a container this process is PID 1, and Linux ignores the
// SIGABRT the runtime's crash path raises against PID 1 — an unhandled boot
// exception printed its banner and then spun the main thread at 100% CPU with
// the container still `running`, so the restart policy never fired. Program.cs
// therefore registers an AppDomain.UnhandledException handler that exits via
// the exit() syscall (which PID 1 CAN perform) before anything can throw.
// Behavioural coverage needs a real crashed process (see the compose comment
// and the issue's live reproductions); this pins the handler's existence so a
// refactor cannot silently reopen the wedge.
var hostProjectDir = FindHostProjectDirectory();
Assert.NotNull(hostProjectDir);
var program = File.ReadAllText(Path.Combine(hostProjectDir!, "Program.cs"));
Assert.Contains("AppDomain.CurrentDomain.UnhandledException +=", program);
Assert.Contains("Environment.Exit(134)", program);
// The registration must precede the first statement that can throw — the
// configuration build is the earliest (appsettings.json is non-optional).
var handlerAt = program.IndexOf("AppDomain.CurrentDomain.UnhandledException +=", StringComparison.Ordinal);
var configAt = program.IndexOf("new ConfigurationBuilder()", StringComparison.Ordinal);
Assert.True(handlerAt >= 0 && configAt >= 0 && handlerAt < configAt,
"the UnhandledException exit handler must be registered before the configuration build");
}
private static string? FindHostProjectDirectory()
{
// Walk up from the test assembly location to find the src directory