dohertj2/ScadaBridge
1
0
Fork 0
Code Issues 2 Pull Requests 1 Actions Packages Projects Releases Wiki Activity

Pre-Serilog boot failure wedges the container (100% CPU spin, restart policy never fires) #34

New Issue
Closed
opened 2026-08-07 11:41:19 -04:00 by dohertj2 · 1 comment
dohertj2 commented 2026-08-07 11:41:19 -04:00
Owner
Copy Link
Copy Source

Symptom

Any unhandled exception thrown in the ScadaBridge Host's pre-Serilog window (the pre-host block in Program.cs — Layer-A ${secret:} expander, StartupValidator, migrators — before the Serilog host is built) does not terminate the process. After printing the crash banner to stderr, the main thread spins at ~100% CPU indefinitely; the container stays running (RestartCount=0), so restart: unless-stopped never fires and the node serves nothing.

The compose comments and plan docs describe this window as "exits with a bare stderr trace and restarts". Observed twice, it does not exit. Monitoring that watches container state sees a healthy-looking running container — so any pre-host failure becomes a silent manual-restart outage.

Reproductions (both live, 2026-08-07 gate)

Recorded in docs/plans/2026-08-07-secrets-central-shared-store-live-gate.md (defect 2), gated on main (gate first pass @ 68f812ea; doc merged through e697477c):

  1. SqlException from the expander's SqlServerSecretsStoreMigrator run (the 0.5.0 virgin-DB 2714 race, since fixed at the library in 0.5.1): banner printed, then the main thread — confirmed as the busy thread — spun at ~100% CPU for ~5 minutes until a manual restart. Container stayed running, RestartCount=0.
  2. StartupValidator InvalidOperationException (check 4's deliberately-short pepper): identical signature — banner printed, 100% CPU, container still running.

Two different exception types from two different pre-host stages, same wedge — so it is not specific to SqlClient or to the new SQL store path.

Why it matters

Fail-closed is only honest if the process actually exits. Every pre-host fail-closed guard (blank/${secret:} connection string refusal, config validation, secret-resolution failures) currently converts a designed clean fail-fast into a wedged, invisible outage: the restart policy cannot recover it and container-state monitoring cannot see it. This is a whole outage class, not a single bug site.

Scope

Pre-existing — this predates the secrets central-shared-SQL-store program and affects the old SQLite path's pre-host failure modes equally; the 2026-08-07 gate merely produced two clean reproductions. Mechanism not root-caused (out of gate scope); the observable and both reproductions are recorded in the gate doc.

Filed at closeout of the central-shared-store program (scadaproj#4).

## Symptom Any unhandled exception thrown in the ScadaBridge Host's **pre-Serilog window** (the pre-host block in `Program.cs` — Layer-A `${secret:}` expander, `StartupValidator`, migrators — before the Serilog host is built) does **not** terminate the process. After printing the crash banner to stderr, the main thread spins at ~100% CPU indefinitely; the container stays `running` (`RestartCount=0`), so `restart: unless-stopped` never fires and the node serves nothing. The compose comments and plan docs describe this window as "exits with a bare stderr trace and restarts". **Observed twice, it does not exit.** Monitoring that watches container state sees a healthy-looking `running` container — so any pre-host failure becomes a **silent manual-restart outage**. ## Reproductions (both live, 2026-08-07 gate) Recorded in `docs/plans/2026-08-07-secrets-central-shared-store-live-gate.md` (defect 2), gated on `main` (gate first pass @ `68f812ea`; doc merged through `e697477c`): 1. **`SqlException` from the expander's `SqlServerSecretsStoreMigrator` run** (the 0.5.0 virgin-DB 2714 race, since fixed at the library in 0.5.1): banner printed, then the main thread — confirmed as the busy thread — spun at ~100% CPU for ~5 minutes until a manual restart. Container stayed `running`, `RestartCount=0`. 2. **`StartupValidator` `InvalidOperationException`** (check 4's deliberately-short pepper): identical signature — banner printed, 100% CPU, container still `running`. Two different exception types from two different pre-host stages, same wedge — so it is not specific to SqlClient or to the new SQL store path. ## Why it matters Fail-closed is only honest if the process actually exits. Every pre-host fail-closed guard (blank/`${secret:}` connection string refusal, config validation, secret-resolution failures) currently converts a designed clean fail-fast into a wedged, invisible outage: the restart policy cannot recover it and container-state monitoring cannot see it. This is a whole outage class, not a single bug site. ## Scope **Pre-existing** — this predates the secrets central-shared-SQL-store program and affects the old SQLite path's pre-host failure modes equally; the 2026-08-07 gate merely produced two clean reproductions. Mechanism not root-caused (out of gate scope); the observable and both reproductions are recorded in the gate doc. Filed at closeout of the central-shared-store program (scadaproj#4).
dohertj2 commented 2026-08-08 05:23:42 -04:00
Author
Owner
Copy Link
Copy Source

FIXED — main @ e9c412e5.

Root cause: dotnet runs as container PID 1, and Linux ignores default-action signals sent to PID 1 — so the runtime's unhandled-exception path (print banner, then abort() → SIGABRT) could never terminate the process and spun instead. Proven deterministically: the identical StartupValidator crash exits 134 under an init process and wedges (100% CPU, container running) without one. Not specific to any exception type or boot stage, exactly as the two gate reproductions suggested.

Fix, two layers:

  1. Program.cs registers an AppDomain.UnhandledException handler before the first statement that can throw: prints the trace, best-effort Serilog flush, Environment.Exit(134) — exit() is a syscall PID 1 can perform, 134 preserves the 128+SIGABRT crash code, and it covers unhandled exceptions on every thread, not just the boot window. It cannot fire under WebApplicationFactory (the test host catches entry-point exceptions itself), so designed boot-refusal exceptions still propagate to tests unchanged.
  2. docker/docker-compose.yml sets init: true on all 8 nodes — for crash paths that bypass the managed event entirely (Environment.FailFast, runtime-internal aborts), plus proper signal handling generally.

The CoordinatedShutdown no-Environment.Exit source guard gained a precise carve-out (exactly one call, only inside the handler); note Environment.Exit still fires the CLR shutdown hook Akka binds via run-by-clr-shutdown-hook = on, so the crash path loses nothing abort() had.

Live verification on the rig image: the repro that wedged now yields Exited (134) with RestartCount climbing under restart: unless-stopped (8 restarts in 25 s), stack trace intact — verified with and without --init, so each layer is proven independently. Full 8-node rig redeployed healthy, docker-init as PID 1, both centrals /health/ready.

Operator note for production overlays: any deployment running this image under its own compose/orchestration should also add init: true (or the platform equivalent) — the in-process handler covers managed exceptions, but only an init process covers FailFast/runtime aborts.

**FIXED** — `main @ e9c412e5`. **Root cause:** dotnet runs as container PID 1, and Linux ignores default-action signals sent to PID 1 — so the runtime's unhandled-exception path (print banner, then `abort()` → SIGABRT) could never terminate the process and spun instead. Proven deterministically: the identical `StartupValidator` crash exits `134` under an init process and wedges (100% CPU, container `running`) without one. Not specific to any exception type or boot stage, exactly as the two gate reproductions suggested. **Fix, two layers:** 1. `Program.cs` registers an `AppDomain.UnhandledException` handler before the first statement that can throw: prints the trace, best-effort Serilog flush, `Environment.Exit(134)` — `exit()` is a syscall PID 1 *can* perform, 134 preserves the 128+SIGABRT crash code, and it covers unhandled exceptions on **every** thread, not just the boot window. It cannot fire under WebApplicationFactory (the test host catches entry-point exceptions itself), so designed boot-refusal exceptions still propagate to tests unchanged. 2. `docker/docker-compose.yml` sets `init: true` on all 8 nodes — for crash paths that bypass the managed event entirely (`Environment.FailFast`, runtime-internal aborts), plus proper signal handling generally. The CoordinatedShutdown `no-Environment.Exit` source guard gained a precise carve-out (exactly one call, only inside the handler); note `Environment.Exit` still fires the CLR shutdown hook Akka binds via `run-by-clr-shutdown-hook = on`, so the crash path loses nothing `abort()` had. **Live verification on the rig image:** the repro that wedged now yields `Exited (134)` with `RestartCount` climbing under `restart: unless-stopped` (8 restarts in 25 s), stack trace intact — verified **with and without** `--init`, so each layer is proven independently. Full 8-node rig redeployed healthy, `docker-init` as PID 1, both centrals `/health/ready`. **Operator note for production overlays:** any deployment running this image under its own compose/orchestration should also add `init: true` (or the platform equivalent) — the in-process handler covers managed exceptions, but only an init process covers `FailFast`/runtime aborts.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
No items
No Label
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
dohertj2 (dohertj2)
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: dohertj2/ScadaBridge#34
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?