Joseph Doherty
e9c412e528
Root cause: dotnet runs as container PID 1 and Linux ignores default-action signals sent to PID 1, so the runtime's unhandled-exception path (banner, then abort() -> SIGABRT) could never terminate the process — it printed the trace and spun the main thread at 100% CPU with the container `running`, so `restart: unless-stopped` never fired. Reproduced deterministically: same StartupValidator throw exits 134 under an init process and wedges without one. Two layers, each covering the other's gap: - Program.cs registers an AppDomain.UnhandledException handler before the first statement that can throw: prints the trace, best-effort flushes Serilog, Environment.Exit(134) — exit() is a syscall PID 1 CAN perform, 134 preserves the 128+SIGABRT crash code, and it covers every thread, not just the boot window. It cannot fire under WebApplicationFactory (the test host catches entry-point exceptions), so the designed boot-refusal exceptions still propagate to tests unchanged. - docker-compose: init: true on all 8 nodes for the crash paths that bypass the managed event (Environment.FailFast, runtime-internal aborts). The CoordinatedShutdown no-Environment.Exit guard gains a precise carve-out (exactly one call, only inside the handler); Environment.Exit still fires the CLR shutdown hook Akka binds via run-by-clr-shutdown-hook = on, so the crash path skips nothing abort() kept. New pin test keeps the handler ahead of the configuration build. Live-verified on the rig image: crash now yields Exited (134) + RestartCount climbing under `unless-stopped`, trace intact, with and without init; full 8-node rig redeployed healthy with docker-init as PID 1. Closes #34. Claude-Session: https://claude.ai/code/session_014WNM4vjoVksyyBraTXSZE1
329 lines
18 KiB
YAML
329 lines
18 KiB
YAML
# ── Clustered secret replication: the pull-only gRPC hub (scadaproj#3) ─────────
|
|
#
|
|
# Central hosts the hub on its EXISTING h2c control-plane listener (CentralGrpcPort
|
|
# 8083, alongside CentralControlService); each site node sweeps it on an interval and
|
|
# writes what it pulls into its OWN local SQLite store. Nothing but ciphertext crosses
|
|
# the wire, so every participating node must resolve the SAME KEK.
|
|
#
|
|
# ENABLED ON FOUR NODES ONLY: the central pair (hub) and the site-a pair (followers).
|
|
# site-b and site-c are deliberately left without it, so the default-OFF posture is
|
|
# proven side by side on one rig — exactly as site-a is the rig's only LocalDb-replicated
|
|
# pair. A node with no Secrets__* override keeps the shipped appsettings default
|
|
# (Replication:Enabled=false, Mode=SqlServer) and composes a plain local store.
|
|
#
|
|
# ALL VALUES HERE ARE DEV-ONLY and committed under the same exception as the mesh PSKs
|
|
# and the ApiKeyPepper above: a local docker rig needs a working credential in source
|
|
# control to boot. Production supplies the KEK out of band (ZB_SECRETS_MASTER_KEY, never
|
|
# committed) and the hub token from appsettings/env — NEVER as a ${secret:} reference,
|
|
# since resolving one is what the hub exists to make possible.
|
|
x-secrets-hub-env: &secrets-hub-env
|
|
# DEV-ONLY KEK — NOT a real key. Identical on all four participating nodes: only
|
|
# ciphertext replicates, so a node with a different KEK fails closed on resolve with a
|
|
# kek_id mismatch that reads like corruption but is a deployment error.
|
|
ZB_SECRETS_MASTER_KEY: "zZiBWuoaVMbJmGXToLk9Lakw0iJozXoL/7Gxac3GwJ4="
|
|
# The appsettings default is the relative "scadabridge-secrets.db", which resolves to
|
|
# /app — inside the image's writable layer, so it is destroyed by any container
|
|
# recreate and unreachable from the host. /app/data is the node's own mounted volume
|
|
# (the one LocalDb already uses on sites; added to the central pair for this).
|
|
Secrets__SqlitePath: "/app/data/scadabridge-secrets.db"
|
|
Secrets__Replication__Enabled: "true"
|
|
Secrets__Replication__Mode: "Grpc"
|
|
# DEV-ONLY shared bearer token — NOT a real secret. Presented by every follower and
|
|
# verified by the hub's fail-closed SecretsHubAuthInterceptor. Must be IDENTICAL on the
|
|
# hub and every follower; an unset token is a startup failure on both halves.
|
|
Secrets__GrpcHub__BearerToken: "secrets-hub-docker-dev-token"
|
|
|
|
# Central-only half: the SHARED SQL-Server secret store (scadaproj#4, Secrets 0.5.0).
|
|
# In Grpc mode BOTH central nodes read and write ONE copy of every row in this database,
|
|
# so the two hub instances serve identical manifests by construction — that is what makes
|
|
# the site-side FallbackEndpoints below safe. ZbSecretsHub is a dedicated database on the
|
|
# rig's existing scadabridge-mssql container (default zbsecrets schema, created by the
|
|
# boot-time SqlServerSecretsStoreMigrator; the database itself + the scadabridge_app grant
|
|
# were provisioned once via sqlcmd — see docs/plans/2026-08-07-secrets-central-shared-store-live-gate.md).
|
|
# Same DEV-ONLY credentials as the ScadaBridge__Database__* strings above. Must be a
|
|
# LITERAL value, never a ${secret:} reference — the pre-host expander needs this string to
|
|
# reach the store that would resolve it (bootstrap circularity; registration rejects it).
|
|
# Site nodes must NEVER carry this key: sites talk to central, not to central's database.
|
|
x-secrets-hub-central-env: &secrets-hub-central-env
|
|
Secrets__SqlServer__ConnectionString: "Server=scadabridge-mssql,1433;Database=ZbSecretsHub;User Id=scadabridge_app;Password=ScadaBridge_Dev1#;TrustServerCertificate=true"
|
|
|
|
# Site half of the same section. The hub client dials Endpoint first and fails over, per
|
|
# call, to FallbackEndpoints in order (sticky on whichever answered last, Secrets 0.5.0).
|
|
# Listing central-b is safe ONLY because both centrals serve the one shared SQL store
|
|
# above — never list endpoints backed by independent stores: failing over to an emptier
|
|
# hub is a silent convergence stop, the exact defect (scadaproj#4) the shared store
|
|
# exists to prevent.
|
|
x-secrets-hub-site-env: &secrets-hub-site-env
|
|
Secrets__GrpcHub__Endpoint: "http://scadabridge-central-a:8083"
|
|
Secrets__GrpcHub__FallbackEndpoints__0: "http://scadabridge-central-b:8083"
|
|
|
|
services:
|
|
central-a:
|
|
image: scadabridge:latest
|
|
# An init process (tini) as PID 1, so a crashed dotnet process actually dies.
|
|
# Without it dotnet IS PID 1, Linux ignores the SIGABRT the runtime's crash path
|
|
# raises against PID 1, and any unhandled boot exception left the container
|
|
# `running` with the main thread spinning at 100% CPU — restart policy never
|
|
# fired (ScadaBridge#34). Belt to Program.cs's UnhandledException handler, which
|
|
# covers managed exceptions but not FailFast/runtime-internal aborts.
|
|
init: true
|
|
# CoordinatedShutdown needs cluster-leave (15s budget) + cluster-exiting +
|
|
# actor-system-terminate + Serilog flush; the 10s SIGTERM default SIGKILLed
|
|
# mid-drain, turning every redeploy into the crash path (review 01 [Medium]).
|
|
stop_grace_period: 30s
|
|
container_name: scadabridge-central-a
|
|
environment:
|
|
# Hub half of the pull-only gRPC secrets hub + the central-only shared SQL store
|
|
# (anchors at the top of this file).
|
|
<<: [*secrets-hub-env, *secrets-hub-central-env]
|
|
SCADABRIDGE_CONFIG: Central
|
|
ASPNETCORE_ENVIRONMENT: Development
|
|
ASPNETCORE_URLS: "http://+:5000"
|
|
# DEV-ONLY local-cluster value — NOT a real secret. The Auth/Config normalization
|
|
# (2026-06-03) made ScadaBridge:InboundApi:ApiKeyPepper a hard Central startup
|
|
# requirement (>=16 chars, per-environment). Real deployments inject a true secret
|
|
# out-of-band (env/secret store), never from source control — see
|
|
# docs/operations/inbound-api-key-reissue.md. Both Central nodes share one pepper.
|
|
ScadaBridge__InboundApi__ApiKeyPepper: "dev-only-insecure-pepper-docker-cluster-0001"
|
|
# DEV-ONLY local-cluster secrets — NOT real secrets. Relocated out of the mounted
|
|
# appsettings.Central.json (G-4 T5 secrets cleanup) so that app-config file carries no
|
|
# plaintext credentials. Env overrides layer over JSON BEFORE the ${secret:} pre-host
|
|
# expander runs, so the dev cluster boots with no KEK/secret store. The SQL password is
|
|
# the same value already committed for the sibling scadabridge-mssql container
|
|
# (infra/docker-compose.yml MSSQL_SA_PASSWORD) — consolidation, not new exposure. Real
|
|
# deployments use ${secret:} tokens (src/.../appsettings.Central.json) + a seeded store.
|
|
ScadaBridge__Database__ConfigurationDb: "Server=scadabridge-mssql,1433;Database=ScadaBridgeConfig;User Id=scadabridge_app;Password=ScadaBridge_Dev1#;TrustServerCertificate=true"
|
|
ScadaBridge__Database__MachineDataDb: "Server=scadabridge-mssql,1433;Database=ScadaBridgeMachineData;User Id=scadabridge_app;Password=ScadaBridge_Dev1#;TrustServerCertificate=true"
|
|
ScadaBridge__Security__Ldap__ServiceAccountPassword: "serviceaccount123"
|
|
ScadaBridge__Security__JwtSigningKey: "scadabridge-dev-jwt-signing-key-must-be-at-least-32-characters-long"
|
|
# DEV-ONLY gRPC control-plane preshared keys, one per site — NOT real secrets.
|
|
# Central verifies/presents these; each site node carries the same value as
|
|
# ScadaBridge:Communication:GrpcPsk in its mounted appsettings.Site.json. Kept as
|
|
# env overrides (not in the mounted central appsettings) so that file stays free of
|
|
# plaintext credentials. Production instead seeds SB-GRPC-PSK-<siteId> into the
|
|
# secret store, which is also the only source that can serve a site added at runtime.
|
|
ScadaBridge__Communication__SitePsks__site-a: "dev-grpc-psk-docker-site-a"
|
|
ScadaBridge__Communication__SitePsks__site-b: "dev-grpc-psk-docker-site-b"
|
|
ScadaBridge__Communication__SitePsks__site-c: "dev-grpc-psk-docker-site-c"
|
|
ports:
|
|
- "9001:5000" # Web UI + Inbound API
|
|
- "9011:8081" # Akka remoting (host access for CLI/debugging)
|
|
- "9013:8083" # gRPC control plane (CentralControlService, T1A.2)
|
|
volumes:
|
|
- ./central-node-a/appsettings.Central.json:/app/appsettings.Central.json:ro
|
|
# Originally added for the gRPC secrets hub's local SQLite store; since the
|
|
# central store moved to the shared SQL-Server database (Secrets 0.5.0,
|
|
# scadaproj#4) the scadabridge-secrets.db here is a pre-0.5.0 residue, but the
|
|
# volume is still needed (inbound-api-keys.sqlite lives on it).
|
|
- ./central-node-a/data:/app/data
|
|
- ./central-node-a/logs:/app/logs
|
|
networks:
|
|
- scadabridge-net
|
|
restart: unless-stopped
|
|
|
|
central-b:
|
|
image: scadabridge:latest
|
|
init: true # PID-1 crash wedge, ScadaBridge#34 — see central-a
|
|
# CoordinatedShutdown needs cluster-leave (15s budget) + cluster-exiting +
|
|
# actor-system-terminate + Serilog flush; the 10s SIGTERM default SIGKILLed
|
|
# mid-drain, turning every redeploy into the crash path (review 01 [Medium]).
|
|
stop_grace_period: 30s
|
|
container_name: scadabridge-central-b
|
|
environment:
|
|
# Hub half of the pull-only gRPC secrets hub + the central-only shared SQL store
|
|
# (anchors at the top of this file).
|
|
<<: [*secrets-hub-env, *secrets-hub-central-env]
|
|
SCADABRIDGE_CONFIG: Central
|
|
ASPNETCORE_ENVIRONMENT: Development
|
|
ASPNETCORE_URLS: "http://+:5000"
|
|
# DEV-ONLY local-cluster value — NOT a real secret. The Auth/Config normalization
|
|
# (2026-06-03) made ScadaBridge:InboundApi:ApiKeyPepper a hard Central startup
|
|
# requirement (>=16 chars, per-environment). Real deployments inject a true secret
|
|
# out-of-band (env/secret store), never from source control — see
|
|
# docs/operations/inbound-api-key-reissue.md. Both Central nodes share one pepper.
|
|
ScadaBridge__InboundApi__ApiKeyPepper: "dev-only-insecure-pepper-docker-cluster-0001"
|
|
# DEV-ONLY local-cluster secrets — NOT real secrets. Relocated out of the mounted
|
|
# appsettings.Central.json (G-4 T5 secrets cleanup) so that app-config file carries no
|
|
# plaintext credentials. Env overrides layer over JSON BEFORE the ${secret:} pre-host
|
|
# expander runs, so the dev cluster boots with no KEK/secret store. The SQL password is
|
|
# the same value already committed for the sibling scadabridge-mssql container
|
|
# (infra/docker-compose.yml MSSQL_SA_PASSWORD) — consolidation, not new exposure. Real
|
|
# deployments use ${secret:} tokens (src/.../appsettings.Central.json) + a seeded store.
|
|
ScadaBridge__Database__ConfigurationDb: "Server=scadabridge-mssql,1433;Database=ScadaBridgeConfig;User Id=scadabridge_app;Password=ScadaBridge_Dev1#;TrustServerCertificate=true"
|
|
ScadaBridge__Database__MachineDataDb: "Server=scadabridge-mssql,1433;Database=ScadaBridgeMachineData;User Id=scadabridge_app;Password=ScadaBridge_Dev1#;TrustServerCertificate=true"
|
|
ScadaBridge__Security__Ldap__ServiceAccountPassword: "serviceaccount123"
|
|
ScadaBridge__Security__JwtSigningKey: "scadabridge-dev-jwt-signing-key-must-be-at-least-32-characters-long"
|
|
# DEV-ONLY gRPC control-plane preshared keys, one per site — NOT real secrets.
|
|
# Central verifies/presents these; each site node carries the same value as
|
|
# ScadaBridge:Communication:GrpcPsk in its mounted appsettings.Site.json. Kept as
|
|
# env overrides (not in the mounted central appsettings) so that file stays free of
|
|
# plaintext credentials. Production instead seeds SB-GRPC-PSK-<siteId> into the
|
|
# secret store, which is also the only source that can serve a site added at runtime.
|
|
ScadaBridge__Communication__SitePsks__site-a: "dev-grpc-psk-docker-site-a"
|
|
ScadaBridge__Communication__SitePsks__site-b: "dev-grpc-psk-docker-site-b"
|
|
ScadaBridge__Communication__SitePsks__site-c: "dev-grpc-psk-docker-site-c"
|
|
ports:
|
|
- "9002:5000" # Web UI + Inbound API
|
|
- "9012:8081" # Akka remoting
|
|
- "9014:8083" # gRPC control plane (CentralControlService, T1A.2)
|
|
volumes:
|
|
- ./central-node-b/appsettings.Central.json:/app/appsettings.Central.json:ro
|
|
# Originally added for the gRPC secrets hub's local SQLite store; since the
|
|
# central store moved to the shared SQL-Server database (Secrets 0.5.0,
|
|
# scadaproj#4) the scadabridge-secrets.db here is a pre-0.5.0 residue, but the
|
|
# volume is still needed (inbound-api-keys.sqlite lives on it).
|
|
- ./central-node-b/data:/app/data
|
|
- ./central-node-b/logs:/app/logs
|
|
networks:
|
|
- scadabridge-net
|
|
restart: unless-stopped
|
|
|
|
site-a-a:
|
|
image: scadabridge:latest
|
|
init: true # PID-1 crash wedge, ScadaBridge#34 — see central-a
|
|
# CoordinatedShutdown needs cluster-leave (15s budget) + cluster-exiting +
|
|
# actor-system-terminate + Serilog flush; the 10s SIGTERM default SIGKILLed
|
|
# mid-drain, turning every redeploy into the crash path (review 01 [Medium]).
|
|
stop_grace_period: 30s
|
|
container_name: scadabridge-site-a-a
|
|
environment:
|
|
# Follower half of the pull-only gRPC secrets hub (anchors at the top of this
|
|
# file). site-b and site-c deliberately carry neither.
|
|
<<: [*secrets-hub-env, *secrets-hub-site-env]
|
|
SCADABRIDGE_CONFIG: Site
|
|
ports:
|
|
- "9021:8082" # Akka remoting (host access for debugging)
|
|
- "9023:8083" # gRPC streaming
|
|
volumes:
|
|
- ./site-a-node-a/appsettings.Site.json:/app/appsettings.Site.json:ro
|
|
- ./site-a-node-a/data:/app/data
|
|
- ./site-a-node-a/logs:/app/logs
|
|
networks:
|
|
- scadabridge-net
|
|
restart: unless-stopped
|
|
|
|
site-a-b:
|
|
image: scadabridge:latest
|
|
init: true # PID-1 crash wedge, ScadaBridge#34 — see central-a
|
|
# CoordinatedShutdown needs cluster-leave (15s budget) + cluster-exiting +
|
|
# actor-system-terminate + Serilog flush; the 10s SIGTERM default SIGKILLed
|
|
# mid-drain, turning every redeploy into the crash path (review 01 [Medium]).
|
|
stop_grace_period: 30s
|
|
container_name: scadabridge-site-a-b
|
|
environment:
|
|
# Follower half of the pull-only gRPC secrets hub (anchors at the top of this
|
|
# file). site-b and site-c deliberately carry neither.
|
|
<<: [*secrets-hub-env, *secrets-hub-site-env]
|
|
SCADABRIDGE_CONFIG: Site
|
|
ports:
|
|
- "9022:8082" # Akka remoting
|
|
- "9024:8083" # gRPC streaming
|
|
volumes:
|
|
- ./site-a-node-b/appsettings.Site.json:/app/appsettings.Site.json:ro
|
|
- ./site-a-node-b/data:/app/data
|
|
- ./site-a-node-b/logs:/app/logs
|
|
networks:
|
|
- scadabridge-net
|
|
restart: unless-stopped
|
|
|
|
site-b-a:
|
|
image: scadabridge:latest
|
|
init: true # PID-1 crash wedge, ScadaBridge#34 — see central-a
|
|
# CoordinatedShutdown needs cluster-leave (15s budget) + cluster-exiting +
|
|
# actor-system-terminate + Serilog flush; the 10s SIGTERM default SIGKILLed
|
|
# mid-drain, turning every redeploy into the crash path (review 01 [Medium]).
|
|
stop_grace_period: 30s
|
|
container_name: scadabridge-site-b-a
|
|
environment:
|
|
SCADABRIDGE_CONFIG: Site
|
|
ports:
|
|
- "9031:8082" # Akka remoting
|
|
- "9033:8083" # gRPC streaming
|
|
volumes:
|
|
- ./site-b-node-a/appsettings.Site.json:/app/appsettings.Site.json:ro
|
|
- ./site-b-node-a/data:/app/data
|
|
- ./site-b-node-a/logs:/app/logs
|
|
networks:
|
|
- scadabridge-net
|
|
restart: unless-stopped
|
|
|
|
site-b-b:
|
|
image: scadabridge:latest
|
|
init: true # PID-1 crash wedge, ScadaBridge#34 — see central-a
|
|
# CoordinatedShutdown needs cluster-leave (15s budget) + cluster-exiting +
|
|
# actor-system-terminate + Serilog flush; the 10s SIGTERM default SIGKILLed
|
|
# mid-drain, turning every redeploy into the crash path (review 01 [Medium]).
|
|
stop_grace_period: 30s
|
|
container_name: scadabridge-site-b-b
|
|
environment:
|
|
SCADABRIDGE_CONFIG: Site
|
|
ports:
|
|
- "9032:8082" # Akka remoting
|
|
- "9034:8083" # gRPC streaming
|
|
volumes:
|
|
- ./site-b-node-b/appsettings.Site.json:/app/appsettings.Site.json:ro
|
|
- ./site-b-node-b/data:/app/data
|
|
- ./site-b-node-b/logs:/app/logs
|
|
networks:
|
|
- scadabridge-net
|
|
restart: unless-stopped
|
|
|
|
site-c-a:
|
|
image: scadabridge:latest
|
|
init: true # PID-1 crash wedge, ScadaBridge#34 — see central-a
|
|
# CoordinatedShutdown needs cluster-leave (15s budget) + cluster-exiting +
|
|
# actor-system-terminate + Serilog flush; the 10s SIGTERM default SIGKILLed
|
|
# mid-drain, turning every redeploy into the crash path (review 01 [Medium]).
|
|
stop_grace_period: 30s
|
|
container_name: scadabridge-site-c-a
|
|
environment:
|
|
SCADABRIDGE_CONFIG: Site
|
|
ports:
|
|
- "9041:8082" # Akka remoting
|
|
- "9043:8083" # gRPC streaming
|
|
volumes:
|
|
- ./site-c-node-a/appsettings.Site.json:/app/appsettings.Site.json:ro
|
|
- ./site-c-node-a/data:/app/data
|
|
- ./site-c-node-a/logs:/app/logs
|
|
networks:
|
|
- scadabridge-net
|
|
restart: unless-stopped
|
|
|
|
site-c-b:
|
|
image: scadabridge:latest
|
|
init: true # PID-1 crash wedge, ScadaBridge#34 — see central-a
|
|
# CoordinatedShutdown needs cluster-leave (15s budget) + cluster-exiting +
|
|
# actor-system-terminate + Serilog flush; the 10s SIGTERM default SIGKILLed
|
|
# mid-drain, turning every redeploy into the crash path (review 01 [Medium]).
|
|
stop_grace_period: 30s
|
|
container_name: scadabridge-site-c-b
|
|
environment:
|
|
SCADABRIDGE_CONFIG: Site
|
|
ports:
|
|
- "9042:8082" # Akka remoting
|
|
- "9044:8083" # gRPC streaming
|
|
volumes:
|
|
- ./site-c-node-b/appsettings.Site.json:/app/appsettings.Site.json:ro
|
|
- ./site-c-node-b/data:/app/data
|
|
- ./site-c-node-b/logs:/app/logs
|
|
networks:
|
|
- scadabridge-net
|
|
restart: unless-stopped
|
|
|
|
traefik:
|
|
image: traefik:v3.4
|
|
container_name: scadabridge-traefik
|
|
ports:
|
|
- "9000:80" # Central load-balanced entrypoint
|
|
- "8180:8080" # Traefik dashboard
|
|
volumes:
|
|
- ./traefik/traefik.yml:/etc/traefik/traefik.yml:ro
|
|
- ./traefik/dynamic.yml:/etc/traefik/dynamic.yml:ro
|
|
networks:
|
|
- scadabridge-net
|
|
restart: unless-stopped
|
|
|
|
networks:
|
|
scadabridge-net:
|
|
external: true
|