Files
ScadaBridge/docker/docker-compose.yml
T
Joseph Doherty da1669f87a fix(docker): give every node an explicit secret store path, not just the hub four
The 0.6.2 adoption set Secrets__SqlitePath inside x-secrets-hub-env, which only the
central pair and site-a merge — clustered replication is deliberately enabled on four
nodes so the default-OFF posture is proven side by side. site-b and site-c merge no
secrets anchor at all, so they fell back to the shipped default and crash-looped at
boot (exit 134, nine times each) while the other four came up clean.

Splitting the store path into its own anchor that all eight merge fixes it. The path
is not optional config: every node has a store, only four have a hub.

Also corrects the assumption behind deleting the key from appsettings.json. Measured
here: with a Secrets section present but no SqlitePath, the effective value is the
RELATIVE "secrets.db", which fails both validation rules — the per-user default does
not take over. A deployment must set it explicitly.

Verified live: all 8 nodes up, zero validation errors, stores at /data on the same
host directory as before, all four pairs converged to one cluster each through a
simultaneous recreate, central-a active / central-b standby, secrets hub sweeps
recovered after the expected boot race.
2026-08-11 10:26:03 -04:00

379 lines
21 KiB
YAML

# EVERY node needs a secret store path; only four take the hub anchor below. Keep this
# anchor separate from that one — folding it in is what crash-looped site-b and site-c on
# 2026-08-11: they merge no secrets anchor, fell back to the shipped default, and were
# rejected at boot.
#
# The value must be absolute AND outside the content root (/app) — Secrets 0.6.2 validates
# both at startup, because a store inside the deployment directory is destroyed by an
# ordinary upgrade (this wiped the MxGateway key store on 2026-08-09). /data is the node's
# own bind mount, the SAME host directory as /app/data, so an existing store is found
# unmoved; only the path used to reach it changes. Do NOT point this back at /app/data —
# absolute is not sufficient, and the boot will fail.
#
# Do NOT delete this key expecting the library's per-user default to take over. Measured on
# this rig: with a Secrets section present but no SqlitePath, the effective value is the
# RELATIVE "secrets.db", which fails both rules. Setting it explicitly is the supported
# posture for a deployment in any case.
x-secrets-store-env: &secrets-store-env
Secrets__SqlitePath: "/data/scadabridge-secrets.db"
# ── Clustered secret replication: the pull-only gRPC hub (scadaproj#3) ─────────
#
# Central hosts the hub on its EXISTING h2c control-plane listener (CentralGrpcPort
# 8083, alongside CentralControlService); each site node sweeps it on an interval and
# writes what it pulls into its OWN local SQLite store. Nothing but ciphertext crosses
# the wire, so every participating node must resolve the SAME KEK.
#
# ENABLED ON FOUR NODES ONLY: the central pair (hub) and the site-a pair (followers).
# site-b and site-c are deliberately left without it, so the default-OFF posture is
# proven side by side on one rig — exactly as site-a is the rig's only LocalDb-replicated
# pair. A node with no Secrets__* override keeps the shipped appsettings default
# (Replication:Enabled=false, Mode=SqlServer) and composes a plain local store.
#
# ALL VALUES HERE ARE DEV-ONLY and committed under the same exception as the mesh PSKs
# and the ApiKeyPepper above: a local docker rig needs a working credential in source
# control to boot. Production supplies the KEK out of band (ZB_SECRETS_MASTER_KEY, never
# committed) and the hub token from appsettings/env — NEVER as a ${secret:} reference,
# since resolving one is what the hub exists to make possible.
x-secrets-hub-env: &secrets-hub-env
# DEV-ONLY KEK — NOT a real key. Identical on all four participating nodes: only
# ciphertext replicates, so a node with a different KEK fails closed on resolve with a
# kek_id mismatch that reads like corruption but is a deployment error.
ZB_SECRETS_MASTER_KEY: "zZiBWuoaVMbJmGXToLk9Lakw0iJozXoL/7Gxac3GwJ4="
Secrets__Replication__Enabled: "true"
Secrets__Replication__Mode: "Grpc"
# DEV-ONLY shared bearer token — NOT a real secret. Presented by every follower and
# verified by the hub's fail-closed SecretsHubAuthInterceptor. Must be IDENTICAL on the
# hub and every follower; an unset token is a startup failure on both halves.
Secrets__GrpcHub__BearerToken: "secrets-hub-docker-dev-token"
# Central-only half: the SHARED SQL-Server secret store (scadaproj#4, Secrets 0.5.0).
# In Grpc mode BOTH central nodes read and write ONE copy of every row in this database,
# so the two hub instances serve identical manifests by construction — that is what makes
# the site-side FallbackEndpoints below safe. ZbSecretsHub is a dedicated database on the
# rig's existing scadabridge-mssql container (default zbsecrets schema, created by the
# boot-time SqlServerSecretsStoreMigrator; the database itself + the scadabridge_app grant
# were provisioned once via sqlcmd — see docs/plans/2026-08-07-secrets-central-shared-store-live-gate.md).
# Same DEV-ONLY credentials as the ScadaBridge__Database__* strings above. Must be a
# LITERAL value, never a ${secret:} reference — the pre-host expander needs this string to
# reach the store that would resolve it (bootstrap circularity; registration rejects it).
# Site nodes must NEVER carry this key: sites talk to central, not to central's database.
x-secrets-hub-central-env: &secrets-hub-central-env
Secrets__SqlServer__ConnectionString: "Server=scadabridge-mssql,1433;Database=ZbSecretsHub;User Id=scadabridge_app;Password=ScadaBridge_Dev1#;TrustServerCertificate=true"
# Site half of the same section. The hub client dials Endpoint first and fails over, per
# call, to FallbackEndpoints in order (sticky on whichever answered last, Secrets 0.5.0).
# Listing central-b is safe ONLY because both centrals serve the one shared SQL store
# above — never list endpoints backed by independent stores: failing over to an emptier
# hub is a silent convergence stop, the exact defect (scadaproj#4) the shared store
# exists to prevent.
x-secrets-hub-site-env: &secrets-hub-site-env
Secrets__GrpcHub__Endpoint: "http://scadabridge-central-a:8083"
Secrets__GrpcHub__FallbackEndpoints__0: "http://scadabridge-central-b:8083"
services:
central-a:
image: scadabridge:latest
# An init process (tini) as PID 1, so a crashed dotnet process actually dies.
# Without it dotnet IS PID 1, Linux ignores the SIGABRT the runtime's crash path
# raises against PID 1, and any unhandled boot exception left the container
# `running` with the main thread spinning at 100% CPU — restart policy never
# fired (ScadaBridge#34). Belt to Program.cs's UnhandledException handler, which
# covers managed exceptions but not FailFast/runtime-internal aborts.
init: true
# CoordinatedShutdown needs cluster-leave (15s budget) + cluster-exiting +
# actor-system-terminate + Serilog flush; the 10s SIGTERM default SIGKILLed
# mid-drain, turning every redeploy into the crash path (review 01 [Medium]).
stop_grace_period: 30s
container_name: scadabridge-central-a
environment:
# Hub half of the pull-only gRPC secrets hub + the central-only shared SQL store
# (anchors at the top of this file).
<<: [*secrets-store-env, *secrets-hub-env, *secrets-hub-central-env]
SCADABRIDGE_CONFIG: Central
ASPNETCORE_ENVIRONMENT: Development
ASPNETCORE_URLS: "http://+:5000"
# DEV-ONLY local-cluster value — NOT a real secret. The Auth/Config normalization
# (2026-06-03) made ScadaBridge:InboundApi:ApiKeyPepper a hard Central startup
# requirement (>=16 chars, per-environment). Real deployments inject a true secret
# out-of-band (env/secret store), never from source control — see
# docs/operations/inbound-api-key-reissue.md. Both Central nodes share one pepper.
ScadaBridge__InboundApi__ApiKeyPepper: "dev-only-insecure-pepper-docker-cluster-0001"
# DEV-ONLY local-cluster secrets — NOT real secrets. Relocated out of the mounted
# appsettings.Central.json (G-4 T5 secrets cleanup) so that app-config file carries no
# plaintext credentials. Env overrides layer over JSON BEFORE the ${secret:} pre-host
# expander runs, so the dev cluster boots with no KEK/secret store. The SQL password is
# the same value already committed for the sibling scadabridge-mssql container
# (infra/docker-compose.yml MSSQL_SA_PASSWORD) — consolidation, not new exposure. Real
# deployments use ${secret:} tokens (src/.../appsettings.Central.json) + a seeded store.
ScadaBridge__Database__ConfigurationDb: "Server=scadabridge-mssql,1433;Database=ScadaBridgeConfig;User Id=scadabridge_app;Password=ScadaBridge_Dev1#;TrustServerCertificate=true"
ScadaBridge__Database__MachineDataDb: "Server=scadabridge-mssql,1433;Database=ScadaBridgeMachineData;User Id=scadabridge_app;Password=ScadaBridge_Dev1#;TrustServerCertificate=true"
ScadaBridge__Security__Ldap__ServiceAccountPassword: "serviceaccount123"
ScadaBridge__Security__JwtSigningKey: "scadabridge-dev-jwt-signing-key-must-be-at-least-32-characters-long"
# DEV-ONLY gRPC control-plane preshared keys, one per site — NOT real secrets.
# Central verifies/presents these; each site node carries the same value as
# ScadaBridge:Communication:GrpcPsk in its mounted appsettings.Site.json. Kept as
# env overrides (not in the mounted central appsettings) so that file stays free of
# plaintext credentials. Production instead seeds SB-GRPC-PSK-<siteId> into the
# secret store, which is also the only source that can serve a site added at runtime.
ScadaBridge__Communication__SitePsks__site-a: "dev-grpc-psk-docker-site-a"
ScadaBridge__Communication__SitePsks__site-b: "dev-grpc-psk-docker-site-b"
ScadaBridge__Communication__SitePsks__site-c: "dev-grpc-psk-docker-site-c"
ports:
- "9001:5000" # Web UI + Inbound API
- "9011:8081" # Akka remoting (host access for CLI/debugging)
- "9013:8083" # gRPC control plane (CentralControlService, T1A.2)
volumes:
- ./central-node-a/appsettings.Central.json:/app/appsettings.Central.json:ro
# Originally added for the gRPC secrets hub's local SQLite store; since the
# central store moved to the shared SQL-Server database (Secrets 0.5.0,
# scadaproj#4) the scadabridge-secrets.db here is a pre-0.5.0 residue, but the
# volume is still needed (inbound-api-keys.sqlite lives on it).
- ./central-node-a/data:/app/data
# Same host directory, mounted a second time OUTSIDE the content root (/app).
# Secrets 0.6.2 refuses a store path under the content root; this bind mount is
# not touched by an image rebuild, but the guard is lexical and cannot know that.
- ./central-node-a/data:/data
- ./central-node-a/logs:/app/logs
networks:
- scadabridge-net
restart: unless-stopped
central-b:
image: scadabridge:latest
init: true # PID-1 crash wedge, ScadaBridge#34 — see central-a
# CoordinatedShutdown needs cluster-leave (15s budget) + cluster-exiting +
# actor-system-terminate + Serilog flush; the 10s SIGTERM default SIGKILLed
# mid-drain, turning every redeploy into the crash path (review 01 [Medium]).
stop_grace_period: 30s
container_name: scadabridge-central-b
environment:
# Hub half of the pull-only gRPC secrets hub + the central-only shared SQL store
# (anchors at the top of this file).
<<: [*secrets-store-env, *secrets-hub-env, *secrets-hub-central-env]
SCADABRIDGE_CONFIG: Central
ASPNETCORE_ENVIRONMENT: Development
ASPNETCORE_URLS: "http://+:5000"
# DEV-ONLY local-cluster value — NOT a real secret. The Auth/Config normalization
# (2026-06-03) made ScadaBridge:InboundApi:ApiKeyPepper a hard Central startup
# requirement (>=16 chars, per-environment). Real deployments inject a true secret
# out-of-band (env/secret store), never from source control — see
# docs/operations/inbound-api-key-reissue.md. Both Central nodes share one pepper.
ScadaBridge__InboundApi__ApiKeyPepper: "dev-only-insecure-pepper-docker-cluster-0001"
# DEV-ONLY local-cluster secrets — NOT real secrets. Relocated out of the mounted
# appsettings.Central.json (G-4 T5 secrets cleanup) so that app-config file carries no
# plaintext credentials. Env overrides layer over JSON BEFORE the ${secret:} pre-host
# expander runs, so the dev cluster boots with no KEK/secret store. The SQL password is
# the same value already committed for the sibling scadabridge-mssql container
# (infra/docker-compose.yml MSSQL_SA_PASSWORD) — consolidation, not new exposure. Real
# deployments use ${secret:} tokens (src/.../appsettings.Central.json) + a seeded store.
ScadaBridge__Database__ConfigurationDb: "Server=scadabridge-mssql,1433;Database=ScadaBridgeConfig;User Id=scadabridge_app;Password=ScadaBridge_Dev1#;TrustServerCertificate=true"
ScadaBridge__Database__MachineDataDb: "Server=scadabridge-mssql,1433;Database=ScadaBridgeMachineData;User Id=scadabridge_app;Password=ScadaBridge_Dev1#;TrustServerCertificate=true"
ScadaBridge__Security__Ldap__ServiceAccountPassword: "serviceaccount123"
ScadaBridge__Security__JwtSigningKey: "scadabridge-dev-jwt-signing-key-must-be-at-least-32-characters-long"
# DEV-ONLY gRPC control-plane preshared keys, one per site — NOT real secrets.
# Central verifies/presents these; each site node carries the same value as
# ScadaBridge:Communication:GrpcPsk in its mounted appsettings.Site.json. Kept as
# env overrides (not in the mounted central appsettings) so that file stays free of
# plaintext credentials. Production instead seeds SB-GRPC-PSK-<siteId> into the
# secret store, which is also the only source that can serve a site added at runtime.
ScadaBridge__Communication__SitePsks__site-a: "dev-grpc-psk-docker-site-a"
ScadaBridge__Communication__SitePsks__site-b: "dev-grpc-psk-docker-site-b"
ScadaBridge__Communication__SitePsks__site-c: "dev-grpc-psk-docker-site-c"
ports:
- "9002:5000" # Web UI + Inbound API
- "9012:8081" # Akka remoting
- "9014:8083" # gRPC control plane (CentralControlService, T1A.2)
volumes:
- ./central-node-b/appsettings.Central.json:/app/appsettings.Central.json:ro
# Originally added for the gRPC secrets hub's local SQLite store; since the
# central store moved to the shared SQL-Server database (Secrets 0.5.0,
# scadaproj#4) the scadabridge-secrets.db here is a pre-0.5.0 residue, but the
# volume is still needed (inbound-api-keys.sqlite lives on it).
- ./central-node-b/data:/app/data
# Same host directory, mounted a second time OUTSIDE the content root (/app).
# Secrets 0.6.2 refuses a store path under the content root; this bind mount is
# not touched by an image rebuild, but the guard is lexical and cannot know that.
- ./central-node-b/data:/data
- ./central-node-b/logs:/app/logs
networks:
- scadabridge-net
restart: unless-stopped
site-a-a:
image: scadabridge:latest
init: true # PID-1 crash wedge, ScadaBridge#34 — see central-a
# CoordinatedShutdown needs cluster-leave (15s budget) + cluster-exiting +
# actor-system-terminate + Serilog flush; the 10s SIGTERM default SIGKILLed
# mid-drain, turning every redeploy into the crash path (review 01 [Medium]).
stop_grace_period: 30s
container_name: scadabridge-site-a-a
environment:
# Follower half of the pull-only gRPC secrets hub (anchors at the top of this
# file). site-b and site-c deliberately carry neither.
<<: [*secrets-store-env, *secrets-hub-env, *secrets-hub-site-env]
SCADABRIDGE_CONFIG: Site
ports:
- "9021:8082" # Akka remoting (host access for debugging)
- "9023:8083" # gRPC streaming
volumes:
- ./site-a-node-a/appsettings.Site.json:/app/appsettings.Site.json:ro
- ./site-a-node-a/data:/app/data
# Same host directory, mounted a second time OUTSIDE the content root (/app).
# Secrets 0.6.2 refuses a store path under the content root; this bind mount is
# not touched by an image rebuild, but the guard is lexical and cannot know that.
- ./site-a-node-a/data:/data
- ./site-a-node-a/logs:/app/logs
networks:
- scadabridge-net
restart: unless-stopped
site-a-b:
image: scadabridge:latest
init: true # PID-1 crash wedge, ScadaBridge#34 — see central-a
# CoordinatedShutdown needs cluster-leave (15s budget) + cluster-exiting +
# actor-system-terminate + Serilog flush; the 10s SIGTERM default SIGKILLed
# mid-drain, turning every redeploy into the crash path (review 01 [Medium]).
stop_grace_period: 30s
container_name: scadabridge-site-a-b
environment:
# Follower half of the pull-only gRPC secrets hub (anchors at the top of this
# file). site-b and site-c deliberately carry neither.
<<: [*secrets-store-env, *secrets-hub-env, *secrets-hub-site-env]
SCADABRIDGE_CONFIG: Site
ports:
- "9022:8082" # Akka remoting
- "9024:8083" # gRPC streaming
volumes:
- ./site-a-node-b/appsettings.Site.json:/app/appsettings.Site.json:ro
- ./site-a-node-b/data:/app/data
# Same host directory, mounted a second time OUTSIDE the content root (/app).
# Secrets 0.6.2 refuses a store path under the content root; this bind mount is
# not touched by an image rebuild, but the guard is lexical and cannot know that.
- ./site-a-node-b/data:/data
- ./site-a-node-b/logs:/app/logs
networks:
- scadabridge-net
restart: unless-stopped
site-b-a:
image: scadabridge:latest
init: true # PID-1 crash wedge, ScadaBridge#34 — see central-a
# CoordinatedShutdown needs cluster-leave (15s budget) + cluster-exiting +
# actor-system-terminate + Serilog flush; the 10s SIGTERM default SIGKILLed
# mid-drain, turning every redeploy into the crash path (review 01 [Medium]).
stop_grace_period: 30s
container_name: scadabridge-site-b-a
environment:
<<: *secrets-store-env
SCADABRIDGE_CONFIG: Site
ports:
- "9031:8082" # Akka remoting
- "9033:8083" # gRPC streaming
volumes:
- ./site-b-node-a/appsettings.Site.json:/app/appsettings.Site.json:ro
- ./site-b-node-a/data:/app/data
# Same host directory, mounted a second time OUTSIDE the content root (/app).
# Secrets 0.6.2 refuses a store path under the content root; this bind mount is
# not touched by an image rebuild, but the guard is lexical and cannot know that.
- ./site-b-node-a/data:/data
- ./site-b-node-a/logs:/app/logs
networks:
- scadabridge-net
restart: unless-stopped
site-b-b:
image: scadabridge:latest
init: true # PID-1 crash wedge, ScadaBridge#34 — see central-a
# CoordinatedShutdown needs cluster-leave (15s budget) + cluster-exiting +
# actor-system-terminate + Serilog flush; the 10s SIGTERM default SIGKILLed
# mid-drain, turning every redeploy into the crash path (review 01 [Medium]).
stop_grace_period: 30s
container_name: scadabridge-site-b-b
environment:
<<: *secrets-store-env
SCADABRIDGE_CONFIG: Site
ports:
- "9032:8082" # Akka remoting
- "9034:8083" # gRPC streaming
volumes:
- ./site-b-node-b/appsettings.Site.json:/app/appsettings.Site.json:ro
- ./site-b-node-b/data:/app/data
# Same host directory, mounted a second time OUTSIDE the content root (/app).
# Secrets 0.6.2 refuses a store path under the content root; this bind mount is
# not touched by an image rebuild, but the guard is lexical and cannot know that.
- ./site-b-node-b/data:/data
- ./site-b-node-b/logs:/app/logs
networks:
- scadabridge-net
restart: unless-stopped
site-c-a:
image: scadabridge:latest
init: true # PID-1 crash wedge, ScadaBridge#34 — see central-a
# CoordinatedShutdown needs cluster-leave (15s budget) + cluster-exiting +
# actor-system-terminate + Serilog flush; the 10s SIGTERM default SIGKILLed
# mid-drain, turning every redeploy into the crash path (review 01 [Medium]).
stop_grace_period: 30s
container_name: scadabridge-site-c-a
environment:
<<: *secrets-store-env
SCADABRIDGE_CONFIG: Site
ports:
- "9041:8082" # Akka remoting
- "9043:8083" # gRPC streaming
volumes:
- ./site-c-node-a/appsettings.Site.json:/app/appsettings.Site.json:ro
- ./site-c-node-a/data:/app/data
# Same host directory, mounted a second time OUTSIDE the content root (/app).
# Secrets 0.6.2 refuses a store path under the content root; this bind mount is
# not touched by an image rebuild, but the guard is lexical and cannot know that.
- ./site-c-node-a/data:/data
- ./site-c-node-a/logs:/app/logs
networks:
- scadabridge-net
restart: unless-stopped
site-c-b:
image: scadabridge:latest
init: true # PID-1 crash wedge, ScadaBridge#34 — see central-a
# CoordinatedShutdown needs cluster-leave (15s budget) + cluster-exiting +
# actor-system-terminate + Serilog flush; the 10s SIGTERM default SIGKILLed
# mid-drain, turning every redeploy into the crash path (review 01 [Medium]).
stop_grace_period: 30s
container_name: scadabridge-site-c-b
environment:
<<: *secrets-store-env
SCADABRIDGE_CONFIG: Site
ports:
- "9042:8082" # Akka remoting
- "9044:8083" # gRPC streaming
volumes:
- ./site-c-node-b/appsettings.Site.json:/app/appsettings.Site.json:ro
- ./site-c-node-b/data:/app/data
# Same host directory, mounted a second time OUTSIDE the content root (/app).
# Secrets 0.6.2 refuses a store path under the content root; this bind mount is
# not touched by an image rebuild, but the guard is lexical and cannot know that.
- ./site-c-node-b/data:/data
- ./site-c-node-b/logs:/app/logs
networks:
- scadabridge-net
restart: unless-stopped
traefik:
image: traefik:v3.4
container_name: scadabridge-traefik
ports:
- "9000:80" # Central load-balanced entrypoint
- "8180:8080" # Traefik dashboard
volumes:
- ./traefik/traefik.yml:/etc/traefik/traefik.yml:ro
- ./traefik/dynamic.yml:/etc/traefik/dynamic.yml:ro
networks:
- scadabridge-net
restart: unless-stopped
networks:
scadabridge-net:
external: true