dohertj2/ScadaBridge
1
0
Fork 0
Code Issues 2 Pull Requests 1 Actions Packages Projects Releases Wiki Activity
Files
43e87a7492f48e479a3191a3e9ae5fe339859352
ScadaBridge/docker/docker-compose.yml
T
Joseph Doherty 9d5cf7100e test(secrets): live gate for the gRPC secrets hub — 3/4 PASS, not merged 
Rig config enabling the pull-only hub on the docker cluster (central pair hosts,
site-a pair follows; site-b and site-c deliberately left off so the default-OFF
posture is proven side by side), plus the gate record.

Checks 1-3 PASS. A central write reaches both site-a nodes in 5 s with a
byte-identical ciphertext row and decrypts correctly on both; a site pair boots
and serves its full last-known-good store with the entire central pair stopped,
warning once per interval without crashing, and resumes convergence unaided when
central returns; a tombstone propagates in under 9 s and survives a pair restart
with central up and sweeping, without resurrecting.

Check 4 FAILS one clause of three. Both auth negatives - absent bearer and wrong
bearer - are denied with a byte-identical Unauthenticated status and detail, and
a fleet-wide grep of all eight nodes' docker logs and on-disk Serilog files finds
ZERO occurrences of the dev token, the dev KEK or either plaintext. But the
criterion also asks for a server-side WARNING on denial, and there is none: the
only record is one Information line per call from Grpc.AspNetCore.Server, because
SecretsHubAuthInterceptor deliberately logs nothing on a denial and warns only
when no token is configured at all. That is a property of the 0.4.0 library, not
of this branch, and it is not patched here - a host-side interceptor would
contradict a documented library decision at the wrong layer and put an unbounded
log write on an unauthenticated endpoint.

The merge condition is 4/4, so this branch is NOT merged. The library's denial
logging is the only thing between this result and a merge.

Two residuals worth carrying: the hub client dials a single endpoint and does not
fail over (observed live, and contrasted against CentralGrpcEndpoints failing over
on the same node in the same minute), and the central pair does not converge with
itself - central-b answered an authenticated GetManifest with an empty manifest
for the whole run while central-a held both secrets. Together those make "which
central node is authoritative for secrets" one question, not two.

Rig config notes: Secrets__SqlitePath points at /app/data because the appsettings
default resolves to /app inside the image's writable layer, so the central pair
gained the per-node data volume the site pairs already had. All values are
dev-only and committed under the same exception the mesh PSKs already use.

Also recorded: a gate-METHOD defect. Seeding the bind-mounted store from the macOS
host is not coherent with the running container - the row was visible to the host
and to a fresh container but never to the node, and was lost outright on restart.
Every store access was redone from a throwaway container. The failure mode is a
convincing false negative that looks exactly like a broken hub.

Claude-Session: https://claude.ai/code/session_014WNM4vjoVksyyBraTXSZE1
2026-08-07 08:01:55 -04:00

298 lines
16 KiB
YAML
Raw Blame History

# ── Clustered secret replication: the pull-only gRPC hub (scadaproj#3) ─────────
#
# Central hosts the hub on its EXISTING h2c control-plane listener (CentralGrpcPort
# 8083, alongside CentralControlService); each site node sweeps it on an interval and
# writes what it pulls into its OWN local SQLite store. Nothing but ciphertext crosses
# the wire, so every participating node must resolve the SAME KEK.
#
# ENABLED ON FOUR NODES ONLY: the central pair (hub) and the site-a pair (followers).
# site-b and site-c are deliberately left without it, so the default-OFF posture is
# proven side by side on one rig — exactly as site-a is the rig's only LocalDb-replicated
# pair. A node with no Secrets__* override keeps the shipped appsettings default
# (Replication:Enabled=false, Mode=SqlServer) and composes a plain local store.
#
# ALL VALUES HERE ARE DEV-ONLY and committed under the same exception as the mesh PSKs
# and the ApiKeyPepper above: a local docker rig needs a working credential in source
# control to boot. Production supplies the KEK out of band (ZB_SECRETS_MASTER_KEY, never
# committed) and the hub token from appsettings/env — NEVER as a ${secret:} reference,
# since resolving one is what the hub exists to make possible.
x-secrets-hub-env: &secrets-hub-env
# DEV-ONLY KEK — NOT a real key. Identical on all four participating nodes: only
# ciphertext replicates, so a node with a different KEK fails closed on resolve with a
# kek_id mismatch that reads like corruption but is a deployment error.
ZB_SECRETS_MASTER_KEY: "zZiBWuoaVMbJmGXToLk9Lakw0iJozXoL/7Gxac3GwJ4="
# The appsettings default is the relative "scadabridge-secrets.db", which resolves to
# /app — inside the image's writable layer, so it is destroyed by any container
# recreate and unreachable from the host. /app/data is the node's own mounted volume
# (the one LocalDb already uses on sites; added to the central pair for this).
Secrets__SqlitePath: "/app/data/scadabridge-secrets.db"
Secrets__Replication__Enabled: "true"
Secrets__Replication__Mode: "Grpc"
# DEV-ONLY shared bearer token — NOT a real secret. Presented by every follower and
# verified by the hub's fail-closed SecretsHubAuthInterceptor. Must be IDENTICAL on the
# hub and every follower; an unset token is a startup failure on both halves.
Secrets__GrpcHub__BearerToken: "secrets-hub-docker-dev-token"
# Site half of the same section. NOTE the asymmetry with
# ScadaBridge:Communication:CentralGrpcEndpoints, which is a LIST that fails over across
# the central pair: the hub client dials a SINGLE endpoint, so a sweep against a stopped
# central-a stalls rather than failing over to central-b. That is survivable — the sweep
# is best-effort and the node keeps serving its full local last-known-good store — but
# secrets stop converging until central-a returns.
x-secrets-hub-site-env: &secrets-hub-site-env
Secrets__GrpcHub__Endpoint: "http://scadabridge-central-a:8083"
services:
central-a:
image: scadabridge:latest
# CoordinatedShutdown needs cluster-leave (15s budget) + cluster-exiting +
# actor-system-terminate + Serilog flush; the 10s SIGTERM default SIGKILLed
# mid-drain, turning every redeploy into the crash path (review 01 [Medium]).
stop_grace_period: 30s
container_name: scadabridge-central-a
environment:
# Hub half of the pull-only gRPC secrets hub (anchor at the top of this file).
<<: *secrets-hub-env
SCADABRIDGE_CONFIG: Central
ASPNETCORE_ENVIRONMENT: Development
ASPNETCORE_URLS: "http://+:5000"
# DEV-ONLY local-cluster value — NOT a real secret. The Auth/Config normalization
# (2026-06-03) made ScadaBridge:InboundApi:ApiKeyPepper a hard Central startup
# requirement (>=16 chars, per-environment). Real deployments inject a true secret
# out-of-band (env/secret store), never from source control — see
# docs/operations/inbound-api-key-reissue.md. Both Central nodes share one pepper.
ScadaBridge__InboundApi__ApiKeyPepper: "dev-only-insecure-pepper-docker-cluster-0001"
# DEV-ONLY local-cluster secrets — NOT real secrets. Relocated out of the mounted
# appsettings.Central.json (G-4 T5 secrets cleanup) so that app-config file carries no
# plaintext credentials. Env overrides layer over JSON BEFORE the ${secret:} pre-host
# expander runs, so the dev cluster boots with no KEK/secret store. The SQL password is
# the same value already committed for the sibling scadabridge-mssql container
# (infra/docker-compose.yml MSSQL_SA_PASSWORD) — consolidation, not new exposure. Real
# deployments use ${secret:} tokens (src/.../appsettings.Central.json) + a seeded store.
ScadaBridge__Database__ConfigurationDb: "Server=scadabridge-mssql,1433;Database=ScadaBridgeConfig;User Id=scadabridge_app;Password=ScadaBridge_Dev1#;TrustServerCertificate=true"
ScadaBridge__Database__MachineDataDb: "Server=scadabridge-mssql,1433;Database=ScadaBridgeMachineData;User Id=scadabridge_app;Password=ScadaBridge_Dev1#;TrustServerCertificate=true"
ScadaBridge__Security__Ldap__ServiceAccountPassword: "serviceaccount123"
ScadaBridge__Security__JwtSigningKey: "scadabridge-dev-jwt-signing-key-must-be-at-least-32-characters-long"
# DEV-ONLY gRPC control-plane preshared keys, one per site — NOT real secrets.
# Central verifies/presents these; each site node carries the same value as
# ScadaBridge:Communication:GrpcPsk in its mounted appsettings.Site.json. Kept as
# env overrides (not in the mounted central appsettings) so that file stays free of
# plaintext credentials. Production instead seeds SB-GRPC-PSK-<siteId> into the
# secret store, which is also the only source that can serve a site added at runtime.
ScadaBridge__Communication__SitePsks__site-a: "dev-grpc-psk-docker-site-a"
ScadaBridge__Communication__SitePsks__site-b: "dev-grpc-psk-docker-site-b"
ScadaBridge__Communication__SitePsks__site-c: "dev-grpc-psk-docker-site-c"
ports:
- "9001:5000" # Web UI + Inbound API
- "9011:8081" # Akka remoting (host access for CLI/debugging)
- "9013:8083" # gRPC control plane (CentralControlService, T1A.2)
volumes:
- ./central-node-a/appsettings.Central.json:/app/appsettings.Central.json:ro
# Added for the gRPC secrets hub: the node's local secret store lives at
# Secrets__SqlitePath=/app/data/scadabridge-secrets.db, so it needs the same
# per-node volume the site nodes already have. Without it the store sits in the
# image's writable layer and is destroyed by every container recreate.
- ./central-node-a/data:/app/data
- ./central-node-a/logs:/app/logs
networks:
- scadabridge-net
restart: unless-stopped
central-b:
image: scadabridge:latest
# CoordinatedShutdown needs cluster-leave (15s budget) + cluster-exiting +
# actor-system-terminate + Serilog flush; the 10s SIGTERM default SIGKILLed
# mid-drain, turning every redeploy into the crash path (review 01 [Medium]).
stop_grace_period: 30s
container_name: scadabridge-central-b
environment:
# Hub half of the pull-only gRPC secrets hub (anchor at the top of this file).
<<: *secrets-hub-env
SCADABRIDGE_CONFIG: Central
ASPNETCORE_ENVIRONMENT: Development
ASPNETCORE_URLS: "http://+:5000"
# DEV-ONLY local-cluster value — NOT a real secret. The Auth/Config normalization
# (2026-06-03) made ScadaBridge:InboundApi:ApiKeyPepper a hard Central startup
# requirement (>=16 chars, per-environment). Real deployments inject a true secret
# out-of-band (env/secret store), never from source control — see
# docs/operations/inbound-api-key-reissue.md. Both Central nodes share one pepper.
ScadaBridge__InboundApi__ApiKeyPepper: "dev-only-insecure-pepper-docker-cluster-0001"
# DEV-ONLY local-cluster secrets — NOT real secrets. Relocated out of the mounted
# appsettings.Central.json (G-4 T5 secrets cleanup) so that app-config file carries no
# plaintext credentials. Env overrides layer over JSON BEFORE the ${secret:} pre-host
# expander runs, so the dev cluster boots with no KEK/secret store. The SQL password is
# the same value already committed for the sibling scadabridge-mssql container
# (infra/docker-compose.yml MSSQL_SA_PASSWORD) — consolidation, not new exposure. Real
# deployments use ${secret:} tokens (src/.../appsettings.Central.json) + a seeded store.
ScadaBridge__Database__ConfigurationDb: "Server=scadabridge-mssql,1433;Database=ScadaBridgeConfig;User Id=scadabridge_app;Password=ScadaBridge_Dev1#;TrustServerCertificate=true"
ScadaBridge__Database__MachineDataDb: "Server=scadabridge-mssql,1433;Database=ScadaBridgeMachineData;User Id=scadabridge_app;Password=ScadaBridge_Dev1#;TrustServerCertificate=true"
ScadaBridge__Security__Ldap__ServiceAccountPassword: "serviceaccount123"
ScadaBridge__Security__JwtSigningKey: "scadabridge-dev-jwt-signing-key-must-be-at-least-32-characters-long"
# DEV-ONLY gRPC control-plane preshared keys, one per site — NOT real secrets.
# Central verifies/presents these; each site node carries the same value as
# ScadaBridge:Communication:GrpcPsk in its mounted appsettings.Site.json. Kept as
# env overrides (not in the mounted central appsettings) so that file stays free of
# plaintext credentials. Production instead seeds SB-GRPC-PSK-<siteId> into the
# secret store, which is also the only source that can serve a site added at runtime.
ScadaBridge__Communication__SitePsks__site-a: "dev-grpc-psk-docker-site-a"
ScadaBridge__Communication__SitePsks__site-b: "dev-grpc-psk-docker-site-b"
ScadaBridge__Communication__SitePsks__site-c: "dev-grpc-psk-docker-site-c"
ports:
- "9002:5000" # Web UI + Inbound API
- "9012:8081" # Akka remoting
- "9014:8083" # gRPC control plane (CentralControlService, T1A.2)
volumes:
- ./central-node-b/appsettings.Central.json:/app/appsettings.Central.json:ro
# Added for the gRPC secrets hub: the node's local secret store lives at
# Secrets__SqlitePath=/app/data/scadabridge-secrets.db, so it needs the same
# per-node volume the site nodes already have. Without it the store sits in the
# image's writable layer and is destroyed by every container recreate.
- ./central-node-b/data:/app/data
- ./central-node-b/logs:/app/logs
networks:
- scadabridge-net
restart: unless-stopped
site-a-a:
image: scadabridge:latest
# CoordinatedShutdown needs cluster-leave (15s budget) + cluster-exiting +
# actor-system-terminate + Serilog flush; the 10s SIGTERM default SIGKILLed
# mid-drain, turning every redeploy into the crash path (review 01 [Medium]).
stop_grace_period: 30s
container_name: scadabridge-site-a-a
environment:
# Follower half of the pull-only gRPC secrets hub (anchors at the top of this
# file). site-b and site-c deliberately carry neither.
<<: [*secrets-hub-env, *secrets-hub-site-env]
SCADABRIDGE_CONFIG: Site
ports:
- "9021:8082" # Akka remoting (host access for debugging)
- "9023:8083" # gRPC streaming
volumes:
- ./site-a-node-a/appsettings.Site.json:/app/appsettings.Site.json:ro
- ./site-a-node-a/data:/app/data
- ./site-a-node-a/logs:/app/logs
networks:
- scadabridge-net
restart: unless-stopped
site-a-b:
image: scadabridge:latest
# CoordinatedShutdown needs cluster-leave (15s budget) + cluster-exiting +
# actor-system-terminate + Serilog flush; the 10s SIGTERM default SIGKILLed
# mid-drain, turning every redeploy into the crash path (review 01 [Medium]).
stop_grace_period: 30s
container_name: scadabridge-site-a-b
environment:
# Follower half of the pull-only gRPC secrets hub (anchors at the top of this
# file). site-b and site-c deliberately carry neither.
<<: [*secrets-hub-env, *secrets-hub-site-env]
SCADABRIDGE_CONFIG: Site
ports:
- "9022:8082" # Akka remoting
- "9024:8083" # gRPC streaming
volumes:
- ./site-a-node-b/appsettings.Site.json:/app/appsettings.Site.json:ro
- ./site-a-node-b/data:/app/data
- ./site-a-node-b/logs:/app/logs
networks:
- scadabridge-net
restart: unless-stopped
site-b-a:
image: scadabridge:latest
# CoordinatedShutdown needs cluster-leave (15s budget) + cluster-exiting +
# actor-system-terminate + Serilog flush; the 10s SIGTERM default SIGKILLed
# mid-drain, turning every redeploy into the crash path (review 01 [Medium]).
stop_grace_period: 30s
container_name: scadabridge-site-b-a
environment:
SCADABRIDGE_CONFIG: Site
ports:
- "9031:8082" # Akka remoting
- "9033:8083" # gRPC streaming
volumes:
- ./site-b-node-a/appsettings.Site.json:/app/appsettings.Site.json:ro
- ./site-b-node-a/data:/app/data
- ./site-b-node-a/logs:/app/logs
networks:
- scadabridge-net
restart: unless-stopped
site-b-b:
image: scadabridge:latest
# CoordinatedShutdown needs cluster-leave (15s budget) + cluster-exiting +
# actor-system-terminate + Serilog flush; the 10s SIGTERM default SIGKILLed
# mid-drain, turning every redeploy into the crash path (review 01 [Medium]).
stop_grace_period: 30s
container_name: scadabridge-site-b-b
environment:
SCADABRIDGE_CONFIG: Site
ports:
- "9032:8082" # Akka remoting
- "9034:8083" # gRPC streaming
volumes:
- ./site-b-node-b/appsettings.Site.json:/app/appsettings.Site.json:ro
- ./site-b-node-b/data:/app/data
- ./site-b-node-b/logs:/app/logs
networks:
- scadabridge-net
restart: unless-stopped
site-c-a:
image: scadabridge:latest
# CoordinatedShutdown needs cluster-leave (15s budget) + cluster-exiting +
# actor-system-terminate + Serilog flush; the 10s SIGTERM default SIGKILLed
# mid-drain, turning every redeploy into the crash path (review 01 [Medium]).
stop_grace_period: 30s
container_name: scadabridge-site-c-a
environment:
SCADABRIDGE_CONFIG: Site
ports:
- "9041:8082" # Akka remoting
- "9043:8083" # gRPC streaming
volumes:
- ./site-c-node-a/appsettings.Site.json:/app/appsettings.Site.json:ro
- ./site-c-node-a/data:/app/data
- ./site-c-node-a/logs:/app/logs
networks:
- scadabridge-net
restart: unless-stopped
site-c-b:
image: scadabridge:latest
# CoordinatedShutdown needs cluster-leave (15s budget) + cluster-exiting +
# actor-system-terminate + Serilog flush; the 10s SIGTERM default SIGKILLed
# mid-drain, turning every redeploy into the crash path (review 01 [Medium]).
stop_grace_period: 30s
container_name: scadabridge-site-c-b
environment:
SCADABRIDGE_CONFIG: Site
ports:
- "9042:8082" # Akka remoting
- "9044:8083" # gRPC streaming
volumes:
- ./site-c-node-b/appsettings.Site.json:/app/appsettings.Site.json:ro
- ./site-c-node-b/data:/app/data
- ./site-c-node-b/logs:/app/logs
networks:
- scadabridge-net
restart: unless-stopped
traefik:
image: traefik:v3.4
container_name: scadabridge-traefik
ports:
- "9000:80" # Central load-balanced entrypoint
- "8180:8080" # Traefik dashboard
volumes:
- ./traefik/traefik.yml:/etc/traefik/traefik.yml:ro
- ./traefik/dynamic.yml:/etc/traefik/dynamic.yml:ro
networks:
- scadabridge-net
restart: unless-stopped
networks:
scadabridge-net:
external: true
Reference in New Issue View Git Blame Copy Permalink