ScadaBridge/docker

ScadaBridge Docker Infrastructure

Local Docker deployment of the full ScadaBridge cluster topology: a 2-node central cluster and three 2-node site clusters.

Cluster Topology

              ┌───────────────────┐
              │  Traefik LB :9000 │  ◄── CLI / Browser
              │  Dashboard :8180  │
              └────────┬──────────┘
                       │ routes to active node
┌──────────────────────┼──────────────────────────────┐
│                  Central Cluster                    │
│                                                     │
│  ┌─────────────────┐     ┌─────────────────┐        │
│  │  central-node-a  │◄──►│  central-node-b  │       │
│  │  (leader/oldest) │     │  (standby)       │       │
│  │  Web UI :9001    │     │  Web UI :9002    │       │
│  │  Akka   :9011    │     │  Akka   :9012    │       │
│  └────────┬─────────┘     └─────────────────┘       │
│           │                                         │
└───────────┼─────────────────────────────────────────┘
            │ Akka.NET Remoting (hub-and-spoke)
            ├──────────────────┬──────────────────┐
            ▼                  ▼                  ▼
┌────────────────────┐ ┌────────────────────┐ ┌────────────────────┐
│  Site-A Cluster    │ │  Site-B Cluster    │ │  Site-C Cluster    │
│  (Test Plant A)    │ │  (Test Plant B)    │ │  (Test Plant C)    │
│                    │ │                    │ │                    │
│  node-a ◄──► node-b│ │  node-a ◄──► node-b│ │  node-a ◄──► node-b│
│  Akka :9021 :9022  │ │  Akka :9031 :9032  │ │  Akka :9041 :9042  │
│  gRPC :9023 :9024  │ │  gRPC :9033 :9034  │ │  gRPC :9043 :9044  │
└────────────────────┘ └────────────────────┘ └────────────────────┘

Central Cluster (active/standby)

Runs the web UI (Blazor Server), Template Engine, Deployment Manager, Security, Inbound API, Management Service, and Health Monitoring. Connects to MS SQL for configuration and machine data, LDAP for authentication, and SMTP for notifications.

Both central nodes run ScadaBridge__InboundApi__ApiKeyStore__Provider: "SqlServer" (docker-compose.yml), so inbound API keys live in ONE shared zbauth schema in ScadaBridgeConfig rather than a per-node SQLite file — a key minted through either node survives a central failover and a revoke is cluster-wide (design 2026-09-02, deferred-work register row 108). Each node's legacy /app/data/inbound-api-keys.sqlite was imported once at the first boot on this provider and renamed *.imported. The former sync-inbound-api-keys.sh file-copy workaround is gone; verify-delmia-notifier.sh's preflight now doubles as the proof that both nodes see the same keys.

Site Clusters (active/standby each)

Each site cluster runs Site Runtime, Data Connection Layer, Store-and-Forward, and Site Event Logging. Sites connect to OPC UA for device data and to the central cluster via Akka.NET remoting. Each site node also hosts a gRPC streaming server (port 8083) that central nodes connect to for real-time attribute value and alarm state streams. Deployed configurations and S&F buffers are stored in local SQLite databases per node.

Site Cluster	Site Identifier	Central UI Name
Site-A	site-a	Test Plant A
Site-B	site-b	Test Plant B
Site-C	site-c	Test Plant C

Port Allocation

Application Nodes

Node	Container Name	Host Web Port	Host Akka Port	Host gRPC Port	Internal Ports
Traefik LB	scadabridge-traefik	9000	—	—	80 (proxy), 8080 (dashboard)
Central A	scadabridge-central-a	9001	9011	—	5000 (web), 8081 (Akka)
Central B	scadabridge-central-b	9002	9012	—	5000 (web), 8081 (Akka)
Site-A A	scadabridge-site-a-a	—	9021	9023	8082 (Akka), 8083 (gRPC)
Site-A B	scadabridge-site-a-b	—	9022	9024	8082 (Akka), 8083 (gRPC)
Site-B A	scadabridge-site-b-a	—	9031	9033	8082 (Akka), 8083 (gRPC)
Site-B B	scadabridge-site-b-b	—	9032	9034	8082 (Akka), 8083 (gRPC)
Site-C A	scadabridge-site-c-a	—	9041	9043	8082 (Akka), 8083 (gRPC)
Site-C B	scadabridge-site-c-b	—	9042	9044	8082 (Akka), 8083 (gRPC)

Port block pattern: 90X1/90X2 (Akka), 90X3/90X4 (gRPC) where X = 0 (central), 2 (site-a), 3 (site-b), 4 (site-c). gRPC streaming ports are used by central nodes to subscribe to real-time site data streams.

Infrastructure Services (from infra/docker-compose.yml)

Service	Container Name	Host Port	Purpose
MS SQL 2022	scadabridge-mssql	1433	Configuration and machine data databases
SMTP (Mailpit)	scadabridge-smtp	1025 / 8025	Email capture (SMTP / web UI)
OPC UA	scadabridge-opcua	50000 / 8080	Simulated OPC UA server (protocol / web UI)
REST API	scadabridge-restapi	5200	External REST API for integration testing

infra/docker-compose.yml no longer runs an LDAP container (its ldap service is commented out, kept only for rollback). The central nodes bind to the local redundant GLAuth pair instead — zb-ldap-primary on host.docker.internal:3893 with zb-ldap-backup on host.docker.internal:3894 as FallbackServers (source of truth: scadaproj/infra/glauth-redundant; see each central node's appsettings.Central.json). This replaces the previously-used shared dev GLAuth at 10.100.0.35:3893 — its serviceaccount password was rotated away (SEC-36) and is no longer valid for this rig.

All containers communicate over the shared scadabridge-net Docker bridge network using container names as hostnames.

Directory Structure

docker/
├── Dockerfile                          # Multi-stage build (shared by all nodes)
├── docker-compose.yml                  # 8-node application stack
├── build.sh                            # Build Docker image
├── deploy.sh                           # Build + deploy all containers
├── seed-sites.sh                       # Create test sites with Akka + gRPC addresses
├── seed-live-probe.sh                  # Seed one deployable live-data instance on site-a (stream traffic)
├── seed-delmia-sim.sh                  # Seed the in-memory Delmia machine stand-in (DelmiaSim01) on site-a
├── seed-delmia-api.sh                  # Seed the DelmiaRecipeDownload inbound method + a scoped API key (token printed once)
├── seed-flow-outbound-demo.sh          # Seed the Flow Debugger outbound-calls demo: RigRest external system, RigDb connection, FlowDemo list, OutboundDemo script + FlowOutboundDemo method/key
├── seed-flow-log-demo.sh               # Seed the Flow Debugger STEP-LOGS demo: FlowLogDemo Call script on SimMachine (logs info/debug/warn, writes an attribute, then throws) — idempotent
├── verify-delmia-notifier.sh           # DelmiaNotifier end-to-end vs the rig: standby relay, active-central crash/takeover, rejoin; step H = Flow Debugger live run + replay (CLI flow-debug); step I = outbound API/DB/notify nodes live + replay; step J = step logs + exception detail live + replay
├── teardown.sh                         # Stop and remove containers
├── central-node-a/
│   ├── appsettings.Central.json        # Central node A configuration
│   └── logs/                           # Serilog file output (gitignored)
├── central-node-b/
│   ├── appsettings.Central.json
│   └── logs/
├── site-a-node-a/
│   ├── appsettings.Site.json           # Site-A node A configuration
│   └── logs/                           # Serilog file output (gitignored)
├── site-a-node-b/
│   ├── appsettings.Site.json
│   └── logs/
├── site-b-node-a/
│   ├── appsettings.Site.json           # Site-B node A configuration
│   └── logs/
├── site-b-node-b/
│   ├── appsettings.Site.json
│   └── logs/
├── site-c-node-a/
│   ├── appsettings.Site.json           # Site-C node A configuration
│   └── logs/
└── site-c-node-b/
    ├── appsettings.Site.json
    └── logs/

SQLite databases live in named docker volumes (scadabridge-<node>-data, one per node — register row 79), not in the node directories; inspect them from the host (read-only) at ~/OrbStack/docker/volumes/<name>/. Any docker/<node>/data/ directory still on disk is a dormant pre-migration (2026-08-18) backup, no longer mounted.

gRPC control-plane keys (dev)

The site gRPC service (SiteStreamService on 8083 — live subscriptions, audit pull, cached-telemetry ingest) is gated by a preshared key, and the gate is fail-closed: a site node with no key refuses every call, and StartupValidator refuses to boot it at all. So the rig carries dev keys, one per site:

Where	Setting	Value
site-{a,b,c}-node-*/appsettings.Site.json	ScadaBridge:Communication:GrpcPsk	dev-grpc-psk-docker-site-{a,b,c}
docker-compose.yml, both central nodes	ScadaBridge__Communication__SitePsks__site-{a,b,c}	same value

Both nodes of a pair carry the same key; each site's key is different from the others'. The central half lives in compose env rather than the mounted appsettings.Central.json, which by convention holds no plaintext credentials. Production uses ${secret:SB-GRPC-PSK-<siteId>} on the site and the matching secret in central's store — see docs/deployment/topology-guide.md.

These are not real secrets and are committed deliberately, exactly like the LocalDb sync key (dev-site-a-localdb-sync-key) beside them. The two are separate keys on purpose: the LocalDb one authenticates the pair partner for database replication, not central.

If you add a site to the rig, add its key in both places or its streams will fail with PermissionDenied.

Commands

Initial Setup

Start infrastructure services first, then build and deploy the application:

# 1. Start test infrastructure (MS SQL, SMTP, OPC UA — LDAP is the local
#    redundant GLAuth pair on the Docker host, not started here; see
#    "Infrastructure Services" above)
cd infra && docker compose up -d && cd ..

# 2. Build and deploy all 8 ScadaBridge nodes
docker/deploy.sh

# 3. Seed test sites (first-time only, after cluster is healthy)
docker/seed-sites.sh

# 4. Seed one deployable live-data instance on site-a, so the rig has real
#    stream traffic for live probes (see "Live-data probe instance" below)
docker/seed-live-probe.sh

Live-data probe instance

seed-live-probe.sh creates the minimum working set the rig needs for any live probe of the data or streaming path: a RigProbe template with two attributes bound to self-changing opc-plc nodes (ns=3;s=SpikeData, ns=3;s=RandomSignedInt32, both ~1 Hz) plus a 5 s heartbeat script, and a deployed rig-probe-1 instance on site-a.

Two traps it exists to avoid:

• The custom tags in infra/opcua/nodes.json are declared but not simulated. Motor.*, Tank.*, Valve.*, TestChildObject.* read 0 / false / epoch forever. Binding to them gives a Connected data connection with Good quality and zero event traffic, which looks healthy and probes nothing. Use the built-in simulated nodes (SpikeData, RandomSignedInt32, StepUp, FastUInt1..5, SlowUInt1..5).
• scadabridge_site_stream_subscribers_active is not data-connection health. It counts open central→site gRPC subscriber streams, so it sits at 0 on a perfectly healthy site with nobody watching. (Register row 54 renamed it from scadabridge_site_connection_up, whose name invited exactly that misreading — it is what made row 48's "0 on all six site nodes" look like a fault when it was healthy steady state.)

For real data-connection health, the same row added two honest gauges:

Series	Meaning
scadabridge_site_data_connections_connected	Connections currently in the Connected state
scadabridge_site_data_connections_total	Connections known to this node, in any state

connected < total is the alert condition — neither number alone separates "a device is down" from "nothing is configured". Both read 0 on the standby node, which is healthy: connections are opened by the deployment path on the active node only, so alert on the active node or on max() across the pair. The health report's dataConnectionEndpoints / dataConnectionTagQuality (via scadabridge health site) remain the per-connection detail.

Verify (metrics are on container port 8084 — all eight nodes as of register row 56, which gave the two central nodes the same dedicated scrape listener the six site nodes already had; the containers ship no curl/wget):

scadabridge debug snapshot --id <instance-id>
scadabridge health site --identifier site-a
scadabridge debug stream --id <instance-id>   # open a stream, then in another shell:

docker exec scadabridge-site-a-a bash -c \
  'exec 3<>/dev/tcp/localhost/8084 && printf "GET /metrics HTTP/1.0\r\nHost: localhost\r\n\r\n" >&3 && cat <&3' \
  | grep -E 'stream_subscribers_active|stream_batch_size|data_connections'

# Central-only series (KPI/outbox counters, bundle-import session gauge + rejection
# counter, alarm-cache gauges) — same recipe, central container:
docker exec scadabridge-central-a bash -c \
  'exec 3<>/dev/tcp/localhost/8084 && printf "GET /metrics HTTP/1.0\r\nHost: localhost\r\n\r\n" >&3 && cat <&3' \
  | grep -E 'import_sessions|alarm_cache'

Central binds 8084 only because docker/central-node-*/appsettings.Central.json now sets ScadaBridge:Node:MetricsPort. Unlike a site node, central reads that key only when present — a central deployment without it is unchanged and still serves /metrics on its web port, since the endpoint rides endpoint routing and is served on every bound listener.

After Code Changes

Rebuild and redeploy. The Docker build cache skips NuGet restore when only source files change:

docker/deploy.sh

Stop Application Nodes

Stops and removes all 8 application containers. Site SQLite databases and log files are preserved in node directories:

docker/teardown.sh

Stop Everything

docker/teardown.sh
cd infra && docker compose down && cd ..

View Logs

# All nodes (follow mode)
docker compose -f docker/docker-compose.yml logs -f

# Single node
docker logs -f scadabridge-central-a

# Filter by site cluster
docker compose -f docker/docker-compose.yml logs -f site-a-a site-a-b
docker compose -f docker/docker-compose.yml logs -f site-b-a site-b-b
docker compose -f docker/docker-compose.yml logs -f site-c-a site-c-b

# Persisted log files
ls docker/central-node-a/logs/

Restart a Single Node

docker restart scadabridge-central-a

Check Cluster Health

# Central node A health check
curl -s http://localhost:9001/health/ready | python3 -m json.tool

# Central node B health check
curl -s http://localhost:9002/health/ready | python3 -m json.tool

CLI Access

The CLI connects to the Central Host's HTTP management API via the Traefik load balancer at http://localhost:9000, which routes to the active central node:

dotnet run --project src/ZB.MOM.WW.ScadaBridge.CLI -- \
    --url http://localhost:9000 \
    --username multi-role --password password \
    template list

Direct access to individual nodes is also available at http://localhost:9001 (central-a) and http://localhost:9002 (central-b).

Note: The multi-role test user has Admin, Design, and Deployment roles. The admin user only has the Admin role and cannot perform design or deployment operations. See infra/glauth/config.toml for all test users and their group memberships.

A recommended ~/.scadabridge/config.json for the Docker test environment:

{
  "managementUrl": "http://localhost:9000"
}

With this config file in place, the URL is automatic:

dotnet run --project src/ZB.MOM.WW.ScadaBridge.CLI -- \
    --username multi-role --password password \
    template list

Clear Site Data

Remove SQLite databases to reset site state (deployed configs, S&F buffers). Data lives in named docker volumes (scadabridge-<node>-data, register row 79 — no longer bind-mounted under docker/<node>/data), so empty the volume with a throwaway container while the pair is stopped:

# Single site
docker compose -f docker/docker-compose.yml stop site-a-a site-a-b
docker run --rm \
    -v scadabridge-site-a-node-a-data:/a -v scadabridge-site-a-node-b-data:/b \
    alpine sh -c 'rm -rf /a/* /b/*'
docker compose -f docker/docker-compose.yml start site-a-a site-a-b

# All sites: repeat with the six scadabridge-site-*-data volumes, or teardown + volume rm
# (see teardown.sh's hint) and redeploy.

To inspect a node's live databases from the host, read them via OrbStack's volume exposure (read-only inspection — don't write through this path): ~/OrbStack/docker/volumes/scadabridge-site-a-node-a-data/site-localdb.db.

Rebuild Image From Scratch (no cache)

docker build --no-cache -t scadabridge:latest -f docker/Dockerfile .

Build Cache

The Dockerfile uses a multi-stage build optimized for fast rebuilds:

1. Restore stage: Copies only .csproj files and runs dotnet restore. This layer is cached as long as no project file changes.
2. Build stage: Copies source code and runs dotnet publish --no-restore. Re-runs on any source change but skips restore.
3. Runtime stage: Uses the slim aspnet:10.0 base image with only the published output.

Typical rebuild after a source-only change takes ~5 seconds (restore cached, only build + publish runs).

Test Users

All test passwords are password. See infra/glauth/config.toml for the full list.

Username	Roles	Use Case
admin	Admin	System administration
designer	Design	Template authoring
deployer	Deployment	Instance deployment (all sites)
multi-role	Admin, Design, Deployment	Full access for testing

Failover Testing

Automated failover drill (failover-drill.sh)

DRILL_MODE=standby bash docker/failover-drill.sh   # default — younger-node crash, active untouched
DRILL_MODE=active  bash docker/failover-drill.sh    # oldest-node crash — survivor must TAKE OVER

The scripted drill (docker kill = SIGKILL, the hard-crash path — a docker stop would take the graceful CoordinatedShutdown path and would not prove crash recovery) has two modes, and since the auto-down decision (2026-07-21) both expect recovery — the cluster runs Akka's AutoDowning provider (auto-down-unreachable-after = 15s), under which the leader among the reachable members downs the unreachable peer, so a crash of either node fails over:

• DRILL_MODE=standby (default) — kills the STANDBY (younger) central node. The active node is untouched: expected result is no routing outage at all (/health/active blips = 0) and member removal on the survivor within ~25s (10s failure-detection threshold + 15s auto-down window; the 2s heartbeat interval is not additive). PASS = the survivor logs the downing/removal within TIMEOUT_S (default 90s) while routing stays up.
• DRILL_MODE=active — kills the ACTIVE (oldest) central node. The survivor must take over while the victim is still down: it auto-downs the dead oldest, becomes the oldest member itself, re-hosts all singletons, and its /health/active goes 200. PASS = survivor active within TIMEOUT_S, then Traefik routing to it. (Under the pre-2026-07-21 keep-oldest strategy this direction was a proven total outage — the younger survivor took DownReachable and downed itself, because Akka's down-if-alone only rescues a side with ≥ 2 members.)

Both modes finish by restarting the victim and confirming it rejoins as a ready standby. The drill exercises downing-on-hard-crash, S3 (single active node routed through Traefik), and the Task 20 restart/rejoin contract. Requires a running cluster (bash docker/deploy.sh) and curl + docker on the host.

Partition trade (accepted). Auto-down is availability-first: in a real network partition (both nodes alive, link cut) each side downs the other and both run active — dual-active until an operator restarts one side after the partition heals. This was an explicit owner decision (2026-07-21): site pairs have no shared lease infrastructure to arbitrate, and a stalled system is a bigger risk than a rare partition. See docs/plans/2026-07-21-auto-down-availability-decision.md.

Seed-node ordering — every node lists ITSELF first (decision 2026-07-22). Akka runs FirstSeedNodeProcess — the only bootstrap path that can form a new cluster when no peer answers InitJoin — exclusively when seed-nodes[0] is the node's own address; every other node runs JoinSeedNodeProcess, which retries InitJoin forever and can never form a cluster. Each shipped node config therefore lists itself first and its partner second (docker/central-node-b/appsettings.Central.json leads with scadabridge-central-b), and StartupValidator fails the boot if that ordering is ever broken. This closes the former registered outage gap, where a lone cold-starting central-b (with central-a down) never came Up and recovery was operator-driven.

Self-first ordering is safe, and the three interesting cases are covered by SelfFirstSeedBootstrapTests (real in-process clusters at production failure-detection timings):

Scenario	Behavior
Lone cold-start, peer dead	Forms alone in ~5s (seed-node-timeout) — operational, unattended
Restart into a live peer	InitJoinAck answers, node rejoins; never islands
Both cold-start simultaneously (mutually reachable)	The InitJoin handshake resolves it before either self-joins → one 2-member cluster

An earlier revision of this README claimed the repo deliberately avoided self-first ordering because simultaneous cold start would produce "two one-node clusters that never merge". That is not what happens while the nodes are mutually reachable — the handshake converges them (measured, row 3 above). Only a genuine boot-time partition splits them, which is the same class auto-down already accepts.

Rejected alternative — an external self-form timer. A watchdog that waits N seconds for membership and then calls Cluster.Join(SelfAddress) was implemented and discarded: it cannot see Akka's join handshake, so it cannot distinguish "no seed answered" from "a seed answered and the join is in flight". On a routine standby restart the peer is alive but the join stalls behind removal of the restarting node's own stale incarnation; a Join(self) issued during TryingToJoin abandons the in-flight join and forms a second cluster at the same address — a permanent split (measured: still split after 90s). Akka's own first-seed process has no such race because it is part of the handshake.

Observed results (auto-down decision verification):

Run 2026-07-21 against a freshly-deployed cluster with SplitBrainResolverStrategy: auto-down (first drill: active=central-a). Both directions recovered.

Direction (DRILL_MODE)	Outcome	Measured
active (oldest-node crash)	PASS — TAKEOVER — central-b auto-downed the dead oldest, went Younger -> Oldest on all 7 singletons, and served /health/active while the victim was still down; restarted victim rejoined as standby.	Survivor active + Traefik routing in 28s (budget ~25s: 10s detection + 15s auto-down + hand-over); victim ready 2s after restart.
standby (younger-node crash)	PASS — active node untouched; survivor downed+removed the crashed member; restarted victim rejoined as standby.	Member removed in 27s; 0 /health/active routing blips; victim ready 2s after restart.

Historical baseline (keep-oldest, run 2026-07-13 on 99544985): standby PASS with member removal in 27s / 0 routing blips; active was a total outage — central-b self-downed ~20s after the kill (live SBR log 2026-07-21: SBR took decision Akka.Cluster.SBR.DownReachable … including myself) and could not re-bootstrap until central-a returned. That result is what motivated the auto-down decision. In-process envelope (FailoverTimingTests) measured full failover at 33.7s.

Central Failover

# Stop the active central node
docker stop scadabridge-central-a

# Verify central-b takes over (check logs for leader election)
docker logs -f scadabridge-central-b

# Access UI on standby node
open http://localhost:9002

# Restore the original node
docker start scadabridge-central-a

Manual failover from the UI (admin-only). Instead of stopping a container, an Administrator can trigger a planned role swap from the Trigger failover button on the central-cluster card at /monitoring/health (via Traefik, http://localhost:9000). The active (oldest Up) node leaves the cluster gracefully, so singletons hand over rather than being killed; the node then restarts under restart: unless-stopped and rejoins as the standby.

• The button is disabled when the pair has no online standby — the same guard is re-enforced server-side, since failing over a lone node is an outage, not a failover.
• Triggering it disconnects the page you clicked it on: Traefik routes the UI to the active node, which is the node being restarted. The page reconnects against the new active node.
• Each invocation writes one Cluster / ManualFailover row to dbo.AuditLog naming the admin and the target address, written before the Leave is issued.

To verify on the rig: press the button, watch central-a restart and central-b's badge flip to Primary, then confirm the audit row landed.

Site Failover

# Stop the active site-a node
docker stop scadabridge-site-a-a

# Verify site-a-b takes over singleton (DeploymentManager)
docker logs -f scadabridge-site-a-b

# Restore
docker start scadabridge-site-a-a

Same pattern applies for site-b (scadabridge-site-b-a/scadabridge-site-b-b) and site-c (scadabridge-site-c-a/scadabridge-site-c-b).

Failover takes approximately 25 seconds (2s heartbeat + 10s detection threshold + 15s stable-after for split-brain resolver).

Manual site failover from the UI (admin-only). Each site card on /monitoring/health carries the same Trigger failover button as the central card. Central and each site are separate Akka clusters, so this is a request relayed over the ClusterClient command/control channel — the site's own communication actor performs the graceful Leave against its site-{SiteId} role and acks the result.

• Unlike central failover, this does not disconnect your page — a site is a different cluster.
• A refusal from the site (no standby, or a command addressed to a different site) reads differently from an unreachable site (Ask timeout); the UI shows the site's own reason. Only the timeout leaves any doubt about whether the failover took effect.
• A site running an older binary has no handler for the command, so it dead-letters and you see "site did not respond".
• Each invocation writes a Cluster / ManualFailover audit row stamped with the site id.