Joseph DohertyScadaBridge Docker Infrastructure
Local Docker deployment of the full ScadaBridge cluster topology: a 2-node central cluster and three 2-node site clusters.
Cluster Topology
┌───────────────────┐
│ Traefik LB :9000 │ ◄── CLI / Browser
│ Dashboard :8180 │
└────────┬──────────┘
│ routes to active node
┌──────────────────────┼──────────────────────────────┐
│ Central Cluster │
│ │
│ ┌─────────────────┐ ┌─────────────────┐ │
│ │ central-node-a │◄──►│ central-node-b │ │
│ │ (leader/oldest) │ │ (standby) │ │
│ │ Web UI :9001 │ │ Web UI :9002 │ │
│ │ Akka :9011 │ │ Akka :9012 │ │
│ └────────┬─────────┘ └─────────────────┘ │
│ │ │
└───────────┼─────────────────────────────────────────┘
│ Akka.NET Remoting (hub-and-spoke)
├──────────────────┬──────────────────┐
▼ ▼ ▼
┌────────────────────┐ ┌────────────────────┐ ┌────────────────────┐
│ Site-A Cluster │ │ Site-B Cluster │ │ Site-C Cluster │
│ (Test Plant A) │ │ (Test Plant B) │ │ (Test Plant C) │
│ │ │ │ │ │
│ node-a ◄──► node-b│ │ node-a ◄──► node-b│ │ node-a ◄──► node-b│
│ Akka :9021 :9022 │ │ Akka :9031 :9032 │ │ Akka :9041 :9042 │
│ gRPC :9023 :9024 │ │ gRPC :9033 :9034 │ │ gRPC :9043 :9044 │
└────────────────────┘ └────────────────────┘ └────────────────────┘
Central Cluster (active/standby)
Runs the web UI (Blazor Server), Template Engine, Deployment Manager, Security, Inbound API, Management Service, and Health Monitoring. Connects to MS SQL for configuration and machine data, LDAP for authentication, and SMTP for notifications.
Site Clusters (active/standby each)
Each site cluster runs Site Runtime, Data Connection Layer, Store-and-Forward, and Site Event Logging. Sites connect to OPC UA for device data and to the central cluster via Akka.NET remoting. Each site node also hosts a gRPC streaming server (port 8083) that central nodes connect to for real-time attribute value and alarm state streams. Deployed configurations and S&F buffers are stored in local SQLite databases per node.
| Site Cluster | Site Identifier | Central UI Name |
|---|---|---|
| Site-A | site-a |
Test Plant A |
| Site-B | site-b |
Test Plant B |
| Site-C | site-c |
Test Plant C |
Port Allocation
Application Nodes
| Node | Container Name | Host Web Port | Host Akka Port | Host gRPC Port | Internal Ports |
|---|---|---|---|---|---|
| Traefik LB | scadabridge-traefik |
9000 | — | — | 80 (proxy), 8080 (dashboard) |
| Central A | scadabridge-central-a |
9001 | 9011 | — | 5000 (web), 8081 (Akka) |
| Central B | scadabridge-central-b |
9002 | 9012 | — | 5000 (web), 8081 (Akka) |
| Site-A A | scadabridge-site-a-a |
— | 9021 | 9023 | 8082 (Akka), 8083 (gRPC) |
| Site-A B | scadabridge-site-a-b |
— | 9022 | 9024 | 8082 (Akka), 8083 (gRPC) |
| Site-B A | scadabridge-site-b-a |
— | 9031 | 9033 | 8082 (Akka), 8083 (gRPC) |
| Site-B B | scadabridge-site-b-b |
— | 9032 | 9034 | 8082 (Akka), 8083 (gRPC) |
| Site-C A | scadabridge-site-c-a |
— | 9041 | 9043 | 8082 (Akka), 8083 (gRPC) |
| Site-C B | scadabridge-site-c-b |
— | 9042 | 9044 | 8082 (Akka), 8083 (gRPC) |
Port block pattern: 90X1/90X2 (Akka), 90X3/90X4 (gRPC) where X = 0 (central), 2 (site-a), 3 (site-b), 4 (site-c). gRPC streaming ports are used by central nodes to subscribe to real-time site data streams.
Infrastructure Services (from infra/docker-compose.yml)
| Service | Container Name | Host Port | Purpose |
|---|---|---|---|
| MS SQL 2022 | scadabridge-mssql |
1433 | Configuration and machine data databases |
| LDAP (GLAuth) | scadabridge-ldap |
3893 | Authentication with test users |
| SMTP (Mailpit) | scadabridge-smtp |
1025 / 8025 | Email capture (SMTP / web UI) |
| OPC UA | scadabridge-opcua |
50000 / 8080 | Simulated OPC UA server (protocol / web UI) |
| REST API | scadabridge-restapi |
5200 | External REST API for integration testing |
All containers communicate over the shared scadabridge-net Docker bridge network using container names as hostnames.
Directory Structure
docker/
├── Dockerfile # Multi-stage build (shared by all nodes)
├── docker-compose.yml # 8-node application stack
├── build.sh # Build Docker image
├── deploy.sh # Build + deploy all containers
├── seed-sites.sh # Create test sites with Akka + gRPC addresses
├── teardown.sh # Stop and remove containers
├── central-node-a/
│ ├── appsettings.Central.json # Central node A configuration
│ └── logs/ # Serilog file output (gitignored)
├── central-node-b/
│ ├── appsettings.Central.json
│ └── logs/
├── site-a-node-a/
│ ├── appsettings.Site.json # Site-A node A configuration
│ ├── data/ # SQLite databases (gitignored)
│ └── logs/
├── site-a-node-b/
│ ├── appsettings.Site.json
│ ├── data/
│ └── logs/
├── site-b-node-a/
│ ├── appsettings.Site.json # Site-B node A configuration
│ ├── data/
│ └── logs/
├── site-b-node-b/
│ ├── appsettings.Site.json
│ ├── data/
│ └── logs/
├── site-c-node-a/
│ ├── appsettings.Site.json # Site-C node A configuration
│ ├── data/
│ └── logs/
└── site-c-node-b/
├── appsettings.Site.json
├── data/
└── logs/
Commands
Initial Setup
Start infrastructure services first, then build and deploy the application:
# 1. Start test infrastructure (MS SQL, LDAP, SMTP, OPC UA)
cd infra && docker compose up -d && cd ..
# 2. Build and deploy all 8 ScadaBridge nodes
docker/deploy.sh
# 3. Seed test sites (first-time only, after cluster is healthy)
docker/seed-sites.sh
After Code Changes
Rebuild and redeploy. The Docker build cache skips NuGet restore when only source files change:
docker/deploy.sh
Stop Application Nodes
Stops and removes all 8 application containers. Site SQLite databases and log files are preserved in node directories:
docker/teardown.sh
Stop Everything
docker/teardown.sh
cd infra && docker compose down && cd ..
View Logs
# All nodes (follow mode)
docker compose -f docker/docker-compose.yml logs -f
# Single node
docker logs -f scadabridge-central-a
# Filter by site cluster
docker compose -f docker/docker-compose.yml logs -f site-a-a site-a-b
docker compose -f docker/docker-compose.yml logs -f site-b-a site-b-b
docker compose -f docker/docker-compose.yml logs -f site-c-a site-c-b
# Persisted log files
ls docker/central-node-a/logs/
Restart a Single Node
docker restart scadabridge-central-a
Check Cluster Health
# Central node A health check
curl -s http://localhost:9001/health/ready | python3 -m json.tool
# Central node B health check
curl -s http://localhost:9002/health/ready | python3 -m json.tool
CLI Access
The CLI connects to the Central Host's HTTP management API via the Traefik load balancer at http://localhost:9000, which routes to the active central node:
dotnet run --project src/ZB.MOM.WW.ScadaBridge.CLI -- \
--url http://localhost:9000 \
--username multi-role --password password \
template list
Direct access to individual nodes is also available at http://localhost:9001 (central-a) and http://localhost:9002 (central-b).
Note: The
multi-roletest user has Admin, Design, and Deployment roles. Theadminuser only has the Admin role and cannot perform design or deployment operations. Seeinfra/glauth/config.tomlfor all test users and their group memberships.
A recommended ~/.scadabridge/config.json for the Docker test environment:
{
"managementUrl": "http://localhost:9000"
}
With this config file in place, the URL is automatic:
dotnet run --project src/ZB.MOM.WW.ScadaBridge.CLI -- \
--username multi-role --password password \
template list
Clear Site Data
Remove SQLite databases to reset site state (deployed configs, S&F buffers):
# Single site
rm -rf docker/site-a-node-a/data docker/site-a-node-b/data
docker restart scadabridge-site-a-a scadabridge-site-a-b
# All sites
rm -rf docker/site-*/data
docker restart scadabridge-site-a-a scadabridge-site-a-b \
scadabridge-site-b-a scadabridge-site-b-b \
scadabridge-site-c-a scadabridge-site-c-b
Rebuild Image From Scratch (no cache)
docker build --no-cache -t scadabridge:latest -f docker/Dockerfile .
Build Cache
The Dockerfile uses a multi-stage build optimized for fast rebuilds:
- Restore stage: Copies only
.csprojfiles and runsdotnet restore. This layer is cached as long as no project file changes. - Build stage: Copies source code and runs
dotnet publish --no-restore. Re-runs on any source change but skips restore. - Runtime stage: Uses the slim
aspnet:10.0base image with only the published output.
Typical rebuild after a source-only change takes ~5 seconds (restore cached, only build + publish runs).
Test Users
All test passwords are password. See infra/glauth/config.toml for the full list.
| Username | Roles | Use Case |
|---|---|---|
admin |
Admin | System administration |
designer |
Design | Template authoring |
deployer |
Deployment | Instance deployment (all sites) |
multi-role |
Admin, Design, Deployment | Full access for testing |
Failover Testing
Automated failover drill (failover-drill.sh)
DRILL_MODE=standby bash docker/failover-drill.sh # default — survivable younger-node crash
DRILL_MODE=active bash docker/failover-drill.sh # oldest-node crash — measures the registered outage gap
The scripted drill (docker kill = SIGKILL, the hard-crash path — a docker stop would take the graceful CoordinatedShutdown path and would not prove crash recovery) has two modes, because under the unified oldest-member semantics the active node IS the oldest, i.e. the one crash two-node keep-oldest cannot survive:
DRILL_MODE=standby(default) — kills the STANDBY (younger) central node. The survivable direction: SBR downs the crashed member and the active node keeps its singletons. Expected result: no routing outage at all (the active node is never touched, so/health/activeblips = 0) and member removal on the survivor within ~25s (10s failure-detection threshold + 15s stable-after; the 2s heartbeat interval is not additive). PASS = the survivor logs the member removal withinTIMEOUT_S(default 90s) while routing stays up.DRILL_MODE=active— kills the ACTIVE (oldest) central node. Expected result: a total central outage until the victim container is restarted — this is the registered deferred keep-oldest decision (master tracker 2026-07-08): keep-oldest downs the partition without the oldest, so the younger survivor downs itself, and it cannot re-form a cluster alone (see the seed-node constraint below). The drill confirms the dark window, then recovery within ~2 min of restarting the victim. The mode exists to make the registered gap observable, not to pretend it is covered.
The drill exercises S1 (SBR downing on hard crash), S3 (single active node routed through Traefik), and the Task 20 restart/rejoin contract. Requires a running cluster (bash docker/deploy.sh) and curl + docker on the host.
Seed-node bootstrap constraint. Only the FIRST seed in Cluster:SeedNodes may self-join to form a new cluster. Both central nodes list scadabridge-central-a first (docker/central-node-a/appsettings.Central.json, docker/central-node-b/appsettings.Central.json), so a lone restarted central-b (with central-a still down) loops on InitJoin forever — it never reaches Up, and /health/active never returns 200. Operator recovery actions: (1) restart the dead first-seed node (central-a) — preferred; or (2) restart the survivor with a self-first seed override (env ScadaBridge__Cluster__SeedNodes__0=akka.tcp://scadabridge@<self-host>:8081, ScadaBridge__Cluster__SeedNodes__1=<peer>). The repo deliberately does NOT ship self-first ordering per node: with both nodes self-first, a simultaneous cold start can let each self-join independently → two one-node clusters that never merge (the cold-start split-brain the identical-seed-order convention exists to prevent). The real remedy is the pending keep-oldest topology/strategy decision (deferred, owner: user).
Observed results (plan R2-01 T3):
PENDING — live drill not yet executed (requires a dedicated docker cluster rebuild; deferred to avoid disrupting the running environment). The two-mode drill script and its per-direction PASS criteria are in place; the empirical timings below are to be filled by running
DRILL_MODE=standbythenDRILL_MODE=activeagainst a freshly-deployed cluster and recording the one-line summaries + the cluster commit SHA.
Direction ( DRILL_MODE)Outcome Measured standby(younger-node crash)pending live run pending live run active(oldest-node crash)pending live run pending live run
Central Failover
# Stop the active central node
docker stop scadabridge-central-a
# Verify central-b takes over (check logs for leader election)
docker logs -f scadabridge-central-b
# Access UI on standby node
open http://localhost:9002
# Restore the original node
docker start scadabridge-central-a
Site Failover
# Stop the active site-a node
docker stop scadabridge-site-a-a
# Verify site-a-b takes over singleton (DeploymentManager)
docker logs -f scadabridge-site-a-b
# Restore
docker start scadabridge-site-a-a
Same pattern applies for site-b (scadabridge-site-b-a/scadabridge-site-b-b) and site-c (scadabridge-site-c-a/scadabridge-site-c-b).
Failover takes approximately 25 seconds (2s heartbeat + 10s detection threshold + 15s stable-after for split-brain resolver).