Files
lmxopcua/docker-dev/docker-compose.yml
T
Joseph Doherty cf6110d02c docs(mesh): ConfigSource/ConfigServe config, rig flip switch, Phase 3 as shipped
Phase 3 Task 9. docker-dev rig: central-1/2 carry ConfigServe (:4055 h2c + committed
dev key) and stay Direct; the four site nodes carry ConfigSource (both central
endpoints + matching key) defaulting to Direct. Flip only the site nodes with
OTOPCUA_CONFIG_MODE=FetchAndCache at 'docker compose up' — central keeps SQL and models
the target topology. Binding :4055 activates the dedicated-h2c Kestrel takeover on
central (re-binds :9000 too). Docs: new docs/Configuration.md ConfigSource/ConfigServe
section; docs/Redundancy.md 'config bytes travel out-of-band' note; the program plan
Phase 3 section + tracking row flipped to CODE-COMPLETE; a CLAUDE.md 'Config source'
section.

Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
2026-07-22 20:06:23 -04:00

669 lines
41 KiB
YAML
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
# docker-dev/ — Mac-friendly single-mesh hub-and-spoke fleet for v2 development + manual UI exercise.
#
# Topology: ONE Akka mesh seeded by `central-1`. Logical separation between
# tenants is by ServerCluster.ClusterId rows (MAIN / SITE-A / SITE-B) in the one
# shared `OtOpcUa` ConfigDb — NOT by separate meshes. All six host nodes join the
# same gossip ring and the central UI deploys to every cluster over it.
#
# Stack:
# sql SQL Server 2022 — hosts the one ConfigDb every node uses
# cluster-seed one-shot mssql-tools job that INSERTs the ServerCluster +
# ClusterNode rows scoping each tenant, then exits (idempotent)
#
# central-1, central-2 OTOPCUA_ROLES=admin,driver — the ONLY UI + deploy
# singleton, plus the MAIN cluster's OPC UA publishers.
# Reachable at http://localhost:9200 (via Traefik).
# Both are Akka seed nodes, each listing ITSELF first
# (2026-07-22) so either can cold-start while the other
# is down; the site nodes seed off central-1 only.
# site-a-1, site-a-2 OTOPCUA_ROLES=driver — driver-only members of the same
# site-b-1, site-b-2 mesh, scoped to SITE-A / SITE-B by ClusterId. They
# serve no UI and authenticate no users; the central
# cluster manages and deploys to them over the mesh.
#
# Auth is real LDAP against the shared GLAuth on the Linux Docker host
# (10.100.0.35:3893, dc=zb,dc=local) — there is no LDAP container here.
# Only the admin-role central nodes carry the Security__Ldap__* block.
# Sign in `multi-role` / `password`.
#
# traefik PathPrefix(`/`) → central-1 / central-2 (the single UI route).
#
# OPC UA endpoints (host-side port → container 4840):
# central-1 :4840 central-2 :4841
# site-a-1 :4842 site-a-2 :4843
# site-b-1 :4844 site-b-2 :4845
#
# Headless deploy: POST http://localhost:9200/api/deployments with the
# X-Api-Key header (Security__DeployApiKey = "docker-dev-deploy-key").
#
# SQL persistence: the otopcua-mssql-data named volume keeps the ConfigDb schema
# + seeded clusters across `docker compose up` cycles; without it a recreate
# silently drops the OtOpcUa database.
#
# Usage:
# docker compose -f docker-dev/docker-compose.yml up -d --build
# open http://localhost:9200 # central Blazor admin UI
# open http://localhost:8089 # Traefik dashboard (8080 is the sister scadalink stack)
#
# Tear-down: docker compose -f docker-dev/docker-compose.yml down -v
name: otopcua-dev
# ── Shared secret-replication env (all six host nodes) ──────────────────────────
# Akka peer-to-peer clustered secret replication rides the existing `otopcua` Akka
# mesh (DistributedPubSub topic `zb-mom-ww-secrets`) across all six host nodes. Two
# hard requirements are wired here, identically, on every node:
# 1. Secrets__Replication__Enabled=true → SecretsRegistration.AddOtOpcUaSecrets
# switches from plain AddZbSecrets to AddZbSecretsAkkaReplication + the eager
# SecretReplicationStarter hosted service, so every node joins anti-entropy.
# 2. ZB_SECRETS_MASTER_KEY → the ONE shared 32-byte base64 KEK. Every node MUST
# carry the SAME key or peers fail closed decrypting each other's rows
# (kek_id mismatch). appsettings.json binds Secrets:MasterKey:Source=Environment
# / EnvVarName=ZB_SECRETS_MASTER_KEY, so this env var IS the KEK on every node.
# The KEK below is a committed DEV-ONLY default (matches the rig's zero-operator-step
# philosophy, like the committed dev SQL password). Override for a fresh key with:
# OTOPCUA_SECRETS_KEK=$(openssl rand -base64 32) docker compose ... up -d
# NEVER reuse this key outside this local dev rig.
# AnnounceInterval is shortened to 5s (default 30s) so anti-entropy convergence is
# observable quickly during dev exercise. Each node keeps its own local SQLite store
# (Secrets:SqlitePath=otopcua-secrets.db, per-container); replication syncs them.
x-secrets-env: &secrets-env
Secrets__Replication__Enabled: "true"
Secrets__Replication__AnnounceInterval: "00:00:05"
ZB_SECRETS_MASTER_KEY: "${OTOPCUA_SECRETS_KEK:-ZYGhIX0luS/XsevpCB2W18jYHMcqO6AjM9oXy+T6Zp4=}"
services:
sql:
image: mcr.microsoft.com/mssql/server:2022-latest
environment:
ACCEPT_EULA: "Y"
SA_PASSWORD: "OtOpcUa!Dev123"
MSSQL_PID: Developer
ports:
- "14330:1433"
# Persist the ConfigDb across container recreates. Without this the dev SQL
# is ephemeral (container writable layer), so a recreate silently drops the
# OtOpcUa database and every host node fails its configdb health check until
# EF auto-migration + cluster-seed rebuild it. The named volume keeps the
# schema + seeded clusters between `docker compose up` cycles.
volumes:
- otopcua-mssql-data:/var/opt/mssql
healthcheck:
test: ["CMD-SHELL", "/opt/mssql-tools18/bin/sqlcmd -S localhost -U sa -P 'OtOpcUa!Dev123' -No -Q 'SELECT 1' || exit 1"]
interval: 10s
timeout: 5s
retries: 20
# ── Migrator (one-shot) ────────────────────────────────────────────────────
# Applies EF Core migrations to the OtOpcUa ConfigDb so a fresh SQL volume gets
# the schema with no operator step (the host nodes deliberately don't auto-
# migrate — production owns schema changes). cluster-seed + every host node
# depend on this completing, so nothing races an in-progress migration.
# Idempotent: a no-op once the schema is current.
migrator:
build:
context: ..
dockerfile: docker-dev/Dockerfile
target: migrator
image: otopcua-migrator:dev
depends_on:
sql:
condition: service_healthy
environment:
OTOPCUA_CONFIG_CONNECTION: "Server=sql,1433;Database=OtOpcUa;User Id=sa;Password=OtOpcUa!Dev123;TrustServerCertificate=True;"
restart: "no"
# ── Cluster seed (one-shot) ────────────────────────────────────────────────
# Runs only after `migrator` completes (so the schema is final — no race), then
# INSERTs the three ServerCluster rows and the six ClusterNode rows that scope
# each tenant inside the shared OtOpcUa ConfigDb. Idempotent — re-runs are no-ops.
cluster-seed:
image: mcr.microsoft.com/mssql-tools:latest
depends_on:
migrator:
condition: service_completed_successfully
volumes:
- ./seed:/seed:ro
entrypoint: ["/bin/bash", "/seed/entrypoint.sh"]
restart: "no"
# A local OpenLDAP container used to live here, but the bitnami/openldap:2.6
# image was retired (manifest gone) and bitnamilegacy/openldap:2.6 crashes
# during LDIF setup (exit 68). Rather than stub auth, the central (admin-role)
# containers bind the shared GLAuth on the Linux Docker host (Security__Ldap__*
# below: 10.100.0.35:3893, dc=zb,dc=local, DevStubMode=false) — so dev auth
# exercises the real LDAP bind + group→role path. Sign in `multi-role` /
# `password` (all roles) or any shared test user / `password`.
# ── Central cluster (2-node fused admin+driver) ─────────────────────────────
# The only UI + deploy singleton; also the MAIN cluster's OPC UA publishers.
# central-1 seeds the single Akka mesh that every other node joins; central-2 is a seed too
# (listing itself first), so it can form the mesh alone if central-1 is down.
central-1: &otopcua-host
build:
context: ..
dockerfile: docker-dev/Dockerfile
target: runtime
image: otopcua-host:dev
# Per-node memory bounds. The full single-mesh stack (6 host nodes) OOM-killed
# central-1 on a loaded host. Each host node measured ~357 MiB idle-solo and
# climbs sharply under deploy/materialise load: a node materialising its full
# cluster slice (e.g. central → MAIN's galaxy mirror + UNS overlay, ~1400 OPC UA
# nodes) peaks well above 1g during a deploy — a 1g cap OOM-kills it (exit 137).
# Cap at 2g (≈peak + headroom) with a 1g reservation. These top-level keys are
# inherited by every service that uses `<<: *otopcua-host` (YAML merge keeps the
# anchor's scalar keys; only the `environment` block is re-declared per service).
# The full 6-node mesh needs ~12g of Docker Desktop VM memory — on a constrained
# host raise the VM memory or run fewer host services.
mem_limit: 2g
mem_reservation: 1g
# Restart supervision. A 2-node keep-oldest SBR downs the LONE survivor when the
# OLDEST node crashes (the survivor is "the side without the oldest" → DownReachable
# → run-coordinated-shutdown-when-down terminates it). Recovery is by exit-and-rejoin:
# the supervisor restarts the exited node and it re-forms/rejoins as a fresh
# incarnation (the ScadaBridge pattern). Without a restart policy the exited node
# stays down and the cluster looks like a total outage. Inherited by every host node
# via the anchor merge.
restart: unless-stopped
depends_on:
sql: { condition: service_healthy }
migrator: { condition: service_completed_successfully }
environment:
<<: *secrets-env
OTOPCUA_ROLES: "admin,driver"
ASPNETCORE_URLS: "http://+:9000"
ConnectionStrings__ConfigDb: "Server=sql,1433;Database=OtOpcUa;User Id=sa;Password=OtOpcUa!Dev123;TrustServerCertificate=True;"
Cluster__Hostname: "0.0.0.0"
Cluster__Port: "4053"
Cluster__PublicHostname: "central-1"
# Per-cluster mesh Phase 3 (config fetch-and-cache). Central SERVES the deployed-configuration
# artifact to FetchAndCache site nodes over a dedicated h2c gRPC listener (:4055), gated by the
# shared node key. Central itself is ALWAYS Direct — it IS the SQL source and must never fetch.
# ConfigServe:ApiKey must be byte-identical to the site nodes' ConfigSource:ApiKey (fail-closed).
# Binding :4055 activates the dedicated-h2c Kestrel takeover on central (it re-binds :9000 too).
ConfigServe__GrpcListenPort: "4055"
ConfigServe__ApiKey: "configserve-docker-dev-key"
ConfigSource__Mode: "Direct"
# Both redundancy peers are seeds (#459 / ScadaBridge parity) so a restarted node can
# re-join the mesh via EITHER peer, not only central-1.
#
# ORDER IS LOAD-BEARING (2026-07-22): each node lists ITSELF first. Akka runs
# FirstSeedNodeProcess — the only path that can form a NEW cluster when no peer answers
# InitJoin — exclusively for seed-nodes[0]; every other node retries InitJoin forever. So a
# node listing its partner first cannot cold-start while that partner is down. Enforced at
# boot by AkkaClusterOptionsValidator; see docs/Redundancy.md.
Cluster__SeedNodes__0: "akka.tcp://otopcua@central-1:4053"
Cluster__SeedNodes__1: "akka.tcp://otopcua@central-2:4053"
Cluster__Roles__0: "admin"
Cluster__Roles__1: "driver"
# Mesh command transport (per-cluster mesh Phase 2). DARK SWITCH: the rig comes up on "Dps",
# the transport this repo has always used, and every node keeps its ClusterClient wiring built
# and its comm actor registered with the receptionist either way. Flip the whole rig with
# OTOPCUA_MESH_MODE=ClusterClient at `docker compose up` — no compose edit, and no rebuild.
#
# Contact points are node ADDRESSES ONLY. "/system/receptionist" is appended at construction
# time, and MeshTransportOptionsValidator refuses to start a node whose contact carries an
# actor-path suffix, because a doubled suffix resolves to nothing and drops every ack in
# silence. Both central nodes are listed so a driver node's client can rotate to whichever one
# is up — that rotation is why the comm actors are per-node and NOT cluster singletons.
MeshTransport__Mode: "${OTOPCUA_MESH_MODE:-Dps}"
MeshTransport__CentralContactPoints__0: "akka.tcp://otopcua@central-1:4053"
MeshTransport__CentralContactPoints__1: "akka.tcp://otopcua@central-2:4053"
Security__Jwt__SigningKey: "docker-dev-signing-key-with-at-least-32-bytes-of-utf8-content-12345"
Security__Jwt__Issuer: "otopcua-dev"
Security__Jwt__Audience: "otopcua-dev"
Security__Ldap__Enabled: "true"
Security__Ldap__DevStubMode: "false"
Security__Ldap__Server: "10.100.0.35"
Security__Ldap__Port: "3893"
Security__Ldap__Transport: "None"
Security__Ldap__AllowInsecure: "true"
Security__Ldap__SearchBase: "dc=zb,dc=local"
Security__Ldap__ServiceAccountDn: "cn=serviceaccount,dc=zb,dc=local"
Security__Ldap__ServiceAccountPassword: "serviceaccount123"
Security__DeployApiKey: "docker-dev-deploy-key"
Security__Auth__DisableLogin: "true"
# Map the shared-GLAuth data-plane groups (ReadOnly/WriteOperate/...) to data-plane role
# strings the OPC UA session ACL gates read. The DB LdapGroupRoleMapping is AdminRole-typed
# (AdminUI roles); this free-form GroupToRole baseline is the path for data-plane roles.
Security__Ldap__GroupToRole__ReadOnly: "ReadOnly"
Security__Ldap__GroupToRole__WriteOperate: "WriteOperate"
Security__Ldap__GroupToRole__WriteTune: "WriteTune"
Security__Ldap__GroupToRole__WriteConfigure: "WriteConfigure"
Security__Ldap__GroupToRole__AlarmAck: "AlarmAck"
# Server-side HistoryRead via the ZB.MOM.WW.HistorianGateway sidecar. DISABLED by default: the
# docker-dev rig has no historian gateway to point at, and the historian-gateway cutover retired the
# old Wonderware Host/Port/SharedSecret keys — the code now reads ServerHistorian__Endpoint and builds
# a Uri from it, so Enabled=true with an empty Endpoint crashes the Host at startup
# (HistorianGatewayClientAdapter.Create -> UriFormatException "The URI is empty").
# To enable: OTOPCUA_HISTORIAN_ENABLED=true + OTOPCUA_HISTORIAN_ENDPOINT=https://<host>:5222, and
# export ServerHistorian__ApiKey=histgw_<id>_<secret> at `docker compose up` (never commit the key).
ServerHistorian__Enabled: "${OTOPCUA_HISTORIAN_ENABLED:-false}"
ServerHistorian__Endpoint: "${OTOPCUA_HISTORIAN_ENDPOINT:-}"
ServerHistorian__ApiKey: "${ServerHistorian__ApiKey:-}"
ServerHistorian__UseTls: "true"
ServerHistorian__AllowUntrustedServerCertificate: "true"
# Pin EF Core + ASP.NET Core to Warning so the per-poll Deployment SELECT /
# "Executed DbCommand" Information|Debug lines stop flooding the Serilog
# pipeline and starving the Akka cluster heartbeat thread. The host logs via
# Serilog (AddZbSerilog → ReadFrom.Configuration); these env vars override
# Serilog:MinimumLevel:Override:* (app/Akka levels are left untouched).
Serilog__MinimumLevel__Override__Microsoft.EntityFrameworkCore: "Warning"
Serilog__MinimumLevel__Override__Microsoft.AspNetCore: "Warning"
GALAXY_MXGW_API_KEY: "${GALAXY_MXGW_API_KEY:-mxgw_otopcua2_GI7-tNozYE6cXGUSgEzL3AHDV7bYcYIHdMwKYgyHdX4}"
# Consolidated pair-local config cache (ZB.MOM.WW.LocalDb). Every driver-role node
# (central is admin,driver) gets one, on a per-node named volume so the cache survives
# container recreates. The central pair leaves replication OFF (default) — only the
# site-a pair replicates, as the enablement demo; site-b is the default-OFF pin.
LocalDb__Path: "/app/data/otopcua-localdb.db"
ports:
- "4840:4840"
volumes:
- otopcua-localdb-central-1:/app/data
central-2:
<<: *otopcua-host
depends_on:
sql: { condition: service_healthy }
central-1: { condition: service_started }
migrator: { condition: service_completed_successfully }
environment:
<<: *secrets-env
OTOPCUA_ROLES: "admin,driver"
ASPNETCORE_URLS: "http://+:9000"
ConnectionStrings__ConfigDb: "Server=sql,1433;Database=OtOpcUa;User Id=sa;Password=OtOpcUa!Dev123;TrustServerCertificate=True;"
Cluster__Hostname: "0.0.0.0"
Cluster__Port: "4053"
Cluster__PublicHostname: "central-2"
# Per-cluster mesh Phase 3 — serves the artifact to FetchAndCache site nodes (see central-1).
# Always Direct; ConfigServe:ApiKey must match the site nodes' ConfigSource:ApiKey.
ConfigServe__GrpcListenPort: "4055"
ConfigServe__ApiKey: "configserve-docker-dev-key"
ConfigSource__Mode: "Direct"
# Both redundancy peers are seeds (#459 / ScadaBridge parity) — see central-1. SELF FIRST:
# central-2 lists itself as seed-nodes[0], which is what lets it cold-start while central-1
# is down (it previously listed central-1 first and simply never came Up in that case).
Cluster__SeedNodes__0: "akka.tcp://otopcua@central-2:4053"
Cluster__SeedNodes__1: "akka.tcp://otopcua@central-1:4053"
Cluster__Roles__0: "admin"
Cluster__Roles__1: "driver"
# Mesh command transport (per-cluster mesh Phase 2). DARK SWITCH: the rig comes up on "Dps",
# the transport this repo has always used, and every node keeps its ClusterClient wiring built
# and its comm actor registered with the receptionist either way. Flip the whole rig with
# OTOPCUA_MESH_MODE=ClusterClient at `docker compose up` — no compose edit, and no rebuild.
#
# Contact points are node ADDRESSES ONLY. "/system/receptionist" is appended at construction
# time, and MeshTransportOptionsValidator refuses to start a node whose contact carries an
# actor-path suffix, because a doubled suffix resolves to nothing and drops every ack in
# silence. Both central nodes are listed so a driver node's client can rotate to whichever one
# is up — that rotation is why the comm actors are per-node and NOT cluster singletons.
MeshTransport__Mode: "${OTOPCUA_MESH_MODE:-Dps}"
MeshTransport__CentralContactPoints__0: "akka.tcp://otopcua@central-1:4053"
MeshTransport__CentralContactPoints__1: "akka.tcp://otopcua@central-2:4053"
Security__Jwt__SigningKey: "docker-dev-signing-key-with-at-least-32-bytes-of-utf8-content-12345"
Security__Jwt__Issuer: "otopcua-dev"
Security__Jwt__Audience: "otopcua-dev"
Security__Ldap__Enabled: "true"
Security__Ldap__DevStubMode: "false"
Security__Ldap__Server: "10.100.0.35"
Security__Ldap__Port: "3893"
Security__Ldap__Transport: "None"
Security__Ldap__AllowInsecure: "true"
Security__Ldap__SearchBase: "dc=zb,dc=local"
Security__Ldap__ServiceAccountDn: "cn=serviceaccount,dc=zb,dc=local"
Security__Ldap__ServiceAccountPassword: "serviceaccount123"
Security__DeployApiKey: "docker-dev-deploy-key"
Security__Auth__DisableLogin: "true"
# Data-plane group→role baseline (see central-1).
Security__Ldap__GroupToRole__ReadOnly: "ReadOnly"
Security__Ldap__GroupToRole__WriteOperate: "WriteOperate"
Security__Ldap__GroupToRole__WriteTune: "WriteTune"
Security__Ldap__GroupToRole__WriteConfigure: "WriteConfigure"
Security__Ldap__GroupToRole__AlarmAck: "AlarmAck"
# Server-side HistoryRead via the ZB.MOM.WW.HistorianGateway sidecar. DISABLED by default: the
# docker-dev rig has no historian gateway to point at, and the historian-gateway cutover retired the
# old Wonderware Host/Port/SharedSecret keys — the code now reads ServerHistorian__Endpoint and builds
# a Uri from it, so Enabled=true with an empty Endpoint crashes the Host at startup
# (HistorianGatewayClientAdapter.Create -> UriFormatException "The URI is empty").
# To enable: OTOPCUA_HISTORIAN_ENABLED=true + OTOPCUA_HISTORIAN_ENDPOINT=https://<host>:5222, and
# export ServerHistorian__ApiKey=histgw_<id>_<secret> at `docker compose up` (never commit the key).
ServerHistorian__Enabled: "${OTOPCUA_HISTORIAN_ENABLED:-false}"
ServerHistorian__Endpoint: "${OTOPCUA_HISTORIAN_ENDPOINT:-}"
ServerHistorian__ApiKey: "${ServerHistorian__ApiKey:-}"
ServerHistorian__UseTls: "true"
ServerHistorian__AllowUntrustedServerCertificate: "true"
# Quiet EF/AspNetCore SQL flood — see central-1 (Serilog override). mem_limit/
# mem_reservation are inherited from the *otopcua-host anchor.
Serilog__MinimumLevel__Override__Microsoft.EntityFrameworkCore: "Warning"
Serilog__MinimumLevel__Override__Microsoft.AspNetCore: "Warning"
GALAXY_MXGW_API_KEY: "${GALAXY_MXGW_API_KEY:-mxgw_otopcua2_GI7-tNozYE6cXGUSgEzL3AHDV7bYcYIHdMwKYgyHdX4}"
# Pair-local config cache; central pair leaves replication OFF (see central-1).
LocalDb__Path: "/app/data/otopcua-localdb.db"
ports:
- "4841:4840"
volumes:
- otopcua-localdb-central-2:/app/data
# ── Site A cluster (2-node driver-only) ─────────────────────────────────────
# Driver-only members of the single mesh, scoped to SITE-A by ClusterId. No UI,
# no user auth — managed + deployed to by the central cluster over the mesh.
# All site nodes seed central-1.
site-a-1:
<<: *otopcua-host
depends_on:
sql: { condition: service_healthy }
central-1: { condition: service_started }
migrator: { condition: service_completed_successfully }
environment:
OTOPCUA_ROLES: "driver"
ConnectionStrings__ConfigDb: "Server=sql,1433;Database=OtOpcUa;User Id=sa;Password=OtOpcUa!Dev123;TrustServerCertificate=True;"
Cluster__Hostname: "0.0.0.0"
Cluster__Port: "4053"
Cluster__PublicHostname: "site-a-1"
# Per-cluster mesh Phase 3 dark switch. Default Direct (reads central SQL). Flip the site nodes to
# FetchAndCache with `OTOPCUA_CONFIG_MODE=FetchAndCache docker compose up`: they then fetch the
# deployed artifact from central over gRPC (endpoints below) and read ONLY the LocalDb cache — no
# central SQL config read. Central stays Direct, so this env var moves ONLY the four site nodes.
# ConfigSource:ApiKey must equal central's ConfigServe:ApiKey (the interceptor is fail-closed).
ConfigSource__Mode: "${OTOPCUA_CONFIG_MODE:-Direct}"
ConfigSource__CentralFetchEndpoints__0: "http://central-1:4055"
ConfigSource__CentralFetchEndpoints__1: "http://central-2:4055"
ConfigSource__ApiKey: "configserve-docker-dev-key"
# Site nodes are deliberately NOT seeds — they join the central pair's mesh. The self-first
# seed rule is conditional for exactly this reason (it binds only when a node's own address
# is in its own seed list), so these configs are exempt rather than broken.
Cluster__SeedNodes__0: "akka.tcp://otopcua@central-1:4053"
Cluster__Roles__0: "driver"
# Mesh command transport (per-cluster mesh Phase 2). DARK SWITCH: the rig comes up on "Dps",
# the transport this repo has always used, and every node keeps its ClusterClient wiring built
# and its comm actor registered with the receptionist either way. Flip the whole rig with
# OTOPCUA_MESH_MODE=ClusterClient at `docker compose up` — no compose edit, and no rebuild.
#
# Contact points are node ADDRESSES ONLY. "/system/receptionist" is appended at construction
# time, and MeshTransportOptionsValidator refuses to start a node whose contact carries an
# actor-path suffix, because a doubled suffix resolves to nothing and drops every ack in
# silence. Both central nodes are listed so a driver node's client can rotate to whichever one
# is up — that rotation is why the comm actors are per-node and NOT cluster singletons.
MeshTransport__Mode: "${OTOPCUA_MESH_MODE:-Dps}"
MeshTransport__CentralContactPoints__0: "akka.tcp://otopcua@central-1:4053"
MeshTransport__CentralContactPoints__1: "akka.tcp://otopcua@central-2:4053"
<<: *secrets-env
# Quiet EF/AspNetCore SQL flood — see central-1 (Serilog override). mem_limit/
# mem_reservation are inherited from the *otopcua-host anchor.
Serilog__MinimumLevel__Override__Microsoft.EntityFrameworkCore: "Warning"
Serilog__MinimumLevel__Override__Microsoft.AspNetCore: "Warning"
# Resolved at runtime by GalaxyDriver.ResolveApiKey when a DriverInstance's
# Gateway.ApiKeySecretRef = "env:GALAXY_MXGW_API_KEY".
GALAXY_MXGW_API_KEY: "${GALAXY_MXGW_API_KEY:-mxgw_otopcua2_GI7-tNozYE6cXGUSgEzL3AHDV7bYcYIHdMwKYgyHdX4}"
# Pair-local config cache + REPLICATION ON — the site-a pair is the enablement demo.
# site-a-1 is the initiator: it binds the h2c sync listener on 9001 AND dials the peer.
# Setting SyncListenPort makes Program.cs add a dedicated Http2-only Kestrel listener and
# re-bind the primary HTTP port (an explicit Listen* otherwise discards ASPNETCORE_URLS).
# Alarm store-and-forward buffer, Phase 2. Enabled on BOTH halves of the pair so the
# buffer is a live, replicated table rather than an empty one — the live gate needs a real
# queue to watch converge, and needs to see that only the Primary drains it.
#
# There is no HistorianGateway on this rig, so the endpoint below is deliberately
# unresolvable: every drain attempt fails, which is exactly the historian-outage state the
# buffer exists for. Note the endpoint must still be a syntactically valid absolute http(s)
# URI or the host refuses to start — ServerHistorianOptionsValidator is consumer-gated, so
# AlarmHistorian:Enabled=true makes it required even while ServerHistorian:Enabled=false.
#
# MaxAttempts is raised far above the production default of 10. With a permanently
# unreachable gateway the default would dead-letter the whole queue about five minutes into
# the outage, turning a buffering test into a dead-letter test.
AlarmHistorian__Enabled: "true"
AlarmHistorian__MaxAttempts: "1000000"
ServerHistorian__Endpoint: "${OTOPCUA_HISTORIAN_ENDPOINT:-http://histgw-absent.invalid:5222}"
# Required whenever the section is consumed: the gateway client validates its own options at
# construction and throws on an empty key, so a keyless node crash-loops during Akka startup
# rather than degrading. DEV-ONLY placeholder — it authenticates nothing, because the endpoint
# above resolves nowhere. NEVER reuse this shape for a real gateway key; supply those via
# ServerHistorian__ApiKey from the environment.
ServerHistorian__ApiKey: "${OTOPCUA_HISTORIAN_APIKEY:-histgw_dev_rig-no-gateway-present}"
# Must agree with the endpoint scheme above — the client throws on a mismatch in EITHER
# direction, so an http endpoint with the default UseTls=true crash-loops the host.
ServerHistorian__UseTls: "false"
LocalDb__Path: "/app/data/otopcua-localdb.db"
LocalDb__SyncListenPort: "9001"
LocalDb__Replication__PeerAddress: "http://site-a-2:9001"
# DEV-ONLY committed key (rig philosophy — like the SQL password + secrets KEK above). It
# MUST be byte-identical on both nodes: the interceptor is fail-closed, so any mismatch
# silently stops the pair converging. NEVER reuse this key outside this local dev rig.
LocalDb__Replication__ApiKey: "dev-site-a-localdb-sync-key"
# Row-count batching against gRPC's 4 MB message cap; artifact chunk rows are ≈171 KB, so
# 16 × 171 KB ≈ 2.7 MB stays under the cap with headroom.
LocalDb__Replication__MaxBatchSize: "16"
ports:
- "4842:4840"
volumes:
- otopcua-localdb-site-a-1:/app/data
site-a-2:
<<: *otopcua-host
depends_on:
sql: { condition: service_healthy }
central-1: { condition: service_started }
migrator: { condition: service_completed_successfully }
environment:
OTOPCUA_ROLES: "driver"
ConnectionStrings__ConfigDb: "Server=sql,1433;Database=OtOpcUa;User Id=sa;Password=OtOpcUa!Dev123;TrustServerCertificate=True;"
Cluster__Hostname: "0.0.0.0"
Cluster__Port: "4053"
Cluster__PublicHostname: "site-a-2"
# Per-cluster mesh Phase 3 dark switch (see site-a-1). Default Direct; OTOPCUA_CONFIG_MODE=FetchAndCache
# flips it to fetch the artifact from central over gRPC and read only the LocalDb cache.
ConfigSource__Mode: "${OTOPCUA_CONFIG_MODE:-Direct}"
ConfigSource__CentralFetchEndpoints__0: "http://central-1:4055"
ConfigSource__CentralFetchEndpoints__1: "http://central-2:4055"
ConfigSource__ApiKey: "configserve-docker-dev-key"
Cluster__SeedNodes__0: "akka.tcp://otopcua@central-1:4053"
Cluster__Roles__0: "driver"
# Mesh command transport (per-cluster mesh Phase 2). DARK SWITCH: the rig comes up on "Dps",
# the transport this repo has always used, and every node keeps its ClusterClient wiring built
# and its comm actor registered with the receptionist either way. Flip the whole rig with
# OTOPCUA_MESH_MODE=ClusterClient at `docker compose up` — no compose edit, and no rebuild.
#
# Contact points are node ADDRESSES ONLY. "/system/receptionist" is appended at construction
# time, and MeshTransportOptionsValidator refuses to start a node whose contact carries an
# actor-path suffix, because a doubled suffix resolves to nothing and drops every ack in
# silence. Both central nodes are listed so a driver node's client can rotate to whichever one
# is up — that rotation is why the comm actors are per-node and NOT cluster singletons.
MeshTransport__Mode: "${OTOPCUA_MESH_MODE:-Dps}"
MeshTransport__CentralContactPoints__0: "akka.tcp://otopcua@central-1:4053"
MeshTransport__CentralContactPoints__1: "akka.tcp://otopcua@central-2:4053"
<<: *secrets-env
Serilog__MinimumLevel__Override__Microsoft.EntityFrameworkCore: "Warning"
Serilog__MinimumLevel__Override__Microsoft.AspNetCore: "Warning"
GALAXY_MXGW_API_KEY: "${GALAXY_MXGW_API_KEY:-mxgw_otopcua2_GI7-tNozYE6cXGUSgEzL3AHDV7bYcYIHdMwKYgyHdX4}"
# site-a-2 is the passive half of the replicating pair: it binds the sync listener on 9001
# but does NOT dial (no PeerAddress). The replication stream is bidirectional, so a-1's
# single dial carries both directions. Same key + MaxBatchSize as a-1 (byte-identical key
# is mandatory — the interceptor fail-closes on a mismatch).
# Alarm store-and-forward buffer, Phase 2. Enabled on BOTH halves of the pair so the
# buffer is a live, replicated table rather than an empty one — the live gate needs a real
# queue to watch converge, and needs to see that only the Primary drains it.
#
# There is no HistorianGateway on this rig, so the endpoint below is deliberately
# unresolvable: every drain attempt fails, which is exactly the historian-outage state the
# buffer exists for. Note the endpoint must still be a syntactically valid absolute http(s)
# URI or the host refuses to start — ServerHistorianOptionsValidator is consumer-gated, so
# AlarmHistorian:Enabled=true makes it required even while ServerHistorian:Enabled=false.
#
# MaxAttempts is raised far above the production default of 10. With a permanently
# unreachable gateway the default would dead-letter the whole queue about five minutes into
# the outage, turning a buffering test into a dead-letter test.
AlarmHistorian__Enabled: "true"
AlarmHistorian__MaxAttempts: "1000000"
ServerHistorian__Endpoint: "${OTOPCUA_HISTORIAN_ENDPOINT:-http://histgw-absent.invalid:5222}"
# Required whenever the section is consumed: the gateway client validates its own options at
# construction and throws on an empty key, so a keyless node crash-loops during Akka startup
# rather than degrading. DEV-ONLY placeholder — it authenticates nothing, because the endpoint
# above resolves nowhere. NEVER reuse this shape for a real gateway key; supply those via
# ServerHistorian__ApiKey from the environment.
ServerHistorian__ApiKey: "${OTOPCUA_HISTORIAN_APIKEY:-histgw_dev_rig-no-gateway-present}"
# Must agree with the endpoint scheme above — the client throws on a mismatch in EITHER
# direction, so an http endpoint with the default UseTls=true crash-loops the host.
ServerHistorian__UseTls: "false"
LocalDb__Path: "/app/data/otopcua-localdb.db"
LocalDb__SyncListenPort: "9001"
LocalDb__Replication__ApiKey: "dev-site-a-localdb-sync-key"
LocalDb__Replication__MaxBatchSize: "16"
ports:
- "4843:4840"
volumes:
- otopcua-localdb-site-a-2:/app/data
# ── Site B cluster (2-node driver-only) ─────────────────────────────────────
site-b-1:
<<: *otopcua-host
depends_on:
sql: { condition: service_healthy }
central-1: { condition: service_started }
migrator: { condition: service_completed_successfully }
environment:
OTOPCUA_ROLES: "driver"
ConnectionStrings__ConfigDb: "Server=sql,1433;Database=OtOpcUa;User Id=sa;Password=OtOpcUa!Dev123;TrustServerCertificate=True;"
Cluster__Hostname: "0.0.0.0"
Cluster__Port: "4053"
Cluster__PublicHostname: "site-b-1"
# Per-cluster mesh Phase 3 dark switch (see site-a-1). Default Direct; OTOPCUA_CONFIG_MODE=FetchAndCache
# flips it to fetch the artifact from central over gRPC and read only the LocalDb cache.
ConfigSource__Mode: "${OTOPCUA_CONFIG_MODE:-Direct}"
ConfigSource__CentralFetchEndpoints__0: "http://central-1:4055"
ConfigSource__CentralFetchEndpoints__1: "http://central-2:4055"
ConfigSource__ApiKey: "configserve-docker-dev-key"
Cluster__SeedNodes__0: "akka.tcp://otopcua@central-1:4053"
Cluster__Roles__0: "driver"
# Mesh command transport (per-cluster mesh Phase 2). DARK SWITCH: the rig comes up on "Dps",
# the transport this repo has always used, and every node keeps its ClusterClient wiring built
# and its comm actor registered with the receptionist either way. Flip the whole rig with
# OTOPCUA_MESH_MODE=ClusterClient at `docker compose up` — no compose edit, and no rebuild.
#
# Contact points are node ADDRESSES ONLY. "/system/receptionist" is appended at construction
# time, and MeshTransportOptionsValidator refuses to start a node whose contact carries an
# actor-path suffix, because a doubled suffix resolves to nothing and drops every ack in
# silence. Both central nodes are listed so a driver node's client can rotate to whichever one
# is up — that rotation is why the comm actors are per-node and NOT cluster singletons.
MeshTransport__Mode: "${OTOPCUA_MESH_MODE:-Dps}"
MeshTransport__CentralContactPoints__0: "akka.tcp://otopcua@central-1:4053"
MeshTransport__CentralContactPoints__1: "akka.tcp://otopcua@central-2:4053"
<<: *secrets-env
Serilog__MinimumLevel__Override__Microsoft.EntityFrameworkCore: "Warning"
Serilog__MinimumLevel__Override__Microsoft.AspNetCore: "Warning"
GALAXY_MXGW_API_KEY: "${GALAXY_MXGW_API_KEY:-mxgw_otopcua2_GI7-tNozYE6cXGUSgEzL3AHDV7bYcYIHdMwKYgyHdX4}"
# site-b is the default-OFF pin: it gets the pair-local cache but NO SyncListenPort and NO
# replication config, so no sync listener binds and ISyncStatus stays disconnected/Healthy.
# This is what the live gate's check 8 verifies — the cache works locally with replication off.
# Alarm store-and-forward with replication OFF — the other half of the site-b pin. The sink
# must work as a plain node-local buffer here, with no sync listener and no peer traffic.
AlarmHistorian__Enabled: "true"
AlarmHistorian__MaxAttempts: "1000000"
ServerHistorian__Endpoint: "${OTOPCUA_HISTORIAN_ENDPOINT:-http://histgw-absent.invalid:5222}"
# Required whenever the section is consumed: the gateway client validates its own options at
# construction and throws on an empty key, so a keyless node crash-loops during Akka startup
# rather than degrading. DEV-ONLY placeholder — it authenticates nothing, because the endpoint
# above resolves nowhere. NEVER reuse this shape for a real gateway key; supply those via
# ServerHistorian__ApiKey from the environment.
ServerHistorian__ApiKey: "${OTOPCUA_HISTORIAN_APIKEY:-histgw_dev_rig-no-gateway-present}"
# Must agree with the endpoint scheme above — the client throws on a mismatch in EITHER
# direction, so an http endpoint with the default UseTls=true crash-loops the host.
ServerHistorian__UseTls: "false"
LocalDb__Path: "/app/data/otopcua-localdb.db"
ports:
- "4844:4840"
volumes:
- otopcua-localdb-site-b-1:/app/data
site-b-2:
<<: *otopcua-host
depends_on:
sql: { condition: service_healthy }
central-1: { condition: service_started }
migrator: { condition: service_completed_successfully }
environment:
OTOPCUA_ROLES: "driver"
ConnectionStrings__ConfigDb: "Server=sql,1433;Database=OtOpcUa;User Id=sa;Password=OtOpcUa!Dev123;TrustServerCertificate=True;"
Cluster__Hostname: "0.0.0.0"
Cluster__Port: "4053"
Cluster__PublicHostname: "site-b-2"
# Per-cluster mesh Phase 3 dark switch (see site-a-1). Default Direct; OTOPCUA_CONFIG_MODE=FetchAndCache
# flips it to fetch the artifact from central over gRPC and read only the LocalDb cache.
ConfigSource__Mode: "${OTOPCUA_CONFIG_MODE:-Direct}"
ConfigSource__CentralFetchEndpoints__0: "http://central-1:4055"
ConfigSource__CentralFetchEndpoints__1: "http://central-2:4055"
ConfigSource__ApiKey: "configserve-docker-dev-key"
Cluster__SeedNodes__0: "akka.tcp://otopcua@central-1:4053"
Cluster__Roles__0: "driver"
# Mesh command transport (per-cluster mesh Phase 2). DARK SWITCH: the rig comes up on "Dps",
# the transport this repo has always used, and every node keeps its ClusterClient wiring built
# and its comm actor registered with the receptionist either way. Flip the whole rig with
# OTOPCUA_MESH_MODE=ClusterClient at `docker compose up` — no compose edit, and no rebuild.
#
# Contact points are node ADDRESSES ONLY. "/system/receptionist" is appended at construction
# time, and MeshTransportOptionsValidator refuses to start a node whose contact carries an
# actor-path suffix, because a doubled suffix resolves to nothing and drops every ack in
# silence. Both central nodes are listed so a driver node's client can rotate to whichever one
# is up — that rotation is why the comm actors are per-node and NOT cluster singletons.
MeshTransport__Mode: "${OTOPCUA_MESH_MODE:-Dps}"
MeshTransport__CentralContactPoints__0: "akka.tcp://otopcua@central-1:4053"
MeshTransport__CentralContactPoints__1: "akka.tcp://otopcua@central-2:4053"
<<: *secrets-env
Serilog__MinimumLevel__Override__Microsoft.EntityFrameworkCore: "Warning"
Serilog__MinimumLevel__Override__Microsoft.AspNetCore: "Warning"
GALAXY_MXGW_API_KEY: "${GALAXY_MXGW_API_KEY:-mxgw_otopcua2_GI7-tNozYE6cXGUSgEzL3AHDV7bYcYIHdMwKYgyHdX4}"
# site-b default-OFF pin (see site-b-1): cache on, replication off.
# Alarm store-and-forward with replication OFF — the other half of the site-b pin. The sink
# must work as a plain node-local buffer here, with no sync listener and no peer traffic.
AlarmHistorian__Enabled: "true"
AlarmHistorian__MaxAttempts: "1000000"
ServerHistorian__Endpoint: "${OTOPCUA_HISTORIAN_ENDPOINT:-http://histgw-absent.invalid:5222}"
# Required whenever the section is consumed: the gateway client validates its own options at
# construction and throws on an empty key, so a keyless node crash-loops during Akka startup
# rather than degrading. DEV-ONLY placeholder — it authenticates nothing, because the endpoint
# above resolves nowhere. NEVER reuse this shape for a real gateway key; supply those via
# ServerHistorian__ApiKey from the environment.
ServerHistorian__ApiKey: "${OTOPCUA_HISTORIAN_APIKEY:-histgw_dev_rig-no-gateway-present}"
# Must agree with the endpoint scheme above — the client throws on a mismatch in EITHER
# direction, so an http endpoint with the default UseTls=true crash-loops the host.
ServerHistorian__UseTls: "false"
LocalDb__Path: "/app/data/otopcua-localdb.db"
ports:
- "4845:4840"
volumes:
- otopcua-localdb-site-b-2:/app/data
traefik:
image: traefik:v3.1
command:
- --entrypoints.web.address=:80
- --providers.file.filename=/etc/traefik/dynamic.yml
- --providers.file.watch=true
- --api.insecure=true
ports:
- "9200:80" # host port 9200 → traefik :80 entrypoint (80 conflicts with scadabridge-traefik)
- "8089:8080" # 8080 conflicts with the sister scadalink dev stack
volumes:
- ./traefik-dynamic.yml:/etc/traefik/dynamic.yml:ro
depends_on:
- central-1
- central-2
volumes:
# SQL Server data dir — persists the OtOpcUa ConfigDb across container recreates.
otopcua-mssql-data:
# Per-node pair-local config-cache (ZB.MOM.WW.LocalDb) files. One volume per node — the
# cache is node-local, not shared; it persists the cached deployment artifact (and, on the
# site-a pair, the replicated oplog state) across container recreates so a node can boot
# from cache after a restart even with central SQL down.
otopcua-localdb-central-1:
otopcua-localdb-central-2:
otopcua-localdb-site-a-1:
otopcua-localdb-site-a-2:
otopcua-localdb-site-b-1:
otopcua-localdb-site-b-2: