Joseph Doherty
3f24d4d6bf
Akka runs FirstSeedNodeProcess -- the only bootstrap path that can form a NEW
cluster when no peer answers InitJoin -- exclusively when seed-nodes[0] is the
node's own address. Every other node runs JoinSeedNodeProcess and retries
InitJoin forever. That is why "both peers are seeds" never meant "either can
cold-start alone", and it is fixed here the way Akka itself intends: each seed
node lists ITSELF first.
- docker-dev central-2 now lists itself as SeedNodes__0 (central-1 was already
self-first). The site nodes are untouched -- they seed off central-1 only and
are deliberately not seeds.
- AkkaClusterOptionsValidator enforces the invariant at boot via the shared
ZB.MOM.WW.Configuration AddValidatedOptions/ValidateOnStart seam. The rule is
CONDITIONAL -- self must be seed[0] only IF self is in the list at all -- or it
would refuse to boot every driver-only site node. Identity is compared on
PublicHostname (falling back to Hostname when blank) AND port, i.e. the address
Akka puts in SelfAddress: matching the 0.0.0.0 bind address would find no seed
anywhere and leave the rule silently inert on the whole docker-dev rig.
- ClusterBootstrapFallback + Cluster:SelfFormAfter are DELETED. The watchdog sat
outside Akka's join handshake, so it could not distinguish "no seed answered"
from "a seed answered and the join is in flight" -- and Cluster.Join(SelfAddress)
is not ignored mid-handshake, it wins. The live gate (ea45ace1) caught it
islanding a node that a manual failover had bounced: InitJoinAck received, no
Welcome inside the window because the peer's ring still held the old
incarnation, watchdog fired, second cluster. The TCP reachability guard patched
that one shape; the race stayed. It was also inert for every non-seed node, so
after this change it can never usefully fire.
- SelfFormBootstrapTests -> SelfFirstSeedBootstrapTests: real in-process clusters
through the production bootstrap at production failure-detection timings, incl.
a falsifiability control proving the OLD peer-first ordering never forms (that
test failed at 11s against the watchdog, which is what proved the retirement),
the restart-into-a-live-peer case that killed the watchdog, and a simultaneous
cold start converging on ONE cluster.
Mirrors ScadaBridge 4a6341d8; anticipates per-cluster-mesh-program Phase 6, which
had this retirement queued.
551 lines
32 KiB
YAML
551 lines
32 KiB
YAML
# docker-dev/ — Mac-friendly single-mesh hub-and-spoke fleet for v2 development + manual UI exercise.
|
||
#
|
||
# Topology: ONE Akka mesh seeded by `central-1`. Logical separation between
|
||
# tenants is by ServerCluster.ClusterId rows (MAIN / SITE-A / SITE-B) in the one
|
||
# shared `OtOpcUa` ConfigDb — NOT by separate meshes. All six host nodes join the
|
||
# same gossip ring and the central UI deploys to every cluster over it.
|
||
#
|
||
# Stack:
|
||
# sql SQL Server 2022 — hosts the one ConfigDb every node uses
|
||
# cluster-seed one-shot mssql-tools job that INSERTs the ServerCluster +
|
||
# ClusterNode rows scoping each tenant, then exits (idempotent)
|
||
#
|
||
# central-1, central-2 OTOPCUA_ROLES=admin,driver — the ONLY UI + deploy
|
||
# singleton, plus the MAIN cluster's OPC UA publishers.
|
||
# Reachable at http://localhost:9200 (via Traefik).
|
||
# Both are Akka seed nodes, each listing ITSELF first
|
||
# (2026-07-22) so either can cold-start while the other
|
||
# is down; the site nodes seed off central-1 only.
|
||
# site-a-1, site-a-2 OTOPCUA_ROLES=driver — driver-only members of the same
|
||
# site-b-1, site-b-2 mesh, scoped to SITE-A / SITE-B by ClusterId. They
|
||
# serve no UI and authenticate no users; the central
|
||
# cluster manages and deploys to them over the mesh.
|
||
#
|
||
# Auth is real LDAP against the shared GLAuth on the Linux Docker host
|
||
# (10.100.0.35:3893, dc=zb,dc=local) — there is no LDAP container here.
|
||
# Only the admin-role central nodes carry the Security__Ldap__* block.
|
||
# Sign in `multi-role` / `password`.
|
||
#
|
||
# traefik PathPrefix(`/`) → central-1 / central-2 (the single UI route).
|
||
#
|
||
# OPC UA endpoints (host-side port → container 4840):
|
||
# central-1 :4840 central-2 :4841
|
||
# site-a-1 :4842 site-a-2 :4843
|
||
# site-b-1 :4844 site-b-2 :4845
|
||
#
|
||
# Headless deploy: POST http://localhost:9200/api/deployments with the
|
||
# X-Api-Key header (Security__DeployApiKey = "docker-dev-deploy-key").
|
||
#
|
||
# SQL persistence: the otopcua-mssql-data named volume keeps the ConfigDb schema
|
||
# + seeded clusters across `docker compose up` cycles; without it a recreate
|
||
# silently drops the OtOpcUa database.
|
||
#
|
||
# Usage:
|
||
# docker compose -f docker-dev/docker-compose.yml up -d --build
|
||
# open http://localhost:9200 # central Blazor admin UI
|
||
# open http://localhost:8089 # Traefik dashboard (8080 is the sister scadalink stack)
|
||
#
|
||
# Tear-down: docker compose -f docker-dev/docker-compose.yml down -v
|
||
|
||
name: otopcua-dev
|
||
|
||
# ── Shared secret-replication env (all six host nodes) ──────────────────────────
|
||
# Akka peer-to-peer clustered secret replication rides the existing `otopcua` Akka
|
||
# mesh (DistributedPubSub topic `zb-mom-ww-secrets`) across all six host nodes. Two
|
||
# hard requirements are wired here, identically, on every node:
|
||
# 1. Secrets__Replication__Enabled=true → SecretsRegistration.AddOtOpcUaSecrets
|
||
# switches from plain AddZbSecrets to AddZbSecretsAkkaReplication + the eager
|
||
# SecretReplicationStarter hosted service, so every node joins anti-entropy.
|
||
# 2. ZB_SECRETS_MASTER_KEY → the ONE shared 32-byte base64 KEK. Every node MUST
|
||
# carry the SAME key or peers fail closed decrypting each other's rows
|
||
# (kek_id mismatch). appsettings.json binds Secrets:MasterKey:Source=Environment
|
||
# / EnvVarName=ZB_SECRETS_MASTER_KEY, so this env var IS the KEK on every node.
|
||
# The KEK below is a committed DEV-ONLY default (matches the rig's zero-operator-step
|
||
# philosophy, like the committed dev SQL password). Override for a fresh key with:
|
||
# OTOPCUA_SECRETS_KEK=$(openssl rand -base64 32) docker compose ... up -d
|
||
# NEVER reuse this key outside this local dev rig.
|
||
# AnnounceInterval is shortened to 5s (default 30s) so anti-entropy convergence is
|
||
# observable quickly during dev exercise. Each node keeps its own local SQLite store
|
||
# (Secrets:SqlitePath=otopcua-secrets.db, per-container); replication syncs them.
|
||
x-secrets-env: &secrets-env
|
||
Secrets__Replication__Enabled: "true"
|
||
Secrets__Replication__AnnounceInterval: "00:00:05"
|
||
ZB_SECRETS_MASTER_KEY: "${OTOPCUA_SECRETS_KEK:-ZYGhIX0luS/XsevpCB2W18jYHMcqO6AjM9oXy+T6Zp4=}"
|
||
|
||
services:
|
||
|
||
sql:
|
||
image: mcr.microsoft.com/mssql/server:2022-latest
|
||
environment:
|
||
ACCEPT_EULA: "Y"
|
||
SA_PASSWORD: "OtOpcUa!Dev123"
|
||
MSSQL_PID: Developer
|
||
ports:
|
||
- "14330:1433"
|
||
# Persist the ConfigDb across container recreates. Without this the dev SQL
|
||
# is ephemeral (container writable layer), so a recreate silently drops the
|
||
# OtOpcUa database and every host node fails its configdb health check until
|
||
# EF auto-migration + cluster-seed rebuild it. The named volume keeps the
|
||
# schema + seeded clusters between `docker compose up` cycles.
|
||
volumes:
|
||
- otopcua-mssql-data:/var/opt/mssql
|
||
healthcheck:
|
||
test: ["CMD-SHELL", "/opt/mssql-tools18/bin/sqlcmd -S localhost -U sa -P 'OtOpcUa!Dev123' -No -Q 'SELECT 1' || exit 1"]
|
||
interval: 10s
|
||
timeout: 5s
|
||
retries: 20
|
||
|
||
# ── Migrator (one-shot) ────────────────────────────────────────────────────
|
||
# Applies EF Core migrations to the OtOpcUa ConfigDb so a fresh SQL volume gets
|
||
# the schema with no operator step (the host nodes deliberately don't auto-
|
||
# migrate — production owns schema changes). cluster-seed + every host node
|
||
# depend on this completing, so nothing races an in-progress migration.
|
||
# Idempotent: a no-op once the schema is current.
|
||
migrator:
|
||
build:
|
||
context: ..
|
||
dockerfile: docker-dev/Dockerfile
|
||
target: migrator
|
||
image: otopcua-migrator:dev
|
||
depends_on:
|
||
sql:
|
||
condition: service_healthy
|
||
environment:
|
||
OTOPCUA_CONFIG_CONNECTION: "Server=sql,1433;Database=OtOpcUa;User Id=sa;Password=OtOpcUa!Dev123;TrustServerCertificate=True;"
|
||
restart: "no"
|
||
|
||
# ── Cluster seed (one-shot) ────────────────────────────────────────────────
|
||
# Runs only after `migrator` completes (so the schema is final — no race), then
|
||
# INSERTs the three ServerCluster rows and the six ClusterNode rows that scope
|
||
# each tenant inside the shared OtOpcUa ConfigDb. Idempotent — re-runs are no-ops.
|
||
cluster-seed:
|
||
image: mcr.microsoft.com/mssql-tools:latest
|
||
depends_on:
|
||
migrator:
|
||
condition: service_completed_successfully
|
||
volumes:
|
||
- ./seed:/seed:ro
|
||
entrypoint: ["/bin/bash", "/seed/entrypoint.sh"]
|
||
restart: "no"
|
||
|
||
# A local OpenLDAP container used to live here, but the bitnami/openldap:2.6
|
||
# image was retired (manifest gone) and bitnamilegacy/openldap:2.6 crashes
|
||
# during LDIF setup (exit 68). Rather than stub auth, the central (admin-role)
|
||
# containers bind the shared GLAuth on the Linux Docker host (Security__Ldap__*
|
||
# below: 10.100.0.35:3893, dc=zb,dc=local, DevStubMode=false) — so dev auth
|
||
# exercises the real LDAP bind + group→role path. Sign in `multi-role` /
|
||
# `password` (all roles) or any shared test user / `password`.
|
||
|
||
# ── Central cluster (2-node fused admin+driver) ─────────────────────────────
|
||
# The only UI + deploy singleton; also the MAIN cluster's OPC UA publishers.
|
||
# central-1 seeds the single Akka mesh that every other node joins; central-2 is a seed too
|
||
# (listing itself first), so it can form the mesh alone if central-1 is down.
|
||
|
||
central-1: &otopcua-host
|
||
build:
|
||
context: ..
|
||
dockerfile: docker-dev/Dockerfile
|
||
target: runtime
|
||
image: otopcua-host:dev
|
||
# Per-node memory bounds. The full single-mesh stack (6 host nodes) OOM-killed
|
||
# central-1 on a loaded host. Each host node measured ~357 MiB idle-solo and
|
||
# climbs sharply under deploy/materialise load: a node materialising its full
|
||
# cluster slice (e.g. central → MAIN's galaxy mirror + UNS overlay, ~1400 OPC UA
|
||
# nodes) peaks well above 1g during a deploy — a 1g cap OOM-kills it (exit 137).
|
||
# Cap at 2g (≈peak + headroom) with a 1g reservation. These top-level keys are
|
||
# inherited by every service that uses `<<: *otopcua-host` (YAML merge keeps the
|
||
# anchor's scalar keys; only the `environment` block is re-declared per service).
|
||
# The full 6-node mesh needs ~12g of Docker Desktop VM memory — on a constrained
|
||
# host raise the VM memory or run fewer host services.
|
||
mem_limit: 2g
|
||
mem_reservation: 1g
|
||
# Restart supervision. A 2-node keep-oldest SBR downs the LONE survivor when the
|
||
# OLDEST node crashes (the survivor is "the side without the oldest" → DownReachable
|
||
# → run-coordinated-shutdown-when-down terminates it). Recovery is by exit-and-rejoin:
|
||
# the supervisor restarts the exited node and it re-forms/rejoins as a fresh
|
||
# incarnation (the ScadaBridge pattern). Without a restart policy the exited node
|
||
# stays down and the cluster looks like a total outage. Inherited by every host node
|
||
# via the anchor merge.
|
||
restart: unless-stopped
|
||
depends_on:
|
||
sql: { condition: service_healthy }
|
||
migrator: { condition: service_completed_successfully }
|
||
environment:
|
||
<<: *secrets-env
|
||
OTOPCUA_ROLES: "admin,driver"
|
||
ASPNETCORE_URLS: "http://+:9000"
|
||
ConnectionStrings__ConfigDb: "Server=sql,1433;Database=OtOpcUa;User Id=sa;Password=OtOpcUa!Dev123;TrustServerCertificate=True;"
|
||
Cluster__Hostname: "0.0.0.0"
|
||
Cluster__Port: "4053"
|
||
Cluster__PublicHostname: "central-1"
|
||
# Both redundancy peers are seeds (#459 / ScadaBridge parity) so a restarted node can
|
||
# re-join the mesh via EITHER peer, not only central-1.
|
||
#
|
||
# ORDER IS LOAD-BEARING (2026-07-22): each node lists ITSELF first. Akka runs
|
||
# FirstSeedNodeProcess — the only path that can form a NEW cluster when no peer answers
|
||
# InitJoin — exclusively for seed-nodes[0]; every other node retries InitJoin forever. So a
|
||
# node listing its partner first cannot cold-start while that partner is down. Enforced at
|
||
# boot by AkkaClusterOptionsValidator; see docs/Redundancy.md.
|
||
Cluster__SeedNodes__0: "akka.tcp://otopcua@central-1:4053"
|
||
Cluster__SeedNodes__1: "akka.tcp://otopcua@central-2:4053"
|
||
Cluster__Roles__0: "admin"
|
||
Cluster__Roles__1: "driver"
|
||
Security__Jwt__SigningKey: "docker-dev-signing-key-with-at-least-32-bytes-of-utf8-content-12345"
|
||
Security__Jwt__Issuer: "otopcua-dev"
|
||
Security__Jwt__Audience: "otopcua-dev"
|
||
Security__Ldap__Enabled: "true"
|
||
Security__Ldap__DevStubMode: "false"
|
||
Security__Ldap__Server: "10.100.0.35"
|
||
Security__Ldap__Port: "3893"
|
||
Security__Ldap__Transport: "None"
|
||
Security__Ldap__AllowInsecure: "true"
|
||
Security__Ldap__SearchBase: "dc=zb,dc=local"
|
||
Security__Ldap__ServiceAccountDn: "cn=serviceaccount,dc=zb,dc=local"
|
||
Security__Ldap__ServiceAccountPassword: "serviceaccount123"
|
||
Security__DeployApiKey: "docker-dev-deploy-key"
|
||
Security__Auth__DisableLogin: "true"
|
||
# Map the shared-GLAuth data-plane groups (ReadOnly/WriteOperate/...) to data-plane role
|
||
# strings the OPC UA session ACL gates read. The DB LdapGroupRoleMapping is AdminRole-typed
|
||
# (AdminUI roles); this free-form GroupToRole baseline is the path for data-plane roles.
|
||
Security__Ldap__GroupToRole__ReadOnly: "ReadOnly"
|
||
Security__Ldap__GroupToRole__WriteOperate: "WriteOperate"
|
||
Security__Ldap__GroupToRole__WriteTune: "WriteTune"
|
||
Security__Ldap__GroupToRole__WriteConfigure: "WriteConfigure"
|
||
Security__Ldap__GroupToRole__AlarmAck: "AlarmAck"
|
||
# Server-side HistoryRead via the ZB.MOM.WW.HistorianGateway sidecar. DISABLED by default: the
|
||
# docker-dev rig has no historian gateway to point at, and the historian-gateway cutover retired the
|
||
# old Wonderware Host/Port/SharedSecret keys — the code now reads ServerHistorian__Endpoint and builds
|
||
# a Uri from it, so Enabled=true with an empty Endpoint crashes the Host at startup
|
||
# (HistorianGatewayClientAdapter.Create -> UriFormatException "The URI is empty").
|
||
# To enable: OTOPCUA_HISTORIAN_ENABLED=true + OTOPCUA_HISTORIAN_ENDPOINT=https://<host>:5222, and
|
||
# export ServerHistorian__ApiKey=histgw_<id>_<secret> at `docker compose up` (never commit the key).
|
||
ServerHistorian__Enabled: "${OTOPCUA_HISTORIAN_ENABLED:-false}"
|
||
ServerHistorian__Endpoint: "${OTOPCUA_HISTORIAN_ENDPOINT:-}"
|
||
ServerHistorian__ApiKey: "${ServerHistorian__ApiKey:-}"
|
||
ServerHistorian__UseTls: "true"
|
||
ServerHistorian__AllowUntrustedServerCertificate: "true"
|
||
# Pin EF Core + ASP.NET Core to Warning so the per-poll Deployment SELECT /
|
||
# "Executed DbCommand" Information|Debug lines stop flooding the Serilog
|
||
# pipeline and starving the Akka cluster heartbeat thread. The host logs via
|
||
# Serilog (AddZbSerilog → ReadFrom.Configuration); these env vars override
|
||
# Serilog:MinimumLevel:Override:* (app/Akka levels are left untouched).
|
||
Serilog__MinimumLevel__Override__Microsoft.EntityFrameworkCore: "Warning"
|
||
Serilog__MinimumLevel__Override__Microsoft.AspNetCore: "Warning"
|
||
GALAXY_MXGW_API_KEY: "${GALAXY_MXGW_API_KEY:-mxgw_otopcua2_GI7-tNozYE6cXGUSgEzL3AHDV7bYcYIHdMwKYgyHdX4}"
|
||
# Consolidated pair-local config cache (ZB.MOM.WW.LocalDb). Every driver-role node
|
||
# (central is admin,driver) gets one, on a per-node named volume so the cache survives
|
||
# container recreates. The central pair leaves replication OFF (default) — only the
|
||
# site-a pair replicates, as the enablement demo; site-b is the default-OFF pin.
|
||
LocalDb__Path: "/app/data/otopcua-localdb.db"
|
||
ports:
|
||
- "4840:4840"
|
||
volumes:
|
||
- otopcua-localdb-central-1:/app/data
|
||
|
||
central-2:
|
||
<<: *otopcua-host
|
||
depends_on:
|
||
sql: { condition: service_healthy }
|
||
central-1: { condition: service_started }
|
||
migrator: { condition: service_completed_successfully }
|
||
environment:
|
||
<<: *secrets-env
|
||
OTOPCUA_ROLES: "admin,driver"
|
||
ASPNETCORE_URLS: "http://+:9000"
|
||
ConnectionStrings__ConfigDb: "Server=sql,1433;Database=OtOpcUa;User Id=sa;Password=OtOpcUa!Dev123;TrustServerCertificate=True;"
|
||
Cluster__Hostname: "0.0.0.0"
|
||
Cluster__Port: "4053"
|
||
Cluster__PublicHostname: "central-2"
|
||
# Both redundancy peers are seeds (#459 / ScadaBridge parity) — see central-1. SELF FIRST:
|
||
# central-2 lists itself as seed-nodes[0], which is what lets it cold-start while central-1
|
||
# is down (it previously listed central-1 first and simply never came Up in that case).
|
||
Cluster__SeedNodes__0: "akka.tcp://otopcua@central-2:4053"
|
||
Cluster__SeedNodes__1: "akka.tcp://otopcua@central-1:4053"
|
||
Cluster__Roles__0: "admin"
|
||
Cluster__Roles__1: "driver"
|
||
Security__Jwt__SigningKey: "docker-dev-signing-key-with-at-least-32-bytes-of-utf8-content-12345"
|
||
Security__Jwt__Issuer: "otopcua-dev"
|
||
Security__Jwt__Audience: "otopcua-dev"
|
||
Security__Ldap__Enabled: "true"
|
||
Security__Ldap__DevStubMode: "false"
|
||
Security__Ldap__Server: "10.100.0.35"
|
||
Security__Ldap__Port: "3893"
|
||
Security__Ldap__Transport: "None"
|
||
Security__Ldap__AllowInsecure: "true"
|
||
Security__Ldap__SearchBase: "dc=zb,dc=local"
|
||
Security__Ldap__ServiceAccountDn: "cn=serviceaccount,dc=zb,dc=local"
|
||
Security__Ldap__ServiceAccountPassword: "serviceaccount123"
|
||
Security__DeployApiKey: "docker-dev-deploy-key"
|
||
Security__Auth__DisableLogin: "true"
|
||
# Data-plane group→role baseline (see central-1).
|
||
Security__Ldap__GroupToRole__ReadOnly: "ReadOnly"
|
||
Security__Ldap__GroupToRole__WriteOperate: "WriteOperate"
|
||
Security__Ldap__GroupToRole__WriteTune: "WriteTune"
|
||
Security__Ldap__GroupToRole__WriteConfigure: "WriteConfigure"
|
||
Security__Ldap__GroupToRole__AlarmAck: "AlarmAck"
|
||
# Server-side HistoryRead via the ZB.MOM.WW.HistorianGateway sidecar. DISABLED by default: the
|
||
# docker-dev rig has no historian gateway to point at, and the historian-gateway cutover retired the
|
||
# old Wonderware Host/Port/SharedSecret keys — the code now reads ServerHistorian__Endpoint and builds
|
||
# a Uri from it, so Enabled=true with an empty Endpoint crashes the Host at startup
|
||
# (HistorianGatewayClientAdapter.Create -> UriFormatException "The URI is empty").
|
||
# To enable: OTOPCUA_HISTORIAN_ENABLED=true + OTOPCUA_HISTORIAN_ENDPOINT=https://<host>:5222, and
|
||
# export ServerHistorian__ApiKey=histgw_<id>_<secret> at `docker compose up` (never commit the key).
|
||
ServerHistorian__Enabled: "${OTOPCUA_HISTORIAN_ENABLED:-false}"
|
||
ServerHistorian__Endpoint: "${OTOPCUA_HISTORIAN_ENDPOINT:-}"
|
||
ServerHistorian__ApiKey: "${ServerHistorian__ApiKey:-}"
|
||
ServerHistorian__UseTls: "true"
|
||
ServerHistorian__AllowUntrustedServerCertificate: "true"
|
||
# Quiet EF/AspNetCore SQL flood — see central-1 (Serilog override). mem_limit/
|
||
# mem_reservation are inherited from the *otopcua-host anchor.
|
||
Serilog__MinimumLevel__Override__Microsoft.EntityFrameworkCore: "Warning"
|
||
Serilog__MinimumLevel__Override__Microsoft.AspNetCore: "Warning"
|
||
GALAXY_MXGW_API_KEY: "${GALAXY_MXGW_API_KEY:-mxgw_otopcua2_GI7-tNozYE6cXGUSgEzL3AHDV7bYcYIHdMwKYgyHdX4}"
|
||
# Pair-local config cache; central pair leaves replication OFF (see central-1).
|
||
LocalDb__Path: "/app/data/otopcua-localdb.db"
|
||
ports:
|
||
- "4841:4840"
|
||
volumes:
|
||
- otopcua-localdb-central-2:/app/data
|
||
|
||
# ── Site A cluster (2-node driver-only) ─────────────────────────────────────
|
||
# Driver-only members of the single mesh, scoped to SITE-A by ClusterId. No UI,
|
||
# no user auth — managed + deployed to by the central cluster over the mesh.
|
||
# All site nodes seed central-1.
|
||
|
||
site-a-1:
|
||
<<: *otopcua-host
|
||
depends_on:
|
||
sql: { condition: service_healthy }
|
||
central-1: { condition: service_started }
|
||
migrator: { condition: service_completed_successfully }
|
||
environment:
|
||
OTOPCUA_ROLES: "driver"
|
||
ConnectionStrings__ConfigDb: "Server=sql,1433;Database=OtOpcUa;User Id=sa;Password=OtOpcUa!Dev123;TrustServerCertificate=True;"
|
||
Cluster__Hostname: "0.0.0.0"
|
||
Cluster__Port: "4053"
|
||
Cluster__PublicHostname: "site-a-1"
|
||
# Site nodes are deliberately NOT seeds — they join the central pair's mesh. The self-first
|
||
# seed rule is conditional for exactly this reason (it binds only when a node's own address
|
||
# is in its own seed list), so these configs are exempt rather than broken.
|
||
Cluster__SeedNodes__0: "akka.tcp://otopcua@central-1:4053"
|
||
Cluster__Roles__0: "driver"
|
||
<<: *secrets-env
|
||
# Quiet EF/AspNetCore SQL flood — see central-1 (Serilog override). mem_limit/
|
||
# mem_reservation are inherited from the *otopcua-host anchor.
|
||
Serilog__MinimumLevel__Override__Microsoft.EntityFrameworkCore: "Warning"
|
||
Serilog__MinimumLevel__Override__Microsoft.AspNetCore: "Warning"
|
||
# Resolved at runtime by GalaxyDriver.ResolveApiKey when a DriverInstance's
|
||
# Gateway.ApiKeySecretRef = "env:GALAXY_MXGW_API_KEY".
|
||
GALAXY_MXGW_API_KEY: "${GALAXY_MXGW_API_KEY:-mxgw_otopcua2_GI7-tNozYE6cXGUSgEzL3AHDV7bYcYIHdMwKYgyHdX4}"
|
||
# Pair-local config cache + REPLICATION ON — the site-a pair is the enablement demo.
|
||
# site-a-1 is the initiator: it binds the h2c sync listener on 9001 AND dials the peer.
|
||
# Setting SyncListenPort makes Program.cs add a dedicated Http2-only Kestrel listener and
|
||
# re-bind the primary HTTP port (an explicit Listen* otherwise discards ASPNETCORE_URLS).
|
||
# Alarm store-and-forward buffer, Phase 2. Enabled on BOTH halves of the pair so the
|
||
# buffer is a live, replicated table rather than an empty one — the live gate needs a real
|
||
# queue to watch converge, and needs to see that only the Primary drains it.
|
||
#
|
||
# There is no HistorianGateway on this rig, so the endpoint below is deliberately
|
||
# unresolvable: every drain attempt fails, which is exactly the historian-outage state the
|
||
# buffer exists for. Note the endpoint must still be a syntactically valid absolute http(s)
|
||
# URI or the host refuses to start — ServerHistorianOptionsValidator is consumer-gated, so
|
||
# AlarmHistorian:Enabled=true makes it required even while ServerHistorian:Enabled=false.
|
||
#
|
||
# MaxAttempts is raised far above the production default of 10. With a permanently
|
||
# unreachable gateway the default would dead-letter the whole queue about five minutes into
|
||
# the outage, turning a buffering test into a dead-letter test.
|
||
AlarmHistorian__Enabled: "true"
|
||
AlarmHistorian__MaxAttempts: "1000000"
|
||
ServerHistorian__Endpoint: "${OTOPCUA_HISTORIAN_ENDPOINT:-http://histgw-absent.invalid:5222}"
|
||
# Required whenever the section is consumed: the gateway client validates its own options at
|
||
# construction and throws on an empty key, so a keyless node crash-loops during Akka startup
|
||
# rather than degrading. DEV-ONLY placeholder — it authenticates nothing, because the endpoint
|
||
# above resolves nowhere. NEVER reuse this shape for a real gateway key; supply those via
|
||
# ServerHistorian__ApiKey from the environment.
|
||
ServerHistorian__ApiKey: "${OTOPCUA_HISTORIAN_APIKEY:-histgw_dev_rig-no-gateway-present}"
|
||
# Must agree with the endpoint scheme above — the client throws on a mismatch in EITHER
|
||
# direction, so an http endpoint with the default UseTls=true crash-loops the host.
|
||
ServerHistorian__UseTls: "false"
|
||
LocalDb__Path: "/app/data/otopcua-localdb.db"
|
||
LocalDb__SyncListenPort: "9001"
|
||
LocalDb__Replication__PeerAddress: "http://site-a-2:9001"
|
||
# DEV-ONLY committed key (rig philosophy — like the SQL password + secrets KEK above). It
|
||
# MUST be byte-identical on both nodes: the interceptor is fail-closed, so any mismatch
|
||
# silently stops the pair converging. NEVER reuse this key outside this local dev rig.
|
||
LocalDb__Replication__ApiKey: "dev-site-a-localdb-sync-key"
|
||
# Row-count batching against gRPC's 4 MB message cap; artifact chunk rows are ≈171 KB, so
|
||
# 16 × 171 KB ≈ 2.7 MB stays under the cap with headroom.
|
||
LocalDb__Replication__MaxBatchSize: "16"
|
||
ports:
|
||
- "4842:4840"
|
||
volumes:
|
||
- otopcua-localdb-site-a-1:/app/data
|
||
|
||
site-a-2:
|
||
<<: *otopcua-host
|
||
depends_on:
|
||
sql: { condition: service_healthy }
|
||
central-1: { condition: service_started }
|
||
migrator: { condition: service_completed_successfully }
|
||
environment:
|
||
OTOPCUA_ROLES: "driver"
|
||
ConnectionStrings__ConfigDb: "Server=sql,1433;Database=OtOpcUa;User Id=sa;Password=OtOpcUa!Dev123;TrustServerCertificate=True;"
|
||
Cluster__Hostname: "0.0.0.0"
|
||
Cluster__Port: "4053"
|
||
Cluster__PublicHostname: "site-a-2"
|
||
Cluster__SeedNodes__0: "akka.tcp://otopcua@central-1:4053"
|
||
Cluster__Roles__0: "driver"
|
||
<<: *secrets-env
|
||
Serilog__MinimumLevel__Override__Microsoft.EntityFrameworkCore: "Warning"
|
||
Serilog__MinimumLevel__Override__Microsoft.AspNetCore: "Warning"
|
||
GALAXY_MXGW_API_KEY: "${GALAXY_MXGW_API_KEY:-mxgw_otopcua2_GI7-tNozYE6cXGUSgEzL3AHDV7bYcYIHdMwKYgyHdX4}"
|
||
# site-a-2 is the passive half of the replicating pair: it binds the sync listener on 9001
|
||
# but does NOT dial (no PeerAddress). The replication stream is bidirectional, so a-1's
|
||
# single dial carries both directions. Same key + MaxBatchSize as a-1 (byte-identical key
|
||
# is mandatory — the interceptor fail-closes on a mismatch).
|
||
# Alarm store-and-forward buffer, Phase 2. Enabled on BOTH halves of the pair so the
|
||
# buffer is a live, replicated table rather than an empty one — the live gate needs a real
|
||
# queue to watch converge, and needs to see that only the Primary drains it.
|
||
#
|
||
# There is no HistorianGateway on this rig, so the endpoint below is deliberately
|
||
# unresolvable: every drain attempt fails, which is exactly the historian-outage state the
|
||
# buffer exists for. Note the endpoint must still be a syntactically valid absolute http(s)
|
||
# URI or the host refuses to start — ServerHistorianOptionsValidator is consumer-gated, so
|
||
# AlarmHistorian:Enabled=true makes it required even while ServerHistorian:Enabled=false.
|
||
#
|
||
# MaxAttempts is raised far above the production default of 10. With a permanently
|
||
# unreachable gateway the default would dead-letter the whole queue about five minutes into
|
||
# the outage, turning a buffering test into a dead-letter test.
|
||
AlarmHistorian__Enabled: "true"
|
||
AlarmHistorian__MaxAttempts: "1000000"
|
||
ServerHistorian__Endpoint: "${OTOPCUA_HISTORIAN_ENDPOINT:-http://histgw-absent.invalid:5222}"
|
||
# Required whenever the section is consumed: the gateway client validates its own options at
|
||
# construction and throws on an empty key, so a keyless node crash-loops during Akka startup
|
||
# rather than degrading. DEV-ONLY placeholder — it authenticates nothing, because the endpoint
|
||
# above resolves nowhere. NEVER reuse this shape for a real gateway key; supply those via
|
||
# ServerHistorian__ApiKey from the environment.
|
||
ServerHistorian__ApiKey: "${OTOPCUA_HISTORIAN_APIKEY:-histgw_dev_rig-no-gateway-present}"
|
||
# Must agree with the endpoint scheme above — the client throws on a mismatch in EITHER
|
||
# direction, so an http endpoint with the default UseTls=true crash-loops the host.
|
||
ServerHistorian__UseTls: "false"
|
||
LocalDb__Path: "/app/data/otopcua-localdb.db"
|
||
LocalDb__SyncListenPort: "9001"
|
||
LocalDb__Replication__ApiKey: "dev-site-a-localdb-sync-key"
|
||
LocalDb__Replication__MaxBatchSize: "16"
|
||
ports:
|
||
- "4843:4840"
|
||
volumes:
|
||
- otopcua-localdb-site-a-2:/app/data
|
||
|
||
# ── Site B cluster (2-node driver-only) ─────────────────────────────────────
|
||
|
||
site-b-1:
|
||
<<: *otopcua-host
|
||
depends_on:
|
||
sql: { condition: service_healthy }
|
||
central-1: { condition: service_started }
|
||
migrator: { condition: service_completed_successfully }
|
||
environment:
|
||
OTOPCUA_ROLES: "driver"
|
||
ConnectionStrings__ConfigDb: "Server=sql,1433;Database=OtOpcUa;User Id=sa;Password=OtOpcUa!Dev123;TrustServerCertificate=True;"
|
||
Cluster__Hostname: "0.0.0.0"
|
||
Cluster__Port: "4053"
|
||
Cluster__PublicHostname: "site-b-1"
|
||
Cluster__SeedNodes__0: "akka.tcp://otopcua@central-1:4053"
|
||
Cluster__Roles__0: "driver"
|
||
<<: *secrets-env
|
||
Serilog__MinimumLevel__Override__Microsoft.EntityFrameworkCore: "Warning"
|
||
Serilog__MinimumLevel__Override__Microsoft.AspNetCore: "Warning"
|
||
GALAXY_MXGW_API_KEY: "${GALAXY_MXGW_API_KEY:-mxgw_otopcua2_GI7-tNozYE6cXGUSgEzL3AHDV7bYcYIHdMwKYgyHdX4}"
|
||
# site-b is the default-OFF pin: it gets the pair-local cache but NO SyncListenPort and NO
|
||
# replication config, so no sync listener binds and ISyncStatus stays disconnected/Healthy.
|
||
# This is what the live gate's check 8 verifies — the cache works locally with replication off.
|
||
# Alarm store-and-forward with replication OFF — the other half of the site-b pin. The sink
|
||
# must work as a plain node-local buffer here, with no sync listener and no peer traffic.
|
||
AlarmHistorian__Enabled: "true"
|
||
AlarmHistorian__MaxAttempts: "1000000"
|
||
ServerHistorian__Endpoint: "${OTOPCUA_HISTORIAN_ENDPOINT:-http://histgw-absent.invalid:5222}"
|
||
# Required whenever the section is consumed: the gateway client validates its own options at
|
||
# construction and throws on an empty key, so a keyless node crash-loops during Akka startup
|
||
# rather than degrading. DEV-ONLY placeholder — it authenticates nothing, because the endpoint
|
||
# above resolves nowhere. NEVER reuse this shape for a real gateway key; supply those via
|
||
# ServerHistorian__ApiKey from the environment.
|
||
ServerHistorian__ApiKey: "${OTOPCUA_HISTORIAN_APIKEY:-histgw_dev_rig-no-gateway-present}"
|
||
# Must agree with the endpoint scheme above — the client throws on a mismatch in EITHER
|
||
# direction, so an http endpoint with the default UseTls=true crash-loops the host.
|
||
ServerHistorian__UseTls: "false"
|
||
LocalDb__Path: "/app/data/otopcua-localdb.db"
|
||
ports:
|
||
- "4844:4840"
|
||
volumes:
|
||
- otopcua-localdb-site-b-1:/app/data
|
||
|
||
site-b-2:
|
||
<<: *otopcua-host
|
||
depends_on:
|
||
sql: { condition: service_healthy }
|
||
central-1: { condition: service_started }
|
||
migrator: { condition: service_completed_successfully }
|
||
environment:
|
||
OTOPCUA_ROLES: "driver"
|
||
ConnectionStrings__ConfigDb: "Server=sql,1433;Database=OtOpcUa;User Id=sa;Password=OtOpcUa!Dev123;TrustServerCertificate=True;"
|
||
Cluster__Hostname: "0.0.0.0"
|
||
Cluster__Port: "4053"
|
||
Cluster__PublicHostname: "site-b-2"
|
||
Cluster__SeedNodes__0: "akka.tcp://otopcua@central-1:4053"
|
||
Cluster__Roles__0: "driver"
|
||
<<: *secrets-env
|
||
Serilog__MinimumLevel__Override__Microsoft.EntityFrameworkCore: "Warning"
|
||
Serilog__MinimumLevel__Override__Microsoft.AspNetCore: "Warning"
|
||
GALAXY_MXGW_API_KEY: "${GALAXY_MXGW_API_KEY:-mxgw_otopcua2_GI7-tNozYE6cXGUSgEzL3AHDV7bYcYIHdMwKYgyHdX4}"
|
||
# site-b default-OFF pin (see site-b-1): cache on, replication off.
|
||
# Alarm store-and-forward with replication OFF — the other half of the site-b pin. The sink
|
||
# must work as a plain node-local buffer here, with no sync listener and no peer traffic.
|
||
AlarmHistorian__Enabled: "true"
|
||
AlarmHistorian__MaxAttempts: "1000000"
|
||
ServerHistorian__Endpoint: "${OTOPCUA_HISTORIAN_ENDPOINT:-http://histgw-absent.invalid:5222}"
|
||
# Required whenever the section is consumed: the gateway client validates its own options at
|
||
# construction and throws on an empty key, so a keyless node crash-loops during Akka startup
|
||
# rather than degrading. DEV-ONLY placeholder — it authenticates nothing, because the endpoint
|
||
# above resolves nowhere. NEVER reuse this shape for a real gateway key; supply those via
|
||
# ServerHistorian__ApiKey from the environment.
|
||
ServerHistorian__ApiKey: "${OTOPCUA_HISTORIAN_APIKEY:-histgw_dev_rig-no-gateway-present}"
|
||
# Must agree with the endpoint scheme above — the client throws on a mismatch in EITHER
|
||
# direction, so an http endpoint with the default UseTls=true crash-loops the host.
|
||
ServerHistorian__UseTls: "false"
|
||
LocalDb__Path: "/app/data/otopcua-localdb.db"
|
||
ports:
|
||
- "4845:4840"
|
||
volumes:
|
||
- otopcua-localdb-site-b-2:/app/data
|
||
|
||
traefik:
|
||
image: traefik:v3.1
|
||
command:
|
||
- --entrypoints.web.address=:80
|
||
- --providers.file.filename=/etc/traefik/dynamic.yml
|
||
- --providers.file.watch=true
|
||
- --api.insecure=true
|
||
ports:
|
||
- "9200:80" # host port 9200 → traefik :80 entrypoint (80 conflicts with scadabridge-traefik)
|
||
- "8089:8080" # 8080 conflicts with the sister scadalink dev stack
|
||
volumes:
|
||
- ./traefik-dynamic.yml:/etc/traefik/dynamic.yml:ro
|
||
depends_on:
|
||
- central-1
|
||
- central-2
|
||
|
||
volumes:
|
||
# SQL Server data dir — persists the OtOpcUa ConfigDb across container recreates.
|
||
otopcua-mssql-data:
|
||
# Per-node pair-local config-cache (ZB.MOM.WW.LocalDb) files. One volume per node — the
|
||
# cache is node-local, not shared; it persists the cached deployment artifact (and, on the
|
||
# site-a pair, the replicated oplog state) across container recreates so a node can boot
|
||
# from cache after a restart even with central SQL down.
|
||
otopcua-localdb-central-1:
|
||
otopcua-localdb-central-2:
|
||
otopcua-localdb-site-a-1:
|
||
otopcua-localdb-site-a-2:
|
||
otopcua-localdb-site-b-1:
|
||
otopcua-localdb-site-b-2:
|