Joseph Doherty
f9f1b8fcee
Gate record: docs/plans/2026-07-20-localdb-phase2-live-gate.md. Checks 1, 2, 5, 6 pass. Checks 3 and 4 are NOT satisfied — the defect they were meant to confirm turned out to be the opposite of what the plan assumed. Three defects crash-looped every driver node before check 1 could even run: 1. An empty ServerHistorian:ApiKey kills the host. ServerHistorianOptions- Validator exists to turn exactly that class of failure into a named OptionsValidationException, but its documented fail tier explicitly excluded ApiKey on the reasoning that a keyless client "degrades — the gateway rejects calls". It does not: the client validates its own options at construction, so the process dies during Akka startup and never makes a call. 2. UseTls disagreeing with the endpoint scheme kills the host too, in both directions (both messages confirmed in the shipped client assembly). Moving an endpoint from https to http without clearing UseTls is an ordinary migration slip. 3. Plaintext h2c was UNREACHABLE. HistorianGatewayClientAdapter forwarded the TLS-only options unconditionally, and AllowUntrustedServerCertificate defaults to false, so it always sent RequireCertificateValidation=true — which the client rejects outright when UseTls=false. Every http:// deployment crashed, though the scheme is documented as the supported way to select h2c, and the only workaround was to assert a certificate posture for a connection that has no certificate. The fourth was the blocker, and it is Phase 2's own: 4. The drain gate deferred to a Primary that cannot deliver. Redundancy roles are elected CLUSTER-WIDE; the alarm queue is PAIR-LOCAL. On the rig the elected driver Primary is central-1 — it carries the driver Akka role, replicates nobody's LocalDb and does not even run the alarm historian — so every driver node logged "Historian drain suspended", including the two site-b nodes that have no peer at all. Nothing drained anywhere, where before Phase 2 it drained fine. The cost is not a duplicate; it is the buffer growing to the capacity wall and evicting the audit trail it exists to protect. Fixed in three layers: a separate ShouldDrainAlarmHistory policy (unknown role drains; the two gates now deliberately disagree, and a test pins that); peer- host matching in DriverHostActor so a node stands down only for a Primary holding its rows; and AddAlarmHistorian short-circuiting the gate when replication is unconfigured — testing BOTH Replication:PeerAddress and SyncListenPort, since only the dialing half sets the former while both halves share the queue. Every one of these follows from the asymmetry: a false allow costs a duplicate row, which at-least-once delivery already accepts and payload-hash ids collapse; a false deny loses data silently. A third vacuous test, caught by the same delete-the-guard discipline: the role-view tests stayed green with the guard removed, because AwaitAssert polls until an assertion passes and the assertion was "reads open" — which is the SEEDED value, satisfied at the first poll before the actor processed anything. They now assert the sequence of published values through a recording view; the control then goes red for exactly the cases that matter. Migration evidence: 11 legacy rows across two deliberately overlapping files converged to exactly 9 identical rows on both nodes, proving D-6's payload-hash identity on real nodes rather than in a fixture. Open design fork, recorded in the gate doc rather than decided here: a pair cannot currently identify its own Primary, so both halves drain. Safe in every topology — nothing loses data — but the gate's de-duplication benefit is unrealised until roles are scoped per pair. Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
538 lines
30 KiB
YAML
538 lines
30 KiB
YAML
# docker-dev/ — Mac-friendly single-mesh hub-and-spoke fleet for v2 development + manual UI exercise.
|
||
#
|
||
# Topology: ONE Akka mesh seeded by `central-1`. Logical separation between
|
||
# tenants is by ServerCluster.ClusterId rows (MAIN / SITE-A / SITE-B) in the one
|
||
# shared `OtOpcUa` ConfigDb — NOT by separate meshes. All six host nodes join the
|
||
# same gossip ring and the central UI deploys to every cluster over it.
|
||
#
|
||
# Stack:
|
||
# sql SQL Server 2022 — hosts the one ConfigDb every node uses
|
||
# cluster-seed one-shot mssql-tools job that INSERTs the ServerCluster +
|
||
# ClusterNode rows scoping each tenant, then exits (idempotent)
|
||
#
|
||
# central-1, central-2 OTOPCUA_ROLES=admin,driver — the ONLY UI + deploy
|
||
# singleton, plus the MAIN cluster's OPC UA publishers.
|
||
# Reachable at http://localhost:9200 (via Traefik).
|
||
# central-1 is the Akka seed node; central-2 joins it.
|
||
# site-a-1, site-a-2 OTOPCUA_ROLES=driver — driver-only members of the same
|
||
# site-b-1, site-b-2 mesh, scoped to SITE-A / SITE-B by ClusterId. They
|
||
# serve no UI and authenticate no users; the central
|
||
# cluster manages and deploys to them over the mesh.
|
||
#
|
||
# Auth is real LDAP against the shared GLAuth on the Linux Docker host
|
||
# (10.100.0.35:3893, dc=zb,dc=local) — there is no LDAP container here.
|
||
# Only the admin-role central nodes carry the Security__Ldap__* block.
|
||
# Sign in `multi-role` / `password`.
|
||
#
|
||
# traefik PathPrefix(`/`) → central-1 / central-2 (the single UI route).
|
||
#
|
||
# OPC UA endpoints (host-side port → container 4840):
|
||
# central-1 :4840 central-2 :4841
|
||
# site-a-1 :4842 site-a-2 :4843
|
||
# site-b-1 :4844 site-b-2 :4845
|
||
#
|
||
# Headless deploy: POST http://localhost:9200/api/deployments with the
|
||
# X-Api-Key header (Security__DeployApiKey = "docker-dev-deploy-key").
|
||
#
|
||
# SQL persistence: the otopcua-mssql-data named volume keeps the ConfigDb schema
|
||
# + seeded clusters across `docker compose up` cycles; without it a recreate
|
||
# silently drops the OtOpcUa database.
|
||
#
|
||
# Usage:
|
||
# docker compose -f docker-dev/docker-compose.yml up -d --build
|
||
# open http://localhost:9200 # central Blazor admin UI
|
||
# open http://localhost:8089 # Traefik dashboard (8080 is the sister scadalink stack)
|
||
#
|
||
# Tear-down: docker compose -f docker-dev/docker-compose.yml down -v
|
||
|
||
name: otopcua-dev
|
||
|
||
# ── Shared secret-replication env (all six host nodes) ──────────────────────────
|
||
# Akka peer-to-peer clustered secret replication rides the existing `otopcua` Akka
|
||
# mesh (DistributedPubSub topic `zb-mom-ww-secrets`) across all six host nodes. Two
|
||
# hard requirements are wired here, identically, on every node:
|
||
# 1. Secrets__Replication__Enabled=true → SecretsRegistration.AddOtOpcUaSecrets
|
||
# switches from plain AddZbSecrets to AddZbSecretsAkkaReplication + the eager
|
||
# SecretReplicationStarter hosted service, so every node joins anti-entropy.
|
||
# 2. ZB_SECRETS_MASTER_KEY → the ONE shared 32-byte base64 KEK. Every node MUST
|
||
# carry the SAME key or peers fail closed decrypting each other's rows
|
||
# (kek_id mismatch). appsettings.json binds Secrets:MasterKey:Source=Environment
|
||
# / EnvVarName=ZB_SECRETS_MASTER_KEY, so this env var IS the KEK on every node.
|
||
# The KEK below is a committed DEV-ONLY default (matches the rig's zero-operator-step
|
||
# philosophy, like the committed dev SQL password). Override for a fresh key with:
|
||
# OTOPCUA_SECRETS_KEK=$(openssl rand -base64 32) docker compose ... up -d
|
||
# NEVER reuse this key outside this local dev rig.
|
||
# AnnounceInterval is shortened to 5s (default 30s) so anti-entropy convergence is
|
||
# observable quickly during dev exercise. Each node keeps its own local SQLite store
|
||
# (Secrets:SqlitePath=otopcua-secrets.db, per-container); replication syncs them.
|
||
x-secrets-env: &secrets-env
|
||
Secrets__Replication__Enabled: "true"
|
||
Secrets__Replication__AnnounceInterval: "00:00:05"
|
||
ZB_SECRETS_MASTER_KEY: "${OTOPCUA_SECRETS_KEK:-ZYGhIX0luS/XsevpCB2W18jYHMcqO6AjM9oXy+T6Zp4=}"
|
||
|
||
services:
|
||
|
||
sql:
|
||
image: mcr.microsoft.com/mssql/server:2022-latest
|
||
environment:
|
||
ACCEPT_EULA: "Y"
|
||
SA_PASSWORD: "OtOpcUa!Dev123"
|
||
MSSQL_PID: Developer
|
||
ports:
|
||
- "14330:1433"
|
||
# Persist the ConfigDb across container recreates. Without this the dev SQL
|
||
# is ephemeral (container writable layer), so a recreate silently drops the
|
||
# OtOpcUa database and every host node fails its configdb health check until
|
||
# EF auto-migration + cluster-seed rebuild it. The named volume keeps the
|
||
# schema + seeded clusters between `docker compose up` cycles.
|
||
volumes:
|
||
- otopcua-mssql-data:/var/opt/mssql
|
||
healthcheck:
|
||
test: ["CMD-SHELL", "/opt/mssql-tools18/bin/sqlcmd -S localhost -U sa -P 'OtOpcUa!Dev123' -No -Q 'SELECT 1' || exit 1"]
|
||
interval: 10s
|
||
timeout: 5s
|
||
retries: 20
|
||
|
||
# ── Migrator (one-shot) ────────────────────────────────────────────────────
|
||
# Applies EF Core migrations to the OtOpcUa ConfigDb so a fresh SQL volume gets
|
||
# the schema with no operator step (the host nodes deliberately don't auto-
|
||
# migrate — production owns schema changes). cluster-seed + every host node
|
||
# depend on this completing, so nothing races an in-progress migration.
|
||
# Idempotent: a no-op once the schema is current.
|
||
migrator:
|
||
build:
|
||
context: ..
|
||
dockerfile: docker-dev/Dockerfile
|
||
target: migrator
|
||
image: otopcua-migrator:dev
|
||
depends_on:
|
||
sql:
|
||
condition: service_healthy
|
||
environment:
|
||
OTOPCUA_CONFIG_CONNECTION: "Server=sql,1433;Database=OtOpcUa;User Id=sa;Password=OtOpcUa!Dev123;TrustServerCertificate=True;"
|
||
restart: "no"
|
||
|
||
# ── Cluster seed (one-shot) ────────────────────────────────────────────────
|
||
# Runs only after `migrator` completes (so the schema is final — no race), then
|
||
# INSERTs the three ServerCluster rows and the six ClusterNode rows that scope
|
||
# each tenant inside the shared OtOpcUa ConfigDb. Idempotent — re-runs are no-ops.
|
||
cluster-seed:
|
||
image: mcr.microsoft.com/mssql-tools:latest
|
||
depends_on:
|
||
migrator:
|
||
condition: service_completed_successfully
|
||
volumes:
|
||
- ./seed:/seed:ro
|
||
entrypoint: ["/bin/bash", "/seed/entrypoint.sh"]
|
||
restart: "no"
|
||
|
||
# A local OpenLDAP container used to live here, but the bitnami/openldap:2.6
|
||
# image was retired (manifest gone) and bitnamilegacy/openldap:2.6 crashes
|
||
# during LDIF setup (exit 68). Rather than stub auth, the central (admin-role)
|
||
# containers bind the shared GLAuth on the Linux Docker host (Security__Ldap__*
|
||
# below: 10.100.0.35:3893, dc=zb,dc=local, DevStubMode=false) — so dev auth
|
||
# exercises the real LDAP bind + group→role path. Sign in `multi-role` /
|
||
# `password` (all roles) or any shared test user / `password`.
|
||
|
||
# ── Central cluster (2-node fused admin+driver) ─────────────────────────────
|
||
# The only UI + deploy singleton; also the MAIN cluster's OPC UA publishers.
|
||
# central-1 seeds the single Akka mesh that every other node joins.
|
||
|
||
central-1: &otopcua-host
|
||
build:
|
||
context: ..
|
||
dockerfile: docker-dev/Dockerfile
|
||
target: runtime
|
||
image: otopcua-host:dev
|
||
# Per-node memory bounds. The full single-mesh stack (6 host nodes) OOM-killed
|
||
# central-1 on a loaded host. Each host node measured ~357 MiB idle-solo and
|
||
# climbs sharply under deploy/materialise load: a node materialising its full
|
||
# cluster slice (e.g. central → MAIN's galaxy mirror + UNS overlay, ~1400 OPC UA
|
||
# nodes) peaks well above 1g during a deploy — a 1g cap OOM-kills it (exit 137).
|
||
# Cap at 2g (≈peak + headroom) with a 1g reservation. These top-level keys are
|
||
# inherited by every service that uses `<<: *otopcua-host` (YAML merge keeps the
|
||
# anchor's scalar keys; only the `environment` block is re-declared per service).
|
||
# The full 6-node mesh needs ~12g of Docker Desktop VM memory — on a constrained
|
||
# host raise the VM memory or run fewer host services.
|
||
mem_limit: 2g
|
||
mem_reservation: 1g
|
||
# Restart supervision. A 2-node keep-oldest SBR downs the LONE survivor when the
|
||
# OLDEST node crashes (the survivor is "the side without the oldest" → DownReachable
|
||
# → run-coordinated-shutdown-when-down terminates it). Recovery is by exit-and-rejoin:
|
||
# the supervisor restarts the exited node and it re-forms/rejoins as a fresh
|
||
# incarnation (the ScadaBridge pattern). Without a restart policy the exited node
|
||
# stays down and the cluster looks like a total outage. Inherited by every host node
|
||
# via the anchor merge.
|
||
restart: unless-stopped
|
||
depends_on:
|
||
sql: { condition: service_healthy }
|
||
migrator: { condition: service_completed_successfully }
|
||
environment:
|
||
<<: *secrets-env
|
||
OTOPCUA_ROLES: "admin,driver"
|
||
ASPNETCORE_URLS: "http://+:9000"
|
||
ConnectionStrings__ConfigDb: "Server=sql,1433;Database=OtOpcUa;User Id=sa;Password=OtOpcUa!Dev123;TrustServerCertificate=True;"
|
||
Cluster__Hostname: "0.0.0.0"
|
||
Cluster__Port: "4053"
|
||
Cluster__PublicHostname: "central-1"
|
||
# Both redundancy peers are seeds (#459 / ScadaBridge parity) so a restarted node can
|
||
# re-join the mesh via EITHER peer, not only central-1. With restart supervision this is
|
||
# the 2-node keep-oldest exit-and-rejoin recovery path.
|
||
Cluster__SeedNodes__0: "akka.tcp://otopcua@central-1:4053"
|
||
Cluster__SeedNodes__1: "akka.tcp://otopcua@central-2:4053"
|
||
Cluster__Roles__0: "admin"
|
||
Cluster__Roles__1: "driver"
|
||
Security__Jwt__SigningKey: "docker-dev-signing-key-with-at-least-32-bytes-of-utf8-content-12345"
|
||
Security__Jwt__Issuer: "otopcua-dev"
|
||
Security__Jwt__Audience: "otopcua-dev"
|
||
Security__Ldap__Enabled: "true"
|
||
Security__Ldap__DevStubMode: "false"
|
||
Security__Ldap__Server: "10.100.0.35"
|
||
Security__Ldap__Port: "3893"
|
||
Security__Ldap__Transport: "None"
|
||
Security__Ldap__AllowInsecure: "true"
|
||
Security__Ldap__SearchBase: "dc=zb,dc=local"
|
||
Security__Ldap__ServiceAccountDn: "cn=serviceaccount,dc=zb,dc=local"
|
||
Security__Ldap__ServiceAccountPassword: "serviceaccount123"
|
||
Security__DeployApiKey: "docker-dev-deploy-key"
|
||
Security__Auth__DisableLogin: "true"
|
||
# Map the shared-GLAuth data-plane groups (ReadOnly/WriteOperate/...) to data-plane role
|
||
# strings the OPC UA session ACL gates read. The DB LdapGroupRoleMapping is AdminRole-typed
|
||
# (AdminUI roles); this free-form GroupToRole baseline is the path for data-plane roles.
|
||
Security__Ldap__GroupToRole__ReadOnly: "ReadOnly"
|
||
Security__Ldap__GroupToRole__WriteOperate: "WriteOperate"
|
||
Security__Ldap__GroupToRole__WriteTune: "WriteTune"
|
||
Security__Ldap__GroupToRole__WriteConfigure: "WriteConfigure"
|
||
Security__Ldap__GroupToRole__AlarmAck: "AlarmAck"
|
||
# Server-side HistoryRead via the ZB.MOM.WW.HistorianGateway sidecar. DISABLED by default: the
|
||
# docker-dev rig has no historian gateway to point at, and the historian-gateway cutover retired the
|
||
# old Wonderware Host/Port/SharedSecret keys — the code now reads ServerHistorian__Endpoint and builds
|
||
# a Uri from it, so Enabled=true with an empty Endpoint crashes the Host at startup
|
||
# (HistorianGatewayClientAdapter.Create -> UriFormatException "The URI is empty").
|
||
# To enable: OTOPCUA_HISTORIAN_ENABLED=true + OTOPCUA_HISTORIAN_ENDPOINT=https://<host>:5222, and
|
||
# export ServerHistorian__ApiKey=histgw_<id>_<secret> at `docker compose up` (never commit the key).
|
||
ServerHistorian__Enabled: "${OTOPCUA_HISTORIAN_ENABLED:-false}"
|
||
ServerHistorian__Endpoint: "${OTOPCUA_HISTORIAN_ENDPOINT:-}"
|
||
ServerHistorian__ApiKey: "${ServerHistorian__ApiKey:-}"
|
||
ServerHistorian__UseTls: "true"
|
||
ServerHistorian__AllowUntrustedServerCertificate: "true"
|
||
# Pin EF Core + ASP.NET Core to Warning so the per-poll Deployment SELECT /
|
||
# "Executed DbCommand" Information|Debug lines stop flooding the Serilog
|
||
# pipeline and starving the Akka cluster heartbeat thread. The host logs via
|
||
# Serilog (AddZbSerilog → ReadFrom.Configuration); these env vars override
|
||
# Serilog:MinimumLevel:Override:* (app/Akka levels are left untouched).
|
||
Serilog__MinimumLevel__Override__Microsoft.EntityFrameworkCore: "Warning"
|
||
Serilog__MinimumLevel__Override__Microsoft.AspNetCore: "Warning"
|
||
GALAXY_MXGW_API_KEY: "${GALAXY_MXGW_API_KEY:-mxgw_otopcua2_GI7-tNozYE6cXGUSgEzL3AHDV7bYcYIHdMwKYgyHdX4}"
|
||
# Consolidated pair-local config cache (ZB.MOM.WW.LocalDb). Every driver-role node
|
||
# (central is admin,driver) gets one, on a per-node named volume so the cache survives
|
||
# container recreates. The central pair leaves replication OFF (default) — only the
|
||
# site-a pair replicates, as the enablement demo; site-b is the default-OFF pin.
|
||
LocalDb__Path: "/app/data/otopcua-localdb.db"
|
||
ports:
|
||
- "4840:4840"
|
||
volumes:
|
||
- otopcua-localdb-central-1:/app/data
|
||
|
||
central-2:
|
||
<<: *otopcua-host
|
||
depends_on:
|
||
sql: { condition: service_healthy }
|
||
central-1: { condition: service_started }
|
||
migrator: { condition: service_completed_successfully }
|
||
environment:
|
||
<<: *secrets-env
|
||
OTOPCUA_ROLES: "admin,driver"
|
||
ASPNETCORE_URLS: "http://+:9000"
|
||
ConnectionStrings__ConfigDb: "Server=sql,1433;Database=OtOpcUa;User Id=sa;Password=OtOpcUa!Dev123;TrustServerCertificate=True;"
|
||
Cluster__Hostname: "0.0.0.0"
|
||
Cluster__Port: "4053"
|
||
Cluster__PublicHostname: "central-2"
|
||
# Both redundancy peers are seeds (#459 / ScadaBridge parity) — see central-1.
|
||
Cluster__SeedNodes__0: "akka.tcp://otopcua@central-1:4053"
|
||
Cluster__SeedNodes__1: "akka.tcp://otopcua@central-2:4053"
|
||
Cluster__Roles__0: "admin"
|
||
Cluster__Roles__1: "driver"
|
||
Security__Jwt__SigningKey: "docker-dev-signing-key-with-at-least-32-bytes-of-utf8-content-12345"
|
||
Security__Jwt__Issuer: "otopcua-dev"
|
||
Security__Jwt__Audience: "otopcua-dev"
|
||
Security__Ldap__Enabled: "true"
|
||
Security__Ldap__DevStubMode: "false"
|
||
Security__Ldap__Server: "10.100.0.35"
|
||
Security__Ldap__Port: "3893"
|
||
Security__Ldap__Transport: "None"
|
||
Security__Ldap__AllowInsecure: "true"
|
||
Security__Ldap__SearchBase: "dc=zb,dc=local"
|
||
Security__Ldap__ServiceAccountDn: "cn=serviceaccount,dc=zb,dc=local"
|
||
Security__Ldap__ServiceAccountPassword: "serviceaccount123"
|
||
Security__DeployApiKey: "docker-dev-deploy-key"
|
||
Security__Auth__DisableLogin: "true"
|
||
# Data-plane group→role baseline (see central-1).
|
||
Security__Ldap__GroupToRole__ReadOnly: "ReadOnly"
|
||
Security__Ldap__GroupToRole__WriteOperate: "WriteOperate"
|
||
Security__Ldap__GroupToRole__WriteTune: "WriteTune"
|
||
Security__Ldap__GroupToRole__WriteConfigure: "WriteConfigure"
|
||
Security__Ldap__GroupToRole__AlarmAck: "AlarmAck"
|
||
# Server-side HistoryRead via the ZB.MOM.WW.HistorianGateway sidecar. DISABLED by default: the
|
||
# docker-dev rig has no historian gateway to point at, and the historian-gateway cutover retired the
|
||
# old Wonderware Host/Port/SharedSecret keys — the code now reads ServerHistorian__Endpoint and builds
|
||
# a Uri from it, so Enabled=true with an empty Endpoint crashes the Host at startup
|
||
# (HistorianGatewayClientAdapter.Create -> UriFormatException "The URI is empty").
|
||
# To enable: OTOPCUA_HISTORIAN_ENABLED=true + OTOPCUA_HISTORIAN_ENDPOINT=https://<host>:5222, and
|
||
# export ServerHistorian__ApiKey=histgw_<id>_<secret> at `docker compose up` (never commit the key).
|
||
ServerHistorian__Enabled: "${OTOPCUA_HISTORIAN_ENABLED:-false}"
|
||
ServerHistorian__Endpoint: "${OTOPCUA_HISTORIAN_ENDPOINT:-}"
|
||
ServerHistorian__ApiKey: "${ServerHistorian__ApiKey:-}"
|
||
ServerHistorian__UseTls: "true"
|
||
ServerHistorian__AllowUntrustedServerCertificate: "true"
|
||
# Quiet EF/AspNetCore SQL flood — see central-1 (Serilog override). mem_limit/
|
||
# mem_reservation are inherited from the *otopcua-host anchor.
|
||
Serilog__MinimumLevel__Override__Microsoft.EntityFrameworkCore: "Warning"
|
||
Serilog__MinimumLevel__Override__Microsoft.AspNetCore: "Warning"
|
||
GALAXY_MXGW_API_KEY: "${GALAXY_MXGW_API_KEY:-mxgw_otopcua2_GI7-tNozYE6cXGUSgEzL3AHDV7bYcYIHdMwKYgyHdX4}"
|
||
# Pair-local config cache; central pair leaves replication OFF (see central-1).
|
||
LocalDb__Path: "/app/data/otopcua-localdb.db"
|
||
ports:
|
||
- "4841:4840"
|
||
volumes:
|
||
- otopcua-localdb-central-2:/app/data
|
||
|
||
# ── Site A cluster (2-node driver-only) ─────────────────────────────────────
|
||
# Driver-only members of the single mesh, scoped to SITE-A by ClusterId. No UI,
|
||
# no user auth — managed + deployed to by the central cluster over the mesh.
|
||
# All site nodes seed central-1.
|
||
|
||
site-a-1:
|
||
<<: *otopcua-host
|
||
depends_on:
|
||
sql: { condition: service_healthy }
|
||
central-1: { condition: service_started }
|
||
migrator: { condition: service_completed_successfully }
|
||
environment:
|
||
OTOPCUA_ROLES: "driver"
|
||
ConnectionStrings__ConfigDb: "Server=sql,1433;Database=OtOpcUa;User Id=sa;Password=OtOpcUa!Dev123;TrustServerCertificate=True;"
|
||
Cluster__Hostname: "0.0.0.0"
|
||
Cluster__Port: "4053"
|
||
Cluster__PublicHostname: "site-a-1"
|
||
Cluster__SeedNodes__0: "akka.tcp://otopcua@central-1:4053"
|
||
Cluster__Roles__0: "driver"
|
||
<<: *secrets-env
|
||
# Quiet EF/AspNetCore SQL flood — see central-1 (Serilog override). mem_limit/
|
||
# mem_reservation are inherited from the *otopcua-host anchor.
|
||
Serilog__MinimumLevel__Override__Microsoft.EntityFrameworkCore: "Warning"
|
||
Serilog__MinimumLevel__Override__Microsoft.AspNetCore: "Warning"
|
||
# Resolved at runtime by GalaxyDriver.ResolveApiKey when a DriverInstance's
|
||
# Gateway.ApiKeySecretRef = "env:GALAXY_MXGW_API_KEY".
|
||
GALAXY_MXGW_API_KEY: "${GALAXY_MXGW_API_KEY:-mxgw_otopcua2_GI7-tNozYE6cXGUSgEzL3AHDV7bYcYIHdMwKYgyHdX4}"
|
||
# Pair-local config cache + REPLICATION ON — the site-a pair is the enablement demo.
|
||
# site-a-1 is the initiator: it binds the h2c sync listener on 9001 AND dials the peer.
|
||
# Setting SyncListenPort makes Program.cs add a dedicated Http2-only Kestrel listener and
|
||
# re-bind the primary HTTP port (an explicit Listen* otherwise discards ASPNETCORE_URLS).
|
||
# Alarm store-and-forward buffer, Phase 2. Enabled on BOTH halves of the pair so the
|
||
# buffer is a live, replicated table rather than an empty one — the live gate needs a real
|
||
# queue to watch converge, and needs to see that only the Primary drains it.
|
||
#
|
||
# There is no HistorianGateway on this rig, so the endpoint below is deliberately
|
||
# unresolvable: every drain attempt fails, which is exactly the historian-outage state the
|
||
# buffer exists for. Note the endpoint must still be a syntactically valid absolute http(s)
|
||
# URI or the host refuses to start — ServerHistorianOptionsValidator is consumer-gated, so
|
||
# AlarmHistorian:Enabled=true makes it required even while ServerHistorian:Enabled=false.
|
||
#
|
||
# MaxAttempts is raised far above the production default of 10. With a permanently
|
||
# unreachable gateway the default would dead-letter the whole queue about five minutes into
|
||
# the outage, turning a buffering test into a dead-letter test.
|
||
AlarmHistorian__Enabled: "true"
|
||
AlarmHistorian__MaxAttempts: "1000000"
|
||
ServerHistorian__Endpoint: "${OTOPCUA_HISTORIAN_ENDPOINT:-http://histgw-absent.invalid:5222}"
|
||
# Required whenever the section is consumed: the gateway client validates its own options at
|
||
# construction and throws on an empty key, so a keyless node crash-loops during Akka startup
|
||
# rather than degrading. DEV-ONLY placeholder — it authenticates nothing, because the endpoint
|
||
# above resolves nowhere. NEVER reuse this shape for a real gateway key; supply those via
|
||
# ServerHistorian__ApiKey from the environment.
|
||
ServerHistorian__ApiKey: "${OTOPCUA_HISTORIAN_APIKEY:-histgw_dev_rig-no-gateway-present}"
|
||
# Must agree with the endpoint scheme above — the client throws on a mismatch in EITHER
|
||
# direction, so an http endpoint with the default UseTls=true crash-loops the host.
|
||
ServerHistorian__UseTls: "false"
|
||
LocalDb__Path: "/app/data/otopcua-localdb.db"
|
||
LocalDb__SyncListenPort: "9001"
|
||
LocalDb__Replication__PeerAddress: "http://site-a-2:9001"
|
||
# DEV-ONLY committed key (rig philosophy — like the SQL password + secrets KEK above). It
|
||
# MUST be byte-identical on both nodes: the interceptor is fail-closed, so any mismatch
|
||
# silently stops the pair converging. NEVER reuse this key outside this local dev rig.
|
||
LocalDb__Replication__ApiKey: "dev-site-a-localdb-sync-key"
|
||
# Row-count batching against gRPC's 4 MB message cap; artifact chunk rows are ≈171 KB, so
|
||
# 16 × 171 KB ≈ 2.7 MB stays under the cap with headroom.
|
||
LocalDb__Replication__MaxBatchSize: "16"
|
||
ports:
|
||
- "4842:4840"
|
||
volumes:
|
||
- otopcua-localdb-site-a-1:/app/data
|
||
|
||
site-a-2:
|
||
<<: *otopcua-host
|
||
depends_on:
|
||
sql: { condition: service_healthy }
|
||
central-1: { condition: service_started }
|
||
migrator: { condition: service_completed_successfully }
|
||
environment:
|
||
OTOPCUA_ROLES: "driver"
|
||
ConnectionStrings__ConfigDb: "Server=sql,1433;Database=OtOpcUa;User Id=sa;Password=OtOpcUa!Dev123;TrustServerCertificate=True;"
|
||
Cluster__Hostname: "0.0.0.0"
|
||
Cluster__Port: "4053"
|
||
Cluster__PublicHostname: "site-a-2"
|
||
Cluster__SeedNodes__0: "akka.tcp://otopcua@central-1:4053"
|
||
Cluster__Roles__0: "driver"
|
||
<<: *secrets-env
|
||
Serilog__MinimumLevel__Override__Microsoft.EntityFrameworkCore: "Warning"
|
||
Serilog__MinimumLevel__Override__Microsoft.AspNetCore: "Warning"
|
||
GALAXY_MXGW_API_KEY: "${GALAXY_MXGW_API_KEY:-mxgw_otopcua2_GI7-tNozYE6cXGUSgEzL3AHDV7bYcYIHdMwKYgyHdX4}"
|
||
# site-a-2 is the passive half of the replicating pair: it binds the sync listener on 9001
|
||
# but does NOT dial (no PeerAddress). The replication stream is bidirectional, so a-1's
|
||
# single dial carries both directions. Same key + MaxBatchSize as a-1 (byte-identical key
|
||
# is mandatory — the interceptor fail-closes on a mismatch).
|
||
# Alarm store-and-forward buffer, Phase 2. Enabled on BOTH halves of the pair so the
|
||
# buffer is a live, replicated table rather than an empty one — the live gate needs a real
|
||
# queue to watch converge, and needs to see that only the Primary drains it.
|
||
#
|
||
# There is no HistorianGateway on this rig, so the endpoint below is deliberately
|
||
# unresolvable: every drain attempt fails, which is exactly the historian-outage state the
|
||
# buffer exists for. Note the endpoint must still be a syntactically valid absolute http(s)
|
||
# URI or the host refuses to start — ServerHistorianOptionsValidator is consumer-gated, so
|
||
# AlarmHistorian:Enabled=true makes it required even while ServerHistorian:Enabled=false.
|
||
#
|
||
# MaxAttempts is raised far above the production default of 10. With a permanently
|
||
# unreachable gateway the default would dead-letter the whole queue about five minutes into
|
||
# the outage, turning a buffering test into a dead-letter test.
|
||
AlarmHistorian__Enabled: "true"
|
||
AlarmHistorian__MaxAttempts: "1000000"
|
||
ServerHistorian__Endpoint: "${OTOPCUA_HISTORIAN_ENDPOINT:-http://histgw-absent.invalid:5222}"
|
||
# Required whenever the section is consumed: the gateway client validates its own options at
|
||
# construction and throws on an empty key, so a keyless node crash-loops during Akka startup
|
||
# rather than degrading. DEV-ONLY placeholder — it authenticates nothing, because the endpoint
|
||
# above resolves nowhere. NEVER reuse this shape for a real gateway key; supply those via
|
||
# ServerHistorian__ApiKey from the environment.
|
||
ServerHistorian__ApiKey: "${OTOPCUA_HISTORIAN_APIKEY:-histgw_dev_rig-no-gateway-present}"
|
||
# Must agree with the endpoint scheme above — the client throws on a mismatch in EITHER
|
||
# direction, so an http endpoint with the default UseTls=true crash-loops the host.
|
||
ServerHistorian__UseTls: "false"
|
||
LocalDb__Path: "/app/data/otopcua-localdb.db"
|
||
LocalDb__SyncListenPort: "9001"
|
||
LocalDb__Replication__ApiKey: "dev-site-a-localdb-sync-key"
|
||
LocalDb__Replication__MaxBatchSize: "16"
|
||
ports:
|
||
- "4843:4840"
|
||
volumes:
|
||
- otopcua-localdb-site-a-2:/app/data
|
||
|
||
# ── Site B cluster (2-node driver-only) ─────────────────────────────────────
|
||
|
||
site-b-1:
|
||
<<: *otopcua-host
|
||
depends_on:
|
||
sql: { condition: service_healthy }
|
||
central-1: { condition: service_started }
|
||
migrator: { condition: service_completed_successfully }
|
||
environment:
|
||
OTOPCUA_ROLES: "driver"
|
||
ConnectionStrings__ConfigDb: "Server=sql,1433;Database=OtOpcUa;User Id=sa;Password=OtOpcUa!Dev123;TrustServerCertificate=True;"
|
||
Cluster__Hostname: "0.0.0.0"
|
||
Cluster__Port: "4053"
|
||
Cluster__PublicHostname: "site-b-1"
|
||
Cluster__SeedNodes__0: "akka.tcp://otopcua@central-1:4053"
|
||
Cluster__Roles__0: "driver"
|
||
<<: *secrets-env
|
||
Serilog__MinimumLevel__Override__Microsoft.EntityFrameworkCore: "Warning"
|
||
Serilog__MinimumLevel__Override__Microsoft.AspNetCore: "Warning"
|
||
GALAXY_MXGW_API_KEY: "${GALAXY_MXGW_API_KEY:-mxgw_otopcua2_GI7-tNozYE6cXGUSgEzL3AHDV7bYcYIHdMwKYgyHdX4}"
|
||
# site-b is the default-OFF pin: it gets the pair-local cache but NO SyncListenPort and NO
|
||
# replication config, so no sync listener binds and ISyncStatus stays disconnected/Healthy.
|
||
# This is what the live gate's check 8 verifies — the cache works locally with replication off.
|
||
# Alarm store-and-forward with replication OFF — the other half of the site-b pin. The sink
|
||
# must work as a plain node-local buffer here, with no sync listener and no peer traffic.
|
||
AlarmHistorian__Enabled: "true"
|
||
AlarmHistorian__MaxAttempts: "1000000"
|
||
ServerHistorian__Endpoint: "${OTOPCUA_HISTORIAN_ENDPOINT:-http://histgw-absent.invalid:5222}"
|
||
# Required whenever the section is consumed: the gateway client validates its own options at
|
||
# construction and throws on an empty key, so a keyless node crash-loops during Akka startup
|
||
# rather than degrading. DEV-ONLY placeholder — it authenticates nothing, because the endpoint
|
||
# above resolves nowhere. NEVER reuse this shape for a real gateway key; supply those via
|
||
# ServerHistorian__ApiKey from the environment.
|
||
ServerHistorian__ApiKey: "${OTOPCUA_HISTORIAN_APIKEY:-histgw_dev_rig-no-gateway-present}"
|
||
# Must agree with the endpoint scheme above — the client throws on a mismatch in EITHER
|
||
# direction, so an http endpoint with the default UseTls=true crash-loops the host.
|
||
ServerHistorian__UseTls: "false"
|
||
LocalDb__Path: "/app/data/otopcua-localdb.db"
|
||
ports:
|
||
- "4844:4840"
|
||
volumes:
|
||
- otopcua-localdb-site-b-1:/app/data
|
||
|
||
site-b-2:
|
||
<<: *otopcua-host
|
||
depends_on:
|
||
sql: { condition: service_healthy }
|
||
central-1: { condition: service_started }
|
||
migrator: { condition: service_completed_successfully }
|
||
environment:
|
||
OTOPCUA_ROLES: "driver"
|
||
ConnectionStrings__ConfigDb: "Server=sql,1433;Database=OtOpcUa;User Id=sa;Password=OtOpcUa!Dev123;TrustServerCertificate=True;"
|
||
Cluster__Hostname: "0.0.0.0"
|
||
Cluster__Port: "4053"
|
||
Cluster__PublicHostname: "site-b-2"
|
||
Cluster__SeedNodes__0: "akka.tcp://otopcua@central-1:4053"
|
||
Cluster__Roles__0: "driver"
|
||
<<: *secrets-env
|
||
Serilog__MinimumLevel__Override__Microsoft.EntityFrameworkCore: "Warning"
|
||
Serilog__MinimumLevel__Override__Microsoft.AspNetCore: "Warning"
|
||
GALAXY_MXGW_API_KEY: "${GALAXY_MXGW_API_KEY:-mxgw_otopcua2_GI7-tNozYE6cXGUSgEzL3AHDV7bYcYIHdMwKYgyHdX4}"
|
||
# site-b default-OFF pin (see site-b-1): cache on, replication off.
|
||
# Alarm store-and-forward with replication OFF — the other half of the site-b pin. The sink
|
||
# must work as a plain node-local buffer here, with no sync listener and no peer traffic.
|
||
AlarmHistorian__Enabled: "true"
|
||
AlarmHistorian__MaxAttempts: "1000000"
|
||
ServerHistorian__Endpoint: "${OTOPCUA_HISTORIAN_ENDPOINT:-http://histgw-absent.invalid:5222}"
|
||
# Required whenever the section is consumed: the gateway client validates its own options at
|
||
# construction and throws on an empty key, so a keyless node crash-loops during Akka startup
|
||
# rather than degrading. DEV-ONLY placeholder — it authenticates nothing, because the endpoint
|
||
# above resolves nowhere. NEVER reuse this shape for a real gateway key; supply those via
|
||
# ServerHistorian__ApiKey from the environment.
|
||
ServerHistorian__ApiKey: "${OTOPCUA_HISTORIAN_APIKEY:-histgw_dev_rig-no-gateway-present}"
|
||
# Must agree with the endpoint scheme above — the client throws on a mismatch in EITHER
|
||
# direction, so an http endpoint with the default UseTls=true crash-loops the host.
|
||
ServerHistorian__UseTls: "false"
|
||
LocalDb__Path: "/app/data/otopcua-localdb.db"
|
||
ports:
|
||
- "4845:4840"
|
||
volumes:
|
||
- otopcua-localdb-site-b-2:/app/data
|
||
|
||
traefik:
|
||
image: traefik:v3.1
|
||
command:
|
||
- --entrypoints.web.address=:80
|
||
- --providers.file.filename=/etc/traefik/dynamic.yml
|
||
- --providers.file.watch=true
|
||
- --api.insecure=true
|
||
ports:
|
||
- "9200:80" # host port 9200 → traefik :80 entrypoint (80 conflicts with scadabridge-traefik)
|
||
- "8089:8080" # 8080 conflicts with the sister scadalink dev stack
|
||
volumes:
|
||
- ./traefik-dynamic.yml:/etc/traefik/dynamic.yml:ro
|
||
depends_on:
|
||
- central-1
|
||
- central-2
|
||
|
||
volumes:
|
||
# SQL Server data dir — persists the OtOpcUa ConfigDb across container recreates.
|
||
otopcua-mssql-data:
|
||
# Per-node pair-local config-cache (ZB.MOM.WW.LocalDb) files. One volume per node — the
|
||
# cache is node-local, not shared; it persists the cached deployment artifact (and, on the
|
||
# site-a pair, the replicated oplog state) across container recreates so a node can boot
|
||
# from cache after a restart even with central SQL down.
|
||
otopcua-localdb-central-1:
|
||
otopcua-localdb-central-2:
|
||
otopcua-localdb-site-a-1:
|
||
otopcua-localdb-site-a-2:
|
||
otopcua-localdb-site-b-1:
|
||
otopcua-localdb-site-b-2:
|