dohertj2/lmxopcua
1
0
Fork 0
Code Issues 1 Pull Requests 1 Actions Packages Projects Releases Wiki Activity

feat(deploy): add site-a + site-b 2-node clusters to docker-dev

Browse Source 
Extends the docker-dev compose with two additional, fully-isolated Akka
clusters representing distinct sites. Each site is a 2-node fused
admin+driver cluster (OTOPCUA_ROLES=admin,driver on both nodes), backed
by its own ConfigDb database so configuration state stays separate from
the main cluster and from the other site.

Cluster isolation: the three meshes share the same Akka system name
"otopcua" and remoting port 4053 (inside each container's own network
namespace), but their seed-node lists are disjoint — main seeds at
admin-a, site-a seeds at site-a-1, site-b seeds at site-b-1 — so gossip
doesn't cross between them.

Layout:
  Main cluster   ConfigDb=OtOpcUa        admin-a, admin-b, driver-a, driver-b
  Site A         ConfigDb=OtOpcUa_SiteA  site-a-1, site-a-2 (fused admin+driver)
  Site B         ConfigDb=OtOpcUa_SiteB  site-b-1, site-b-2 (fused admin+driver)

OPC UA endpoints exposed on host ports 4840-4845. Admin UIs reachable
through Traefik via Host-header routing:
  http://localhost               → main cluster (PathPrefix default)
  http://site-a.localhost        → site A
  http://site-b.localhost        → site B

`*.localhost` auto-resolves on macOS; Linux users add the two hosts to
/etc/hosts (or rely on the resolver's RFC 6761 behaviour).
This commit is contained in:
Joseph Doherty
2026-05-26 13:59:23 -04:00
parent a1a7646b33
commit 961e09430a
3 changed files with 209 additions and 26 deletions
Download Patch File Download Diff File Expand all files Collapse all files

50
docker-dev/README.md
View File

@@ -1,20 +1,41 @@
# docker-dev # docker-dev
Mac-friendly four-node OtOpcUa fleet for manual UI exercise + integration smoke tests. Spins up an Akka cluster + SQL Server + OpenLDAP + Traefik in front of two admin nodes. Mac-friendly multi-cluster OtOpcUa fleet for manual UI exercise + integration smoke tests. Spins up **three isolated Akka clusters** + SQL Server + OpenLDAP + Traefik on the same Compose network. Each cluster has its own ConfigDb database and its own seed-node list, so Akka.Cluster gossip doesn't cross between them even though they share the same system name `otopcua`.
## Stack ## Stack
### Shared infrastructure
| Service | Role | Ports | | Service | Role | Ports |
|---|---|---| |---|---|---|
| `sql` | SQL Server 2022 (`ConfigDb` backing store) | host `14330` → container `1433` | | `sql` | SQL Server 2022 (hosts all per-cluster ConfigDb databases) | host `14330` → container `1433` |
| `ldap` | OpenLDAP with dev users `alice` / `bob` | host `3893` → container `1389` | | `ldap` | OpenLDAP with dev users `alice` / `bob` | host `3893` → container `1389` |
| `admin-a` | OtOpcUa.Host, `OTOPCUA_ROLES=admin`, cluster seed | internal `9000` | | `traefik` | Routes :80 by Host header / PathPrefix | host `80`, dashboard `8080` |
| `admin-b` | OtOpcUa.Host, `OTOPCUA_ROLES=admin`, joins admin-a | internal `9000` |
| `driver-a` | OtOpcUa.Host, `OTOPCUA_ROLES=driver` | host `4840` → container `4840` |
| `driver-b` | OtOpcUa.Host, `OTOPCUA_ROLES=driver` | host `4841` → container `4840` |
| `traefik` | Routes `:80` to whichever admin-* currently passes `/health/active` | host `80`, dashboard `8080` |
All six containers share an Akka cluster bound to port `4053` inside the Compose network. The Akka `PublicHostname` of each container matches its Compose service name; the seed-node list points at `admin-a` so the other three join via that. ### Main cluster — split admin/driver roles (ConfigDb: `OtOpcUa`)
| Service | Role | Ports |
|---|---|---|
| `admin-a` | `OTOPCUA_ROLES=admin`, cluster seed | internal `9000` |
| `admin-b` | `OTOPCUA_ROLES=admin`, joins admin-a | internal `9000` |
| `driver-a` | `OTOPCUA_ROLES=driver` | host `4840` → container `4840` |
| `driver-b` | `OTOPCUA_ROLES=driver` | host `4841` → container `4840` |
### Site A cluster — 2-node fused admin+driver (ConfigDb: `OtOpcUa_SiteA`)
| Service | Role | Ports |
|---|---|---|
| `site-a-1` | `OTOPCUA_ROLES=admin,driver`, cluster seed | host `4842` → container `4840` |
| `site-a-2` | `OTOPCUA_ROLES=admin,driver`, joins site-a-1 | host `4843` → container `4840` |
### Site B cluster — 2-node fused admin+driver (ConfigDb: `OtOpcUa_SiteB`)
| Service | Role | Ports |
|---|---|---|
| `site-b-1` | `OTOPCUA_ROLES=admin,driver`, cluster seed | host `4844` → container `4840` |
| `site-b-2` | `OTOPCUA_ROLES=admin,driver`, joins site-b-1 | host `4845` → container `4840` |
All containers bind Akka remoting to port `4053` inside their own network namespace; the `PublicHostname` of each matches its Compose service name. Cluster isolation is enforced purely by disjoint seed lists.
## Bring up ## Bring up
@@ -22,12 +43,16 @@ All six containers share an Akka cluster bound to port `4053` inside the Compose
# from the repo root # from the repo root
docker compose -f docker-dev/docker-compose.yml up -d --build docker compose -f docker-dev/docker-compose.yml up -d --build
# wait ~15 seconds for SQL to come up + the cluster to form # wait ~20 seconds for SQL to come up + all three clusters to form
open http://localhost # Blazor admin UI via Traefik open http://localhost # main cluster admin UI
open http://site-a.localhost # site A admin UI
open http://site-b.localhost # site B admin UI
open http://localhost:8080 # Traefik dashboard open http://localhost:8080 # Traefik dashboard
``` ```
On macOS, `*.localhost` resolves to `127.0.0.1` automatically. On Linux add `127.0.0.1 site-a.localhost site-b.localhost` to `/etc/hosts` if your resolver doesn't.
The first build takes a few minutes (.NET SDK image + restore + publish). Subsequent rebuilds are faster with Docker's layer cache. The first build takes a few minutes (.NET SDK image + restore + publish). Subsequent rebuilds are faster with Docker's layer cache.
## Auth (dev only) ## Auth (dev only)
@@ -58,5 +83,8 @@ The `-v` drops the SQL + LDAP volumes; remove it to keep ConfigDb state across r
## Notes ## Notes
- This compose is for the **local Mac/Linux developer rig**. The team's CI + soak runs go to the remote docker host at `10.100.0.35` (see `docs/v2/dev-environment.md`); the file there mirrors this one with adjusted port bindings. - This compose is for the **local Mac/Linux developer rig**. The team's CI + soak runs go to the remote docker host at `10.100.0.35` (see `docs/v2/dev-environment.md`); the file there mirrors this one with adjusted port bindings.
- The OPC UA driver endpoints (`opc.tcp://localhost:4840`, `opc.tcp://localhost:4841`) are reachable directly from the host — Traefik is only in front of the admin HTTP surface. - The OPC UA driver endpoints are reachable directly from the host (Traefik is only in front of the admin HTTP surface):
- Main: `opc.tcp://localhost:4840` (driver-a), `opc.tcp://localhost:4841` (driver-b)
- Site A: `opc.tcp://localhost:4842` (site-a-1), `opc.tcp://localhost:4843` (site-a-2)
- Site B: `opc.tcp://localhost:4844` (site-b-1), `opc.tcp://localhost:4845` (site-b-2)
- Galaxy + Wonderware drivers can't run in Linux containers (they need the Windows-only mxaccessgw + Historian SDK). On non-Windows, `DriverInstanceActor.ShouldStub(driverType, roles)` returns `true` for those types and the actor goes straight to a `Stubbed` state that returns deterministic success. - Galaxy + Wonderware drivers can't run in Linux containers (they need the Windows-only mxaccessgw + Historian SDK). On non-Windows, `DriverInstanceActor.ShouldStub(driverType, roles)` returns `true` for those types and the actor goes straight to a `Stubbed` state that returns deterministic success.

131
docker-dev/docker-compose.yml
View File

@@ -1,17 +1,36 @@
# docker-dev/ — Mac-friendly four-node fleet for v2 development + manual UI exercise. # docker-dev/ — Mac-friendly multi-cluster fleet for v2 development + manual UI exercise.
# #
# Stack: # Stack (3 separate Akka clusters sharing the same SQL + LDAP):
# sql SQL Server 2022 (ConfigDb backing store) # sql SQL Server 2022 (per-cluster ConfigDb databases)
# ldap OpenLDAP with the dev users from C:\publish\glauth\auth.md mirrored in # ldap OpenLDAP with the dev users from C:\publish\glauth\auth.md mirrored in
# admin-a OtOpcUa.Host with OTOPCUA_ROLES=admin (cluster seed) #
# Main cluster (existing — split-role admin / driver pair on a single Akka mesh):
# admin-a OtOpcUa.Host with OTOPCUA_ROLES=admin (seed)
# admin-b OtOpcUa.Host with OTOPCUA_ROLES=admin (joins admin-a) # admin-b OtOpcUa.Host with OTOPCUA_ROLES=admin (joins admin-a)
# driver-a OtOpcUa.Host with OTOPCUA_ROLES=driver (joins via admin-a) # driver-a OtOpcUa.Host with OTOPCUA_ROLES=driver (joins via admin-a)
# driver-b OtOpcUa.Host with OTOPCUA_ROLES=driver (joins via admin-a) # driver-b OtOpcUa.Host with OTOPCUA_ROLES=driver (joins via admin-a)
# traefik Routes :80 to whichever admin-* currently passes /health/active # ConfigDb: OtOpcUa
#
# Site A cluster (2-node fused admin+driver — its own ConfigDb + seed):
# site-a-1, site-a-2 OTOPCUA_ROLES=admin,driver, seed = site-a-1
# ConfigDb: OtOpcUa_SiteA
#
# Site B cluster (2-node fused admin+driver — its own ConfigDb + seed):
# site-b-1, site-b-2 OTOPCUA_ROLES=admin,driver, seed = site-b-1
# ConfigDb: OtOpcUa_SiteB
#
# traefik PathPrefix → main cluster admin-a/admin-b; Host(`site-a.localhost`) →
# site-a-*; Host(`site-b.localhost`) → site-b-*. Add the two site hosts to
# your /etc/hosts (or rely on macOS `.localhost` auto-resolution).
#
# Cluster isolation is enforced by disjoint seed-node lists — Akka.Cluster gossip won't
# cross between the three meshes even though they share the same system name "otopcua".
# #
# Usage: # Usage:
# docker compose -f docker-dev/docker-compose.yml up -d --build # docker compose -f docker-dev/docker-compose.yml up -d --build
# open http://localhost # Blazor admin UI via Traefik # open http://localhost # main cluster Blazor admin UI
# open http://site-a.localhost # site A admin UI
# open http://site-b.localhost # site B admin UI
# open http://localhost:8080 # Traefik dashboard # open http://localhost:8080 # Traefik dashboard
# #
# Tear-down: docker compose -f docker-dev/docker-compose.yml down -v # Tear-down: docker compose -f docker-dev/docker-compose.yml down -v
@@ -113,6 +132,102 @@ services:
ports: ports:
- "4841:4840" - "4841:4840"
# ── Site A cluster (2-node fused admin+driver) ──────────────────────────────
# Its own ConfigDb (OtOpcUa_SiteA) + its own seed (site-a-1) → Akka isolation
# from the main cluster and from site B. Both nodes carry both roles.
site-a-1:
<<: *otopcua-host
environment:
OTOPCUA_ROLES: "admin,driver"
ASPNETCORE_URLS: "http://+:9000"
ConnectionStrings__ConfigDb: "Server=sql,1433;Database=OtOpcUa_SiteA;User Id=sa;Password=OtOpcUa!Dev123;TrustServerCertificate=True;"
Cluster__Hostname: "0.0.0.0"
Cluster__Port: "4053"
Cluster__PublicHostname: "site-a-1"
Cluster__SeedNodes__0: "akka.tcp://otopcua@site-a-1:4053"
Cluster__Roles__0: "admin"
Cluster__Roles__1: "driver"
Security__Jwt__SigningKey: "docker-dev-signing-key-with-at-least-32-bytes-of-utf8-content-12345"
Security__Jwt__Issuer: "otopcua-dev"
Security__Jwt__Audience: "otopcua-dev"
Authentication__Ldap__Server: "ldap"
Authentication__Ldap__Port: "1389"
Authentication__Ldap__AllowInsecureLdap: "true"
ports:
- "4842:4840"
site-a-2:
<<: *otopcua-host
depends_on:
sql: { condition: service_healthy }
site-a-1: { condition: service_started }
environment:
OTOPCUA_ROLES: "admin,driver"
ASPNETCORE_URLS: "http://+:9000"
ConnectionStrings__ConfigDb: "Server=sql,1433;Database=OtOpcUa_SiteA;User Id=sa;Password=OtOpcUa!Dev123;TrustServerCertificate=True;"
Cluster__Hostname: "0.0.0.0"
Cluster__Port: "4053"
Cluster__PublicHostname: "site-a-2"
Cluster__SeedNodes__0: "akka.tcp://otopcua@site-a-1:4053"
Cluster__Roles__0: "admin"
Cluster__Roles__1: "driver"
Security__Jwt__SigningKey: "docker-dev-signing-key-with-at-least-32-bytes-of-utf8-content-12345"
Security__Jwt__Issuer: "otopcua-dev"
Security__Jwt__Audience: "otopcua-dev"
Authentication__Ldap__Server: "ldap"
Authentication__Ldap__Port: "1389"
Authentication__Ldap__AllowInsecureLdap: "true"
ports:
- "4843:4840"
# ── Site B cluster (2-node fused admin+driver) ──────────────────────────────
site-b-1:
<<: *otopcua-host
environment:
OTOPCUA_ROLES: "admin,driver"
ASPNETCORE_URLS: "http://+:9000"
ConnectionStrings__ConfigDb: "Server=sql,1433;Database=OtOpcUa_SiteB;User Id=sa;Password=OtOpcUa!Dev123;TrustServerCertificate=True;"
Cluster__Hostname: "0.0.0.0"
Cluster__Port: "4053"
Cluster__PublicHostname: "site-b-1"
Cluster__SeedNodes__0: "akka.tcp://otopcua@site-b-1:4053"
Cluster__Roles__0: "admin"
Cluster__Roles__1: "driver"
Security__Jwt__SigningKey: "docker-dev-signing-key-with-at-least-32-bytes-of-utf8-content-12345"
Security__Jwt__Issuer: "otopcua-dev"
Security__Jwt__Audience: "otopcua-dev"
Authentication__Ldap__Server: "ldap"
Authentication__Ldap__Port: "1389"
Authentication__Ldap__AllowInsecureLdap: "true"
ports:
- "4844:4840"
site-b-2:
<<: *otopcua-host
depends_on:
sql: { condition: service_healthy }
site-b-1: { condition: service_started }
environment:
OTOPCUA_ROLES: "admin,driver"
ASPNETCORE_URLS: "http://+:9000"
ConnectionStrings__ConfigDb: "Server=sql,1433;Database=OtOpcUa_SiteB;User Id=sa;Password=OtOpcUa!Dev123;TrustServerCertificate=True;"
Cluster__Hostname: "0.0.0.0"
Cluster__Port: "4053"
Cluster__PublicHostname: "site-b-2"
Cluster__SeedNodes__0: "akka.tcp://otopcua@site-b-1:4053"
Cluster__Roles__0: "admin"
Cluster__Roles__1: "driver"
Security__Jwt__SigningKey: "docker-dev-signing-key-with-at-least-32-bytes-of-utf8-content-12345"
Security__Jwt__Issuer: "otopcua-dev"
Security__Jwt__Audience: "otopcua-dev"
Authentication__Ldap__Server: "ldap"
Authentication__Ldap__Port: "1389"
Authentication__Ldap__AllowInsecureLdap: "true"
ports:
- "4845:4840"
traefik: traefik:
image: traefik:v3.1 image: traefik:v3.1
command: command:
@@ -128,3 +243,7 @@ services:
depends_on: depends_on:
- admin-a - admin-a
- admin-b - admin-b
- site-a-1
- site-a-2
- site-b-1
- site-b-2

42
docker-dev/traefik-dynamic.yml
View File

@@ -1,6 +1,12 @@
# docker-dev companion to scripts/install/traefik-dynamic.yml. Same routing rules, # docker-dev companion to scripts/install/traefik-dynamic.yml. Routes three
# but the upstream targets are the Compose service names (admin-a, admin-b) on # Akka clusters that share the Compose network:
# port 9000 instead of the Windows hostnames a bare-metal deployment would use. #
# - Main cluster (default): PathPrefix(`/`) → admin-a / admin-b.
# - Site A cluster: Host(`site-a.localhost`) → site-a-1 / site-a-2.
# - Site B cluster: Host(`site-b.localhost`) → site-b-1 / site-b-2.
#
# Host-header rules are more specific than PathPrefix, so they win over the
# default router for the site hostnames automatically — no priority field needed.
http: http:
routers: routers:
@@ -9,6 +15,16 @@ http:
rule: "PathPrefix(`/`)" rule: "PathPrefix(`/`)"
service: otopcua-admin service: otopcua-admin
otopcua-site-a:
entryPoints: ["web"]
rule: "Host(`site-a.localhost`)"
service: otopcua-site-a
otopcua-site-b:
entryPoints: ["web"]
rule: "Host(`site-b.localhost`)"
service: otopcua-site-b
services: services:
otopcua-admin: otopcua-admin:
loadBalancer: loadBalancer:
@@ -19,3 +35,23 @@ http:
path: /health/active path: /health/active
interval: 5s interval: 5s
timeout: 2s timeout: 2s
otopcua-site-a:
loadBalancer:
servers:
- url: "http://site-a-1:9000"
- url: "http://site-a-2:9000"
healthCheck:
path: /health/active
interval: 5s
timeout: 2s
otopcua-site-b:
loadBalancer:
servers:
- url: "http://site-b-1:9000"
- url: "http://site-b-2:9000"
healthCheck:
path: /health/active
interval: 5s
timeout: 2s