lmxopcua/docker-dev/README.md

# docker-dev

Mac-friendly OtOpcUa fleet for manual UI exercise + integration smoke tests. Spins up **one single Akka mesh** (hub-and-spoke topology) + SQL Server + Traefik on the same Compose network. All six host nodes share the single `OtOpcUa` ConfigDb — logical separation between MAIN, SITE-A, and SITE-B is enforced by per-row `ServerCluster.ClusterId` scoping, not by mesh isolation.

## Stack

### Shared infrastructure

| Service | Role | Ports |
|---|---|---|
| `sql` | SQL Server 2022 — single `OtOpcUa` ConfigDb shared by all nodes | host `14330` → container `1433` |
| `traefik` | Routes `:80` by PathPrefix to central admin nodes | host `80`, dashboard `8089` |

Authentication uses the **shared GLAuth** on the Linux Docker host at `10.100.0.35:3893` (baseDN `dc=zb,dc=local`). Only the central admin nodes authenticate users. Sign in as `multi-role` / `password` to get all three OtOpcUa roles (Administrator, Designer, Viewer), or use any other shared test user with password `password`. Group→role mappings are seeded by `seed/seed-clusters.sql` (`OtOpcUa-Admins`→Administrator, `OtOpcUa-Designers`→Designer, `OtOpcUa-Viewers`→Viewer). The shared GLAuth source of truth and deploy runbook live in `scadaproj/infra/glauth/`.

### Central nodes — fused admin+driver (MAIN cluster, UI + deploy singleton)

| Service | Roles | Ports |
|---|---|---|
| `central-1` | `OTOPCUA_ROLES=admin,driver`, Akka mesh seed | host `4840` → container `4840`; internal `9000` |
| `central-2` | `OTOPCUA_ROLES=admin,driver`, joins central-1 | host `4841` → container `4840`; internal `9000` |

`central-1` and `central-2` are the **only** nodes that host the Admin UI and the deploy singleton. They are also the OPC UA publishers for the MAIN cluster. Traefik routes all `PathPrefix(/)` traffic to whichever central node has the leader role.

### Site A nodes — driver-only (SITE-A cluster)

| Service | Roles | Ports |
|---|---|---|
| `site-a-1` | `OTOPCUA_ROLES=driver`, joins the single mesh | host `4842` → container `4840` |
| `site-a-2` | `OTOPCUA_ROLES=driver` **+** a deliberately miscased `Cluster__Roles__0=Driver` (see below) | host `4843` → container `4840` |

### Site B nodes — driver-only (SITE-B cluster)

| Service | Roles | Ports |
|---|---|---|
| `site-b-1` | `OTOPCUA_ROLES=driver`, joins the single mesh | host `4844` → container `4840` |
| `site-b-2` | **`Cluster:Roles`-ONLY** — no `OTOPCUA_ROLES` at all (see below) | host `4845` → container `4840` |

Site nodes serve no UI and authenticate no users. The central cluster manages and deploys to them over the shared Akka mesh. All six nodes bind Akka remoting to port `4053` inside their own network namespace; `PublicHostname` for each matches its Compose service name.

### Role-authority standing pins (archreview R3-04)

Every node above sets **both** `OTOPCUA_ROLES` and the matching `Cluster__Roles__*` env vars,
consistently — which is exactly why neither R3-04 finding (01/S-12, 01/C-9) could reproduce on this
rig as originally configured. Two nodes deliberately deviate, as standing pins:

- **`site-b-2` is `Cluster:Roles`-only.** No `OTOPCUA_ROLES` is set at all; `Cluster__Roles__0=driver`
  / `Cluster__Roles__1=cluster-SITE-B` are its sole role source. **Expected to FAIL every deploy on a
  pre-R3-04 build** (`Program.cs` read `OTOPCUA_ROLES` only, so this node's host-side conditional
  wiring mounted no driver stack at all — no LocalDb, no OPC UA endpoint, no ConfigDb, the deny-all
  `NullOpcUaUserAuthenticator` — while its Akka gossip still looked healthy) — the exact outage
  01/S-12 describes. Post-fix it behaves identically to `site-b-1`.
- **`site-a-2` pins a miscased `Cluster__Roles__0=Driver`** (capital D; `cluster-SITE-A` keeps its
  own spelling verbatim, by design). Post-fix `EffectiveRoles` normalizes it silently and this node
  behaves identically to `site-a-1`. Pre-fix, all four role-scoped validators (`SplitTopologyTransportValidator`
  and three siblings) compared roles case-sensitively, so three of the four silently skipped every
  driver-scoped boot-time rule for this node, and `site-a` elected **no Primary at all**
  (`RedundancyStateActor.SelectDriverPrimary` matches roles ordinally against gossip).

## Multi-tenancy

All six host nodes write to the same `OtOpcUa` ConfigDb. The `ServerCluster` table differentiates the three logical clusters: each maps to one row, and each `ClusterNode` row's `ClusterId` ties the runtime node back to its owning cluster scope.

Two one-shot Compose services bootstrap the DB on bring-up: `migrator` applies the EF Core migrations (so a fresh SQL volume gets the schema with no operator step — the host nodes deliberately do **not** auto-migrate, since production owns schema changes), then `cluster-seed` (image `mcr.microsoft.com/mssql-tools`) INSERTs the rows below. `cluster-seed` and every host node `depend_on` the `migrator` completing (`service_completed_successfully`), so the seed never races an in-progress migration. The seed is **idempotent** — `IF NOT EXISTS` guards every insert — so re-runs on `docker compose up` are no-ops:

| Logical cluster | `ServerCluster.ClusterId` | `ClusterNode.NodeId` rows |
|---|---|---|
| Main | `MAIN` | `central-1`, `central-2` (OPC UA publishers + admin UI) |
| Site A | `SITE-A` | `site-a-1`, `site-a-2` |
| Site B | `SITE-B` | `site-b-1`, `site-b-2` |

Each `ClusterNode.NodeId` matches the node's `Cluster__PublicHostname` env value (Compose service name) — that's the lookup the runtime uses to resolve its own membership. `ApplicationUri` follows the `urn:OtOpcUa:<NodeId>` convention.

The SQL lives at `seed/seed-clusters.sql`; the wait-and-apply wrapper lives at `seed/entrypoint.sh`. To re-seed manually:

```bash
docker compose -f docker-dev/docker-compose.yml run --rm cluster-seed
```

### Galaxy / MxAccess gateway

The seed also pre-creates a `SystemPlatform` Namespace + a `GalaxyMxGateway` DriverInstance in the MAIN cluster pointing at `http://10.100.0.48:5120`. The API key is resolved from the `GALAXY_MXGW_API_KEY` env var set on every driver-role container in compose; override via `GALAXY_MXGW_API_KEY=... docker compose up -d` to swap keys without editing the compose file.

The DriverHost actor doesn't spawn drivers from raw DriverInstance rows on its own — the v2 deploy lifecycle requires a *sealed Deployment* before drivers materialise. After first bring-up, sign in to the Admin UI and click **Deploy current configuration** on `/deployments` to compose the seeded rows into an artifact and dispatch it. The Galaxy driver instance will start its gRPC connection to the gateway on the next deploy ack.

## Bring up

```bash
# from the repo root
docker compose -f docker-dev/docker-compose.yml up -d --build

# the one-shot migrator + cluster-seed bootstrap the DB; watch the seed finish:
docker compose -f docker-dev/docker-compose.yml logs -f cluster-seed   # ^C once it prints "[cluster-seed] done."

open http://localhost:9200                 # Admin UI (Traefik → central-1 or central-2)
open http://localhost:8089                 # Traefik dashboard
```

The first build takes a few minutes (.NET SDK image + restore + publish). **No manual schema step is needed** — on a fresh SQL volume the one-shot `migrator` service applies the EF migrations (the host nodes deliberately don't auto-migrate, since production owns schema changes), then `cluster-seed` populates the cluster/namespace/driver rows. `cluster-seed` and the host nodes wait for the migrator via `service_completed_successfully`, so nothing races an in-progress migration. A plain `docker compose ... up -d` on an existing volume is a fast no-op for both — the named SQL volume keeps the schema + rows across restarts; only `down -v` wipes them, after which the next `up` re-migrates + re-seeds automatically.

## Auth (dev only)

Central nodes authenticate against the shared GLAuth at `10.100.0.35:3893` (baseDN `dc=zb,dc=local`). `DevStubMode` is **not** active. Sign in with any test user (password `password`); `multi-role` / `password` returns all three roles (Administrator, Designer, Viewer). Group→role mappings are seeded by `seed/seed-clusters.sql`. The GLAuth source of truth + deploy runbook is in `scadaproj/infra/glauth/`. **Do not** enable `DevStubMode` outside local debugging — production must always bind a real LDAP backend.

The LDAP service-account bind password is read from the host env var `OTOPCUA_LDAP_BIND_PASSWORD` (compose default `serviceaccount123`, the committed dev password of the shared GLAuth, so the rig works with nothing set). Export it before `docker compose up` to bind with a different password (#619).

## Headless deploy

```bash
./docker-dev/deploy.sh                  # trigger a deploy (no request body)
./docker-dev/deploy.sh artifact.json    # …sending that JSON file as the request body
```

`deploy.sh` polls `http://localhost:9200/health/ready` (every 2 s, up to 120 s) and only then
issues the single, un-retried `POST http://localhost:9200/api/deployments` with
`X-Api-Key: docker-dev-deploy-key`. Override the URL / key / timeout via `OTOPCUA_ADMIN_URL`,
`OTOPCUA_DEPLOY_API_KEY`, `OTOPCUA_READY_TIMEOUT_SECONDS`.

The body is optional and the no-body form is fully supported (verified on this rig 2026-08-18):
the minimal-API `DeployRequest?` parameter binds a missing body *and* a `Content-Length: 0` body
as **null** — it does not 400 — and the handler defaults the audit field to
`CreatedBy = "rest-api"`. A bare `./docker-dev/deploy.sh` answers `202 Accepted` with the new
`deploymentId` + `revisionHash`, `200` when there is nothing to deploy, and `409` while another
deployment is still in flight.

### First-POST 404 — mechanism CONFIRMED (2026-08-18)

The first headless POST after `up -d` was seen answering **404** ~40 s into boot (2026-08-17 live
gate), clearing on retry. Neither end can produce it: the AdminUI maps every route synchronously
before `RunAsync` (`Program.cs:740` maps the deploy API, `:803` runs) and that endpoint's only
failure modes are 503 / 401 / 422 / 409 / 500 — it has no 404 path; and Traefik answers a *matched*
router whose servers are all down with **503 "no available server"** (the checked-in
`traefik-dynamic.yml` router is `PathPrefix(/)` → `otopcua-admin` with a `/health/active`
healthCheck). Traefik's `api@internal` router (`PathPrefix(/api)`) is bound to the `:8080`
entrypoint (host 8089), not `web`/`:9200`, so it does not shadow `/api/deployments` either.

**The mechanism is the window between Traefik binding the `web` entrypoint and loading the
dynamic file-provider config** (`--providers.file.filename=/etc/traefik/dynamic.yml`). The
entrypoint accepts connections *before* the router table exists, and until it does, every request
gets Go's default `http.NotFound` handler. Reproduced 2026-08-18 by rapid-polling `:9200` across a
`docker restart otopcua-dev-traefik-1`: the port goes from connection-refused straight to **404 for
~65–160 ms**, then 200. The response body is exactly `404 page not found` (19 bytes), and the
access log — with the `--accesslog=true --accesslog.fields.defaultmode=keep` flags this service
carries for precisely this purpose — records it as:

```
192.168.97.1 - - [18/Aug/2026:11:58:26 +0000] "GET /health/ready HTTP/1.1" 404 19 "-" "-" 1 "-" "-" 0ms
```

**Empty `RouterName`, empty `ServiceName`, `0ms`** — nothing matched and no backend was dialled.
The competing candidate is **excluded**: a healthCheck'd service with no healthy server yet would
be a *matched* router (`otopcua-admin@file`) answering **503**, and zero 503s appear anywhere in
the log. Note the window has two sub-phases — the earliest 404s are not access-logged at all
(the accesslog handler is itself part of the not-yet-built dynamic chain), later ones are; both are
the same client-visible 404, so **absence from the access log does not mean absence of the 404**.

The readiness gate in `deploy.sh` closes the race (it loops on any non-200, and a 200 through
`:9200` proves both a matched router and a live backend), so a 404 *after* the gate is new evidence
and the script fails loudly saying so. A plain `docker compose up -d` also hides the window on this
rig without the gate helping: `up -d` blocks on the `depends_on: service_healthy` conditions and
returns ~30 s *after* Traefik starts, by which time the router table is long loaded.

## QuestDB historian overlay

`docker-compose.questdb.yml` is an **optional overlay** (not part of the default rig) that adds one service, `questdb`
(`questdb/questdb:10.0.1`, container `otopcua-dev-questdb-1`, named volume `otopcua-questdb-data`), for the `ServerHistorian:Backend=QuestDb`
historian ([`docs/Historian.md`](../docs/Historian.md#questdb-backend)). Host ports: PG wire **`18812`** (→ 8812) and the web console / REST **`19000`**
(→ 9000), chosen so they do not collide with AdminUI `:9000`/`:9200`.

```bash
# Bring up ONLY questdb (--no-deps: nothing else is touched):
docker compose -f docker-dev/docker-compose.yml -f docker-dev/docker-compose.questdb.yml up -d --no-deps questdb

# Point the central nodes at it (the QuestDb connection string already defaults to the `questdb` service;
# override with OTOPCUA_HISTORIAN_QUESTDB). Keep the overlay -f flag on every compose command for these nodes:
OTOPCUA_HISTORIAN_ENABLED=true OTOPCUA_HISTORIAN_BACKEND=QuestDb OTOPCUA_CONTHIST_ENABLED=true \
  docker compose -f docker-dev/docker-compose.yml -f docker-dev/docker-compose.questdb.yml up -d central-1 central-2

# Inspect from the host (REST, no client needed):
curl -G localhost:19000/exec --data-urlencode "query=SELECT tag, count() n FROM hist_sample WHERE NOT deleted GROUP BY tag"

# Integration tests (skipped when the variable is absent):
export HISTORIAN_QUESTDB_TEST_CONNECTION='Host=localhost;Port=18812;Username=admin;Password=quest;Database=qdb;ServerCompatibilityMode=NoTypeLoading'
dotnet test tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb.IntegrationTests
```

Tear down with `docker compose -f docker-dev/docker-compose.yml -f docker-dev/docker-compose.questdb.yml rm -sf questdb` (add `docker volume rm otopcua-questdb-data`
to drop the data). The `admin`/`quest` credentials are QuestDB's stock defaults, not secrets.

## Tear down

```bash
docker compose -f docker-dev/docker-compose.yml down -v
```

The `-v` drops the SQL volume; remove it to keep ConfigDb state across restarts. There is no local LDAP volume — LDAP is the shared external GLAuth on `10.100.0.35:3893`.

## Failover smoke

1. Watch the Traefik dashboard at `http://localhost:8089`. Exactly ONE of `central-1` / `central-2` is UP in the `otopcua-admin` service — the admin role-leader. The standby is DOWN by design: `/health/active` answers 503 on it, which is how Traefik pins browser traffic to the leader. (Before `ZB.MOM.WW.Health` 0.2.1 the standby answered 200, so both showed UP and this pinning silently never worked — see `lmxopcua#494`.)
2. `docker compose -f docker-dev/docker-compose.yml stop central-1` — `central-2` picks up the admin role-leader within ~15 s (Akka split-brain stable-after) and Traefik flips it to UP. Verified 2026-07-24: the swap completed inside 20 s with **no gap** — `http://localhost:9200/` answered 200 continuously across the failover.
3. `docker compose -f docker-dev/docker-compose.yml start central-1` — `central-1` rejoins and **reclaims** the admin role-leader, and Traefik flips back. It does not stay with `central-2`: Akka's `RoleLeader` is the lowest-address member, so the outcome is deterministic rather than sticky.

> **Cold-boot caveat.** Starting both central nodes simultaneously from stopped can leave each self-forming its own 1-member cluster (self-first seed lists), in which case BOTH are their own role-leader and both answer `/health/active` 200. Check `entries["akka"].data.memberCount` on `/health/ready` — it should read 2. Restarting either node makes it join the survivor.

## Resource limits & dev logging

The full single-mesh stack (`central-1`/`central-2` + the four site nodes) can OOM-kill `central-1` on a loaded host. Two settings in the compose file guard against that:

- **EF Core + ASP.NET Core logs are pinned to `Warning`** on every host node (`Serilog__MinimumLevel__Override__Microsoft.EntityFrameworkCore` / `…Microsoft.AspNetCore` = `Warning`). The host logs via Serilog (`AddZbSerilog` → `ReadFrom.Configuration`), and in `Development` the default level is `Debug` — without these overrides every Deployment-poll emits an `Executed DbCommand` / `SELECT … FROM [Deployment]` line, flooding the Serilog pipeline and starving the Akka cluster heartbeat thread. Application + Akka log levels are left untouched, so this only silences the per-poll SQL chatter. To temporarily restore the SQL log flood for debugging, drop those two env vars (or set them back to `Information`) on the node you're inspecting.
- **Each host node has `mem_limit: 1g`** (`mem_reservation: 512m`). A quiet solo `central-1` measures ~357 MiB; the limit leaves headroom for the deploy/UI load and per-cluster driver subscriptions that push a fully-loaded node higher. The limit/reservation live on the `&otopcua-host` anchor, so all six host services inherit them; `sql`, `traefik`, and the one-shot `migrator`/`cluster-seed` are left unbounded.

The full six-node host stack therefore needs roughly **6 GiB** of Docker Desktop VM memory just for the host nodes (plus SQL Server's own footprint on top). On a constrained host, either raise the Docker Desktop VM memory or run fewer host services (e.g. just `central-1` + `central-2`, or a single central node) rather than the full mesh.

## Notes

- This compose is for the **local Mac/Linux developer rig**. The team's CI + soak runs go to the remote docker host at `10.100.0.35` (see `docs/v2/dev-environment.md`); the file there mirrors this one with adjusted port bindings.
- The OPC UA endpoints are reachable directly from the host (Traefik is only in front of the admin HTTP surface):
  - Main: `opc.tcp://localhost:4840` (central-1), `opc.tcp://localhost:4841` (central-2)
  - Site A: `opc.tcp://localhost:4842` (site-a-1), `opc.tcp://localhost:4843` (site-a-2)
  - Site B: `opc.tcp://localhost:4844` (site-b-1), `opc.tcp://localhost:4845` (site-b-2)
- Galaxy + Wonderware drivers can't run in Linux containers (they need the Windows-only mxaccessgw + Historian SDK). On non-Windows, `DriverInstanceActor.ShouldStub(driverType, roles)` returns `true` for those types and the actor goes straight to a `Stubbed` state that returns deterministic success.
- SQL persistence: ConfigDb state survives container restarts (named Docker volume). Drop the volume with `down -v` for a clean slate.