From 8dd9da7d4d9a10ee0c0af1fb17ce7275d9b87e96 Mon Sep 17 00:00:00 2001 From: Joseph Doherty Date: Fri, 24 Jul 2026 11:30:50 -0400 Subject: [PATCH] =?UTF-8?q?fix(health):=20bump=20to=20ZB.MOM.WW.Health=200?= =?UTF-8?q?.2.1=20=E2=80=94=20Traefik=20leader-pinning=20now=20works?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit 0.2.1 makes the shared ActiveNodeHealthCheck return Unhealthy (503), not Degraded (200), for a node that carries the role but is not the role leader. That is what the shared health spec always specified, what docs/ServiceHosting.md, docs/v2/Architecture-v2.md and the 2026-05-26 alignment design all describe ("200 only on the Admin-role leader; 503 elsewhere"), and what docker-dev/traefik-dynamic.yml + scripts/install/traefik-dynamic.yml assume when they use /health/active as the load-balancer probe. The implementation returned Degraded, which MapZbHealth maps to 200, so BOTH central nodes always passed the probe and Traefik load-balanced the admin UI across the pair instead of pinning it to the leader. The docs were right; the code was wrong. Closes the Traefik half of #494. No OtOpcUa code change — package pin only. Live-verified on docker-dev, with the MAIN pair genuinely formed (memberCount 2, both members naming one leader): central-1 (leader) /health/active -> 200 traefik UP central-2 (standby) /health/active -> 503 traefik DOWN Failover drill (README step 2), which had never actually exercised anything: stopping central-1 flipped Traefik to central-2 within 20 s and http://localhost:9200/ answered 200 continuously across the swap — no outage. Restarting central-1 reclaimed the role-leader and Traefik flipped back. README corrected on two counts: the drill now states that exactly ONE backend is UP by design, and step 3 no longer claims central-2 keeps leadership — RoleLeader is the lowest-address member, so central-1 deterministically reclaims it. Added the cold-boot caveat that starting both nodes at once can leave each self-forming a 1-member cluster, in which case both are their own leader and both answer 200. STILL OPEN in #494: driver-only nodes. The check is scoped to the admin role and returns Healthy for any node that lacks it, so all four site nodes answer 200 and the real per-Cluster redundancy Primary (oldest Up driver member, IsDriverPrimary) is still not what this tier reports. --- Directory.Packages.props | 6 +++--- docker-dev/README.md | 8 +++++--- 2 files changed, 8 insertions(+), 6 deletions(-) diff --git a/Directory.Packages.props b/Directory.Packages.props index 9e7443a6..ccb9daa8 100644 --- a/Directory.Packages.props +++ b/Directory.Packages.props @@ -140,7 +140,7 @@ - + - - + + diff --git a/docker-dev/README.md b/docker-dev/README.md index 0b6f02b0..fa55c9b8 100644 --- a/docker-dev/README.md +++ b/docker-dev/README.md @@ -100,9 +100,11 @@ The `-v` drops the SQL volume; remove it to keep ConfigDb state across restarts. ## Failover smoke -1. Watch the Traefik dashboard at `http://localhost:8089`. Both `central-1` and `central-2` should be listed as healthy in the `otopcua-admin` service. -2. `docker compose -f docker-dev/docker-compose.yml stop central-1` — `central-2` should pick up the admin role-leader within ~15 s (Akka split-brain stable-after). Traefik will route traffic to `central-2` once its `/health/active` returns 200. -3. `docker compose -f docker-dev/docker-compose.yml start central-1` — `central-1` rejoins as a follower; `central-2` keeps the leader role until something disturbs it. +1. Watch the Traefik dashboard at `http://localhost:8089`. Exactly ONE of `central-1` / `central-2` is UP in the `otopcua-admin` service — the admin role-leader. The standby is DOWN by design: `/health/active` answers 503 on it, which is how Traefik pins browser traffic to the leader. (Before `ZB.MOM.WW.Health` 0.2.1 the standby answered 200, so both showed UP and this pinning silently never worked — see `lmxopcua#494`.) +2. `docker compose -f docker-dev/docker-compose.yml stop central-1` — `central-2` picks up the admin role-leader within ~15 s (Akka split-brain stable-after) and Traefik flips it to UP. Verified 2026-07-24: the swap completed inside 20 s with **no gap** — `http://localhost:9200/` answered 200 continuously across the failover. +3. `docker compose -f docker-dev/docker-compose.yml start central-1` — `central-1` rejoins and **reclaims** the admin role-leader, and Traefik flips back. It does not stay with `central-2`: Akka's `RoleLeader` is the lowest-address member, so the outcome is deterministic rather than sticky. + +> **Cold-boot caveat.** Starting both central nodes simultaneously from stopped can leave each self-forming its own 1-member cluster (self-first seed lists), in which case BOTH are their own role-leader and both answer `/health/active` 200. Check `entries["akka"].data.memberCount` on `/health/ready` — it should read 2. Restarting either node makes it join the survivor. ## Resource limits & dev logging