dohertj2/lmxopcua
1
0
Fork 0
Code Issues 12 Pull Requests 1 Actions Packages Projects Releases Wiki Activity

/health/active returns 200 on every node — the tier cannot identify the active node #494

New Issue
Closed
opened 2026-07-24 10:40:38 -04:00 by dohertj2 · 2 comments
dohertj2 commented 2026-07-24 10:40:38 -04:00
Owner
Copy Link
Copy Source

Summary

/health/active returns 200 on every node, including standbys and nodes that are not the redundancy Primary. A consumer cannot use it to tell which node of a pair is active, which is what the tier exists to answer.

Found during the live acceptance run for the family overview dashboard (scadaproj docs/plans/2026-07-22-overview-dashboard-impl-plan.md, Phase 4). The dashboard reads /health/active and maps 200 → Active, 503 → Standby — the mapping MapZbHealth implies, and the one ScadaBridge already satisfies. Against OtOpcUa it therefore shows all six rig nodes as Active.

Live evidence

Rig: docker-dev, six nodes, three healthy 2-node meshes (2 members · 0 unreachable, both members of each pair agreeing on one leader). Build = feat/mesh-phase6 + the Health 0.2.0 bump.

central-1 /health/active → 200   overall Healthy    admin-leader Healthy
central-2 /health/active → 200   overall Degraded   admin-leader Degraded
                                   "Role 'admin' member but not leader."
site-a-1  /health/active → 200   overall Healthy    admin-leader Healthy
site-a-2  /health/active → 200   overall Healthy    admin-leader Healthy
                                   "Active for role 'admin' (or not a role member)."
site-b-1  /health/active → 200   …same
site-b-2  /health/active → 200   …same

For contrast, ScadaBridge on the same rig host:

scadabridge-central-a → 200    scadabridge-central-b → 503
scadabridge-site-a-a  → 200    scadabridge-site-a-b  → 503

Why every node answers 200

Two independent causes, both in src/Server/ZB.MOM.WW.OtOpcUa.Host/Health/HealthEndpoints.cs:

.AddTypeActivatedCheck<ActiveNodeHealthCheck>(
    "admin-leader",
    failureStatus: null,
    tags: new[] { ZbHealthTags.Active },
    args: "admin")

1. On the standby admin, the check returns Degraded, and Degraded is a 200.
The shared ActiveNodeHealthCheck role-filtered mode maps "carries the role but is not the role leader" to HealthStatus.Degraded (ActiveNodeDecision.Evaluate). MapZbHealth uses the ASP.NET default status mapping — Healthy/Degraded → 200, Unhealthy → 503 — so central-2 is indistinguishable from central-1 over HTTP. Only the body differs.

2. On driver-only nodes the check passes vacuously.
Same Evaluate: if (!hasRole) return HealthStatus.Healthy. The four site nodes carry driver + cluster-SITE-* and no admin role, so they report Healthy with the description "Active for role 'admin' (or not a role member)". The tier says nothing at all about them.

Note this is the shared library behaving exactly as documented for its role-filtered constructor — this is about which question OtOpcUa asks it, not a bug in ZB.MOM.WW.Health.Akka.

The deeper mismatch

The check is scoped to RoleLeader("admin"). Per CLAUDE.md, the redundancy Primary is now the oldest Up driver member, per application Cluster (NodeRedundancyState.IsDriverPrimary, ServiceLevel 250 vs 240) — and the same doc records that oldest-Up and RoleLeader (lowest address) diverge after any node restart. So even for the central pair, admin-leader is not asking about the node whose ServiceLevel reads 250.

After Phase 6 there is one Primary per Cluster, so the tier should be able to answer per-cluster: exactly one 200 in MAIN, one in SITE-A, one in SITE-B.

Suggested direction (not prescriptive)

Make /health/active mean "this node is the redundancy Primary for its Cluster":

  • Register a check backed by IsDriverPrimary rather than RoleLeader("admin").
  • Return Unhealthy — not Degraded — on a non-primary node, so the tier is a 503. If the role-filtered shared mode is still wanted elsewhere, ZB.MOM.WW.Health also offers the role-less mode, which already returns Unhealthy for a non-leader (that is the mode ScadaBridge uses, and why it reads correctly).

Impact to check before changing this

docker-dev/traefik-dynamic.yml uses /health/active as its load-balancer health check for the central UI:

healthCheck:
  path: /health/active
  interval: 5s

Today both central nodes pass, so Traefik load-balances the AdminUI across the pair (with a sticky cookie). If a standby starts returning 503, Traefik will drop it from the pool and send all UI traffic to the primary. That may well be desirable — the deploy singleton lives on one node — but it is a behaviour change, and the same pattern may exist in scripts/install/traefik-dynamic.yml. Worth deciding deliberately rather than discovering.

Not blocking the dashboard

The overview dashboard renders OtOpcUa correctly in every other respect — leader chip per cluster group, member counts, no split-brain, per-check detail. Only the Active/Standby badge is wrong, and it is wrong because it faithfully reports what the endpoint says.

## Summary `/health/active` returns **200 on every node**, including standbys and nodes that are not the redundancy Primary. A consumer cannot use it to tell which node of a pair is active, which is what the tier exists to answer. Found during the live acceptance run for the family overview dashboard (`scadaproj docs/plans/2026-07-22-overview-dashboard-impl-plan.md`, Phase 4). The dashboard reads `/health/active` and maps **200 → Active, 503 → Standby** — the mapping `MapZbHealth` implies, and the one ScadaBridge already satisfies. Against OtOpcUa it therefore shows all six rig nodes as Active. ## Live evidence Rig: `docker-dev`, six nodes, three healthy 2-node meshes (`2 members · 0 unreachable`, both members of each pair agreeing on one leader). Build = `feat/mesh-phase6` + the Health 0.2.0 bump. ``` central-1 /health/active → 200 overall Healthy admin-leader Healthy central-2 /health/active → 200 overall Degraded admin-leader Degraded "Role 'admin' member but not leader." site-a-1 /health/active → 200 overall Healthy admin-leader Healthy site-a-2 /health/active → 200 overall Healthy admin-leader Healthy "Active for role 'admin' (or not a role member)." site-b-1 /health/active → 200 …same site-b-2 /health/active → 200 …same ``` For contrast, ScadaBridge on the same rig host: ``` scadabridge-central-a → 200 scadabridge-central-b → 503 scadabridge-site-a-a → 200 scadabridge-site-a-b → 503 ``` ## Why every node answers 200 Two independent causes, both in `src/Server/ZB.MOM.WW.OtOpcUa.Host/Health/HealthEndpoints.cs`: ```csharp .AddTypeActivatedCheck<ActiveNodeHealthCheck>( "admin-leader", failureStatus: null, tags: new[] { ZbHealthTags.Active }, args: "admin") ``` **1. On the standby admin, the check returns `Degraded`, and `Degraded` is a 200.** The shared `ActiveNodeHealthCheck` role-filtered mode maps "carries the role but is not the role leader" to `HealthStatus.Degraded` (`ActiveNodeDecision.Evaluate`). `MapZbHealth` uses the ASP.NET default status mapping — Healthy/Degraded → 200, Unhealthy → 503 — so `central-2` is indistinguishable from `central-1` over HTTP. Only the body differs. **2. On driver-only nodes the check passes vacuously.** Same `Evaluate`: `if (!hasRole) return HealthStatus.Healthy`. The four site nodes carry `driver` + `cluster-SITE-*` and no `admin` role, so they report Healthy with the description *"Active for role 'admin' (or not a role member)"*. The tier says nothing at all about them. Note this is the shared library behaving exactly as documented for its role-filtered constructor — this is about which question OtOpcUa asks it, not a bug in `ZB.MOM.WW.Health.Akka`. ## The deeper mismatch The check is scoped to `RoleLeader("admin")`. Per `CLAUDE.md`, the redundancy Primary is now **the oldest Up `driver` member, per application `Cluster`** (`NodeRedundancyState.IsDriverPrimary`, `ServiceLevel` 250 vs 240) — and the same doc records that oldest-Up and `RoleLeader` (lowest address) **diverge after any node restart**. So even for the central pair, `admin-leader` is not asking about the node whose `ServiceLevel` reads 250. After Phase 6 there is one Primary **per `Cluster`**, so the tier should be able to answer per-cluster: exactly one 200 in `MAIN`, one in `SITE-A`, one in `SITE-B`. ## Suggested direction (not prescriptive) Make `/health/active` mean "this node is the redundancy Primary for its `Cluster`": - Register a check backed by `IsDriverPrimary` rather than `RoleLeader("admin")`. - Return **`Unhealthy`** — not `Degraded` — on a non-primary node, so the tier is a 503. If the role-filtered shared mode is still wanted elsewhere, `ZB.MOM.WW.Health` also offers the role-less mode, which already returns Unhealthy for a non-leader (that is the mode ScadaBridge uses, and why it reads correctly). ### Impact to check before changing this `docker-dev/traefik-dynamic.yml` uses `/health/active` as its **load-balancer health check** for the central UI: ```yaml healthCheck: path: /health/active interval: 5s ``` Today both central nodes pass, so Traefik load-balances the AdminUI across the pair (with a sticky cookie). If a standby starts returning 503, Traefik will drop it from the pool and send all UI traffic to the primary. That may well be desirable — the deploy singleton lives on one node — but it is a behaviour change, and the same pattern may exist in `scripts/install/traefik-dynamic.yml`. Worth deciding deliberately rather than discovering. ## Not blocking the dashboard The overview dashboard renders OtOpcUa correctly in every other respect — leader chip per cluster group, member counts, no split-brain, per-check detail. Only the Active/Standby badge is wrong, and it is wrong because it faithfully reports what the endpoint says.
dohertj2 commented 2026-07-24 11:31:14 -04:00
Author
Owner
Copy Link
Copy Source

Traefik half fixed — ZB.MOM.WW.Health 0.2.1

Root cause was in the shared library, not here. ActiveNodeDecision.Evaluate returned Degraded for "carries the role but is not the role leader", and MapZbHealth maps Degraded to 200. The shared health spec has always said the active tier "Fails (503) on a standby or role-member-but-not-leader node" — so the library contradicted its own spec, and this repo's docs (docs/ServiceHosting.md, docs/v2/Architecture-v2.md, the 2026-05-26 alignment design's "200 only on the Admin-role leader; 503 elsewhere") were describing behaviour that never shipped.

Fixed in ZB.MOM.WW.Health 0.2.1 (published to the Gitea feed); this repo bumped in 8dd9da7d. Only OtOpcUa is affected — it is the sole consumer of the role-filtered mode.

Live verification on docker-dev

With the MAIN pair genuinely formed (memberCount 2, both members naming one leader):

central-1 (leader)   /health/active → 200    traefik: UP
central-2 (standby)  /health/active → 503    traefik: DOWN

The failover drill in docker-dev/README.md step 2 — which had never actually exercised anything, since both backends always passed — now works:

  • stopped central-1 at 15:27:41; Traefik had flipped to central-2 by 15:28:01 (< 20 s)
  • http://localhost:9200/ answered 200 continuously across the swap, no outage
  • restarting central-1 reclaimed the role-leader and Traefik flipped back

README corrected on two counts: exactly one backend is UP by design, and step 3's claim that "central-2 keeps the leader role" is wrong — RoleLeader is the lowest-address member, so central-1 deterministically reclaims it.

Still open in this issue

Driver-only nodes. The check is scoped to RoleLeader("admin") and returns Healthy for any node lacking the role, so all four site nodes still answer /health/active 200:

site-a-1 → 200   site-a-2 → 200
site-b-1 → 200   site-b-2 → 200

The real per-Cluster redundancy Primary is the oldest Up driver member (IsDriverPrimary, ServiceLevel 250 vs 240), which this tier never consults. Traefik doesn't route to site nodes so nothing is broken operationally, but any consumer selecting the active node by this tier — the overview dashboard does — still reads every site node as Active.

Cold-boot caveat worth recording

Starting both central nodes simultaneously from stopped can leave each self-forming its own 1-member cluster (self-first seed lists), in which case both ARE their own role-leader and both correctly answer 200. Check entries["akka"].data.memberCount on /health/ready before concluding the fix regressed. Restarting either node makes it join the survivor. Related: scadaproj docs/plans/2026-07-22-initjoin-selfform-fallback.md.

## Traefik half fixed — `ZB.MOM.WW.Health` 0.2.1 Root cause was in the shared library, not here. `ActiveNodeDecision.Evaluate` returned `Degraded` for "carries the role but is not the role leader", and `MapZbHealth` maps Degraded to **200**. The shared health spec has always said the active tier "Fails (503) on a standby or role-member-but-not-leader node" — so the library contradicted its own spec, and this repo's docs (`docs/ServiceHosting.md`, `docs/v2/Architecture-v2.md`, the 2026-05-26 alignment design's "200 only on the Admin-role leader; **503 elsewhere**") were describing behaviour that never shipped. Fixed in `ZB.MOM.WW.Health` **0.2.1** (published to the Gitea feed); this repo bumped in `8dd9da7d`. Only OtOpcUa is affected — it is the sole consumer of the role-filtered mode. ### Live verification on docker-dev With the MAIN pair genuinely formed (`memberCount` 2, both members naming one leader): ``` central-1 (leader) /health/active → 200 traefik: UP central-2 (standby) /health/active → 503 traefik: DOWN ``` The failover drill in `docker-dev/README.md` step 2 — which had never actually exercised anything, since both backends always passed — now works: - stopped `central-1` at 15:27:41; Traefik had flipped to `central-2` by 15:28:01 (< 20 s) - `http://localhost:9200/` answered **200 continuously** across the swap, no outage - restarting `central-1` reclaimed the role-leader and Traefik flipped back README corrected on two counts: exactly one backend is UP by design, and step 3's claim that "central-2 keeps the leader role" is wrong — `RoleLeader` is the lowest-address member, so `central-1` deterministically reclaims it. ### Still open in this issue **Driver-only nodes.** The check is scoped to `RoleLeader("admin")` and returns Healthy for any node lacking the role, so all four site nodes still answer `/health/active` 200: ``` site-a-1 → 200 site-a-2 → 200 site-b-1 → 200 site-b-2 → 200 ``` The real per-`Cluster` redundancy Primary is the oldest Up `driver` member (`IsDriverPrimary`, ServiceLevel 250 vs 240), which this tier never consults. Traefik doesn't route to site nodes so nothing is broken operationally, but any consumer selecting the active node by this tier — the overview dashboard does — still reads every site node as Active. ### Cold-boot caveat worth recording Starting both central nodes simultaneously from stopped can leave each self-forming its own 1-member cluster (self-first seed lists), in which case both ARE their own role-leader and both correctly answer 200. Check `entries["akka"].data.memberCount` on `/health/ready` before concluding the fix regressed. Restarting either node makes it join the survivor. Related: `scadaproj docs/plans/2026-07-22-initjoin-selfform-fallback.md`.
dohertj2 commented 2026-07-24 12:45:50 -04:00
Author
Owner
Copy Link
Copy Source

Closed — site nodes fixed, and the earlier fix corrected

45504861 replaces the shared ActiveNodeHealthCheck(role: "admin") with a local ClusterPrimaryHealthCheck (cluster-primary).

One rule: this node is active iff it is the oldest Up member carrying its own active role, where the active role is admin when the node has it and driver otherwise.

  • A fused admin node answers for admin → Traefik pins the AdminUI to the node hosting the cluster singletons.
  • A driver-only site node answers for driver → exactly SelectDriverPrimary, the same election behind IsDriverPrimary and the ServiceLevel 250/240 split. The tier and the ServiceLevel can no longer disagree.

Per-mesh scoping is free after Phase 6 — ClusterState.Members already contains only the node's own Cluster, so exactly one node answers 200 per Cluster by construction.

SelectDriverPrimary is generalised to SelectOldestUpMemberOfRole so the age-ordering rule has a single implementation.

The RoleLeader half was not theoretical

Worth recording, because the earlier fix (8dd9da7d, ZB.MOM.WW.Health 0.2.1) made the tier a real 503 but still selected by RoleLeader. On the rebuilt rig the two orderings diverge right now:

akka leader (lowest address) = central-1
oldest admin member          = central-2   ← hosts the singletons

So 0.2.1 alone would have pinned Traefik to central-1 — the node not hosting the work. Exactly the failure mode RedundancyStateActor.SelectDriverPrimary was written to avoid, reappearing in the health tier. With this change Traefik correctly routes to central-2.

Live verification

Exactly one 200 per mesh:

MAIN     central-2 → 200    central-1 → 503     traefik: central-2 UP, central-1 DOWN
SITE-A   site-a-1  → 200    site-a-2  → 503
SITE-B   site-b-1  → 200    site-b-2  → 503

cluster-primary on site-a-1:

{ "status": "Healthy",
  "description": "Primary for role 'driver' (oldest Up member).",
  "data": { "activeRole": "driver",
            "selfAddress": "akka.tcp://otopcua@site-a-1:4053",
            "primary": "akka.tcp://otopcua@site-a-1:4053" } }

The overview dashboard now renders one Active + one Standby for all three OtOpcUa Clusters.

Follow-up worth considering (not filed)

Two of the four family apps now need "oldest Up member of role X" and both had to avoid the shared ActiveNodeHealthCheck to get it — ScadaBridge wrote OldestNodeActiveHealthCheck for the same reason (leadership diverges from singleton host), and this repo now has ClusterPrimaryHealthCheck. That is a candidate for promotion into ZB.MOM.WW.Health.Akka as an oldest-member mode, retiring the RoleLeader-based one. Left alone here rather than designing a shared API off two samples.

## Closed — site nodes fixed, and the earlier fix corrected `45504861` replaces the shared `ActiveNodeHealthCheck(role: "admin")` with a local `ClusterPrimaryHealthCheck` (`cluster-primary`). **One rule:** this node is active iff it is the **oldest Up member carrying its own active role**, where the active role is `admin` when the node has it and `driver` otherwise. - A fused admin node answers for `admin` → Traefik pins the AdminUI to the node hosting the cluster singletons. - A driver-only site node answers for `driver` → exactly `SelectDriverPrimary`, the same election behind `IsDriverPrimary` and the ServiceLevel 250/240 split. The tier and the ServiceLevel can no longer disagree. Per-mesh scoping is free after Phase 6 — `ClusterState.Members` already contains only the node's own `Cluster`, so exactly one node answers 200 per `Cluster` by construction. `SelectDriverPrimary` is generalised to `SelectOldestUpMemberOfRole` so the age-ordering rule has a single implementation. ### The RoleLeader half was not theoretical Worth recording, because the earlier fix (`8dd9da7d`, `ZB.MOM.WW.Health` 0.2.1) made the tier a real 503 but **still selected by `RoleLeader`**. On the rebuilt rig the two orderings diverge right now: ``` akka leader (lowest address) = central-1 oldest admin member = central-2 ← hosts the singletons ``` So 0.2.1 alone would have pinned Traefik to `central-1` — the node *not* hosting the work. Exactly the failure mode `RedundancyStateActor.SelectDriverPrimary` was written to avoid, reappearing in the health tier. With this change Traefik correctly routes to `central-2`. ### Live verification Exactly one 200 per mesh: ``` MAIN central-2 → 200 central-1 → 503 traefik: central-2 UP, central-1 DOWN SITE-A site-a-1 → 200 site-a-2 → 503 SITE-B site-b-1 → 200 site-b-2 → 503 ``` `cluster-primary` on site-a-1: ```json { "status": "Healthy", "description": "Primary for role 'driver' (oldest Up member).", "data": { "activeRole": "driver", "selfAddress": "akka.tcp://otopcua@site-a-1:4053", "primary": "akka.tcp://otopcua@site-a-1:4053" } } ``` The overview dashboard now renders one Active + one Standby for all three OtOpcUa `Cluster`s. ### Follow-up worth considering (not filed) Two of the four family apps now need "oldest Up member of role X" and both had to avoid the shared `ActiveNodeHealthCheck` to get it — ScadaBridge wrote `OldestNodeActiveHealthCheck` for the same reason (leadership diverges from singleton host), and this repo now has `ClusterPrimaryHealthCheck`. That is a candidate for promotion into `ZB.MOM.WW.Health.Akka` as an oldest-member mode, retiring the `RoleLeader`-based one. Left alone here rather than designing a shared API off two samples.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
auto-managed
queue management
 cross-driver
queue management
 driver/abcip
queue management
 driver/ablegacy
queue management
 driver/focas
queue management
 driver/opcuaclient
queue management
 driver/s7
queue management
 driver/twincat
queue management
 phase/1
queue management
 phase/2
queue management
 phase/3
queue management
 phase/4
queue management
 phase/5
queue management
 phase/6
queue management
 queue/blocked
queue management
 queue/done
queue management
 queue/failed
queue management
 queue/in-progress
queue management
 queue/queued
queue management
No Label
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
dohertj2 (dohertj2)
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: dohertj2/lmxopcua#494
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?