docs: Health 0.2.1 active-tier fix — index row + Phase 4 ledger
This commit is contained in:
@@ -143,7 +143,7 @@ each project's **code-verified current state**, and the **gaps** between. See
|
|||||||
|---|---|---|---|---|
|
|---|---|---|---|---|
|
||||||
| Auth (login / identity / authz) | Adopted (lib `0.1.4`; all 3 apps, merged to **local default** main/master + **pushed to origin** (gitea)) | Shared `ZB.MOM.WW.Auth` lib | [`components/auth/`](components/auth/) | [`ZB.MOM.WW.Auth/`](ZB.MOM.WW.Auth/) |
|
| Auth (login / identity / authz) | Adopted (lib `0.1.4`; all 3 apps, merged to **local default** main/master + **pushed to origin** (gitea)) | Shared `ZB.MOM.WW.Auth` lib | [`components/auth/`](components/auth/) | [`ZB.MOM.WW.Auth/`](ZB.MOM.WW.Auth/) |
|
||||||
| UI Theme (layout / tokens / components) | Adopted (feed `0.2.0`–`0.3.1`; **all 4 apps pinned `0.3.1`**, pushed to origin) | Shared `ZB.MOM.WW.Theme` RCL | [`components/ui-theme/`](components/ui-theme/) | [`ZB.MOM.WW.Theme/`](ZB.MOM.WW.Theme/) |
|
| UI Theme (layout / tokens / components) | Adopted (feed `0.2.0`–`0.3.1`; **all 4 apps pinned `0.3.1`**, pushed to origin) | Shared `ZB.MOM.WW.Theme` RCL | [`components/ui-theme/`](components/ui-theme/) | [`ZB.MOM.WW.Theme/`](ZB.MOM.WW.Theme/) |
|
||||||
| Health (readiness / liveness / active-node) | **Adopted** (lib `0.1.0`, on the feed; **all 4 apps** reference it + wire `MapZbHealth` — verified 2026-07-18; the prior "not yet adopted" claim was stale) | Shared `ZB.MOM.WW.Health` lib | [`components/health/`](components/health/) | [`ZB.MOM.WW.Health/`](ZB.MOM.WW.Health/) |
|
| Health (readiness / liveness / active-node) | **Adopted** (feed carries `0.1.0`/`0.2.0`/`0.2.1`; **all 4 apps** reference it + wire `MapZbHealth`). **`0.2.0` (2026-07-24)** added an optional per-entry `data` object to the canonical health JSON and made `AkkaClusterHealthCheck` publish its cluster view (`leader`/`selfAddress`/`selfRoles`/`memberCount`/`unreachableCount`) — additive, so pre-0.2.0 payloads still parse. **`0.2.1` (2026-07-24) is a behaviour fix with a live consumer impact:** the active tier now returns **Unhealthy (503)** on a role-member-but-not-leader node instead of `Degraded`, which the spec maps to 200. The old value made `/health/active` answer 200 on *every* node, so it could not distinguish active from standby over HTTP at all — and OtOpcUa's Traefik routes the admin UI by exactly this probe, so its leader-pinning had silently never worked ([`lmxopcua#494`](https://gitea.dohertylan.com/dohertj2/lmxopcua/issues/494)). **Only OtOpcUa is affected** — it is the sole user of the role-filtered mode; ScadaBridge deliberately uses its own `OldestNodeActiveHealthCheck` (leadership is address-ordered and diverges from the singleton host), and mxgw/HistorianGateway have no Akka cluster. | Shared `ZB.MOM.WW.Health` lib | [`components/health/`](components/health/) | [`ZB.MOM.WW.Health/`](ZB.MOM.WW.Health/) |
|
||||||
| Observability (metrics / traces / logs) | **Adopted** (libs `0.1.0`, on the feed; **all 4 apps** call `AddZbTelemetry`; `AddZbSerilog` in all but ScadaBridge, which keeps its own factory by design) | Shared `ZB.MOM.WW.Telemetry` lib + `.Serilog` | [`components/observability/`](components/observability/) | [`ZB.MOM.WW.Telemetry/`](ZB.MOM.WW.Telemetry/) |
|
| Observability (metrics / traces / logs) | **Adopted** (libs `0.1.0`, on the feed; **all 4 apps** call `AddZbTelemetry`; `AddZbSerilog` in all but ScadaBridge, which keeps its own factory by design) | Shared `ZB.MOM.WW.Telemetry` lib + `.Serilog` | [`components/observability/`](components/observability/) | [`ZB.MOM.WW.Telemetry/`](ZB.MOM.WW.Telemetry/) |
|
||||||
| Config + validation (options / startup validation) | Adopted (lib `0.1.0`, on the feed; **all 4 apps**, pushed — the "local only / not yet pushed" caveat was stale) | Shared `ZB.MOM.WW.Configuration` lib | [`components/configuration/`](components/configuration/) | [`ZB.MOM.WW.Configuration/`](ZB.MOM.WW.Configuration/) |
|
| Config + validation (options / startup validation) | Adopted (lib `0.1.0`, on the feed; **all 4 apps**, pushed — the "local only / not yet pushed" caveat was stale) | Shared `ZB.MOM.WW.Configuration` lib | [`components/configuration/`](components/configuration/) | [`ZB.MOM.WW.Configuration/`](ZB.MOM.WW.Configuration/) |
|
||||||
| Audit (event model + writer seam) | Adopted (lib `0.1.0`, on the feed; **all 4 apps**, pushed to origin) | Shared `ZB.MOM.WW.Audit` lib | [`components/audit/`](components/audit/) | [`ZB.MOM.WW.Audit/`](ZB.MOM.WW.Audit/) |
|
| Audit (event model + writer seam) | Adopted (lib `0.1.0`, on the feed; **all 4 apps**, pushed to origin) | Shared `ZB.MOM.WW.Audit` lib | [`components/audit/`](components/audit/) | [`ZB.MOM.WW.Audit/`](ZB.MOM.WW.Audit/) |
|
||||||
|
|||||||
@@ -220,10 +220,12 @@
|
|||||||
"OUTSTANDING: the OtOpcUa docker-dev rig still runs a pre-0.2.0 build. Redeploying it needs a decision because the repo is checked out on feat/mesh-phase6, which carries 4 commits and an untracked Phase 7 plan beyond the base of feat/health-0.2.0-bump.",
|
"OUTSTANDING: the OtOpcUa docker-dev rig still runs a pre-0.2.0 build. Redeploying it needs a decision because the repo is checked out on feat/mesh-phase6, which carries 4 commits and an untracked Phase 7 plan beyond the base of feat/health-0.2.0-bump.",
|
||||||
"PRODUCT BUG found by the live rig: LeaderResolver and InstanceCard.ClusterDataLine both keyed the cluster view on the check NAME 'akka-cluster'. ScadaBridge registers the shared AkkaClusterHealthCheck under that name; OtOpcUa registers the SAME check as 'akka'. The result was a silent miss \u2014 OtOpcUa groups rendered with no leader chip and no evidence line, which is indistinguishable from a group that simply has not reported yet. Both now locate the entry by the DATA KEY ('leader'), which is the part of the contract the shared check owns. Regression tests cover akka-cluster / akka / an arbitrary future name, plus an entry that publishes data WITHOUT a leader (ScadaBridge's localdb check) so scanning cannot pick the wrong entry. Unit tests and review had both missed it because every fixture used ScadaBridge's name.",
|
"PRODUCT BUG found by the live rig: LeaderResolver and InstanceCard.ClusterDataLine both keyed the cluster view on the check NAME 'akka-cluster'. ScadaBridge registers the shared AkkaClusterHealthCheck under that name; OtOpcUa registers the SAME check as 'akka'. The result was a silent miss \u2014 OtOpcUa groups rendered with no leader chip and no evidence line, which is indistinguishable from a group that simply has not reported yet. Both now locate the entry by the DATA KEY ('leader'), which is the part of the contract the shared check owns. Regression tests cover akka-cluster / akka / an arbitrary future name, plus an entry that publishes data WITHOUT a leader (ScadaBridge's localdb check) so scanning cannot pick the wrong entry. Unit tests and review had both missed it because every fixture used ScadaBridge's name.",
|
||||||
"OTOPCUA CONTRACT GAP (not a dashboard bug, worth filing): /health/active does not answer 'am I the active node'. On the standby admin (central-2) the admin-leader check returns Degraded ('Role admin member but not leader'), and ASP.NET maps Degraded to 200 \u2014 so a standby looks Active. On driver-only nodes it returns Healthy with the description 'Active for role admin (or not a role member)', i.e. it passes vacuously for every node that is not an admin at all. Meanwhile CLAUDE.md documents the real election as the oldest Up driver member per Cluster (IsDriverPrimary, ServiceLevel 250 vs 240), which this tier never consults. For the tier to mean what MapZbHealth's status mapping implies, a non-active node must return Unhealthy (503) \u2014 which is exactly what ScadaBridge's SitePairActiveNodeHealthCheck does, and why ScadaBridge reads correctly. FILED as lmxopcua#494 (https://gitea.dohertylan.com/dohertj2/lmxopcua/issues/494), with the traefik-dynamic.yml LB-healthcheck impact flagged: docker-dev uses /health/active as its load-balancer probe, so making a standby return 503 would drop it from the UI pool.",
|
"OTOPCUA CONTRACT GAP (not a dashboard bug, worth filing): /health/active does not answer 'am I the active node'. On the standby admin (central-2) the admin-leader check returns Degraded ('Role admin member but not leader'), and ASP.NET maps Degraded to 200 \u2014 so a standby looks Active. On driver-only nodes it returns Healthy with the description 'Active for role admin (or not a role member)', i.e. it passes vacuously for every node that is not an admin at all. Meanwhile CLAUDE.md documents the real election as the oldest Up driver member per Cluster (IsDriverPrimary, ServiceLevel 250 vs 240), which this tier never consults. For the tier to mean what MapZbHealth's status mapping implies, a non-active node must return Unhealthy (503) \u2014 which is exactly what ScadaBridge's SitePairActiveNodeHealthCheck does, and why ScadaBridge reads correctly. FILED as lmxopcua#494 (https://gitea.dohertylan.com/dohertj2/lmxopcua/issues/494), with the traefik-dynamic.yml LB-healthcheck impact flagged: docker-dev uses /health/active as its load-balancer probe, so making a standby return 503 would drop it from the UI pool.",
|
||||||
"OtOpcUa rig rebuilt from a new branch feat/mesh-phase6-health-0.2.0 = feat/mesh-phase6 + a cherry-pick of the 3-line Health 0.2.0 bump (2e515c34). Checking out feat/health-0.2.0-bump directly would have regressed the rig past four commits of in-progress Phase 6/7 work. All three meshes formed cleanly (2 members, 0 unreachable, both members agreeing on one leader) \u2014 OtOpcUa's per-pair self-first seeding did NOT hit the cold-boot self-form race that ScadaBridge did."
|
"OtOpcUa rig rebuilt from a new branch feat/mesh-phase6-health-0.2.0 = feat/mesh-phase6 + a cherry-pick of the 3-line Health 0.2.0 bump (2e515c34). Checking out feat/health-0.2.0-bump directly would have regressed the rig past four commits of in-progress Phase 6/7 work. All three meshes formed cleanly (2 members, 0 unreachable, both members agreeing on one leader) \u2014 OtOpcUa's per-pair self-first seeding did NOT hit the cold-boot self-form race that ScadaBridge did.",
|
||||||
|
"TRAEFIK LEADER-PINNING FIXED (ZB.MOM.WW.Health 0.2.1, published + restore-verified; OtOpcUa bumped at 8dd9da7d). Root cause was in the shared library: ActiveNodeDecision.Evaluate returned Degraded for role-member-but-not-leader, and MapZbHealth maps Degraded to 200 \u2014 so /health/active answered 200 on every node and OtOpcUa's Traefik kept the standby in the admin-UI pool. The shared spec had always said the tier 'Fails (503) on a standby or role-member-but-not-leader node', and OtOpcUa's own docs said '200 only on the Admin-role leader; 503 elsewhere'; the docs were right and the code was wrong. Live-verified: central-1 200/traefik UP, central-2 503/traefik DOWN; the README failover drill (which had never exercised anything) flipped Traefik to central-2 within 20s of stopping central-1 with the UI answering 200 continuously across the swap, and central-1 reclaimed the role-leader on restart. Blast radius one consumer \u2014 ScadaBridge uses its own OldestNodeActiveHealthCheck, mxgw/HG have no Akka.",
|
||||||
|
"RIG OPERATIONS LANDMINE: the docker-dev migrator (dotnet ef database update) HANGS under x86 emulation \u2014 22 minutes at 0.00% CPU with no output, and all six node containers sit in Created because they depend_on service_completed_successfully. Worked around by stopping the migrator and docker-start-ing the nodes directly, which is safe when the change is a package-version bump with no EF model change. Worth knowing before assuming a rebuild is merely slow."
|
||||||
],
|
],
|
||||||
"phase4_results": {
|
"phase4_results": {
|
||||||
"1_all_render_single_active_per_pair": "PARTIAL. ScadaBridge PASS: all four pairs render one Active + one Standby (active 200 / 503). OtOpcUa CANNOT PASS as built: all six nodes answer /health/active 200 because that tier does not gate on the redundancy Primary \u2014 see the OtOpcUa active-tier finding.",
|
"1_all_render_single_active_per_pair": "PARTIAL, and now partial for a smaller reason. ScadaBridge PASS (all four pairs: one Active + one Standby). OtOpcUa central pair PASS after ZB.MOM.WW.Health 0.2.1 \u2014 central-1 Active / central-2 Standby, live-verified. OtOpcUa's four DRIVER-ONLY site nodes still all read Active: the admin-leader check is scoped to the admin role and returns Healthy for any node lacking it, so it never consults the real per-Cluster Primary (IsDriverPrimary). Tracked in lmxopcua#494, Traefik half closed.",
|
||||||
"2_leader_chip_and_split_brain": "PASS for BOTH products after the rig redeploy and the LeaderResolver fix. All seven cluster groups (ScadaBridge central/site-a/b/c, OtOpcUa MAIN/SITE-A/SITE-B) render a Leader chip with zero split-brain flags. Verified earlier that the chip MOVES when the leader is stopped and that the split-brain flag appears while a pair is self-formed and clears once it genuinely joins.",
|
"2_leader_chip_and_split_brain": "PASS for BOTH products after the rig redeploy and the LeaderResolver fix. All seven cluster groups (ScadaBridge central/site-a/b/c, OtOpcUa MAIN/SITE-A/SITE-B) render a Leader chip with zero split-brain flags. Verified earlier that the chip MOVES when the leader is stopped and that the split-brain flag appears while a pair is self-formed and clears once it genuinely joins.",
|
||||||
"3_kill_and_restart": "PASS. Stopped scadabridge-site-c-a 12:01:27; Unreachable by 12:02:33 (two-strike damping = 2 failing sweeps). Restarted 12:04:01; Up again by 12:04:16, i.e. on the first good poll (recovery is deliberately asymmetric)",
|
"3_kill_and_restart": "PASS. Stopped scadabridge-site-c-a 12:01:27; Unreachable by 12:02:33 (two-strike damping = 2 failing sweeps). Restarted 12:04:01; Up again by 12:04:16, i.e. on the first good poll (recovery is deliberately asymmetric)",
|
||||||
"4_degraded_with_failing_entry": "PASS. Canned Degraded and Unhealthy endpoints render as Degraded/Down with the failing entry and its description visible, and check-dot ok/warn/bad classes per check",
|
"4_degraded_with_failing_entry": "PASS. Canned Degraded and Unhealthy endpoints render as Degraded/Down with the failing entry and its description visible, and check-dot ok/warn/bad classes per check",
|
||||||
|
|||||||
Reference in New Issue
Block a user