diff --git a/ZB.MOM.WW.Overview/CLAUDE.md b/ZB.MOM.WW.Overview/CLAUDE.md index 6ec41df..24e7481 100644 --- a/ZB.MOM.WW.Overview/CLAUDE.md +++ b/ZB.MOM.WW.Overview/CLAUDE.md @@ -2,7 +2,7 @@ The hosted family dashboard (Blazor Server). Build/test from this directory: `dotnet build ZB.MOM.WW.Overview.slnx`, `dotnet test ZB.MOM.WW.Overview.slnx` (fully offline). -**Backlog / open work:** lives in Gitea issues on `dohertj2/scadaproj` (`tea issues list --repo dohertj2/scadaproj`) — currently #14 (re-run the two OtOpcUa-blocked Phase 4 acceptance checks). The history below is a record, not a status tracker; file new follow-ups as issues. +**Backlog / open work:** lives in Gitea issues on `dohertj2/scadaproj` (`tea issues list --repo dohertj2/scadaproj`) — none open for Overview as of 2026-11-12 (#14, the last Phase 4 checks, closed). The history below is a record, not a status tracker; file new follow-ups as issues. ## Probe behaviour (P3 review fixes, scadaproj#48) @@ -23,8 +23,10 @@ every registered instance's anonymous `/health/ready` + `/health/active`, caches the `ZB.MOM.WW.Health` `0.2.0`/`0.3.0` per-entry `data` work exists to feed. **No login, strictly read-only, holds no credentials** — a requirement, not an omission; it references no Auth/Audit/ Secrets package. Design + impl plan: `docs/plans/2026-07-22-overview-dashboard-{design,impl-plan}.md` -(ledger 19/20; the open task is Phase 4 live acceptance, recorded in `41013dc` as 7 of 9 design §9 -checks passing, 2 blocked on an OtOpcUa rig redeploy → #14). +(ledger 20/20. Phase 4 live acceptance was recorded in `41013dc` as 7 of 9 design §9 checks passing, with 2 +blocked on an OtOpcUa rig redeploy. Those two, one Active per pair and the leader/split-brain chips, were re-run +against an OtOpcUa master `412da27c` rig on 2026-11-12 and passed: scadaproj#14, +`docs/plans/evidence/2026-11-12-overview-phase4/README.md`.) The `Applications` registry ships **empty** on purpose: a deployment supplying none fails startup naming the key rather than booting an empty dashboard. Per-deployment overlays are committed — diff --git a/docs/plans/2026-07-22-overview-dashboard-design.md b/docs/plans/2026-07-22-overview-dashboard-design.md index 79ac0bf..8d05b0e 100644 --- a/docs/plans/2026-07-22-overview-dashboard-design.md +++ b/docs/plans/2026-07-22-overview-dashboard-design.md @@ -147,7 +147,7 @@ Single page (`/`), `ThemeShell` chassis with its own product name/accent so it r ## 9. Acceptance (v1 definition of done) - The §4 ScadaBridge site-node health PR is merged and site rows render with full check detail + Active/Standby (no special-cased rows anywhere). -- Registry pointed at the live docker-dev rigs (OtOpcUa 6-node + ScadaBridge local cluster) renders every instance correctly, including: each Active/Standby pair (central **and** site pairs) showing exactly one Active; **each cluster group showing its current leader, and the leader chip moving when the leader node is stopped**; killing a node flips its card to Unreachable within ~2 cadences and back on restart; a Degraded check (e.g. stop a dependency) shows Warn with the failing entry visible; management links open the right app UIs; the page is reachable with no login. (OtOpcUa one-Active + leader-chip checks still to re-run → #14) +- Registry pointed at the live docker-dev rigs (OtOpcUa 6-node + ScadaBridge local cluster) renders every instance correctly, including: each Active/Standby pair (central **and** site pairs) showing exactly one Active; **each cluster group showing its current leader, and the leader chip moving when the leader node is stopped**; killing a node flips its card to Unreachable within ~2 cadences and back on restart; a Degraded check (e.g. stop a dependency) shows Warn with the failing entry visible; management links open the right app UIs; the page is reachable with no login. (OtOpcUa one-Active + leader-chip/split-brain checks re-run and PASSED 2026-11-12, scadaproj#14 — `docs/plans/evidence/2026-11-12-overview-phase4/`) - **Cache/staleness proven live:** first page load renders instantly from cache (no poll on request); with auto-refresh paused (or the poller stopped) past `StaleAfterSeconds`, every card flips to Stale, and resumes cleanly. - Boot fails with a clear `ConfigPreflight` message on a malformed registry. - `dotnet test` green: options validator, status derivation (incl. flap damping, staleness marking, leader extraction + disagreement flag, JSON parsing golden tests against captured real payloads from all four apps — with and without `data`), snapshot store, and a bUnit render test of `InstanceCard`. diff --git a/docs/plans/2026-07-22-overview-dashboard-impl-plan.md.tasks.json b/docs/plans/2026-07-22-overview-dashboard-impl-plan.md.tasks.json index 33253e5..65a8ccb 100644 --- a/docs/plans/2026-07-22-overview-dashboard-impl-plan.md.tasks.json +++ b/docs/plans/2026-07-22-overview-dashboard-impl-plan.md.tasks.json @@ -155,7 +155,7 @@ { "id": "20", "subject": "4 Live acceptance (design \u00a79)", - "status": "pending", + "status": "completed", "blockedBy": [ "7", "9", @@ -236,6 +236,13 @@ "7_management_links_no_login": "PASS. 'Open CentralUI' opened http://localhost:9001 in a new tab (target=_blank rel=noopener); the dashboard itself never presented or redirected to a login", "8_malformed_registry": "PASS both branches. No registry -> ConfigPreflight names Overview:Applications:0:Name and :Instances:0:BaseUrl. Stale window inside the poll interval -> validator names the keys and the consequence", "9_health_and_metrics": "PASS. /health/ready, /health/active, /healthz, /metrics all 200; 16 gauge series with application/node/group labels" + }, + "phase4_rerun_2026_11_12": { + "issue": "scadaproj#14", + "evidence": "docs/plans/evidence/2026-11-12-overview-phase4/README.md", + "rig": "OtOpcUa docker-dev from master 412da27c (Health 0.3.0), own compose project; Overview main d93ca7b in a container on the rig network", + "1_all_render_single_active_per_pair": "PASS. Exactly one /health/active 200 per mesh: MAIN central-2, SITE-A site-a-1, SITE-B site-b-1; partners 503. Dashboard: one ACTIVE + one STANDBY in all three groups.", + "2_leader_chip_and_split_brain": "PASS. LEADER chip on every group from data.leader; network-partitioning site-b-2 from site-b-1 produced two 1-member clusters, both Active, and the SPLIT BRAIN chip; it cleared once site-b-2 was restarted and rejoined (healing the network alone does not merge two Akka clusters - documented OtOpcUa recovery). Stopping site-a-1 moved the SITE-A chip to site-a-2." } } } \ No newline at end of file diff --git a/docs/plans/evidence/2026-11-12-overview-phase4/14-check1-baseline.png b/docs/plans/evidence/2026-11-12-overview-phase4/14-check1-baseline.png new file mode 100644 index 0000000..413a675 Binary files /dev/null and b/docs/plans/evidence/2026-11-12-overview-phase4/14-check1-baseline.png differ diff --git a/docs/plans/evidence/2026-11-12-overview-phase4/14-check2-cleared.png b/docs/plans/evidence/2026-11-12-overview-phase4/14-check2-cleared.png new file mode 100644 index 0000000..3c6e270 Binary files /dev/null and b/docs/plans/evidence/2026-11-12-overview-phase4/14-check2-cleared.png differ diff --git a/docs/plans/evidence/2026-11-12-overview-phase4/14-check2-leader-moved.png b/docs/plans/evidence/2026-11-12-overview-phase4/14-check2-leader-moved.png new file mode 100644 index 0000000..0346704 Binary files /dev/null and b/docs/plans/evidence/2026-11-12-overview-phase4/14-check2-leader-moved.png differ diff --git a/docs/plans/evidence/2026-11-12-overview-phase4/14-check2-splitbrain.png b/docs/plans/evidence/2026-11-12-overview-phase4/14-check2-splitbrain.png new file mode 100644 index 0000000..62ad4ed Binary files /dev/null and b/docs/plans/evidence/2026-11-12-overview-phase4/14-check2-splitbrain.png differ diff --git a/docs/plans/evidence/2026-11-12-overview-phase4/README.md b/docs/plans/evidence/2026-11-12-overview-phase4/README.md new file mode 100644 index 0000000..8ec0a74 --- /dev/null +++ b/docs/plans/evidence/2026-11-12-overview-phase4/README.md @@ -0,0 +1,65 @@ +# Overview Phase 4 — the two OtOpcUa-blocked §9 checks, re-run 2026-11-12 (scadaproj#14) + +Design §9 (`docs/plans/2026-07-22-overview-dashboard-design.md`), plan task 20. On 2026-07-24 two of the nine checks +passed for ScadaBridge but were blocked for OtOpcUa (pre-Health-0.2.0 build: every node Active, no `data`, no leader +chip). Both are re-run here against a current OtOpcUa rig. **Result: both PASS.** + +## Setup + +| | | +|---|---| +| OtOpcUa | docker-dev rig built from `master` `412da27c` (Telemetry 0.3.0; pins `ZB.MOM.WW.Health` 0.3.0), run as its own compose project `otopcua-g761` (remapped host ports, image `otopcua-host:g761`) so it did not touch the shared `otopcua-dev` rig. Three independent 2-node meshes: MAIN `central-1/2`, SITE-A `site-a-1/2`, SITE-B `site-b-1/2`. | +| Overview | scadaproj `main` `d93ca7b` (Overview last changed `6123d03`), published framework-dependent and run in an `aspnet:10.0` container joined to the rig network (`otopcua-g761_default`), so every node is addressed by container DNS name. Leader resolution is per group and host-only, so a host-run dashboard on `localhost:*` URLs could not tell pair members apart. Registry: six instances, three groups (MAIN on `:9000`, SITE-A/SITE-B on `:8080`), `HasActiveRole: true`, poll 5 s. | +| Clock | All times UTC 2026-11-13 02:49–02:55 (= 2026-11-12 evening US Eastern). | + +## Check 1 — every Active/Standby pair shows exactly one Active — PASS + +`GET /health/active` on all six nodes (raw bodies: [`active-baseline.txt`](active-baseline.txt)): + +| node | HTTP | akka `memberCount` | akka `leader` | `cluster-primary` | +|---|---|---|---|---| +| central-1 | 503 | 2 | central-1 | Standby: active node for role 'admin' is central-2 | +| central-2 | **200** | 2 | central-1 | Active node for role 'admin' (oldest Up member) | +| site-a-1 | **200** | 2 | site-a-1 | Active node for role 'driver' (oldest Up member) | +| site-a-2 | 503 | 2 | site-a-1 | Standby: active node for role 'driver' is site-a-1 | +| site-b-1 | **200** | 2 | site-b-1 | Active node for role 'driver' (oldest Up member) | +| site-b-2 | 503 | 2 | site-b-1 | Standby: active node for role 'driver' is site-b-1 | + +Exactly one 200 per mesh. The dashboard renders one ACTIVE + one STANDBY in all three groups +([`14-check1-baseline.png`](14-check1-baseline.png)). MAIN shows the documented leader/active divergence: Akka leader +`central-1` (lowest address) carries LEADER + STANDBY, and `central-2` (oldest Up `admin` member, the singleton host) +is ACTIVE. + +## Check 2 — leader chip, and a split-brain chip that appears and clears — PASS + +- **Leader chip.** Every group shows `LEADER · ` from the akka entry's `data.leader` (MAIN `central-1`, + SITE-A `site-a-1`, SITE-B `site-b-1`) — [`14-check1-baseline.png`](14-check1-baseline.png). +- **Split brain forced.** At 02:49:45 `site-b-2` was disconnected from the rig network (`docker network disconnect`), + after first being attached to a private network `g761-probe` (alias `site-b-2`) shared only with the dashboard, so + the pair lost each other but the dashboard kept reaching both. Within ~25 s each side downed the other (auto-down, + the accepted availability-over-partition-safety trade): both became 1-member clusters, both answered + `/health/active` 200 and each named itself leader ([`split-timeline.txt`](split-timeline.txt)). The dashboard showed + `SPLIT BRAIN · AKKA.TCP://OTOPCUA@SITE-B-1:4053 VS AKKA.TCP://OTOPCUA@SITE-B-2:4053` on SITE-B with both cards + ACTIVE ([`14-check2-splitbrain.png`](14-check2-splitbrain.png)). +- **Clears.** Reconnecting the network (02:51:41) does **not** end the split. Two formed Akka clusters never merge, + and both nodes stayed 1-member and Active for the next 70 s. This is the documented recovery in OtOpcUa + `docs/Redundancy.md` ("after the link heals, restart one side"), not a dashboard or product defect. After + `docker restart` of `site-b-2` (02:53:10) it rejoined; at 02:53:25 site-b-1 was 200 and site-b-2 503, both + `memberCount` 2, both naming `site-b-1` leader, and the chip was gone ([`14-check2-cleared.png`](14-check2-cleared.png); + raw [`active-after-rejoin.txt`](active-after-rejoin.txt): three 200s, one per mesh). +- **Leader chip moves** (the §9 wording). Stopping SITE-A's leader `site-a-1` (02:53:44) moved the chip to + `LEADER · SITE-A-2`, with `site-a-2` ACTIVE and `site-a-1` UNREACHABLE ([`14-check2-leader-moved.png`](14-check2-leader-moved.png)). + `site-a-1` was restarted at 02:54:24 and rejoined. + +## Observations (none is a defect) + +- The group header's "N members · M unreachable" counts **registered** instances (`Overview.razor` `group.Count()`), + not Akka membership, so it read "2 members · 0 unreachable" during the split while each node reported + `memberCount` 1. Akka's own count is on each card's check detail. Working as designed. +- Only the resolved group leader gets a LEADER badge; during the split `site-b-2` (which named itself leader) had + none, and the disagreement is carried by the split-brain chip instead. + +## Cleanup + +The `otopcua-g761` rig, the `g761-probe` network and the dashboard container were removed after the +lmxopcua#761 gate run that shared the rig. diff --git a/docs/plans/evidence/2026-11-12-overview-phase4/active-after-rejoin.txt b/docs/plans/evidence/2026-11-12-overview-phase4/active-after-rejoin.txt new file mode 100644 index 0000000..e861d58 --- /dev/null +++ b/docs/plans/evidence/2026-11-12-overview-phase4/active-after-rejoin.txt @@ -0,0 +1,18 @@ +== central-1:9000 /health/active +{"status":"Unhealthy","totalDurationMs":3.7296,"entries":{"configdb":{"status":"Healthy","description":"Database query probe succeeded.","durationMs":3.3947},"akka":{"status":"Healthy","description":"Akka cluster member status: Up","durationMs":0.0208,"data":{"selfAddress":"akka.tcp://otopcua@central-1:4053","selfRoles":["admin","cluster-MAIN","driver"],"memberCount":2,"unreachableCount":0,"leader":"akka.tcp://otopcua@central-1:4053"}},"cluster-primary":{"status":"Unhealthy","description":"Standby: active node for role \u0027admin\u0027 is akka.tcp://otopcua@central-2:4053.","durationMs":0.0147,"data":{"selfAddress":"akka.tcp://otopcua@central-1:4053","activeRole":"admin","activeNode":"akka.tcp://otopcua@central-2:4053"}},"localdb-replication":{"status":"Healthy","description":"LocalDb replication is not configured on this node (default-OFF).","durationMs":0.0993}}} +HTTP 503 +== central-2:9000 /health/active +{"status":"Healthy","totalDurationMs":1.0577,"entries":{"configdb":{"status":"Healthy","description":"Database query probe succeeded.","durationMs":0.8938},"akka":{"status":"Healthy","description":"Akka cluster member status: Up","durationMs":0.0115,"data":{"selfAddress":"akka.tcp://otopcua@central-2:4053","selfRoles":["admin","cluster-MAIN","driver"],"memberCount":2,"unreachableCount":0,"leader":"akka.tcp://otopcua@central-1:4053"}},"cluster-primary":{"status":"Healthy","description":"Active node for role \u0027admin\u0027 (oldest Up member).","durationMs":0.0152,"data":{"selfAddress":"akka.tcp://otopcua@central-2:4053","activeRole":"admin","activeNode":"akka.tcp://otopcua@central-2:4053"}},"localdb-replication":{"status":"Healthy","description":"LocalDb replication is not configured on this node (default-OFF).","durationMs":0.0886}}} +HTTP 200 +== site-a-1:8080 /health/active +{"status":"Healthy","totalDurationMs":0.2295,"entries":{"opcua-endpoint":{"status":"Healthy","description":"OPC UA endpoint is listening","durationMs":0.0085,"data":{"status":"Started"}},"akka":{"status":"Healthy","description":"Akka cluster member status: Up","durationMs":0.0116,"data":{"selfAddress":"akka.tcp://otopcua@site-a-1:4053","selfRoles":["cluster-SITE-A","driver"],"memberCount":2,"unreachableCount":0,"leader":"akka.tcp://otopcua@site-a-1:4053"}},"cluster-primary":{"status":"Healthy","description":"Active node for role \u0027driver\u0027 (oldest Up member).","durationMs":0.0228,"data":{"selfAddress":"akka.tcp://otopcua@site-a-1:4053","activeRole":"driver","activeNode":"akka.tcp://otopcua@site-a-1:4053"}},"localdb-replication":{"status":"Healthy","description":"LocalDb replication is connected; oplog backlog 0.","durationMs":0.097}}} +HTTP 200 +== site-a-2:8080 /health/active +{"status":"Unhealthy","totalDurationMs":0.3031,"entries":{"opcua-endpoint":{"status":"Healthy","description":"OPC UA endpoint is listening","durationMs":0.0023,"data":{"status":"Started"}},"akka":{"status":"Healthy","description":"Akka cluster member status: Up","durationMs":0.0182,"data":{"selfAddress":"akka.tcp://otopcua@site-a-2:4053","selfRoles":["cluster-SITE-A","driver"],"memberCount":2,"unreachableCount":0,"leader":"akka.tcp://otopcua@site-a-1:4053"}},"cluster-primary":{"status":"Unhealthy","description":"Standby: active node for role \u0027driver\u0027 is akka.tcp://otopcua@site-a-1:4053.","durationMs":0.0151,"data":{"selfAddress":"akka.tcp://otopcua@site-a-2:4053","activeRole":"driver","activeNode":"akka.tcp://otopcua@site-a-1:4053"}},"localdb-replication":{"status":"Healthy","description":"LocalDb replication is connected; oplog backlog 0.","durationMs":0.244}}} +HTTP 503 +== site-b-1:8080 /health/active +{"status":"Healthy","totalDurationMs":0.135,"entries":{"opcua-endpoint":{"status":"Healthy","description":"OPC UA endpoint is listening","durationMs":0.001,"data":{"status":"Started"}},"akka":{"status":"Healthy","description":"Akka cluster member status: Up","durationMs":0.0107,"data":{"selfAddress":"akka.tcp://otopcua@site-b-1:4053","selfRoles":["cluster-SITE-B","driver"],"memberCount":2,"unreachableCount":0,"leader":"akka.tcp://otopcua@site-b-1:4053"}},"cluster-primary":{"status":"Healthy","description":"Active node for role \u0027driver\u0027 (oldest Up member).","durationMs":0.0135,"data":{"selfAddress":"akka.tcp://otopcua@site-b-1:4053","activeRole":"driver","activeNode":"akka.tcp://otopcua@site-b-1:4053"}},"localdb-replication":{"status":"Healthy","description":"LocalDb replication is not configured on this node (default-OFF).","durationMs":0.0677}}} +HTTP 200 +== site-b-2:8080 /health/active + +HTTP 000 diff --git a/docs/plans/evidence/2026-11-12-overview-phase4/active-baseline.txt b/docs/plans/evidence/2026-11-12-overview-phase4/active-baseline.txt new file mode 100644 index 0000000..53ffeb7 --- /dev/null +++ b/docs/plans/evidence/2026-11-12-overview-phase4/active-baseline.txt @@ -0,0 +1,18 @@ +== central-1:9000 /health/active +{"status":"Unhealthy","totalDurationMs":1.9007,"entries":{"configdb":{"status":"Healthy","description":"Database query probe succeeded.","durationMs":1.5868},"akka":{"status":"Healthy","description":"Akka cluster member status: Up","durationMs":0.0205,"data":{"selfAddress":"akka.tcp://otopcua@central-1:4053","selfRoles":["admin","cluster-MAIN","driver"],"memberCount":2,"unreachableCount":0,"leader":"akka.tcp://otopcua@central-1:4053"}},"cluster-primary":{"status":"Unhealthy","description":"Standby: active node for role \u0027admin\u0027 is akka.tcp://otopcua@central-2:4053.","durationMs":0.0282,"data":{"selfAddress":"akka.tcp://otopcua@central-1:4053","activeRole":"admin","activeNode":"akka.tcp://otopcua@central-2:4053"}},"localdb-replication":{"status":"Healthy","description":"LocalDb replication is not configured on this node (default-OFF).","durationMs":0.1166}}} +HTTP 503 +== central-2:9000 /health/active +{"status":"Healthy","totalDurationMs":1.5114,"entries":{"configdb":{"status":"Healthy","description":"Database query probe succeeded.","durationMs":1.3051},"akka":{"status":"Healthy","description":"Akka cluster member status: Up","durationMs":0.0189,"data":{"selfAddress":"akka.tcp://otopcua@central-2:4053","selfRoles":["admin","cluster-MAIN","driver"],"memberCount":2,"unreachableCount":0,"leader":"akka.tcp://otopcua@central-1:4053"}},"cluster-primary":{"status":"Healthy","description":"Active node for role \u0027admin\u0027 (oldest Up member).","durationMs":0.0208,"data":{"selfAddress":"akka.tcp://otopcua@central-2:4053","activeRole":"admin","activeNode":"akka.tcp://otopcua@central-2:4053"}},"localdb-replication":{"status":"Healthy","description":"LocalDb replication is not configured on this node (default-OFF).","durationMs":0.0995}}} +HTTP 200 +== site-a-1:8080 /health/active +{"status":"Healthy","totalDurationMs":0.2578,"entries":{"opcua-endpoint":{"status":"Healthy","description":"OPC UA endpoint is listening","durationMs":0.0019,"data":{"status":"Started"}},"akka":{"status":"Healthy","description":"Akka cluster member status: Up","durationMs":0.0261,"data":{"selfAddress":"akka.tcp://otopcua@site-a-1:4053","selfRoles":["cluster-SITE-A","driver"],"memberCount":2,"unreachableCount":0,"leader":"akka.tcp://otopcua@site-a-1:4053"}},"cluster-primary":{"status":"Healthy","description":"Active node for role \u0027driver\u0027 (oldest Up member).","durationMs":0.0282,"data":{"selfAddress":"akka.tcp://otopcua@site-a-1:4053","activeRole":"driver","activeNode":"akka.tcp://otopcua@site-a-1:4053"}},"localdb-replication":{"status":"Healthy","description":"LocalDb replication is connected; oplog backlog 0.","durationMs":0.0855}}} +HTTP 200 +== site-a-2:8080 /health/active +{"status":"Unhealthy","totalDurationMs":0.4101,"entries":{"opcua-endpoint":{"status":"Healthy","description":"OPC UA endpoint is listening","durationMs":0.0008,"data":{"status":"Started"}},"akka":{"status":"Healthy","description":"Akka cluster member status: Up","durationMs":0.0298,"data":{"selfAddress":"akka.tcp://otopcua@site-a-2:4053","selfRoles":["cluster-SITE-A","driver"],"memberCount":2,"unreachableCount":0,"leader":"akka.tcp://otopcua@site-a-1:4053"}},"cluster-primary":{"status":"Unhealthy","description":"Standby: active node for role \u0027driver\u0027 is akka.tcp://otopcua@site-a-1:4053.","durationMs":0.0314,"data":{"selfAddress":"akka.tcp://otopcua@site-a-2:4053","activeRole":"driver","activeNode":"akka.tcp://otopcua@site-a-1:4053"}},"localdb-replication":{"status":"Healthy","description":"LocalDb replication is connected; oplog backlog 0.","durationMs":0.1005}}} +HTTP 503 +== site-b-1:8080 /health/active +{"status":"Healthy","totalDurationMs":0.2829,"entries":{"opcua-endpoint":{"status":"Healthy","description":"OPC UA endpoint is listening","durationMs":0.0016,"data":{"status":"Started"}},"akka":{"status":"Healthy","description":"Akka cluster member status: Up","durationMs":0.0338,"data":{"selfAddress":"akka.tcp://otopcua@site-b-1:4053","selfRoles":["cluster-SITE-B","driver"],"memberCount":2,"unreachableCount":0,"leader":"akka.tcp://otopcua@site-b-1:4053"}},"cluster-primary":{"status":"Healthy","description":"Active node for role \u0027driver\u0027 (oldest Up member).","durationMs":0.0242,"data":{"selfAddress":"akka.tcp://otopcua@site-b-1:4053","activeRole":"driver","activeNode":"akka.tcp://otopcua@site-b-1:4053"}},"localdb-replication":{"status":"Healthy","description":"LocalDb replication is not configured on this node (default-OFF).","durationMs":0.118}}} +HTTP 200 +== site-b-2:8080 /health/active +{"status":"Unhealthy","totalDurationMs":0.4468,"entries":{"opcua-endpoint":{"status":"Healthy","description":"OPC UA endpoint is listening","durationMs":0.0016,"data":{"status":"Started"}},"akka":{"status":"Healthy","description":"Akka cluster member status: Up","durationMs":0.037,"data":{"selfAddress":"akka.tcp://otopcua@site-b-2:4053","selfRoles":["cluster-SITE-B","driver"],"memberCount":2,"unreachableCount":0,"leader":"akka.tcp://otopcua@site-b-1:4053"}},"cluster-primary":{"status":"Unhealthy","description":"Standby: active node for role \u0027driver\u0027 is akka.tcp://otopcua@site-b-1:4053.","durationMs":0.0288,"data":{"selfAddress":"akka.tcp://otopcua@site-b-2:4053","activeRole":"driver","activeNode":"akka.tcp://otopcua@site-b-1:4053"}},"localdb-replication":{"status":"Healthy","description":"LocalDb replication is not configured on this node (default-OFF).","durationMs":0.0971}}} +HTTP 503 diff --git a/docs/plans/evidence/2026-11-12-overview-phase4/active-split-default-network.txt b/docs/plans/evidence/2026-11-12-overview-phase4/active-split-default-network.txt new file mode 100644 index 0000000..3dba3f6 --- /dev/null +++ b/docs/plans/evidence/2026-11-12-overview-phase4/active-split-default-network.txt @@ -0,0 +1,18 @@ +== central-1:9000 /health/active +{"status":"Unhealthy","totalDurationMs":1.5402,"entries":{"configdb":{"status":"Healthy","description":"Database query probe succeeded.","durationMs":1.2492},"akka":{"status":"Healthy","description":"Akka cluster member status: Up","durationMs":0.011,"data":{"selfAddress":"akka.tcp://otopcua@central-1:4053","selfRoles":["admin","cluster-MAIN","driver"],"memberCount":2,"unreachableCount":0,"leader":"akka.tcp://otopcua@central-1:4053"}},"cluster-primary":{"status":"Unhealthy","description":"Standby: active node for role \u0027admin\u0027 is akka.tcp://otopcua@central-2:4053.","durationMs":0.0137,"data":{"selfAddress":"akka.tcp://otopcua@central-1:4053","activeRole":"admin","activeNode":"akka.tcp://otopcua@central-2:4053"}},"localdb-replication":{"status":"Healthy","description":"LocalDb replication is not configured on this node (default-OFF).","durationMs":0.1257}}} +HTTP 503 +== central-2:9000 /health/active +{"status":"Healthy","totalDurationMs":1.1343,"entries":{"configdb":{"status":"Healthy","description":"Database query probe succeeded.","durationMs":0.9846},"akka":{"status":"Healthy","description":"Akka cluster member status: Up","durationMs":0.0102,"data":{"selfAddress":"akka.tcp://otopcua@central-2:4053","selfRoles":["admin","cluster-MAIN","driver"],"memberCount":2,"unreachableCount":0,"leader":"akka.tcp://otopcua@central-1:4053"}},"cluster-primary":{"status":"Healthy","description":"Active node for role \u0027admin\u0027 (oldest Up member).","durationMs":0.0137,"data":{"selfAddress":"akka.tcp://otopcua@central-2:4053","activeRole":"admin","activeNode":"akka.tcp://otopcua@central-2:4053"}},"localdb-replication":{"status":"Healthy","description":"LocalDb replication is not configured on this node (default-OFF).","durationMs":0.0771}}} +HTTP 200 +== site-a-1:8080 /health/active +{"status":"Healthy","totalDurationMs":0.669,"entries":{"opcua-endpoint":{"status":"Healthy","description":"OPC UA endpoint is listening","durationMs":0.003,"data":{"status":"Started"}},"akka":{"status":"Healthy","description":"Akka cluster member status: Up","durationMs":0.0148,"data":{"selfAddress":"akka.tcp://otopcua@site-a-1:4053","selfRoles":["cluster-SITE-A","driver"],"memberCount":2,"unreachableCount":0,"leader":"akka.tcp://otopcua@site-a-1:4053"}},"cluster-primary":{"status":"Healthy","description":"Active node for role \u0027driver\u0027 (oldest Up member).","durationMs":0.4781,"data":{"selfAddress":"akka.tcp://otopcua@site-a-1:4053","activeRole":"driver","activeNode":"akka.tcp://otopcua@site-a-1:4053"}},"localdb-replication":{"status":"Healthy","description":"LocalDb replication is connected; oplog backlog 0.","durationMs":0.0967}}} +HTTP 200 +== site-a-2:8080 /health/active +{"status":"Unhealthy","totalDurationMs":0.4653,"entries":{"opcua-endpoint":{"status":"Healthy","description":"OPC UA endpoint is listening","durationMs":0.0018,"data":{"status":"Started"}},"akka":{"status":"Healthy","description":"Akka cluster member status: Up","durationMs":0.0144,"data":{"selfAddress":"akka.tcp://otopcua@site-a-2:4053","selfRoles":["cluster-SITE-A","driver"],"memberCount":2,"unreachableCount":0,"leader":"akka.tcp://otopcua@site-a-1:4053"}},"cluster-primary":{"status":"Unhealthy","description":"Standby: active node for role \u0027driver\u0027 is akka.tcp://otopcua@site-a-1:4053.","durationMs":0.0364,"data":{"selfAddress":"akka.tcp://otopcua@site-a-2:4053","activeRole":"driver","activeNode":"akka.tcp://otopcua@site-a-1:4053"}},"localdb-replication":{"status":"Healthy","description":"LocalDb replication is connected; oplog backlog 0.","durationMs":0.1502}}} +HTTP 503 +== site-b-1:8080 /health/active +{"status":"Healthy","totalDurationMs":1.2128,"entries":{"opcua-endpoint":{"status":"Healthy","description":"OPC UA endpoint is listening","durationMs":0.0012,"data":{"status":"Started"}},"akka":{"status":"Healthy","description":"Akka cluster member status: Up","durationMs":0.014,"data":{"selfAddress":"akka.tcp://otopcua@site-b-1:4053","selfRoles":["cluster-SITE-B","driver"],"memberCount":1,"unreachableCount":0,"leader":"akka.tcp://otopcua@site-b-1:4053"}},"cluster-primary":{"status":"Healthy","description":"Active node for role \u0027driver\u0027 (oldest Up member).","durationMs":0.0217,"data":{"selfAddress":"akka.tcp://otopcua@site-b-1:4053","activeRole":"driver","activeNode":"akka.tcp://otopcua@site-b-1:4053"}},"localdb-replication":{"status":"Healthy","description":"LocalDb replication is not configured on this node (default-OFF).","durationMs":0.7106}}} +HTTP 200 +== site-b-2:8080 /health/active + +HTTP 000 diff --git a/docs/plans/evidence/2026-11-12-overview-phase4/split-timeline.txt b/docs/plans/evidence/2026-11-12-overview-phase4/split-timeline.txt new file mode 100644 index 0000000..b635e67 --- /dev/null +++ b/docs/plans/evidence/2026-11-12-overview-phase4/split-timeline.txt @@ -0,0 +1,40 @@ +# All times UTC 2026-11-13 (= 2026-11-12 evening US Eastern). Probe: GET /health/active from inside the rig +# network (site-b-2 via the g761-probe network while partitioned). Format: node HTTP members unreachable leader | cluster-primary. +02:49:45 site-b-2 disconnected from otopcua-g761_default (stays reachable to the dashboard via g761-probe) +site-b-1 200 members 2 unreach 0 leader akka.tcp://otopcua@site-b-1:4053 | Active node for role 'driver' (oldest Up member). +site-b-2 503 members 2 unreach 0 leader akka.tcp://otopcua@site-b-1:4053 | Standby: active node for role 'driver' is akka.tcp://otopcua@site-b-1:4053. +02:49:54 +site-b-1 200 members 2 unreach 1 leader akka.tcp://otopcua@site-b-1:4053 | Active node for role 'driver' (oldest Up member). +site-b-2 503 members 2 unreach 1 leader akka.tcp://otopcua@site-b-2:4053 | Standby: active node for role 'driver' is akka.tcp://otopcua@site-b-1:4053. +02:50:05 +site-b-1 200 members 1 unreach 0 leader akka.tcp://otopcua@site-b-1:4053 | Active node for role 'driver' (oldest Up member). +site-b-2 200 members 1 unreach 0 leader akka.tcp://otopcua@site-b-2:4053 | Active node for role 'driver' (oldest Up member). +02:50:18 (unchanged at 02:50:30, 02:50:41, 02:50:53; screenshot 14-check2-splitbrain.png at 02:51:10) +healed network at 02:51:41 +site-b-1 200 members 1 unreach 0 leader akka.tcp://otopcua@site-b-1:4053 | Active node for role 'driver' (oldest Up member). +site-b-2 200 members 1 unreach 0 leader akka.tcp://otopcua@site-b-2:4053 | Active node for role 'driver' (oldest Up member). +02:51:43 +site-b-1 200 members 1 unreach 0 leader akka.tcp://otopcua@site-b-1:4053 | Active node for role 'driver' (oldest Up member). +site-b-2 200 members 1 unreach 0 leader akka.tcp://otopcua@site-b-2:4053 | Active node for role 'driver' (oldest Up member). +02:51:54 +site-b-1 200 members 1 unreach 0 leader akka.tcp://otopcua@site-b-1:4053 | Active node for role 'driver' (oldest Up member). +site-b-2 200 members 1 unreach 0 leader akka.tcp://otopcua@site-b-2:4053 | Active node for role 'driver' (oldest Up member). +02:52:05 +site-b-1 200 members 1 unreach 0 leader akka.tcp://otopcua@site-b-1:4053 | Active node for role 'driver' (oldest Up member). +site-b-2 200 members 1 unreach 0 leader akka.tcp://otopcua@site-b-2:4053 | Active node for role 'driver' (oldest Up member). +02:52:16 +site-b-1 200 members 1 unreach 0 leader akka.tcp://otopcua@site-b-1:4053 | Active node for role 'driver' (oldest Up member). +site-b-2 200 members 1 unreach 0 leader akka.tcp://otopcua@site-b-2:4053 | Active node for role 'driver' (oldest Up member). +02:52:27 +site-b-1 200 members 1 unreach 0 leader akka.tcp://otopcua@site-b-1:4053 | Active node for role 'driver' (oldest Up member). +site-b-2 200 members 1 unreach 0 leader akka.tcp://otopcua@site-b-2:4053 | Active node for role 'driver' (oldest Up member). +02:52:38 +site-b-1 200 members 1 unreach 0 leader akka.tcp://otopcua@site-b-1:4053 | Active node for role 'driver' (oldest Up member). +site-b-2 200 members 1 unreach 0 leader akka.tcp://otopcua@site-b-2:4053 | Active node for role 'driver' (oldest Up member). +02:52:50 +restarted site-b-2 at 02:53:10 +site-b-1 200 members 2 unreach 0 leader akka.tcp://otopcua@site-b-1:4053 | Active node for role 'driver' (oldest Up member). +site-b-2 503 members 2 unreach 0 leader akka.tcp://otopcua@site-b-1:4053 | Standby: active node for role 'driver' is akka.tcp://otopcua@site-b-1:4053. +02:53:25 +stop site-a-1 at 02:53:44 +start site-a-1 at 02:54:24