Files
scadaproj/docs/plans/evidence/2026-11-12-overview-phase4/README.md
T
Joseph Doherty 4909946104 docs(overview): Phase 4 live acceptance complete - OtOpcUa one-Active + leader/split-brain chips pass (#14)
Re-ran the two design 9 checks that were blocked for OtOpcUa on 2026-07-24, against a
docker-dev rig from OtOpcUa master 412da27c (Health 0.3.0) with the dashboard in a
container on the rig network:

1. one Active per pair: exactly one /health/active 200 per mesh (MAIN central-2,
   SITE-A site-a-1, SITE-B site-b-1), dashboard shows one ACTIVE + one STANDBY each;
2. leader chip on every group; network-partitioning site-b-2 from site-b-1 produced
   two 1-member clusters, both Active, and the SPLIT BRAIN chip, which cleared once
   site-b-2 was restarted (healing the link alone does not merge two Akka clusters,
   the documented OtOpcUa recovery); stopping site-a-1 moved the SITE-A chip.

Evidence (screenshots, /health/active bodies, timeline) under
docs/plans/evidence/2026-11-12-overview-phase4/. Ledger task 20 -> completed (20/20).

Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi
2026-11-12 22:47:13 -05:00

66 lines
5.1 KiB
Markdown
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
# Overview Phase 4 — the two OtOpcUa-blocked §9 checks, re-run 2026-11-12 (scadaproj#14)
Design §9 (`docs/plans/2026-07-22-overview-dashboard-design.md`), plan task 20. On 2026-07-24 two of the nine checks
passed for ScadaBridge but were blocked for OtOpcUa (pre-Health-0.2.0 build: every node Active, no `data`, no leader
chip). Both are re-run here against a current OtOpcUa rig. **Result: both PASS.**
## Setup
| | |
|---|---|
| OtOpcUa | docker-dev rig built from `master` `412da27c` (Telemetry 0.3.0; pins `ZB.MOM.WW.Health` 0.3.0), run as its own compose project `otopcua-g761` (remapped host ports, image `otopcua-host:g761`) so it did not touch the shared `otopcua-dev` rig. Three independent 2-node meshes: MAIN `central-1/2`, SITE-A `site-a-1/2`, SITE-B `site-b-1/2`. |
| Overview | scadaproj `main` `d93ca7b` (Overview last changed `6123d03`), published framework-dependent and run in an `aspnet:10.0` container joined to the rig network (`otopcua-g761_default`), so every node is addressed by container DNS name. Leader resolution is per group and host-only, so a host-run dashboard on `localhost:*` URLs could not tell pair members apart. Registry: six instances, three groups (MAIN on `:9000`, SITE-A/SITE-B on `:8080`), `HasActiveRole: true`, poll 5 s. |
| Clock | All times UTC 2026-11-13 02:49–02:55 (= 2026-11-12 evening US Eastern). |
## Check 1 — every Active/Standby pair shows exactly one Active — PASS
`GET /health/active` on all six nodes (raw bodies: [`active-baseline.txt`](active-baseline.txt)):
| node | HTTP | akka `memberCount` | akka `leader` | `cluster-primary` |
|---|---|---|---|---|
| central-1 | 503 | 2 | central-1 | Standby: active node for role 'admin' is central-2 |
| central-2 | **200** | 2 | central-1 | Active node for role 'admin' (oldest Up member) |
| site-a-1 | **200** | 2 | site-a-1 | Active node for role 'driver' (oldest Up member) |
| site-a-2 | 503 | 2 | site-a-1 | Standby: active node for role 'driver' is site-a-1 |
| site-b-1 | **200** | 2 | site-b-1 | Active node for role 'driver' (oldest Up member) |
| site-b-2 | 503 | 2 | site-b-1 | Standby: active node for role 'driver' is site-b-1 |
Exactly one 200 per mesh. The dashboard renders one ACTIVE + one STANDBY in all three groups
([`14-check1-baseline.png`](14-check1-baseline.png)). MAIN shows the documented leader/active divergence: Akka leader
`central-1` (lowest address) carries LEADER + STANDBY, and `central-2` (oldest Up `admin` member, the singleton host)
is ACTIVE.
## Check 2 — leader chip, and a split-brain chip that appears and clears — PASS
- **Leader chip.** Every group shows `LEADER · <node>` from the akka entry's `data.leader` (MAIN `central-1`,
SITE-A `site-a-1`, SITE-B `site-b-1`) — [`14-check1-baseline.png`](14-check1-baseline.png).
- **Split brain forced.** At 02:49:45 `site-b-2` was disconnected from the rig network (`docker network disconnect`),
after first being attached to a private network `g761-probe` (alias `site-b-2`) shared only with the dashboard, so
the pair lost each other but the dashboard kept reaching both. Within ~25 s each side downed the other (auto-down,
the accepted availability-over-partition-safety trade): both became 1-member clusters, both answered
`/health/active` 200 and each named itself leader ([`split-timeline.txt`](split-timeline.txt)). The dashboard showed
`SPLIT BRAIN · AKKA.TCP://OTOPCUA@SITE-B-1:4053 VS AKKA.TCP://OTOPCUA@SITE-B-2:4053` on SITE-B with both cards
ACTIVE ([`14-check2-splitbrain.png`](14-check2-splitbrain.png)).
- **Clears.** Reconnecting the network (02:51:41) does **not** end the split. Two formed Akka clusters never merge,
and both nodes stayed 1-member and Active for the next 70 s. This is the documented recovery in OtOpcUa
`docs/Redundancy.md` ("after the link heals, restart one side"), not a dashboard or product defect. After
`docker restart` of `site-b-2` (02:53:10) it rejoined; at 02:53:25 site-b-1 was 200 and site-b-2 503, both
`memberCount` 2, both naming `site-b-1` leader, and the chip was gone ([`14-check2-cleared.png`](14-check2-cleared.png);
raw [`active-after-rejoin.txt`](active-after-rejoin.txt): three 200s, one per mesh).
- **Leader chip moves** (the §9 wording). Stopping SITE-A's leader `site-a-1` (02:53:44) moved the chip to
`LEADER · SITE-A-2`, with `site-a-2` ACTIVE and `site-a-1` UNREACHABLE ([`14-check2-leader-moved.png`](14-check2-leader-moved.png)).
`site-a-1` was restarted at 02:54:24 and rejoined.
## Observations (none is a defect)
- The group header's "N members · M unreachable" counts **registered** instances (`Overview.razor` `group.Count()`),
not Akka membership, so it read "2 members · 0 unreachable" during the split while each node reported
`memberCount` 1. Akka's own count is on each card's check detail. Working as designed.
- Only the resolved group leader gets a LEADER badge; during the split `site-b-2` (which named itself leader) had
none, and the disagreement is carried by the split-brain chip instead.
## Cleanup
The `otopcua-g761` rig, the `g761-probe` network and the dashboard container were removed after the
lmxopcua#761 gate run that shared the rig.