4909946104
Re-ran the two design 9 checks that were blocked for OtOpcUa on 2026-07-24, against a docker-dev rig from OtOpcUa master 412da27c (Health 0.3.0) with the dashboard in a container on the rig network: 1. one Active per pair: exactly one /health/active 200 per mesh (MAIN central-2, SITE-A site-a-1, SITE-B site-b-1), dashboard shows one ACTIVE + one STANDBY each; 2. leader chip on every group; network-partitioning site-b-2 from site-b-1 produced two 1-member clusters, both Active, and the SPLIT BRAIN chip, which cleared once site-b-2 was restarted (healing the link alone does not merge two Akka clusters, the documented OtOpcUa recovery); stopping site-a-1 moved the SITE-A chip. Evidence (screenshots, /health/active bodies, timeline) under docs/plans/evidence/2026-11-12-overview-phase4/. Ledger task 20 -> completed (20/20). Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi
66 lines
5.1 KiB
Markdown
66 lines
5.1 KiB
Markdown
# Overview Phase 4 — the two OtOpcUa-blocked §9 checks, re-run 2026-11-12 (scadaproj#14)
|
||
|
||
Design §9 (`docs/plans/2026-07-22-overview-dashboard-design.md`), plan task 20. On 2026-07-24 two of the nine checks
|
||
passed for ScadaBridge but were blocked for OtOpcUa (pre-Health-0.2.0 build: every node Active, no `data`, no leader
|
||
chip). Both are re-run here against a current OtOpcUa rig. **Result: both PASS.**
|
||
|
||
## Setup
|
||
|
||
| | |
|
||
|---|---|
|
||
| OtOpcUa | docker-dev rig built from `master` `412da27c` (Telemetry 0.3.0; pins `ZB.MOM.WW.Health` 0.3.0), run as its own compose project `otopcua-g761` (remapped host ports, image `otopcua-host:g761`) so it did not touch the shared `otopcua-dev` rig. Three independent 2-node meshes: MAIN `central-1/2`, SITE-A `site-a-1/2`, SITE-B `site-b-1/2`. |
|
||
| Overview | scadaproj `main` `d93ca7b` (Overview last changed `6123d03`), published framework-dependent and run in an `aspnet:10.0` container joined to the rig network (`otopcua-g761_default`), so every node is addressed by container DNS name. Leader resolution is per group and host-only, so a host-run dashboard on `localhost:*` URLs could not tell pair members apart. Registry: six instances, three groups (MAIN on `:9000`, SITE-A/SITE-B on `:8080`), `HasActiveRole: true`, poll 5 s. |
|
||
| Clock | All times UTC 2026-11-13 02:49–02:55 (= 2026-11-12 evening US Eastern). |
|
||
|
||
## Check 1 — every Active/Standby pair shows exactly one Active — PASS
|
||
|
||
`GET /health/active` on all six nodes (raw bodies: [`active-baseline.txt`](active-baseline.txt)):
|
||
|
||
| node | HTTP | akka `memberCount` | akka `leader` | `cluster-primary` |
|
||
|---|---|---|---|---|
|
||
| central-1 | 503 | 2 | central-1 | Standby: active node for role 'admin' is central-2 |
|
||
| central-2 | **200** | 2 | central-1 | Active node for role 'admin' (oldest Up member) |
|
||
| site-a-1 | **200** | 2 | site-a-1 | Active node for role 'driver' (oldest Up member) |
|
||
| site-a-2 | 503 | 2 | site-a-1 | Standby: active node for role 'driver' is site-a-1 |
|
||
| site-b-1 | **200** | 2 | site-b-1 | Active node for role 'driver' (oldest Up member) |
|
||
| site-b-2 | 503 | 2 | site-b-1 | Standby: active node for role 'driver' is site-b-1 |
|
||
|
||
Exactly one 200 per mesh. The dashboard renders one ACTIVE + one STANDBY in all three groups
|
||
([`14-check1-baseline.png`](14-check1-baseline.png)). MAIN shows the documented leader/active divergence: Akka leader
|
||
`central-1` (lowest address) carries LEADER + STANDBY, and `central-2` (oldest Up `admin` member, the singleton host)
|
||
is ACTIVE.
|
||
|
||
## Check 2 — leader chip, and a split-brain chip that appears and clears — PASS
|
||
|
||
- **Leader chip.** Every group shows `LEADER · <node>` from the akka entry's `data.leader` (MAIN `central-1`,
|
||
SITE-A `site-a-1`, SITE-B `site-b-1`) — [`14-check1-baseline.png`](14-check1-baseline.png).
|
||
- **Split brain forced.** At 02:49:45 `site-b-2` was disconnected from the rig network (`docker network disconnect`),
|
||
after first being attached to a private network `g761-probe` (alias `site-b-2`) shared only with the dashboard, so
|
||
the pair lost each other but the dashboard kept reaching both. Within ~25 s each side downed the other (auto-down,
|
||
the accepted availability-over-partition-safety trade): both became 1-member clusters, both answered
|
||
`/health/active` 200 and each named itself leader ([`split-timeline.txt`](split-timeline.txt)). The dashboard showed
|
||
`SPLIT BRAIN · AKKA.TCP://OTOPCUA@SITE-B-1:4053 VS AKKA.TCP://OTOPCUA@SITE-B-2:4053` on SITE-B with both cards
|
||
ACTIVE ([`14-check2-splitbrain.png`](14-check2-splitbrain.png)).
|
||
- **Clears.** Reconnecting the network (02:51:41) does **not** end the split. Two formed Akka clusters never merge,
|
||
and both nodes stayed 1-member and Active for the next 70 s. This is the documented recovery in OtOpcUa
|
||
`docs/Redundancy.md` ("after the link heals, restart one side"), not a dashboard or product defect. After
|
||
`docker restart` of `site-b-2` (02:53:10) it rejoined; at 02:53:25 site-b-1 was 200 and site-b-2 503, both
|
||
`memberCount` 2, both naming `site-b-1` leader, and the chip was gone ([`14-check2-cleared.png`](14-check2-cleared.png);
|
||
raw [`active-after-rejoin.txt`](active-after-rejoin.txt): three 200s, one per mesh).
|
||
- **Leader chip moves** (the §9 wording). Stopping SITE-A's leader `site-a-1` (02:53:44) moved the chip to
|
||
`LEADER · SITE-A-2`, with `site-a-2` ACTIVE and `site-a-1` UNREACHABLE ([`14-check2-leader-moved.png`](14-check2-leader-moved.png)).
|
||
`site-a-1` was restarted at 02:54:24 and rejoined.
|
||
|
||
## Observations (none is a defect)
|
||
|
||
- The group header's "N members · M unreachable" counts **registered** instances (`Overview.razor` `group.Count()`),
|
||
not Akka membership, so it read "2 members · 0 unreachable" during the split while each node reported
|
||
`memberCount` 1. Akka's own count is on each card's check detail. Working as designed.
|
||
- Only the resolved group leader gets a LEADER badge; during the split `site-b-2` (which named itself leader) had
|
||
none, and the disagreement is carried by the split-brain chip instead.
|
||
|
||
## Cleanup
|
||
|
||
The `otopcua-g761` rig, the `g761-probe` network and the dashboard container were removed after the
|
||
lmxopcua#761 gate run that shared the rig.
|