From a78425ea8ff0056d1ff9f599b64f1dce2d5202f2 Mon Sep 17 00:00:00 2001 From: Joseph Doherty Date: Wed, 22 Jul 2026 07:19:34 -0400 Subject: [PATCH] =?UTF-8?q?docs(plans):=20record=20the=20Task=208=20live?= =?UTF-8?q?=20finding=20=E2=80=94=20self-form=20races=20the=20join=20hands?= =?UTF-8?q?hake?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Corrects the plan's behavior table (the mid-join-handshake row was FALSE) and notes the mesh-program Phase 6 convergence on ScadaBridge's self-first seed ordering, which retires the watchdog + guard entirely. Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW --- docs/plans/2026-07-22-per-cluster-mesh-program.md | 8 ++++++-- ...026-07-22-selfform-fallback-and-manual-failover.md | 2 +- ...elfform-fallback-and-manual-failover.md.tasks.json | 11 ++++++----- 3 files changed, 13 insertions(+), 8 deletions(-) diff --git a/docs/plans/2026-07-22-per-cluster-mesh-program.md b/docs/plans/2026-07-22-per-cluster-mesh-program.md index e1ef53ef..7b7bbacd 100644 --- a/docs/plans/2026-07-22-per-cluster-mesh-program.md +++ b/docs/plans/2026-07-22-per-cluster-mesh-program.md @@ -143,8 +143,12 @@ observability). kill-and-reconnect of the central dialer recovers every stream. ### Phase 6 — Mesh partition + co-location topology -**Scope:** per-cluster seed nodes (each pair node lists itself + partner — the `SelfFormAfter` -fallback then covers both), cluster-scoped roles `cluster-{ClusterId}` + singleton re-scoping, +**Scope:** per-cluster seed nodes — **adopt ScadaBridge's self-first ordering (each node lists +ITSELF as `seed-nodes[0]`, partner second) and RETIRE the `SelfFormAfter` watchdog + TCP +reachability guard** (2026-07-22 execution finding: the watchdog races Akka's join handshake — +`Join(self)` during an in-flight join islands the node, hit live here and fixed with the guard; +ScadaBridge proved self-first ordering is the race-free form of the same semantics, enforced by +a startup-validator rule — port that rule too); cluster-scoped roles `cluster-{ClusterId}` + singleton re-scoping, central pair keeps the admin singletons; **docker-dev rig rewritten** to model the real topology — including the co-location port table above (both products' compose files on shared per-site networks, real LocalDb sync ports); remove the ClusterRedundancy page's mesh-scope caveat (the diff --git a/docs/plans/2026-07-22-selfform-fallback-and-manual-failover.md b/docs/plans/2026-07-22-selfform-fallback-and-manual-failover.md index d7d2a30d..9a89c3fe 100644 --- a/docs/plans/2026-07-22-selfform-fallback-and-manual-failover.md +++ b/docs/plans/2026-07-22-selfform-fallback-and-manual-failover.md @@ -30,7 +30,7 @@ | Peer boots after survivor self-formed | Its InitJoin is answered → joins as youngest. No island. | | Both pair nodes cold-start simultaneously, mutually unreachable | Both self-form → dual-active (same partition class auto-down accepts; restart one side) | | `SelfFormAfter` null/`≤0` | Disabled — today's wait-forever behavior | -| Window expires mid-join-handshake | Benign: Akka ignores `Join` once joined | +| ~~Window expires mid-join-handshake~~ | ~~Benign: Akka ignores `Join` once joined~~ **FALSE — proven live (Task 8): `Join(self)` during an in-flight join abandons it and islands the node. Fixed by the TCP reachability guard: never self-form while any seed peer accepts a connection; wait a window and re-arm. ScadaBridge rejected the watchdog entirely for self-first seed ordering; OtOpcUa converges at mesh-program Phase 6.** | **Manual failover rules:** graceful `Leave` of the **oldest Up `driver` member** (identical query to `RedundancyStateActor.SelectDriverPrimary` — `Member.AgeOrdering`, `Leaving` excluded via `Status == Up`), never `Down`. The leaving node hands over via the cluster-leave phases, `CoordinatedShutdown` runs, `ActorSystemTerminationWatchdog` exits the process, the supervisor restarts it, and it rejoins as youngest → the survivor is Primary (ServiceLevel 250 moves; OPC UA clients re-select). Admin-only; peer guard (<2 Up driver members → disabled); confirmation dialog; audited via the shared `ZB.MOM.WW.Audit.IAuditWriter` seam before the Leave. **Mesh-scope caveat:** until the per-cluster mesh lands, the election is mesh-wide — the button acts on THE Primary of the whole Akka cluster, not per-`ServerCluster` row; the UI must say so (mirrors the KNOWN LIMITATION in `docs/Redundancy.md:84-99`). diff --git a/docs/plans/2026-07-22-selfform-fallback-and-manual-failover.md.tasks.json b/docs/plans/2026-07-22-selfform-fallback-and-manual-failover.md.tasks.json index 2ca29190..e301e5d1 100644 --- a/docs/plans/2026-07-22-selfform-fallback-and-manual-failover.md.tasks.json +++ b/docs/plans/2026-07-22-selfform-fallback-and-manual-failover.md.tasks.json @@ -3,11 +3,12 @@ "tasks": [ {"id": 0, "subject": "Task 1: SelfFormAfter on AkkaClusterOptions", "status": "completed"}, {"id": 1, "subject": "Task 2: ClusterBootstrapFallback + AddStartup wiring", "status": "completed", "blockedBy": [0]}, - {"id": 2, "subject": "Task 3: SelfFormBootstrapTests — 4 scenarios incl. site-node island guard", "status": "completed", "blockedBy": [1], "notes": "Teeth proven: scenario 2 run with selfFormAfter:null failed (red) before restore."}, - {"id": 3, "subject": "Task 4: Docs — Redundancy.md bootstrap section + mesh design note", "status": "pending", "blockedBy": [1]}, - {"id": 4, "subject": "Task 5: Full verification + optional docker-dev live check", "status": "pending", "blockedBy": [2, 3]}, - {"id": 5, "subject": "Task 6: Manual failover service + cluster-level test", "status": "pending", "blockedBy": [4]}, - {"id": 6, "subject": "Task 7: ClusterRedundancy live panel + failover button + bUnit tests", "status": "pending", "blockedBy": [5]} + {"id": 2, "subject": "Task 3: SelfFormBootstrapTests — 4 scenarios incl. site-node island guard", "status": "completed", "blockedBy": [1], "notes": "Teeth proven: scenario 2 run with selfFormAfter:null failed (red) before restore. A 5th test was added by Task 8."}, + {"id": 3, "subject": "Task 4: Docs — Redundancy.md bootstrap section + mesh design note", "status": "completed", "blockedBy": [1]}, + {"id": 4, "subject": "Task 5: Full verification + optional docker-dev live check", "status": "completed", "blockedBy": [2, 3], "notes": "Live gate PASSED for the cold-start case (defect observed on old image, fix on new, island guard on a real site node, one six-node cluster after recovery)."}, + {"id": 5, "subject": "Task 6: Manual failover service + cluster-level test", "status": "completed", "blockedBy": [4], "notes": "4/4. Teeth proven: sabotaging FailOverCore to address order turned the target + parity tests red."}, + {"id": 6, "subject": "Task 7: ClusterRedundancy live panel + failover button + tests", "status": "completed", "blockedBy": [5], "notes": "PLAN DEVIATION: plan said bUnit; repo has no bUnit (PageAuthorizationGuardTests documents itself as the substitute). Adapted to a pure ManualFailoverPageModel + 9 unit tests, razor as thin shell. Policy is FleetAdmin, not the ConfigEditor the neighbouring cluster pages use (this restarts a node; ConfigEditor admits Designer). Live-verified: panel, confirm dialog, execution, singleton handover, audit row."}, + {"id": 7, "subject": "Task 8 (NEW, from live gate): reachability guard — don't self-form while a seed peer is reachable", "status": "completed", "blockedBy": [6], "notes": "DEFECT: the Task 7 live drill caught the self-form fallback islanding the node the failover bounced. central-1 restarted, got InitJoinAck from central-2 at 10:49:05 (join in flight, healthy) but no Welcome inside the window (peer's ring still held the old incarnation until 10:49:06); at 10:49:15 the fallback fired and formed a SECOND cluster. The plan's 'Window expires mid-join-handshake -> benign, Akka ignores Join once joined' row is FALSE — the node had not joined, and Join(self) wins. Manual failover deliberately produces that restart, so the two halves of the plan collided. FIX (user chose the reachability guard): TCP-probe the other seed addresses before self-forming; a reachable peer means wait another window and re-arm. Proven three ways: (1) new regression test whose 'peer' is a bare TcpListener speaking no Akka — RED without the guard; (2) live, a throwaway node seeded at sql:1433 (TCP-reachable, not Akka) logged the guard every 10s and never self-formed; (3) positive control, the same node seeded at a dead port self-formed at +10s. Failover re-drill on the rig: no island, all six nodes one cluster."} ], "lastUpdated": "2026-07-22T00:00:00Z" }