docs(plans): record the Task 8 live finding — self-form races the join handshake

Corrects the plan's behavior table (the mid-join-handshake row was FALSE) and
notes the mesh-program Phase 6 convergence on ScadaBridge's self-first seed
ordering, which retires the watchdog + guard entirely.

Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW

docs/plans/2026-07-22-per-cluster-mesh-program.md

@@ -143,8 +143,12 @@ observability).
 kill-and-reconnect of the central dialer recovers every stream.
 
 ### Phase 6 — Mesh partition + co-location topology
-**Scope:** per-cluster seed nodes (each pair node lists itself + partner — the `SelfFormAfter`
+**Scope:** per-cluster seed nodes — **adopt ScadaBridge's self-first ordering (each node lists
-fallback then covers both), cluster-scoped roles `cluster-{ClusterId}` + singleton re-scoping,
+ITSELF as `seed-nodes[0]`, partner second) and RETIRE the `SelfFormAfter` watchdog + TCP
+reachability guard** (2026-07-22 execution finding: the watchdog races Akka's join handshake —
+`Join(self)` during an in-flight join islands the node, hit live here and fixed with the guard;
+ScadaBridge proved self-first ordering is the race-free form of the same semantics, enforced by
+a startup-validator rule — port that rule too); cluster-scoped roles `cluster-{ClusterId}` + singleton re-scoping,
 central pair keeps the admin singletons; **docker-dev rig rewritten** to model the real topology —
 including the co-location port table above (both products' compose files on shared per-site
 networks, real LocalDb sync ports); remove the ClusterRedundancy page's mesh-scope caveat (the

docs/plans/2026-07-22-selfform-fallback-and-manual-failover.md

@@ -30,7 +30,7 @@
 | Peer boots after survivor self-formed | Its InitJoin is answered → joins as youngest. No island. |
 | Both pair nodes cold-start simultaneously, mutually unreachable | Both self-form → dual-active (same partition class auto-down accepts; restart one side) |
 | `SelfFormAfter` null/`≤0` | Disabled — today's wait-forever behavior |
-| Window expires mid-join-handshake | Benign: Akka ignores `Join` once joined |
+| ~~Window expires mid-join-handshake~~ | ~~Benign: Akka ignores `Join` once joined~~ **FALSE — proven live (Task 8): `Join(self)` during an in-flight join abandons it and islands the node. Fixed by the TCP reachability guard: never self-form while any seed peer accepts a connection; wait a window and re-arm. ScadaBridge rejected the watchdog entirely for self-first seed ordering; OtOpcUa converges at mesh-program Phase 6.** |
 
 **Manual failover rules:** graceful `Leave` of the **oldest Up `driver` member** (identical query to `RedundancyStateActor.SelectDriverPrimary` — `Member.AgeOrdering`, `Leaving` excluded via `Status == Up`), never `Down`. The leaving node hands over via the cluster-leave phases, `CoordinatedShutdown` runs, `ActorSystemTerminationWatchdog` exits the process, the supervisor restarts it, and it rejoins as youngest → the survivor is Primary (ServiceLevel 250 moves; OPC UA clients re-select). Admin-only; peer guard (<2 Up driver members → disabled); confirmation dialog; audited via the shared `ZB.MOM.WW.Audit.IAuditWriter` seam before the Leave. **Mesh-scope caveat:** until the per-cluster mesh lands, the election is mesh-wide — the button acts on THE Primary of the whole Akka cluster, not per-`ServerCluster` row; the UI must say so (mirrors the KNOWN LIMITATION in `docs/Redundancy.md:84-99`).
 

docs/plans/2026-07-22-selfform-fallback-and-manual-failover.md.tasks.json

@@ -3,11 +3,12 @@
   "tasks": [
     {"id": 0, "subject": "Task 1: SelfFormAfter on AkkaClusterOptions", "status": "completed"},
     {"id": 1, "subject": "Task 2: ClusterBootstrapFallback + AddStartup wiring", "status": "completed", "blockedBy": [0]},
-    {"id": 2, "subject": "Task 3: SelfFormBootstrapTests — 4 scenarios incl. site-node island guard", "status": "completed", "blockedBy": [1], "notes": "Teeth proven: scenario 2 run with selfFormAfter:null failed (red) before restore."},
+    {"id": 2, "subject": "Task 3: SelfFormBootstrapTests — 4 scenarios incl. site-node island guard", "status": "completed", "blockedBy": [1], "notes": "Teeth proven: scenario 2 run with selfFormAfter:null failed (red) before restore. A 5th test was added by Task 8."},
-    {"id": 3, "subject": "Task 4: Docs — Redundancy.md bootstrap section + mesh design note", "status": "pending", "blockedBy": [1]},
+    {"id": 3, "subject": "Task 4: Docs — Redundancy.md bootstrap section + mesh design note", "status": "completed", "blockedBy": [1]},
-    {"id": 4, "subject": "Task 5: Full verification + optional docker-dev live check", "status": "pending", "blockedBy": [2, 3]},
+    {"id": 4, "subject": "Task 5: Full verification + optional docker-dev live check", "status": "completed", "blockedBy": [2, 3], "notes": "Live gate PASSED for the cold-start case (defect observed on old image, fix on new, island guard on a real site node, one six-node cluster after recovery)."},
-    {"id": 5, "subject": "Task 6: Manual failover service + cluster-level test", "status": "pending", "blockedBy": [4]},
+    {"id": 5, "subject": "Task 6: Manual failover service + cluster-level test", "status": "completed", "blockedBy": [4], "notes": "4/4. Teeth proven: sabotaging FailOverCore to address order turned the target + parity tests red."},
-    {"id": 6, "subject": "Task 7: ClusterRedundancy live panel + failover button + bUnit tests", "status": "pending", "blockedBy": [5]}
+    {"id": 6, "subject": "Task 7: ClusterRedundancy live panel + failover button + tests", "status": "completed", "blockedBy": [5], "notes": "PLAN DEVIATION: plan said bUnit; repo has no bUnit (PageAuthorizationGuardTests documents itself as the substitute). Adapted to a pure ManualFailoverPageModel + 9 unit tests, razor as thin shell. Policy is FleetAdmin, not the ConfigEditor the neighbouring cluster pages use (this restarts a node; ConfigEditor admits Designer). Live-verified: panel, confirm dialog, execution, singleton handover, audit row."},
+    {"id": 7, "subject": "Task 8 (NEW, from live gate): reachability guard — don't self-form while a seed peer is reachable", "status": "completed", "blockedBy": [6], "notes": "DEFECT: the Task 7 live drill caught the self-form fallback islanding the node the failover bounced. central-1 restarted, got InitJoinAck from central-2 at 10:49:05 (join in flight, healthy) but no Welcome inside the window (peer's ring still held the old incarnation until 10:49:06); at 10:49:15 the fallback fired and formed a SECOND cluster. The plan's 'Window expires mid-join-handshake -> benign, Akka ignores Join once joined' row is FALSE — the node had not joined, and Join(self) wins. Manual failover deliberately produces that restart, so the two halves of the plan collided. FIX (user chose the reachability guard): TCP-probe the other seed addresses before self-forming; a reachable peer means wait another window and re-arm. Proven three ways: (1) new regression test whose 'peer' is a bare TcpListener speaking no Akka — RED without the guard; (2) live, a throwaway node seeded at sql:1433 (TCP-reachable, not Akka) logged the guard every 10s and never self-formed; (3) positive control, the same node seeded at a dead port self-formed at +10s. Failover re-drill on the rig: no island, all six nodes one cluster."}
   ],
   "lastUpdated": "2026-07-22T00:00:00Z"
 }