docs(cluster): self-first seed ordering replaces the self-form fallback

docs/Redundancy.md's bootstrap section is rewritten around the mechanism that is
actually in force: which of Akka's two bootstrap processes runs is decided by
whether seed-nodes[0] is this node's own address, so the ORDER of Cluster:SeedNodes
is the fix, not a timer. Adds the per-process table, the shipped self-first
example, why the validator is conditional (site nodes are not seeds) and why it
matches PublicHostname rather than the 0.0.0.0 bind address.

The retirement is documented rather than erased: a new subsection explains that a
watchdog outside the join handshake cannot tell "no seed answered" from "a seed
answered and the join is in flight", and the ea45ace1 live-gate finding is kept
verbatim as the evidence that produced that conclusion (InitJoinAck at 10:49:05,
old incarnation retired 10:49:06, watchdog fired 10:49:15, second cluster). The
watchdog's own earlier PASS is kept too — it did pass what it was gated on.

Also: a live gate for the NEW mechanism is registered as outstanding (not
re-drilled on the docker-dev rig since the swap), and the plan doc that shipped
the watchdog gets a superseded banner rather than an edit, so the execution record
and its Task 8 finding stay readable.

Ripple: CLAUDE.md cluster section, docs/Configuration.md + docs/v2/Cluster.md +
docs/ServiceHosting.md SeedNodes entries, the per-cluster-mesh design doc's two
"CLOSED by SelfFormAfter" notes, and mesh-program Phase 6 (which had this
convergence queued — now marked done early) + Phase 7's live-gate name.

docs/plans/2026-07-21-per-cluster-mesh-design.md

@@ -85,11 +85,13 @@ Their failover drill has a mode that exists *"to make the registered gap observa
 it is covered."* **A two-node keep-oldest mesh has an acknowledged total-outage hole.** If OtOpcUa
 adopts 2-node meshes, this must be decided deliberately, not inherited.
 
-> **CLOSED 2026-07-22.** The `InitJoin` half of this gap is fixed in both repos by the self-form
-> fallback: `Cluster:SelfFormAfter` (default 10 s) lets a node that is one of its own seeds form a
-> cluster on itself when no seed answers. See `docs/Redundancy.md` §"Bootstrap: the self-form
-> fallback" — including the island guard that keeps today's non-seed site nodes waiting, which the
-> per-cluster mesh removes by making every pair node a seed of its own mesh.
+> **CLOSED 2026-07-22.** The `InitJoin` half of this gap is fixed in both repos by **self-first seed
+> ordering**: every node that is one of its own seeds lists ITSELF as `seed-nodes[0]`, which is the
+> only ordering under which Akka runs `FirstSeedNodeProcess` and can form a cluster with no peer
+> answering. (An earlier `Cluster:SelfFormAfter` watchdog was tried here and retired — it raced the
+> join handshake.) See `docs/Redundancy.md` §"Bootstrap: self-first seed ordering". Today's
+> driver-only site nodes are seeded by `central-1` only and are exempt; the per-cluster mesh removes
+> even that exception by making every pair node a seed of its own mesh.
 
 ### Both inter-cluster transports are unauthenticated
 
@@ -261,11 +263,14 @@ with Phase 6/7, which makes every mesh exactly two nodes.
 
 **The bootstrap half of the gap is also closed (2026-07-22).** Downing was only ever one of the two
 outage modes: even with `auto-down`, a node cold-starting while its peer was dead looped on `InitJoin`
-forever, because Akka lets only the first listed seed form a new cluster. `Cluster:SelfFormAfter`
-(default 10 s, `ClusterBootstrapFallback`) now self-forms in that case, guarded so a node absent from
-its own seed list never islands itself. Pinned by `SelfFormBootstrapTests`; see `docs/Redundancy.md`
-§"Bootstrap: the self-form fallback". Under this design each pair node is a seed of its own two-node
-mesh, so the fallback covers both sides and the current site-node exception disappears.
+forever, because Akka lets only `seed-nodes[0]` form a new cluster. The fix is **self-first seed
+ordering** — each seed node lists itself first, enforced at boot by `AkkaClusterOptionsValidator`.
+The `Cluster:SelfFormAfter` watchdog first shipped for this is deleted: a timer outside Akka's join
+handshake cannot tell "no seed answered" from "a seed answered and the join is in flight", and it
+islanded a manually-failed-over node live. Pinned by `SelfFirstSeedBootstrapTests`; see
+`docs/Redundancy.md` §"Bootstrap: self-first seed ordering". Under this design each pair node is a
+seed of its own two-node mesh, so the ordering covers both sides and the current site-node exemption
+disappears.
 
 ### 6.3 DECIDED — match ScadaBridge's auth posture for now
 

docs/plans/2026-07-22-per-cluster-mesh-program.md

@@ -46,8 +46,9 @@ Two independent 2-node Akka clusters per site — they share hardware, never a m
 | OtOpcUa LocalDb sync port | OtOpcUa driver | h2c LocalDb pair replication (default off/0 today — Phase 6 assigns a real per-site port) |
 
 **Aligned HA posture (both products, per VM — already true or landing via the selfform plan):**
-auto-down downing (15 s window), oldest-Up active/primary election, `SelfFormAfter` 10 s
-self-form fallback, termination-watchdog → process exit → `sc.exe failure` restart recovery.
+auto-down downing (15 s window), oldest-Up active/primary election, self-first seed ordering
+(the `SelfFormAfter` self-form fallback that briefly stood in for it was retired 2026-07-22),
+termination-watchdog → process exit → `sc.exe failure` restart recovery.
 One failover story for operators regardless of product.
 
 **Prerequisite ordering:** the fallback/manual-failover plan
@@ -143,12 +144,14 @@ observability).
 kill-and-reconnect of the central dialer recovers every stream.
 
 ### Phase 6 — Mesh partition + co-location topology
-**Scope:** per-cluster seed nodes — **adopt ScadaBridge's self-first ordering (each node lists
-ITSELF as `seed-nodes[0]`, partner second) and RETIRE the `SelfFormAfter` watchdog + TCP
-reachability guard** (2026-07-22 execution finding: the watchdog races Akka's join handshake —
-`Join(self)` during an in-flight join islands the node, hit live here and fixed with the guard;
-ScadaBridge proved self-first ordering is the race-free form of the same semantics, enforced by
-a startup-validator rule — port that rule too); cluster-scoped roles `cluster-{ClusterId}` + singleton re-scoping,
+**Scope:** per-cluster seed nodes — ~~adopt ScadaBridge's self-first ordering and RETIRE the
+`SelfFormAfter` watchdog + TCP reachability guard~~ **DONE EARLY 2026-07-22** (docker-dev
+`central-2` swapped to self-first, `ClusterBootstrapFallback`/`SelfFormAfter` deleted,
+`AkkaClusterOptionsValidator` ports ScadaBridge's startup rule in its conditional form, and
+`SelfFirstSeedBootstrapTests` replaces `SelfFormBootstrapTests`; see `docs/Redundancy.md`
+§"Bootstrap: self-first seed ordering"). What remains for this phase is the per-pair
+`Cluster__SeedNodes__*` matrix once the meshes actually split — every node in a pair is then a seed
+of its own mesh, so the site-node exemption disappears; cluster-scoped roles `cluster-{ClusterId}` + singleton re-scoping,
 central pair keeps the admin singletons; **docker-dev rig rewritten** to model the real topology —
 including the co-location port table above (both products' compose files on shared per-site
 networks, real LocalDb sync ports); remove the ClusterRedundancy page's mesh-scope caveat (the
@@ -163,12 +166,13 @@ cross separate meshes).
 ### Phase 7 — Failover drill + live gates
 **Scope:** the drill ScadaBridge already has (`failover-drill.sh` analogue) run per pair, both
 directions; **close the two outstanding live gates**: (a) auto-down 1-vs-1 crash-the-oldest
-(deferred since Phase 0a — finally testable, every mesh is exactly two nodes), (b) `SelfFormAfter`
+(deferred since Phase 0a — finally testable, every mesh is exactly two nodes), (b) self-first
 lone-cold-start on the real per-pair topology; manual-failover button re-verified per pair;
 operator runbook for the co-located site (one page covering both products' failover on the same
 two VMs).
 **Exit gate:** drill green on every pair type (central, site); runbook merged; design doc §7
-table fully marked DONE.
+table fully marked DONE. (Live gate (b) is now the **self-first cold-start-alone** drill, not
+`SelfFormAfter` — the watchdog it named was retired 2026-07-22.)
 
 ---
 

docs/plans/2026-07-22-per-cluster-mesh-program.md.tasks.json

@@ -9,7 +9,7 @@
     {"id": 4, "subject": "Phase 4: cut driver-side ConfigDb — EfAlarmConditionStateStore to LocalDb, DbHealthProbeActor ServiceLevel input, OpcUaPublishActor audit (live gate)", "status": "pending", "blockedBy": [3]},
     {"id": 5, "subject": "Phase 5: gRPC oneof stream contract; migrate 7 observability topics; AdminUI reconnect story", "status": "pending", "blockedBy": [2]},
     {"id": 6, "subject": "Phase 6: mesh partition — per-pair seeds, cluster-scoped roles/singletons, co-location rig rewrite, secrets replication re-scope", "status": "pending", "blockedBy": [0, 4, 5]},
-    {"id": 7, "subject": "Phase 7: failover drill per pair (incl. shared-VM failure) + close auto-down 1v1 and SelfFormAfter live gates + operator runbook", "status": "pending", "blockedBy": [6]}
+    {"id": 7, "subject": "Phase 7: failover drill per pair (incl. shared-VM failure) + close auto-down 1v1 and self-first cold-start-alone live gates + operator runbook", "status": "pending", "blockedBy": [6]}
   ],
   "lastUpdated": "2026-07-22T00:00:00Z"
 }

docs/plans/2026-07-22-selfform-fallback-and-manual-failover.md

@@ -1,5 +1,22 @@
 # OtOpcUa: InitJoin Self-Form Fallback + Manual Failover Control — Implementation Plan
 
+> ## ⚠ SUPERSEDED IN PART (2026-07-22, same day)
+>
+> **Half (1) of this plan — the `Cluster:SelfFormAfter` self-form watchdog — was retired within hours
+> of landing.** `ClusterBootstrapFallback`, the `SelfFormAfter` option and `SelfFormBootstrapTests`
+> are **deleted**; the cold-start-alone goal is now met by **self-first seed ordering** (each seed
+> node lists ITSELF as `seed-nodes[0]`), enforced at boot by `AkkaClusterOptionsValidator` and pinned
+> by `SelfFirstSeedBootstrapTests`. Reason: a timer outside Akka's join handshake cannot distinguish
+> "no seed answered" from "a seed answered and the join is in flight", and `Cluster.Join(SelfAddress)`
+> is not ignored mid-handshake — it wins. Task 8 below caught exactly that live and patched the one
+> observed shape with a TCP reachability guard; the race remained. Mesh-program Phase 6 had this
+> convergence queued and it was pulled forward. See `docs/Redundancy.md` §"Bootstrap: self-first seed
+> ordering" and ScadaBridge `4a6341d8`.
+>
+> **Half (2) — manual failover — is unaffected and still current.** Tasks below are kept verbatim as
+> the execution record, including the Task 8 live-gate evidence, which is the whole reason the
+> watchdog was rejected. Do not execute the Task 1/2/3 code as written.
+
 > **For Claude:** REQUIRED SUB-SKILL: Use superpowers-extended-cc:executing-plans to implement this plan task-by-task.
 >
 > Shared cross-repo design: `~/Desktop/scadaproj/docs/plans/2026-07-22-initjoin-selfform-fallback.md` (design rationale, MNTR assessment, behavior spec). The ScadaBridge half lives in `~/Desktop/ScadaBridge/docs/plans/2026-07-22-selfform-fallback-and-manual-failover.md`. This plan is self-contained for execution.