docs(plans): record the Task 8 live finding — self-form races the join handshake

Corrects the plan's behavior table (the mid-join-handshake row was FALSE) and
notes the mesh-program Phase 6 convergence on ScadaBridge's self-first seed
ordering, which retires the watchdog + guard entirely.

Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW

CLAUDE.md

@@ -224,6 +224,8 @@ The server supports non-transparent warm/hot redundancy via the `Redundancy` sec
 - **Downing is `auto-down`, not `keep-oldest`.** `Cluster:SplitBrainResolverStrategy` (default `auto-down`) selects the provider via `ServiceCollectionExtensions.BuildDowningHocon`; an unknown value fails host start. A two-node pair running the previous `keep-oldest` + `down-if-alone` **could not survive a crash of the oldest node**: Akka's `KeepOldest.OldestDecision` only lets `down-if-alone` rescue a side holding ≥ 2 members, so the 1-vs-1 survivor downs *itself* and exits. `down-if-alone` is a 3+-node feature. The accepted trade for `auto-down` is dual-active during a genuine network partition. **Its live gate is still open** — the pathology only appears 1-vs-1 and docker-dev is a single six-node mesh, so the crash-the-oldest drill is deferred to the per-cluster mesh work.
 - **The Primary is the oldest Up `driver` member, not the role leader.** `RedundancyStateActor` previously used `ClusterState.RoleLeader("driver")` (lowest *address*), while `ClusterSingletonManager` places singletons on the *oldest*. They agree only on a freshly-formed cluster and diverge after any restart, so the Primary-gated data plane (writes, alarm acks, alerts emit, alarm-history drain) could enable on a node not hosting the singletons. `NodeRedundancyState.IsRoleLeaderForDriver` was renamed **`IsDriverPrimary`**.
 
+**A third bootstrap gap closed 2026-07-22 — downing was never the whole story.** Even under `auto-down`, a node cold-starting while its peer was dead **never came Up**: Akka lets only the **first** listed seed form a *new* cluster, so every other node loops on `InitJoin` forever. `Cluster:SelfFormAfter` (`TimeSpan?`, default 10 s, `null`/`≤0` disables) arms `ClusterBootstrapFallback` from `WithOtOpcUaClusterBootstrap` via an Akka.Hosting `AddStartup` task — it waits that long for membership, then `Cluster.Join(SelfAddress)`. **It fires only when this node's own address is in its own `SeedNodes`**: a node seeded only by someone else (today's docker-dev site nodes, which list only `central-1`) must wait rather than island itself permanently. Pinned by `SelfFormBootstrapTests`, whose two negative cases each carry a positive-control self-join so they cannot pass vacuously. See `docs/Redundancy.md` §"Bootstrap: the self-form fallback".
+
 Assert the **effective** cluster config off a running `ActorSystem` (as `SplitBrainResolverActivationTests` does), never the typed options or the akka.conf text — several HOCON fragments compete and the losing one still reads correctly in the file.
 
 **Still per-Akka-cluster, not per application `Cluster`.** One Primary is elected across the whole Akka mesh, so a fleet running several application clusters in one mesh (what docker-dev models) gets one Primary among all driver nodes. The fix is the per-cluster mesh workstream: `docs/plans/2026-07-21-per-cluster-mesh-design.md` (Phases 0a/0b done; 1–7 not started, Phase 1 planned in `…-phase1.md`).

docs/Redundancy.md

@@ -159,7 +159,7 @@ Per-node identity comes from `appsettings.json` + the `OTOPCUA_ROLES` env var:
 OTOPCUA_ROLES=admin,driver
 ```
 
-Both nodes share the same `ConfigDb` connection string; `Cluster.PublicHostname` + `Roles` are what makes them distinct in cluster gossip. The first node bootstraps the cluster (its address goes in `SeedNodes`); the second node joins via the same `SeedNodes` list.
+Both nodes share the same `ConfigDb` connection string; `Cluster.PublicHostname` + `Roles` are what makes them distinct in cluster gossip. The first node bootstraps the cluster (its address goes in `SeedNodes`); the second node joins via the same `SeedNodes` list. List **both** peers in `SeedNodes` on **both** nodes so either can cold-start alone — see [Bootstrap: the self-form fallback](#bootstrap-the-self-form-fallback) for why the seed list alone is not enough, and what `Cluster:SelfFormAfter` adds.
 
 There is no longer a `Node:NodeId` setting and no `ClusterNode.RedundancyRole` column (the V2 migration dropped it — primary/secondary is now derived from cluster membership age). NodeId is derived as `host:port` of the cluster `PublicHostname` (see `ClusterRoleInfo.LocalNode` for the formula).
 
@@ -241,10 +241,9 @@ two-node pair that choice means *any* crash of the oldest node is a full outage.
    `IHostApplicationLifetime.StopApplication()` so the process exits and the supervisor restarts it instead
    of idling forever with a dead actor system. Under `auto-down` the *survivor* no longer needs this
    (it is never the one downed); the **downed** node still does.
-3. **Both peers listed in `SeedNodes`** on every node, so a restarted node can re-join via either peer.
-   Note the residual bootstrap constraint: only the **first** seed may form a cluster alone, so a node
-   cold-starting while its peer is dead still waits in `InitJoin`. Auto-down removes the crash outage, not
-   this one.
+3. **Both peers listed in `SeedNodes`** on every node, so a restarted node can re-join via either peer —
+   plus the **self-form fallback** below, without which a node cold-starting while its peer is dead waits
+   in `InitJoin` forever (auto-down removes the crash outage, not that one).
 
 > **On `HardKillFailoverTests`:** that in-process test hard-kills the oldest and asserts the survivor becomes
 > sole `driver` role-leader. It passed even under `keep-oldest` — because it simulates the crash with
@@ -259,8 +258,36 @@ bootstrap and reading `akka.cluster.downing-provider-class` back off the running
 the typed option or the akka.conf text is not sufficient, because several HOCON fragments compete and the
 losing one still looks correct in the file.
 
-There is no operator-driven role swap during a partition; failover is what the cluster and supervisor do
-automatically.
+Failover during a partition is what the cluster and supervisor do automatically. For a *planned* swap —
+patching, draining a node — see [Manual failover](#manual-failover) below.
+
+## Manual failover
+
+The cluster redundancy page (`/clusters/{id}/redundancy`) carries a **Live redundancy** panel — the current
+driver Primary and every Up `driver` member, read from live cluster state — and an admin-gated **Trigger
+failover** button.
+
+The mechanism is a graceful cluster **`Leave`** of the current Primary, never a `Down`:
+`IManualFailoverService.FailOverDriverPrimaryAsync` selects the **oldest Up `driver` member** — the identical
+query to `RedundancyStateActor.SelectDriverPrimary`, pinned together by
+`ManualFailoverServiceTests.Parity_with_SelectDriverPrimary` so the node acted on is exactly the node the
+election names. The leaving node hands its singletons over through the cluster-leave phases,
+`CoordinatedShutdown` runs, `ActorSystemTerminationWatchdog` exits the process, the supervisor restarts it,
+and it rejoins as the **youngest** member — so the survivor is now the oldest, becomes Primary, and
+advertises `ServiceLevel` 250. OPC UA clients re-select.
+
+| Rule | Behaviour |
+|---|---|
+| Authorization | `AdminUiPolicies.FleetAdmin` (**Administrator only**) — gated in markup *and* re-checked server-side before the call. Deliberately not `ConfigEditor`, which the neighbouring cluster pages use: this restarts a production node rather than editing config, and ConfigEditor also admits Designer. |
+| Peer guard | Disabled, with the reason as its tooltip, when fewer than two Up driver members exist — with one node a "failover" is a shutdown. Re-evaluated inside the service against live state, so a stale page cannot bypass it. |
+| Confirmation | A dialog naming the node, and what follows (restart, ServiceLevel 250 moves, clients re-select). |
+| Audit | Written through the shared `ZB.MOM.WW.Audit.IAuditWriter` seam **before** the Leave — action `cluster.manual-failover`, actor, target. A *refused* failover changes nothing and writes nothing. |
+
+> **Mesh-scope caveat.** Until the per-cluster mesh work lands
+> (`docs/plans/2026-07-21-per-cluster-mesh-design.md`), the Primary is elected once per Akka mesh rather than
+> per application `Cluster` row — so on a fleet running several clusters in one mesh this button acts on the
+> whole mesh's Primary, which may belong to a different cluster than the page you are on. The panel says so.
+> Phase 6 of the mesh program removes both the caveat and this notice.
 
 > **Live gate outstanding.** The auto-down change is verified by unit tests against the effective
 > configuration, and by the sister project's live drill on an equivalent Akka version and topology. It has
@@ -270,6 +297,67 @@ automatically.
 > alongside the per-cluster mesh work, which makes every mesh exactly two nodes. See
 > `docs/plans/2026-07-21-per-cluster-mesh-design.md` §6.2 / Phase 0a.
 
+## Bootstrap: the self-form fallback
+
+Listing both peers in `SeedNodes` does **not** mean either node can cold-start alone. Akka lets only the
+**first** listed seed form a *new* cluster; every other node sends `InitJoin` to the seeds and retries —
+forever — until one of them answers. A node that boots while its peer is dead therefore never comes Up,
+and no downing strategy helps: **auto-down removes the crash outage, not this one.**
+
+`Cluster:SelfFormAfter` closes that gap. `ClusterBootstrapFallback.Arm` (armed for every host built through
+`WithOtOpcUaClusterBootstrap`, via an Akka.Hosting `AddStartup` task) waits that long for cluster membership
+and, on expiry, calls `Cluster.Join(SelfAddress)` — the node forms a cluster on itself and becomes
+operational unattended.
+
+| `Cluster:SelfFormAfter` | Behaviour |
+|---|---|
+| `"00:00:10"` (**default**) | A node with no membership after 10 s self-forms. A live peer answers `InitJoin` in milliseconds, so on any normal boot the fallback never fires. |
+| `null` / `≤ 0` | Disabled — the pre-2026-07-22 behaviour: wait on `InitJoin` indefinitely. |
+
+**Two guards keep the fallback from creating the split it exists to avoid.**
+
+**1 — Reachability.** An expired window is not evidence the peer is down, so before self-forming the
+fallback TCP-probes the other seed addresses; if any accepts a connection it waits another window instead.
+This was added after the live gate below caught the fallback islanding a node: a node bounced by a manual
+failover restarted, received `InitJoinAck` from its live peer — a join in flight and healthy — but did not
+get the Welcome inside the window, because the peer's ring still held the node's previous incarnation
+(`Exiting` → `Down` → `Removed`). The fallback fired on the timer and the node formed a **second cluster**.
+The original design assumed `Cluster.Join(SelfAddress)` would be ignored mid-handshake; **it is not — it
+wins.** Since manual failover deliberately produces exactly that restart, without this guard every manual
+failover could island the node it bounced. Pinned by
+`SelfFormBootstrapTests.Reachable_peer_suppresses_self_forming_until_it_goes_away`, whose "peer" is a bare
+`TcpListener` that speaks no Akka: reachable, never answering a join.
+
+**2 — Seed membership.** The fallback fires **only when this node's own address
+appears in its own `SeedNodes`**. A node that is not one of its own seeds is never legitimately first, and
+if it self-formed, a later-booting real seed would form a second cluster the two could never merge. That is
+exactly the current docker-dev topology: the site-a/site-b driver nodes list only `central-1` as a seed, so
+they deliberately keep waiting rather than islanding themselves. Under the future per-cluster mesh
+(`docs/plans/2026-07-21-per-cluster-mesh-design.md`) each pair node *is* a seed of its own two-node mesh, so
+the fallback covers both sides of every pair and the site-node exception disappears.
+
+Sequential recovery is island-free by construction: Akka's join protocol prefers an existing cluster (a
+booting node only self-forms when **no** seed answered), so a peer booting after the survivor self-formed
+simply joins it as the youngest member. The residual risk is both pair nodes cold-starting inside the
+window while mutually unreachable — both self-form, which is the same dual-active class the `auto-down`
+strategy already accepts, with the same recovery (restart one side).
+
+Pinned by `SelfFormBootstrapTests` (`tests/Core/ZB.MOM.WW.OtOpcUa.Cluster.Tests/`), which starts real hosts
+through the production bootstrap: a lone non-first seed comes Up; a disabled window keeps waiting; a node
+absent from its own seed list never self-forms; a node whose peer is merely *reachable* keeps waiting. Every
+negative case carries a positive control (an explicit self-join, or dropping the peer listener) so none can
+pass merely because the node was unformable.
+
+> **Live gate: PASSED (docker-dev, 2026-07-22).** Drilled on the six-node rig, defect first and fix
+> second. On the pre-change image, `central-2` started alone (central-1 stopped) and logged **zero** join
+> events in 70 s — the `InitJoin` loop, observed rather than argued. On the rebuilt image the same node
+> logged the self-form warning **10 s** after start and `Leader is moving node [central-2] to [Up]`, then
+> came up with all four admin singletons. A `site-a-2` recreated while central-1 was still down logged
+> *"self-form fallback inactive: this node … is not in its own seed list"* and stayed out — the island
+> guard on the real topology. Restarting `central-1` produced `Welcome from [central-2]`: it joined the
+> self-formed cluster instead of starting a second one, and the four site nodes then joined through it —
+> six members, **one** cluster.
+
 ## Primary data-plane gate (writes, acks, alerts emit)
 
 Three data-plane surfaces are **Primary-gated** so a warm standby never touches a shared field device or duplicates a fleet-wide publish: inbound operator **writes** (`DriverHostActor.HandleRouteNodeWrite`), native-alarm **acks** (`HandleRouteNativeAlarmAck`), and the native/scripted **alerts emit** (`ForwardNativeAlarm` / `ScriptedAlarmHostActor.OnEngineEmission`). All three route through one policy — `PrimaryGatePolicy.ShouldServiceAsPrimary(localRole, driverMemberCount)` (`OtOpcUa.Runtime.Drivers`):

docs/plans/2026-07-21-per-cluster-mesh-design.md

@@ -85,6 +85,12 @@ Their failover drill has a mode that exists *"to make the registered gap observa
 it is covered."* **A two-node keep-oldest mesh has an acknowledged total-outage hole.** If OtOpcUa
 adopts 2-node meshes, this must be decided deliberately, not inherited.
 
+> **CLOSED 2026-07-22.** The `InitJoin` half of this gap is fixed in both repos by the self-form
+> fallback: `Cluster:SelfFormAfter` (default 10 s) lets a node that is one of its own seeds form a
+> cluster on itself when no seed answers. See `docs/Redundancy.md` §"Bootstrap: the self-form
+> fallback" — including the island guard that keeps today's non-seed site nodes waiting, which the
+> per-cluster mesh removes by making every pair node a seed of its own mesh.
+
 ### Both inter-cluster transports are unauthenticated
 
 Akka remoting has no TLS, no secure cookie, no `trusted-selection-paths`. The gRPC listener is h2c
@@ -253,6 +259,14 @@ rather than the configured one. See `docs/Redundancy.md`.
 1-vs-1 split, and `docker-dev` is a single six-node mesh. The crash-the-oldest drill therefore lands
 with Phase 6/7, which makes every mesh exactly two nodes.
 
+**The bootstrap half of the gap is also closed (2026-07-22).** Downing was only ever one of the two
+outage modes: even with `auto-down`, a node cold-starting while its peer was dead looped on `InitJoin`
+forever, because Akka lets only the first listed seed form a new cluster. `Cluster:SelfFormAfter`
+(default 10 s, `ClusterBootstrapFallback`) now self-forms in that case, guarded so a node absent from
+its own seed list never islands itself. Pinned by `SelfFormBootstrapTests`; see `docs/Redundancy.md`
+§"Bootstrap: the self-form fallback". Under this design each pair node is a seed of its own two-node
+mesh, so the fallback covers both sides and the current site-node exception disappears.
+
 ### 6.3 DECIDED — match ScadaBridge's auth posture for now
 
 Inter-cluster transports stay unauthenticated and unencrypted, as theirs are: the boundary assumes a

docs/plans/2026-07-22-per-cluster-mesh-program.md

@@ -0,0 +1,198 @@
+# Per-Cluster Mesh Program — Align OtOpcUa's Akka Topology with ScadaBridge
+
+> **For Claude:** This is a PROGRAM plan (phase roadmap + gates), not a bite-sized task plan.
+> The authoritative design is `docs/plans/2026-07-21-per-cluster-mesh-design.md` (decisions settled
+> 2026-07-21; Phases 0a/0b DONE). Per that design's own instruction, **each phase gets its own
+> detailed plan, written when the phase starts** (superpowers writing-plans → executing-plans),
+> so plans are authored against current code, not against a forecast. This document sequences the
+> phases, fixes the deployment topology (co-location with ScadaBridge — NEW constraint 2026-07-22),
+> and defines each phase's entry/exit gates.
+
+**Goal:** OtOpcUa runs the same Akka.NET mesh shape as ScadaBridge — **one Akka mesh per
+application `Cluster`, two nodes max**, central ↔ cluster joined by explicit transports
+(ClusterClient + gRPC + fetch), driver nodes holding **no ConfigDb connection** — so that every
+Primary-gated decision and every gated resource share one pair-local scope, and both products
+present one operational model on the shared site hardware.
+
+**Why now (2026-07-22):** OtOpcUa cluster pairs will run **on the same two Windows VMs as the
+ScadaBridge site nodes**. That deployment makes the current single-fleet-mesh design actively
+wrong for sites (a site's OtOpcUa nodes would gossip across the WAN to central and elect ONE
+Primary fleet-wide), and makes the ScadaBridge shape the obviously correct one: each site's two
+VMs host two independent, identically-postured 2-node clusters (one per product), surviving alone
+on the site LAN. It also means driver nodes must not depend on reaching central SQL — **sites
+have no SQL Server**, so §6.1's "driver nodes never connect to the ConfigDb" stops being an
+architectural preference and becomes a deployment requirement.
+
+---
+
+## Deployment topology (the co-location constraint, NEW)
+
+Each **site**: 2 Windows VMs. Each VM runs one ScadaBridge site node AND one OtOpcUa driver node.
+Two independent 2-node Akka clusters per site — they share hardware, never a mesh. **Central**:
+2 Windows VMs, each running a ScadaBridge central node and an OtOpcUa central node
+(`admin,driver`), again as separate pairs.
+
+**Per-VM port allocation (no collisions — verify against actual deployment configs in Phase 6):**
+
+| Port | Owner | Purpose |
+|---|---|---|
+| 8081 / 8082 | ScadaBridge | Akka remoting (central / site) |
+| 8083 | ScadaBridge site | gRPC h2c (streams + LocalDb sync) |
+| 8084 | ScadaBridge site | metrics |
+| 5000 (+Traefik) | ScadaBridge central | UI + Inbound API |
+| **4053** | OtOpcUa | Akka remoting |
+| **4840** | OtOpcUa | OPC UA endpoint |
+| OtOpcUa AdminUI port | OtOpcUa central (admin) | AdminUI (driver-only site nodes host no UI) |
+| OtOpcUa LocalDb sync port | OtOpcUa driver | h2c LocalDb pair replication (default off/0 today — Phase 6 assigns a real per-site port) |
+
+**Aligned HA posture (both products, per VM — already true or landing via the selfform plan):**
+auto-down downing (15 s window), oldest-Up active/primary election, `SelfFormAfter` 10 s
+self-form fallback, termination-watchdog → process exit → `sc.exe failure` restart recovery.
+One failover story for operators regardless of product.
+
+**Prerequisite ordering:** the fallback/manual-failover plan
+(`docs/plans/2026-07-22-selfform-fallback-and-manual-failover.md`) executes **before** this
+program — it is small, independent, and Phase 6 depends on its semantics (under per-pair meshes
+every node lists itself + partner as seeds, so the fallback covers both nodes of every pair;
+the site-node island guard then simply never triggers).
+
+---
+
+## What changes, in one table (from the design doc)
+
+| Aspect | Today (single fleet mesh) | Target (ScadaBridge shape) |
+|---|---|---|
+| Mesh | one gossip ring, all nodes, seeded by central-1 | one 2-node mesh per application `Cluster`; same ActorSystem name; separation by seed-node partitioning |
+| Roles | `admin` / `driver` fleet-wide | `driver` + cluster-specific `cluster-{ClusterId}`; singletons scoped to the cluster role |
+| Primary election | mesh-wide oldest Up driver (Phase 0b) | pair-local by construction — same rule, correct scope |
+| Command/control | 9 DPS topics + singleton over gossip | ClusterClient (one receptionist actor per side); central discovers nodes from `ClusterNode` rows |
+| Deploy | notify (DPS) + node fetches from ConfigDb | notify via ClusterClient; artifact fetched **from central**, cached in LocalDb |
+| Driver ConfigDb connection | direct EF connection to central SQL | **none** — LocalDb is the steady-state config store |
+| Live telemetry | 7 observability DPS topics | one gRPC stream contract (`oneof` event), central dials each cluster node |
+| Rig | six-node single mesh (`docker-dev`) | per-cluster meshes; rig models the real topology |
+
+---
+
+## Phases
+
+Each phase below is one row of design-doc §7, expanded with entry/exit gates. **Execution recipe
+per phase:** (1) invoke writing-plans in this repo to produce
+`docs/plans/2026-07-2X-mesh-phase-N-<name>.md` from the scope notes here + the design doc §
+references, exploring current code first; (2) execute it task-by-task; (3) run the phase's exit
+gate; (4) update this file's status column and the design doc's §7 table.
+
+### Phase 1 — `ClusterNode` address columns + DB-sourced ack set
+**Scope:** `ClusterNode` gains Akka + gRPC address columns (mirroring ScadaBridge's `Site`
+entity `NodeAAddress`/`GrpcNodeAAddress` pattern, but per-node rows); EF migration; AdminUI node
+edit surfaces the fields; `ConfigPublishCoordinator` derives its expected-ack set from
+`ClusterNode` rows instead of `Akka.Cluster.State.Members` filtered by role (design §3 fact 3 —
+this removes the coordinator's one genuinely mesh-bound dependency).
+**Independent of the split:** yes — safe on the current mesh.
+**Exit gate:** deploy on the unchanged docker-dev rig completes with the coordinator's expected-ack
+set proven DB-sourced (test: a `ClusterNode` row present but node down → deploy reports that node
+missing; a node up but row absent → its ack is not expected).
+
+### Phase 2 — Comm actors + ClusterClient transport
+**Scope:** one receptionist-registered actor per side (`/user/central-communication`,
+`/user/cluster-communication`), registered **per node, not as a singleton** (contact rotation);
+ClusterClient central → cluster carrying deploy notify + acks + driver-control; the ScadaBridge
+idioms copied verbatim: sender-preserving `Tell(new ClusterClient.Send(...), Sender)` Ask relay,
+typed-failure reply for every unhandled message, no central buffering toward unreachable clusters
+(drop + warn), central discovers contacts from Phase 1's `ClusterNode` rows (60 s refresh +
+admin-change refresh), clusters know central from appsettings (static, restart to change).
+**Frame-size guard:** anything carrying payload sets both the frame limit and
+`log-frame-size-exceeding` (design §8) — the deploy path stays payload-free by design.
+**Exit gate:** on the still-single-mesh rig, deploy notify + acks and AdminUI Reconnect/Restart
+flow over ClusterClient (DPS paths deleted or dark-switched), including an Ask timing out cleanly
+against a stopped node.
+
+### Phase 3 — Config fetch-and-cache from central
+**Scope:** the deploy artifact is served **by central** (transport per its own phase plan — the
+ScadaBridge analogue is token-gated HTTP; decide HTTP vs a gRPC fetch RPC when planning) and
+cached in LocalDb; driver nodes read config exclusively from LocalDb (boot-from-cache becomes the
+normal path — design §6.1); chunking, SHA-256 verify, newest-2 retention, pair replication carry
+over unchanged. **This phase changes a running data path — it gets its own live gate** (design §7
+note): deploy lands on a driver pair with central SQL stopped mid-fetch → retry lands; #485
+last-known-good semantics re-proven on the new path.
+**Exit gate:** live gate green on the rig; a driver node with an empty LocalDb and reachable
+central boots into the current config; with central down it boots last-known-good.
+
+### Phase 4 — Cut the driver-side ConfigDb connection
+**Scope (design §6.1 audit table):** re-home `EfAlarmConditionStateStore` to LocalDb (pair-local
+state, same journey as the Phase-2 alarm S&F buffer); **resolve the `DbHealthProbeActor`
+question** — driver nodes have no DB to probe, and DB health currently feeds ServiceLevel tiering,
+so define the replacement health input (candidate: central-reachability via the Phase 2/3
+transports) — this is a client-visible ServiceLevel semantics change and must be documented in
+`docs/Redundancy.md` + the interop playbook; audit `OpcUaPublishActor`'s ConfigDb use (TBD in the
+design) and re-source it; registration cleanup in `ServiceCollectionExtensions`; driver-role
+`Program.cs` branch registers no EF ConfigDb context at all (mirror ScadaBridge's central-only
+`AddConfigurationDatabase`).
+**Exit gate:** its own live gate — a driver pair runs a full deploy + alarm + historian cycle with
+**no ConfigDb connection string configured at all**; grep-level proof no driver-branch service can
+resolve the ConfigDb context.
+
+### Phase 5 — gRPC stream contract for live telemetry
+**Scope:** one server-streaming contract carrying a `oneof` event, **cluster nodes host the gRPC
+server, central dials in** (the inverted direction is the load-bearing ScadaBridge finding —
+design §2); migrate the seven observability topics (`alerts`, `driver-health`,
+`driver-resilience-status`, `fleet-status`, `script-logs`, plus redundancy-state distribution and
+deployment-acks if Phase 2 left them on DPS); additive-only field evolution, contract locked by
+test; per-panel reconnect story for the AdminUI (design §8 — losing gossip loses free fleet
+observability).
+**Exit gate:** all AdminUI live panels green against a pair with DPS telemetry topics deleted;
+kill-and-reconnect of the central dialer recovers every stream.
+
+### Phase 6 — Mesh partition + co-location topology
+**Scope:** per-cluster seed nodes — **adopt ScadaBridge's self-first ordering (each node lists
+ITSELF as `seed-nodes[0]`, partner second) and RETIRE the `SelfFormAfter` watchdog + TCP
+reachability guard** (2026-07-22 execution finding: the watchdog races Akka's join handshake —
+`Join(self)` during an in-flight join islands the node, hit live here and fixed with the guard;
+ScadaBridge proved self-first ordering is the race-free form of the same semantics, enforced by
+a startup-validator rule — port that rule too); cluster-scoped roles `cluster-{ClusterId}` + singleton re-scoping,
+central pair keeps the admin singletons; **docker-dev rig rewritten** to model the real topology —
+including the co-location port table above (both products' compose files on shared per-site
+networks, real LocalDb sync ports); remove the ClusterRedundancy page's mesh-scope caveat (the
+election is pair-local now) and the fallback's site-node island-guard docs note (moot — every
+node is a seed of its own mesh); `Cluster__SeedNodes__*` env matrix per pair.
+**Exit gate:** rig up in the new shape; every existing live-gated behavior re-verified per pair
+(deploy, redundancy 250/240 per pair — **two Primaries fleet-wide, one per pair, by design**);
+secrets Akka replication re-verified or re-scoped (it rides DPS on the current single mesh — its
+topology must be re-decided here, likely SQL-hub mode like ScadaBridge, since pub/sub cannot
+cross separate meshes).
+
+### Phase 7 — Failover drill + live gates
+**Scope:** the drill ScadaBridge already has (`failover-drill.sh` analogue) run per pair, both
+directions; **close the two outstanding live gates**: (a) auto-down 1-vs-1 crash-the-oldest
+(deferred since Phase 0a — finally testable, every mesh is exactly two nodes), (b) `SelfFormAfter`
+lone-cold-start on the real per-pair topology; manual-failover button re-verified per pair;
+operator runbook for the co-located site (one page covering both products' failover on the same
+two VMs).
+**Exit gate:** drill green on every pair type (central, site); runbook merged; design doc §7
+table fully marked DONE.
+
+---
+
+## Risks carried from the design (§8, unchanged — re-read before each phase plan)
+
+LocalDb becomes load-bearing for config (blast radius of the #485 class rises);
+`DbHealthProbeActor` feeds a client-visible value; the rig models the doomed topology until
+Phase 6; losing gossip loses free observability (7 panels); 128 KB ClusterClient frame drop is
+silent unless both knobs are set; never stack app-level LWW on LocalDb's HLC. **New (this
+program):** co-located VMs mean a VM loss now takes out one node of BOTH products at once — the
+drill in Phase 7 must include the shared-VM failure (both products fail over together), and
+resource sizing on the site VMs should be checked once both products run the full stack.
+
+## Tracking
+
+| Phase | Status |
+|---|---|
+| 0a downing strategy | DONE 2026-07-21 (live gate → Phase 7) |
+| 0b oldest-Up election | DONE 2026-07-21 |
+| Prereq: selfform-fallback + manual-failover plan | plan written 2026-07-22, not executed |
+| 1 ClusterNode columns + DB ack set | not started — plan to be written |
+| 2 ClusterClient transport | not started |
+| 3 fetch-and-cache | not started |
+| 4 cut driver ConfigDb | not started |
+| 5 gRPC telemetry | not started |
+| 6 mesh partition + co-location | not started |
+| 7 drill + live gates | not started |

docs/plans/2026-07-22-per-cluster-mesh-program.md.tasks.json

@@ -0,0 +1,15 @@
+{
+  "planPath": "docs/plans/2026-07-22-per-cluster-mesh-program.md",
+  "note": "PROGRAM plan: each task = write that phase's detailed plan (writing-plans), execute it (executing-plans), run its exit gate, update the tracking tables. Prereq task 0 is the separate selfform-fallback plan in this repo.",
+  "tasks": [
+    {"id": 0, "subject": "Prereq: execute 2026-07-22-selfform-fallback-and-manual-failover.md (7 tasks)", "status": "pending"},
+    {"id": 1, "subject": "Phase 1: ClusterNode Akka+gRPC address columns; ConfigPublishCoordinator ack set from DB", "status": "pending"},
+    {"id": 2, "subject": "Phase 2: comm actors + receptionist + ClusterClient transport (deploy notify/acks, driver-control)", "status": "pending", "blockedBy": [1]},
+    {"id": 3, "subject": "Phase 3: config fetch-and-cache from central; LocalDb steady-state config store (live gate)", "status": "pending", "blockedBy": [2]},
+    {"id": 4, "subject": "Phase 4: cut driver-side ConfigDb — EfAlarmConditionStateStore to LocalDb, DbHealthProbeActor ServiceLevel input, OpcUaPublishActor audit (live gate)", "status": "pending", "blockedBy": [3]},
+    {"id": 5, "subject": "Phase 5: gRPC oneof stream contract; migrate 7 observability topics; AdminUI reconnect story", "status": "pending", "blockedBy": [2]},
+    {"id": 6, "subject": "Phase 6: mesh partition — per-pair seeds, cluster-scoped roles/singletons, co-location rig rewrite, secrets replication re-scope", "status": "pending", "blockedBy": [0, 4, 5]},
+    {"id": 7, "subject": "Phase 7: failover drill per pair (incl. shared-VM failure) + close auto-down 1v1 and SelfFormAfter live gates + operator runbook", "status": "pending", "blockedBy": [6]}
+  ],
+  "lastUpdated": "2026-07-22T00:00:00Z"
+}

docs/plans/2026-07-22-selfform-fallback-and-manual-failover.md

@@ -0,0 +1,388 @@
+# OtOpcUa: InitJoin Self-Form Fallback + Manual Failover Control — Implementation Plan
+
+> **For Claude:** REQUIRED SUB-SKILL: Use superpowers-extended-cc:executing-plans to implement this plan task-by-task.
+>
+> Shared cross-repo design: `~/Desktop/scadaproj/docs/plans/2026-07-22-initjoin-selfform-fallback.md` (design rationale, MNTR assessment, behavior spec). The ScadaBridge half lives in `~/Desktop/ScadaBridge/docs/plans/2026-07-22-selfform-fallback-and-manual-failover.md`. This plan is self-contained for execution.
+>
+> **⚠ This plan is expected to GROW** — see "Reserved for additions" at the end. Append new tasks there (and to the `.tasks.json`) rather than renumbering existing ones.
+
+**Goal:** (1) Either node of a 2-node OtOpcUa pair can cold-start alone and become operational, unattended — a configurable, fast (default 10 s) InitJoin timeout falls back to self-forming a cluster, with a hard guard so site nodes seeded only by central can never island themselves. (2) An admin-only "Trigger failover" control on the Cluster Redundancy page performs a graceful, audited swap of the driver Primary.
+
+**Architecture:** New `AkkaClusterOptions.SelfFormAfter` (`TimeSpan?`, default 10 s, `null`/`≤0` disables, bound from the `Cluster` section) arms `ClusterBootstrapFallback` via an Akka.Hosting startup task inside `WithOtOpcUaClusterBootstrap` — so production AND the Cluster.Tests hosts get it identically. Wait for membership via `RegisterOnMemberUp`; on expiry, `Cluster.Join(SelfAddress)` — only if this node's own address is in its own `SeedNodes`. Manual failover = graceful `Cluster.Leave(<oldest Up driver member>)` via a new `IManualFailoverService` in ControlPlane, surfaced on `/clusters/{id}/redundancy`.
+
+**Tech Stack:** .NET 10, Akka.NET 1.5.62, Akka.Hosting 1.5.62 (`AddStartup`), Blazor Server (AdminUI, InteractiveServer), bUnit, xunit. No new packages.
+
+**Branch:** `feat/selfform-fallback` off `master`.
+
+---
+
+## Design essentials (from the shared design doc)
+
+**The defect:** Akka only lets the FIRST listed seed self-join to form a *new* cluster; every other node loops on `InitJoin` forever — "auto-down removes the crash outage, not this one" (`docs/Redundancy.md:244-247`). A lone cold-starting `central-2` (or a pair node under the future per-cluster mesh) never comes Up.
+
+**Behavior spec:**
+
+| Scenario | Behavior with fallback |
+|---|---|
+| Peer alive (any boot order) | Normal seed join in ms — fallback never fires |
+| Lone cold-start, self IS in own `SeedNodes` | After `SelfFormAfter`: warn log + `Cluster.Join(SelfAddress)` → Up alone (`min-nr-of-members=1`) |
+| Lone cold-start, self NOT in own `SeedNodes` | Fallback inert (info log). **This is the docker-dev site-node topology** — site-a/b nodes list only `central-1`; self-forming there would island them permanently, so they must wait. |
+| Peer boots after survivor self-formed | Its InitJoin is answered → joins as youngest. No island. |
+| Both pair nodes cold-start simultaneously, mutually unreachable | Both self-form → dual-active (same partition class auto-down accepts; restart one side) |
+| `SelfFormAfter` null/`≤0` | Disabled — today's wait-forever behavior |
+| ~~Window expires mid-join-handshake~~ | ~~Benign: Akka ignores `Join` once joined~~ **FALSE — proven live (Task 8): `Join(self)` during an in-flight join abandons it and islands the node. Fixed by the TCP reachability guard: never self-form while any seed peer accepts a connection; wait a window and re-arm. ScadaBridge rejected the watchdog entirely for self-first seed ordering; OtOpcUa converges at mesh-program Phase 6.** |
+
+**Manual failover rules:** graceful `Leave` of the **oldest Up `driver` member** (identical query to `RedundancyStateActor.SelectDriverPrimary` — `Member.AgeOrdering`, `Leaving` excluded via `Status == Up`), never `Down`. The leaving node hands over via the cluster-leave phases, `CoordinatedShutdown` runs, `ActorSystemTerminationWatchdog` exits the process, the supervisor restarts it, and it rejoins as youngest → the survivor is Primary (ServiceLevel 250 moves; OPC UA clients re-select). Admin-only; peer guard (<2 Up driver members → disabled); confirmation dialog; audited via the shared `ZB.MOM.WW.Audit.IAuditWriter` seam before the Leave. **Mesh-scope caveat:** until the per-cluster mesh lands, the election is mesh-wide — the button acts on THE Primary of the whole Akka cluster, not per-`ServerCluster` row; the UI must say so (mirrors the KNOWN LIMITATION in `docs/Redundancy.md:84-99`).
+
+**Multi-node TestKit:** assessed, NOT used — everything is deterministic in-process via real hosts through `WithOtOpcUaClusterBootstrap` (the `SplitBrainResolverActivationTests` pattern), and the Akka TestKit family is xunit-v2-only while this repo's integration tree is xunit.v3. Full verdict in the shared design doc.
+
+---
+
+### Task 1 (B1): `SelfFormAfter` on `AkkaClusterOptions`
+
+**Classification:** small
+**Estimated implement time:** ~3 min
+**Parallelizable with:** none (first task)
+
+**Files:**
+- Modify: `src/Core/ZB.MOM.WW.OtOpcUa.Cluster/AkkaClusterOptions.cs`
+
+**Step 1: Branch**
+
+```bash
+cd ~/Desktop/OtOpcUa && git checkout master && git checkout -b feat/selfform-fallback
+```
+
+**Step 2: Add the property** (after `SplitBrainResolverStrategy`, mirroring its XML-doc depth):
+
+```csharp
+    /// <summary>
+    /// Bootstrap self-form fallback window (decision 2026-07-22, scadaproj/akka_failover.md §6.1).
+    /// Akka only lets the FIRST listed seed form a new cluster; a non-first seed cold-starting
+    /// while its peer is down loops on InitJoin forever ("auto-down removes the crash outage,
+    /// not this one" — docs/Redundancy.md). When this node has waited longer than this window
+    /// without cluster membership it joins itself — but ONLY if its own address appears in
+    /// <see cref="SeedNodes"/>; a non-seed node (e.g. a site node whose only seed is central-1)
+    /// stays waiting, because self-forming there creates a permanent island. Default 10s
+    /// (same-datacenter pair; a live peer answers InitJoin in milliseconds). <c>null</c> or a
+    /// non-positive value disables the fallback. Accepted trade: both pair nodes cold-starting
+    /// inside the window while mutually unreachable form two clusters — the same dual-active
+    /// class the auto-down strategy already accepts, same recovery (restart one side).
+    /// </summary>
+    public TimeSpan? SelfFormAfter { get; set; } = TimeSpan.FromSeconds(10);
+```
+
+**Step 3:** `dotnet build src/Core/ZB.MOM.WW.OtOpcUa.Cluster` — 0 warnings. Commit:
+
+```bash
+git add -A && git commit -m "feat(cluster): SelfFormAfter option (bootstrap self-form fallback window)"
+```
+
+(The default-value pin test lands with Task 3's test file — the property is inert until Task 2.)
+
+---
+
+### Task 2 (B2): `ClusterBootstrapFallback` + `AddStartup` wiring
+
+**Classification:** high-risk (cluster formation behavior)
+**Estimated implement time:** ~5 min
+**Parallelizable with:** none
+
+**Files:**
+- Create: `src/Core/ZB.MOM.WW.OtOpcUa.Cluster/ClusterBootstrapFallback.cs`
+- Modify: `src/Core/ZB.MOM.WW.OtOpcUa.Cluster/ServiceCollectionExtensions.cs` (`WithOtOpcUaClusterBootstrap`, after the `AddHocon(BuildDowningHocon(options), HoconAddMode.Prepend)` call at line 93)
+
+**Step 1: Implement the fallback class** (deliberately duplicated from ScadaBridge's, like the termination watchdogs — see that repo's `ClusterBootstrapFallback.cs` for the full doc comment to adapt):
+
+```csharp
+using Akka.Actor;
+using Microsoft.Extensions.Logging;
+
+namespace ZB.MOM.WW.OtOpcUa.Cluster;
+
+/// <summary>
+/// InitJoin self-form fallback (decision 2026-07-22, scadaproj/akka_failover.md §6.1). Akka only
+/// lets the FIRST listed seed form a NEW cluster; every other node retries InitJoin forever —
+/// "auto-down removes the crash outage, not this one" (docs/Redundancy.md). Waits
+/// <see cref="AkkaClusterOptions.SelfFormAfter"/> for membership, then joins itself.
+/// <para><b>Island safety:</b> fires ONLY when this node's own address is in its own
+/// <see cref="AkkaClusterOptions.SeedNodes"/> — a site node seeded only by central-1 must wait,
+/// not island. Sequential recovery is island-free: Akka's join protocol prefers an existing
+/// cluster (a booting node only self-forms when NO seed answers InitJoin).</para>
+/// <para><b>Benign race:</b> Akka ignores <c>Join</c> once the node has joined a cluster this
+/// incarnation. Residual risk: both pair nodes cold-starting inside the window while mutually
+/// unreachable self-form separately — the same dual-active class auto-down accepts.</para>
+/// </summary>
+public static class ClusterBootstrapFallback
+{
+    public static void Arm(ActorSystem system, AkkaClusterOptions options, ILogger logger)
+    {
+        if (options.SelfFormAfter is not { } window || window <= TimeSpan.Zero)
+        {
+            logger.LogInformation(
+                "Cluster self-form fallback disabled (SelfFormAfter not set) — a node cold-starting "
+                + "while its peer is down will wait on InitJoin indefinitely.");
+            return;
+        }
+
+        var cluster = Akka.Cluster.Cluster.Get(system);
+        var self = cluster.SelfAddress;
+        var isSeed = options.SeedNodes.Any(s => TryParseAddress(s, out var a) && a.Equals(self));
+        if (!isSeed)
+        {
+            logger.LogInformation(
+                "Cluster self-form fallback inactive: this node ({Self}) is not in its own seed list "
+                + "[{Seeds}] — self-forming here would island it from the real seeds.",
+                self, string.Join(", ", options.SeedNodes));
+            return;
+        }
+
+        var joined = new TaskCompletionSource(TaskCreationOptions.RunContinuationsAsynchronously);
+        cluster.RegisterOnMemberUp(() => joined.TrySetResult());
+
+        _ = Task.Run(async () =>
+        {
+            var winner = await Task.WhenAny(joined.Task, Task.Delay(window));
+            if (winner == joined.Task || system.WhenTerminated.IsCompleted)
+                return;
+            logger.LogWarning(
+                "No cluster membership after {Window} — no seed answered InitJoin (peer down at boot). "
+                + "Self-forming a cluster at {Self} so this node becomes operational; if the peer was "
+                + "merely partitioned (not dead), the pair is now dual-active — restart one side after "
+                + "the partition heals (accepted availability-first trade, decision 2026-07-22).",
+                window, self);
+            cluster.Join(self);
+        });
+    }
+
+    private static bool TryParseAddress(string seed, out Address address)
+    {
+        try { address = Address.Parse(seed); return true; }
+        catch { address = default!; return false; }
+    }
+}
+```
+
+(If the Cluster project lacks a `Microsoft.Extensions.Logging.Abstractions` reference, add the `PackageReference` — check `Directory.Packages.props` for the pinned version first.)
+
+**Step 2: Wire via Akka.Hosting startup hook** — in `WithOtOpcUaClusterBootstrap`, after the downing HOCON line (~93):
+
+```csharp
+        // InitJoin self-form fallback (decision 2026-07-22): registered as an Akka.Hosting
+        // startup task so every host built through this bootstrap — production AND the test
+        // hosts in Cluster.Tests — arms it identically. Guarded inside Arm: disabled when
+        // SelfFormAfter is unset; inert when this node is not in its own seed list (site nodes
+        // seeded only by central-1 must wait, not island).
+        var fallbackLogger = (serviceProvider.GetService<ILoggerFactory>()
+                              ?? Microsoft.Extensions.Logging.Abstractions.NullLoggerFactory.Instance)
+            .CreateLogger("ZB.MOM.WW.OtOpcUa.Cluster.ClusterBootstrapFallback");
+        builder.AddStartup((system, _) =>
+        {
+            ClusterBootstrapFallback.Arm(system, options, fallbackLogger);
+            return Task.CompletedTask;
+        });
+```
+
+(Akka.Hosting 1.5.62 has `AddStartup`. If the overload signature differs, alternative: register an `IHostedService` in `AddOtOpcUaCluster` mirroring `ActorSystemTerminationWatchdog`'s lazy-accessor pattern — but prefer `AddStartup` so test hosts arm it for free.)
+
+**Step 3:** `dotnet build src/Core/ZB.MOM.WW.OtOpcUa.Cluster` — 0 warnings.
+
+**Step 4: Commit**
+
+```bash
+git add -A && git commit -m "feat(cluster): InitJoin self-form fallback armed via Akka.Hosting startup task"
+```
+
+---
+
+### Task 3 (B3): `SelfFormBootstrapTests` — 4 scenarios through the production bootstrap
+
+**Classification:** standard
+**Estimated implement time:** ~5 min
+**Parallelizable with:** Task 4
+
+**Files:**
+- Create: `tests/Core/ZB.MOM.WW.OtOpcUa.Cluster.Tests/SelfFormBootstrapTests.cs`
+
+**Step 1: Write the tests** — follow `SplitBrainResolverActivationTests`' pattern (real `Host.CreateDefaultBuilder()` + `AddAkka` + `WithOtOpcUaClusterBootstrap`; that file's remarks explain why testing through the production bootstrap is mandatory — a test that re-creates the wiring pins the test's wiring, not the shipped one). Helper + scenarios:
+
+```csharp
+public sealed class SelfFormBootstrapTests
+{
+    private static int FreePort()
+    {
+        var l = new System.Net.Sockets.TcpListener(System.Net.IPAddress.Loopback, 0);
+        l.Start();
+        var p = ((System.Net.IPEndPoint)l.LocalEndpoint).Port;
+        l.Stop();
+        return p;
+    }
+
+    private static async Task<IHost> StartNodeAsync(
+        string systemName, int selfPort, int peerPort, TimeSpan? selfFormAfter, bool selfInSeeds = true)
+    {
+        var options = new AkkaClusterOptions
+        {
+            SystemName = systemName, Hostname = "127.0.0.1", PublicHostname = "127.0.0.1",
+            Port = selfPort, Roles = new[] { "admin", "driver" },
+            SelfFormAfter = selfFormAfter,
+            SeedNodes = selfInSeeds
+                ? new[] { $"akka.tcp://{systemName}@127.0.0.1:{peerPort}",
+                          $"akka.tcp://{systemName}@127.0.0.1:{selfPort}" }   // self SECOND — only the fallback can form it
+                : new[] { $"akka.tcp://{systemName}@127.0.0.1:{peerPort}" },
+        };
+        var builder = Host.CreateDefaultBuilder();
+        builder.ConfigureServices(services =>
+        {
+            services.AddSingleton<IOptions<AkkaClusterOptions>>(Options.Create(options));
+            services.AddAkka(systemName, (ab, sp) => ab.WithOtOpcUaClusterBootstrap(sp));
+        });
+        var host = builder.Build();
+        await host.StartAsync();
+        return host;
+    }
+
+    // 1) SelfFormAfter_defaults_to_ten_seconds
+    //      new AkkaClusterOptions().SelfFormAfter.ShouldBe(TimeSpan.FromSeconds(10));
+    // 2) Lone_non_first_seed_self_forms_after_the_window
+    //      dead peer port, window 2s → poll Cluster.Get(system).State until 1 Up member ≤ 20s.
+    // 3) Disabled_fallback_keeps_waiting  (window null → State.Members empty after 6s;
+    //      POSITIVE CONTROL: cluster.Join(cluster.SelfAddress) → 1 Up ≤ 20s — proves the node
+    //      was formable and only the fallback was absent.)
+    // 4) Site_node_seeded_only_by_central_never_self_forms  (selfInSeeds:false, window 1s →
+    //      Members empty after 5s; positive control join. THE guard test: this is the exact
+    //      docker-dev topology — site nodes list only central-1.)
+}
+```
+
+Use a unique `systemName` per test (`otopcua-selfform-1` …) so concurrent hosts cannot cross-join; dispose each host in `finally` (`await host.StopAsync(); host.Dispose();`).
+
+**Step 2: Prove the key assertion has teeth** — before relying on scenario 2, temporarily run it with `selfFormAfter: null` and confirm it times out (red), then restore. Record that in the task notes.
+
+**Step 3: Run**
+
+```bash
+dotnet test tests/Core/ZB.MOM.WW.OtOpcUa.Cluster.Tests --filter "FullyQualifiedName~SelfFormBootstrap"
+```
+Expected: 4 PASS.
+
+**Step 4: Commit**
+
+```bash
+git add -A && git commit -m "test(cluster): self-form fallback — lone-seed forms, disabled waits, site-node guard"
+```
+
+---
+
+### Task 4 (B4): Docs — Redundancy.md + mesh design note
+
+**Classification:** small
+**Estimated implement time:** ~4 min
+**Parallelizable with:** Task 3
+
+**Files:**
+- Modify: `docs/Redundancy.md` (~lines 162, 244–247) — replace the bootstrap-constraint text ("Auto-down removes the crash outage, not this one") with `SelfFormAfter` semantics, the 10 s default, the site-node island guard (why driver-only nodes seeded by central-1 deliberately do NOT self-form under the CURRENT one-mesh topology), and the note that under the future per-cluster mesh each pair node is a seed so the fallback covers both.
+- Modify: `docs/plans/2026-07-21-per-cluster-mesh-design.md` §6.2/§7 — one-line note: the "registered outage gap" is now closed by `SelfFormAfter` (link Redundancy.md).
+- Modify: `CLAUDE.md` — brief note in the redundancy-corrections area.
+
+Commit: `docs(redundancy): SelfFormAfter closes the seed-node bootstrap gap`.
+
+---
+
+### Task 5 (B5): Full verification + optional docker-dev live check
+
+**Classification:** standard
+**Estimated implement time:** ~5 min (suite runtime dominates)
+**Parallelizable with:** none
+
+```bash
+cd ~/Desktop/OtOpcUa
+dotnet build ZB.MOM.WW.OtOpcUa.slnx     # 0 warnings
+dotnet test  ZB.MOM.WW.OtOpcUa.slnx     # green vs the pre-existing baseline
+```
+
+Optional live check (docker-dev six-node mesh): `docker compose stop central-1 central-2 && docker compose start central-2` → central-2 (a listed seed) self-forms in ~10 s; site nodes (non-seeds) keep waiting — the island guard on the real topology. Then `docker compose start central-1` → joins central-2's cluster, mesh re-forms. If the rig isn't running, record deferred-live.
+
+---
+
+### Task 6 (D4): Manual failover service + cluster-level test
+
+**Classification:** high-risk
+**Estimated implement time:** ~5 min
+**Parallelizable with:** none
+
+**Files:**
+- Create: `src/Server/ZB.MOM.WW.OtOpcUa.ControlPlane/Redundancy/IManualFailoverService.cs`
+- Create: `src/Server/ZB.MOM.WW.OtOpcUa.ControlPlane/Redundancy/ManualFailoverService.cs`
+- Modify: the ControlPlane/Host DI registration point (find where `RedundancyStateActor`-adjacent services register; admin-role branch of `Program.cs`)
+- Test: `tests/Server/ZB.MOM.WW.OtOpcUa.ControlPlane.Tests/ManualFailoverServiceTests.cs`
+
+**Step 1: Failing tests** — reuse the two-node fixture pattern from `RedundancyPrimaryElectionTests` (node A older on the **higher** port so age ≠ address order; that fixture even has a guard test asserting the divergence — keep it honest here too):
+
+- `FailOver_targets_the_oldest_driver_member_not_the_role_leader`: service picks node A (oldest, higher port) even though B is `RoleLeader("driver")`; A's system terminates (graceful exit); B's next `RedundancyStateActor` snapshot names B Primary.
+- `FailOver_refuses_when_no_driver_peer_exists`: single driver member → returns null, no Leave, node still Up (positive assert).
+- `Parity_with_SelectDriverPrimary`: on the same fixture, the service's target equals the member `RedundancyStateActor` names Primary — pins the two 5-line queries together.
+
+**Step 2: Implement** — same shape as ScadaBridge's `FailOverCore` but role `"driver"`:
+
+```csharp
+public static Address? FailOverCore(ActorSystem system, bool dryRun = false)
+{
+    var cluster = Akka.Cluster.Cluster.Get(system);
+    // Intentionally identical to RedundancyStateActor.SelectDriverPrimary (oldest Up driver,
+    // Member.AgeOrdering) — the node acted on must be exactly the one the election names.
+    // Pinned by ManualFailoverServiceTests.Parity_with_SelectDriverPrimary.
+    var drivers = cluster.State.Members
+        .Where(m => m.Status == MemberStatus.Up && m.Roles.Contains("driver"))
+        .OrderBy(m => m, Member.AgeOrdering)
+        .ToList();
+    if (drivers.Count < 2) return null;
+    var primary = drivers[0];
+    if (!dryRun) cluster.Leave(primary.Address);
+    return primary.Address;
+}
+```
+
+Async wrapper `FailOverDriverPrimaryAsync(string actor)`: dry-run peer guard → audit via the shared `ZB.MOM.WW.Audit.IAuditWriter` seam (action `cluster.manual-failover`, actor, target in `DetailsJson` — copy the call shape from an existing audited AdminUI admin action) → real Leave → warn log → return address string.
+
+**Step 3:** Tests PASS → commit `feat(redundancy): manual failover service — graceful Leave of the driver Primary`.
+
+---
+
+### Task 7 (D5): ClusterRedundancy page — live panel + failover button + bUnit tests
+
+**Classification:** standard
+**Estimated implement time:** ~5 min
+**Parallelizable with:** none
+
+**Files:**
+- Modify: `src/Server/ZB.MOM.WW.OtOpcUa.AdminUI/Components/Pages/Clusters/ClusterRedundancy.razor` (page is already `@rendermode RenderMode.InteractiveServer` — no rendermode work; and it lives in the AdminUI host, not an RCL, so the static-router wrapper concern from lmxopcua#483 does not apply)
+- Test: the AdminUI bUnit suite (place beside the existing Clusters page tests)
+- Modify: `docs/Redundancy.md` — "Manual failover" subsection (button semantics, mesh-wide scope caveat)
+
+**Step 1: Failing bUnit tests:** (a) button hidden under the read-only policy (`AdminUiPolicies.AuthenticatedRead`), visible under the admin/write policy — check `AdminUiPolicies` for the exact policy the mutating Clusters pages use (e.g. `ClusterEdit.razor`) and use the same one; (b) disabled with tooltip when the live snapshot shows <2 Up driver members; (c) confirm flow calls a fake `IManualFailoverService` exactly once.
+
+**Step 2: Implement.** Add a "Live redundancy" panel above the static config table: current driver Primary + Up driver members from a `GetSnapshot()` read-only companion on `IManualFailoverService` (sourced from live `Cluster.State`; do NOT subscribe to the DPS topic for v1), the mesh-scope notice ("in the current single-mesh topology this acts on the whole mesh's Primary"), and the admin-gated **Trigger failover** button with confirm dialog (states: the Primary node restarts, ServiceLevel 250 moves to the survivor, OPC UA clients re-select). On confirm: call the service with the authenticated user name; surface the returned target address.
+
+**Step 3:** Tests PASS → commit `feat(adminui): manual failover control on the cluster redundancy page`.
+
+**Live check (fold into the Task 5 gate when docker-dev is up):** press the button, ServiceLevel 250 moves (old Primary drops off, survivor publishes 250), denial meters stay quiet, audit event recorded.
+
+---
+
+## Reserved for additions (this plan is expected to grow)
+
+Append new tasks below as **Task 8+** (and extend `.tasks.json`) — do not renumber Tasks 1–7. Candidate items already flagged in the family docs, awaiting a decision on which to include:
+
+- **Configurable downing window** — promote `ServiceCollectionExtensions.DowningStableAfter` (hard-coded 15 s) to `AkkaClusterOptions` (e.g. `Cluster:DowningStableAfter`), keeping the pinned invariant window > `acceptable-heartbeat-pause` (`akka_failover.md` §6.2 calls this out; ScadaBridge's equivalent `Cluster:StableAfter` is already config).
+- **Failure-detector tuning for faster failover** — expose `heartbeat-interval` / `acceptable-heartbeat-pause` (currently fixed in `Resources/akka.conf`) as options, per the availability-first tuning table in `akka_failover.md` §2.
+- ~~Auto-down 1-vs-1 live gate~~ / ~~per-cluster mesh interplay~~ — **now owned by the per-cluster mesh PROGRAM plan** (`docs/plans/2026-07-22-per-cluster-mesh-program.md`, added 2026-07-22): Phases 1–7 align OtOpcUa's mesh with ScadaBridge's shape (one 2-node mesh per application Cluster, co-located on the ScadaBridge VMs). That program lists THIS plan as its prerequisite (execute this one first); its Phase 6 removes the ClusterRedundancy mesh-scope caveat + the site-node island-guard docs note, and Phase 7 closes the auto-down 1-vs-1 and `SelfFormAfter` live gates on the real per-pair topology.
+- *(add further items here)*
+
+---
+
+## Completion
+
+- Merge decision via the finishing-a-development-branch flow (family convention: ff-merge to `master` + push to gitea `lmxopcua`, or PR — ask the user).
+- After merge: update `scadaproj/akka_failover.md` §6.1 status, `scadaproj/CLAUDE.md` index row, and memory `ha-availability-over-partition-safety` (tracked in the scadaproj index plan).
+- Verification-before-completion applies throughout; live gates may be recorded deferred-live if docker-dev is down.

docs/plans/2026-07-22-selfform-fallback-and-manual-failover.md.tasks.json

@@ -0,0 +1,14 @@
+{
+  "planPath": "docs/plans/2026-07-22-selfform-fallback-and-manual-failover.md",
+  "tasks": [
+    {"id": 0, "subject": "Task 1: SelfFormAfter on AkkaClusterOptions", "status": "completed"},
+    {"id": 1, "subject": "Task 2: ClusterBootstrapFallback + AddStartup wiring", "status": "completed", "blockedBy": [0]},
+    {"id": 2, "subject": "Task 3: SelfFormBootstrapTests — 4 scenarios incl. site-node island guard", "status": "completed", "blockedBy": [1], "notes": "Teeth proven: scenario 2 run with selfFormAfter:null failed (red) before restore. A 5th test was added by Task 8."},
+    {"id": 3, "subject": "Task 4: Docs — Redundancy.md bootstrap section + mesh design note", "status": "completed", "blockedBy": [1]},
+    {"id": 4, "subject": "Task 5: Full verification + optional docker-dev live check", "status": "completed", "blockedBy": [2, 3], "notes": "Live gate PASSED for the cold-start case (defect observed on old image, fix on new, island guard on a real site node, one six-node cluster after recovery)."},
+    {"id": 5, "subject": "Task 6: Manual failover service + cluster-level test", "status": "completed", "blockedBy": [4], "notes": "4/4. Teeth proven: sabotaging FailOverCore to address order turned the target + parity tests red."},
+    {"id": 6, "subject": "Task 7: ClusterRedundancy live panel + failover button + tests", "status": "completed", "blockedBy": [5], "notes": "PLAN DEVIATION: plan said bUnit; repo has no bUnit (PageAuthorizationGuardTests documents itself as the substitute). Adapted to a pure ManualFailoverPageModel + 9 unit tests, razor as thin shell. Policy is FleetAdmin, not the ConfigEditor the neighbouring cluster pages use (this restarts a node; ConfigEditor admits Designer). Live-verified: panel, confirm dialog, execution, singleton handover, audit row."},
+    {"id": 7, "subject": "Task 8 (NEW, from live gate): reachability guard — don't self-form while a seed peer is reachable", "status": "completed", "blockedBy": [6], "notes": "DEFECT: the Task 7 live drill caught the self-form fallback islanding the node the failover bounced. central-1 restarted, got InitJoinAck from central-2 at 10:49:05 (join in flight, healthy) but no Welcome inside the window (peer's ring still held the old incarnation until 10:49:06); at 10:49:15 the fallback fired and formed a SECOND cluster. The plan's 'Window expires mid-join-handshake -> benign, Akka ignores Join once joined' row is FALSE — the node had not joined, and Join(self) wins. Manual failover deliberately produces that restart, so the two halves of the plan collided. FIX (user chose the reachability guard): TCP-probe the other seed addresses before self-forming; a reachable peer means wait another window and re-arm. Proven three ways: (1) new regression test whose 'peer' is a bare TcpListener speaking no Akka — RED without the guard; (2) live, a throwaway node seeded at sql:1433 (TCP-reachable, not Akka) logged the guard every 10s and never self-formed; (3) positive control, the same node seeded at a dead port self-formed at +10s. Failover re-drill on the rig: no island, all six nodes one cluster."}
+  ],
+  "lastUpdated": "2026-07-22T00:00:00Z"
+}

src/Core/ZB.MOM.WW.OtOpcUa.Cluster/AkkaClusterOptions.cs

@@ -55,4 +55,19 @@ public sealed class AkkaClusterOptions
     ///     </para>
     /// </remarks>
     public string SplitBrainResolverStrategy { get; set; } = "auto-down";
+
+    /// <summary>
+    /// Bootstrap self-form fallback window (decision 2026-07-22, scadaproj/akka_failover.md §6.1).
+    /// Akka only lets the FIRST listed seed form a new cluster; a non-first seed cold-starting
+    /// while its peer is down loops on InitJoin forever ("auto-down removes the crash outage,
+    /// not this one" — docs/Redundancy.md). When this node has waited longer than this window
+    /// without cluster membership it joins itself — but ONLY if its own address appears in
+    /// <see cref="SeedNodes"/>; a non-seed node (e.g. a site node whose only seed is central-1)
+    /// stays waiting, because self-forming there creates a permanent island. Default 10s
+    /// (same-datacenter pair; a live peer answers InitJoin in milliseconds). <c>null</c> or a
+    /// non-positive value disables the fallback. Accepted trade: both pair nodes cold-starting
+    /// inside the window while mutually unreachable form two clusters — the same dual-active
+    /// class the auto-down strategy already accepts, same recovery (restart one side).
+    /// </summary>
+    public TimeSpan? SelfFormAfter { get; set; } = TimeSpan.FromSeconds(10);
 }

src/Core/ZB.MOM.WW.OtOpcUa.Cluster/ClusterBootstrapFallback.cs

@@ -0,0 +1,206 @@
+using Akka.Actor;
+using Microsoft.Extensions.Logging;
+
+namespace ZB.MOM.WW.OtOpcUa.Cluster;
+
+/// <summary>
+/// InitJoin self-form fallback (decision 2026-07-22, scadaproj/akka_failover.md §6.1).
+/// Akka only lets the FIRST listed seed form a NEW cluster; every other node retries InitJoin
+/// forever. So "both nodes are seed nodes" (<see cref="AkkaClusterOptions.SeedNodes"/>) does NOT
+/// mean either can cold-start alone — a non-first seed booting while its peer is down waits
+/// indefinitely ("auto-down removes the crash outage, not this one" — <c>docs/Redundancy.md</c>).
+/// This watchdog waits <see cref="AkkaClusterOptions.SelfFormAfter"/> for membership; on expiry it
+/// forms a cluster on itself.
+///
+/// <para><b>Island safety.</b> Fires ONLY when this node's own address is in its own seed list.
+/// A node that is not a seed (never legitimately first) must keep waiting: if it self-formed, a
+/// later-booting real seed would form a second cluster and the two can never merge. That is the
+/// current docker-dev site-node topology — site-a/site-b nodes list only central-1. For nodes that
+/// ARE seeds, sequential recovery is island-free — Akka's join protocol prefers an existing cluster
+/// (a booting node only self-forms when NO seed answers InitJoin), so a peer booting after this
+/// node self-formed simply joins it.</para>
+///
+/// <para><b>Reachability guard (added 2026-07-22 after a live-gate failure).</b> An expired window
+/// is NOT sufficient evidence that the peer is down, and joining on the timer alone islands nodes.
+/// Drilled on the docker-dev rig: a node bounced by a manual failover restarted, received
+/// <c>InitJoinAck</c> from its live peer — a join in flight and healthy — but did not get the
+/// Welcome inside the window, because the peer's ring still held the node's previous incarnation
+/// (Exiting → Down → Removed). The fallback fired anyway and the node formed a SECOND cluster,
+/// islanded until an operator restarted it. So <c>Join(SelfAddress)</c> is <b>not</b> ignored
+/// mid-handshake — it wins, which is the opposite of what the original design assumed. Before
+/// self-forming, this therefore TCP-probes the other seed addresses: if any accepts a connection
+/// the peer is alive, a join is likely already in flight, and the fallback waits another window
+/// instead. That is also exactly what the fallback claims to detect — "no seed answered InitJoin
+/// (peer down at boot)".</para>
+///
+/// <para><b>Residual risk.</b> Both pair nodes cold-starting inside the window while mutually
+/// unreachable (a boot-time partition): both self-form, the same dual-active class the auto-down
+/// downing strategy already accepts, with the same recovery (restart one side).</para>
+///
+/// <para>Deliberately duplicated from ScadaBridge's equivalent (like the termination watchdogs) —
+/// the two repos share the behavior, not a package.</para>
+/// </summary>
+public static class ClusterBootstrapFallback
+{
+    /// <summary>
+    /// Arms the fallback on a freshly created actor system. Safe to call unconditionally: it no-ops
+    /// (with an explanatory log) when the fallback is disabled or when this node is not one of its
+    /// own seeds.
+    /// </summary>
+    /// <param name="system">The actor system whose cluster membership is being watched.</param>
+    /// <param name="options">The bound cluster options carrying the window and the seed list.</param>
+    /// <param name="logger">Logger for the armed / inert / self-forming decisions.</param>
+    /// <param name="peerProbe">
+    /// Overrides the reachability probe (host, port) → reachable. Production passes
+    /// <see langword="null"/> for the real TCP connect; tests substitute it to drive the guard
+    /// deterministically.
+    /// </param>
+    public static void Arm(
+        ActorSystem system,
+        AkkaClusterOptions options,
+        ILogger logger,
+        Func<string, int, Task<bool>>? peerProbe = null)
+    {
+        ArgumentNullException.ThrowIfNull(system);
+        ArgumentNullException.ThrowIfNull(options);
+        ArgumentNullException.ThrowIfNull(logger);
+
+        if (options.SelfFormAfter is not { } window || window <= TimeSpan.Zero)
+        {
+            logger.LogInformation(
+                "Cluster self-form fallback disabled (SelfFormAfter not set) — a node cold-starting "
+                + "while its peer is down will wait on InitJoin indefinitely.");
+            return;
+        }
+
+        var cluster = Akka.Cluster.Cluster.Get(system);
+        var self = cluster.SelfAddress;
+        var isSeed = options.SeedNodes.Any(s => TryParseAddress(s, out var a) && a.Equals(self));
+        if (!isSeed)
+        {
+            logger.LogInformation(
+                "Cluster self-form fallback inactive: this node ({Self}) is not in its own seed list "
+                + "[{Seeds}] — self-forming here would island it from the real seeds.",
+                self,
+                string.Join(", ", options.SeedNodes));
+            return;
+        }
+
+        // Every seed that is not this node. These are the addresses whose reachability decides
+        // whether an expired window means "peer is down" or "peer is alive and we are mid-join".
+        var peerSeeds = options.SeedNodes
+            .Select(s => TryParseAddress(s, out var a) ? a : null)
+            .Where(a => a is not null && !a.Equals(self))
+            .Select(a => (Host: a!.Host ?? string.Empty, Port: a!.Port ?? 0))
+            .Where(p => !string.IsNullOrWhiteSpace(p.Host) && p.Port > 0)
+            .ToList();
+
+        var probe = peerProbe ?? TryConnectAsync;
+
+        var joined = new TaskCompletionSource(TaskCreationOptions.RunContinuationsAsynchronously);
+        cluster.RegisterOnMemberUp(() => joined.TrySetResult());
+
+        _ = Task.Run(async () =>
+        {
+            while (true)
+            {
+                var winner = await Task.WhenAny(joined.Task, Task.Delay(window));
+                if (winner == joined.Task || system.WhenTerminated.IsCompleted)
+                {
+                    return;
+                }
+
+                var reachable = await AnyReachableAsync(peerSeeds, probe);
+                if (reachable is not null)
+                {
+                    // Do NOT self-form. A reachable peer means either a join already in flight (the
+                    // failure this guard exists for) or a peer about to answer — self-forming here
+                    // creates a second cluster that can never merge.
+                    logger.LogInformation(
+                        "Self-form window ({Window}) expired, but seed peer {Peer} is reachable — a join is "
+                        + "most likely already in flight (a node re-joining after a restart is not admitted "
+                        + "until its previous incarnation is removed). Continuing to wait rather than "
+                        + "forming a second cluster.",
+                        window,
+                        reachable);
+                    continue;
+                }
+
+                logger.LogWarning(
+                    "No cluster membership after {Window} and no seed peer is reachable — the peer is down "
+                    + "at boot. Self-forming a cluster at {Self} so this node becomes operational; if the "
+                    + "peer was merely partitioned (not dead), the pair is now dual-active — restart one "
+                    + "side after the partition heals (accepted availability-first trade, decision "
+                    + "2026-07-22).",
+                    window,
+                    self);
+                cluster.Join(self);
+                return;
+            }
+        });
+    }
+
+    /// <summary>How long a single peer-reachability connect attempt may take.</summary>
+    public static readonly TimeSpan ProbeTimeout = TimeSpan.FromSeconds(2);
+
+    /// <summary>
+    /// Returns the first reachable peer as <c>host:port</c>, or <see langword="null"/> when none
+    /// answered — including when there are no peer seeds at all (a lone-seed config, where an
+    /// expired window really does mean this node is on its own).
+    /// </summary>
+    private static async Task<string?> AnyReachableAsync(
+        IReadOnlyList<(string Host, int Port)> peers,
+        Func<string, int, Task<bool>> probe)
+    {
+        foreach (var (host, port) in peers)
+        {
+            bool ok;
+            try
+            {
+                ok = await probe(host, port);
+            }
+            catch (Exception)
+            {
+                // A probe that cannot even be attempted (DNS gone, socket exhaustion) is treated as
+                // unreachable — it must not throw out of the watchdog and disarm the fallback.
+                ok = false;
+            }
+
+            if (ok) return $"{host}:{port}";
+        }
+
+        return null;
+    }
+
+    private static async Task<bool> TryConnectAsync(string host, int port)
+    {
+        using var client = new System.Net.Sockets.TcpClient();
+        using var cts = new CancellationTokenSource(ProbeTimeout);
+        try
+        {
+            await client.ConnectAsync(host, port, cts.Token);
+            return client.Connected;
+        }
+        catch (Exception)
+        {
+            // Refused, unresolvable, or timed out — all mean "not reachable right now".
+            return false;
+        }
+    }
+
+    private static bool TryParseAddress(string seed, out Address address)
+    {
+        try
+        {
+            address = Address.Parse(seed);
+            return true;
+        }
+        catch (Exception)
+        {
+            // A malformed seed entry must not disarm the fallback; bad seed URIs surface from
+            // Akka's own bootstrap.
+            address = Address.AllSystems;
+            return false;
+        }
+    }
+}

src/Core/ZB.MOM.WW.OtOpcUa.Cluster/ServiceCollectionExtensions.cs

@@ -92,6 +92,23 @@ public static class ServiceCollectionExtensions
         // ActorSystem for exactly that reason: the effective value is the only one worth asserting.
         builder.AddHocon(BuildDowningHocon(options), HoconAddMode.Prepend);
 
+        // InitJoin self-form fallback (decision 2026-07-22): registered as an Akka.Hosting startup
+        // task so every host built through this bootstrap — production AND the test hosts in
+        // Cluster.Tests — arms it identically. Guarded inside Arm: disabled when SelfFormAfter is
+        // unset; inert when this node is not in its own seed list (site nodes seeded only by
+        // central-1 must wait, not island).
+        // ILoggerFactory is fully qualified: `using Microsoft.Extensions.Logging` would make the
+        // LogLevel in ConfigureLoggers above ambiguous with Akka.Event.LogLevel.
+        var fallbackLogger =
+            (serviceProvider.GetService<Microsoft.Extensions.Logging.ILoggerFactory>()
+             ?? Microsoft.Extensions.Logging.Abstractions.NullLoggerFactory.Instance)
+            .CreateLogger("ZB.MOM.WW.OtOpcUa.Cluster.ClusterBootstrapFallback");
+        builder.AddStartup((system, _) =>
+        {
+            ClusterBootstrapFallback.Arm(system, options, fallbackLogger);
+            return Task.CompletedTask;
+        });
+
         return builder;
     }
 

src/Server/ZB.MOM.WW.OtOpcUa.AdminUI/Components/Pages/Clusters/ClusterRedundancy.razor

@@ -1,10 +1,16 @@
 @page "/clusters/{ClusterId}/redundancy"
 @attribute [Authorize(Policy = AdminUiPolicies.AuthenticatedRead)]
 @rendermode RenderMode.InteractiveServer
+@using Microsoft.AspNetCore.Authorization
 @using Microsoft.EntityFrameworkCore
+@using ZB.MOM.WW.OtOpcUa.AdminUI.Redundancy
 @using ZB.MOM.WW.OtOpcUa.Configuration
 @using ZB.MOM.WW.OtOpcUa.Configuration.Entities
+@using ZB.MOM.WW.OtOpcUa.ControlPlane.Redundancy
 @inject IDbContextFactory<OtOpcUaConfigDbContext> DbFactory
+@inject IManualFailoverService FailoverService
+@inject AuthenticationStateProvider AuthState
+@inject IAuthorizationService AuthorizationService
 
 @if (!_loaded)
 {
@@ -45,6 +51,67 @@ else
         </div>
     </section>
 
+    <section class="panel rise mt-3" style="animation-delay:.11s">
+        <div class="panel-head">Live redundancy</div>
+        <div style="padding:1rem">
+            <div class="kv">
+                <span class="k">Driver Primary</span>
+                <span class="v mono">@(_failover.Snapshot?.PrimaryAddress ?? "—")</span>
+            </div>
+            <div class="kv">
+                <span class="k">Up driver members</span>
+                <span class="v mono">
+                    @(_failover.Snapshot is { DriverAddresses.Count: > 0 } s
+                        ? string.Join(", ", s.DriverAddresses)
+                        : "—")
+                </span>
+            </div>
+
+            <p class="text-muted small mt-2 mb-0">
+                Read live from cluster state on this node. <strong>Mesh-wide scope:</strong> the Primary is
+                elected once per Akka mesh, not per cluster row — in the current single-mesh topology this
+                acts on the whole mesh's Primary, which may be a node of another
+                <span class="mono">Cluster</span>. See <span class="mono">docs/Redundancy.md</span>.
+            </p>
+
+            <AuthorizeView Policy="@ManualFailoverPageModel.RequiredPolicy">
+                <Authorized>
+                    <div class="mt-3">
+                        <button class="btn btn-sm btn-outline-danger"
+                                disabled="@(!_failover.CanFailOver)"
+                                title="@(_failover.DisabledReason ?? "Gracefully move the driver Primary to its peer")"
+                                @onclick="() => _failover.RequestFailover()">
+                            Trigger failover
+                        </button>
+                        @if (_failover.DisabledReason is { } reason)
+                        {
+                            <span class="text-muted small ms-2">@reason</span>
+                        }
+                    </div>
+
+                    @if (_failover.ConfirmOpen)
+                    {
+                        <div class="panel notice mt-3">
+                            <strong>Fail over the driver Primary?</strong>
+                            <ul class="mb-2 mt-2">
+                                <li><span class="mono">@(_failover.Snapshot?.PrimaryAddress)</span> leaves the cluster and its process restarts.</li>
+                                <li>Its peer becomes Primary and advertises <span class="mono">ServiceLevel</span> 250.</li>
+                                <li>Connected OPC UA clients re-select the new Primary.</li>
+                            </ul>
+                            <button class="btn btn-sm btn-danger" @onclick="ConfirmFailoverAsync">Confirm failover</button>
+                            <button class="btn btn-sm btn-outline-secondary ms-2" @onclick="() => _failover.CancelFailover()">Cancel</button>
+                        </div>
+                    }
+                </Authorized>
+            </AuthorizeView>
+
+            @if (_failover.StatusMessage is { } msg)
+            {
+                <div class="mt-3 @(_failover.StatusIsError ? "text-danger" : "text-success")">@msg</div>
+            }
+        </div>
+    </section>
+
     <section class="panel rise mt-3" style="animation-delay:.14s">
         <div class="panel-head">Node service-level configuration</div>
         @if (_nodes is null || _nodes.Count == 0)
@@ -91,8 +158,16 @@ else
     private ServerCluster? _cluster;
     private List<ClusterNode>? _nodes;
 
+    // Everything consequential about the failover control (peer guard, confirm flow, outcome text)
+    // lives in this pure model rather than in the markup — the repo has no bUnit, so logic left in a
+    // .razor is verified only by driving the page. Covered by ManualFailoverPageModelTests.
+    private ManualFailoverPageModel _failover = default!;
+
     protected override async Task OnInitializedAsync()
     {
+        _failover = new ManualFailoverPageModel(FailoverService);
+        _failover.Refresh();
+
         await using var db = await DbFactory.CreateDbContextAsync();
         _cluster = await db.ServerClusters.AsNoTracking()
             .FirstOrDefaultAsync(c => c.ClusterId == ClusterId);
@@ -105,4 +180,22 @@ else
         }
         _loaded = true;
     }
+
+    /// <summary>
+    /// Defense-in-depth: the button is FleetAdmin-gated in markup, but this handler runs on the
+    /// server circuit — re-check the policy before bouncing a production node (the same pattern the
+    /// certificate-store actions use).
+    /// </summary>
+    private async Task ConfirmFailoverAsync()
+    {
+        var authState = await AuthState.GetAuthenticationStateAsync();
+        if (!(await AuthorizationService.AuthorizeAsync(
+                authState.User, null, ManualFailoverPageModel.RequiredPolicy)).Succeeded)
+        {
+            _failover.CancelFailover();
+            return;
+        }
+
+        await _failover.ConfirmFailoverAsync(authState.User.Identity?.Name ?? "system");
+    }
 }

src/Server/ZB.MOM.WW.OtOpcUa.AdminUI/Redundancy/ManualFailoverPageModel.cs

@@ -0,0 +1,134 @@
+using ZB.MOM.WW.OtOpcUa.ControlPlane.Redundancy;
+using ZB.MOM.WW.OtOpcUa.Security.Auth;
+
+namespace ZB.MOM.WW.OtOpcUa.AdminUI.Redundancy;
+
+/// <summary>
+/// Presentation state for the Trigger-failover control on the cluster redundancy page.
+/// </summary>
+/// <remarks>
+///     <para>
+///         A pure model behind a thin razor shell — the same split the driver-typed tag editors use,
+///         and for the same reason: this repo has no bUnit (see
+///         <c>PageAuthorizationGuardTests</c>), so anything left inside the <c>.razor</c> is verified
+///         only by driving the page live. The peer guard, the confirm flow, and the
+///         refused-vs-succeeded outcome are consequential enough to be unit-testable, so they live
+///         here.
+///     </para>
+///     <para>
+///         What is deliberately NOT here: the markup authorization gate. It is expressed in the razor
+///         as <c>&lt;AuthorizeView Policy="@ManualFailoverPageModel.RequiredPolicy"&gt;</c> and
+///         re-checked server-side before the call, so <see cref="RequiredPolicy"/> is the single
+///         source of truth for both.
+///     </para>
+/// </remarks>
+public sealed class ManualFailoverPageModel
+{
+    /// <summary>
+    /// The policy gating the control, in markup and in the server-side re-check.
+    /// </summary>
+    /// <remarks>
+    /// <see cref="AdminUiPolicies.FleetAdmin"/> (Administrator only) rather than the
+    /// <see cref="AdminUiPolicies.ConfigEditor"/> the neighbouring cluster-authoring pages use:
+    /// this does not edit configuration, it restarts a production node. ConfigEditor also admits the
+    /// Designer role, which must not be able to bounce the Primary.
+    /// </remarks>
+    public const string RequiredPolicy = AdminUiPolicies.FleetAdmin;
+
+    private readonly IManualFailoverService _service;
+
+    /// <summary>Creates the model over the failover service.</summary>
+    /// <param name="service">The manual-failover seam; faked in tests.</param>
+    public ManualFailoverPageModel(IManualFailoverService service)
+        => _service = service ?? throw new ArgumentNullException(nameof(service));
+
+    /// <summary>The last read cluster snapshot, or <see langword="null"/> if it could not be read.</summary>
+    public ManualFailoverSnapshot? Snapshot { get; private set; }
+
+    /// <summary>Whether the confirmation dialog is open.</summary>
+    public bool ConfirmOpen { get; private set; }
+
+    /// <summary>Status text from the last completed action, if any.</summary>
+    public string? StatusMessage { get; private set; }
+
+    /// <summary>Whether <see cref="StatusMessage"/> reports a failure.</summary>
+    public bool StatusIsError { get; private set; }
+
+    /// <summary>Whether the button is enabled — a peer must exist to take over.</summary>
+    public bool CanFailOver => Snapshot?.CanFailOver == true;
+
+    /// <summary>
+    /// Why the button is disabled, or <see langword="null"/> when it is enabled. Rendered as the
+    /// button's tooltip so a disabled control explains itself.
+    /// </summary>
+    public string? DisabledReason => Snapshot is null
+        ? "Cluster state is unavailable on this node."
+        : Snapshot.CanFailOver
+            ? null
+            : $"Failover needs a driver peer to take over; this mesh has {Snapshot.DriverAddresses.Count} Up driver member(s).";
+
+    /// <summary>Re-reads live cluster state. Never throws — an unreadable cluster disables the button.</summary>
+    public void Refresh()
+    {
+        try
+        {
+            Snapshot = _service.GetSnapshot();
+        }
+        catch (Exception ex)
+        {
+            // A node whose ActorSystem is not yet up (or not a cluster provider) must render the page,
+            // not 500. The button stays disabled with DisabledReason explaining why.
+            Snapshot = null;
+            StatusIsError = true;
+            StatusMessage = $"Could not read cluster state: {ex.Message}";
+        }
+    }
+
+    /// <summary>Opens the confirmation dialog. No-op when the peer guard is closed.</summary>
+    public void RequestFailover()
+    {
+        if (!CanFailOver) return;
+        StatusMessage = null;
+        StatusIsError = false;
+        ConfirmOpen = true;
+    }
+
+    /// <summary>Closes the confirmation dialog without acting.</summary>
+    public void CancelFailover() => ConfirmOpen = false;
+
+    /// <summary>
+    /// Performs the failover the dialog is confirming.
+    /// </summary>
+    /// <param name="actor">The authenticated user name, recorded in the audit event.</param>
+    public async Task ConfirmFailoverAsync(string actor)
+    {
+        if (!ConfirmOpen) return;
+        ConfirmOpen = false;
+
+        try
+        {
+            var target = await _service.FailOverDriverPrimaryAsync(actor).ConfigureAwait(false);
+            if (target is null)
+            {
+                // The service re-evaluates the peer guard against live state, which may have changed
+                // since Refresh(). A refusal is not an error — but it must not read as a success.
+                StatusIsError = true;
+                StatusMessage = "Failover refused: no driver peer is available to take over.";
+            }
+            else
+            {
+                StatusIsError = false;
+                StatusMessage =
+                    $"Failover requested. {target} is leaving the cluster; its peer takes over as Primary "
+                    + "and the node restarts as the youngest member.";
+            }
+        }
+        catch (Exception ex)
+        {
+            StatusIsError = true;
+            StatusMessage = $"Failover failed: {ex.Message}";
+        }
+
+        Refresh();
+    }
+}

src/Server/ZB.MOM.WW.OtOpcUa.ControlPlane/Redundancy/IManualFailoverService.cs

@@ -0,0 +1,54 @@
+namespace ZB.MOM.WW.OtOpcUa.ControlPlane.Redundancy;
+
+/// <summary>
+/// A read-only view of the cluster's driver membership, as the manual-failover control sees it.
+/// </summary>
+/// <param name="PrimaryAddress">
+/// The current driver Primary — the oldest Up <c>driver</c> member, the same node
+/// <see cref="RedundancyStateActor.SelectDriverPrimary"/> names — or <see langword="null"/> when
+/// there is no Up driver member at all.
+/// </param>
+/// <param name="DriverAddresses">
+/// Every Up <c>driver</c> member, oldest first. A manual failover needs at least two: with one
+/// there is nothing to fail over to.
+/// </param>
+public sealed record ManualFailoverSnapshot(string? PrimaryAddress, IReadOnlyList<string> DriverAddresses)
+{
+    /// <summary>Whether a failover can be performed — i.e. a peer exists to take over.</summary>
+    public bool CanFailOver => DriverAddresses.Count >= 2;
+}
+
+/// <summary>
+/// Operator-initiated, graceful swap of the driver Primary.
+/// </summary>
+/// <remarks>
+///     <para>
+///         The mechanism is a cluster <b>Leave</b> of the current Primary, never a <c>Down</c>. The
+///         leaving node hands its singletons over through the normal cluster-leave phases,
+///         <c>CoordinatedShutdown</c> runs, <c>ActorSystemTerminationWatchdog</c> exits the process,
+///         the service supervisor restarts it, and it rejoins as the youngest member — so the
+///         survivor becomes the oldest Up driver member and therefore Primary. ServiceLevel 250
+///         moves with it and OPC UA clients re-select.
+///     </para>
+///     <para>
+///         <b>Mesh-scope caveat.</b> Until the per-cluster mesh work lands
+///         (<c>docs/plans/2026-07-21-per-cluster-mesh-design.md</c>) the Primary is elected once per
+///         Akka mesh, not per application <c>Cluster</c> row. On a fleet running several application
+///         clusters in one mesh this acts on the whole mesh's Primary. The UI says so.
+///     </para>
+/// </remarks>
+public interface IManualFailoverService
+{
+    /// <summary>Reads the current driver membership from live cluster state.</summary>
+    ManualFailoverSnapshot GetSnapshot();
+
+    /// <summary>
+    /// Gracefully removes the current driver Primary from the cluster so its peer takes over.
+    /// </summary>
+    /// <param name="actor">The authenticated identity performing the failover; recorded in the audit event.</param>
+    /// <returns>
+    /// The address of the node asked to leave, or <see langword="null"/> when the failover was
+    /// refused because no driver peer exists (in which case nothing was changed and nothing audited).
+    /// </returns>
+    Task<string?> FailOverDriverPrimaryAsync(string actor);
+}

src/Server/ZB.MOM.WW.OtOpcUa.ControlPlane/Redundancy/ManualFailoverService.cs

@@ -0,0 +1,128 @@
+using System.Text.Json;
+using Akka.Actor;
+using Akka.Cluster;
+using Microsoft.Extensions.Logging;
+using ZB.MOM.WW.Audit;
+
+namespace ZB.MOM.WW.OtOpcUa.ControlPlane.Redundancy;
+
+/// <summary>
+/// Default <see cref="IManualFailoverService"/> — a graceful cluster <c>Leave</c> of the oldest Up
+/// <c>driver</c> member. See the interface for the full behaviour and the mesh-scope caveat.
+/// </summary>
+public sealed class ManualFailoverService : IManualFailoverService
+{
+    /// <summary>The audit <see cref="AuditEvent.Action"/> stamped onto every manual failover.</summary>
+    public const string AuditAction = "cluster.manual-failover";
+
+    /// <summary>The audit <see cref="AuditEvent.Category"/> stamped onto every manual failover.</summary>
+    public const string AuditCategory = "Redundancy";
+
+    private readonly Func<ActorSystem> _system;
+    private readonly IAuditWriter _audit;
+    private readonly ILogger<ManualFailoverService> _log;
+
+    /// <summary>
+    /// Creates the service. The ActorSystem is taken as a factory, never as a constructed instance:
+    /// this is registered before Akka's hosted service has built the system, so resolving eagerly
+    /// would race startup (the same lazy-accessor pattern <c>ActorSystemTerminationWatchdog</c> uses).
+    /// </summary>
+    /// <param name="system">Lazy accessor for the node's ActorSystem.</param>
+    /// <param name="audit">The shared audit seam; a failover is recorded before it is performed.</param>
+    /// <param name="log">Logger for the failover decision.</param>
+    public ManualFailoverService(Func<ActorSystem> system, IAuditWriter audit, ILogger<ManualFailoverService> log)
+    {
+        _system = system ?? throw new ArgumentNullException(nameof(system));
+        _audit = audit ?? throw new ArgumentNullException(nameof(audit));
+        _log = log ?? throw new ArgumentNullException(nameof(log));
+    }
+
+    /// <inheritdoc/>
+    public ManualFailoverSnapshot GetSnapshot() => SnapshotCore(_system());
+
+    /// <inheritdoc/>
+    public async Task<string?> FailOverDriverPrimaryAsync(string actor)
+    {
+        ArgumentException.ThrowIfNullOrWhiteSpace(actor);
+
+        var system = _system();
+
+        // Dry run first so the peer guard is evaluated before anything is audited: a refused
+        // failover changed nothing and must not leave an audit row implying it did.
+        var target = FailOverCore(system, dryRun: true);
+        if (target is null)
+        {
+            _log.LogWarning(
+                "Manual failover refused for {Actor}: fewer than two Up driver members — there is no peer to take over.",
+                actor);
+            return null;
+        }
+
+        await _audit.WriteAsync(new AuditEvent
+        {
+            EventId = Guid.NewGuid(),
+            OccurredAtUtc = DateTimeOffset.UtcNow,
+            Actor = actor,
+            Action = AuditAction,
+            Category = AuditCategory,
+            SourceNode = target.ToString(),
+            Outcome = AuditOutcome.Success,
+            DetailsJson = JsonSerializer.Serialize(new
+            {
+                target = target.ToString(),
+                driverMembers = SnapshotCore(system).DriverAddresses,
+            }),
+        }).ConfigureAwait(false);
+
+        _log.LogWarning(
+            "Manual failover requested by {Actor}: asking the driver Primary {Target} to leave the cluster. "
+            + "It hands over its singletons, shuts down, and is restarted by its supervisor as the youngest "
+            + "member; its peer becomes Primary and advertises the primary ServiceLevel.",
+            actor,
+            target);
+
+        FailOverCore(system, dryRun: false);
+        return target.ToString();
+    }
+
+    /// <summary>
+    /// Selects — and, unless <paramref name="dryRun"/>, gracefully removes — the driver Primary.
+    /// </summary>
+    /// <param name="system">The ActorSystem whose cluster is acted on.</param>
+    /// <param name="dryRun">When <see langword="true"/>, selects the target without leaving.</param>
+    /// <returns>The target address, or <see langword="null"/> when fewer than two Up driver members exist.</returns>
+    public static Address? FailOverCore(ActorSystem system, bool dryRun = false)
+    {
+        ArgumentNullException.ThrowIfNull(system);
+
+        var cluster = Akka.Cluster.Cluster.Get(system);
+
+        // Intentionally identical to RedundancyStateActor.SelectDriverPrimary (oldest Up driver,
+        // Member.AgeOrdering) — the node acted on must be exactly the one the election names, or a
+        // failover would remove a node that was not Primary and change nothing. Pinned by
+        // ManualFailoverServiceTests.Parity_with_SelectDriverPrimary.
+        var drivers = OrderedDrivers(cluster.State.Members);
+
+        // A failover needs somewhere to fail over TO. With a single driver member, leaving would be
+        // a shutdown dressed up as a failover.
+        if (drivers.Count < 2) return null;
+
+        var primary = drivers[0];
+        if (!dryRun) cluster.Leave(primary.Address);
+        return primary.Address;
+    }
+
+    private static ManualFailoverSnapshot SnapshotCore(ActorSystem system)
+    {
+        var drivers = OrderedDrivers(Akka.Cluster.Cluster.Get(system).State.Members);
+
+        return new ManualFailoverSnapshot(
+            drivers.FirstOrDefault()?.Address.ToString(),
+            drivers.Select(m => m.Address.ToString()).ToList());
+    }
+
+    private static List<Member> OrderedDrivers(IEnumerable<Member> members) => members
+        .Where(m => m.Status == MemberStatus.Up && m.Roles.Contains(RedundancyStateActor.DriverRole))
+        .OrderBy(m => m, Member.AgeOrdering)
+        .ToList();
+}

src/Server/ZB.MOM.WW.OtOpcUa.Host/Program.cs

@@ -13,6 +13,7 @@ using ZB.MOM.WW.OtOpcUa.AdminUI.ScriptAnalysis;
 using ZB.MOM.WW.OtOpcUa.Cluster;
 using ZB.MOM.WW.OtOpcUa.Configuration;
 using ZB.MOM.WW.OtOpcUa.ControlPlane;
+using ZB.MOM.WW.OtOpcUa.ControlPlane.Redundancy;
 using ZB.MOM.WW.OtOpcUa.Commons.OpcUa;
 using ZB.MOM.WW.OtOpcUa.Commons.Engines;
 using ZB.MOM.WW.OtOpcUa.Core.Scripting;
@@ -372,6 +373,13 @@ if (hasAdmin)
     builder.Services.AddCascadingAuthenticationState();
     builder.Services.AddSignalR();
     builder.Services.AddOtOpcUaAdminClients();
+    // Manual failover backs the Trigger-failover control on the cluster redundancy page. Admin-only:
+    // it is driven from the UI, and the node it acts on is chosen from live cluster state, so any
+    // admin node can drive it regardless of whether that node is the Primary. The ActorSystem comes
+    // through the Func<> accessor below (never resolved at construction — this runs before Akka's
+    // hosted service has built the system).
+    builder.Services.TryAddSingleton<Func<ActorSystem>>(sp => () => sp.GetRequiredService<ActorSystem>());
+    builder.Services.AddSingleton<IManualFailoverService, ManualFailoverService>();
 }
 
 // Registered unconditionally: driver-role nodes resolve Layer-B DriverConfig secrets and have no auth/DP/AdminUI.

tests/Core/ZB.MOM.WW.OtOpcUa.Cluster.Tests/SelfFormBootstrapTests.cs

@@ -0,0 +1,259 @@
+using System.Net;
+using System.Net.Sockets;
+using Akka.Actor;
+using Akka.Cluster;
+using Akka.Hosting;
+using Microsoft.Extensions.DependencyInjection;
+using Microsoft.Extensions.Hosting;
+using Microsoft.Extensions.Options;
+using Shouldly;
+using Xunit;
+
+namespace ZB.MOM.WW.OtOpcUa.Cluster.Tests;
+
+/// <summary>
+/// Guards the InitJoin self-form fallback — whether a node whose peer is down at boot ever becomes
+/// operational.
+/// </summary>
+/// <remarks>
+///     <para>
+///         <b>Why these tests start real hosts through the production bootstrap.</b> Like
+///         <see cref="SplitBrainResolverActivationTests"/>, the question is what a running node
+///         actually does, not what an options object holds. Every host here is built with
+///         <see cref="ServiceCollectionExtensions.WithOtOpcUaClusterBootstrap"/> — the shipped entry
+///         point — so a change that stops arming the fallback (or arms it from a place production
+///         does not reach) turns these red. A test that called
+///         <see cref="ClusterBootstrapFallback.Arm"/> itself would pin the test's wiring instead.
+///     </para>
+///     <para>
+///         <b>Seed ordering is load-bearing.</b> The node under test lists itself SECOND in its own
+///         seed list, behind a dead peer port. Akka only lets the FIRST listed seed self-join, so
+///         nothing but the fallback can bring these nodes Up — which is what makes the negative
+///         scenarios (disabled window, non-seed node) meaningful rather than vacuous. Each of those
+///         carries a positive control (an explicit <c>Cluster.Join(SelfAddress)</c>) proving the node
+///         was formable all along and only the fallback was absent.
+///     </para>
+/// </remarks>
+public sealed class SelfFormBootstrapTests
+{
+    private static int FreePort()
+    {
+        var listener = new TcpListener(IPAddress.Loopback, 0);
+        listener.Start();
+        var port = ((IPEndPoint)listener.LocalEndpoint).Port;
+        listener.Stop();
+        return port;
+    }
+
+    private static async Task<IHost> StartNodeAsync(
+        string systemName,
+        int selfPort,
+        int peerPort,
+        TimeSpan? selfFormAfter,
+        bool selfInSeeds = true)
+    {
+        var options = new AkkaClusterOptions
+        {
+            SystemName = systemName,
+            Hostname = "127.0.0.1",
+            PublicHostname = "127.0.0.1",
+            Port = selfPort,
+            Roles = new[] { "admin", "driver" },
+            SelfFormAfter = selfFormAfter,
+
+            // Self listed SECOND, behind a port nothing is listening on: Akka lets only the first
+            // seed self-join, so this node can never form a cluster on its own except via the
+            // fallback.
+            SeedNodes = selfInSeeds
+                ? new[]
+                {
+                    $"akka.tcp://{systemName}@127.0.0.1:{peerPort}",
+                    $"akka.tcp://{systemName}@127.0.0.1:{selfPort}",
+                }
+                : new[] { $"akka.tcp://{systemName}@127.0.0.1:{peerPort}" },
+        };
+
+        var builder = Host.CreateDefaultBuilder();
+        builder.ConfigureServices(services =>
+        {
+            services.AddSingleton<IOptions<AkkaClusterOptions>>(Options.Create(options));
+            services.AddAkka(systemName, (ab, sp) => ab.WithOtOpcUaClusterBootstrap(sp));
+        });
+
+        var host = builder.Build();
+        await host.StartAsync();
+        return host;
+    }
+
+    /// <summary>Polls cluster state until it holds at least one Up member, or the deadline passes.</summary>
+    private static async Task<bool> WaitForUpMemberAsync(ActorSystem system, TimeSpan timeout)
+    {
+        var cluster = Akka.Cluster.Cluster.Get(system);
+        var deadline = DateTime.UtcNow + timeout;
+        while (DateTime.UtcNow < deadline)
+        {
+            if (cluster.State.Members.Any(m => m.Status == MemberStatus.Up))
+            {
+                return true;
+            }
+
+            await Task.Delay(200);
+        }
+
+        return cluster.State.Members.Any(m => m.Status == MemberStatus.Up);
+    }
+
+    private static async Task StopAsync(IHost host)
+    {
+        await host.StopAsync();
+        host.Dispose();
+    }
+
+    /// <summary>
+    /// An unconfigured deployment gets the fallback: a pair node that cold-starts while its peer is
+    /// down must come up on its own, not wait for an operator.
+    /// </summary>
+    [Fact]
+    public void SelfFormAfter_defaults_to_ten_seconds()
+    {
+        new AkkaClusterOptions().SelfFormAfter.ShouldBe(TimeSpan.FromSeconds(10));
+    }
+
+    /// <summary>
+    /// The core scenario: a node listed as a non-first seed, booting with its peer dead, becomes Up
+    /// on its own after the window. Without the fallback it loops on InitJoin forever.
+    /// </summary>
+    [Fact]
+    public async Task Lone_non_first_seed_self_forms_after_the_window()
+    {
+        var selfPort = FreePort();
+        var peerPort = FreePort();
+        var host = await StartNodeAsync(
+            "otopcua-selfform-1", selfPort, peerPort, TimeSpan.FromSeconds(2));
+        try
+        {
+            var system = host.Services.GetRequiredService<ActorSystem>();
+
+            (await WaitForUpMemberAsync(system, TimeSpan.FromSeconds(20)))
+                .ShouldBeTrue("a lone non-first seed must self-form and come Up unattended");
+        }
+        finally
+        {
+            await StopAsync(host);
+        }
+    }
+
+    /// <summary>
+    /// The opt-out is real: with the window unset the node keeps waiting on InitJoin. The positive
+    /// control proves the node was formable and only the fallback was missing — otherwise this test
+    /// would pass just as happily against a node that could never join anything.
+    /// </summary>
+    [Fact]
+    public async Task Disabled_fallback_keeps_waiting()
+    {
+        var selfPort = FreePort();
+        var peerPort = FreePort();
+        var host = await StartNodeAsync("otopcua-selfform-2", selfPort, peerPort, selfFormAfter: null);
+        try
+        {
+            var system = host.Services.GetRequiredService<ActorSystem>();
+
+            (await WaitForUpMemberAsync(system, TimeSpan.FromSeconds(6)))
+                .ShouldBeFalse("with SelfFormAfter unset the node must keep waiting on InitJoin");
+
+            // Positive control: the node WAS formable all along.
+            var cluster = Akka.Cluster.Cluster.Get(system);
+            cluster.Join(cluster.SelfAddress);
+            (await WaitForUpMemberAsync(system, TimeSpan.FromSeconds(20)))
+                .ShouldBeTrue("positive control — an explicit self-join must bring this node Up");
+        }
+        finally
+        {
+            await StopAsync(host);
+        }
+    }
+
+    /// <summary>
+    /// Regression for the live-gate islanding defect (2026-07-22). A peer that is <b>reachable</b>
+    /// but not yet admitting the join must NOT be treated as "peer down at boot".
+    /// </summary>
+    /// <remarks>
+    ///     <para>
+    ///         On the docker-dev rig, a node bounced by a manual failover restarted, received
+    ///         <c>InitJoinAck</c> from its live peer — the join was in flight and healthy — but did
+    ///         not get the Welcome inside the window, because the peer's ring still held the node's
+    ///         previous incarnation. The fallback fired on the timer and the node formed a
+    ///         <i>second</i> cluster, islanded until an operator restarted it. The original design
+    ///         assumed <c>Join(SelfAddress)</c> would be ignored mid-handshake; it is not.
+    ///     </para>
+    ///     <para>
+    ///         The peer here is a bare <see cref="TcpListener"/> that accepts connections and speaks
+    ///         no Akka at all — reachable at the transport level, never answering a join. That is the
+    ///         defect's shape with nothing faked: under the pre-fix code this node self-formed on the
+    ///         timer; under the guard it keeps waiting. The positive control then proves the guard is
+    ///         a <i>guard</i> and not a disablement: drop the listener and the same node self-forms.
+    ///     </para>
+    /// </remarks>
+    [Fact]
+    public async Task Reachable_peer_suppresses_self_forming_until_it_goes_away()
+    {
+        var selfPort = FreePort();
+        var peerPort = FreePort();
+
+        // A peer that accepts TCP and does nothing else — reachable, but no join will ever complete.
+        var peer = new TcpListener(IPAddress.Loopback, peerPort);
+        peer.Start();
+
+        var host = await StartNodeAsync(
+            "otopcua-selfform-4", selfPort, peerPort, TimeSpan.FromSeconds(2));
+        try
+        {
+            var system = host.Services.GetRequiredService<ActorSystem>();
+
+            // Several windows pass. Pre-fix this self-formed after the first one.
+            (await WaitForUpMemberAsync(system, TimeSpan.FromSeconds(10)))
+                .ShouldBeFalse("a reachable peer means a join may be in flight — self-forming here islands this node");
+
+            // Positive control: the peer really goes away, and now the fallback does its job.
+            peer.Stop();
+
+            (await WaitForUpMemberAsync(system, TimeSpan.FromSeconds(30)))
+                .ShouldBeTrue("once the peer is genuinely gone the fallback must still self-form");
+        }
+        finally
+        {
+            try { peer.Stop(); } catch (SocketException) { /* already stopped by the test body */ }
+            await StopAsync(host);
+        }
+    }
+
+    /// <summary>
+    /// THE island guard. A node that is not in its own seed list — the current docker-dev site-node
+    /// topology, where site-a/site-b list only central-1 — must never self-form: it would island
+    /// itself from the real seeds permanently. Positive control as above.
+    /// </summary>
+    [Fact]
+    public async Task Site_node_seeded_only_by_central_never_self_forms()
+    {
+        var selfPort = FreePort();
+        var peerPort = FreePort();
+        var host = await StartNodeAsync(
+            "otopcua-selfform-3", selfPort, peerPort, TimeSpan.FromSeconds(1), selfInSeeds: false);
+        try
+        {
+            var system = host.Services.GetRequiredService<ActorSystem>();
+
+            (await WaitForUpMemberAsync(system, TimeSpan.FromSeconds(5)))
+                .ShouldBeFalse("a node absent from its own seed list must wait, not island itself");
+
+            var cluster = Akka.Cluster.Cluster.Get(system);
+            cluster.Join(cluster.SelfAddress);
+            (await WaitForUpMemberAsync(system, TimeSpan.FromSeconds(20)))
+                .ShouldBeTrue("positive control — an explicit self-join must bring this node Up");
+        }
+        finally
+        {
+            await StopAsync(host);
+        }
+    }
+}

tests/Server/ZB.MOM.WW.OtOpcUa.AdminUI.Tests/Redundancy/ManualFailoverPageModelTests.cs

@@ -0,0 +1,185 @@
+using Shouldly;
+using Xunit;
+using ZB.MOM.WW.OtOpcUa.AdminUI.Redundancy;
+using ZB.MOM.WW.OtOpcUa.ControlPlane.Redundancy;
+using ZB.MOM.WW.OtOpcUa.Security.Auth;
+
+namespace ZB.MOM.WW.OtOpcUa.AdminUI.Tests.Redundancy;
+
+/// <summary>
+/// Covers the Trigger-failover control's behaviour. The repo has no bUnit (see
+/// <c>PageAuthorizationGuardTests</c>), so the consequential parts — the peer guard, the confirm
+/// flow, and the refused-vs-succeeded outcome — live in a pure model that the razor is a shell over,
+/// and are tested here rather than left to live verification alone.
+/// </summary>
+public sealed class ManualFailoverPageModelTests
+{
+    private sealed class FakeFailoverService : IManualFailoverService
+    {
+        public ManualFailoverSnapshot Next { get; set; } = new("akka.tcp://otopcua@a:4053", new[]
+        {
+            "akka.tcp://otopcua@a:4053", "akka.tcp://otopcua@b:4053",
+        });
+
+        public int FailOverCalls { get; private set; }
+        public List<string> Actors { get; } = new();
+        public string? Result { get; set; } = "akka.tcp://otopcua@a:4053";
+        public Exception? Throw { get; set; }
+        public Exception? ThrowOnSnapshot { get; set; }
+
+        public ManualFailoverSnapshot GetSnapshot()
+            => ThrowOnSnapshot is not null ? throw ThrowOnSnapshot : Next;
+
+        public Task<string?> FailOverDriverPrimaryAsync(string actor)
+        {
+            FailOverCalls++;
+            Actors.Add(actor);
+            if (Throw is not null) throw Throw;
+            return Task.FromResult(Result);
+        }
+    }
+
+    private static (ManualFailoverPageModel Model, FakeFailoverService Service) Build()
+    {
+        var svc = new FakeFailoverService();
+        var model = new ManualFailoverPageModel(svc);
+        model.Refresh();
+        return (model, svc);
+    }
+
+    /// <summary>
+    /// The control is Administrator-only. Not <see cref="AdminUiPolicies.ConfigEditor"/> — which the
+    /// neighbouring cluster-authoring pages use, and which also admits Designer: this restarts a
+    /// production node rather than editing configuration.
+    /// </summary>
+    [Fact]
+    public void Control_is_gated_on_the_fleet_admin_policy()
+    {
+        ManualFailoverPageModel.RequiredPolicy.ShouldBe(AdminUiPolicies.FleetAdmin);
+        ManualFailoverPageModel.RequiredPolicy.ShouldNotBe(AdminUiPolicies.ConfigEditor);
+    }
+
+    /// <summary>With a peer present the button is live and explains nothing away.</summary>
+    [Fact]
+    public void Button_is_enabled_when_a_driver_peer_exists()
+    {
+        var (model, _) = Build();
+
+        model.CanFailOver.ShouldBeTrue();
+        model.DisabledReason.ShouldBeNull();
+        model.Snapshot!.PrimaryAddress.ShouldBe("akka.tcp://otopcua@a:4053");
+    }
+
+    /// <summary>
+    /// THE peer guard. On a lone driver node a "failover" is a shutdown: the button must be disabled
+    /// and say why, and requesting it must not even open the dialog.
+    /// </summary>
+    [Fact]
+    public void Button_is_disabled_with_a_reason_when_there_is_no_peer()
+    {
+        var (model, svc) = Build();
+        svc.Next = new ManualFailoverSnapshot("akka.tcp://otopcua@a:4053", new[] { "akka.tcp://otopcua@a:4053" });
+        model.Refresh();
+
+        model.CanFailOver.ShouldBeFalse();
+        model.DisabledReason.ShouldNotBeNull().ShouldContain("1 Up driver member");
+
+        model.RequestFailover();
+        model.ConfirmOpen.ShouldBeFalse("a guarded-off control must not open its confirmation dialog");
+    }
+
+    /// <summary>The confirm flow calls the service exactly once, with the authenticated user.</summary>
+    [Fact]
+    public async Task Confirm_flow_calls_the_service_exactly_once()
+    {
+        var (model, svc) = Build();
+
+        model.RequestFailover();
+        model.ConfirmOpen.ShouldBeTrue();
+        await model.ConfirmFailoverAsync("alice");
+
+        svc.FailOverCalls.ShouldBe(1);
+        svc.Actors.ShouldBe(new[] { "alice" });
+        model.ConfirmOpen.ShouldBeFalse();
+        model.StatusIsError.ShouldBeFalse();
+        model.StatusMessage.ShouldNotBeNull().ShouldContain("akka.tcp://otopcua@a:4053");
+    }
+
+    /// <summary>
+    /// Confirming without an open dialog does nothing — a stray double-submit on the circuit must not
+    /// bounce a second node.
+    /// </summary>
+    [Fact]
+    public async Task Confirming_twice_only_fails_over_once()
+    {
+        var (model, svc) = Build();
+
+        model.RequestFailover();
+        await model.ConfirmFailoverAsync("alice");
+        await model.ConfirmFailoverAsync("alice");
+
+        svc.FailOverCalls.ShouldBe(1);
+    }
+
+    /// <summary>Cancelling closes the dialog and calls nothing.</summary>
+    [Fact]
+    public async Task Cancel_does_not_call_the_service()
+    {
+        var (model, svc) = Build();
+
+        model.RequestFailover();
+        model.CancelFailover();
+        await model.ConfirmFailoverAsync("alice");
+
+        model.ConfirmOpen.ShouldBeFalse();
+        svc.FailOverCalls.ShouldBe(0);
+    }
+
+    /// <summary>
+    /// The service re-evaluates the peer guard against live state, which can have changed since the
+    /// page rendered. A refusal must surface as a failure — reporting it as a success would tell an
+    /// operator the Primary moved when it did not.
+    /// </summary>
+    [Fact]
+    public async Task Refusal_is_reported_as_an_error_not_a_success()
+    {
+        var (model, svc) = Build();
+        svc.Result = null;
+
+        model.RequestFailover();
+        await model.ConfirmFailoverAsync("alice");
+
+        model.StatusIsError.ShouldBeTrue();
+        model.StatusMessage.ShouldNotBeNull().ShouldContain("refused");
+    }
+
+    /// <summary>A throwing service surfaces as an error, not an unhandled circuit exception.</summary>
+    [Fact]
+    public async Task Service_failure_is_surfaced_not_thrown()
+    {
+        var (model, svc) = Build();
+        svc.Throw = new InvalidOperationException("cluster gone");
+
+        model.RequestFailover();
+        await model.ConfirmFailoverAsync("alice");
+
+        model.StatusIsError.ShouldBeTrue();
+        model.StatusMessage.ShouldNotBeNull().ShouldContain("cluster gone");
+    }
+
+    /// <summary>
+    /// An unreadable cluster renders a disabled control, not a 500 — the page is also the place an
+    /// operator looks when the node is unhealthy.
+    /// </summary>
+    [Fact]
+    public void Unreadable_cluster_state_disables_the_control_instead_of_throwing()
+    {
+        var svc = new FakeFailoverService { ThrowOnSnapshot = new InvalidOperationException("no cluster") };
+        var model = new ManualFailoverPageModel(svc);
+
+        Should.NotThrow(model.Refresh);
+        model.Snapshot.ShouldBeNull();
+        model.CanFailOver.ShouldBeFalse();
+        model.DisabledReason.ShouldNotBeNull().ShouldContain("unavailable");
+    }
+}

tests/Server/ZB.MOM.WW.OtOpcUa.ControlPlane.Tests/ManualFailoverServiceTests.cs

@@ -0,0 +1,226 @@
+using Akka.Actor;
+using Akka.Cluster;
+using Akka.Configuration;
+using Microsoft.Extensions.Logging.Abstractions;
+using Shouldly;
+using Xunit;
+using ZB.MOM.WW.Audit;
+using ZB.MOM.WW.OtOpcUa.Commons.Messages.Redundancy;
+using ZB.MOM.WW.OtOpcUa.ControlPlane.Redundancy;
+
+namespace ZB.MOM.WW.OtOpcUa.ControlPlane.Tests;
+
+/// <summary>
+/// Pins <b>which</b> node a manual failover removes, on a real two-node cluster built so that the
+/// oldest member is not the lowest-addressed one.
+/// </summary>
+/// <remarks>
+///     <para>
+///         The fixture mirrors <see cref="RedundancyPrimaryElectionTests"/> deliberately, and for the
+///         same reason: age order and address order coincide on a freshly-formed cluster and diverge
+///         after any restart. A failover that targeted the <i>role leader</i> (lowest address) rather
+///         than the oldest member would, after a restart, remove the node that is <b>not</b> hosting
+///         the singletons — leaving the Primary exactly where it was while taking an innocent node
+///         down. Node A therefore binds the <b>higher</b> port and joins first.
+///     </para>
+///     <para>
+///         The parity test is the durable one: it pins the service's target to the member
+///         <see cref="RedundancyStateActor"/> actually names Primary, so the two five-line queries
+///         cannot drift apart silently.
+///     </para>
+/// </remarks>
+public sealed class ManualFailoverServiceTests : IAsyncLifetime
+{
+    // Distinct from RedundancyPrimaryElectionTests' 19530/19531 — xunit may run the two classes in
+    // parallel, and both need deterministic (non-zero) ports.
+    private const int OldestPort = 19_541;   // joins FIRST  -> oldest, but the HIGHER address
+    private const int YoungestPort = 19_540; // joins SECOND -> role leader, the LOWER address
+
+    private ActorSystem? _oldest;
+    private ActorSystem? _youngest;
+
+    private static Config NodeConfig(int port, string systemName, int seedPort) =>
+        ConfigurationFactory.ParseString($$"""
+        akka {
+            loglevel = "WARNING"
+            actor.provider = "Akka.Cluster.ClusterActorRefProvider, Akka.Cluster"
+            remote.dot-netty.tcp {
+                hostname = "127.0.0.1"
+                port = {{port}}
+            }
+            cluster {
+                seed-nodes = ["akka.tcp://{{systemName}}@127.0.0.1:{{seedPort}}"]
+                roles = ["admin", "driver"]
+                min-nr-of-members = 1
+                run-coordinated-shutdown-when-down = off
+                downing-provider-class = ""
+            }
+        }
+        """);
+
+    public async Task InitializeAsync()
+    {
+        // Order matters: the seed forms the cluster and is therefore the oldest member.
+        _oldest = ActorSystem.Create("manual-failover", NodeConfig(OldestPort, "manual-failover", OldestPort));
+        await WaitForUpAsync(_oldest, expectedMembers: 1);
+
+        _youngest = ActorSystem.Create("manual-failover", NodeConfig(YoungestPort, "manual-failover", OldestPort));
+        await WaitForUpAsync(_oldest, expectedMembers: 2);
+        await WaitForUpAsync(_youngest, expectedMembers: 2);
+    }
+
+    public async Task DisposeAsync()
+    {
+        if (_youngest is not null) await _youngest.Terminate();
+        if (_oldest is not null) await _oldest.Terminate();
+    }
+
+    private static async Task WaitForUpAsync(ActorSystem system, int expectedMembers)
+    {
+        var cluster = Akka.Cluster.Cluster.Get(system);
+        var deadline = DateTime.UtcNow + TimeSpan.FromSeconds(30);
+        while (DateTime.UtcNow < deadline)
+        {
+            if (cluster.State.Members.Count(m => m.Status == MemberStatus.Up) >= expectedMembers) return;
+            await Task.Delay(100);
+        }
+
+        throw new TimeoutException(
+            $"cluster on port {cluster.SelfAddress.Port} never saw {expectedMembers} Up members "
+            + $"(saw {cluster.State.Members.Count(m => m.Status == MemberStatus.Up)})");
+    }
+
+    private static (ManualFailoverService Service, RecordingAuditWriter Audit) BuildService(ActorSystem system)
+    {
+        var audit = new RecordingAuditWriter();
+        return (new ManualFailoverService(() => system, audit, NullLogger<ManualFailoverService>.Instance), audit);
+    }
+
+    /// <summary>
+    /// Fixture sanity check: if age order and address order did not actually diverge, the assertion
+    /// below would pass for the wrong reason.
+    /// </summary>
+    [Fact]
+    public void Fixture_actually_produces_divergent_age_and_address_ordering()
+    {
+        Akka.Cluster.Cluster.Get(_oldest!).State.RoleLeader("driver")!.Port.ShouldBe(
+            YoungestPort,
+            "the fixture is only meaningful if the lowest-addressed node is the YOUNGER one");
+    }
+
+    /// <summary>
+    /// The failover removes the oldest driver member — the node hosting the singletons — not the
+    /// role leader, and the survivor is then elected Primary.
+    /// </summary>
+    [Fact]
+    public async Task FailOver_targets_the_oldest_driver_member_not_the_role_leader()
+    {
+        // Driven from the survivor, as production does: the admin node asks the Primary to leave.
+        var (service, audit) = BuildService(_youngest!);
+
+        var target = await service.FailOverDriverPrimaryAsync("tester");
+
+        target.ShouldNotBeNull();
+        target.ShouldContain(
+            $":{OldestPort}",
+            Case.Sensitive,
+            "the failover must remove the OLDEST driver member (which holds the singletons), not the role leader");
+
+        // The leaving node runs CoordinatedShutdown and terminates — that is what makes the process
+        // exit and be restarted as the youngest member.
+        await _oldest!.WhenTerminated.WaitAsync(TimeSpan.FromSeconds(30));
+
+        // ... and the survivor is now what RedundancyStateActor names Primary.
+        var snapshot = await CaptureSnapshotAsync(_youngest!);
+        var primaries = snapshot.Where(n => n.Role == RedundancyRole.Primary).ToList();
+        primaries.Count.ShouldBe(1, "exactly one node may be Primary after the swap");
+        primaries[0].NodeId.Value.ShouldBe($"127.0.0.1:{YoungestPort}");
+
+        audit.Events.Count.ShouldBe(1, "a manual failover must be audited exactly once");
+        audit.Events[0].Action.ShouldBe(ManualFailoverService.AuditAction);
+        audit.Events[0].Actor.ShouldBe("tester");
+        audit.Events[0].SourceNode.ShouldNotBeNull().ShouldContain($":{OldestPort}");
+    }
+
+    /// <summary>
+    /// With no peer to take over, a failover is a disguised shutdown. It must refuse — and must not
+    /// audit, because nothing happened.
+    /// </summary>
+    [Fact]
+    public async Task FailOver_refuses_when_no_driver_peer_exists()
+    {
+        const int LonePort = 19_542;
+        var lone = ActorSystem.Create("manual-failover-lone", NodeConfig(LonePort, "manual-failover-lone", LonePort));
+        try
+        {
+            await WaitForUpAsync(lone, expectedMembers: 1);
+            var (service, audit) = BuildService(lone);
+
+            var result = await service.FailOverDriverPrimaryAsync("tester");
+
+            result.ShouldBeNull("a single driver member has nowhere to fail over to");
+            audit.Events.ShouldBeEmpty("a refused failover changed nothing and must not be audited");
+
+            // Positive assert: the node is still a live Up member — the refusal did not half-leave it.
+            Akka.Cluster.Cluster.Get(lone).State.Members
+                .Count(m => m.Status == MemberStatus.Up)
+                .ShouldBe(1);
+            lone.WhenTerminated.IsCompleted.ShouldBeFalse();
+        }
+        finally
+        {
+            await lone.Terminate();
+        }
+    }
+
+    /// <summary>
+    /// The service's target and <see cref="RedundancyStateActor.SelectDriverPrimary"/> are two copies
+    /// of the same query. This pins them together, so a change to one that is not made to the other
+    /// fails here rather than in production, where it would mean failing over the wrong node.
+    /// </summary>
+    [Fact]
+    public async Task Parity_with_SelectDriverPrimary()
+    {
+        var elected = RedundancyStateActor.SelectDriverPrimary(
+            Akka.Cluster.Cluster.Get(_oldest!).State.Members);
+
+        var target = ManualFailoverService.FailOverCore(_oldest!, dryRun: true);
+        var snapshotPrimary = BuildService(_oldest!).Service.GetSnapshot().PrimaryAddress;
+
+        elected.ShouldNotBeNull();
+        target.ShouldBe(elected, "the node acted on must be exactly the node the election names Primary");
+        snapshotPrimary.ShouldBe(elected.ToString(), "the UI panel must name the same node the button acts on");
+
+        // The dry run really was dry.
+        await Task.Delay(500);
+        Akka.Cluster.Cluster.Get(_oldest!).State.Members
+            .Count(m => m.Status == MemberStatus.Up)
+            .ShouldBe(2, "a dry run must not remove anyone");
+    }
+
+    private static async Task<IReadOnlyList<NodeRedundancyState>> CaptureSnapshotAsync(ActorSystem system)
+    {
+        var tcs = new TaskCompletionSource<RedundancyStateChanged>(
+            TaskCreationOptions.RunContinuationsAsynchronously);
+
+        system.ActorOf(
+            RedundancyStateActor.Props(broadcast: msg =>
+            {
+                if (msg is RedundancyStateChanged changed) tcs.TrySetResult(changed);
+            }),
+            $"redundancy-{Guid.NewGuid():N}");
+
+        return (await tcs.Task.WaitAsync(TimeSpan.FromSeconds(15))).Nodes;
+    }
+
+    private sealed class RecordingAuditWriter : IAuditWriter
+    {
+        public List<AuditEvent> Events { get; } = new();
+
+        public Task WriteAsync(AuditEvent auditEvent, CancellationToken cancellationToken = default)
+        {
+            Events.Add(auditEvent);
+            return Task.CompletedTask;
+        }
+    }
+}