docs(redundancy): SelfFormAfter closes the seed-node bootstrap gap
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
This commit is contained in:
Joseph Doherty
@@ -224,6 +224,8 @@ The server supports non-transparent warm/hot redundancy via the `Redundancy` sec
|
|||||||
- **Downing is `auto-down`, not `keep-oldest`.** `Cluster:SplitBrainResolverStrategy` (default `auto-down`) selects the provider via `ServiceCollectionExtensions.BuildDowningHocon`; an unknown value fails host start. A two-node pair running the previous `keep-oldest` + `down-if-alone` **could not survive a crash of the oldest node**: Akka's `KeepOldest.OldestDecision` only lets `down-if-alone` rescue a side holding ≥ 2 members, so the 1-vs-1 survivor downs *itself* and exits. `down-if-alone` is a 3+-node feature. The accepted trade for `auto-down` is dual-active during a genuine network partition. **Its live gate is still open** — the pathology only appears 1-vs-1 and docker-dev is a single six-node mesh, so the crash-the-oldest drill is deferred to the per-cluster mesh work.
|
- **Downing is `auto-down`, not `keep-oldest`.** `Cluster:SplitBrainResolverStrategy` (default `auto-down`) selects the provider via `ServiceCollectionExtensions.BuildDowningHocon`; an unknown value fails host start. A two-node pair running the previous `keep-oldest` + `down-if-alone` **could not survive a crash of the oldest node**: Akka's `KeepOldest.OldestDecision` only lets `down-if-alone` rescue a side holding ≥ 2 members, so the 1-vs-1 survivor downs *itself* and exits. `down-if-alone` is a 3+-node feature. The accepted trade for `auto-down` is dual-active during a genuine network partition. **Its live gate is still open** — the pathology only appears 1-vs-1 and docker-dev is a single six-node mesh, so the crash-the-oldest drill is deferred to the per-cluster mesh work.
|
||||||
- **The Primary is the oldest Up `driver` member, not the role leader.** `RedundancyStateActor` previously used `ClusterState.RoleLeader("driver")` (lowest *address*), while `ClusterSingletonManager` places singletons on the *oldest*. They agree only on a freshly-formed cluster and diverge after any restart, so the Primary-gated data plane (writes, alarm acks, alerts emit, alarm-history drain) could enable on a node not hosting the singletons. `NodeRedundancyState.IsRoleLeaderForDriver` was renamed **`IsDriverPrimary`**.
|
- **The Primary is the oldest Up `driver` member, not the role leader.** `RedundancyStateActor` previously used `ClusterState.RoleLeader("driver")` (lowest *address*), while `ClusterSingletonManager` places singletons on the *oldest*. They agree only on a freshly-formed cluster and diverge after any restart, so the Primary-gated data plane (writes, alarm acks, alerts emit, alarm-history drain) could enable on a node not hosting the singletons. `NodeRedundancyState.IsRoleLeaderForDriver` was renamed **`IsDriverPrimary`**.
|
||||||
|
|
||||||
|
**A third bootstrap gap closed 2026-07-22 — downing was never the whole story.** Even under `auto-down`, a node cold-starting while its peer was dead **never came Up**: Akka lets only the **first** listed seed form a *new* cluster, so every other node loops on `InitJoin` forever. `Cluster:SelfFormAfter` (`TimeSpan?`, default 10 s, `null`/`≤0` disables) arms `ClusterBootstrapFallback` from `WithOtOpcUaClusterBootstrap` via an Akka.Hosting `AddStartup` task — it waits that long for membership, then `Cluster.Join(SelfAddress)`. **It fires only when this node's own address is in its own `SeedNodes`**: a node seeded only by someone else (today's docker-dev site nodes, which list only `central-1`) must wait rather than island itself permanently. Pinned by `SelfFormBootstrapTests`, whose two negative cases each carry a positive-control self-join so they cannot pass vacuously. See `docs/Redundancy.md` §"Bootstrap: the self-form fallback".
|
||||||
|
|
||||||
Assert the **effective** cluster config off a running `ActorSystem` (as `SplitBrainResolverActivationTests` does), never the typed options or the akka.conf text — several HOCON fragments compete and the losing one still reads correctly in the file.
|
Assert the **effective** cluster config off a running `ActorSystem` (as `SplitBrainResolverActivationTests` does), never the typed options or the akka.conf text — several HOCON fragments compete and the losing one still reads correctly in the file.
|
||||||
|
|
||||||
**Still per-Akka-cluster, not per application `Cluster`.** One Primary is elected across the whole Akka mesh, so a fleet running several application clusters in one mesh (what docker-dev models) gets one Primary among all driver nodes. The fix is the per-cluster mesh workstream: `docs/plans/2026-07-21-per-cluster-mesh-design.md` (Phases 0a/0b done; 1–7 not started, Phase 1 planned in `…-phase1.md`).
|
**Still per-Akka-cluster, not per application `Cluster`.** One Primary is elected across the whole Akka mesh, so a fleet running several application clusters in one mesh (what docker-dev models) gets one Primary among all driver nodes. The fix is the per-cluster mesh workstream: `docs/plans/2026-07-21-per-cluster-mesh-design.md` (Phases 0a/0b done; 1–7 not started, Phase 1 planned in `…-phase1.md`).
|
||||||
|
|||||||
+40
-5
@@ -159,7 +159,7 @@ Per-node identity comes from `appsettings.json` + the `OTOPCUA_ROLES` env var:
|
|||||||
OTOPCUA_ROLES=admin,driver
|
OTOPCUA_ROLES=admin,driver
|
||||||
```
|
```
|
||||||
|
|
||||||
Both nodes share the same `ConfigDb` connection string; `Cluster.PublicHostname` + `Roles` are what makes them distinct in cluster gossip. The first node bootstraps the cluster (its address goes in `SeedNodes`); the second node joins via the same `SeedNodes` list.
|
Both nodes share the same `ConfigDb` connection string; `Cluster.PublicHostname` + `Roles` are what makes them distinct in cluster gossip. The first node bootstraps the cluster (its address goes in `SeedNodes`); the second node joins via the same `SeedNodes` list. List **both** peers in `SeedNodes` on **both** nodes so either can cold-start alone — see [Bootstrap: the self-form fallback](#bootstrap-the-self-form-fallback) for why the seed list alone is not enough, and what `Cluster:SelfFormAfter` adds.
|
||||||
|
|
||||||
There is no longer a `Node:NodeId` setting and no `ClusterNode.RedundancyRole` column (the V2 migration dropped it — primary/secondary is now derived from cluster membership age). NodeId is derived as `host:port` of the cluster `PublicHostname` (see `ClusterRoleInfo.LocalNode` for the formula).
|
There is no longer a `Node:NodeId` setting and no `ClusterNode.RedundancyRole` column (the V2 migration dropped it — primary/secondary is now derived from cluster membership age). NodeId is derived as `host:port` of the cluster `PublicHostname` (see `ClusterRoleInfo.LocalNode` for the formula).
|
||||||
|
|
||||||
@@ -241,10 +241,9 @@ two-node pair that choice means *any* crash of the oldest node is a full outage.
|
|||||||
`IHostApplicationLifetime.StopApplication()` so the process exits and the supervisor restarts it instead
|
`IHostApplicationLifetime.StopApplication()` so the process exits and the supervisor restarts it instead
|
||||||
of idling forever with a dead actor system. Under `auto-down` the *survivor* no longer needs this
|
of idling forever with a dead actor system. Under `auto-down` the *survivor* no longer needs this
|
||||||
(it is never the one downed); the **downed** node still does.
|
(it is never the one downed); the **downed** node still does.
|
||||||
3. **Both peers listed in `SeedNodes`** on every node, so a restarted node can re-join via either peer.
|
3. **Both peers listed in `SeedNodes`** on every node, so a restarted node can re-join via either peer —
|
||||||
Note the residual bootstrap constraint: only the **first** seed may form a cluster alone, so a node
|
plus the **self-form fallback** below, without which a node cold-starting while its peer is dead waits
|
||||||
cold-starting while its peer is dead still waits in `InitJoin`. Auto-down removes the crash outage, not
|
in `InitJoin` forever (auto-down removes the crash outage, not that one).
|
||||||
this one.
|
|
||||||
|
|
||||||
> **On `HardKillFailoverTests`:** that in-process test hard-kills the oldest and asserts the survivor becomes
|
> **On `HardKillFailoverTests`:** that in-process test hard-kills the oldest and asserts the survivor becomes
|
||||||
> sole `driver` role-leader. It passed even under `keep-oldest` — because it simulates the crash with
|
> sole `driver` role-leader. It passed even under `keep-oldest` — because it simulates the crash with
|
||||||
@@ -270,6 +269,42 @@ automatically.
|
|||||||
> alongside the per-cluster mesh work, which makes every mesh exactly two nodes. See
|
> alongside the per-cluster mesh work, which makes every mesh exactly two nodes. See
|
||||||
> `docs/plans/2026-07-21-per-cluster-mesh-design.md` §6.2 / Phase 0a.
|
> `docs/plans/2026-07-21-per-cluster-mesh-design.md` §6.2 / Phase 0a.
|
||||||
|
|
||||||
|
## Bootstrap: the self-form fallback
|
||||||
|
|
||||||
|
Listing both peers in `SeedNodes` does **not** mean either node can cold-start alone. Akka lets only the
|
||||||
|
**first** listed seed form a *new* cluster; every other node sends `InitJoin` to the seeds and retries —
|
||||||
|
forever — until one of them answers. A node that boots while its peer is dead therefore never comes Up,
|
||||||
|
and no downing strategy helps: **auto-down removes the crash outage, not this one.**
|
||||||
|
|
||||||
|
`Cluster:SelfFormAfter` closes that gap. `ClusterBootstrapFallback.Arm` (armed for every host built through
|
||||||
|
`WithOtOpcUaClusterBootstrap`, via an Akka.Hosting `AddStartup` task) waits that long for cluster membership
|
||||||
|
and, on expiry, calls `Cluster.Join(SelfAddress)` — the node forms a cluster on itself and becomes
|
||||||
|
operational unattended.
|
||||||
|
|
||||||
|
| `Cluster:SelfFormAfter` | Behaviour |
|
||||||
|
|---|---|
|
||||||
|
| `"00:00:10"` (**default**) | A node with no membership after 10 s self-forms. A live peer answers `InitJoin` in milliseconds, so on any normal boot the fallback never fires. |
|
||||||
|
| `null` / `≤ 0` | Disabled — the pre-2026-07-22 behaviour: wait on `InitJoin` indefinitely. |
|
||||||
|
|
||||||
|
**The island guard is the load-bearing part.** The fallback fires **only when this node's own address
|
||||||
|
appears in its own `SeedNodes`**. A node that is not one of its own seeds is never legitimately first, and
|
||||||
|
if it self-formed, a later-booting real seed would form a second cluster the two could never merge. That is
|
||||||
|
exactly the current docker-dev topology: the site-a/site-b driver nodes list only `central-1` as a seed, so
|
||||||
|
they deliberately keep waiting rather than islanding themselves. Under the future per-cluster mesh
|
||||||
|
(`docs/plans/2026-07-21-per-cluster-mesh-design.md`) each pair node *is* a seed of its own two-node mesh, so
|
||||||
|
the fallback covers both sides of every pair and the site-node exception disappears.
|
||||||
|
|
||||||
|
Sequential recovery is island-free by construction: Akka's join protocol prefers an existing cluster (a
|
||||||
|
booting node only self-forms when **no** seed answered), so a peer booting after the survivor self-formed
|
||||||
|
simply joins it as the youngest member. The residual risk is both pair nodes cold-starting inside the
|
||||||
|
window while mutually unreachable — both self-form, which is the same dual-active class the `auto-down`
|
||||||
|
strategy already accepts, with the same recovery (restart one side).
|
||||||
|
|
||||||
|
Pinned by `SelfFormBootstrapTests` (`tests/Core/ZB.MOM.WW.OtOpcUa.Cluster.Tests/`), which starts real hosts
|
||||||
|
through the production bootstrap: a lone non-first seed comes Up; a disabled window keeps waiting; a node
|
||||||
|
absent from its own seed list never self-forms. The two negative cases each carry a positive control (an
|
||||||
|
explicit self-join) so they cannot pass merely because the node was unformable.
|
||||||
|
|
||||||
## Primary data-plane gate (writes, acks, alerts emit)
|
## Primary data-plane gate (writes, acks, alerts emit)
|
||||||
|
|
||||||
Three data-plane surfaces are **Primary-gated** so a warm standby never touches a shared field device or duplicates a fleet-wide publish: inbound operator **writes** (`DriverHostActor.HandleRouteNodeWrite`), native-alarm **acks** (`HandleRouteNativeAlarmAck`), and the native/scripted **alerts emit** (`ForwardNativeAlarm` / `ScriptedAlarmHostActor.OnEngineEmission`). All three route through one policy — `PrimaryGatePolicy.ShouldServiceAsPrimary(localRole, driverMemberCount)` (`OtOpcUa.Runtime.Drivers`):
|
Three data-plane surfaces are **Primary-gated** so a warm standby never touches a shared field device or duplicates a fleet-wide publish: inbound operator **writes** (`DriverHostActor.HandleRouteNodeWrite`), native-alarm **acks** (`HandleRouteNativeAlarmAck`), and the native/scripted **alerts emit** (`ForwardNativeAlarm` / `ScriptedAlarmHostActor.OnEngineEmission`). All three route through one policy — `PrimaryGatePolicy.ShouldServiceAsPrimary(localRole, driverMemberCount)` (`OtOpcUa.Runtime.Drivers`):
|
||||||
|
|||||||
@@ -85,6 +85,12 @@ Their failover drill has a mode that exists *"to make the registered gap observa
|
|||||||
it is covered."* **A two-node keep-oldest mesh has an acknowledged total-outage hole.** If OtOpcUa
|
it is covered."* **A two-node keep-oldest mesh has an acknowledged total-outage hole.** If OtOpcUa
|
||||||
adopts 2-node meshes, this must be decided deliberately, not inherited.
|
adopts 2-node meshes, this must be decided deliberately, not inherited.
|
||||||
|
|
||||||
|
> **CLOSED 2026-07-22.** The `InitJoin` half of this gap is fixed in both repos by the self-form
|
||||||
|
> fallback: `Cluster:SelfFormAfter` (default 10 s) lets a node that is one of its own seeds form a
|
||||||
|
> cluster on itself when no seed answers. See `docs/Redundancy.md` §"Bootstrap: the self-form
|
||||||
|
> fallback" — including the island guard that keeps today's non-seed site nodes waiting, which the
|
||||||
|
> per-cluster mesh removes by making every pair node a seed of its own mesh.
|
||||||
|
|
||||||
### Both inter-cluster transports are unauthenticated
|
### Both inter-cluster transports are unauthenticated
|
||||||
|
|
||||||
Akka remoting has no TLS, no secure cookie, no `trusted-selection-paths`. The gRPC listener is h2c
|
Akka remoting has no TLS, no secure cookie, no `trusted-selection-paths`. The gRPC listener is h2c
|
||||||
@@ -253,6 +259,14 @@ rather than the configured one. See `docs/Redundancy.md`.
|
|||||||
1-vs-1 split, and `docker-dev` is a single six-node mesh. The crash-the-oldest drill therefore lands
|
1-vs-1 split, and `docker-dev` is a single six-node mesh. The crash-the-oldest drill therefore lands
|
||||||
with Phase 6/7, which makes every mesh exactly two nodes.
|
with Phase 6/7, which makes every mesh exactly two nodes.
|
||||||
|
|
||||||
|
**The bootstrap half of the gap is also closed (2026-07-22).** Downing was only ever one of the two
|
||||||
|
outage modes: even with `auto-down`, a node cold-starting while its peer was dead looped on `InitJoin`
|
||||||
|
forever, because Akka lets only the first listed seed form a new cluster. `Cluster:SelfFormAfter`
|
||||||
|
(default 10 s, `ClusterBootstrapFallback`) now self-forms in that case, guarded so a node absent from
|
||||||
|
its own seed list never islands itself. Pinned by `SelfFormBootstrapTests`; see `docs/Redundancy.md`
|
||||||
|
§"Bootstrap: the self-form fallback". Under this design each pair node is a seed of its own two-node
|
||||||
|
mesh, so the fallback covers both sides and the current site-node exception disappears.
|
||||||
|
|
||||||
### 6.3 DECIDED — match ScadaBridge's auth posture for now
|
### 6.3 DECIDED — match ScadaBridge's auth posture for now
|
||||||
|
|
||||||
Inter-cluster transports stay unauthenticated and unencrypted, as theirs are: the boundary assumes a
|
Inter-cluster transports stay unauthenticated and unencrypted, as theirs are: the boundary assumes a
|
||||||
|
|||||||
@@ -0,0 +1,194 @@
|
|||||||
|
# Per-Cluster Mesh Program — Align OtOpcUa's Akka Topology with ScadaBridge
|
||||||
|
|
||||||
|
> **For Claude:** This is a PROGRAM plan (phase roadmap + gates), not a bite-sized task plan.
|
||||||
|
> The authoritative design is `docs/plans/2026-07-21-per-cluster-mesh-design.md` (decisions settled
|
||||||
|
> 2026-07-21; Phases 0a/0b DONE). Per that design's own instruction, **each phase gets its own
|
||||||
|
> detailed plan, written when the phase starts** (superpowers writing-plans → executing-plans),
|
||||||
|
> so plans are authored against current code, not against a forecast. This document sequences the
|
||||||
|
> phases, fixes the deployment topology (co-location with ScadaBridge — NEW constraint 2026-07-22),
|
||||||
|
> and defines each phase's entry/exit gates.
|
||||||
|
|
||||||
|
**Goal:** OtOpcUa runs the same Akka.NET mesh shape as ScadaBridge — **one Akka mesh per
|
||||||
|
application `Cluster`, two nodes max**, central ↔ cluster joined by explicit transports
|
||||||
|
(ClusterClient + gRPC + fetch), driver nodes holding **no ConfigDb connection** — so that every
|
||||||
|
Primary-gated decision and every gated resource share one pair-local scope, and both products
|
||||||
|
present one operational model on the shared site hardware.
|
||||||
|
|
||||||
|
**Why now (2026-07-22):** OtOpcUa cluster pairs will run **on the same two Windows VMs as the
|
||||||
|
ScadaBridge site nodes**. That deployment makes the current single-fleet-mesh design actively
|
||||||
|
wrong for sites (a site's OtOpcUa nodes would gossip across the WAN to central and elect ONE
|
||||||
|
Primary fleet-wide), and makes the ScadaBridge shape the obviously correct one: each site's two
|
||||||
|
VMs host two independent, identically-postured 2-node clusters (one per product), surviving alone
|
||||||
|
on the site LAN. It also means driver nodes must not depend on reaching central SQL — **sites
|
||||||
|
have no SQL Server**, so §6.1's "driver nodes never connect to the ConfigDb" stops being an
|
||||||
|
architectural preference and becomes a deployment requirement.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Deployment topology (the co-location constraint, NEW)
|
||||||
|
|
||||||
|
Each **site**: 2 Windows VMs. Each VM runs one ScadaBridge site node AND one OtOpcUa driver node.
|
||||||
|
Two independent 2-node Akka clusters per site — they share hardware, never a mesh. **Central**:
|
||||||
|
2 Windows VMs, each running a ScadaBridge central node and an OtOpcUa central node
|
||||||
|
(`admin,driver`), again as separate pairs.
|
||||||
|
|
||||||
|
**Per-VM port allocation (no collisions — verify against actual deployment configs in Phase 6):**
|
||||||
|
|
||||||
|
| Port | Owner | Purpose |
|
||||||
|
|---|---|---|
|
||||||
|
| 8081 / 8082 | ScadaBridge | Akka remoting (central / site) |
|
||||||
|
| 8083 | ScadaBridge site | gRPC h2c (streams + LocalDb sync) |
|
||||||
|
| 8084 | ScadaBridge site | metrics |
|
||||||
|
| 5000 (+Traefik) | ScadaBridge central | UI + Inbound API |
|
||||||
|
| **4053** | OtOpcUa | Akka remoting |
|
||||||
|
| **4840** | OtOpcUa | OPC UA endpoint |
|
||||||
|
| OtOpcUa AdminUI port | OtOpcUa central (admin) | AdminUI (driver-only site nodes host no UI) |
|
||||||
|
| OtOpcUa LocalDb sync port | OtOpcUa driver | h2c LocalDb pair replication (default off/0 today — Phase 6 assigns a real per-site port) |
|
||||||
|
|
||||||
|
**Aligned HA posture (both products, per VM — already true or landing via the selfform plan):**
|
||||||
|
auto-down downing (15 s window), oldest-Up active/primary election, `SelfFormAfter` 10 s
|
||||||
|
self-form fallback, termination-watchdog → process exit → `sc.exe failure` restart recovery.
|
||||||
|
One failover story for operators regardless of product.
|
||||||
|
|
||||||
|
**Prerequisite ordering:** the fallback/manual-failover plan
|
||||||
|
(`docs/plans/2026-07-22-selfform-fallback-and-manual-failover.md`) executes **before** this
|
||||||
|
program — it is small, independent, and Phase 6 depends on its semantics (under per-pair meshes
|
||||||
|
every node lists itself + partner as seeds, so the fallback covers both nodes of every pair;
|
||||||
|
the site-node island guard then simply never triggers).
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## What changes, in one table (from the design doc)
|
||||||
|
|
||||||
|
| Aspect | Today (single fleet mesh) | Target (ScadaBridge shape) |
|
||||||
|
|---|---|---|
|
||||||
|
| Mesh | one gossip ring, all nodes, seeded by central-1 | one 2-node mesh per application `Cluster`; same ActorSystem name; separation by seed-node partitioning |
|
||||||
|
| Roles | `admin` / `driver` fleet-wide | `driver` + cluster-specific `cluster-{ClusterId}`; singletons scoped to the cluster role |
|
||||||
|
| Primary election | mesh-wide oldest Up driver (Phase 0b) | pair-local by construction — same rule, correct scope |
|
||||||
|
| Command/control | 9 DPS topics + singleton over gossip | ClusterClient (one receptionist actor per side); central discovers nodes from `ClusterNode` rows |
|
||||||
|
| Deploy | notify (DPS) + node fetches from ConfigDb | notify via ClusterClient; artifact fetched **from central**, cached in LocalDb |
|
||||||
|
| Driver ConfigDb connection | direct EF connection to central SQL | **none** — LocalDb is the steady-state config store |
|
||||||
|
| Live telemetry | 7 observability DPS topics | one gRPC stream contract (`oneof` event), central dials each cluster node |
|
||||||
|
| Rig | six-node single mesh (`docker-dev`) | per-cluster meshes; rig models the real topology |
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Phases
|
||||||
|
|
||||||
|
Each phase below is one row of design-doc §7, expanded with entry/exit gates. **Execution recipe
|
||||||
|
per phase:** (1) invoke writing-plans in this repo to produce
|
||||||
|
`docs/plans/2026-07-2X-mesh-phase-N-<name>.md` from the scope notes here + the design doc §
|
||||||
|
references, exploring current code first; (2) execute it task-by-task; (3) run the phase's exit
|
||||||
|
gate; (4) update this file's status column and the design doc's §7 table.
|
||||||
|
|
||||||
|
### Phase 1 — `ClusterNode` address columns + DB-sourced ack set
|
||||||
|
**Scope:** `ClusterNode` gains Akka + gRPC address columns (mirroring ScadaBridge's `Site`
|
||||||
|
entity `NodeAAddress`/`GrpcNodeAAddress` pattern, but per-node rows); EF migration; AdminUI node
|
||||||
|
edit surfaces the fields; `ConfigPublishCoordinator` derives its expected-ack set from
|
||||||
|
`ClusterNode` rows instead of `Akka.Cluster.State.Members` filtered by role (design §3 fact 3 —
|
||||||
|
this removes the coordinator's one genuinely mesh-bound dependency).
|
||||||
|
**Independent of the split:** yes — safe on the current mesh.
|
||||||
|
**Exit gate:** deploy on the unchanged docker-dev rig completes with the coordinator's expected-ack
|
||||||
|
set proven DB-sourced (test: a `ClusterNode` row present but node down → deploy reports that node
|
||||||
|
missing; a node up but row absent → its ack is not expected).
|
||||||
|
|
||||||
|
### Phase 2 — Comm actors + ClusterClient transport
|
||||||
|
**Scope:** one receptionist-registered actor per side (`/user/central-communication`,
|
||||||
|
`/user/cluster-communication`), registered **per node, not as a singleton** (contact rotation);
|
||||||
|
ClusterClient central → cluster carrying deploy notify + acks + driver-control; the ScadaBridge
|
||||||
|
idioms copied verbatim: sender-preserving `Tell(new ClusterClient.Send(...), Sender)` Ask relay,
|
||||||
|
typed-failure reply for every unhandled message, no central buffering toward unreachable clusters
|
||||||
|
(drop + warn), central discovers contacts from Phase 1's `ClusterNode` rows (60 s refresh +
|
||||||
|
admin-change refresh), clusters know central from appsettings (static, restart to change).
|
||||||
|
**Frame-size guard:** anything carrying payload sets both the frame limit and
|
||||||
|
`log-frame-size-exceeding` (design §8) — the deploy path stays payload-free by design.
|
||||||
|
**Exit gate:** on the still-single-mesh rig, deploy notify + acks and AdminUI Reconnect/Restart
|
||||||
|
flow over ClusterClient (DPS paths deleted or dark-switched), including an Ask timing out cleanly
|
||||||
|
against a stopped node.
|
||||||
|
|
||||||
|
### Phase 3 — Config fetch-and-cache from central
|
||||||
|
**Scope:** the deploy artifact is served **by central** (transport per its own phase plan — the
|
||||||
|
ScadaBridge analogue is token-gated HTTP; decide HTTP vs a gRPC fetch RPC when planning) and
|
||||||
|
cached in LocalDb; driver nodes read config exclusively from LocalDb (boot-from-cache becomes the
|
||||||
|
normal path — design §6.1); chunking, SHA-256 verify, newest-2 retention, pair replication carry
|
||||||
|
over unchanged. **This phase changes a running data path — it gets its own live gate** (design §7
|
||||||
|
note): deploy lands on a driver pair with central SQL stopped mid-fetch → retry lands; #485
|
||||||
|
last-known-good semantics re-proven on the new path.
|
||||||
|
**Exit gate:** live gate green on the rig; a driver node with an empty LocalDb and reachable
|
||||||
|
central boots into the current config; with central down it boots last-known-good.
|
||||||
|
|
||||||
|
### Phase 4 — Cut the driver-side ConfigDb connection
|
||||||
|
**Scope (design §6.1 audit table):** re-home `EfAlarmConditionStateStore` to LocalDb (pair-local
|
||||||
|
state, same journey as the Phase-2 alarm S&F buffer); **resolve the `DbHealthProbeActor`
|
||||||
|
question** — driver nodes have no DB to probe, and DB health currently feeds ServiceLevel tiering,
|
||||||
|
so define the replacement health input (candidate: central-reachability via the Phase 2/3
|
||||||
|
transports) — this is a client-visible ServiceLevel semantics change and must be documented in
|
||||||
|
`docs/Redundancy.md` + the interop playbook; audit `OpcUaPublishActor`'s ConfigDb use (TBD in the
|
||||||
|
design) and re-source it; registration cleanup in `ServiceCollectionExtensions`; driver-role
|
||||||
|
`Program.cs` branch registers no EF ConfigDb context at all (mirror ScadaBridge's central-only
|
||||||
|
`AddConfigurationDatabase`).
|
||||||
|
**Exit gate:** its own live gate — a driver pair runs a full deploy + alarm + historian cycle with
|
||||||
|
**no ConfigDb connection string configured at all**; grep-level proof no driver-branch service can
|
||||||
|
resolve the ConfigDb context.
|
||||||
|
|
||||||
|
### Phase 5 — gRPC stream contract for live telemetry
|
||||||
|
**Scope:** one server-streaming contract carrying a `oneof` event, **cluster nodes host the gRPC
|
||||||
|
server, central dials in** (the inverted direction is the load-bearing ScadaBridge finding —
|
||||||
|
design §2); migrate the seven observability topics (`alerts`, `driver-health`,
|
||||||
|
`driver-resilience-status`, `fleet-status`, `script-logs`, plus redundancy-state distribution and
|
||||||
|
deployment-acks if Phase 2 left them on DPS); additive-only field evolution, contract locked by
|
||||||
|
test; per-panel reconnect story for the AdminUI (design §8 — losing gossip loses free fleet
|
||||||
|
observability).
|
||||||
|
**Exit gate:** all AdminUI live panels green against a pair with DPS telemetry topics deleted;
|
||||||
|
kill-and-reconnect of the central dialer recovers every stream.
|
||||||
|
|
||||||
|
### Phase 6 — Mesh partition + co-location topology
|
||||||
|
**Scope:** per-cluster seed nodes (each pair node lists itself + partner — the `SelfFormAfter`
|
||||||
|
fallback then covers both), cluster-scoped roles `cluster-{ClusterId}` + singleton re-scoping,
|
||||||
|
central pair keeps the admin singletons; **docker-dev rig rewritten** to model the real topology —
|
||||||
|
including the co-location port table above (both products' compose files on shared per-site
|
||||||
|
networks, real LocalDb sync ports); remove the ClusterRedundancy page's mesh-scope caveat (the
|
||||||
|
election is pair-local now) and the fallback's site-node island-guard docs note (moot — every
|
||||||
|
node is a seed of its own mesh); `Cluster__SeedNodes__*` env matrix per pair.
|
||||||
|
**Exit gate:** rig up in the new shape; every existing live-gated behavior re-verified per pair
|
||||||
|
(deploy, redundancy 250/240 per pair — **two Primaries fleet-wide, one per pair, by design**);
|
||||||
|
secrets Akka replication re-verified or re-scoped (it rides DPS on the current single mesh — its
|
||||||
|
topology must be re-decided here, likely SQL-hub mode like ScadaBridge, since pub/sub cannot
|
||||||
|
cross separate meshes).
|
||||||
|
|
||||||
|
### Phase 7 — Failover drill + live gates
|
||||||
|
**Scope:** the drill ScadaBridge already has (`failover-drill.sh` analogue) run per pair, both
|
||||||
|
directions; **close the two outstanding live gates**: (a) auto-down 1-vs-1 crash-the-oldest
|
||||||
|
(deferred since Phase 0a — finally testable, every mesh is exactly two nodes), (b) `SelfFormAfter`
|
||||||
|
lone-cold-start on the real per-pair topology; manual-failover button re-verified per pair;
|
||||||
|
operator runbook for the co-located site (one page covering both products' failover on the same
|
||||||
|
two VMs).
|
||||||
|
**Exit gate:** drill green on every pair type (central, site); runbook merged; design doc §7
|
||||||
|
table fully marked DONE.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Risks carried from the design (§8, unchanged — re-read before each phase plan)
|
||||||
|
|
||||||
|
LocalDb becomes load-bearing for config (blast radius of the #485 class rises);
|
||||||
|
`DbHealthProbeActor` feeds a client-visible value; the rig models the doomed topology until
|
||||||
|
Phase 6; losing gossip loses free observability (7 panels); 128 KB ClusterClient frame drop is
|
||||||
|
silent unless both knobs are set; never stack app-level LWW on LocalDb's HLC. **New (this
|
||||||
|
program):** co-located VMs mean a VM loss now takes out one node of BOTH products at once — the
|
||||||
|
drill in Phase 7 must include the shared-VM failure (both products fail over together), and
|
||||||
|
resource sizing on the site VMs should be checked once both products run the full stack.
|
||||||
|
|
||||||
|
## Tracking
|
||||||
|
|
||||||
|
| Phase | Status |
|
||||||
|
|---|---|
|
||||||
|
| 0a downing strategy | DONE 2026-07-21 (live gate → Phase 7) |
|
||||||
|
| 0b oldest-Up election | DONE 2026-07-21 |
|
||||||
|
| Prereq: selfform-fallback + manual-failover plan | plan written 2026-07-22, not executed |
|
||||||
|
| 1 ClusterNode columns + DB ack set | not started — plan to be written |
|
||||||
|
| 2 ClusterClient transport | not started |
|
||||||
|
| 3 fetch-and-cache | not started |
|
||||||
|
| 4 cut driver ConfigDb | not started |
|
||||||
|
| 5 gRPC telemetry | not started |
|
||||||
|
| 6 mesh partition + co-location | not started |
|
||||||
|
| 7 drill + live gates | not started |
|
||||||
@@ -0,0 +1,15 @@
|
|||||||
|
{
|
||||||
|
"planPath": "docs/plans/2026-07-22-per-cluster-mesh-program.md",
|
||||||
|
"note": "PROGRAM plan: each task = write that phase's detailed plan (writing-plans), execute it (executing-plans), run its exit gate, update the tracking tables. Prereq task 0 is the separate selfform-fallback plan in this repo.",
|
||||||
|
"tasks": [
|
||||||
|
{"id": 0, "subject": "Prereq: execute 2026-07-22-selfform-fallback-and-manual-failover.md (7 tasks)", "status": "pending"},
|
||||||
|
{"id": 1, "subject": "Phase 1: ClusterNode Akka+gRPC address columns; ConfigPublishCoordinator ack set from DB", "status": "pending"},
|
||||||
|
{"id": 2, "subject": "Phase 2: comm actors + receptionist + ClusterClient transport (deploy notify/acks, driver-control)", "status": "pending", "blockedBy": [1]},
|
||||||
|
{"id": 3, "subject": "Phase 3: config fetch-and-cache from central; LocalDb steady-state config store (live gate)", "status": "pending", "blockedBy": [2]},
|
||||||
|
{"id": 4, "subject": "Phase 4: cut driver-side ConfigDb — EfAlarmConditionStateStore to LocalDb, DbHealthProbeActor ServiceLevel input, OpcUaPublishActor audit (live gate)", "status": "pending", "blockedBy": [3]},
|
||||||
|
{"id": 5, "subject": "Phase 5: gRPC oneof stream contract; migrate 7 observability topics; AdminUI reconnect story", "status": "pending", "blockedBy": [2]},
|
||||||
|
{"id": 6, "subject": "Phase 6: mesh partition — per-pair seeds, cluster-scoped roles/singletons, co-location rig rewrite, secrets replication re-scope", "status": "pending", "blockedBy": [0, 4, 5]},
|
||||||
|
{"id": 7, "subject": "Phase 7: failover drill per pair (incl. shared-VM failure) + close auto-down 1v1 and SelfFormAfter live gates + operator runbook", "status": "pending", "blockedBy": [6]}
|
||||||
|
],
|
||||||
|
"lastUpdated": "2026-07-22T00:00:00Z"
|
||||||
|
}
|
||||||
@@ -0,0 +1,388 @@
|
|||||||
|
# OtOpcUa: InitJoin Self-Form Fallback + Manual Failover Control — Implementation Plan
|
||||||
|
|
||||||
|
> **For Claude:** REQUIRED SUB-SKILL: Use superpowers-extended-cc:executing-plans to implement this plan task-by-task.
|
||||||
|
>
|
||||||
|
> Shared cross-repo design: `~/Desktop/scadaproj/docs/plans/2026-07-22-initjoin-selfform-fallback.md` (design rationale, MNTR assessment, behavior spec). The ScadaBridge half lives in `~/Desktop/ScadaBridge/docs/plans/2026-07-22-selfform-fallback-and-manual-failover.md`. This plan is self-contained for execution.
|
||||||
|
>
|
||||||
|
> **⚠ This plan is expected to GROW** — see "Reserved for additions" at the end. Append new tasks there (and to the `.tasks.json`) rather than renumbering existing ones.
|
||||||
|
|
||||||
|
**Goal:** (1) Either node of a 2-node OtOpcUa pair can cold-start alone and become operational, unattended — a configurable, fast (default 10 s) InitJoin timeout falls back to self-forming a cluster, with a hard guard so site nodes seeded only by central can never island themselves. (2) An admin-only "Trigger failover" control on the Cluster Redundancy page performs a graceful, audited swap of the driver Primary.
|
||||||
|
|
||||||
|
**Architecture:** New `AkkaClusterOptions.SelfFormAfter` (`TimeSpan?`, default 10 s, `null`/`≤0` disables, bound from the `Cluster` section) arms `ClusterBootstrapFallback` via an Akka.Hosting startup task inside `WithOtOpcUaClusterBootstrap` — so production AND the Cluster.Tests hosts get it identically. Wait for membership via `RegisterOnMemberUp`; on expiry, `Cluster.Join(SelfAddress)` — only if this node's own address is in its own `SeedNodes`. Manual failover = graceful `Cluster.Leave(<oldest Up driver member>)` via a new `IManualFailoverService` in ControlPlane, surfaced on `/clusters/{id}/redundancy`.
|
||||||
|
|
||||||
|
**Tech Stack:** .NET 10, Akka.NET 1.5.62, Akka.Hosting 1.5.62 (`AddStartup`), Blazor Server (AdminUI, InteractiveServer), bUnit, xunit. No new packages.
|
||||||
|
|
||||||
|
**Branch:** `feat/selfform-fallback` off `master`.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Design essentials (from the shared design doc)
|
||||||
|
|
||||||
|
**The defect:** Akka only lets the FIRST listed seed self-join to form a *new* cluster; every other node loops on `InitJoin` forever — "auto-down removes the crash outage, not this one" (`docs/Redundancy.md:244-247`). A lone cold-starting `central-2` (or a pair node under the future per-cluster mesh) never comes Up.
|
||||||
|
|
||||||
|
**Behavior spec:**
|
||||||
|
|
||||||
|
| Scenario | Behavior with fallback |
|
||||||
|
|---|---|
|
||||||
|
| Peer alive (any boot order) | Normal seed join in ms — fallback never fires |
|
||||||
|
| Lone cold-start, self IS in own `SeedNodes` | After `SelfFormAfter`: warn log + `Cluster.Join(SelfAddress)` → Up alone (`min-nr-of-members=1`) |
|
||||||
|
| Lone cold-start, self NOT in own `SeedNodes` | Fallback inert (info log). **This is the docker-dev site-node topology** — site-a/b nodes list only `central-1`; self-forming there would island them permanently, so they must wait. |
|
||||||
|
| Peer boots after survivor self-formed | Its InitJoin is answered → joins as youngest. No island. |
|
||||||
|
| Both pair nodes cold-start simultaneously, mutually unreachable | Both self-form → dual-active (same partition class auto-down accepts; restart one side) |
|
||||||
|
| `SelfFormAfter` null/`≤0` | Disabled — today's wait-forever behavior |
|
||||||
|
| Window expires mid-join-handshake | Benign: Akka ignores `Join` once joined |
|
||||||
|
|
||||||
|
**Manual failover rules:** graceful `Leave` of the **oldest Up `driver` member** (identical query to `RedundancyStateActor.SelectDriverPrimary` — `Member.AgeOrdering`, `Leaving` excluded via `Status == Up`), never `Down`. The leaving node hands over via the cluster-leave phases, `CoordinatedShutdown` runs, `ActorSystemTerminationWatchdog` exits the process, the supervisor restarts it, and it rejoins as youngest → the survivor is Primary (ServiceLevel 250 moves; OPC UA clients re-select). Admin-only; peer guard (<2 Up driver members → disabled); confirmation dialog; audited via the shared `ZB.MOM.WW.Audit.IAuditWriter` seam before the Leave. **Mesh-scope caveat:** until the per-cluster mesh lands, the election is mesh-wide — the button acts on THE Primary of the whole Akka cluster, not per-`ServerCluster` row; the UI must say so (mirrors the KNOWN LIMITATION in `docs/Redundancy.md:84-99`).
|
||||||
|
|
||||||
|
**Multi-node TestKit:** assessed, NOT used — everything is deterministic in-process via real hosts through `WithOtOpcUaClusterBootstrap` (the `SplitBrainResolverActivationTests` pattern), and the Akka TestKit family is xunit-v2-only while this repo's integration tree is xunit.v3. Full verdict in the shared design doc.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### Task 1 (B1): `SelfFormAfter` on `AkkaClusterOptions`
|
||||||
|
|
||||||
|
**Classification:** small
|
||||||
|
**Estimated implement time:** ~3 min
|
||||||
|
**Parallelizable with:** none (first task)
|
||||||
|
|
||||||
|
**Files:**
|
||||||
|
- Modify: `src/Core/ZB.MOM.WW.OtOpcUa.Cluster/AkkaClusterOptions.cs`
|
||||||
|
|
||||||
|
**Step 1: Branch**
|
||||||
|
|
||||||
|
```bash
|
||||||
|
cd ~/Desktop/OtOpcUa && git checkout master && git checkout -b feat/selfform-fallback
|
||||||
|
```
|
||||||
|
|
||||||
|
**Step 2: Add the property** (after `SplitBrainResolverStrategy`, mirroring its XML-doc depth):
|
||||||
|
|
||||||
|
```csharp
|
||||||
|
/// <summary>
|
||||||
|
/// Bootstrap self-form fallback window (decision 2026-07-22, scadaproj/akka_failover.md §6.1).
|
||||||
|
/// Akka only lets the FIRST listed seed form a new cluster; a non-first seed cold-starting
|
||||||
|
/// while its peer is down loops on InitJoin forever ("auto-down removes the crash outage,
|
||||||
|
/// not this one" — docs/Redundancy.md). When this node has waited longer than this window
|
||||||
|
/// without cluster membership it joins itself — but ONLY if its own address appears in
|
||||||
|
/// <see cref="SeedNodes"/>; a non-seed node (e.g. a site node whose only seed is central-1)
|
||||||
|
/// stays waiting, because self-forming there creates a permanent island. Default 10s
|
||||||
|
/// (same-datacenter pair; a live peer answers InitJoin in milliseconds). <c>null</c> or a
|
||||||
|
/// non-positive value disables the fallback. Accepted trade: both pair nodes cold-starting
|
||||||
|
/// inside the window while mutually unreachable form two clusters — the same dual-active
|
||||||
|
/// class the auto-down strategy already accepts, same recovery (restart one side).
|
||||||
|
/// </summary>
|
||||||
|
public TimeSpan? SelfFormAfter { get; set; } = TimeSpan.FromSeconds(10);
|
||||||
|
```
|
||||||
|
|
||||||
|
**Step 3:** `dotnet build src/Core/ZB.MOM.WW.OtOpcUa.Cluster` — 0 warnings. Commit:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
git add -A && git commit -m "feat(cluster): SelfFormAfter option (bootstrap self-form fallback window)"
|
||||||
|
```
|
||||||
|
|
||||||
|
(The default-value pin test lands with Task 3's test file — the property is inert until Task 2.)
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### Task 2 (B2): `ClusterBootstrapFallback` + `AddStartup` wiring
|
||||||
|
|
||||||
|
**Classification:** high-risk (cluster formation behavior)
|
||||||
|
**Estimated implement time:** ~5 min
|
||||||
|
**Parallelizable with:** none
|
||||||
|
|
||||||
|
**Files:**
|
||||||
|
- Create: `src/Core/ZB.MOM.WW.OtOpcUa.Cluster/ClusterBootstrapFallback.cs`
|
||||||
|
- Modify: `src/Core/ZB.MOM.WW.OtOpcUa.Cluster/ServiceCollectionExtensions.cs` (`WithOtOpcUaClusterBootstrap`, after the `AddHocon(BuildDowningHocon(options), HoconAddMode.Prepend)` call at line 93)
|
||||||
|
|
||||||
|
**Step 1: Implement the fallback class** (deliberately duplicated from ScadaBridge's, like the termination watchdogs — see that repo's `ClusterBootstrapFallback.cs` for the full doc comment to adapt):
|
||||||
|
|
||||||
|
```csharp
|
||||||
|
using Akka.Actor;
|
||||||
|
using Microsoft.Extensions.Logging;
|
||||||
|
|
||||||
|
namespace ZB.MOM.WW.OtOpcUa.Cluster;
|
||||||
|
|
||||||
|
/// <summary>
|
||||||
|
/// InitJoin self-form fallback (decision 2026-07-22, scadaproj/akka_failover.md §6.1). Akka only
|
||||||
|
/// lets the FIRST listed seed form a NEW cluster; every other node retries InitJoin forever —
|
||||||
|
/// "auto-down removes the crash outage, not this one" (docs/Redundancy.md). Waits
|
||||||
|
/// <see cref="AkkaClusterOptions.SelfFormAfter"/> for membership, then joins itself.
|
||||||
|
/// <para><b>Island safety:</b> fires ONLY when this node's own address is in its own
|
||||||
|
/// <see cref="AkkaClusterOptions.SeedNodes"/> — a site node seeded only by central-1 must wait,
|
||||||
|
/// not island. Sequential recovery is island-free: Akka's join protocol prefers an existing
|
||||||
|
/// cluster (a booting node only self-forms when NO seed answers InitJoin).</para>
|
||||||
|
/// <para><b>Benign race:</b> Akka ignores <c>Join</c> once the node has joined a cluster this
|
||||||
|
/// incarnation. Residual risk: both pair nodes cold-starting inside the window while mutually
|
||||||
|
/// unreachable self-form separately — the same dual-active class auto-down accepts.</para>
|
||||||
|
/// </summary>
|
||||||
|
public static class ClusterBootstrapFallback
|
||||||
|
{
|
||||||
|
public static void Arm(ActorSystem system, AkkaClusterOptions options, ILogger logger)
|
||||||
|
{
|
||||||
|
if (options.SelfFormAfter is not { } window || window <= TimeSpan.Zero)
|
||||||
|
{
|
||||||
|
logger.LogInformation(
|
||||||
|
"Cluster self-form fallback disabled (SelfFormAfter not set) — a node cold-starting "
|
||||||
|
+ "while its peer is down will wait on InitJoin indefinitely.");
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
var cluster = Akka.Cluster.Cluster.Get(system);
|
||||||
|
var self = cluster.SelfAddress;
|
||||||
|
var isSeed = options.SeedNodes.Any(s => TryParseAddress(s, out var a) && a.Equals(self));
|
||||||
|
if (!isSeed)
|
||||||
|
{
|
||||||
|
logger.LogInformation(
|
||||||
|
"Cluster self-form fallback inactive: this node ({Self}) is not in its own seed list "
|
||||||
|
+ "[{Seeds}] — self-forming here would island it from the real seeds.",
|
||||||
|
self, string.Join(", ", options.SeedNodes));
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
var joined = new TaskCompletionSource(TaskCreationOptions.RunContinuationsAsynchronously);
|
||||||
|
cluster.RegisterOnMemberUp(() => joined.TrySetResult());
|
||||||
|
|
||||||
|
_ = Task.Run(async () =>
|
||||||
|
{
|
||||||
|
var winner = await Task.WhenAny(joined.Task, Task.Delay(window));
|
||||||
|
if (winner == joined.Task || system.WhenTerminated.IsCompleted)
|
||||||
|
return;
|
||||||
|
logger.LogWarning(
|
||||||
|
"No cluster membership after {Window} — no seed answered InitJoin (peer down at boot). "
|
||||||
|
+ "Self-forming a cluster at {Self} so this node becomes operational; if the peer was "
|
||||||
|
+ "merely partitioned (not dead), the pair is now dual-active — restart one side after "
|
||||||
|
+ "the partition heals (accepted availability-first trade, decision 2026-07-22).",
|
||||||
|
window, self);
|
||||||
|
cluster.Join(self);
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
||||||
|
private static bool TryParseAddress(string seed, out Address address)
|
||||||
|
{
|
||||||
|
try { address = Address.Parse(seed); return true; }
|
||||||
|
catch { address = default!; return false; }
|
||||||
|
}
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
(If the Cluster project lacks a `Microsoft.Extensions.Logging.Abstractions` reference, add the `PackageReference` — check `Directory.Packages.props` for the pinned version first.)
|
||||||
|
|
||||||
|
**Step 2: Wire via Akka.Hosting startup hook** — in `WithOtOpcUaClusterBootstrap`, after the downing HOCON line (~93):
|
||||||
|
|
||||||
|
```csharp
|
||||||
|
// InitJoin self-form fallback (decision 2026-07-22): registered as an Akka.Hosting
|
||||||
|
// startup task so every host built through this bootstrap — production AND the test
|
||||||
|
// hosts in Cluster.Tests — arms it identically. Guarded inside Arm: disabled when
|
||||||
|
// SelfFormAfter is unset; inert when this node is not in its own seed list (site nodes
|
||||||
|
// seeded only by central-1 must wait, not island).
|
||||||
|
var fallbackLogger = (serviceProvider.GetService<ILoggerFactory>()
|
||||||
|
?? Microsoft.Extensions.Logging.Abstractions.NullLoggerFactory.Instance)
|
||||||
|
.CreateLogger("ZB.MOM.WW.OtOpcUa.Cluster.ClusterBootstrapFallback");
|
||||||
|
builder.AddStartup((system, _) =>
|
||||||
|
{
|
||||||
|
ClusterBootstrapFallback.Arm(system, options, fallbackLogger);
|
||||||
|
return Task.CompletedTask;
|
||||||
|
});
|
||||||
|
```
|
||||||
|
|
||||||
|
(Akka.Hosting 1.5.62 has `AddStartup`. If the overload signature differs, alternative: register an `IHostedService` in `AddOtOpcUaCluster` mirroring `ActorSystemTerminationWatchdog`'s lazy-accessor pattern — but prefer `AddStartup` so test hosts arm it for free.)
|
||||||
|
|
||||||
|
**Step 3:** `dotnet build src/Core/ZB.MOM.WW.OtOpcUa.Cluster` — 0 warnings.
|
||||||
|
|
||||||
|
**Step 4: Commit**
|
||||||
|
|
||||||
|
```bash
|
||||||
|
git add -A && git commit -m "feat(cluster): InitJoin self-form fallback armed via Akka.Hosting startup task"
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### Task 3 (B3): `SelfFormBootstrapTests` — 4 scenarios through the production bootstrap
|
||||||
|
|
||||||
|
**Classification:** standard
|
||||||
|
**Estimated implement time:** ~5 min
|
||||||
|
**Parallelizable with:** Task 4
|
||||||
|
|
||||||
|
**Files:**
|
||||||
|
- Create: `tests/Core/ZB.MOM.WW.OtOpcUa.Cluster.Tests/SelfFormBootstrapTests.cs`
|
||||||
|
|
||||||
|
**Step 1: Write the tests** — follow `SplitBrainResolverActivationTests`' pattern (real `Host.CreateDefaultBuilder()` + `AddAkka` + `WithOtOpcUaClusterBootstrap`; that file's remarks explain why testing through the production bootstrap is mandatory — a test that re-creates the wiring pins the test's wiring, not the shipped one). Helper + scenarios:
|
||||||
|
|
||||||
|
```csharp
|
||||||
|
public sealed class SelfFormBootstrapTests
|
||||||
|
{
|
||||||
|
private static int FreePort()
|
||||||
|
{
|
||||||
|
var l = new System.Net.Sockets.TcpListener(System.Net.IPAddress.Loopback, 0);
|
||||||
|
l.Start();
|
||||||
|
var p = ((System.Net.IPEndPoint)l.LocalEndpoint).Port;
|
||||||
|
l.Stop();
|
||||||
|
return p;
|
||||||
|
}
|
||||||
|
|
||||||
|
private static async Task<IHost> StartNodeAsync(
|
||||||
|
string systemName, int selfPort, int peerPort, TimeSpan? selfFormAfter, bool selfInSeeds = true)
|
||||||
|
{
|
||||||
|
var options = new AkkaClusterOptions
|
||||||
|
{
|
||||||
|
SystemName = systemName, Hostname = "127.0.0.1", PublicHostname = "127.0.0.1",
|
||||||
|
Port = selfPort, Roles = new[] { "admin", "driver" },
|
||||||
|
SelfFormAfter = selfFormAfter,
|
||||||
|
SeedNodes = selfInSeeds
|
||||||
|
? new[] { $"akka.tcp://{systemName}@127.0.0.1:{peerPort}",
|
||||||
|
$"akka.tcp://{systemName}@127.0.0.1:{selfPort}" } // self SECOND — only the fallback can form it
|
||||||
|
: new[] { $"akka.tcp://{systemName}@127.0.0.1:{peerPort}" },
|
||||||
|
};
|
||||||
|
var builder = Host.CreateDefaultBuilder();
|
||||||
|
builder.ConfigureServices(services =>
|
||||||
|
{
|
||||||
|
services.AddSingleton<IOptions<AkkaClusterOptions>>(Options.Create(options));
|
||||||
|
services.AddAkka(systemName, (ab, sp) => ab.WithOtOpcUaClusterBootstrap(sp));
|
||||||
|
});
|
||||||
|
var host = builder.Build();
|
||||||
|
await host.StartAsync();
|
||||||
|
return host;
|
||||||
|
}
|
||||||
|
|
||||||
|
// 1) SelfFormAfter_defaults_to_ten_seconds
|
||||||
|
// new AkkaClusterOptions().SelfFormAfter.ShouldBe(TimeSpan.FromSeconds(10));
|
||||||
|
// 2) Lone_non_first_seed_self_forms_after_the_window
|
||||||
|
// dead peer port, window 2s → poll Cluster.Get(system).State until 1 Up member ≤ 20s.
|
||||||
|
// 3) Disabled_fallback_keeps_waiting (window null → State.Members empty after 6s;
|
||||||
|
// POSITIVE CONTROL: cluster.Join(cluster.SelfAddress) → 1 Up ≤ 20s — proves the node
|
||||||
|
// was formable and only the fallback was absent.)
|
||||||
|
// 4) Site_node_seeded_only_by_central_never_self_forms (selfInSeeds:false, window 1s →
|
||||||
|
// Members empty after 5s; positive control join. THE guard test: this is the exact
|
||||||
|
// docker-dev topology — site nodes list only central-1.)
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
Use a unique `systemName` per test (`otopcua-selfform-1` …) so concurrent hosts cannot cross-join; dispose each host in `finally` (`await host.StopAsync(); host.Dispose();`).
|
||||||
|
|
||||||
|
**Step 2: Prove the key assertion has teeth** — before relying on scenario 2, temporarily run it with `selfFormAfter: null` and confirm it times out (red), then restore. Record that in the task notes.
|
||||||
|
|
||||||
|
**Step 3: Run**
|
||||||
|
|
||||||
|
```bash
|
||||||
|
dotnet test tests/Core/ZB.MOM.WW.OtOpcUa.Cluster.Tests --filter "FullyQualifiedName~SelfFormBootstrap"
|
||||||
|
```
|
||||||
|
Expected: 4 PASS.
|
||||||
|
|
||||||
|
**Step 4: Commit**
|
||||||
|
|
||||||
|
```bash
|
||||||
|
git add -A && git commit -m "test(cluster): self-form fallback — lone-seed forms, disabled waits, site-node guard"
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### Task 4 (B4): Docs — Redundancy.md + mesh design note
|
||||||
|
|
||||||
|
**Classification:** small
|
||||||
|
**Estimated implement time:** ~4 min
|
||||||
|
**Parallelizable with:** Task 3
|
||||||
|
|
||||||
|
**Files:**
|
||||||
|
- Modify: `docs/Redundancy.md` (~lines 162, 244–247) — replace the bootstrap-constraint text ("Auto-down removes the crash outage, not this one") with `SelfFormAfter` semantics, the 10 s default, the site-node island guard (why driver-only nodes seeded by central-1 deliberately do NOT self-form under the CURRENT one-mesh topology), and the note that under the future per-cluster mesh each pair node is a seed so the fallback covers both.
|
||||||
|
- Modify: `docs/plans/2026-07-21-per-cluster-mesh-design.md` §6.2/§7 — one-line note: the "registered outage gap" is now closed by `SelfFormAfter` (link Redundancy.md).
|
||||||
|
- Modify: `CLAUDE.md` — brief note in the redundancy-corrections area.
|
||||||
|
|
||||||
|
Commit: `docs(redundancy): SelfFormAfter closes the seed-node bootstrap gap`.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### Task 5 (B5): Full verification + optional docker-dev live check
|
||||||
|
|
||||||
|
**Classification:** standard
|
||||||
|
**Estimated implement time:** ~5 min (suite runtime dominates)
|
||||||
|
**Parallelizable with:** none
|
||||||
|
|
||||||
|
```bash
|
||||||
|
cd ~/Desktop/OtOpcUa
|
||||||
|
dotnet build ZB.MOM.WW.OtOpcUa.slnx # 0 warnings
|
||||||
|
dotnet test ZB.MOM.WW.OtOpcUa.slnx # green vs the pre-existing baseline
|
||||||
|
```
|
||||||
|
|
||||||
|
Optional live check (docker-dev six-node mesh): `docker compose stop central-1 central-2 && docker compose start central-2` → central-2 (a listed seed) self-forms in ~10 s; site nodes (non-seeds) keep waiting — the island guard on the real topology. Then `docker compose start central-1` → joins central-2's cluster, mesh re-forms. If the rig isn't running, record deferred-live.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### Task 6 (D4): Manual failover service + cluster-level test
|
||||||
|
|
||||||
|
**Classification:** high-risk
|
||||||
|
**Estimated implement time:** ~5 min
|
||||||
|
**Parallelizable with:** none
|
||||||
|
|
||||||
|
**Files:**
|
||||||
|
- Create: `src/Server/ZB.MOM.WW.OtOpcUa.ControlPlane/Redundancy/IManualFailoverService.cs`
|
||||||
|
- Create: `src/Server/ZB.MOM.WW.OtOpcUa.ControlPlane/Redundancy/ManualFailoverService.cs`
|
||||||
|
- Modify: the ControlPlane/Host DI registration point (find where `RedundancyStateActor`-adjacent services register; admin-role branch of `Program.cs`)
|
||||||
|
- Test: `tests/Server/ZB.MOM.WW.OtOpcUa.ControlPlane.Tests/ManualFailoverServiceTests.cs`
|
||||||
|
|
||||||
|
**Step 1: Failing tests** — reuse the two-node fixture pattern from `RedundancyPrimaryElectionTests` (node A older on the **higher** port so age ≠ address order; that fixture even has a guard test asserting the divergence — keep it honest here too):
|
||||||
|
|
||||||
|
- `FailOver_targets_the_oldest_driver_member_not_the_role_leader`: service picks node A (oldest, higher port) even though B is `RoleLeader("driver")`; A's system terminates (graceful exit); B's next `RedundancyStateActor` snapshot names B Primary.
|
||||||
|
- `FailOver_refuses_when_no_driver_peer_exists`: single driver member → returns null, no Leave, node still Up (positive assert).
|
||||||
|
- `Parity_with_SelectDriverPrimary`: on the same fixture, the service's target equals the member `RedundancyStateActor` names Primary — pins the two 5-line queries together.
|
||||||
|
|
||||||
|
**Step 2: Implement** — same shape as ScadaBridge's `FailOverCore` but role `"driver"`:
|
||||||
|
|
||||||
|
```csharp
|
||||||
|
public static Address? FailOverCore(ActorSystem system, bool dryRun = false)
|
||||||
|
{
|
||||||
|
var cluster = Akka.Cluster.Cluster.Get(system);
|
||||||
|
// Intentionally identical to RedundancyStateActor.SelectDriverPrimary (oldest Up driver,
|
||||||
|
// Member.AgeOrdering) — the node acted on must be exactly the one the election names.
|
||||||
|
// Pinned by ManualFailoverServiceTests.Parity_with_SelectDriverPrimary.
|
||||||
|
var drivers = cluster.State.Members
|
||||||
|
.Where(m => m.Status == MemberStatus.Up && m.Roles.Contains("driver"))
|
||||||
|
.OrderBy(m => m, Member.AgeOrdering)
|
||||||
|
.ToList();
|
||||||
|
if (drivers.Count < 2) return null;
|
||||||
|
var primary = drivers[0];
|
||||||
|
if (!dryRun) cluster.Leave(primary.Address);
|
||||||
|
return primary.Address;
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
Async wrapper `FailOverDriverPrimaryAsync(string actor)`: dry-run peer guard → audit via the shared `ZB.MOM.WW.Audit.IAuditWriter` seam (action `cluster.manual-failover`, actor, target in `DetailsJson` — copy the call shape from an existing audited AdminUI admin action) → real Leave → warn log → return address string.
|
||||||
|
|
||||||
|
**Step 3:** Tests PASS → commit `feat(redundancy): manual failover service — graceful Leave of the driver Primary`.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### Task 7 (D5): ClusterRedundancy page — live panel + failover button + bUnit tests
|
||||||
|
|
||||||
|
**Classification:** standard
|
||||||
|
**Estimated implement time:** ~5 min
|
||||||
|
**Parallelizable with:** none
|
||||||
|
|
||||||
|
**Files:**
|
||||||
|
- Modify: `src/Server/ZB.MOM.WW.OtOpcUa.AdminUI/Components/Pages/Clusters/ClusterRedundancy.razor` (page is already `@rendermode RenderMode.InteractiveServer` — no rendermode work; and it lives in the AdminUI host, not an RCL, so the static-router wrapper concern from lmxopcua#483 does not apply)
|
||||||
|
- Test: the AdminUI bUnit suite (place beside the existing Clusters page tests)
|
||||||
|
- Modify: `docs/Redundancy.md` — "Manual failover" subsection (button semantics, mesh-wide scope caveat)
|
||||||
|
|
||||||
|
**Step 1: Failing bUnit tests:** (a) button hidden under the read-only policy (`AdminUiPolicies.AuthenticatedRead`), visible under the admin/write policy — check `AdminUiPolicies` for the exact policy the mutating Clusters pages use (e.g. `ClusterEdit.razor`) and use the same one; (b) disabled with tooltip when the live snapshot shows <2 Up driver members; (c) confirm flow calls a fake `IManualFailoverService` exactly once.
|
||||||
|
|
||||||
|
**Step 2: Implement.** Add a "Live redundancy" panel above the static config table: current driver Primary + Up driver members from a `GetSnapshot()` read-only companion on `IManualFailoverService` (sourced from live `Cluster.State`; do NOT subscribe to the DPS topic for v1), the mesh-scope notice ("in the current single-mesh topology this acts on the whole mesh's Primary"), and the admin-gated **Trigger failover** button with confirm dialog (states: the Primary node restarts, ServiceLevel 250 moves to the survivor, OPC UA clients re-select). On confirm: call the service with the authenticated user name; surface the returned target address.
|
||||||
|
|
||||||
|
**Step 3:** Tests PASS → commit `feat(adminui): manual failover control on the cluster redundancy page`.
|
||||||
|
|
||||||
|
**Live check (fold into the Task 5 gate when docker-dev is up):** press the button, ServiceLevel 250 moves (old Primary drops off, survivor publishes 250), denial meters stay quiet, audit event recorded.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Reserved for additions (this plan is expected to grow)
|
||||||
|
|
||||||
|
Append new tasks below as **Task 8+** (and extend `.tasks.json`) — do not renumber Tasks 1–7. Candidate items already flagged in the family docs, awaiting a decision on which to include:
|
||||||
|
|
||||||
|
- **Configurable downing window** — promote `ServiceCollectionExtensions.DowningStableAfter` (hard-coded 15 s) to `AkkaClusterOptions` (e.g. `Cluster:DowningStableAfter`), keeping the pinned invariant window > `acceptable-heartbeat-pause` (`akka_failover.md` §6.2 calls this out; ScadaBridge's equivalent `Cluster:StableAfter` is already config).
|
||||||
|
- **Failure-detector tuning for faster failover** — expose `heartbeat-interval` / `acceptable-heartbeat-pause` (currently fixed in `Resources/akka.conf`) as options, per the availability-first tuning table in `akka_failover.md` §2.
|
||||||
|
- ~~Auto-down 1-vs-1 live gate~~ / ~~per-cluster mesh interplay~~ — **now owned by the per-cluster mesh PROGRAM plan** (`docs/plans/2026-07-22-per-cluster-mesh-program.md`, added 2026-07-22): Phases 1–7 align OtOpcUa's mesh with ScadaBridge's shape (one 2-node mesh per application Cluster, co-located on the ScadaBridge VMs). That program lists THIS plan as its prerequisite (execute this one first); its Phase 6 removes the ClusterRedundancy mesh-scope caveat + the site-node island-guard docs note, and Phase 7 closes the auto-down 1-vs-1 and `SelfFormAfter` live gates on the real per-pair topology.
|
||||||
|
- *(add further items here)*
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Completion
|
||||||
|
|
||||||
|
- Merge decision via the finishing-a-development-branch flow (family convention: ff-merge to `master` + push to gitea `lmxopcua`, or PR — ask the user).
|
||||||
|
- After merge: update `scadaproj/akka_failover.md` §6.1 status, `scadaproj/CLAUDE.md` index row, and memory `ha-availability-over-partition-safety` (tracked in the scadaproj index plan).
|
||||||
|
- Verification-before-completion applies throughout; live gates may be recorded deferred-live if docker-dev is down.
|
||||||
@@ -0,0 +1,13 @@
|
|||||||
|
{
|
||||||
|
"planPath": "docs/plans/2026-07-22-selfform-fallback-and-manual-failover.md",
|
||||||
|
"tasks": [
|
||||||
|
{"id": 0, "subject": "Task 1: SelfFormAfter on AkkaClusterOptions", "status": "completed"},
|
||||||
|
{"id": 1, "subject": "Task 2: ClusterBootstrapFallback + AddStartup wiring", "status": "completed", "blockedBy": [0]},
|
||||||
|
{"id": 2, "subject": "Task 3: SelfFormBootstrapTests — 4 scenarios incl. site-node island guard", "status": "completed", "blockedBy": [1], "notes": "Teeth proven: scenario 2 run with selfFormAfter:null failed (red) before restore."},
|
||||||
|
{"id": 3, "subject": "Task 4: Docs — Redundancy.md bootstrap section + mesh design note", "status": "pending", "blockedBy": [1]},
|
||||||
|
{"id": 4, "subject": "Task 5: Full verification + optional docker-dev live check", "status": "pending", "blockedBy": [2, 3]},
|
||||||
|
{"id": 5, "subject": "Task 6: Manual failover service + cluster-level test", "status": "pending", "blockedBy": [4]},
|
||||||
|
{"id": 6, "subject": "Task 7: ClusterRedundancy live panel + failover button + bUnit tests", "status": "pending", "blockedBy": [5]}
|
||||||
|
],
|
||||||
|
"lastUpdated": "2026-07-22T00:00:00Z"
|
||||||
|
}
|
||||||
Reference in New Issue
Block a user