Compare commits
2 Commits
a78425ea8f
...
9af935f237
| Author | SHA1 | Date | |
|---|---|---|---|
 Joseph Doherty
|
9af935f237 | ||
|
Joseph Doherty
|
3f24d4d6bf |
@@ -224,7 +224,7 @@ The server supports non-transparent warm/hot redundancy via the `Redundancy` sec
|
||||
- **Downing is `auto-down`, not `keep-oldest`.** `Cluster:SplitBrainResolverStrategy` (default `auto-down`) selects the provider via `ServiceCollectionExtensions.BuildDowningHocon`; an unknown value fails host start. A two-node pair running the previous `keep-oldest` + `down-if-alone` **could not survive a crash of the oldest node**: Akka's `KeepOldest.OldestDecision` only lets `down-if-alone` rescue a side holding ≥ 2 members, so the 1-vs-1 survivor downs *itself* and exits. `down-if-alone` is a 3+-node feature. The accepted trade for `auto-down` is dual-active during a genuine network partition. **Its live gate is still open** — the pathology only appears 1-vs-1 and docker-dev is a single six-node mesh, so the crash-the-oldest drill is deferred to the per-cluster mesh work.
|
||||
- **The Primary is the oldest Up `driver` member, not the role leader.** `RedundancyStateActor` previously used `ClusterState.RoleLeader("driver")` (lowest *address*), while `ClusterSingletonManager` places singletons on the *oldest*. They agree only on a freshly-formed cluster and diverge after any restart, so the Primary-gated data plane (writes, alarm acks, alerts emit, alarm-history drain) could enable on a node not hosting the singletons. `NodeRedundancyState.IsRoleLeaderForDriver` was renamed **`IsDriverPrimary`**.
|
||||
|
||||
**A third bootstrap gap closed 2026-07-22 — downing was never the whole story.** Even under `auto-down`, a node cold-starting while its peer was dead **never came Up**: Akka lets only the **first** listed seed form a *new* cluster, so every other node loops on `InitJoin` forever. `Cluster:SelfFormAfter` (`TimeSpan?`, default 10 s, `null`/`≤0` disables) arms `ClusterBootstrapFallback` from `WithOtOpcUaClusterBootstrap` via an Akka.Hosting `AddStartup` task — it waits that long for membership, then `Cluster.Join(SelfAddress)`. **It fires only when this node's own address is in its own `SeedNodes`**: a node seeded only by someone else (today's docker-dev site nodes, which list only `central-1`) must wait rather than island itself permanently. Pinned by `SelfFormBootstrapTests`, whose two negative cases each carry a positive-control self-join so they cannot pass vacuously. See `docs/Redundancy.md` §"Bootstrap: the self-form fallback".
|
||||
**A third bootstrap gap closed 2026-07-22 — downing was never the whole story, and the fix changed twice.** Even under `auto-down`, a node cold-starting while its peer was dead **never came Up**: Akka runs `FirstSeedNodeProcess` — the only bootstrap path that can form a *new* cluster when no peer answers `InitJoin` — exclusively when `seed-nodes[0]` is the node’s own address; every other node runs `JoinSeedNodeProcess` and retries `InitJoin` forever. **`Cluster:SeedNodes` is therefore ORDERED: every node that is one of its own seeds lists ITSELF first, partner second** (docker-dev `central-2` was swapped; site nodes list only `central-1` and are legitimately not seeds). `AkkaClusterOptionsValidator` (`AddValidatedOptions`/`ValidateOnStart` in `AddOtOpcUaCluster`) enforces it at boot — **conditionally** (self must be entry 0 only *if* self is in the list, or every site node would refuse to start), matching on `PublicHostname`+`Port`, never the `0.0.0.0` bind address. The earlier fix — a `Cluster:SelfFormAfter` timer calling `Cluster.Join(SelfAddress)` — is **deleted**: it sat outside Akka’s join handshake, so it could not tell "no seed answered" from "a seed answered and the join is in flight", and it islanded a manually-failed-over node live before a TCP reachability guard patched that one shape. Pinned by `SelfFirstSeedBootstrapTests` (real in-process clusters, incl. a falsifiability control proving peer-first ordering never forms) + `AkkaClusterOptionsValidatorTests`. See `docs/Redundancy.md` §"Bootstrap: self-first seed ordering".
|
||||
|
||||
Assert the **effective** cluster config off a running `ActorSystem` (as `SplitBrainResolverActivationTests` does), never the typed options or the akka.conf text — several HOCON fragments compete and the losing one still reads correctly in the file.
|
||||
|
||||
|
||||
@@ -13,7 +13,9 @@
|
||||
# central-1, central-2 OTOPCUA_ROLES=admin,driver — the ONLY UI + deploy
|
||||
# singleton, plus the MAIN cluster's OPC UA publishers.
|
||||
# Reachable at http://localhost:9200 (via Traefik).
|
||||
# central-1 is the Akka seed node; central-2 joins it.
|
||||
# Both are Akka seed nodes, each listing ITSELF first
|
||||
# (2026-07-22) so either can cold-start while the other
|
||||
# is down; the site nodes seed off central-1 only.
|
||||
# site-a-1, site-a-2 OTOPCUA_ROLES=driver — driver-only members of the same
|
||||
# site-b-1, site-b-2 mesh, scoped to SITE-A / SITE-B by ClusterId. They
|
||||
# serve no UI and authenticate no users; the central
|
||||
@@ -136,7 +138,8 @@ services:
|
||||
|
||||
# ── Central cluster (2-node fused admin+driver) ─────────────────────────────
|
||||
# The only UI + deploy singleton; also the MAIN cluster's OPC UA publishers.
|
||||
# central-1 seeds the single Akka mesh that every other node joins.
|
||||
# central-1 seeds the single Akka mesh that every other node joins; central-2 is a seed too
|
||||
# (listing itself first), so it can form the mesh alone if central-1 is down.
|
||||
|
||||
central-1: &otopcua-host
|
||||
build:
|
||||
@@ -176,8 +179,13 @@ services:
|
||||
Cluster__Port: "4053"
|
||||
Cluster__PublicHostname: "central-1"
|
||||
# Both redundancy peers are seeds (#459 / ScadaBridge parity) so a restarted node can
|
||||
# re-join the mesh via EITHER peer, not only central-1. With restart supervision this is
|
||||
# the 2-node keep-oldest exit-and-rejoin recovery path.
|
||||
# re-join the mesh via EITHER peer, not only central-1.
|
||||
#
|
||||
# ORDER IS LOAD-BEARING (2026-07-22): each node lists ITSELF first. Akka runs
|
||||
# FirstSeedNodeProcess — the only path that can form a NEW cluster when no peer answers
|
||||
# InitJoin — exclusively for seed-nodes[0]; every other node retries InitJoin forever. So a
|
||||
# node listing its partner first cannot cold-start while that partner is down. Enforced at
|
||||
# boot by AkkaClusterOptionsValidator; see docs/Redundancy.md.
|
||||
Cluster__SeedNodes__0: "akka.tcp://otopcua@central-1:4053"
|
||||
Cluster__SeedNodes__1: "akka.tcp://otopcua@central-2:4053"
|
||||
Cluster__Roles__0: "admin"
|
||||
@@ -248,9 +256,11 @@ services:
|
||||
Cluster__Hostname: "0.0.0.0"
|
||||
Cluster__Port: "4053"
|
||||
Cluster__PublicHostname: "central-2"
|
||||
# Both redundancy peers are seeds (#459 / ScadaBridge parity) — see central-1.
|
||||
Cluster__SeedNodes__0: "akka.tcp://otopcua@central-1:4053"
|
||||
Cluster__SeedNodes__1: "akka.tcp://otopcua@central-2:4053"
|
||||
# Both redundancy peers are seeds (#459 / ScadaBridge parity) — see central-1. SELF FIRST:
|
||||
# central-2 lists itself as seed-nodes[0], which is what lets it cold-start while central-1
|
||||
# is down (it previously listed central-1 first and simply never came Up in that case).
|
||||
Cluster__SeedNodes__0: "akka.tcp://otopcua@central-2:4053"
|
||||
Cluster__SeedNodes__1: "akka.tcp://otopcua@central-1:4053"
|
||||
Cluster__Roles__0: "admin"
|
||||
Cluster__Roles__1: "driver"
|
||||
Security__Jwt__SigningKey: "docker-dev-signing-key-with-at-least-32-bytes-of-utf8-content-12345"
|
||||
@@ -314,6 +324,9 @@ services:
|
||||
Cluster__Hostname: "0.0.0.0"
|
||||
Cluster__Port: "4053"
|
||||
Cluster__PublicHostname: "site-a-1"
|
||||
# Site nodes are deliberately NOT seeds — they join the central pair's mesh. The self-first
|
||||
# seed rule is conditional for exactly this reason (it binds only when a node's own address
|
||||
# is in its own seed list), so these configs are exempt rather than broken.
|
||||
Cluster__SeedNodes__0: "akka.tcp://otopcua@central-1:4053"
|
||||
Cluster__Roles__0: "driver"
|
||||
<<: *secrets-env
|
||||
|
||||
@@ -103,7 +103,7 @@ The checked-in `appsettings*.json` files are deliberately thin: they carry only
|
||||
| `Hostname` | string | `0.0.0.0` | Bind hostname. |
|
||||
| `Port` | int | `4053` | Cluster transport port. |
|
||||
| `PublicHostname` | string | `127.0.0.1` | Hostname advertised in cluster gossip; must be reachable by peers. |
|
||||
| `SeedNodes` | string[] | `[]` | Seed nodes for bootstrapping. |
|
||||
| `SeedNodes` | string[] | `[]` | Seed nodes for bootstrapping. **Ordered:** a node that appears in its own list must be entry 0 (only `seed-nodes[0]` can form a new cluster), or the host refuses to start — see [Redundancy.md § Bootstrap: self-first seed ordering](Redundancy.md#bootstrap-self-first-seed-ordering). |
|
||||
| `Roles` | string[] | `[]` | Cluster roles for this node. When empty, falls back to `OTOPCUA_ROLES`. Allowed values: `admin`, `driver`, `dev`. |
|
||||
|
||||
> The full redundancy model (ServiceLevel tiers, split-brain, peer discovery) is in [`Redundancy.md`](Redundancy.md). The OPC UA peer-URI advertising lives in the `OpcUa:PeerApplicationUris` key above.
|
||||
|
||||
+81
-57
@@ -143,13 +143,18 @@ The admin singleton is the cluster's only `RedundancyStateActor`. If the admin l
|
||||
|
||||
Per-node identity comes from `appsettings.json` + the `OTOPCUA_ROLES` env var:
|
||||
|
||||
```json
|
||||
```jsonc
|
||||
{
|
||||
"Cluster": {
|
||||
"Hostname": "0.0.0.0",
|
||||
"Port": 4053,
|
||||
"PublicHostname": "node-a.lan",
|
||||
"SeedNodes": ["akka.tcp://otopcua@node-a.lan:4053"],
|
||||
// Self FIRST, partner second — see "Bootstrap: self-first seed ordering".
|
||||
// node-b.lan's own config lists node-b.lan first and node-a.lan second.
|
||||
"SeedNodes": [
|
||||
"akka.tcp://otopcua@node-a.lan:4053",
|
||||
"akka.tcp://otopcua@node-b.lan:4053"
|
||||
],
|
||||
"Roles": ["admin", "driver"]
|
||||
}
|
||||
}
|
||||
@@ -159,7 +164,7 @@ Per-node identity comes from `appsettings.json` + the `OTOPCUA_ROLES` env var:
|
||||
OTOPCUA_ROLES=admin,driver
|
||||
```
|
||||
|
||||
Both nodes share the same `ConfigDb` connection string; `Cluster.PublicHostname` + `Roles` are what makes them distinct in cluster gossip. The first node bootstraps the cluster (its address goes in `SeedNodes`); the second node joins via the same `SeedNodes` list. List **both** peers in `SeedNodes` on **both** nodes so either can cold-start alone — see [Bootstrap: the self-form fallback](#bootstrap-the-self-form-fallback) for why the seed list alone is not enough, and what `Cluster:SelfFormAfter` adds.
|
||||
Both nodes share the same `ConfigDb` connection string; `Cluster.PublicHostname` + `Roles` are what makes them distinct in cluster gossip. List **both** peers in `SeedNodes` on **both** nodes so either can cold-start alone — **and list each node ITSELF first**: the order decides which Akka bootstrap process runs, and a node that lists its partner first can never form the cluster while that partner is down. See [Bootstrap: self-first seed ordering](#bootstrap-self-first-seed-ordering).
|
||||
|
||||
There is no longer a `Node:NodeId` setting and no `ClusterNode.RedundancyRole` column (the V2 migration dropped it — primary/secondary is now derived from cluster membership age). NodeId is derived as `host:port` of the cluster `PublicHostname` (see `ClusterRoleInfo.LocalNode` for the formula).
|
||||
|
||||
@@ -241,9 +246,10 @@ two-node pair that choice means *any* crash of the oldest node is a full outage.
|
||||
`IHostApplicationLifetime.StopApplication()` so the process exits and the supervisor restarts it instead
|
||||
of idling forever with a dead actor system. Under `auto-down` the *survivor* no longer needs this
|
||||
(it is never the one downed); the **downed** node still does.
|
||||
3. **Both peers listed in `SeedNodes`** on every node, so a restarted node can re-join via either peer —
|
||||
plus the **self-form fallback** below, without which a node cold-starting while its peer is dead waits
|
||||
in `InitJoin` forever (auto-down removes the crash outage, not that one).
|
||||
3. **Both peers listed in `SeedNodes`** on every node, **each node listing itself first**, so a restarted
|
||||
node can re-join via either peer and a node cold-starting while its peer is dead still forms the
|
||||
cluster. Without the self-first ordering it waits in `InitJoin` forever (auto-down removes the crash
|
||||
outage, not that one) — see [Bootstrap: self-first seed ordering](#bootstrap-self-first-seed-ordering).
|
||||
|
||||
> **On `HardKillFailoverTests`:** that in-process test hard-kills the oldest and asserts the survivor becomes
|
||||
> sole `driver` role-leader. It passed even under `keep-oldest` — because it simulates the crash with
|
||||
@@ -297,66 +303,84 @@ advertises `ServiceLevel` 250. OPC UA clients re-select.
|
||||
> alongside the per-cluster mesh work, which makes every mesh exactly two nodes. See
|
||||
> `docs/plans/2026-07-21-per-cluster-mesh-design.md` §6.2 / Phase 0a.
|
||||
|
||||
## Bootstrap: the self-form fallback
|
||||
## Bootstrap: self-first seed ordering
|
||||
|
||||
Listing both peers in `SeedNodes` does **not** mean either node can cold-start alone. Akka lets only the
|
||||
**first** listed seed form a *new* cluster; every other node sends `InitJoin` to the seeds and retries —
|
||||
forever — until one of them answers. A node that boots while its peer is dead therefore never comes Up,
|
||||
and no downing strategy helps: **auto-down removes the crash outage, not this one.**
|
||||
Listing both peers in `SeedNodes` does **not** mean either node can cold-start alone — and neither does any
|
||||
downing strategy: **auto-down removes the crash outage, not this one.** Akka runs a different bootstrap
|
||||
process depending on whether `seed-nodes[0]` is the node's own address:
|
||||
|
||||
`Cluster:SelfFormAfter` closes that gap. `ClusterBootstrapFallback.Arm` (armed for every host built through
|
||||
`WithOtOpcUaClusterBootstrap`, via an Akka.Hosting `AddStartup` task) waits that long for cluster membership
|
||||
and, on expiry, calls `Cluster.Join(SelfAddress)` — the node forms a cluster on itself and becomes
|
||||
operational unattended.
|
||||
| `seed-nodes[0]` is… | Process Akka runs | Can it form a NEW cluster? |
|
||||
|---|---|---|
|
||||
| this node's own address | `FirstSeedNodeProcess` | **Yes** — it `InitJoin`s the other seeds and self-joins once `seed-node-timeout` (5 s) passes with nobody answering. |
|
||||
| some other node | `JoinSeedNodeProcess` | **No** — it retries `InitJoin` forever, however long the peer stays down. |
|
||||
|
||||
| `Cluster:SelfFormAfter` | Behaviour |
|
||||
|---|---|
|
||||
| `"00:00:10"` (**default**) | A node with no membership after 10 s self-forms. A live peer answers `InitJoin` in milliseconds, so on any normal boot the fallback never fires. |
|
||||
| `null` / `≤ 0` | Disabled — the pre-2026-07-22 behaviour: wait on `InitJoin` indefinitely. |
|
||||
So the ordering *is* the mechanism. **Every node that is a seed of its own mesh lists ITSELF first and its
|
||||
partner second** (decision 2026-07-22):
|
||||
|
||||
**Two guards keep the fallback from creating the split it exists to avoid.**
|
||||
```jsonc
|
||||
// central-1 // central-2
|
||||
"SeedNodes": [ "SeedNodes": [
|
||||
"akka.tcp://otopcua@central-1:4053", "akka.tcp://otopcua@central-2:4053",
|
||||
"akka.tcp://otopcua@central-2:4053" "akka.tcp://otopcua@central-1:4053"
|
||||
] ]
|
||||
```
|
||||
|
||||
**1 — Reachability.** An expired window is not evidence the peer is down, so before self-forming the
|
||||
fallback TCP-probes the other seed addresses; if any accepts a connection it waits another window instead.
|
||||
This was added after the live gate below caught the fallback islanding a node: a node bounced by a manual
|
||||
failover restarted, received `InitJoinAck` from its live peer — a join in flight and healthy — but did not
|
||||
get the Welcome inside the window, because the peer's ring still held the node's previous incarnation
|
||||
(`Exiting` → `Down` → `Removed`). The fallback fired on the timer and the node formed a **second cluster**.
|
||||
The original design assumed `Cluster.Join(SelfAddress)` would be ignored mid-handshake; **it is not — it
|
||||
wins.** Since manual failover deliberately produces exactly that restart, without this guard every manual
|
||||
failover could island the node it bounced. Pinned by
|
||||
`SelfFormBootstrapTests.Reachable_peer_suppresses_self_forming_until_it_goes_away`, whose "peer" is a bare
|
||||
`TcpListener` that speaks no Akka: reachable, never answering a join.
|
||||
The rule is **conditional**: it binds only when a node's own address appears in its own seed list. A
|
||||
driver-only site node is seeded solely by `central-1` (today's docker-dev topology) — it is legitimately
|
||||
not a seed of anything and is exempt. An unconditional "self must be first" rule would refuse to boot every
|
||||
site node in the fleet.
|
||||
|
||||
**2 — Seed membership.** The fallback fires **only when this node's own address
|
||||
appears in its own `SeedNodes`**. A node that is not one of its own seeds is never legitimately first, and
|
||||
if it self-formed, a later-booting real seed would form a second cluster the two could never merge. That is
|
||||
exactly the current docker-dev topology: the site-a/site-b driver nodes list only `central-1` as a seed, so
|
||||
they deliberately keep waiting rather than islanding themselves. Under the future per-cluster mesh
|
||||
(`docs/plans/2026-07-21-per-cluster-mesh-design.md`) each pair node *is* a seed of its own two-node mesh, so
|
||||
the fallback covers both sides of every pair and the site-node exception disappears.
|
||||
`AkkaClusterOptionsValidator` enforces it at startup (`AddValidatedOptions` → `ValidateOnStart` in
|
||||
`AddOtOpcUaCluster`), because the invariant otherwise fails **silently**: the process runs, the port
|
||||
listens, the node simply never becomes a member. Identity is compared on `Cluster:PublicHostname` (falling
|
||||
back to `Cluster:Hostname` when blank) **and** `Cluster:Port` — the address Akka puts in `SelfAddress`.
|
||||
Comparing against `Cluster:Hostname` would be worse than wrong: in docker-dev it is `0.0.0.0`, which matches
|
||||
no seed URI anywhere, so the rule would silently exempt the entire rig.
|
||||
|
||||
Sequential recovery is island-free by construction: Akka's join protocol prefers an existing cluster (a
|
||||
booting node only self-forms when **no** seed answered), so a peer booting after the survivor self-formed
|
||||
simply joins it as the youngest member. The residual risk is both pair nodes cold-starting inside the
|
||||
window while mutually unreachable — both self-form, which is the same dual-active class the `auto-down`
|
||||
strategy already accepts, with the same recovery (restart one side).
|
||||
Sequential recovery is island-free by construction: a `FirstSeedNodeProcess` node self-joins only when **no**
|
||||
seed answered, so a peer booting after the survivor formed simply joins it as the youngest member. Both
|
||||
nodes cold-starting *simultaneously* converge on one cluster while they are mutually reachable — the
|
||||
`InitJoin` handshake resolves it before either self-join deadline expires. The residual risk is a boot-time
|
||||
**partition** (both cold-starting, mutually unreachable): both form, which is the same dual-active class the
|
||||
`auto-down` strategy already accepts, with the same recovery (restart one side).
|
||||
|
||||
Pinned by `SelfFormBootstrapTests` (`tests/Core/ZB.MOM.WW.OtOpcUa.Cluster.Tests/`), which starts real hosts
|
||||
through the production bootstrap: a lone non-first seed comes Up; a disabled window keeps waiting; a node
|
||||
absent from its own seed list never self-forms; a node whose peer is merely *reachable* keeps waiting. Every
|
||||
negative case carries a positive control (an explicit self-join, or dropping the peer listener) so none can
|
||||
pass merely because the node was unformable.
|
||||
Pinned by `SelfFirstSeedBootstrapTests` (`tests/Core/ZB.MOM.WW.OtOpcUa.Cluster.Tests/`), which starts real
|
||||
in-process clusters through the production bootstrap at production failure-detection timings: a lone
|
||||
cold-start with a dead peer forms alone; a restarted node rejoins its live peer; two nodes cold-starting
|
||||
together converge on one cluster. Its **falsifiability control** — the old peer-first ordering, same
|
||||
scenario, never comes Up — is what keeps the other three from going vacuous, and carries a positive-control
|
||||
self-join proving the node was formable all along. `AkkaClusterOptionsValidatorTests` covers the validator,
|
||||
including the site-node exemption and the `0.0.0.0` identity trap.
|
||||
|
||||
> **Live gate: PASSED (docker-dev, 2026-07-22).** Drilled on the six-node rig, defect first and fix
|
||||
> second. On the pre-change image, `central-2` started alone (central-1 stopped) and logged **zero** join
|
||||
> events in 70 s — the `InitJoin` loop, observed rather than argued. On the rebuilt image the same node
|
||||
> logged the self-form warning **10 s** after start and `Leader is moving node [central-2] to [Up]`, then
|
||||
> came up with all four admin singletons. A `site-a-2` recreated while central-1 was still down logged
|
||||
> *"self-form fallback inactive: this node … is not in its own seed list"* and stayed out — the island
|
||||
> guard on the real topology. Restarting `central-1` produced `Welcome from [central-2]`: it joined the
|
||||
> self-formed cluster instead of starting a second one, and the four site nodes then joined through it —
|
||||
> six members, **one** cluster.
|
||||
### Why the self-form watchdog was retired (2026-07-22)
|
||||
|
||||
The first fix for this gap was `Cluster:SelfFormAfter` + `ClusterBootstrapFallback`: wait 10 s for
|
||||
membership, then call `Cluster.Join(SelfAddress)`. **Both are deleted.** A timer sits *outside* Akka's join
|
||||
handshake, so it cannot distinguish "no seed answered" from "a seed answered and the join is in flight" —
|
||||
and `Cluster.Join(SelfAddress)` is **not** ignored mid-handshake: it wins.
|
||||
|
||||
> **Live-gate finding (docker-dev, 2026-07-22 — preserved because it is the evidence).** The watchdog
|
||||
> islanded a node that manual failover had bounced. `central-1` restarted, received `InitJoinAck` from
|
||||
> `central-2` at 10:49:05 — the join was in flight and healthy — but no Welcome arrived inside the window,
|
||||
> because the peer's ring still held its previous incarnation (`Exiting` → `Down` → `Removed`, retired at
|
||||
> 10:49:06). At 10:49:15 the watchdog fired and the node formed a **second cluster**, islanded until an
|
||||
> operator restarted it. Manual failover deliberately produces exactly that restart, so the two halves of
|
||||
> that plan collided. A TCP reachability guard (`ea45ace1`) patched that one shape — never self-form while
|
||||
> a seed peer accepts a connection — and the earlier live gate for the watchdog itself had genuinely
|
||||
> passed: `central-2` cold-started alone in 10 s and a non-seed `site-a-2` correctly stayed out.
|
||||
|
||||
The reachability guard closed the observed failure, not the class: a peer can be TCP-reachable and not
|
||||
answering, or answering and slow, and the timer cannot tell. Self-first ordering has no such race because
|
||||
the decision is made *inside* the handshake, by Akka, on the actual `InitJoin` replies. It was also inert
|
||||
for every non-seed node, so after this change it could never usefully fire. `SelfFormBootstrapTests` is
|
||||
replaced by `SelfFirstSeedBootstrapTests`, whose `Restarting_node_rejoins_its_live_peer_instead_of_islanding`
|
||||
is the drilled scenario above.
|
||||
|
||||
> **Live gate outstanding for the new mechanism.** Self-first ordering is verified here by in-process
|
||||
> clusters at production timings, and on the sister project (ScadaBridge `4a6341d8`) on real clusters *and*
|
||||
> live docker containers. It has **not** been re-drilled on the OtOpcUa docker-dev rig since the swap — the
|
||||
> `central-2` cold-start-alone drill should be repeated on the rebuilt image. Owned by
|
||||
> `docs/plans/2026-07-22-per-cluster-mesh-program.md` Phase 7 alongside the auto-down 1-vs-1 gate.
|
||||
|
||||
## Primary data-plane gate (writes, acks, alerts emit)
|
||||
|
||||
|
||||
@@ -56,6 +56,7 @@ The host joins an Akka.NET cluster bound to the address in `appsettings.json::Cl
|
||||
}
|
||||
```
|
||||
|
||||
- `SeedNodes` is **ordered**: a node listed in its own seed list must be entry 0, since Akka only lets `seed-nodes[0]` form a new cluster. `AkkaClusterOptionsValidator` fails startup otherwise — see [Redundancy.md § Bootstrap: self-first seed ordering](Redundancy.md#bootstrap-self-first-seed-ordering).
|
||||
- `WithOtOpcUaClusterBootstrap` (in `OtOpcUa.Cluster`) loads the embedded HOCON (split-brain resolver, pinned dispatcher, failure detector tuning) and overlays remote endpoint + cluster options.
|
||||
- All cluster singletons + per-node actors live on this single ActorSystem — there is no second Akka instance.
|
||||
|
||||
|
||||
@@ -85,11 +85,13 @@ Their failover drill has a mode that exists *"to make the registered gap observa
|
||||
it is covered."* **A two-node keep-oldest mesh has an acknowledged total-outage hole.** If OtOpcUa
|
||||
adopts 2-node meshes, this must be decided deliberately, not inherited.
|
||||
|
||||
> **CLOSED 2026-07-22.** The `InitJoin` half of this gap is fixed in both repos by the self-form
|
||||
> fallback: `Cluster:SelfFormAfter` (default 10 s) lets a node that is one of its own seeds form a
|
||||
> cluster on itself when no seed answers. See `docs/Redundancy.md` §"Bootstrap: the self-form
|
||||
> fallback" — including the island guard that keeps today's non-seed site nodes waiting, which the
|
||||
> per-cluster mesh removes by making every pair node a seed of its own mesh.
|
||||
> **CLOSED 2026-07-22.** The `InitJoin` half of this gap is fixed in both repos by **self-first seed
|
||||
> ordering**: every node that is one of its own seeds lists ITSELF as `seed-nodes[0]`, which is the
|
||||
> only ordering under which Akka runs `FirstSeedNodeProcess` and can form a cluster with no peer
|
||||
> answering. (An earlier `Cluster:SelfFormAfter` watchdog was tried here and retired — it raced the
|
||||
> join handshake.) See `docs/Redundancy.md` §"Bootstrap: self-first seed ordering". Today's
|
||||
> driver-only site nodes are seeded by `central-1` only and are exempt; the per-cluster mesh removes
|
||||
> even that exception by making every pair node a seed of its own mesh.
|
||||
|
||||
### Both inter-cluster transports are unauthenticated
|
||||
|
||||
@@ -261,11 +263,14 @@ with Phase 6/7, which makes every mesh exactly two nodes.
|
||||
|
||||
**The bootstrap half of the gap is also closed (2026-07-22).** Downing was only ever one of the two
|
||||
outage modes: even with `auto-down`, a node cold-starting while its peer was dead looped on `InitJoin`
|
||||
forever, because Akka lets only the first listed seed form a new cluster. `Cluster:SelfFormAfter`
|
||||
(default 10 s, `ClusterBootstrapFallback`) now self-forms in that case, guarded so a node absent from
|
||||
its own seed list never islands itself. Pinned by `SelfFormBootstrapTests`; see `docs/Redundancy.md`
|
||||
§"Bootstrap: the self-form fallback". Under this design each pair node is a seed of its own two-node
|
||||
mesh, so the fallback covers both sides and the current site-node exception disappears.
|
||||
forever, because Akka lets only `seed-nodes[0]` form a new cluster. The fix is **self-first seed
|
||||
ordering** — each seed node lists itself first, enforced at boot by `AkkaClusterOptionsValidator`.
|
||||
The `Cluster:SelfFormAfter` watchdog first shipped for this is deleted: a timer outside Akka's join
|
||||
handshake cannot tell "no seed answered" from "a seed answered and the join is in flight", and it
|
||||
islanded a manually-failed-over node live. Pinned by `SelfFirstSeedBootstrapTests`; see
|
||||
`docs/Redundancy.md` §"Bootstrap: self-first seed ordering". Under this design each pair node is a
|
||||
seed of its own two-node mesh, so the ordering covers both sides and the current site-node exemption
|
||||
disappears.
|
||||
|
||||
### 6.3 DECIDED — match ScadaBridge's auth posture for now
|
||||
|
||||
|
||||
@@ -46,8 +46,9 @@ Two independent 2-node Akka clusters per site — they share hardware, never a m
|
||||
| OtOpcUa LocalDb sync port | OtOpcUa driver | h2c LocalDb pair replication (default off/0 today — Phase 6 assigns a real per-site port) |
|
||||
|
||||
**Aligned HA posture (both products, per VM — already true or landing via the selfform plan):**
|
||||
auto-down downing (15 s window), oldest-Up active/primary election, `SelfFormAfter` 10 s
|
||||
self-form fallback, termination-watchdog → process exit → `sc.exe failure` restart recovery.
|
||||
auto-down downing (15 s window), oldest-Up active/primary election, self-first seed ordering
|
||||
(the `SelfFormAfter` self-form fallback that briefly stood in for it was retired 2026-07-22),
|
||||
termination-watchdog → process exit → `sc.exe failure` restart recovery.
|
||||
One failover story for operators regardless of product.
|
||||
|
||||
**Prerequisite ordering:** the fallback/manual-failover plan
|
||||
@@ -143,12 +144,14 @@ observability).
|
||||
kill-and-reconnect of the central dialer recovers every stream.
|
||||
|
||||
### Phase 6 — Mesh partition + co-location topology
|
||||
**Scope:** per-cluster seed nodes — **adopt ScadaBridge's self-first ordering (each node lists
|
||||
ITSELF as `seed-nodes[0]`, partner second) and RETIRE the `SelfFormAfter` watchdog + TCP
|
||||
reachability guard** (2026-07-22 execution finding: the watchdog races Akka's join handshake —
|
||||
`Join(self)` during an in-flight join islands the node, hit live here and fixed with the guard;
|
||||
ScadaBridge proved self-first ordering is the race-free form of the same semantics, enforced by
|
||||
a startup-validator rule — port that rule too); cluster-scoped roles `cluster-{ClusterId}` + singleton re-scoping,
|
||||
**Scope:** per-cluster seed nodes — ~~adopt ScadaBridge's self-first ordering and RETIRE the
|
||||
`SelfFormAfter` watchdog + TCP reachability guard~~ **DONE EARLY 2026-07-22** (docker-dev
|
||||
`central-2` swapped to self-first, `ClusterBootstrapFallback`/`SelfFormAfter` deleted,
|
||||
`AkkaClusterOptionsValidator` ports ScadaBridge's startup rule in its conditional form, and
|
||||
`SelfFirstSeedBootstrapTests` replaces `SelfFormBootstrapTests`; see `docs/Redundancy.md`
|
||||
§"Bootstrap: self-first seed ordering"). What remains for this phase is the per-pair
|
||||
`Cluster__SeedNodes__*` matrix once the meshes actually split — every node in a pair is then a seed
|
||||
of its own mesh, so the site-node exemption disappears; cluster-scoped roles `cluster-{ClusterId}` + singleton re-scoping,
|
||||
central pair keeps the admin singletons; **docker-dev rig rewritten** to model the real topology —
|
||||
including the co-location port table above (both products' compose files on shared per-site
|
||||
networks, real LocalDb sync ports); remove the ClusterRedundancy page's mesh-scope caveat (the
|
||||
@@ -163,12 +166,13 @@ cross separate meshes).
|
||||
### Phase 7 — Failover drill + live gates
|
||||
**Scope:** the drill ScadaBridge already has (`failover-drill.sh` analogue) run per pair, both
|
||||
directions; **close the two outstanding live gates**: (a) auto-down 1-vs-1 crash-the-oldest
|
||||
(deferred since Phase 0a — finally testable, every mesh is exactly two nodes), (b) `SelfFormAfter`
|
||||
(deferred since Phase 0a — finally testable, every mesh is exactly two nodes), (b) self-first
|
||||
lone-cold-start on the real per-pair topology; manual-failover button re-verified per pair;
|
||||
operator runbook for the co-located site (one page covering both products' failover on the same
|
||||
two VMs).
|
||||
**Exit gate:** drill green on every pair type (central, site); runbook merged; design doc §7
|
||||
table fully marked DONE.
|
||||
table fully marked DONE. (Live gate (b) is now the **self-first cold-start-alone** drill, not
|
||||
`SelfFormAfter` — the watchdog it named was retired 2026-07-22.)
|
||||
|
||||
---
|
||||
|
||||
|
||||
@@ -9,7 +9,7 @@
|
||||
{"id": 4, "subject": "Phase 4: cut driver-side ConfigDb — EfAlarmConditionStateStore to LocalDb, DbHealthProbeActor ServiceLevel input, OpcUaPublishActor audit (live gate)", "status": "pending", "blockedBy": [3]},
|
||||
{"id": 5, "subject": "Phase 5: gRPC oneof stream contract; migrate 7 observability topics; AdminUI reconnect story", "status": "pending", "blockedBy": [2]},
|
||||
{"id": 6, "subject": "Phase 6: mesh partition — per-pair seeds, cluster-scoped roles/singletons, co-location rig rewrite, secrets replication re-scope", "status": "pending", "blockedBy": [0, 4, 5]},
|
||||
{"id": 7, "subject": "Phase 7: failover drill per pair (incl. shared-VM failure) + close auto-down 1v1 and SelfFormAfter live gates + operator runbook", "status": "pending", "blockedBy": [6]}
|
||||
{"id": 7, "subject": "Phase 7: failover drill per pair (incl. shared-VM failure) + close auto-down 1v1 and self-first cold-start-alone live gates + operator runbook", "status": "pending", "blockedBy": [6]}
|
||||
],
|
||||
"lastUpdated": "2026-07-22T00:00:00Z"
|
||||
}
|
||||
|
||||
@@ -1,5 +1,22 @@
|
||||
# OtOpcUa: InitJoin Self-Form Fallback + Manual Failover Control — Implementation Plan
|
||||
|
||||
> ## ⚠ SUPERSEDED IN PART (2026-07-22, same day)
|
||||
>
|
||||
> **Half (1) of this plan — the `Cluster:SelfFormAfter` self-form watchdog — was retired within hours
|
||||
> of landing.** `ClusterBootstrapFallback`, the `SelfFormAfter` option and `SelfFormBootstrapTests`
|
||||
> are **deleted**; the cold-start-alone goal is now met by **self-first seed ordering** (each seed
|
||||
> node lists ITSELF as `seed-nodes[0]`), enforced at boot by `AkkaClusterOptionsValidator` and pinned
|
||||
> by `SelfFirstSeedBootstrapTests`. Reason: a timer outside Akka's join handshake cannot distinguish
|
||||
> "no seed answered" from "a seed answered and the join is in flight", and `Cluster.Join(SelfAddress)`
|
||||
> is not ignored mid-handshake — it wins. Task 8 below caught exactly that live and patched the one
|
||||
> observed shape with a TCP reachability guard; the race remained. Mesh-program Phase 6 had this
|
||||
> convergence queued and it was pulled forward. See `docs/Redundancy.md` §"Bootstrap: self-first seed
|
||||
> ordering" and ScadaBridge `4a6341d8`.
|
||||
>
|
||||
> **Half (2) — manual failover — is unaffected and still current.** Tasks below are kept verbatim as
|
||||
> the execution record, including the Task 8 live-gate evidence, which is the whole reason the
|
||||
> watchdog was rejected. Do not execute the Task 1/2/3 code as written.
|
||||
|
||||
> **For Claude:** REQUIRED SUB-SKILL: Use superpowers-extended-cc:executing-plans to implement this plan task-by-task.
|
||||
>
|
||||
> Shared cross-repo design: `~/Desktop/scadaproj/docs/plans/2026-07-22-initjoin-selfform-fallback.md` (design rationale, MNTR assessment, behavior spec). The ScadaBridge half lives in `~/Desktop/ScadaBridge/docs/plans/2026-07-22-selfform-fallback-and-manual-failover.md`. This plan is self-contained for execution.
|
||||
|
||||
+1
-1
@@ -64,7 +64,7 @@ The Cluster.Tests project verifies these key values stay correct (`HoconLoaderTe
|
||||
- `Hostname`: interface to bind. `0.0.0.0` listens on every interface.
|
||||
- `Port`: TCP port for cluster gossip. Default 4053.
|
||||
- `PublicHostname`: address advertised in cluster gossip. Must be reachable by every other node.
|
||||
- `SeedNodes`: where new nodes go to join. List one (or two) stable nodes. First node bootstraps the cluster from its own address.
|
||||
- `SeedNodes`: where new nodes go to join. List one (or two) stable nodes. **Order is load-bearing:** Akka only lets `seed-nodes[0]` form a *new* cluster (`FirstSeedNodeProcess`); every other node retries `InitJoin` forever. A node that is one of its own seeds must therefore list ITSELF first, or it can never cold-start while its peer is down — enforced at boot by `AkkaClusterOptionsValidator`, explained in [Redundancy.md § Bootstrap: self-first seed ordering](../Redundancy.md#bootstrap-self-first-seed-ordering). Nodes seeded only by someone else (driver-only site nodes) are exempt.
|
||||
- `Roles`: free-form tags Akka gossip propagates. v2 uses `admin` + `driver`; per-role wiring in `Program.cs` reads `OTOPCUA_ROLES` env var, not this list — these two should stay in sync.
|
||||
|
||||
Per-role overlay files (`appsettings.admin.json`, `appsettings.driver.json`, `appsettings.admin-driver.json`) layer on top of base `appsettings.json` based on the parsed `OTOPCUA_ROLES` (alphabetical, joined by `-`). See [ServiceHosting.md § Per-role configuration overlays](../ServiceHosting.md#per-role-configuration-overlays).
|
||||
|
||||
@@ -20,7 +20,31 @@ public sealed class AkkaClusterOptions
|
||||
/// </summary>
|
||||
public string PublicHostname { get; set; } = "127.0.0.1";
|
||||
|
||||
/// <summary>Gets or sets the seed nodes for cluster bootstrapping.</summary>
|
||||
/// <summary>
|
||||
/// Gets or sets the seed nodes for cluster bootstrapping.
|
||||
/// </summary>
|
||||
/// <remarks>
|
||||
/// <para>
|
||||
/// <b>ORDER IS LOAD-BEARING (decision 2026-07-22): a node that is one of its own seeds
|
||||
/// must list ITSELF first.</b> Akka runs <c>FirstSeedNodeProcess</c> — the only bootstrap
|
||||
/// path that can form a NEW cluster when no peer answers <c>InitJoin</c> — exclusively
|
||||
/// when <c>seed-nodes[0]</c> is this node's own address; any other node runs
|
||||
/// <c>JoinSeedNodeProcess</c>, which retries <c>InitJoin</c> forever and can never form a
|
||||
/// cluster. So listing both peers does NOT mean either can cold-start alone: a node that
|
||||
/// lists its partner first never comes Up while that partner is down. Self-first ordering
|
||||
/// closes that gap inside Akka's own handshake — unlike the retired
|
||||
/// <c>Cluster:SelfFormAfter</c> watchdog, which sat outside it and could not tell "no
|
||||
/// seed answered" from "a seed answered and the join is in flight".
|
||||
/// </para>
|
||||
/// <para>
|
||||
/// The rule is <b>conditional</b>: it binds only when this node's own address appears in
|
||||
/// this list. A node seeded exclusively by someone else — today's docker-dev site nodes,
|
||||
/// which list only <c>central-1</c> — is legitimately not a seed and is exempt. Enforced
|
||||
/// at boot by <see cref="AkkaClusterOptionsValidator"/>; identity is
|
||||
/// <see cref="PublicHostname"/> + <see cref="Port"/> (what Akka puts in
|
||||
/// <c>SelfAddress</c>), never the <see cref="Hostname"/> bind address.
|
||||
/// </para>
|
||||
/// </remarks>
|
||||
public string[] SeedNodes { get; set; } = Array.Empty<string>();
|
||||
|
||||
/// <summary>
|
||||
@@ -55,19 +79,4 @@ public sealed class AkkaClusterOptions
|
||||
/// </para>
|
||||
/// </remarks>
|
||||
public string SplitBrainResolverStrategy { get; set; } = "auto-down";
|
||||
|
||||
/// <summary>
|
||||
/// Bootstrap self-form fallback window (decision 2026-07-22, scadaproj/akka_failover.md §6.1).
|
||||
/// Akka only lets the FIRST listed seed form a new cluster; a non-first seed cold-starting
|
||||
/// while its peer is down loops on InitJoin forever ("auto-down removes the crash outage,
|
||||
/// not this one" — docs/Redundancy.md). When this node has waited longer than this window
|
||||
/// without cluster membership it joins itself — but ONLY if its own address appears in
|
||||
/// <see cref="SeedNodes"/>; a non-seed node (e.g. a site node whose only seed is central-1)
|
||||
/// stays waiting, because self-forming there creates a permanent island. Default 10s
|
||||
/// (same-datacenter pair; a live peer answers InitJoin in milliseconds). <c>null</c> or a
|
||||
/// non-positive value disables the fallback. Accepted trade: both pair nodes cold-starting
|
||||
/// inside the window while mutually unreachable form two clusters — the same dual-active
|
||||
/// class the auto-down strategy already accepts, same recovery (restart one side).
|
||||
/// </summary>
|
||||
public TimeSpan? SelfFormAfter { get; set; } = TimeSpan.FromSeconds(10);
|
||||
}
|
||||
|
||||
@@ -0,0 +1,96 @@
|
||||
using Akka.Actor;
|
||||
using ZB.MOM.WW.Configuration;
|
||||
|
||||
namespace ZB.MOM.WW.OtOpcUa.Cluster;
|
||||
|
||||
/// <summary>
|
||||
/// Fail-fast startup validator for <see cref="AkkaClusterOptions"/>, built on the shared
|
||||
/// <c>ZB.MOM.WW.Configuration</c> <see cref="OptionsValidatorBase{TOptions}"/> and wired through
|
||||
/// <c>AddValidatedOptions</c> in <see cref="ServiceCollectionExtensions.AddOtOpcUaCluster"/>.
|
||||
/// </summary>
|
||||
/// <remarks>
|
||||
/// <para>
|
||||
/// <b>Self-first seed ordering (decision 2026-07-22).</b> Akka runs
|
||||
/// <c>FirstSeedNodeProcess</c> — the only bootstrap path that can form a NEW cluster when no
|
||||
/// peer answers <c>InitJoin</c> — exclusively when <c>seed-nodes[0]</c> is this node's own
|
||||
/// address. Every other node runs <c>JoinSeedNodeProcess</c> and retries <c>InitJoin</c>
|
||||
/// forever. A node that lists its partner first therefore cannot cold-start while that
|
||||
/// partner is down, and the failure is <i>silent</i>: the process runs, the port listens, the
|
||||
/// node simply never becomes a member. Enforced here so it is loud, at boot, on the node
|
||||
/// that has the problem.
|
||||
/// </para>
|
||||
/// <para>
|
||||
/// <b>The rule is conditional and must stay that way.</b> It binds only when this node's own
|
||||
/// address appears in its own seed list. A driver-only site node is seeded solely by
|
||||
/// <c>central-1</c> and is legitimately not a seed of anything; an unconditional "self must
|
||||
/// be first" rule would refuse to boot every one of them.
|
||||
/// </para>
|
||||
/// <para>
|
||||
/// <b>Identity is the advertised address, never the bind address.</b> In docker-dev
|
||||
/// <c>Cluster:Hostname</c> is <c>0.0.0.0</c> and <c>Cluster:PublicHostname</c> is the
|
||||
/// container DNS name — the value Akka puts in <c>SelfAddress</c> and the only one a seed URI
|
||||
/// can match. Comparing against <c>Hostname</c> would match nothing anywhere, silently exempt
|
||||
/// every node, and leave this rule inert. <c>PublicHostname</c> falling back to
|
||||
/// <c>Hostname</c> when blank mirrors Akka's own <c>public-hostname</c> semantics.
|
||||
/// </para>
|
||||
/// </remarks>
|
||||
public sealed class AkkaClusterOptionsValidator : OptionsValidatorBase<AkkaClusterOptions>
|
||||
{
|
||||
/// <inheritdoc />
|
||||
protected override void Validate(ValidationBuilder builder, AkkaClusterOptions options)
|
||||
{
|
||||
ArgumentNullException.ThrowIfNull(options);
|
||||
|
||||
var seeds = options.SeedNodes ?? Array.Empty<string>();
|
||||
if (seeds.Length == 0)
|
||||
{
|
||||
// Single-node / dev default: nothing is claimed about ordering, so there is nothing to
|
||||
// enforce. (Akka bootstraps from an empty seed list by waiting for an explicit join.)
|
||||
return;
|
||||
}
|
||||
|
||||
var selfHost = string.IsNullOrWhiteSpace(options.PublicHostname)
|
||||
? options.Hostname
|
||||
: options.PublicHostname;
|
||||
|
||||
var selfIndex = Array.FindIndex(seeds, s => IsSelf(s, selfHost, options.Port));
|
||||
if (selfIndex <= 0)
|
||||
{
|
||||
// -1 = this node is not one of its own seeds (exempt); 0 = correctly ordered.
|
||||
return;
|
||||
}
|
||||
|
||||
builder.Add(
|
||||
$"Cluster:SeedNodes must list this node itself first: entry {selfIndex} is this node "
|
||||
+ $"('{seeds[selfIndex]}') but entry 0 is '{seeds[0]}'. Akka only lets seed-nodes[0] form "
|
||||
+ "a new cluster (FirstSeedNodeProcess); every other node retries InitJoin forever, so "
|
||||
+ "with the partner listed first this node can never start while its peer is down. Swap "
|
||||
+ "the entries — see docs/Redundancy.md → 'Bootstrap: self-first seed ordering'.");
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// True when <paramref name="seed"/> addresses this node itself — host AND port.
|
||||
/// </summary>
|
||||
/// <remarks>
|
||||
/// Host comparison is case-insensitive because DNS names are, but is otherwise exact: Akka does
|
||||
/// no DNS canonicalisation either, so <c>central-2</c> and <c>central-2.example.com</c> are
|
||||
/// genuinely different seed identities to the cluster. A seed entry Akka itself cannot parse is
|
||||
/// treated as "not this node" — reporting it is Akka's job, with Akka's wording, and this rule
|
||||
/// must not turn a parse problem into a misleading ordering complaint.
|
||||
/// </remarks>
|
||||
private static bool IsSelf(string seed, string selfHost, int selfPort)
|
||||
{
|
||||
Address address;
|
||||
try
|
||||
{
|
||||
address = Address.Parse(seed);
|
||||
}
|
||||
catch (Exception)
|
||||
{
|
||||
return false;
|
||||
}
|
||||
|
||||
return address.Port == selfPort
|
||||
&& string.Equals(address.Host, selfHost, StringComparison.OrdinalIgnoreCase);
|
||||
}
|
||||
}
|
||||
@@ -1,206 +0,0 @@
|
||||
using Akka.Actor;
|
||||
using Microsoft.Extensions.Logging;
|
||||
|
||||
namespace ZB.MOM.WW.OtOpcUa.Cluster;
|
||||
|
||||
/// <summary>
|
||||
/// InitJoin self-form fallback (decision 2026-07-22, scadaproj/akka_failover.md §6.1).
|
||||
/// Akka only lets the FIRST listed seed form a NEW cluster; every other node retries InitJoin
|
||||
/// forever. So "both nodes are seed nodes" (<see cref="AkkaClusterOptions.SeedNodes"/>) does NOT
|
||||
/// mean either can cold-start alone — a non-first seed booting while its peer is down waits
|
||||
/// indefinitely ("auto-down removes the crash outage, not this one" — <c>docs/Redundancy.md</c>).
|
||||
/// This watchdog waits <see cref="AkkaClusterOptions.SelfFormAfter"/> for membership; on expiry it
|
||||
/// forms a cluster on itself.
|
||||
///
|
||||
/// <para><b>Island safety.</b> Fires ONLY when this node's own address is in its own seed list.
|
||||
/// A node that is not a seed (never legitimately first) must keep waiting: if it self-formed, a
|
||||
/// later-booting real seed would form a second cluster and the two can never merge. That is the
|
||||
/// current docker-dev site-node topology — site-a/site-b nodes list only central-1. For nodes that
|
||||
/// ARE seeds, sequential recovery is island-free — Akka's join protocol prefers an existing cluster
|
||||
/// (a booting node only self-forms when NO seed answers InitJoin), so a peer booting after this
|
||||
/// node self-formed simply joins it.</para>
|
||||
///
|
||||
/// <para><b>Reachability guard (added 2026-07-22 after a live-gate failure).</b> An expired window
|
||||
/// is NOT sufficient evidence that the peer is down, and joining on the timer alone islands nodes.
|
||||
/// Drilled on the docker-dev rig: a node bounced by a manual failover restarted, received
|
||||
/// <c>InitJoinAck</c> from its live peer — a join in flight and healthy — but did not get the
|
||||
/// Welcome inside the window, because the peer's ring still held the node's previous incarnation
|
||||
/// (Exiting → Down → Removed). The fallback fired anyway and the node formed a SECOND cluster,
|
||||
/// islanded until an operator restarted it. So <c>Join(SelfAddress)</c> is <b>not</b> ignored
|
||||
/// mid-handshake — it wins, which is the opposite of what the original design assumed. Before
|
||||
/// self-forming, this therefore TCP-probes the other seed addresses: if any accepts a connection
|
||||
/// the peer is alive, a join is likely already in flight, and the fallback waits another window
|
||||
/// instead. That is also exactly what the fallback claims to detect — "no seed answered InitJoin
|
||||
/// (peer down at boot)".</para>
|
||||
///
|
||||
/// <para><b>Residual risk.</b> Both pair nodes cold-starting inside the window while mutually
|
||||
/// unreachable (a boot-time partition): both self-form, the same dual-active class the auto-down
|
||||
/// downing strategy already accepts, with the same recovery (restart one side).</para>
|
||||
///
|
||||
/// <para>Deliberately duplicated from ScadaBridge's equivalent (like the termination watchdogs) —
|
||||
/// the two repos share the behavior, not a package.</para>
|
||||
/// </summary>
|
||||
public static class ClusterBootstrapFallback
|
||||
{
|
||||
/// <summary>
|
||||
/// Arms the fallback on a freshly created actor system. Safe to call unconditionally: it no-ops
|
||||
/// (with an explanatory log) when the fallback is disabled or when this node is not one of its
|
||||
/// own seeds.
|
||||
/// </summary>
|
||||
/// <param name="system">The actor system whose cluster membership is being watched.</param>
|
||||
/// <param name="options">The bound cluster options carrying the window and the seed list.</param>
|
||||
/// <param name="logger">Logger for the armed / inert / self-forming decisions.</param>
|
||||
/// <param name="peerProbe">
|
||||
/// Overrides the reachability probe (host, port) → reachable. Production passes
|
||||
/// <see langword="null"/> for the real TCP connect; tests substitute it to drive the guard
|
||||
/// deterministically.
|
||||
/// </param>
|
||||
public static void Arm(
|
||||
ActorSystem system,
|
||||
AkkaClusterOptions options,
|
||||
ILogger logger,
|
||||
Func<string, int, Task<bool>>? peerProbe = null)
|
||||
{
|
||||
ArgumentNullException.ThrowIfNull(system);
|
||||
ArgumentNullException.ThrowIfNull(options);
|
||||
ArgumentNullException.ThrowIfNull(logger);
|
||||
|
||||
if (options.SelfFormAfter is not { } window || window <= TimeSpan.Zero)
|
||||
{
|
||||
logger.LogInformation(
|
||||
"Cluster self-form fallback disabled (SelfFormAfter not set) — a node cold-starting "
|
||||
+ "while its peer is down will wait on InitJoin indefinitely.");
|
||||
return;
|
||||
}
|
||||
|
||||
var cluster = Akka.Cluster.Cluster.Get(system);
|
||||
var self = cluster.SelfAddress;
|
||||
var isSeed = options.SeedNodes.Any(s => TryParseAddress(s, out var a) && a.Equals(self));
|
||||
if (!isSeed)
|
||||
{
|
||||
logger.LogInformation(
|
||||
"Cluster self-form fallback inactive: this node ({Self}) is not in its own seed list "
|
||||
+ "[{Seeds}] — self-forming here would island it from the real seeds.",
|
||||
self,
|
||||
string.Join(", ", options.SeedNodes));
|
||||
return;
|
||||
}
|
||||
|
||||
// Every seed that is not this node. These are the addresses whose reachability decides
|
||||
// whether an expired window means "peer is down" or "peer is alive and we are mid-join".
|
||||
var peerSeeds = options.SeedNodes
|
||||
.Select(s => TryParseAddress(s, out var a) ? a : null)
|
||||
.Where(a => a is not null && !a.Equals(self))
|
||||
.Select(a => (Host: a!.Host ?? string.Empty, Port: a!.Port ?? 0))
|
||||
.Where(p => !string.IsNullOrWhiteSpace(p.Host) && p.Port > 0)
|
||||
.ToList();
|
||||
|
||||
var probe = peerProbe ?? TryConnectAsync;
|
||||
|
||||
var joined = new TaskCompletionSource(TaskCreationOptions.RunContinuationsAsynchronously);
|
||||
cluster.RegisterOnMemberUp(() => joined.TrySetResult());
|
||||
|
||||
_ = Task.Run(async () =>
|
||||
{
|
||||
while (true)
|
||||
{
|
||||
var winner = await Task.WhenAny(joined.Task, Task.Delay(window));
|
||||
if (winner == joined.Task || system.WhenTerminated.IsCompleted)
|
||||
{
|
||||
return;
|
||||
}
|
||||
|
||||
var reachable = await AnyReachableAsync(peerSeeds, probe);
|
||||
if (reachable is not null)
|
||||
{
|
||||
// Do NOT self-form. A reachable peer means either a join already in flight (the
|
||||
// failure this guard exists for) or a peer about to answer — self-forming here
|
||||
// creates a second cluster that can never merge.
|
||||
logger.LogInformation(
|
||||
"Self-form window ({Window}) expired, but seed peer {Peer} is reachable — a join is "
|
||||
+ "most likely already in flight (a node re-joining after a restart is not admitted "
|
||||
+ "until its previous incarnation is removed). Continuing to wait rather than "
|
||||
+ "forming a second cluster.",
|
||||
window,
|
||||
reachable);
|
||||
continue;
|
||||
}
|
||||
|
||||
logger.LogWarning(
|
||||
"No cluster membership after {Window} and no seed peer is reachable — the peer is down "
|
||||
+ "at boot. Self-forming a cluster at {Self} so this node becomes operational; if the "
|
||||
+ "peer was merely partitioned (not dead), the pair is now dual-active — restart one "
|
||||
+ "side after the partition heals (accepted availability-first trade, decision "
|
||||
+ "2026-07-22).",
|
||||
window,
|
||||
self);
|
||||
cluster.Join(self);
|
||||
return;
|
||||
}
|
||||
});
|
||||
}
|
||||
|
||||
/// <summary>How long a single peer-reachability connect attempt may take.</summary>
|
||||
public static readonly TimeSpan ProbeTimeout = TimeSpan.FromSeconds(2);
|
||||
|
||||
/// <summary>
|
||||
/// Returns the first reachable peer as <c>host:port</c>, or <see langword="null"/> when none
|
||||
/// answered — including when there are no peer seeds at all (a lone-seed config, where an
|
||||
/// expired window really does mean this node is on its own).
|
||||
/// </summary>
|
||||
private static async Task<string?> AnyReachableAsync(
|
||||
IReadOnlyList<(string Host, int Port)> peers,
|
||||
Func<string, int, Task<bool>> probe)
|
||||
{
|
||||
foreach (var (host, port) in peers)
|
||||
{
|
||||
bool ok;
|
||||
try
|
||||
{
|
||||
ok = await probe(host, port);
|
||||
}
|
||||
catch (Exception)
|
||||
{
|
||||
// A probe that cannot even be attempted (DNS gone, socket exhaustion) is treated as
|
||||
// unreachable — it must not throw out of the watchdog and disarm the fallback.
|
||||
ok = false;
|
||||
}
|
||||
|
||||
if (ok) return $"{host}:{port}";
|
||||
}
|
||||
|
||||
return null;
|
||||
}
|
||||
|
||||
private static async Task<bool> TryConnectAsync(string host, int port)
|
||||
{
|
||||
using var client = new System.Net.Sockets.TcpClient();
|
||||
using var cts = new CancellationTokenSource(ProbeTimeout);
|
||||
try
|
||||
{
|
||||
await client.ConnectAsync(host, port, cts.Token);
|
||||
return client.Connected;
|
||||
}
|
||||
catch (Exception)
|
||||
{
|
||||
// Refused, unresolvable, or timed out — all mean "not reachable right now".
|
||||
return false;
|
||||
}
|
||||
}
|
||||
|
||||
private static bool TryParseAddress(string seed, out Address address)
|
||||
{
|
||||
try
|
||||
{
|
||||
address = Address.Parse(seed);
|
||||
return true;
|
||||
}
|
||||
catch (Exception)
|
||||
{
|
||||
// A malformed seed entry must not disarm the fallback; bad seed URIs surface from
|
||||
// Akka's own bootstrap.
|
||||
address = Address.AllSystems;
|
||||
return false;
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -7,6 +7,7 @@ using Akka.Remote.Hosting;
|
||||
using Microsoft.Extensions.Configuration;
|
||||
using Microsoft.Extensions.DependencyInjection;
|
||||
using Microsoft.Extensions.Options;
|
||||
using ZB.MOM.WW.Configuration;
|
||||
using ZB.MOM.WW.OtOpcUa.Commons.Interfaces;
|
||||
|
||||
namespace ZB.MOM.WW.OtOpcUa.Cluster;
|
||||
@@ -19,13 +20,19 @@ public static class ServiceCollectionExtensions
|
||||
/// configurator via <see cref="WithOtOpcUaClusterBootstrap"/> — keeping the entire Akka graph
|
||||
/// under Akka.Hosting's management so cluster singletons land on the same ActorSystem.
|
||||
/// </summary>
|
||||
/// <remarks>
|
||||
/// The binding is validated at startup (<c>ValidateOnStart</c>) by
|
||||
/// <see cref="AkkaClusterOptionsValidator"/>, so a seed list that cannot bootstrap this node —
|
||||
/// self listed behind its partner — fails the host loudly instead of leaving it running but
|
||||
/// permanently outside the cluster.
|
||||
/// </remarks>
|
||||
/// <param name="services">The service collection to configure.</param>
|
||||
/// <param name="configuration">The application configuration containing cluster options.</param>
|
||||
/// <returns>The same service collection, for chaining.</returns>
|
||||
public static IServiceCollection AddOtOpcUaCluster(this IServiceCollection services, IConfiguration configuration)
|
||||
{
|
||||
services.AddOptions<AkkaClusterOptions>()
|
||||
.Bind(configuration.GetSection(AkkaClusterOptions.SectionName));
|
||||
services.AddValidatedOptions<AkkaClusterOptions, AkkaClusterOptionsValidator>(
|
||||
configuration, AkkaClusterOptions.SectionName);
|
||||
|
||||
services.AddSingleton<IClusterRoleInfo, ClusterRoleInfo>();
|
||||
|
||||
@@ -92,22 +99,13 @@ public static class ServiceCollectionExtensions
|
||||
// ActorSystem for exactly that reason: the effective value is the only one worth asserting.
|
||||
builder.AddHocon(BuildDowningHocon(options), HoconAddMode.Prepend);
|
||||
|
||||
// InitJoin self-form fallback (decision 2026-07-22): registered as an Akka.Hosting startup
|
||||
// task so every host built through this bootstrap — production AND the test hosts in
|
||||
// Cluster.Tests — arms it identically. Guarded inside Arm: disabled when SelfFormAfter is
|
||||
// unset; inert when this node is not in its own seed list (site nodes seeded only by
|
||||
// central-1 must wait, not island).
|
||||
// ILoggerFactory is fully qualified: `using Microsoft.Extensions.Logging` would make the
|
||||
// LogLevel in ConfigureLoggers above ambiguous with Akka.Event.LogLevel.
|
||||
var fallbackLogger =
|
||||
(serviceProvider.GetService<Microsoft.Extensions.Logging.ILoggerFactory>()
|
||||
?? Microsoft.Extensions.Logging.Abstractions.NullLoggerFactory.Instance)
|
||||
.CreateLogger("ZB.MOM.WW.OtOpcUa.Cluster.ClusterBootstrapFallback");
|
||||
builder.AddStartup((system, _) =>
|
||||
{
|
||||
ClusterBootstrapFallback.Arm(system, options, fallbackLogger);
|
||||
return Task.CompletedTask;
|
||||
});
|
||||
// NOTE (2026-07-22): no bootstrap watchdog is armed here any more. The cold-start-alone gap
|
||||
// it existed for — Akka lets only seed-nodes[0] form a NEW cluster — is now closed by
|
||||
// self-first seed ordering (AkkaClusterOptions.SeedNodes), enforced at boot by
|
||||
// AkkaClusterOptionsValidator. The retired ClusterBootstrapFallback timer sat OUTSIDE Akka's
|
||||
// join handshake, so it could not distinguish "no seed answered" from "a seed answered and
|
||||
// the join is in flight", and Cluster.Join(SelfAddress) is not ignored mid-handshake — it
|
||||
// wins. See docs/Redundancy.md → "Bootstrap: self-first seed ordering".
|
||||
|
||||
return builder;
|
||||
}
|
||||
|
||||
@@ -14,6 +14,10 @@
|
||||
<PackageReference Include="Akka.Remote.Hosting"/>
|
||||
<PackageReference Include="Microsoft.Extensions.Hosting"/>
|
||||
<PackageReference Include="Microsoft.Extensions.Options.ConfigurationExtensions"/>
|
||||
<!-- Shared options-validation primitives (OptionsValidatorBase / AddValidatedOptions), the
|
||||
same seam the Host's Ldap/OpcUa/Historian validators use. Pulled in here because the
|
||||
self-first seed-ordering rule belongs with the options it validates. -->
|
||||
<PackageReference Include="ZB.MOM.WW.Configuration"/>
|
||||
</ItemGroup>
|
||||
|
||||
<ItemGroup>
|
||||
|
||||
@@ -0,0 +1,177 @@
|
||||
using Shouldly;
|
||||
using Xunit;
|
||||
|
||||
namespace ZB.MOM.WW.OtOpcUa.Cluster.Tests;
|
||||
|
||||
/// <summary>
|
||||
/// Guards the self-first seed-ordering rule at the only moment it can still be fixed cheaply: boot.
|
||||
/// </summary>
|
||||
/// <remarks>
|
||||
/// <para>
|
||||
/// The invariant (decision 2026-07-22) is <b>conditional</b>: <i>if</i> a node's own address
|
||||
/// appears in its own <see cref="AkkaClusterOptions.SeedNodes"/>, it must be entry 0. Akka
|
||||
/// runs <c>FirstSeedNodeProcess</c> — the only path that can form a NEW cluster when no peer
|
||||
/// answers <c>InitJoin</c> — exclusively for <c>seed-nodes[0]</c>; every other node retries
|
||||
/// <c>InitJoin</c> forever. A node listed second therefore cannot cold-start while its peer
|
||||
/// is down, and it fails <i>silently</i>: the process is healthy, the port is open, the node
|
||||
/// simply never becomes a cluster member. That is why the rule is enforced loudly here.
|
||||
/// </para>
|
||||
/// <para>
|
||||
/// The conditional form is required, not incidental: a flat "self must be first" rule would
|
||||
/// reject every driver-only site node, which is seeded solely by <c>central-1</c> and is
|
||||
/// legitimately not a seed of anything —
|
||||
/// see <see cref="Node_absent_from_its_own_seed_list_is_exempt"/>.
|
||||
/// </para>
|
||||
/// </remarks>
|
||||
public sealed class AkkaClusterOptionsValidatorTests
|
||||
{
|
||||
private static AkkaClusterOptions CentralNode(string publicHostname, params string[] seeds) =>
|
||||
new()
|
||||
{
|
||||
// The docker-dev shape: bind to every interface, advertise the container DNS name.
|
||||
Hostname = "0.0.0.0",
|
||||
PublicHostname = publicHostname,
|
||||
Port = 4053,
|
||||
Roles = new[] { "admin", "driver" },
|
||||
SeedNodes = seeds,
|
||||
};
|
||||
|
||||
private static string Seed(string host, int port = 4053) => $"akka.tcp://otopcua@{host}:{port}";
|
||||
|
||||
/// <summary>The shipped central-1 / central-2 shape: own address first, partner second.</summary>
|
||||
[Fact]
|
||||
public void Self_first_seed_order_passes()
|
||||
{
|
||||
var options = CentralNode("central-2", Seed("central-2"), Seed("central-1"));
|
||||
|
||||
var result = new AkkaClusterOptionsValidator().Validate(null, options);
|
||||
|
||||
result.Failed.ShouldBeFalse(result.FailureMessage);
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// The pre-2026-07-22 ordering — partner first — is the boot-alone outage gap, and must not
|
||||
/// start. This is the shape <c>docker-dev</c>'s <c>central-2</c> carried.
|
||||
/// </summary>
|
||||
[Fact]
|
||||
public void Peer_first_seed_order_fails()
|
||||
{
|
||||
var options = CentralNode("central-2", Seed("central-1"), Seed("central-2"));
|
||||
|
||||
var result = new AkkaClusterOptionsValidator().Validate(null, options);
|
||||
|
||||
result.Failed.ShouldBeTrue();
|
||||
result.FailureMessage.ShouldNotBeNull().ShouldContain("SeedNodes");
|
||||
result.FailureMessage.ShouldContain("must list this node itself first");
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// THE conditional exemption. A driver-only site node lists only <c>central-1</c> — it is not a
|
||||
/// seed at all, is never legitimately first, and must pass. An unconditional "self must be
|
||||
/// first" rule would refuse to boot every site node in the fleet.
|
||||
/// </summary>
|
||||
[Fact]
|
||||
public void Node_absent_from_its_own_seed_list_is_exempt()
|
||||
{
|
||||
var options = CentralNode("site-a-1", Seed("central-1"));
|
||||
|
||||
var result = new AkkaClusterOptionsValidator().Validate(null, options);
|
||||
|
||||
result.Failed.ShouldBeFalse(result.FailureMessage);
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// The bind address is NOT the node's identity. In docker-dev <c>Cluster:Hostname</c> is
|
||||
/// <c>0.0.0.0</c> and <c>Cluster:PublicHostname</c> is the container name — the value Akka puts
|
||||
/// in <c>SelfAddress</c> and therefore the only one a seed entry can match. A validator that
|
||||
/// compared against <c>Hostname</c> would find no match anywhere, silently exempt every node,
|
||||
/// and pass this misordered config.
|
||||
/// </summary>
|
||||
[Fact]
|
||||
public void Identity_is_the_public_hostname_not_the_bind_address()
|
||||
{
|
||||
var options = CentralNode("central-2", Seed("central-1"), Seed("central-2"));
|
||||
options.Hostname.ShouldBe("0.0.0.0", "the trap only exists when the bind address is a wildcard");
|
||||
|
||||
var result = new AkkaClusterOptionsValidator().Validate(null, options);
|
||||
|
||||
result.Failed.ShouldBeTrue("matching on the 0.0.0.0 bind address would make this rule vacuous");
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// With <c>PublicHostname</c> blank Akka advertises the bind hostname, so that is the identity to
|
||||
/// match — otherwise a loopback/bare-metal pair configured that way would be silently exempt.
|
||||
/// </summary>
|
||||
[Fact]
|
||||
public void Identity_falls_back_to_the_bind_hostname_when_no_public_hostname_is_set()
|
||||
{
|
||||
var options = CentralNode("central-2", Seed("node-a"), Seed("node-b"));
|
||||
options.Hostname = "node-b";
|
||||
options.PublicHostname = string.Empty;
|
||||
|
||||
var result = new AkkaClusterOptionsValidator().Validate(null, options);
|
||||
|
||||
result.Failed.ShouldBeTrue();
|
||||
result.FailureMessage.ShouldNotBeNull().ShouldContain("must list this node itself first");
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Host AND port: a two-node install sharing one hostname and differing only by port (a
|
||||
/// loopback/dev pair) is a real topology, and matching on host alone would clear the misordered
|
||||
/// node.
|
||||
/// </summary>
|
||||
[Fact]
|
||||
public void Self_match_compares_host_and_port()
|
||||
{
|
||||
var options = CentralNode("127.0.0.1", Seed("127.0.0.1", 4053), Seed("127.0.0.1", 4054));
|
||||
options.Port = 4054;
|
||||
|
||||
var result = new AkkaClusterOptionsValidator().Validate(null, options);
|
||||
|
||||
result.Failed.ShouldBeTrue("this node is 127.0.0.1:4054, which is the SECOND seed");
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Positive control for the pair above: the same host on the same port really is a match, so the
|
||||
/// port comparison cannot be passing merely by never matching anything.
|
||||
/// </summary>
|
||||
[Fact]
|
||||
public void Self_match_on_the_same_host_and_port_passes()
|
||||
{
|
||||
var options = CentralNode("127.0.0.1", Seed("127.0.0.1", 4054), Seed("127.0.0.1", 4053));
|
||||
options.Port = 4054;
|
||||
|
||||
var result = new AkkaClusterOptionsValidator().Validate(null, options);
|
||||
|
||||
result.Failed.ShouldBeFalse(result.FailureMessage);
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// An empty seed list is the single-node/dev default and says nothing about ordering — the rule
|
||||
/// has nothing to bind to and must not invent a failure.
|
||||
/// </summary>
|
||||
[Fact]
|
||||
public void Empty_seed_list_passes()
|
||||
{
|
||||
var options = CentralNode("central-1");
|
||||
|
||||
var result = new AkkaClusterOptionsValidator().Validate(null, options);
|
||||
|
||||
result.Failed.ShouldBeFalse(result.FailureMessage);
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// A malformed seed URI is Akka's error to report, with Akka's wording. This rule must not
|
||||
/// crash on it, and must not turn a parse problem into a misleading "ordering" complaint —
|
||||
/// here the ordering is correct and only a later entry is unparseable.
|
||||
/// </summary>
|
||||
[Fact]
|
||||
public void Malformed_seed_entry_does_not_throw()
|
||||
{
|
||||
var options = CentralNode("central-1", Seed("central-1"), "not-an-akka-address");
|
||||
|
||||
var result = new AkkaClusterOptionsValidator().Validate(null, options);
|
||||
|
||||
result.Failed.ShouldBeFalse(result.FailureMessage);
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,287 @@
|
||||
using System.Net;
|
||||
using System.Net.Sockets;
|
||||
using Akka.Actor;
|
||||
using Akka.Cluster;
|
||||
using Akka.Hosting;
|
||||
using Microsoft.Extensions.DependencyInjection;
|
||||
using Microsoft.Extensions.Hosting;
|
||||
using Microsoft.Extensions.Options;
|
||||
using Shouldly;
|
||||
using Xunit;
|
||||
|
||||
namespace ZB.MOM.WW.OtOpcUa.Cluster.Tests;
|
||||
|
||||
/// <summary>
|
||||
/// Guards the self-first seed-ordering invariant (decision 2026-07-22): every node that is a seed
|
||||
/// of its own mesh lists ITSELF as <c>SeedNodes[0]</c> and its partner second.
|
||||
/// </summary>
|
||||
/// <remarks>
|
||||
/// <para>
|
||||
/// <b>Why the ordering is the mechanism.</b> Akka runs a different bootstrap process
|
||||
/// depending on whether <c>seed-nodes[0]</c> is this node's own address. When it is,
|
||||
/// <c>FirstSeedNodeProcess</c> runs: it <c>InitJoin</c>s the other seeds and self-joins once
|
||||
/// <c>seed-node-timeout</c> (5 s) passes with nobody answering. When it is not,
|
||||
/// <c>JoinSeedNodeProcess</c> runs, and that process can never form a new cluster — it
|
||||
/// retries <c>InitJoin</c> forever. That is why "both peers are listed in SeedNodes" never
|
||||
/// meant "either can cold-start alone", and why a node listed second stayed out of the
|
||||
/// cluster indefinitely while its peer was down.
|
||||
/// </para>
|
||||
/// <para>
|
||||
/// <b>Why not the self-form watchdog this replaces.</b> <c>ClusterBootstrapFallback</c>
|
||||
/// waited <c>Cluster:SelfFormAfter</c> for membership and then called
|
||||
/// <c>Cluster.Join(SelfAddress)</c>. A timer outside Akka's join handshake cannot
|
||||
/// distinguish "no seed answered" from "a seed answered and the join is in flight", and
|
||||
/// <c>Join(SelfAddress)</c> is <b>not</b> ignored mid-handshake — it wins. The live gate on
|
||||
/// the docker-dev rig (2026-07-22) caught exactly that: a node bounced by a manual failover
|
||||
/// restarted, got <c>InitJoinAck</c> from its live peer, but no Welcome inside the window
|
||||
/// because the peer's ring still held its previous incarnation
|
||||
/// (<c>Exiting</c> → <c>Down</c> → <c>Removed</c>); the watchdog fired and formed a SECOND
|
||||
/// cluster. A TCP reachability guard patched that one shape; the race stayed.
|
||||
/// <see cref="Restarting_node_rejoins_its_live_peer_instead_of_islanding"/> is that scenario,
|
||||
/// and self-first ordering has no such race because the decision is made inside the
|
||||
/// handshake by Akka itself.
|
||||
/// </para>
|
||||
/// <para>
|
||||
/// <b>Why these start real hosts through the production bootstrap.</b> Same reason as
|
||||
/// <see cref="SplitBrainResolverActivationTests"/>: the question is what a running node does.
|
||||
/// Every node here is built with
|
||||
/// <see cref="ServiceCollectionExtensions.WithOtOpcUaClusterBootstrap"/> — the shipped entry
|
||||
/// point — so these run at the production failure-detection envelope (heartbeat 2 s,
|
||||
/// acceptable pause 10 s, auto-down 15 s, down-removal-margin 15 s) from
|
||||
/// <c>Resources/akka.conf</c> rather than at timings invented by the test.
|
||||
/// </para>
|
||||
/// </remarks>
|
||||
public sealed class SelfFirstSeedBootstrapTests
|
||||
{
|
||||
private static int FreePort()
|
||||
{
|
||||
var listener = new TcpListener(IPAddress.Loopback, 0);
|
||||
listener.Start();
|
||||
var port = ((IPEndPoint)listener.LocalEndpoint).Port;
|
||||
listener.Stop();
|
||||
return port;
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Starts a node the way every shipped node config is written: its own address first, the
|
||||
/// partner second.
|
||||
/// </summary>
|
||||
/// <param name="systemName">Actor-system name; also the system part of every seed address.</param>
|
||||
/// <param name="selfPort">This node's remoting port.</param>
|
||||
/// <param name="peerPort">The partner's remoting port (may have nothing listening on it).</param>
|
||||
/// <param name="selfFirst">
|
||||
/// <see langword="false"/> reproduces the pre-2026-07-22 ordering (partner first), which is what
|
||||
/// makes <see cref="Peer_first_ordering_is_the_outage_gap_and_never_forms"/> a falsifiability
|
||||
/// control rather than a restatement of the fix.
|
||||
/// </param>
|
||||
private static async Task<IHost> StartNodeAsync(
|
||||
string systemName,
|
||||
int selfPort,
|
||||
int peerPort,
|
||||
bool selfFirst = true)
|
||||
{
|
||||
var self = $"akka.tcp://{systemName}@127.0.0.1:{selfPort}";
|
||||
var peer = $"akka.tcp://{systemName}@127.0.0.1:{peerPort}";
|
||||
var options = new AkkaClusterOptions
|
||||
{
|
||||
SystemName = systemName,
|
||||
Hostname = "127.0.0.1",
|
||||
PublicHostname = "127.0.0.1",
|
||||
Port = selfPort,
|
||||
Roles = new[] { "admin", "driver" },
|
||||
SeedNodes = selfFirst ? new[] { self, peer } : new[] { peer, self },
|
||||
};
|
||||
|
||||
var builder = Host.CreateDefaultBuilder();
|
||||
builder.ConfigureServices(services =>
|
||||
{
|
||||
services.AddSingleton<IOptions<AkkaClusterOptions>>(Options.Create(options));
|
||||
services.AddAkka(systemName, (ab, sp) =>
|
||||
{
|
||||
ab.WithOtOpcUaClusterBootstrap(sp);
|
||||
|
||||
// Crash simulation, test-only: with this off, ActorSystem.Terminate() skips
|
||||
// CoordinatedShutdown and so gossips no Leave — a process death, not a graceful
|
||||
// drain. Without it "restart the standby" would degenerate into the easy path where
|
||||
// the peer had already removed the old member before the new one dialled in.
|
||||
// Prepend is Akka.Hosting's highest-precedence merge mode (see BuildDowningHocon).
|
||||
ab.AddHocon(
|
||||
"akka.coordinated-shutdown.run-by-actor-system-terminate = off",
|
||||
HoconAddMode.Prepend);
|
||||
});
|
||||
});
|
||||
|
||||
var host = builder.Build();
|
||||
await host.StartAsync();
|
||||
return host;
|
||||
}
|
||||
|
||||
/// <summary>Polls until the node sees <paramref name="expected"/> Up members, or the deadline passes.</summary>
|
||||
private static async Task<bool> WaitForUpMembersAsync(IHost host, int expected, TimeSpan timeout)
|
||||
{
|
||||
var cluster = Akka.Cluster.Cluster.Get(host.Services.GetRequiredService<ActorSystem>());
|
||||
var deadline = DateTime.UtcNow + timeout;
|
||||
while (DateTime.UtcNow < deadline)
|
||||
{
|
||||
if (cluster.State.Members.Count(m => m.Status == MemberStatus.Up) >= expected)
|
||||
{
|
||||
return true;
|
||||
}
|
||||
|
||||
await Task.Delay(200);
|
||||
}
|
||||
|
||||
return cluster.State.Members.Count(m => m.Status == MemberStatus.Up) >= expected;
|
||||
}
|
||||
|
||||
private static async Task StopAsync(IHost host)
|
||||
{
|
||||
try
|
||||
{
|
||||
await host.StopAsync();
|
||||
}
|
||||
catch (Exception)
|
||||
{
|
||||
// Teardown only. A host whose ActorSystem was terminated by a crash-simulation cannot
|
||||
// run CoordinatedShutdown again; that must not fail the test that already asserted.
|
||||
}
|
||||
|
||||
host.Dispose();
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// The unattended cold-start-alone guarantee: a node whose partner is dead forms the cluster by
|
||||
/// itself, through Akka's own <c>FirstSeedNodeProcess</c> — no watchdog, no timer, no operator.
|
||||
/// </summary>
|
||||
[Fact]
|
||||
public async Task Lone_cold_start_forms_a_cluster_when_the_peer_is_dead()
|
||||
{
|
||||
var selfPort = FreePort();
|
||||
var deadPeerPort = FreePort(); // nothing is listening there
|
||||
var host = await StartNodeAsync("otopcua-selffirst-1", selfPort, deadPeerPort);
|
||||
try
|
||||
{
|
||||
// seed-node-timeout is Akka's 5 s default; 30 s is slack for a loaded CI box.
|
||||
(await WaitForUpMembersAsync(host, 1, TimeSpan.FromSeconds(30)))
|
||||
.ShouldBeTrue("a self-first seed must form a cluster alone when no peer answers InitJoin");
|
||||
}
|
||||
finally
|
||||
{
|
||||
await StopAsync(host);
|
||||
}
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// FALSIFIABILITY CONTROL for the test above. With the OLD ordering — partner first — the
|
||||
/// identical scenario never comes Up, because <c>JoinSeedNodeProcess</c> has no self-join path.
|
||||
/// If this test ever starts passing quickly, the seed ordering has stopped being the thing doing
|
||||
/// the work and the suite above has gone vacuous.
|
||||
/// </summary>
|
||||
/// <remarks>
|
||||
/// This is also the test that fails while the retired <c>ClusterBootstrapFallback</c> watchdog
|
||||
/// is still armed: that timer would bring this node Up at <c>SelfFormAfter</c> (10 s) despite the
|
||||
/// ordering, which is precisely the "something other than Akka's handshake is deciding" that the
|
||||
/// retirement removes.
|
||||
/// </remarks>
|
||||
[Fact]
|
||||
public async Task Peer_first_ordering_is_the_outage_gap_and_never_forms()
|
||||
{
|
||||
var selfPort = FreePort();
|
||||
var deadPeerPort = FreePort();
|
||||
var host = await StartNodeAsync("otopcua-selffirst-2", selfPort, deadPeerPort, selfFirst: false);
|
||||
try
|
||||
{
|
||||
// 3x the 5 s seed-node-timeout, and longer than the retired watchdog's 10 s window.
|
||||
(await WaitForUpMembersAsync(host, 1, TimeSpan.FromSeconds(15)))
|
||||
.ShouldBeFalse("a node listed behind its peer cannot form a cluster — it InitJoin-loops forever");
|
||||
|
||||
var cluster = Akka.Cluster.Cluster.Get(host.Services.GetRequiredService<ActorSystem>());
|
||||
cluster.State.Members.ShouldBeEmpty();
|
||||
|
||||
// Positive control: the node was formable all along; only the ordering blocked it.
|
||||
cluster.Join(cluster.SelfAddress);
|
||||
(await WaitForUpMembersAsync(host, 1, TimeSpan.FromSeconds(30)))
|
||||
.ShouldBeTrue("positive control — an explicit self-join must bring this node Up");
|
||||
}
|
||||
finally
|
||||
{
|
||||
await StopAsync(host);
|
||||
}
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// The case that killed the self-form watchdog (live gate, docker-dev, 2026-07-22): a routine
|
||||
/// standby restart while the peer is alive must REJOIN, never island. Under self-first ordering
|
||||
/// the restarted node's <c>FirstSeedNodeProcess</c> only self-joins when no seed answers, so the
|
||||
/// live peer's <c>InitJoinAck</c> keeps it on the join path however long the peer takes to
|
||||
/// retire its previous incarnation.
|
||||
/// </summary>
|
||||
[Fact]
|
||||
public async Task Restarting_node_rejoins_its_live_peer_instead_of_islanding()
|
||||
{
|
||||
const string systemName = "otopcua-selffirst-3";
|
||||
var portA = FreePort();
|
||||
var portB = FreePort();
|
||||
|
||||
var nodeA = await StartNodeAsync(systemName, portA, portB);
|
||||
IHost? nodeB = null;
|
||||
IHost? nodeB2 = null;
|
||||
try
|
||||
{
|
||||
(await WaitForUpMembersAsync(nodeA, 1, TimeSpan.FromSeconds(30))).ShouldBeTrue();
|
||||
|
||||
nodeB = await StartNodeAsync(systemName, portB, portA);
|
||||
(await WaitForUpMembersAsync(nodeA, 2, TimeSpan.FromSeconds(60)))
|
||||
.ShouldBeTrue("the pair must converge before the restart is meaningful");
|
||||
|
||||
// Hard-crash B and immediately restart it at the SAME address, as a service restart or
|
||||
// container recreate does. No Leave gossip: A still holds B's previous incarnation.
|
||||
await nodeB.Services.GetRequiredService<ActorSystem>().Terminate()
|
||||
.WaitAsync(TimeSpan.FromSeconds(30), TestContext.Current.CancellationToken);
|
||||
nodeB2 = await StartNodeAsync(systemName, portB, portA);
|
||||
|
||||
// ONE cluster of two, never two clusters of one. The budget covers auto-down (15 s) plus
|
||||
// down-removal-margin (15 s) before the new incarnation is admitted.
|
||||
(await WaitForUpMembersAsync(nodeB2, 2, TimeSpan.FromSeconds(120)))
|
||||
.ShouldBeTrue("the restarted node must rejoin its live peer, not form a second cluster");
|
||||
(await WaitForUpMembersAsync(nodeA, 2, TimeSpan.FromSeconds(120)))
|
||||
.ShouldBeTrue("the surviving node must see the restarted peer as a member of ITS cluster");
|
||||
}
|
||||
finally
|
||||
{
|
||||
if (nodeB2 is not null) await StopAsync(nodeB2);
|
||||
if (nodeB is not null) await StopAsync(nodeB);
|
||||
await StopAsync(nodeA);
|
||||
}
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// The obvious objection to self-first on BOTH nodes: does a simultaneous cold start produce two
|
||||
/// clusters? While the two are mutually reachable it does not — each runs
|
||||
/// <c>FirstSeedNodeProcess</c>, and the <c>InitJoin</c> handshake resolves which one forms before
|
||||
/// either self-join deadline expires. (A genuine boot-time PARTITION would still split, the same
|
||||
/// dual-active class the <c>auto-down</c> strategy already accepts, with the same recovery.)
|
||||
/// </summary>
|
||||
[Fact]
|
||||
public async Task Both_nodes_cold_starting_together_converge_on_one_cluster()
|
||||
{
|
||||
const string systemName = "otopcua-selffirst-4";
|
||||
var portA = FreePort();
|
||||
var portB = FreePort();
|
||||
|
||||
var nodeA = await StartNodeAsync(systemName, portA, portB);
|
||||
var nodeB = await StartNodeAsync(systemName, portB, portA);
|
||||
try
|
||||
{
|
||||
(await WaitForUpMembersAsync(nodeA, 2, TimeSpan.FromSeconds(60)))
|
||||
.ShouldBeTrue("both nodes cold-starting self-first must converge, not split");
|
||||
(await WaitForUpMembersAsync(nodeB, 2, TimeSpan.FromSeconds(60)))
|
||||
.ShouldBeTrue("both nodes cold-starting self-first must converge, not split");
|
||||
}
|
||||
finally
|
||||
{
|
||||
await StopAsync(nodeB);
|
||||
await StopAsync(nodeA);
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -1,259 +0,0 @@
|
||||
using System.Net;
|
||||
using System.Net.Sockets;
|
||||
using Akka.Actor;
|
||||
using Akka.Cluster;
|
||||
using Akka.Hosting;
|
||||
using Microsoft.Extensions.DependencyInjection;
|
||||
using Microsoft.Extensions.Hosting;
|
||||
using Microsoft.Extensions.Options;
|
||||
using Shouldly;
|
||||
using Xunit;
|
||||
|
||||
namespace ZB.MOM.WW.OtOpcUa.Cluster.Tests;
|
||||
|
||||
/// <summary>
|
||||
/// Guards the InitJoin self-form fallback — whether a node whose peer is down at boot ever becomes
|
||||
/// operational.
|
||||
/// </summary>
|
||||
/// <remarks>
|
||||
/// <para>
|
||||
/// <b>Why these tests start real hosts through the production bootstrap.</b> Like
|
||||
/// <see cref="SplitBrainResolverActivationTests"/>, the question is what a running node
|
||||
/// actually does, not what an options object holds. Every host here is built with
|
||||
/// <see cref="ServiceCollectionExtensions.WithOtOpcUaClusterBootstrap"/> — the shipped entry
|
||||
/// point — so a change that stops arming the fallback (or arms it from a place production
|
||||
/// does not reach) turns these red. A test that called
|
||||
/// <see cref="ClusterBootstrapFallback.Arm"/> itself would pin the test's wiring instead.
|
||||
/// </para>
|
||||
/// <para>
|
||||
/// <b>Seed ordering is load-bearing.</b> The node under test lists itself SECOND in its own
|
||||
/// seed list, behind a dead peer port. Akka only lets the FIRST listed seed self-join, so
|
||||
/// nothing but the fallback can bring these nodes Up — which is what makes the negative
|
||||
/// scenarios (disabled window, non-seed node) meaningful rather than vacuous. Each of those
|
||||
/// carries a positive control (an explicit <c>Cluster.Join(SelfAddress)</c>) proving the node
|
||||
/// was formable all along and only the fallback was absent.
|
||||
/// </para>
|
||||
/// </remarks>
|
||||
public sealed class SelfFormBootstrapTests
|
||||
{
|
||||
private static int FreePort()
|
||||
{
|
||||
var listener = new TcpListener(IPAddress.Loopback, 0);
|
||||
listener.Start();
|
||||
var port = ((IPEndPoint)listener.LocalEndpoint).Port;
|
||||
listener.Stop();
|
||||
return port;
|
||||
}
|
||||
|
||||
private static async Task<IHost> StartNodeAsync(
|
||||
string systemName,
|
||||
int selfPort,
|
||||
int peerPort,
|
||||
TimeSpan? selfFormAfter,
|
||||
bool selfInSeeds = true)
|
||||
{
|
||||
var options = new AkkaClusterOptions
|
||||
{
|
||||
SystemName = systemName,
|
||||
Hostname = "127.0.0.1",
|
||||
PublicHostname = "127.0.0.1",
|
||||
Port = selfPort,
|
||||
Roles = new[] { "admin", "driver" },
|
||||
SelfFormAfter = selfFormAfter,
|
||||
|
||||
// Self listed SECOND, behind a port nothing is listening on: Akka lets only the first
|
||||
// seed self-join, so this node can never form a cluster on its own except via the
|
||||
// fallback.
|
||||
SeedNodes = selfInSeeds
|
||||
? new[]
|
||||
{
|
||||
$"akka.tcp://{systemName}@127.0.0.1:{peerPort}",
|
||||
$"akka.tcp://{systemName}@127.0.0.1:{selfPort}",
|
||||
}
|
||||
: new[] { $"akka.tcp://{systemName}@127.0.0.1:{peerPort}" },
|
||||
};
|
||||
|
||||
var builder = Host.CreateDefaultBuilder();
|
||||
builder.ConfigureServices(services =>
|
||||
{
|
||||
services.AddSingleton<IOptions<AkkaClusterOptions>>(Options.Create(options));
|
||||
services.AddAkka(systemName, (ab, sp) => ab.WithOtOpcUaClusterBootstrap(sp));
|
||||
});
|
||||
|
||||
var host = builder.Build();
|
||||
await host.StartAsync();
|
||||
return host;
|
||||
}
|
||||
|
||||
/// <summary>Polls cluster state until it holds at least one Up member, or the deadline passes.</summary>
|
||||
private static async Task<bool> WaitForUpMemberAsync(ActorSystem system, TimeSpan timeout)
|
||||
{
|
||||
var cluster = Akka.Cluster.Cluster.Get(system);
|
||||
var deadline = DateTime.UtcNow + timeout;
|
||||
while (DateTime.UtcNow < deadline)
|
||||
{
|
||||
if (cluster.State.Members.Any(m => m.Status == MemberStatus.Up))
|
||||
{
|
||||
return true;
|
||||
}
|
||||
|
||||
await Task.Delay(200);
|
||||
}
|
||||
|
||||
return cluster.State.Members.Any(m => m.Status == MemberStatus.Up);
|
||||
}
|
||||
|
||||
private static async Task StopAsync(IHost host)
|
||||
{
|
||||
await host.StopAsync();
|
||||
host.Dispose();
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// An unconfigured deployment gets the fallback: a pair node that cold-starts while its peer is
|
||||
/// down must come up on its own, not wait for an operator.
|
||||
/// </summary>
|
||||
[Fact]
|
||||
public void SelfFormAfter_defaults_to_ten_seconds()
|
||||
{
|
||||
new AkkaClusterOptions().SelfFormAfter.ShouldBe(TimeSpan.FromSeconds(10));
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// The core scenario: a node listed as a non-first seed, booting with its peer dead, becomes Up
|
||||
/// on its own after the window. Without the fallback it loops on InitJoin forever.
|
||||
/// </summary>
|
||||
[Fact]
|
||||
public async Task Lone_non_first_seed_self_forms_after_the_window()
|
||||
{
|
||||
var selfPort = FreePort();
|
||||
var peerPort = FreePort();
|
||||
var host = await StartNodeAsync(
|
||||
"otopcua-selfform-1", selfPort, peerPort, TimeSpan.FromSeconds(2));
|
||||
try
|
||||
{
|
||||
var system = host.Services.GetRequiredService<ActorSystem>();
|
||||
|
||||
(await WaitForUpMemberAsync(system, TimeSpan.FromSeconds(20)))
|
||||
.ShouldBeTrue("a lone non-first seed must self-form and come Up unattended");
|
||||
}
|
||||
finally
|
||||
{
|
||||
await StopAsync(host);
|
||||
}
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// The opt-out is real: with the window unset the node keeps waiting on InitJoin. The positive
|
||||
/// control proves the node was formable and only the fallback was missing — otherwise this test
|
||||
/// would pass just as happily against a node that could never join anything.
|
||||
/// </summary>
|
||||
[Fact]
|
||||
public async Task Disabled_fallback_keeps_waiting()
|
||||
{
|
||||
var selfPort = FreePort();
|
||||
var peerPort = FreePort();
|
||||
var host = await StartNodeAsync("otopcua-selfform-2", selfPort, peerPort, selfFormAfter: null);
|
||||
try
|
||||
{
|
||||
var system = host.Services.GetRequiredService<ActorSystem>();
|
||||
|
||||
(await WaitForUpMemberAsync(system, TimeSpan.FromSeconds(6)))
|
||||
.ShouldBeFalse("with SelfFormAfter unset the node must keep waiting on InitJoin");
|
||||
|
||||
// Positive control: the node WAS formable all along.
|
||||
var cluster = Akka.Cluster.Cluster.Get(system);
|
||||
cluster.Join(cluster.SelfAddress);
|
||||
(await WaitForUpMemberAsync(system, TimeSpan.FromSeconds(20)))
|
||||
.ShouldBeTrue("positive control — an explicit self-join must bring this node Up");
|
||||
}
|
||||
finally
|
||||
{
|
||||
await StopAsync(host);
|
||||
}
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Regression for the live-gate islanding defect (2026-07-22). A peer that is <b>reachable</b>
|
||||
/// but not yet admitting the join must NOT be treated as "peer down at boot".
|
||||
/// </summary>
|
||||
/// <remarks>
|
||||
/// <para>
|
||||
/// On the docker-dev rig, a node bounced by a manual failover restarted, received
|
||||
/// <c>InitJoinAck</c> from its live peer — the join was in flight and healthy — but did
|
||||
/// not get the Welcome inside the window, because the peer's ring still held the node's
|
||||
/// previous incarnation. The fallback fired on the timer and the node formed a
|
||||
/// <i>second</i> cluster, islanded until an operator restarted it. The original design
|
||||
/// assumed <c>Join(SelfAddress)</c> would be ignored mid-handshake; it is not.
|
||||
/// </para>
|
||||
/// <para>
|
||||
/// The peer here is a bare <see cref="TcpListener"/> that accepts connections and speaks
|
||||
/// no Akka at all — reachable at the transport level, never answering a join. That is the
|
||||
/// defect's shape with nothing faked: under the pre-fix code this node self-formed on the
|
||||
/// timer; under the guard it keeps waiting. The positive control then proves the guard is
|
||||
/// a <i>guard</i> and not a disablement: drop the listener and the same node self-forms.
|
||||
/// </para>
|
||||
/// </remarks>
|
||||
[Fact]
|
||||
public async Task Reachable_peer_suppresses_self_forming_until_it_goes_away()
|
||||
{
|
||||
var selfPort = FreePort();
|
||||
var peerPort = FreePort();
|
||||
|
||||
// A peer that accepts TCP and does nothing else — reachable, but no join will ever complete.
|
||||
var peer = new TcpListener(IPAddress.Loopback, peerPort);
|
||||
peer.Start();
|
||||
|
||||
var host = await StartNodeAsync(
|
||||
"otopcua-selfform-4", selfPort, peerPort, TimeSpan.FromSeconds(2));
|
||||
try
|
||||
{
|
||||
var system = host.Services.GetRequiredService<ActorSystem>();
|
||||
|
||||
// Several windows pass. Pre-fix this self-formed after the first one.
|
||||
(await WaitForUpMemberAsync(system, TimeSpan.FromSeconds(10)))
|
||||
.ShouldBeFalse("a reachable peer means a join may be in flight — self-forming here islands this node");
|
||||
|
||||
// Positive control: the peer really goes away, and now the fallback does its job.
|
||||
peer.Stop();
|
||||
|
||||
(await WaitForUpMemberAsync(system, TimeSpan.FromSeconds(30)))
|
||||
.ShouldBeTrue("once the peer is genuinely gone the fallback must still self-form");
|
||||
}
|
||||
finally
|
||||
{
|
||||
try { peer.Stop(); } catch (SocketException) { /* already stopped by the test body */ }
|
||||
await StopAsync(host);
|
||||
}
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// THE island guard. A node that is not in its own seed list — the current docker-dev site-node
|
||||
/// topology, where site-a/site-b list only central-1 — must never self-form: it would island
|
||||
/// itself from the real seeds permanently. Positive control as above.
|
||||
/// </summary>
|
||||
[Fact]
|
||||
public async Task Site_node_seeded_only_by_central_never_self_forms()
|
||||
{
|
||||
var selfPort = FreePort();
|
||||
var peerPort = FreePort();
|
||||
var host = await StartNodeAsync(
|
||||
"otopcua-selfform-3", selfPort, peerPort, TimeSpan.FromSeconds(1), selfInSeeds: false);
|
||||
try
|
||||
{
|
||||
var system = host.Services.GetRequiredService<ActorSystem>();
|
||||
|
||||
(await WaitForUpMemberAsync(system, TimeSpan.FromSeconds(5)))
|
||||
.ShouldBeFalse("a node absent from its own seed list must wait, not island itself");
|
||||
|
||||
var cluster = Akka.Cluster.Cluster.Get(system);
|
||||
cluster.Join(cluster.SelfAddress);
|
||||
(await WaitForUpMemberAsync(system, TimeSpan.FromSeconds(20)))
|
||||
.ShouldBeTrue("positive control — an explicit self-join must bring this node Up");
|
||||
}
|
||||
finally
|
||||
{
|
||||
await StopAsync(host);
|
||||
}
|
||||
}
|
||||
}
|
||||
Reference in New Issue
Block a user