docs(redundancy): SelfFormAfter closes the seed-node bootstrap gap
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
This commit is contained in:
+40
-5
@@ -159,7 +159,7 @@ Per-node identity comes from `appsettings.json` + the `OTOPCUA_ROLES` env var:
|
||||
OTOPCUA_ROLES=admin,driver
|
||||
```
|
||||
|
||||
Both nodes share the same `ConfigDb` connection string; `Cluster.PublicHostname` + `Roles` are what makes them distinct in cluster gossip. The first node bootstraps the cluster (its address goes in `SeedNodes`); the second node joins via the same `SeedNodes` list.
|
||||
Both nodes share the same `ConfigDb` connection string; `Cluster.PublicHostname` + `Roles` are what makes them distinct in cluster gossip. The first node bootstraps the cluster (its address goes in `SeedNodes`); the second node joins via the same `SeedNodes` list. List **both** peers in `SeedNodes` on **both** nodes so either can cold-start alone — see [Bootstrap: the self-form fallback](#bootstrap-the-self-form-fallback) for why the seed list alone is not enough, and what `Cluster:SelfFormAfter` adds.
|
||||
|
||||
There is no longer a `Node:NodeId` setting and no `ClusterNode.RedundancyRole` column (the V2 migration dropped it — primary/secondary is now derived from cluster membership age). NodeId is derived as `host:port` of the cluster `PublicHostname` (see `ClusterRoleInfo.LocalNode` for the formula).
|
||||
|
||||
@@ -241,10 +241,9 @@ two-node pair that choice means *any* crash of the oldest node is a full outage.
|
||||
`IHostApplicationLifetime.StopApplication()` so the process exits and the supervisor restarts it instead
|
||||
of idling forever with a dead actor system. Under `auto-down` the *survivor* no longer needs this
|
||||
(it is never the one downed); the **downed** node still does.
|
||||
3. **Both peers listed in `SeedNodes`** on every node, so a restarted node can re-join via either peer.
|
||||
Note the residual bootstrap constraint: only the **first** seed may form a cluster alone, so a node
|
||||
cold-starting while its peer is dead still waits in `InitJoin`. Auto-down removes the crash outage, not
|
||||
this one.
|
||||
3. **Both peers listed in `SeedNodes`** on every node, so a restarted node can re-join via either peer —
|
||||
plus the **self-form fallback** below, without which a node cold-starting while its peer is dead waits
|
||||
in `InitJoin` forever (auto-down removes the crash outage, not that one).
|
||||
|
||||
> **On `HardKillFailoverTests`:** that in-process test hard-kills the oldest and asserts the survivor becomes
|
||||
> sole `driver` role-leader. It passed even under `keep-oldest` — because it simulates the crash with
|
||||
@@ -270,6 +269,42 @@ automatically.
|
||||
> alongside the per-cluster mesh work, which makes every mesh exactly two nodes. See
|
||||
> `docs/plans/2026-07-21-per-cluster-mesh-design.md` §6.2 / Phase 0a.
|
||||
|
||||
## Bootstrap: the self-form fallback
|
||||
|
||||
Listing both peers in `SeedNodes` does **not** mean either node can cold-start alone. Akka lets only the
|
||||
**first** listed seed form a *new* cluster; every other node sends `InitJoin` to the seeds and retries —
|
||||
forever — until one of them answers. A node that boots while its peer is dead therefore never comes Up,
|
||||
and no downing strategy helps: **auto-down removes the crash outage, not this one.**
|
||||
|
||||
`Cluster:SelfFormAfter` closes that gap. `ClusterBootstrapFallback.Arm` (armed for every host built through
|
||||
`WithOtOpcUaClusterBootstrap`, via an Akka.Hosting `AddStartup` task) waits that long for cluster membership
|
||||
and, on expiry, calls `Cluster.Join(SelfAddress)` — the node forms a cluster on itself and becomes
|
||||
operational unattended.
|
||||
|
||||
| `Cluster:SelfFormAfter` | Behaviour |
|
||||
|---|---|
|
||||
| `"00:00:10"` (**default**) | A node with no membership after 10 s self-forms. A live peer answers `InitJoin` in milliseconds, so on any normal boot the fallback never fires. |
|
||||
| `null` / `≤ 0` | Disabled — the pre-2026-07-22 behaviour: wait on `InitJoin` indefinitely. |
|
||||
|
||||
**The island guard is the load-bearing part.** The fallback fires **only when this node's own address
|
||||
appears in its own `SeedNodes`**. A node that is not one of its own seeds is never legitimately first, and
|
||||
if it self-formed, a later-booting real seed would form a second cluster the two could never merge. That is
|
||||
exactly the current docker-dev topology: the site-a/site-b driver nodes list only `central-1` as a seed, so
|
||||
they deliberately keep waiting rather than islanding themselves. Under the future per-cluster mesh
|
||||
(`docs/plans/2026-07-21-per-cluster-mesh-design.md`) each pair node *is* a seed of its own two-node mesh, so
|
||||
the fallback covers both sides of every pair and the site-node exception disappears.
|
||||
|
||||
Sequential recovery is island-free by construction: Akka's join protocol prefers an existing cluster (a
|
||||
booting node only self-forms when **no** seed answered), so a peer booting after the survivor self-formed
|
||||
simply joins it as the youngest member. The residual risk is both pair nodes cold-starting inside the
|
||||
window while mutually unreachable — both self-form, which is the same dual-active class the `auto-down`
|
||||
strategy already accepts, with the same recovery (restart one side).
|
||||
|
||||
Pinned by `SelfFormBootstrapTests` (`tests/Core/ZB.MOM.WW.OtOpcUa.Cluster.Tests/`), which starts real hosts
|
||||
through the production bootstrap: a lone non-first seed comes Up; a disabled window keeps waiting; a node
|
||||
absent from its own seed list never self-forms. The two negative cases each carry a positive control (an
|
||||
explicit self-join) so they cannot pass merely because the node was unformable.
|
||||
|
||||
## Primary data-plane gate (writes, acks, alerts emit)
|
||||
|
||||
Three data-plane surfaces are **Primary-gated** so a warm standby never touches a shared field device or duplicates a fleet-wide publish: inbound operator **writes** (`DriverHostActor.HandleRouteNodeWrite`), native-alarm **acks** (`HandleRouteNativeAlarmAck`), and the native/scripted **alerts emit** (`ForwardNativeAlarm` / `ScriptedAlarmHostActor.OnEngineEmission`). All three route through one policy — `PrimaryGatePolicy.ShouldServiceAsPrimary(localRole, driverMemberCount)` (`OtOpcUa.Runtime.Drivers`):
|
||||
|
||||
Reference in New Issue
Block a user