diff --git a/CLAUDE.md b/CLAUDE.md index 5cca4ce7..f8974e6d 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -224,7 +224,7 @@ The server supports non-transparent warm/hot redundancy via the `Redundancy` sec - **Downing is `auto-down`, not `keep-oldest`.** `Cluster:SplitBrainResolverStrategy` (default `auto-down`) selects the provider via `ServiceCollectionExtensions.BuildDowningHocon`; an unknown value fails host start. A two-node pair running the previous `keep-oldest` + `down-if-alone` **could not survive a crash of the oldest node**: Akka's `KeepOldest.OldestDecision` only lets `down-if-alone` rescue a side holding ≥ 2 members, so the 1-vs-1 survivor downs *itself* and exits. `down-if-alone` is a 3+-node feature. The accepted trade for `auto-down` is dual-active during a genuine network partition. **Its live gate is still open** — the pathology only appears 1-vs-1 and docker-dev is a single six-node mesh, so the crash-the-oldest drill is deferred to the per-cluster mesh work. - **The Primary is the oldest Up `driver` member, not the role leader.** `RedundancyStateActor` previously used `ClusterState.RoleLeader("driver")` (lowest *address*), while `ClusterSingletonManager` places singletons on the *oldest*. They agree only on a freshly-formed cluster and diverge after any restart, so the Primary-gated data plane (writes, alarm acks, alerts emit, alarm-history drain) could enable on a node not hosting the singletons. `NodeRedundancyState.IsRoleLeaderForDriver` was renamed **`IsDriverPrimary`**. -**A third bootstrap gap closed 2026-07-22 — downing was never the whole story.** Even under `auto-down`, a node cold-starting while its peer was dead **never came Up**: Akka lets only the **first** listed seed form a *new* cluster, so every other node loops on `InitJoin` forever. `Cluster:SelfFormAfter` (`TimeSpan?`, default 10 s, `null`/`≤0` disables) arms `ClusterBootstrapFallback` from `WithOtOpcUaClusterBootstrap` via an Akka.Hosting `AddStartup` task — it waits that long for membership, then `Cluster.Join(SelfAddress)`. **It fires only when this node's own address is in its own `SeedNodes`**: a node seeded only by someone else (today's docker-dev site nodes, which list only `central-1`) must wait rather than island itself permanently. Pinned by `SelfFormBootstrapTests`, whose two negative cases each carry a positive-control self-join so they cannot pass vacuously. See `docs/Redundancy.md` §"Bootstrap: the self-form fallback". +**A third bootstrap gap closed 2026-07-22 — downing was never the whole story, and the fix changed twice.** Even under `auto-down`, a node cold-starting while its peer was dead **never came Up**: Akka runs `FirstSeedNodeProcess` — the only bootstrap path that can form a *new* cluster when no peer answers `InitJoin` — exclusively when `seed-nodes[0]` is the node’s own address; every other node runs `JoinSeedNodeProcess` and retries `InitJoin` forever. **`Cluster:SeedNodes` is therefore ORDERED: every node that is one of its own seeds lists ITSELF first, partner second** (docker-dev `central-2` was swapped; site nodes list only `central-1` and are legitimately not seeds). `AkkaClusterOptionsValidator` (`AddValidatedOptions`/`ValidateOnStart` in `AddOtOpcUaCluster`) enforces it at boot — **conditionally** (self must be entry 0 only *if* self is in the list, or every site node would refuse to start), matching on `PublicHostname`+`Port`, never the `0.0.0.0` bind address. The earlier fix — a `Cluster:SelfFormAfter` timer calling `Cluster.Join(SelfAddress)` — is **deleted**: it sat outside Akka’s join handshake, so it could not tell "no seed answered" from "a seed answered and the join is in flight", and it islanded a manually-failed-over node live before a TCP reachability guard patched that one shape. Pinned by `SelfFirstSeedBootstrapTests` (real in-process clusters, incl. a falsifiability control proving peer-first ordering never forms) + `AkkaClusterOptionsValidatorTests`. See `docs/Redundancy.md` §"Bootstrap: self-first seed ordering". Assert the **effective** cluster config off a running `ActorSystem` (as `SplitBrainResolverActivationTests` does), never the typed options or the akka.conf text — several HOCON fragments compete and the losing one still reads correctly in the file. diff --git a/docs/Configuration.md b/docs/Configuration.md index c4b1ecb3..06ec59e6 100644 --- a/docs/Configuration.md +++ b/docs/Configuration.md @@ -103,7 +103,7 @@ The checked-in `appsettings*.json` files are deliberately thin: they carry only | `Hostname` | string | `0.0.0.0` | Bind hostname. | | `Port` | int | `4053` | Cluster transport port. | | `PublicHostname` | string | `127.0.0.1` | Hostname advertised in cluster gossip; must be reachable by peers. | -| `SeedNodes` | string[] | `[]` | Seed nodes for bootstrapping. | +| `SeedNodes` | string[] | `[]` | Seed nodes for bootstrapping. **Ordered:** a node that appears in its own list must be entry 0 (only `seed-nodes[0]` can form a new cluster), or the host refuses to start — see [Redundancy.md § Bootstrap: self-first seed ordering](Redundancy.md#bootstrap-self-first-seed-ordering). | | `Roles` | string[] | `[]` | Cluster roles for this node. When empty, falls back to `OTOPCUA_ROLES`. Allowed values: `admin`, `driver`, `dev`. | > The full redundancy model (ServiceLevel tiers, split-brain, peer discovery) is in [`Redundancy.md`](Redundancy.md). The OPC UA peer-URI advertising lives in the `OpcUa:PeerApplicationUris` key above. diff --git a/docs/Redundancy.md b/docs/Redundancy.md index f5831ebd..bd8e3f07 100644 --- a/docs/Redundancy.md +++ b/docs/Redundancy.md @@ -143,13 +143,18 @@ The admin singleton is the cluster's only `RedundancyStateActor`. If the admin l Per-node identity comes from `appsettings.json` + the `OTOPCUA_ROLES` env var: -```json +```jsonc { "Cluster": { "Hostname": "0.0.0.0", "Port": 4053, "PublicHostname": "node-a.lan", - "SeedNodes": ["akka.tcp://otopcua@node-a.lan:4053"], + // Self FIRST, partner second — see "Bootstrap: self-first seed ordering". + // node-b.lan's own config lists node-b.lan first and node-a.lan second. + "SeedNodes": [ + "akka.tcp://otopcua@node-a.lan:4053", + "akka.tcp://otopcua@node-b.lan:4053" + ], "Roles": ["admin", "driver"] } } @@ -159,7 +164,7 @@ Per-node identity comes from `appsettings.json` + the `OTOPCUA_ROLES` env var: OTOPCUA_ROLES=admin,driver ``` -Both nodes share the same `ConfigDb` connection string; `Cluster.PublicHostname` + `Roles` are what makes them distinct in cluster gossip. The first node bootstraps the cluster (its address goes in `SeedNodes`); the second node joins via the same `SeedNodes` list. List **both** peers in `SeedNodes` on **both** nodes so either can cold-start alone — see [Bootstrap: the self-form fallback](#bootstrap-the-self-form-fallback) for why the seed list alone is not enough, and what `Cluster:SelfFormAfter` adds. +Both nodes share the same `ConfigDb` connection string; `Cluster.PublicHostname` + `Roles` are what makes them distinct in cluster gossip. List **both** peers in `SeedNodes` on **both** nodes so either can cold-start alone — **and list each node ITSELF first**: the order decides which Akka bootstrap process runs, and a node that lists its partner first can never form the cluster while that partner is down. See [Bootstrap: self-first seed ordering](#bootstrap-self-first-seed-ordering). There is no longer a `Node:NodeId` setting and no `ClusterNode.RedundancyRole` column (the V2 migration dropped it — primary/secondary is now derived from cluster membership age). NodeId is derived as `host:port` of the cluster `PublicHostname` (see `ClusterRoleInfo.LocalNode` for the formula). @@ -241,9 +246,10 @@ two-node pair that choice means *any* crash of the oldest node is a full outage. `IHostApplicationLifetime.StopApplication()` so the process exits and the supervisor restarts it instead of idling forever with a dead actor system. Under `auto-down` the *survivor* no longer needs this (it is never the one downed); the **downed** node still does. -3. **Both peers listed in `SeedNodes`** on every node, so a restarted node can re-join via either peer — - plus the **self-form fallback** below, without which a node cold-starting while its peer is dead waits - in `InitJoin` forever (auto-down removes the crash outage, not that one). +3. **Both peers listed in `SeedNodes`** on every node, **each node listing itself first**, so a restarted + node can re-join via either peer and a node cold-starting while its peer is dead still forms the + cluster. Without the self-first ordering it waits in `InitJoin` forever (auto-down removes the crash + outage, not that one) — see [Bootstrap: self-first seed ordering](#bootstrap-self-first-seed-ordering). > **On `HardKillFailoverTests`:** that in-process test hard-kills the oldest and asserts the survivor becomes > sole `driver` role-leader. It passed even under `keep-oldest` — because it simulates the crash with @@ -297,66 +303,84 @@ advertises `ServiceLevel` 250. OPC UA clients re-select. > alongside the per-cluster mesh work, which makes every mesh exactly two nodes. See > `docs/plans/2026-07-21-per-cluster-mesh-design.md` §6.2 / Phase 0a. -## Bootstrap: the self-form fallback +## Bootstrap: self-first seed ordering -Listing both peers in `SeedNodes` does **not** mean either node can cold-start alone. Akka lets only the -**first** listed seed form a *new* cluster; every other node sends `InitJoin` to the seeds and retries — -forever — until one of them answers. A node that boots while its peer is dead therefore never comes Up, -and no downing strategy helps: **auto-down removes the crash outage, not this one.** +Listing both peers in `SeedNodes` does **not** mean either node can cold-start alone — and neither does any +downing strategy: **auto-down removes the crash outage, not this one.** Akka runs a different bootstrap +process depending on whether `seed-nodes[0]` is the node's own address: -`Cluster:SelfFormAfter` closes that gap. `ClusterBootstrapFallback.Arm` (armed for every host built through -`WithOtOpcUaClusterBootstrap`, via an Akka.Hosting `AddStartup` task) waits that long for cluster membership -and, on expiry, calls `Cluster.Join(SelfAddress)` — the node forms a cluster on itself and becomes -operational unattended. +| `seed-nodes[0]` is… | Process Akka runs | Can it form a NEW cluster? | +|---|---|---| +| this node's own address | `FirstSeedNodeProcess` | **Yes** — it `InitJoin`s the other seeds and self-joins once `seed-node-timeout` (5 s) passes with nobody answering. | +| some other node | `JoinSeedNodeProcess` | **No** — it retries `InitJoin` forever, however long the peer stays down. | -| `Cluster:SelfFormAfter` | Behaviour | -|---|---| -| `"00:00:10"` (**default**) | A node with no membership after 10 s self-forms. A live peer answers `InitJoin` in milliseconds, so on any normal boot the fallback never fires. | -| `null` / `≤ 0` | Disabled — the pre-2026-07-22 behaviour: wait on `InitJoin` indefinitely. | +So the ordering *is* the mechanism. **Every node that is a seed of its own mesh lists ITSELF first and its +partner second** (decision 2026-07-22): -**Two guards keep the fallback from creating the split it exists to avoid.** +```jsonc +// central-1 // central-2 +"SeedNodes": [ "SeedNodes": [ + "akka.tcp://otopcua@central-1:4053", "akka.tcp://otopcua@central-2:4053", + "akka.tcp://otopcua@central-2:4053" "akka.tcp://otopcua@central-1:4053" +] ] +``` -**1 — Reachability.** An expired window is not evidence the peer is down, so before self-forming the -fallback TCP-probes the other seed addresses; if any accepts a connection it waits another window instead. -This was added after the live gate below caught the fallback islanding a node: a node bounced by a manual -failover restarted, received `InitJoinAck` from its live peer — a join in flight and healthy — but did not -get the Welcome inside the window, because the peer's ring still held the node's previous incarnation -(`Exiting` → `Down` → `Removed`). The fallback fired on the timer and the node formed a **second cluster**. -The original design assumed `Cluster.Join(SelfAddress)` would be ignored mid-handshake; **it is not — it -wins.** Since manual failover deliberately produces exactly that restart, without this guard every manual -failover could island the node it bounced. Pinned by -`SelfFormBootstrapTests.Reachable_peer_suppresses_self_forming_until_it_goes_away`, whose "peer" is a bare -`TcpListener` that speaks no Akka: reachable, never answering a join. +The rule is **conditional**: it binds only when a node's own address appears in its own seed list. A +driver-only site node is seeded solely by `central-1` (today's docker-dev topology) — it is legitimately +not a seed of anything and is exempt. An unconditional "self must be first" rule would refuse to boot every +site node in the fleet. -**2 — Seed membership.** The fallback fires **only when this node's own address -appears in its own `SeedNodes`**. A node that is not one of its own seeds is never legitimately first, and -if it self-formed, a later-booting real seed would form a second cluster the two could never merge. That is -exactly the current docker-dev topology: the site-a/site-b driver nodes list only `central-1` as a seed, so -they deliberately keep waiting rather than islanding themselves. Under the future per-cluster mesh -(`docs/plans/2026-07-21-per-cluster-mesh-design.md`) each pair node *is* a seed of its own two-node mesh, so -the fallback covers both sides of every pair and the site-node exception disappears. +`AkkaClusterOptionsValidator` enforces it at startup (`AddValidatedOptions` → `ValidateOnStart` in +`AddOtOpcUaCluster`), because the invariant otherwise fails **silently**: the process runs, the port +listens, the node simply never becomes a member. Identity is compared on `Cluster:PublicHostname` (falling +back to `Cluster:Hostname` when blank) **and** `Cluster:Port` — the address Akka puts in `SelfAddress`. +Comparing against `Cluster:Hostname` would be worse than wrong: in docker-dev it is `0.0.0.0`, which matches +no seed URI anywhere, so the rule would silently exempt the entire rig. -Sequential recovery is island-free by construction: Akka's join protocol prefers an existing cluster (a -booting node only self-forms when **no** seed answered), so a peer booting after the survivor self-formed -simply joins it as the youngest member. The residual risk is both pair nodes cold-starting inside the -window while mutually unreachable — both self-form, which is the same dual-active class the `auto-down` -strategy already accepts, with the same recovery (restart one side). +Sequential recovery is island-free by construction: a `FirstSeedNodeProcess` node self-joins only when **no** +seed answered, so a peer booting after the survivor formed simply joins it as the youngest member. Both +nodes cold-starting *simultaneously* converge on one cluster while they are mutually reachable — the +`InitJoin` handshake resolves it before either self-join deadline expires. The residual risk is a boot-time +**partition** (both cold-starting, mutually unreachable): both form, which is the same dual-active class the +`auto-down` strategy already accepts, with the same recovery (restart one side). -Pinned by `SelfFormBootstrapTests` (`tests/Core/ZB.MOM.WW.OtOpcUa.Cluster.Tests/`), which starts real hosts -through the production bootstrap: a lone non-first seed comes Up; a disabled window keeps waiting; a node -absent from its own seed list never self-forms; a node whose peer is merely *reachable* keeps waiting. Every -negative case carries a positive control (an explicit self-join, or dropping the peer listener) so none can -pass merely because the node was unformable. +Pinned by `SelfFirstSeedBootstrapTests` (`tests/Core/ZB.MOM.WW.OtOpcUa.Cluster.Tests/`), which starts real +in-process clusters through the production bootstrap at production failure-detection timings: a lone +cold-start with a dead peer forms alone; a restarted node rejoins its live peer; two nodes cold-starting +together converge on one cluster. Its **falsifiability control** — the old peer-first ordering, same +scenario, never comes Up — is what keeps the other three from going vacuous, and carries a positive-control +self-join proving the node was formable all along. `AkkaClusterOptionsValidatorTests` covers the validator, +including the site-node exemption and the `0.0.0.0` identity trap. -> **Live gate: PASSED (docker-dev, 2026-07-22).** Drilled on the six-node rig, defect first and fix -> second. On the pre-change image, `central-2` started alone (central-1 stopped) and logged **zero** join -> events in 70 s — the `InitJoin` loop, observed rather than argued. On the rebuilt image the same node -> logged the self-form warning **10 s** after start and `Leader is moving node [central-2] to [Up]`, then -> came up with all four admin singletons. A `site-a-2` recreated while central-1 was still down logged -> *"self-form fallback inactive: this node … is not in its own seed list"* and stayed out — the island -> guard on the real topology. Restarting `central-1` produced `Welcome from [central-2]`: it joined the -> self-formed cluster instead of starting a second one, and the four site nodes then joined through it — -> six members, **one** cluster. +### Why the self-form watchdog was retired (2026-07-22) + +The first fix for this gap was `Cluster:SelfFormAfter` + `ClusterBootstrapFallback`: wait 10 s for +membership, then call `Cluster.Join(SelfAddress)`. **Both are deleted.** A timer sits *outside* Akka's join +handshake, so it cannot distinguish "no seed answered" from "a seed answered and the join is in flight" — +and `Cluster.Join(SelfAddress)` is **not** ignored mid-handshake: it wins. + +> **Live-gate finding (docker-dev, 2026-07-22 — preserved because it is the evidence).** The watchdog +> islanded a node that manual failover had bounced. `central-1` restarted, received `InitJoinAck` from +> `central-2` at 10:49:05 — the join was in flight and healthy — but no Welcome arrived inside the window, +> because the peer's ring still held its previous incarnation (`Exiting` → `Down` → `Removed`, retired at +> 10:49:06). At 10:49:15 the watchdog fired and the node formed a **second cluster**, islanded until an +> operator restarted it. Manual failover deliberately produces exactly that restart, so the two halves of +> that plan collided. A TCP reachability guard (`ea45ace1`) patched that one shape — never self-form while +> a seed peer accepts a connection — and the earlier live gate for the watchdog itself had genuinely +> passed: `central-2` cold-started alone in 10 s and a non-seed `site-a-2` correctly stayed out. + +The reachability guard closed the observed failure, not the class: a peer can be TCP-reachable and not +answering, or answering and slow, and the timer cannot tell. Self-first ordering has no such race because +the decision is made *inside* the handshake, by Akka, on the actual `InitJoin` replies. It was also inert +for every non-seed node, so after this change it could never usefully fire. `SelfFormBootstrapTests` is +replaced by `SelfFirstSeedBootstrapTests`, whose `Restarting_node_rejoins_its_live_peer_instead_of_islanding` +is the drilled scenario above. + +> **Live gate outstanding for the new mechanism.** Self-first ordering is verified here by in-process +> clusters at production timings, and on the sister project (ScadaBridge `4a6341d8`) on real clusters *and* +> live docker containers. It has **not** been re-drilled on the OtOpcUa docker-dev rig since the swap — the +> `central-2` cold-start-alone drill should be repeated on the rebuilt image. Owned by +> `docs/plans/2026-07-22-per-cluster-mesh-program.md` Phase 7 alongside the auto-down 1-vs-1 gate. ## Primary data-plane gate (writes, acks, alerts emit) diff --git a/docs/ServiceHosting.md b/docs/ServiceHosting.md index 6bc89eb5..fed0b328 100644 --- a/docs/ServiceHosting.md +++ b/docs/ServiceHosting.md @@ -56,6 +56,7 @@ The host joins an Akka.NET cluster bound to the address in `appsettings.json::Cl } ``` +- `SeedNodes` is **ordered**: a node listed in its own seed list must be entry 0, since Akka only lets `seed-nodes[0]` form a new cluster. `AkkaClusterOptionsValidator` fails startup otherwise — see [Redundancy.md § Bootstrap: self-first seed ordering](Redundancy.md#bootstrap-self-first-seed-ordering). - `WithOtOpcUaClusterBootstrap` (in `OtOpcUa.Cluster`) loads the embedded HOCON (split-brain resolver, pinned dispatcher, failure detector tuning) and overlays remote endpoint + cluster options. - All cluster singletons + per-node actors live on this single ActorSystem — there is no second Akka instance. diff --git a/docs/plans/2026-07-21-per-cluster-mesh-design.md b/docs/plans/2026-07-21-per-cluster-mesh-design.md index 71362557..2de3ebd2 100644 --- a/docs/plans/2026-07-21-per-cluster-mesh-design.md +++ b/docs/plans/2026-07-21-per-cluster-mesh-design.md @@ -85,11 +85,13 @@ Their failover drill has a mode that exists *"to make the registered gap observa it is covered."* **A two-node keep-oldest mesh has an acknowledged total-outage hole.** If OtOpcUa adopts 2-node meshes, this must be decided deliberately, not inherited. -> **CLOSED 2026-07-22.** The `InitJoin` half of this gap is fixed in both repos by the self-form -> fallback: `Cluster:SelfFormAfter` (default 10 s) lets a node that is one of its own seeds form a -> cluster on itself when no seed answers. See `docs/Redundancy.md` §"Bootstrap: the self-form -> fallback" — including the island guard that keeps today's non-seed site nodes waiting, which the -> per-cluster mesh removes by making every pair node a seed of its own mesh. +> **CLOSED 2026-07-22.** The `InitJoin` half of this gap is fixed in both repos by **self-first seed +> ordering**: every node that is one of its own seeds lists ITSELF as `seed-nodes[0]`, which is the +> only ordering under which Akka runs `FirstSeedNodeProcess` and can form a cluster with no peer +> answering. (An earlier `Cluster:SelfFormAfter` watchdog was tried here and retired — it raced the +> join handshake.) See `docs/Redundancy.md` §"Bootstrap: self-first seed ordering". Today's +> driver-only site nodes are seeded by `central-1` only and are exempt; the per-cluster mesh removes +> even that exception by making every pair node a seed of its own mesh. ### Both inter-cluster transports are unauthenticated @@ -261,11 +263,14 @@ with Phase 6/7, which makes every mesh exactly two nodes. **The bootstrap half of the gap is also closed (2026-07-22).** Downing was only ever one of the two outage modes: even with `auto-down`, a node cold-starting while its peer was dead looped on `InitJoin` -forever, because Akka lets only the first listed seed form a new cluster. `Cluster:SelfFormAfter` -(default 10 s, `ClusterBootstrapFallback`) now self-forms in that case, guarded so a node absent from -its own seed list never islands itself. Pinned by `SelfFormBootstrapTests`; see `docs/Redundancy.md` -§"Bootstrap: the self-form fallback". Under this design each pair node is a seed of its own two-node -mesh, so the fallback covers both sides and the current site-node exception disappears. +forever, because Akka lets only `seed-nodes[0]` form a new cluster. The fix is **self-first seed +ordering** — each seed node lists itself first, enforced at boot by `AkkaClusterOptionsValidator`. +The `Cluster:SelfFormAfter` watchdog first shipped for this is deleted: a timer outside Akka's join +handshake cannot tell "no seed answered" from "a seed answered and the join is in flight", and it +islanded a manually-failed-over node live. Pinned by `SelfFirstSeedBootstrapTests`; see +`docs/Redundancy.md` §"Bootstrap: self-first seed ordering". Under this design each pair node is a +seed of its own two-node mesh, so the ordering covers both sides and the current site-node exemption +disappears. ### 6.3 DECIDED — match ScadaBridge's auth posture for now diff --git a/docs/plans/2026-07-22-per-cluster-mesh-program.md b/docs/plans/2026-07-22-per-cluster-mesh-program.md index 7b7bbacd..e5c46b22 100644 --- a/docs/plans/2026-07-22-per-cluster-mesh-program.md +++ b/docs/plans/2026-07-22-per-cluster-mesh-program.md @@ -46,8 +46,9 @@ Two independent 2-node Akka clusters per site — they share hardware, never a m | OtOpcUa LocalDb sync port | OtOpcUa driver | h2c LocalDb pair replication (default off/0 today — Phase 6 assigns a real per-site port) | **Aligned HA posture (both products, per VM — already true or landing via the selfform plan):** -auto-down downing (15 s window), oldest-Up active/primary election, `SelfFormAfter` 10 s -self-form fallback, termination-watchdog → process exit → `sc.exe failure` restart recovery. +auto-down downing (15 s window), oldest-Up active/primary election, self-first seed ordering +(the `SelfFormAfter` self-form fallback that briefly stood in for it was retired 2026-07-22), +termination-watchdog → process exit → `sc.exe failure` restart recovery. One failover story for operators regardless of product. **Prerequisite ordering:** the fallback/manual-failover plan @@ -143,12 +144,14 @@ observability). kill-and-reconnect of the central dialer recovers every stream. ### Phase 6 — Mesh partition + co-location topology -**Scope:** per-cluster seed nodes — **adopt ScadaBridge's self-first ordering (each node lists -ITSELF as `seed-nodes[0]`, partner second) and RETIRE the `SelfFormAfter` watchdog + TCP -reachability guard** (2026-07-22 execution finding: the watchdog races Akka's join handshake — -`Join(self)` during an in-flight join islands the node, hit live here and fixed with the guard; -ScadaBridge proved self-first ordering is the race-free form of the same semantics, enforced by -a startup-validator rule — port that rule too); cluster-scoped roles `cluster-{ClusterId}` + singleton re-scoping, +**Scope:** per-cluster seed nodes — ~~adopt ScadaBridge's self-first ordering and RETIRE the +`SelfFormAfter` watchdog + TCP reachability guard~~ **DONE EARLY 2026-07-22** (docker-dev +`central-2` swapped to self-first, `ClusterBootstrapFallback`/`SelfFormAfter` deleted, +`AkkaClusterOptionsValidator` ports ScadaBridge's startup rule in its conditional form, and +`SelfFirstSeedBootstrapTests` replaces `SelfFormBootstrapTests`; see `docs/Redundancy.md` +§"Bootstrap: self-first seed ordering"). What remains for this phase is the per-pair +`Cluster__SeedNodes__*` matrix once the meshes actually split — every node in a pair is then a seed +of its own mesh, so the site-node exemption disappears; cluster-scoped roles `cluster-{ClusterId}` + singleton re-scoping, central pair keeps the admin singletons; **docker-dev rig rewritten** to model the real topology — including the co-location port table above (both products' compose files on shared per-site networks, real LocalDb sync ports); remove the ClusterRedundancy page's mesh-scope caveat (the @@ -163,12 +166,13 @@ cross separate meshes). ### Phase 7 — Failover drill + live gates **Scope:** the drill ScadaBridge already has (`failover-drill.sh` analogue) run per pair, both directions; **close the two outstanding live gates**: (a) auto-down 1-vs-1 crash-the-oldest -(deferred since Phase 0a — finally testable, every mesh is exactly two nodes), (b) `SelfFormAfter` +(deferred since Phase 0a — finally testable, every mesh is exactly two nodes), (b) self-first lone-cold-start on the real per-pair topology; manual-failover button re-verified per pair; operator runbook for the co-located site (one page covering both products' failover on the same two VMs). **Exit gate:** drill green on every pair type (central, site); runbook merged; design doc §7 -table fully marked DONE. +table fully marked DONE. (Live gate (b) is now the **self-first cold-start-alone** drill, not +`SelfFormAfter` — the watchdog it named was retired 2026-07-22.) --- diff --git a/docs/plans/2026-07-22-per-cluster-mesh-program.md.tasks.json b/docs/plans/2026-07-22-per-cluster-mesh-program.md.tasks.json index 697e7ac5..8ac7f801 100644 --- a/docs/plans/2026-07-22-per-cluster-mesh-program.md.tasks.json +++ b/docs/plans/2026-07-22-per-cluster-mesh-program.md.tasks.json @@ -9,7 +9,7 @@ {"id": 4, "subject": "Phase 4: cut driver-side ConfigDb — EfAlarmConditionStateStore to LocalDb, DbHealthProbeActor ServiceLevel input, OpcUaPublishActor audit (live gate)", "status": "pending", "blockedBy": [3]}, {"id": 5, "subject": "Phase 5: gRPC oneof stream contract; migrate 7 observability topics; AdminUI reconnect story", "status": "pending", "blockedBy": [2]}, {"id": 6, "subject": "Phase 6: mesh partition — per-pair seeds, cluster-scoped roles/singletons, co-location rig rewrite, secrets replication re-scope", "status": "pending", "blockedBy": [0, 4, 5]}, - {"id": 7, "subject": "Phase 7: failover drill per pair (incl. shared-VM failure) + close auto-down 1v1 and SelfFormAfter live gates + operator runbook", "status": "pending", "blockedBy": [6]} + {"id": 7, "subject": "Phase 7: failover drill per pair (incl. shared-VM failure) + close auto-down 1v1 and self-first cold-start-alone live gates + operator runbook", "status": "pending", "blockedBy": [6]} ], "lastUpdated": "2026-07-22T00:00:00Z" } diff --git a/docs/plans/2026-07-22-selfform-fallback-and-manual-failover.md b/docs/plans/2026-07-22-selfform-fallback-and-manual-failover.md index 9a89c3fe..36cd7170 100644 --- a/docs/plans/2026-07-22-selfform-fallback-and-manual-failover.md +++ b/docs/plans/2026-07-22-selfform-fallback-and-manual-failover.md @@ -1,5 +1,22 @@ # OtOpcUa: InitJoin Self-Form Fallback + Manual Failover Control — Implementation Plan +> ## ⚠ SUPERSEDED IN PART (2026-07-22, same day) +> +> **Half (1) of this plan — the `Cluster:SelfFormAfter` self-form watchdog — was retired within hours +> of landing.** `ClusterBootstrapFallback`, the `SelfFormAfter` option and `SelfFormBootstrapTests` +> are **deleted**; the cold-start-alone goal is now met by **self-first seed ordering** (each seed +> node lists ITSELF as `seed-nodes[0]`), enforced at boot by `AkkaClusterOptionsValidator` and pinned +> by `SelfFirstSeedBootstrapTests`. Reason: a timer outside Akka's join handshake cannot distinguish +> "no seed answered" from "a seed answered and the join is in flight", and `Cluster.Join(SelfAddress)` +> is not ignored mid-handshake — it wins. Task 8 below caught exactly that live and patched the one +> observed shape with a TCP reachability guard; the race remained. Mesh-program Phase 6 had this +> convergence queued and it was pulled forward. See `docs/Redundancy.md` §"Bootstrap: self-first seed +> ordering" and ScadaBridge `4a6341d8`. +> +> **Half (2) — manual failover — is unaffected and still current.** Tasks below are kept verbatim as +> the execution record, including the Task 8 live-gate evidence, which is the whole reason the +> watchdog was rejected. Do not execute the Task 1/2/3 code as written. + > **For Claude:** REQUIRED SUB-SKILL: Use superpowers-extended-cc:executing-plans to implement this plan task-by-task. > > Shared cross-repo design: `~/Desktop/scadaproj/docs/plans/2026-07-22-initjoin-selfform-fallback.md` (design rationale, MNTR assessment, behavior spec). The ScadaBridge half lives in `~/Desktop/ScadaBridge/docs/plans/2026-07-22-selfform-fallback-and-manual-failover.md`. This plan is self-contained for execution. diff --git a/docs/v2/Cluster.md b/docs/v2/Cluster.md index 2840a340..61f1b398 100644 --- a/docs/v2/Cluster.md +++ b/docs/v2/Cluster.md @@ -64,7 +64,7 @@ The Cluster.Tests project verifies these key values stay correct (`HoconLoaderTe - `Hostname`: interface to bind. `0.0.0.0` listens on every interface. - `Port`: TCP port for cluster gossip. Default 4053. - `PublicHostname`: address advertised in cluster gossip. Must be reachable by every other node. -- `SeedNodes`: where new nodes go to join. List one (or two) stable nodes. First node bootstraps the cluster from its own address. +- `SeedNodes`: where new nodes go to join. List one (or two) stable nodes. **Order is load-bearing:** Akka only lets `seed-nodes[0]` form a *new* cluster (`FirstSeedNodeProcess`); every other node retries `InitJoin` forever. A node that is one of its own seeds must therefore list ITSELF first, or it can never cold-start while its peer is down — enforced at boot by `AkkaClusterOptionsValidator`, explained in [Redundancy.md § Bootstrap: self-first seed ordering](../Redundancy.md#bootstrap-self-first-seed-ordering). Nodes seeded only by someone else (driver-only site nodes) are exempt. - `Roles`: free-form tags Akka gossip propagates. v2 uses `admin` + `driver`; per-role wiring in `Program.cs` reads `OTOPCUA_ROLES` env var, not this list — these two should stay in sync. Per-role overlay files (`appsettings.admin.json`, `appsettings.driver.json`, `appsettings.admin-driver.json`) layer on top of base `appsettings.json` based on the parsed `OTOPCUA_ROLES` (alphabetical, joined by `-`). See [ServiceHosting.md § Per-role configuration overlays](../ServiceHosting.md#per-role-configuration-overlays).