The live gate for the manual-failover control caught the self-form fallback
forming a SECOND cluster. A node bounced by a failover restarted, received
InitJoinAck from its live peer — the join was in flight and healthy — but did
not get the Welcome inside the 10s window, because the peer's ring still held
the node's previous incarnation (Exiting -> Down -> Removed). The fallback
fired on the timer and the node islanded itself until an operator restarted it.
The original design assumed Cluster.Join(SelfAddress) would be ignored
mid-handshake. It is not — it wins. And since manual failover deliberately
produces that restart, every failover could island the node it bounced.
Before self-forming, TCP-probe the other seed addresses; a reachable peer means
wait another window instead. That is also what the fallback claims to detect:
'no seed answered InitJoin (peer down at boot)'.
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
RedundancyStateActor derived the driver Primary from ClusterState.RoleLeader("driver").
Akka offers two different notions of a "first" member and they are not the same
one: role leader is the lowest-ADDRESSED Up member (host, then port), while
ClusterSingletonManager places singletons on the OLDEST — lowest up-number.
They agree on a freshly-formed cluster, which is why every existing test passed.
They diverge after any restart: the restarted node re-joins as the youngest while
keeping its address, so if it holds the lower address it becomes role leader while
the singletons stay put. The snapshot would then name a Primary that is not
hosting the work, and every Primary-gated surface follows it — inbound device
writes, native-alarm acks, the fleet-wide alerts emit, and the alarm-history
drain would all enable on the wrong node while the node actually running the
singletons stayed gated off.
BuildSnapshot now selects the oldest Up member carrying the driver role, matching
singleton placement. Leaving members are excluded: a node handing its singletons
over must not be named Primary.
NodeRedundancyState.IsRoleLeaderForDriver is renamed IsDriverPrimary, and
NodeHealthInputs.IsDriverRoleLeader likewise. Keeping the old names would have
left the wire contract asserting a derivation the code no longer uses — the same
drift that made this defect invisible.
Proven by a real two-node cluster rather than a mock. RedundancyPrimaryElectionTests
binds the first-joining node to the HIGHER port, so oldest and lowest-address name
different nodes, and includes a fixture assertion that the divergence actually
occurred — without it the real assertion could pass for the wrong reason. Positive
control: restoring the RoleLeader derivation turns exactly the two election tests
red while the fixture check stays green.
Runtime.Tests 440 passed, ControlPlane.Tests 82, Cluster.Tests 36.
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
Ports the sister project's live-proven fix (ScadaBridge cf3bd52f). OtOpcUa ran
the identical configuration it indicts: keep-oldest with down-if-alone = on.
Akka.NET 1.5.62's KeepOldest.OldestDecision only lets the down-if-alone branch
rescue a side holding >= 2 members. A two-node cluster losing a peer is always a
1-vs-1 split, so the branch never fires, the lone survivor falls through to
DownReachable and downs ITSELF, and run-coordinated-shutdown-when-down then
terminates it. down-if-alone is a 3+-node feature and does not do what its name
suggests for a pair: a crash of the oldest node is a total outage, which is the
exact failure the redundancy pair exists to absorb.
Cluster:SplitBrainResolverStrategy now selects the provider, defaulting to
auto-down: Akka's AutoDowning with auto-down-unreachable-after = 15s, so the
leader among the reachable members downs the unreachable peer and a crash of
either node fails over in place. keep-oldest remains available for deployments
that would rather take an outage than ever run dual-active during a real
partition. An unrecognised value fails the host at startup rather than falling
through to the fatal default.
The tests assert the EFFECTIVE configuration — they start a real host through
WithOtOpcUaClusterBootstrap and read akka.cluster.downing-provider-class back off
the running ActorSystem. This is not incidental. The first draft inlined the
three calls the bootstrap makes instead of calling it, which pinned the test's
own wiring: sabotaging production's HOCON precedence left all 36 green. The
prior file had the same shape at a smaller scale, asserting only that
BuildClusterOptions returned a KeepOldestOption — true, and true of a
configuration that cannot fail over. Positive control: making BuildDowningHocon
emit nothing for auto-down turns exactly the two effective-config guards red.
Measured while verifying rather than assumed: HoconAddMode.Append also wins here,
purely because it is added last, so the mode name is not the guarantee. The
comment now says so instead of asserting a precedence rule that does not hold.
Not yet live-drilled on OtOpcUa. The docker-dev rig is a single six-node mesh
where the 1-vs-1 pathology cannot occur; the kill-the-oldest drill belongs with
the per-cluster mesh work that makes every mesh exactly two nodes (design doc
6.2 / Phase 0a). Recorded as an outstanding gate in docs/Redundancy.md.
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
Akka.Cluster.Hosting's WithClustering enables an SBR downing provider by default
(applies SplitBrainResolverOption.Default when ClusterOptions.SplitBrainResolver is
null), which reads the pre-existing akka.conf keep-oldest block. So the cluster was
NOT running NoDowning before Critical 1 and hard-crash failover already worked — the
typed KeepOldestOption is reinforcing/explicit-in-code, not the sole activator.
Corrects the inaccurate 'HOCON inert / NoDowning / never fails over' framing in:
- ServiceCollectionExtensions.BuildClusterOptions XML comment
- akka.conf split-brain-resolver comment
- docs/Redundancy.md Split-brain section
- SplitBrainResolverActivationTests summary + assertion message (+ method rename)
No code revert (the typed option is correct belt-and-suspenders). Cluster.Tests 29/29.
Surfaced by the #9 hard-kill failover negative control.
The split-brain-resolver HOCON block in akka.conf was inert — nothing
registered a downing provider, so the cluster ran Akka's default NoDowning:
a hard-crashed node was never downed, cluster singletons + the driver
role-leader never failed over, and a partition left both redundancy sides
at ServiceLevel 240 indefinitely.
Activate the resolver via the typed ClusterOptions.SplitBrainResolver
(KeepOldestOption { DownIfAlone = true }) in a new testable
BuildClusterOptions helper. keep-oldest + down-if-alone is correct for a
2-node warm-redundancy pair. HOCON keeps stable-after (typed option can't
express it) + a cross-reference comment. Deterministic unit guard added;
end-to-end verified the resolved ActorSystem config carries the provider
and stable-after survives the HOCON/Hosting merge. Redundancy.md corrected.
Hard-kill failover integration test tracked as a follow-up (effort-M harness).
Code review at HEAD 7286d320. Cluster-001 (SeedFromCurrentState reads from one
snapshot), Cluster-003 (HoconLoader double-dispose), Cluster-004 (stale akka.conf
header), Cluster-005 (ServiceLevelCalculator tests added to Cluster.Tests). Cluster-002
deferred (no production caller).
All five suppressed advisories are now resolved at baseline/resolved versions,
so every NuGetAuditSuppress is removed repo-wide:
- System.Security.Cryptography.Xml (GHSA-37gx-xxp4-5rgx / GHSA-w3x6-4m5h-cxqf)
-> fixed by the .NET 10 baseline (10.0.6)
- OPCFoundation Opc.Ua.Core (GHSA-h958-fxgg-g7w3) -> fixed at resolved 1.5.378.106
Two were still live and are now patched via direct security pins:
- OpenTelemetry.Api 1.9.0 -> 1.15.3 (GHSA-g94r-2vxg-569j) pinned in Cluster;
Runtime/ControlPlane/AdminUI + tests inherit via project reference
- Tmds.DBus.Protocol 0.20.0 -> 0.21.3 (GHSA-xrw6-gwf8-vvr9) pinned in Client.UI
Also correct the Historian sidecar runtime comments (x86 -> x64, matching the
csproj PlatformTarget). Solution audit: 0 vulnerable packages; full build clean.
Adds <summary>, <param>, <typeparam>, and <inheritdoc/> tags to public
members surfaced by commentchecker — resolves 5,847 of 5,869 issues
(99.6%) across three /fixdocs passes.
Joseph Doherty