fix(historian): capture while the recording Primary is unreachable (#672)

A SIGKILLed Primary stays an unreachable-but-not-downed member until
auto-down, and the redundancy singleton usually dies with it, so the
standby's cached snapshot kept naming it Primary and nobody captured for
~30 s. DriverHostActor now holds its own ReachabilityEvent (+MemberUp)
subscription, caches the last snapshot, and excludes unreachable peers
from the historization peer-is-Primary predicate only, republishing just
the capture verdict on a reachability change. Alarm drain and the Primary
data-plane verdict are unchanged. Gated by
ContinuousHistorization:CaptureWhilePrimaryUnreachable (default true).

Tests: message-driven cases with real ReachabilityEvents (incl. a
falsification control on the drain/data-plane verdicts) plus a real
two-node cluster crash test with downing disabled.
This commit is contained in:
Joseph Doherty
2026-10-11 08:26:39 -04:00
parent db6a53036b
commit 1b2c36eb87
11 changed files with 573 additions and 21 deletions
+10 -1
View File
@@ -546,7 +546,16 @@ Under warm/hot redundancy both cluster nodes run `ScriptedAlarmHostActor` and ev
node. ⚠️ **The peer it stands down for must record the same refs** — a Primary in its own per-cluster
mesh — not merely hold the same rows; a node with no cluster-scoped role never stands down, because a
fleet-wide election's Primary usually runs a different cluster's configuration and deferring to it would
end this cluster's historization silently. See [Historian.md](Historian.md).
end this cluster's historization silently. **Crash failover (Gitea #672):** a SIGKILLed Primary stays an
unreachable-but-not-downed member until auto-down (10 s failure detector + 15 s), and the redundancy
singleton usually died with it, so the standby's cached snapshot kept naming it Primary and nobody
captured for ~30 s. `DriverHostActor` now follows `ClusterEvent.ReachabilityEvent` and excludes an
**unreachable** peer from the historization peer-is-Primary question only (re-deriving the capture verdict
from the cached snapshot), shrinking the gap to the failure-detector window; gated by
`ContinuousHistorization:CaptureWhilePrimaryUnreachable` (default `true`). The alarm drain and the Primary
data-plane verdict are **unchanged** — they still wait for a snapshot. The cost is duplicate capture while a
live Primary is merely unreachable, and from detection onward under a real partition. See
[Historian.md](Historian.md) §"A crashed Primary: capture gap (#672)".
Net effect: each alarm transition appears **once** on `/alerts` and would historize once, not once per node.