fix(historian): capture while the recording Primary is unreachable (#672)

A SIGKILLed Primary stays an unreachable-but-not-downed member until
auto-down, and the redundancy singleton usually dies with it, so the
standby's cached snapshot kept naming it Primary and nobody captured for
~30 s. DriverHostActor now holds its own ReachabilityEvent (+MemberUp)
subscription, caches the last snapshot, and excludes unreachable peers
from the historization peer-is-Primary predicate only, republishing just
the capture verdict on a reachability change. Alarm drain and the Primary
data-plane verdict are unchanged. Gated by
ContinuousHistorization:CaptureWhilePrimaryUnreachable (default true).

Tests: message-driven cases with real ReachabilityEvents (incl. a
falsification control on the drain/data-plane verdicts) plus a real
two-node cluster crash test with downing disabled.
This commit is contained in:
Joseph Doherty
2026-10-11 08:26:39 -04:00
parent db6a53036b
commit 1b2c36eb87
11 changed files with 573 additions and 21 deletions
@@ -98,6 +98,13 @@ public static class PrimaryGatePolicy
/// promoted node takes to see its first snapshot. That is inherent to any non-duplicating
/// design; the alternative — both nodes capturing — is what #537 was filed about.
/// </para>
/// <para>
/// Gitea #672: on a CRASH the newly promoted node may not see a snapshot until auto-down (the
/// redundancy singleton died with the old Primary), so the caller excludes a Primary the cluster
/// failure detector reports unreachable from <paramref name="addressSpaceSharingPeerIsPrimary"/> —
/// shrinking the gap to the failure-detector window. This method is unchanged by that; it only
/// sees the narrower input.
/// </para>
/// </remarks>
/// <param name="localRole">This node's last-known redundancy role, or <c>null</c> when unknown.</param>
/// <param name="addressSpaceSharingPeerIsPrimary">