fix(historian): capture while the recording Primary is unreachable (#672)

A SIGKILLed Primary stays an unreachable-but-not-downed member until
auto-down, and the redundancy singleton usually dies with it, so the
standby's cached snapshot kept naming it Primary and nobody captured for
~30 s. DriverHostActor now holds its own ReachabilityEvent (+MemberUp)
subscription, caches the last snapshot, and excludes unreachable peers
from the historization peer-is-Primary predicate only, republishing just
the capture verdict on a reachability change. Alarm drain and the Primary
data-plane verdict are unchanged. Gated by
ContinuousHistorization:CaptureWhilePrimaryUnreachable (default true).

Tests: message-driven cases with real ReachabilityEvents (incl. a
falsification control on the drain/data-plane verdicts) plus a real
two-node cluster crash test with downing disabled.
This commit is contained in:
Joseph Doherty
2026-10-11 08:26:39 -04:00
parent db6a53036b
commit 1b2c36eb87
11 changed files with 573 additions and 21 deletions
@@ -54,6 +54,7 @@ public sealed class ContinuousHistorizationOptionsTests
Assert.Equal(8, options.DrainConcurrency);
Assert.Equal(500_000, options.Capacity);
Assert.Equal(5, options.MaxWriteAttempts);
Assert.True(options.CaptureWhilePrimaryUnreachable); // Gitea #672
}
/// <summary>A fully-configured enabled recorder produces no warnings.</summary>