fix(historian): capture while the recording Primary is unreachable (#672)

A SIGKILLed Primary stays an unreachable-but-not-downed member until
auto-down, and the redundancy singleton usually dies with it, so the
standby's cached snapshot kept naming it Primary and nobody captured for
~30 s. DriverHostActor now holds its own ReachabilityEvent (+MemberUp)
subscription, caches the last snapshot, and excludes unreachable peers
from the historization peer-is-Primary predicate only, republishing just
the capture verdict on a reachability change. Alarm drain and the Primary
data-plane verdict are unchanged. Gated by
ContinuousHistorization:CaptureWhilePrimaryUnreachable (default true).

Tests: message-driven cases with real ReachabilityEvents (incl. a
falsification control on the drain/data-plane verdicts) plus a real
two-node cluster crash test with downing disabled.
This commit is contained in:
Joseph Doherty
2026-10-11 08:26:39 -04:00
parent db6a53036b
commit 1b2c36eb87
11 changed files with 573 additions and 21 deletions
@@ -129,6 +129,25 @@ public sealed class ContinuousHistorizationOptions
/// <summary>Cap (seconds) on the exponential retry backoff after repeated drain failures. Defaults to 30.</summary>
public double MaxBackoffSeconds { get; init; } = 30;
/// <summary>
/// Gitea #672: when <c>true</c> (the default), a redundant node whose capture is stood down for its
/// mesh-partner Primary (Gitea #537) resumes capturing as soon as the cluster failure detector reports
/// that Primary <b>unreachable</b>, instead of waiting for auto-down and a fresh redundancy snapshot.
/// <para>
/// A crashed Primary otherwise leaves a capture hole of failure-detection + auto-down (~25 s on the
/// shipped <c>akka.conf</c>): the redundancy singleton usually died with it, so the standby's last
/// snapshot keeps naming the dead node Primary. With this on, the hole shrinks to the failure-detector
/// window (~10–12 s). The cost is duplicates: during a genuine partition both nodes capture from
/// detection onward, and a briefly-unreachable-but-alive Primary overlaps the standby. SqlServer and
/// QuestDB absorb a duplicate of a source-timestamped sample idempotently; a driver with no source
/// timestamp (e.g. Modbus, stamped with each node's own poll time) writes a duplicate row on every
/// backend, and the Gateway backend stores duplicates. <c>false</c> restores the pre-#672 behaviour.
/// Only the historization capture verdict is affected — never the alarm drain or the Primary data-plane
/// gate.
/// </para>
/// </summary>
public bool CaptureWhilePrimaryUnreachable { get; init; } = true;
/// <summary>Returns operator-facing misconfiguration warnings for an <c>Enabled</c> recorder
/// (empty when disabled or correctly configured). Pure — the registration logs each entry.</summary>
/// <returns>Zero or more human-readable warning messages (never carrying secret values).</returns>