fix(historian): capture while the recording Primary is unreachable (#672)

A SIGKILLed Primary stays an unreachable-but-not-downed member until
auto-down, and the redundancy singleton usually dies with it, so the
standby's cached snapshot kept naming it Primary and nobody captured for
~30 s. DriverHostActor now holds its own ReachabilityEvent (+MemberUp)
subscription, caches the last snapshot, and excludes unreachable peers
from the historization peer-is-Primary predicate only, republishing just
the capture verdict on a reachability change. Alarm drain and the Primary
data-plane verdict are unchanged. Gated by
ContinuousHistorization:CaptureWhilePrimaryUnreachable (default true).

Tests: message-driven cases with real ReachabilityEvents (incl. a
falsification control on the drain/data-plane verdicts) plus a real
two-node cluster crash test with downing disabled.
This commit is contained in:
Joseph Doherty
2026-10-11 08:26:39 -04:00
parent db6a53036b
commit 1b2c36eb87
11 changed files with 573 additions and 21 deletions
+16
View File
@@ -1408,6 +1408,7 @@ only the recorder + outbox knobs:
| `MaxWriteAttempts` | `5` | #538: consecutive failed drain passes one tag may accumulate — counted **only across passes where other tags wrote successfully** — before its queued entries are abandoned so the FIFO head can advance |
| `MinBackoffSeconds` | `1` | Initial retry backoff after a failed drain pass |
| `MaxBackoffSeconds` | `30` | Cap on the exponential retry backoff |
| `CaptureWhilePrimaryUnreachable` | `true` | #672: a standby stood down for its mesh Primary (#537) captures as soon as that Primary is reported UNREACHABLE, instead of after auto-down — crash-failover gap ~30 s ⇒ ~10–12 s, at the cost of duplicate windows (see #537/#672 below). `false` = pre-#672 |
### Tag auto-provisioning (`IHistorianProvisioning` EnsureTags hook)
@@ -1547,6 +1548,21 @@ tag); evidence in the same plan doc. Two things to know before touching either.*
`RedundancyRoleView` behind **`IHistorizationRoleView`**, registered unconditionally on driver nodes
(NOT alongside the alarm `IRedundancyRoleView`, which only exists when that section is enabled — the
#511 registration accident).
- **#672 — an UNREACHABLE recording Primary does not count (crash-failover gap).** A SIGKILLed Primary
stays a member (unreachable, not downed) for 10 s failure detection + 15 s auto-down, and the
redundancy singleton died with it, so the standby's cached snapshot kept naming it Primary and
*nobody* captured for ~30 s. `DriverHostActor` now holds its **own** `ReachabilityEvent` (+`MemberUp`)
subscription — not inside `StartPrimaryVerdictPublishing`, which early-returns with no primary-role
source — handled in **all three** behaviours, caches the last snapshot, and on a reachability change
re-derives and republishes **only** the historization verdict with unreachable peers excluded. Gap ⇒ the
failure-detector window (~10–12 s). Gated by `ContinuousHistorization:CaptureWhilePrimaryUnreachable`
(default `true`). ⚠️ **The alarm drain and `ShouldServiceAsPrimary`/`PublishPrimaryVerdict` must NOT
adopt this** (a false allow there is a dual claimant, not a duplicate) — the tests assert both stay
put. ⚠️ A **removed** member deliberately stays marked unreachable (only `MemberUp` at that address
clears it): un-marking it would re-derive from a cached snapshot still naming it Primary and close the
gate on a dead node. Cost: duplicates while a live Primary is merely unreachable and under a real
partition — idempotent on SqlServer/QuestDB only for source-timestamped drivers (Modbus stamps each
node's own poll time ⇒ duplicate rows on every backend); Gateway stores duplicates.
Re-confirmed live in 2026-06 and **since fixed (#600.1)**: `EnsureTags` provisioning used to fire only for
newly-ADDED historized tags, so flipping an existing tag's `historianTagname` (or historization, or DataType)