fix(historian): capture while the recording Primary is unreachable (#672)
A SIGKILLed Primary stays an unreachable-but-not-downed member until auto-down, and the redundancy singleton usually dies with it, so the standby's cached snapshot kept naming it Primary and nobody captured for ~30 s. DriverHostActor now holds its own ReachabilityEvent (+MemberUp) subscription, caches the last snapshot, and excludes unreachable peers from the historization peer-is-Primary predicate only, republishing just the capture verdict on a reachability change. Alarm drain and the Primary data-plane verdict are unchanged. Gated by ContinuousHistorization:CaptureWhilePrimaryUnreachable (default true). Tests: message-driven cases with real ReachabilityEvents (incl. a falsification control on the drain/data-plane verdicts) plus a real two-node cluster crash test with downing disabled.
This commit is contained in:
@@ -1408,6 +1408,7 @@ only the recorder + outbox knobs:
|
||||
| `MaxWriteAttempts` | `5` | #538: consecutive failed drain passes one tag may accumulate — counted **only across passes where other tags wrote successfully** — before its queued entries are abandoned so the FIFO head can advance |
|
||||
| `MinBackoffSeconds` | `1` | Initial retry backoff after a failed drain pass |
|
||||
| `MaxBackoffSeconds` | `30` | Cap on the exponential retry backoff |
|
||||
| `CaptureWhilePrimaryUnreachable` | `true` | #672: a standby stood down for its mesh Primary (#537) captures as soon as that Primary is reported UNREACHABLE, instead of after auto-down — crash-failover gap ~30 s ⇒ ~10–12 s, at the cost of duplicate windows (see #537/#672 below). `false` = pre-#672 |
|
||||
|
||||
### Tag auto-provisioning (`IHistorianProvisioning` EnsureTags hook)
|
||||
|
||||
@@ -1547,6 +1548,21 @@ tag); evidence in the same plan doc. Two things to know before touching either.*
|
||||
`RedundancyRoleView` behind **`IHistorizationRoleView`**, registered unconditionally on driver nodes
|
||||
(NOT alongside the alarm `IRedundancyRoleView`, which only exists when that section is enabled — the
|
||||
#511 registration accident).
|
||||
- **#672 — an UNREACHABLE recording Primary does not count (crash-failover gap).** A SIGKILLed Primary
|
||||
stays a member (unreachable, not downed) for 10 s failure detection + 15 s auto-down, and the
|
||||
redundancy singleton died with it, so the standby's cached snapshot kept naming it Primary and
|
||||
*nobody* captured for ~30 s. `DriverHostActor` now holds its **own** `ReachabilityEvent` (+`MemberUp`)
|
||||
subscription — not inside `StartPrimaryVerdictPublishing`, which early-returns with no primary-role
|
||||
source — handled in **all three** behaviours, caches the last snapshot, and on a reachability change
|
||||
re-derives and republishes **only** the historization verdict with unreachable peers excluded. Gap ⇒ the
|
||||
failure-detector window (~10–12 s). Gated by `ContinuousHistorization:CaptureWhilePrimaryUnreachable`
|
||||
(default `true`). ⚠️ **The alarm drain and `ShouldServiceAsPrimary`/`PublishPrimaryVerdict` must NOT
|
||||
adopt this** (a false allow there is a dual claimant, not a duplicate) — the tests assert both stay
|
||||
put. ⚠️ A **removed** member deliberately stays marked unreachable (only `MemberUp` at that address
|
||||
clears it): un-marking it would re-derive from a cached snapshot still naming it Primary and close the
|
||||
gate on a dead node. Cost: duplicates while a live Primary is merely unreachable and under a real
|
||||
partition — idempotent on SqlServer/QuestDB only for source-timestamped drivers (Modbus stamps each
|
||||
node's own poll time ⇒ duplicate rows on every backend); Gateway stores duplicates.
|
||||
|
||||
Re-confirmed live in 2026-06 and **since fixed (#600.1)**: `EnsureTags` provisioning used to fire only for
|
||||
newly-ADDED historized tags, so flipping an existing tag's `historianTagname` (or historization, or DataType)
|
||||
|
||||
Reference in New Issue
Block a user