fix(historian): capture while the recording Primary is unreachable (#672)
A SIGKILLed Primary stays an unreachable-but-not-downed member until auto-down, and the redundancy singleton usually dies with it, so the standby's cached snapshot kept naming it Primary and nobody captured for ~30 s. DriverHostActor now holds its own ReachabilityEvent (+MemberUp) subscription, caches the last snapshot, and excludes unreachable peers from the historization peer-is-Primary predicate only, republishing just the capture verdict on a reachability change. Alarm drain and the Primary data-plane verdict are unchanged. Gated by ContinuousHistorization:CaptureWhilePrimaryUnreachable (default true). Tests: message-driven cases with real ReachabilityEvents (incl. a falsification control on the drain/data-plane verdicts) plus a real two-node cluster crash test with downing disabled.
This commit is contained in:
@@ -242,6 +242,7 @@ and is registered the same way for every backend.
|
||||
| `DrainConcurrency` | int | `8` | Historian tags a single drain pass writes **concurrently**. One peeked batch is grouped per historian tag and each group is an independent gateway round-trip, so a sequential pass cost `tags × latency`; this window makes it `⌈tags / concurrency⌉ × latency` while bounding how many round-trips can be outstanding against the gateway at once. `1` restores the fully sequential drain; a value `< 1` is clamped to 1 (with a `Validate()` warning). **The ack is unchanged** — still all-or-nothing after every group has reported, and the #538 per-tag quarantine still keys on "another group wrote this pass". |
|
||||
| `Capacity` | int | `500000` | Max un-acked outbox entries before the drop-oldest policy sheds the front of the queue; `0` = unbounded. Roughly 50 MB of FasterLog. The #538 **disk backstop** for the one case the per-tag quarantine deliberately cannot resolve — a historian unreachable or refusing *every* write, where nothing proves any particular tag is at fault so the recorder correctly retains. |
|
||||
| `MaxWriteAttempts` | int | `5` | #538 poison-tag budget: consecutive failed drain passes one historian tag may accumulate before its queued entries are abandoned and the FIFO head is allowed to advance. **Counted only across passes where *other* tags wrote successfully**, so an ordinary gateway outage never advances it. Clamped to at least 1; each abandonment logs a Warning naming the tag. |
|
||||
| `CaptureWhilePrimaryUnreachable` | bool | `true` | #672: a standby whose capture is stood down for its mesh-partner Primary (#537) resumes capturing as soon as the cluster failure detector reports that Primary **unreachable**, instead of waiting for auto-down + a fresh redundancy snapshot. Shrinks a crash-failover capture gap from ~25 s to the failure-detector window (~10–12 s), at the cost of short duplicate windows — see [the crash-failover note](#a-crashed-primary-capture-gap-672). `false` restores the pre-#672 behaviour. Never affects the alarm drain or the Primary data-plane gate. |
|
||||
| `MinBackoffSeconds` | double | `1` | Initial retry backoff after a failed drain pass. |
|
||||
| `MaxBackoffSeconds` | double | `30` | Cap on the exponential retry backoff. |
|
||||
|
||||
@@ -381,6 +382,37 @@ trend is not. The residual cost is a capture gap across a failover, bounded by h
|
||||
node takes to see its first redundancy snapshot — inherent to any non-duplicating design. Suppressed
|
||||
values are counted in `RecorderStatus.SuppressedNotPrimary`, and each gate transition logs once.
|
||||
|
||||
### A crashed Primary: capture gap (#672)
|
||||
|
||||
On a **crash** (SIGKILL / `docker kill`) that "first snapshot" can be a long way off. The dead Primary
|
||||
stays a cluster member — *unreachable, not downed* — for the failure-detector window plus
|
||||
`auto-down-unreachable-after` (`acceptable-heartbeat-pause` 10 s + 15 s in `akka.conf`), and the
|
||||
redundancy singleton almost always lived on it (it is the oldest member), so no snapshot is published
|
||||
until auto-down re-homes the singleton. The standby's cached snapshot keeps naming the dead node Primary
|
||||
and **nobody captures** for ~30 s (measured on the QuestDB live gate).
|
||||
|
||||
`DriverHostActor` therefore follows cluster reachability (its own `ClusterEvent.ReachabilityEvent`
|
||||
subscription) and, with `ContinuousHistorization:CaptureWhilePrimaryUnreachable` on (the default),
|
||||
**excludes an unreachable peer from the historization "a mesh-partner Primary is recording" question**,
|
||||
re-deriving and republishing only the capture verdict from the last snapshot. A node at the same address
|
||||
coming Up again clears the mark; a reachable-again Primary closes the gate again. It logs
|
||||
`capturing historized values while recording Primary <id> is unreachable` once on entry and once on
|
||||
revert. The alarm drain and the Primary data-plane verdict (writes, native acks, the Sparkplug host claim)
|
||||
are **not** touched — a false allow there is a dual claimant, not a duplicate sample.
|
||||
|
||||
Trade-offs, all in the duplicate direction #537 already accepts:
|
||||
|
||||
- **The gap shrinks to the failure-detector window** (~10–12 s), not to zero — detection is the floor.
|
||||
- **Duplicates during the overlap.** A Primary that is only *briefly* unreachable (GC pause, network blip)
|
||||
overlaps the standby until it is reachable again, and under a genuine **partition** both sides capture
|
||||
from detection onward (as they already did after auto-down, now ~15 s earlier). SqlServer and QuestDB
|
||||
absorb a duplicate of a **source-timestamped** sample idempotently (`(TagId, SourceTimeUtc)` key /
|
||||
dedup), but a driver with **no source timestamp** (e.g. Modbus: `SourceTimestampUtc` is null, so each
|
||||
node stamps its own poll time) writes a duplicate row on **every** backend, and the **Gateway** backend
|
||||
stores duplicates of everything.
|
||||
- Set `CaptureWhilePrimaryUnreachable=false` to keep the pre-#672 behaviour (no duplicates from this
|
||||
path, the full ~30 s hole on a crash).
|
||||
|
||||
---
|
||||
|
||||
## Tag auto-provisioning (EnsureTags)
|
||||
|
||||
Reference in New Issue
Block a user