fix(historian): capture while the recording Primary is unreachable (#672)

A SIGKILLed Primary stays an unreachable-but-not-downed member until
auto-down, and the redundancy singleton usually dies with it, so the
standby's cached snapshot kept naming it Primary and nobody captured for
~30 s. DriverHostActor now holds its own ReachabilityEvent (+MemberUp)
subscription, caches the last snapshot, and excludes unreachable peers
from the historization peer-is-Primary predicate only, republishing just
the capture verdict on a reachability change. Alarm drain and the Primary
data-plane verdict are unchanged. Gated by
ContinuousHistorization:CaptureWhilePrimaryUnreachable (default true).

Tests: message-driven cases with real ReachabilityEvents (incl. a
falsification control on the drain/data-plane verdicts) plus a real
two-node cluster crash test with downing disabled.
This commit is contained in:
Joseph Doherty
2026-10-11 08:26:39 -04:00
parent db6a53036b
commit 1b2c36eb87
11 changed files with 573 additions and 21 deletions
+32
View File
@@ -242,6 +242,7 @@ and is registered the same way for every backend.
| `DrainConcurrency` | int | `8` | Historian tags a single drain pass writes **concurrently**. One peeked batch is grouped per historian tag and each group is an independent gateway round-trip, so a sequential pass cost `tags × latency`; this window makes it `⌈tags / concurrency⌉ × latency` while bounding how many round-trips can be outstanding against the gateway at once. `1` restores the fully sequential drain; a value `< 1` is clamped to 1 (with a `Validate()` warning). **The ack is unchanged** — still all-or-nothing after every group has reported, and the #538 per-tag quarantine still keys on "another group wrote this pass". |
| `Capacity` | int | `500000` | Max un-acked outbox entries before the drop-oldest policy sheds the front of the queue; `0` = unbounded. Roughly 50 MB of FasterLog. The #538 **disk backstop** for the one case the per-tag quarantine deliberately cannot resolve — a historian unreachable or refusing *every* write, where nothing proves any particular tag is at fault so the recorder correctly retains. |
| `MaxWriteAttempts` | int | `5` | #538 poison-tag budget: consecutive failed drain passes one historian tag may accumulate before its queued entries are abandoned and the FIFO head is allowed to advance. **Counted only across passes where *other* tags wrote successfully**, so an ordinary gateway outage never advances it. Clamped to at least 1; each abandonment logs a Warning naming the tag. |
| `CaptureWhilePrimaryUnreachable` | bool | `true` | #672: a standby whose capture is stood down for its mesh-partner Primary (#537) resumes capturing as soon as the cluster failure detector reports that Primary **unreachable**, instead of waiting for auto-down + a fresh redundancy snapshot. Shrinks a crash-failover capture gap from ~25 s to the failure-detector window (~10–12 s), at the cost of short duplicate windows — see [the crash-failover note](#a-crashed-primary-capture-gap-672). `false` restores the pre-#672 behaviour. Never affects the alarm drain or the Primary data-plane gate. |
| `MinBackoffSeconds` | double | `1` | Initial retry backoff after a failed drain pass. |
| `MaxBackoffSeconds` | double | `30` | Cap on the exponential retry backoff. |
@@ -381,6 +382,37 @@ trend is not. The residual cost is a capture gap across a failover, bounded by h
node takes to see its first redundancy snapshot — inherent to any non-duplicating design. Suppressed
values are counted in `RecorderStatus.SuppressedNotPrimary`, and each gate transition logs once.
### A crashed Primary: capture gap (#672)
On a **crash** (SIGKILL / `docker kill`) that "first snapshot" can be a long way off. The dead Primary
stays a cluster member — *unreachable, not downed* — for the failure-detector window plus
`auto-down-unreachable-after` (`acceptable-heartbeat-pause` 10 s + 15 s in `akka.conf`), and the
redundancy singleton almost always lived on it (it is the oldest member), so no snapshot is published
until auto-down re-homes the singleton. The standby's cached snapshot keeps naming the dead node Primary
and **nobody captures** for ~30 s (measured on the QuestDB live gate).
`DriverHostActor` therefore follows cluster reachability (its own `ClusterEvent.ReachabilityEvent`
subscription) and, with `ContinuousHistorization:CaptureWhilePrimaryUnreachable` on (the default),
**excludes an unreachable peer from the historization "a mesh-partner Primary is recording" question**,
re-deriving and republishing only the capture verdict from the last snapshot. A node at the same address
coming Up again clears the mark; a reachable-again Primary closes the gate again. It logs
`capturing historized values while recording Primary <id> is unreachable` once on entry and once on
revert. The alarm drain and the Primary data-plane verdict (writes, native acks, the Sparkplug host claim)
are **not** touched — a false allow there is a dual claimant, not a duplicate sample.
Trade-offs, all in the duplicate direction #537 already accepts:
- **The gap shrinks to the failure-detector window** (~10–12 s), not to zero — detection is the floor.
- **Duplicates during the overlap.** A Primary that is only *briefly* unreachable (GC pause, network blip)
overlaps the standby until it is reachable again, and under a genuine **partition** both sides capture
from detection onward (as they already did after auto-down, now ~15 s earlier). SqlServer and QuestDB
absorb a duplicate of a **source-timestamped** sample idempotently (`(TagId, SourceTimeUtc)` key /
dedup), but a driver with **no source timestamp** (e.g. Modbus: `SourceTimestampUtc` is null, so each
node stamps its own poll time) writes a duplicate row on **every** backend, and the **Gateway** backend
stores duplicates of everything.
- Set `CaptureWhilePrimaryUnreachable=false` to keep the pre-#672 behaviour (no duplicates from this
path, the full ~30 s hole on a crash).
---
## Tag auto-provisioning (EnsureTags)