fix(historian): capture while the recording Primary is unreachable (#672)

A SIGKILLed Primary stays an unreachable-but-not-downed member until
auto-down, and the redundancy singleton usually dies with it, so the
standby's cached snapshot kept naming it Primary and nobody captured for
~30 s. DriverHostActor now holds its own ReachabilityEvent (+MemberUp)
subscription, caches the last snapshot, and excludes unreachable peers
from the historization peer-is-Primary predicate only, republishing just
the capture verdict on a reachability change. Alarm drain and the Primary
data-plane verdict are unchanged. Gated by
ContinuousHistorization:CaptureWhilePrimaryUnreachable (default true).

Tests: message-driven cases with real ReachabilityEvents (incl. a
falsification control on the drain/data-plane verdicts) plus a real
two-node cluster crash test with downing disabled.

docs/Configuration.md

@@ -447,6 +447,7 @@ Only the knobs; the backend connection comes from `ServerHistorian`. Outbox reco
 | `ContinuousHistorization:DrainConcurrency` | `8` | Historian tags one drain pass writes **concurrently** — a batch is grouped per tag and each group is its own gateway round-trip, so this turns a sequential `tags × latency` pass into `⌈tags / concurrency⌉ × latency` while bounding outstanding round-trips. `1` = fully sequential; `< 1` is clamped to 1 with a `Validate()` warning. The ack is unchanged (all-or-nothing after every group reports), as is the #538 quarantine rule. |
 | `ContinuousHistorization:Capacity` | `500000` | Max un-acked entries before drop-oldest; `0` = unbounded. The #538 disk backstop for a historian refusing **every** write — the one case the per-tag quarantine cannot resolve. |
 | `ContinuousHistorization:MaxWriteAttempts` | `5` | #538 poison-tag budget: consecutive failed passes one tag may accumulate — **counted only across passes where *other* tags wrote successfully** — before its entries are abandoned so the FIFO head can advance. |
+| `ContinuousHistorization:CaptureWhilePrimaryUnreachable` | `true` | #672: a standby stood down for its mesh-partner Primary (#537) resumes capture as soon as the failure detector reports that Primary **unreachable**, rather than after auto-down + a fresh snapshot — a crash-failover gap of ~25–30 s becomes the failure-detector window (~10–12 s). Cost: duplicate capture while a live Primary is merely unreachable and under a real partition (absorbed idempotently on SqlServer/QuestDB only for source-timestamped drivers; Modbus-style poll-time stamps and the Gateway backend store duplicates). Affects only the capture verdict. `false` = pre-#672 behaviour. |
 | `ContinuousHistorization:MinBackoffSeconds` / `MaxBackoffSeconds` | `1` / `30` | Initial and capped exponential retry backoff after a failed drain pass. |
 
 ### `LocalDb` (pair-local config cache — driver role only)

docs/Historian.md

@@ -242,6 +242,7 @@ and is registered the same way for every backend.
 | `DrainConcurrency` | int | `8` | Historian tags a single drain pass writes **concurrently**. One peeked batch is grouped per historian tag and each group is an independent gateway round-trip, so a sequential pass cost `tags × latency`; this window makes it `⌈tags / concurrency⌉ × latency` while bounding how many round-trips can be outstanding against the gateway at once. `1` restores the fully sequential drain; a value `< 1` is clamped to 1 (with a `Validate()` warning). **The ack is unchanged** — still all-or-nothing after every group has reported, and the #538 per-tag quarantine still keys on "another group wrote this pass". |
 | `Capacity` | int | `500000` | Max un-acked outbox entries before the drop-oldest policy sheds the front of the queue; `0` = unbounded. Roughly 50 MB of FasterLog. The #538 **disk backstop** for the one case the per-tag quarantine deliberately cannot resolve — a historian unreachable or refusing *every* write, where nothing proves any particular tag is at fault so the recorder correctly retains. |
 | `MaxWriteAttempts` | int | `5` | #538 poison-tag budget: consecutive failed drain passes one historian tag may accumulate before its queued entries are abandoned and the FIFO head is allowed to advance. **Counted only across passes where *other* tags wrote successfully**, so an ordinary gateway outage never advances it. Clamped to at least 1; each abandonment logs a Warning naming the tag. |
+| `CaptureWhilePrimaryUnreachable` | bool | `true` | #672: a standby whose capture is stood down for its mesh-partner Primary (#537) resumes capturing as soon as the cluster failure detector reports that Primary **unreachable**, instead of waiting for auto-down + a fresh redundancy snapshot. Shrinks a crash-failover capture gap from ~25 s to the failure-detector window (~10–12 s), at the cost of short duplicate windows — see [the crash-failover note](#a-crashed-primary-capture-gap-672). `false` restores the pre-#672 behaviour. Never affects the alarm drain or the Primary data-plane gate. |
 | `MinBackoffSeconds` | double | `1` | Initial retry backoff after a failed drain pass. |
 | `MaxBackoffSeconds` | double | `30` | Cap on the exponential retry backoff. |
 
@@ -381,6 +382,37 @@ trend is not. The residual cost is a capture gap across a failover, bounded by h
 node takes to see its first redundancy snapshot — inherent to any non-duplicating design. Suppressed
 values are counted in `RecorderStatus.SuppressedNotPrimary`, and each gate transition logs once.
 
+### A crashed Primary: capture gap (#672)
+
+On a **crash** (SIGKILL / `docker kill`) that "first snapshot" can be a long way off. The dead Primary
+stays a cluster member — *unreachable, not downed* — for the failure-detector window plus
+`auto-down-unreachable-after` (`acceptable-heartbeat-pause` 10 s + 15 s in `akka.conf`), and the
+redundancy singleton almost always lived on it (it is the oldest member), so no snapshot is published
+until auto-down re-homes the singleton. The standby's cached snapshot keeps naming the dead node Primary
+and **nobody captures** for ~30 s (measured on the QuestDB live gate).
+
+`DriverHostActor` therefore follows cluster reachability (its own `ClusterEvent.ReachabilityEvent`
+subscription) and, with `ContinuousHistorization:CaptureWhilePrimaryUnreachable` on (the default),
+**excludes an unreachable peer from the historization "a mesh-partner Primary is recording" question**,
+re-deriving and republishing only the capture verdict from the last snapshot. A node at the same address
+coming Up again clears the mark; a reachable-again Primary closes the gate again. It logs
+`capturing historized values while recording Primary <id> is unreachable` once on entry and once on
+revert. The alarm drain and the Primary data-plane verdict (writes, native acks, the Sparkplug host claim)
+are **not** touched — a false allow there is a dual claimant, not a duplicate sample.
+
+Trade-offs, all in the duplicate direction #537 already accepts:
+
+- **The gap shrinks to the failure-detector window** (~10–12 s), not to zero — detection is the floor.
+- **Duplicates during the overlap.** A Primary that is only *briefly* unreachable (GC pause, network blip)
+  overlaps the standby until it is reachable again, and under a genuine **partition** both sides capture
+  from detection onward (as they already did after auto-down, now ~15 s earlier). SqlServer and QuestDB
+  absorb a duplicate of a **source-timestamped** sample idempotently (`(TagId, SourceTimeUtc)` key /
+  dedup), but a driver with **no source timestamp** (e.g. Modbus: `SourceTimestampUtc` is null, so each
+  node stamps its own poll time) writes a duplicate row on **every** backend, and the **Gateway** backend
+  stores duplicates of everything.
+- Set `CaptureWhilePrimaryUnreachable=false` to keep the pre-#672 behaviour (no duplicates from this
+  path, the full ~30 s hole on a crash).
+
 ---
 
 ## Tag auto-provisioning (EnsureTags)

docs/Redundancy.md

@@ -546,7 +546,16 @@ Under warm/hot redundancy both cluster nodes run `ScriptedAlarmHostActor` and ev
   node. ⚠️ **The peer it stands down for must record the same refs** — a Primary in its own per-cluster
   mesh — not merely hold the same rows; a node with no cluster-scoped role never stands down, because a
   fleet-wide election's Primary usually runs a different cluster's configuration and deferring to it would
-  end this cluster's historization silently. See [Historian.md](Historian.md).
+  end this cluster's historization silently. **Crash failover (Gitea #672):** a SIGKILLed Primary stays an
+  unreachable-but-not-downed member until auto-down (10 s failure detector + 15 s), and the redundancy
+  singleton usually died with it, so the standby's cached snapshot kept naming it Primary and nobody
+  captured for ~30 s. `DriverHostActor` now follows `ClusterEvent.ReachabilityEvent` and excludes an
+  **unreachable** peer from the historization peer-is-Primary question only (re-deriving the capture verdict
+  from the cached snapshot), shrinking the gap to the failure-detector window; gated by
+  `ContinuousHistorization:CaptureWhilePrimaryUnreachable` (default `true`). The alarm drain and the Primary
+  data-plane verdict are **unchanged** — they still wait for a snapshot. The cost is duplicate capture while a
+  live Primary is merely unreachable, and from detection onward under a real partition. See
+  [Historian.md](Historian.md) §"A crashed Primary: capture gap (#672)".
 
 Net effect: each alarm transition appears **once** on `/alerts` and would historize once, not once per node.