historization: crash failover leaves a ~30 s capture gap (Primary unreachable but not downed) #672

New Issue

2026-10-11T04:41:43-04:00

dohertj2 commented

2026-10-11 04:41:43 -04:00

Found by the QuestDB historian live gate (docs/plans/2026-10-10-questdb-historian-live-gate.md, §Gate findings). The gap is backend-independent.

Observed: I ran docker kill (SIGKILL) on the recording Primary (central-2) at 07:45:58. After that:

The next historized FastUInt1 sample is at 07:46:29.
Central-1 logged "now the recording Primary" only at 07:46:30, when the killed node rejoined as a new incarnation.
About 30 s of values were captured by neither node.

Cause: #537 gates capture to the recording Primary. A crashed Primary stays unreachable but not downed for the Akka failure-detector window plus auto-down-unreachable-after = 15s. During that window the standby is not promoted, so nobody captures.

Contrast: a clean docker stop hands over immediately: leg 4 of the same gate showed 1× recording with no gap across two failovers. Earlier gates (#491/#537, the SqlServer gate) exercised only the clean path.

Options to consider:

Let the standby capture into its outbox while the Primary is unreachable. This accepts a duplicate window, which insert-if-absent backends (SqlServer, QuestDB) absorb idempotently; the Gateway backend would not.
Tie the capture gate to reachability rather than membership.
Document the gap as the accepted cost of #537.

Found by the QuestDB historian live gate (`docs/plans/2026-10-10-questdb-historian-live-gate.md`, §Gate findings). The gap is backend-independent. **Observed:** I ran `docker kill` (SIGKILL) on the recording Primary (central-2) at 07:45:58. After that: - The next historized FastUInt1 sample is at 07:46:29. - Central-1 logged "now the recording Primary" only at 07:46:30, when the killed node rejoined as a new incarnation. - About 30 s of values were captured by neither node. **Cause:** #537 gates capture to the recording Primary. A crashed Primary stays *unreachable but not downed* for the Akka failure-detector window plus `auto-down-unreachable-after = 15s`. During that window the standby is not promoted, so nobody captures. **Contrast:** a clean `docker stop` hands over immediately: leg 4 of the same gate showed 1× recording with no gap across two failovers. Earlier gates (#491/#537, the SqlServer gate) exercised only the clean path. **Options to consider:** - Let the standby capture into its outbox while the Primary is unreachable. This accepts a duplicate window, which insert-if-absent backends (SqlServer, QuestDB) absorb idempotently; the Gateway backend would not. - Tie the capture gate to reachability rather than membership. - Document the gap as the accepted cost of #537.

dohertj2 referenced this issue

2026-10-11 04:41:48 -04:00

historian: QuestDB backend (design + plan) #666

dohertj2 referenced this issue from a commit

2026-10-11 10:17:47 -04:00

fix(historian): capture while the recording Primary is unreachable (#672)

dohertj2 referenced this issue from a commit

2026-10-11 10:17:47 -04:00

Merge fix/672-capture-gap: capture while the recording Primary is unreachable (#672)

dohertj2 commented

2026-10-11 10:18:00 -04:00

Fixed in 91434e7f (merged to master).

DriverHostActor now keeps its own ReachabilityEvent + MemberUp subscription (InitialStateAsEvents, handled in Steady/Applying/Stale), caches the last redundancy snapshot, and on a reachability change re-derives and republishes only the historization verdict with unreachable peers excluded. A standby stood down for its mesh Primary (#537) therefore starts capturing as soon as that Primary is reported UNREACHABLE instead of after auto-down.

Knob ContinuousHistorization:CaptureWhilePrimaryUnreachable (default true; false = pre-#672).
The alarm drain gate and the data-plane Primary gate (ShouldServiceAsPrimary / PublishPrimaryVerdict) are deliberately unchanged — tests pin both.
A removed member stays marked unreachable until MemberUp at that address (un-marking it would re-close the gate on a dead node).
Cost (documented in Historian.md/Redundancy.md): duplicate windows while a live Primary is merely unreachable; idempotent on SqlServer/QuestDB only for source-timestamped drivers.

Live gate (docker-dev): SIGKILL of the recording Primary — capture gap 13 s (was ~31 s before the fix).

Follow-up filed for a few minor test-strengthening items.

Fixed in `91434e7f` (merged to master). `DriverHostActor` now keeps its own `ReachabilityEvent` + `MemberUp` subscription (`InitialStateAsEvents`, handled in Steady/Applying/Stale), caches the last redundancy snapshot, and on a reachability change re-derives and republishes **only** the historization verdict with unreachable peers excluded. A standby stood down for its mesh Primary (#537) therefore starts capturing as soon as that Primary is reported UNREACHABLE instead of after auto-down. - Knob `ContinuousHistorization:CaptureWhilePrimaryUnreachable` (default `true`; `false` = pre-#672). - The alarm drain gate and the data-plane Primary gate (`ShouldServiceAsPrimary` / `PublishPrimaryVerdict`) are deliberately unchanged — tests pin both. - A removed member stays marked unreachable until `MemberUp` at that address (un-marking it would re-close the gate on a dead node). - Cost (documented in Historian.md/Redundancy.md): duplicate windows while a live Primary is merely unreachable; idempotent on SqlServer/QuestDB only for source-timestamped drivers. **Live gate (docker-dev):** SIGKILL of the recording Primary — capture gap **13 s** (was ~31 s before the fix). Follow-up filed for a few minor test-strengthening items.

dohertj2 closed this issue

2026-10-11 10:18:00 -04:00

dohertj2 referenced this issue

2026-10-11 10:18:15 -04:00

Small follow-ups from the #667-#673 batch (retention cancel path, #672 tests, FOCAS flake, cosmetics, rig cert) #678

dohertj2 referenced this issue from a commit

2026-10-11 13:20:15 -04:00

test(runtime): #678.2 #672 initial-state unreachable peer + strengthen Self_unreachable_is_ignored

dohertj2 referenced this issue from a commit

2026-10-11 13:20:15 -04:00

Merge fix/678-small: retention cancel/fallback tests, #672 tests, FOCAS race, TTL log wording, debug overlay, role-overlay suffix fix (#678)

dohertj2 referenced this issue