Continuous historization: both pair nodes write duplicate live values (no Primary gate) #537

New Issue

2026-08-01T19:58:20-04:00

dohertj2 commented

2026-08-01 19:58:20 -04:00

Split out of #491's live gate (2026-08-01) — a design decision that has not been made, recorded rather than silently changed.

ContinuousHistorizationRecorder has no Primary gate. Both nodes of a redundant pair tap their own dependency mux, append to their own outbox, and drain to the historian — so the historian receives duplicate live values for every historized tag.

Measured on the docker-dev central pair against a real gateway: the gateway's WriteLiveValues{outcome="ok"} counter advanced twice per value with both centrals up, and halved when central-2 was stopped.

This contrasts with the alarm store-and-forward sink, which is Primary-gated — SqliteStoreAndForwardSink/LocalDbStoreAndForwardSink take a drainGate delegate fed from the IRedundancyRoleView singleton that DriverHostActor publishes its PrimaryGatePolicy verdict to, precisely so a pair does not double-deliver (a gated node reports HistorianDrainState.NotPrimary).

The decision is not obviously "add the gate":

Duplicate analog samples may be harmless — the two nodes stamp the same driver TimestampUtc for the same mux fan-out, so the historian may fold them. That is unverified.
Gating introduces a failover window with no capture at all, which the alarm buffer tolerates (it is a durable queue that drains later) but a live-value recorder does not — the outbox on the newly-demoted node would hold values nobody drains, while the newly-promoted node has no history of the gap.
A third option is to gate the drain rather than the capture, so both nodes keep appending durably and only the Primary writes — matching the alarm sink exactly, and making a failover lose nothing.

Needs: confirm the historian's same-timestamp folding behaviour, then pick between "leave duplicated", "gate the drain" (probably right), and "gate the capture" (probably wrong).

Evidence: docs/plans/2026-08-01-continuous-historization-live-gate.md §"Secondary finding".

**Split out of #491's live gate (2026-08-01) — a design decision that has not been made, recorded rather than silently changed.** `ContinuousHistorizationRecorder` has **no Primary gate**. Both nodes of a redundant pair tap their own dependency mux, append to their own outbox, and drain to the historian — so the historian receives **duplicate live values for every historized tag**. Measured on the docker-dev central pair against a real gateway: the gateway's `WriteLiveValues{outcome="ok"}` counter advanced **twice per value** with both centrals up, and **halved** when `central-2` was stopped. This contrasts with the alarm store-and-forward sink, which *is* Primary-gated — `SqliteStoreAndForwardSink`/`LocalDbStoreAndForwardSink` take a `drainGate` delegate fed from the `IRedundancyRoleView` singleton that `DriverHostActor` publishes its `PrimaryGatePolicy` verdict to, precisely so a pair does not double-deliver (a gated node reports `HistorianDrainState.NotPrimary`). **The decision is not obviously "add the gate":** - Duplicate analog samples may be harmless — the two nodes stamp the same driver `TimestampUtc` for the same mux fan-out, so the historian may fold them. That is unverified. - Gating introduces a **failover window with no capture at all**, which the alarm buffer tolerates (it is a durable queue that drains later) but a live-value recorder does not — the outbox on the newly-demoted node would hold values nobody drains, while the newly-promoted node has no history of the gap. - A third option is to gate the **drain** rather than the **capture**, so both nodes keep appending durably and only the Primary writes — matching the alarm sink exactly, and making a failover lose nothing. Needs: confirm the historian's same-timestamp folding behaviour, then pick between "leave duplicated", "gate the drain" (probably right), and "gate the capture" (probably wrong). Evidence: `docs/plans/2026-08-01-continuous-historization-live-gate.md` §"Secondary finding".

dohertj2 referenced this issue

2026-08-04 11:29:42 -04:00

test(historian): live end-to-end gate for continuous-historization value capture #491

dohertj2 referenced this issue from a commit

2026-08-04 12:36:25 -04:00

fix(historian): gate historization capture on the Primary, quarantine a poison tag (#537, #538)

dohertj2 referenced this issue from a commit

2026-08-04 12:36:25 -04:00

docs(historian): live gate for #537 and #538 — both PASSED

dohertj2 commented

2026-08-04 12:37:05 -04:00

FIXED + live-gated — e7fbe977 (fix) / 4bdce910 (gate evidence), master 4bdce910.

The decision went to gating the capture, not the drain — the reverse of what this issue leaned
towards ("gate the drain … probably right"). That preference came from matching the alarm sink, but the
two queues are not the same shape: the alarm buffer lives in LocalDb and replicates, so the Secondary
already holds the Primary's rows and gating the drain loses nothing. The historization outbox is a
node-local FasterLog that replicates to nobody, so a drain-only gate leaves the standby appending forever
to a queue no one empties — unbounded disk, then a flood of stale duplicates the moment it is promoted.
Gating the capture leaves it holding nothing, and entries a node already captured are still drained by
that node, so a demotion loses nothing recorded.

The failover window this issue flagged is real and accepted: it is bounded by how long the promoted node
takes to see its first redundancy snapshot, and the gate is read per value rather than latched so
promotion resumes capture on the very next sample. The live gate exercises exactly that.

⚠️ The peer that matters is one recording the SAME REFS, not one holding the same rows. A node stands
down only for a Primary in its own per-cluster mesh; a node carrying no cluster-scoped role cannot
establish that and never stands down. Under a fleet-wide election the elected Primary is usually running a
different cluster's configuration, and deferring to it would end this cluster's historization silently —
the drain would report success on an empty outbox forever. That is the same scope mismatch that once
suspended the alarm drain on every node of the docker-dev rig at once.

An unknown role captures, matching the alarm drain: a duplicate sample is recoverable, a hole in the trend
is not.

New verdict on RedundancyRoleView behind IHistorizationRoleView, registered unconditionally on
driver nodes rather than alongside the alarm section's IRedundancyRoleView — piggybacking on that would
leave any pair running historization without alarm history ungated, which is the #511 registration accident.

The historian's same-timestamp folding behaviour, which this issue asked to confirm first, turned out not
to be on the critical path: it would only decide whether the duplicates were tolerable, and the fix
removes them either way.

Live gate (2026-08-04, docker-dev central pair + real gateway + real AVEVA Historian)

Leg	Observation
Both nodes up	Secondary logged the suppression line once; its outbox file froze while the Primary's kept growing
Steady state	6 value changes x 2 tags -> `WriteLiveValues{ok}` +12 exactly (was +24 pre-fix)
Failover	Primary stopped -> survivor logged `now the recording Primary … resuming value capture`, captured every subsequent change
Return	Restarted node demoted, logged suppression; rate stayed at one node

Offline tests are falsified: latching the gate at construction reddens only
The_capture_gate_is_re_read_per_value_so_promotion_resumes_capture; treating any Primary as a scope peer
reddens only the two scope-mismatch cases.

Evidence: docs/plans/2026-08-01-continuous-historization-live-gate.md §"Live gate for the two fixes".
Design: docs/Historian.md §"Only the recording Primary captures"; docs/Redundancy.md.

**FIXED + live-gated** — `e7fbe977` (fix) / `4bdce910` (gate evidence), master `4bdce910`. The decision went to **gating the capture, not the drain** — the reverse of what this issue leaned towards ("gate the drain … probably right"). That preference came from matching the alarm sink, but the two queues are not the same shape: the alarm buffer lives in LocalDb and **replicates**, so the Secondary already holds the Primary's rows and gating the drain loses nothing. The historization outbox is a node-local FasterLog that replicates to nobody, so a drain-only gate leaves the standby appending forever to a queue no one empties — unbounded disk, then a flood of stale duplicates the moment it is promoted. Gating the capture leaves it holding nothing, and entries a node already captured are still drained by that node, so a demotion loses nothing recorded. The failover window this issue flagged is real and accepted: it is bounded by how long the promoted node takes to see its first redundancy snapshot, and the gate is read **per value** rather than latched so promotion resumes capture on the very next sample. The live gate exercises exactly that. ⚠️ **The peer that matters is one recording the SAME REFS, not one holding the same rows.** A node stands down only for a Primary in its own per-cluster mesh; a node carrying no cluster-scoped role cannot establish that and never stands down. Under a fleet-wide election the elected Primary is usually running a different cluster's configuration, and deferring to it would end this cluster's historization *silently* — the drain would report success on an empty outbox forever. That is the same scope mismatch that once suspended the alarm drain on every node of the docker-dev rig at once. An unknown role captures, matching the alarm drain: a duplicate sample is recoverable, a hole in the trend is not. New verdict on `RedundancyRoleView` behind `IHistorizationRoleView`, registered **unconditionally** on driver nodes rather than alongside the alarm section's `IRedundancyRoleView` — piggybacking on that would leave any pair running historization without alarm history ungated, which is the #511 registration accident. The historian's same-timestamp folding behaviour, which this issue asked to confirm first, turned out not to be on the critical path: it would only decide whether the *duplicates* were tolerable, and the fix removes them either way. ### Live gate (2026-08-04, docker-dev central pair + real gateway + real AVEVA Historian) | Leg | Observation | |---|---| | Both nodes up | Secondary logged the suppression line once; its outbox file froze while the Primary's kept growing | | Steady state | 6 value changes x 2 tags -> `WriteLiveValues{ok}` **+12 exactly** (was +24 pre-fix) | | Failover | Primary stopped -> survivor logged `now the recording Primary … resuming value capture`, captured every subsequent change | | Return | Restarted node demoted, logged suppression; rate stayed at one node | Offline tests are falsified: latching the gate at construction reddens only `The_capture_gate_is_re_read_per_value_so_promotion_resumes_capture`; treating any Primary as a scope peer reddens only the two scope-mismatch cases. Evidence: `docs/plans/2026-08-01-continuous-historization-live-gate.md` §"Live gate for the two fixes". Design: `docs/Historian.md` §"Only the recording Primary captures"; `docs/Redundancy.md`.

dohertj2 closed this issue

2026-08-04 12:37:05 -04:00

dohertj2 referenced this issue

2026-10-11 04:41:43 -04:00

historization: crash failover leaves a ~30 s capture gap (Primary unreachable but not downed) #672

dohertj2 referenced this issue

2026-10-11 10:18:00 -04:00

historization: crash failover leaves a ~30 s capture gap (Primary unreachable but not downed) #672

Sign in to join this conversation.