fix(historian): gate historization capture on the Primary, quarantine a poison tag (#537, #538)

Both defects were found by #491's live gate against a real HistorianGateway and filed rather than
fixed at the time, because each one's fix is a design decision rather than a patch.

#537 — both nodes of a redundant pair captured every value, so the historian received duplicate
live samples for every historized tag (measured: the gateway's write counter advanced twice per
value and halved when one central was stopped).

The recorder now reads a capture gate per value, fed from the new
PrimaryGatePolicy.ShouldRecordHistorizedValues through IHistorizationRoleView — a third verdict on
the same RedundancyRoleView object, registered unconditionally on driver nodes rather than
alongside the alarm section's IRedundancyRoleView (which only exists when that section is enabled;
piggybacking is the #511 registration accident).

Two things are deliberate and easy to get backwards:

  * It gates the CAPTURE, not the drain — the opposite of the alarm sink. That queue is in LocalDb
    and replicates, so the Secondary already holds the Primary's rows. The historization outbox is
    a node-local FasterLog replicating to nobody, so a drain-only gate would leave the standby
    appending forever to a queue no one empties: unbounded disk, then a flood of stale duplicates
    the moment it is promoted. Entries a node already captured are still drained by that node.

  * The peer it stands down for must RECORD THE SAME REFS, not merely hold the same rows — a
    Primary in this node's own per-cluster mesh. A node carrying no cluster-scoped role cannot
    establish that and never stands down: under a fleet-wide election the elected Primary is
    usually running a different cluster's configuration, and deferring to it would end this
    cluster's historization silently, with the drain reporting success on an empty outbox forever.
    That is the scope mismatch that once suspended the alarm drain on every node of the rig.

An unknown role captures, matching the alarm drain — a duplicate sample is recoverable, a hole in
the trend is not.

#538 — the drain writes every per-tag group and then acks all-or-nothing (deliberate: RemoveAsync
truncates THROUGH an id, so partial acking could drop older unwritten entries). One permanently
failing tag therefore pinned the FIFO head while every healthy tag in the batch was re-submitted
to the historian on every pass, indefinitely — duplicate samples and an outbox that never
truncated, observed live at 8.3 MB from a single unprovisionable tag.

The drain now tracks consecutive failures per historian tag and abandons that tag's entries once
it exceeds MaxWriteAttempts (new, default 5), so the head advances. The counter only advances on a
pass where some OTHER tag wrote successfully: that is what separates "this tag is poison" from
"the backend is down". A count-only quarantine looks identical on the poison-tag case and silently
discards the whole backlog during an ordinary gateway restart, turning a duplicate-samples bug
into a lost-history one. Abandonment is also deferred until every failing tag is over budget, so
nothing is discarded on a pass that could not have advanced the head anyway. Each abandonment logs
a Warning naming the tag; the total rides RecorderStatus.AbandonedEntries.

Capacity also gains a non-zero default (500,000 entries) as the disk backstop for the one case the
quarantine deliberately cannot resolve — a historian refusing every write, where retaining is
correct. Set 0 to restore unbounded retention.

Every new test was falsified by reverting the production change it guards, and each one reddens
alone:

  * drop the backend-reachable evidence -> only A_backend_that_fails_every_tag_abandons_nothing
  * never abandon -> only A_permanently_failing_tag_is_abandoned_so_healthy_tags_stop_being_rewritten
  * latch the capture gate at construction -> only The_capture_gate_is_re_read_per_value...
  * treat any Primary as a scope peer -> only the two scope-mismatch cases

Full suite green. (Four OpcUaClient.Browser failures during the run were the opc-plc fixture being
down on the shared Docker host; brought it back up and they pass.)

Claude-Session: https://claude.ai/code/session_015p7wGqy3YpZNCpDzTpGMKo

docs/Historian.md

@@ -133,16 +133,7 @@ source it from there.
 > value. Fixed; pinned by a regression test whose outbox fake completes on a **pool** thread (a fake
 > returning `ValueTask.CompletedTask`, or even `await Task.Yield()`, cannot reproduce it).
 >
-> ⚠️ **One permanently-failing tag re-writes every healthy tag, forever
-> ([#538](https://gitea.dohertylan.com/dohertj2/lmxopcua/issues/538)).** `DrainOnceAsync` writes **every**
-> per-tag group in a batch and then acks **all-or-nothing**, so a tag whose historian tag does not exist
-> stops the FIFO head advancing while the healthy tags' values are re-submitted on every drain pass —
-> duplicate samples at identical timestamps, and an outbox that never truncates (`Capacity` defaults to
-> `0` = unbounded; observed at 8.3 MB). The all-or-nothing ack itself is deliberate.
->
-> ⚠️ **Both nodes of a redundant pair historize the same value
-> ([#537](https://gitea.dohertylan.com/dohertj2/lmxopcua/issues/537))** — the recorder has no Primary gate,
-> unlike the alarm store-and-forward sink, so the historian receives duplicate live values per tag.
+> The gate also found the two defects below, both since fixed.
 >
 > **Related, separately open:** `EnsureTags` provisioning fires only for **newly-added** historized
 > tags (`AddressSpaceApplier` iterates the added set), so flipping an **existing** tag to
@@ -151,6 +142,71 @@ source it from there.
 
 ---
 
+## A poison tag no longer pins the outbox (#538)
+
+`DrainOnceAsync` peeks a batch, writes **every** per-tag group in it, and then acks **all-or-nothing** —
+because the outbox's `RemoveAsync` truncates the FIFO head *through* the acked id, so acking one tag's
+entries could silently drop older, unwritten entries belonging to another. That much is deliberate and
+unchanged.
+
+What was wrong is what happened when a tag failed **permanently**: the head never advanced, so every
+**healthy** tag in the batch was re-submitted to the historian on every drain pass, indefinitely —
+duplicate samples at identical timestamps, and an outbox that never truncated (observed live at 8.3 MB
+from a single unprovisionable tag, with the gateway's `ok` **and** `error` counters climbing together —
+the signature of this bug).
+
+The drain now tracks **consecutive failures per historian tag** and abandons a tag's queued entries once
+it exceeds `ContinuousHistorization:MaxWriteAttempts` (default 5), letting the head advance. Two
+properties matter:
+
+- **A pass where NOTHING wrote abandons nothing.** The counter only advances on a pass where at least one
+  *other* tag wrote successfully — that is what distinguishes "this tag is poison" from "the backend is
+  down". A count-only quarantine looks identical on the poison-tag case and silently discards the whole
+  backlog during an ordinary gateway restart, turning a duplicate-samples bug into a lost-history one.
+  Pinned by a falsification-control test (`A_backend_that_fails_every_tag_abandons_nothing`) that goes red
+  when the evidence check is removed while the poison-tag test stays green.
+- **A tag over budget waits for the others.** Abandonment is deferred until *every* failing tag in the
+  batch is over budget, so a tag is never discarded on a pass that could not have advanced the head anyway.
+
+Each abandonment logs a Warning naming the tag and the discarded sample count, and the total rides
+`RecorderStatus.AbandonedEntries`. The most common cause is a tag the historian has no definition for —
+easy to reach, because `EnsureTags` provisioning runs only for newly-**added** historized tags, so a
+changed `historianTagname` is never provisioned (Issue 5 above).
+
+`ContinuousHistorization:Capacity` now also defaults to **500,000** entries rather than unbounded. It is
+the disk backstop for the one case the quarantine deliberately cannot resolve — a historian refusing
+*every* write, where retaining is correct. Set `0` to restore unbounded retention.
+
+## Only the recording Primary captures (#537)
+
+Both nodes of a redundant pair serve the same address space and see the same dependency-mux fan-out, so
+an ungated recorder wrote **every sample twice** (measured: the gateway's write counter advanced twice
+per value with both centrals up, and halved when one was stopped).
+
+The recorder now reads a capture gate per value, fed from `PrimaryGatePolicy.ShouldRecordHistorizedValues`
+via `IHistorizationRoleView` — the same shape as the alarm store-and-forward sink's `drainGate`, with two
+deliberate differences:
+
+- **It gates the CAPTURE, not the drain.** The alarm queue lives in LocalDb and *replicates*, so the
+  Secondary already holds the Primary's rows and gating the drain loses nothing. The historization outbox
+  is a node-local FasterLog replicating to nobody: gating only its drain would leave the standby appending
+  forever to a queue no one empties — unbounded disk, and a flood of stale duplicates the moment it is
+  promoted. Entries a node already captured are still drained by that node, so a demotion loses nothing
+  recorded.
+- **The relevant peer is one that records the same refs**, not one that holds the same rows. A node stands
+  down only for a Primary in its own per-cluster mesh. A node carrying no cluster-scoped role cannot
+  establish that, so it never stands down: under a fleet-wide election the elected Primary is usually
+  running a different cluster's configuration, and deferring to it would end this cluster's historization
+  silently — the drain would report success on an empty outbox forever. That is the same scope mismatch
+  that once suspended the alarm drain on every node of the docker-dev rig at once.
+
+An unknown role **captures**, matching the alarm drain: a duplicate sample is recoverable, a hole in the
+trend is not. The residual cost is a capture gap across a failover, bounded by how long the newly promoted
+node takes to see its first redundancy snapshot — inherent to any non-duplicating design. Suppressed
+values are counted in `RecorderStatus.SuppressedNotPrimary`, and each gate transition logs once.
+
+---
+
 ## Tag auto-provisioning (EnsureTags)
 
 When a deploy adds a historized tag, the server auto-ensures it exists in the historian via the gateway's

docs/Redundancy.md

@@ -516,6 +516,20 @@ Under warm/hot redundancy both cluster nodes run `ScriptedAlarmHostActor` and ev
 - **`HistorianAdapterActor` historization** — likewise Primary-gated so alarm historization is exactly-once across all alarm sources. The actor subscribes to the `alerts` DPS topic and translates each `AlarmTransitionEvent` → `AlarmHistorianEvent` before enqueuing it on the sink; scripted alarms therefore historize exactly once regardless of cluster size.
 - **The alarm sink's DRAIN** — a second, independent gate on the same decision. The enqueue gate above is not sufficient any more: since Phase 2 the store-and-forward buffer lives in the replicated LocalDb, so the Secondary holds a full copy of the Primary's queued rows and an ungated drain there would re-deliver every event continuously. `LocalDbStoreAndForwardSink` takes a `drainGate` fed from `IRedundancyRoleView` — a singleton `DriverHostActor` publishes its `PrimaryGatePolicy` verdict to, because the drain runs on a timer and cannot receive `RedundancyStateChanged` itself. A gated-off node reports `HistorianDrainState.NotPrimary`. Delivery is **at-least-once across a failover** by design: rows the old Primary delivered just before dying may not have replicated their deletes yet. See [AlarmHistorian.md](AlarmHistorian.md).
 
+- **Continuous-historization value CAPTURE** (Gitea #537) — a third gate on the same role rule, and the
+  one that gates the *opposite* half of its pipeline. Both pair nodes serve the same address space and see
+  the same dependency-mux fan-out, so an ungated `ContinuousHistorizationRecorder` wrote every live sample
+  twice. It now reads a capture gate per value, fed from `PrimaryGatePolicy.ShouldRecordHistorizedValues`
+  via `IHistorizationRoleView` (a third verdict on the same `RedundancyRoleView` object, registered
+  unconditionally on driver nodes rather than alongside the alarm section's `IRedundancyRoleView`).
+  Gating the **capture** rather than the drain is deliberate: unlike the alarm queue, the historization
+  outbox is a node-local FasterLog that replicates to nobody, so a drain-only gate would leave the standby
+  appending forever to a queue no one empties. Values a node already captured are still drained by that
+  node. ⚠️ **The peer it stands down for must record the same refs** — a Primary in its own per-cluster
+  mesh — not merely hold the same rows; a node with no cluster-scoped role never stands down, because a
+  fleet-wide election's Primary usually runs a different cluster's configuration and deferring to it would
+  end this cluster's historization silently. See [Historian.md](Historian.md).
+
 Net effect: each alarm transition appears **once** on `/alerts` and would historize once, not once per node.
 
 See [ScriptedAlarms.md](ScriptedAlarms.md) and [AlarmTracking.md](AlarmTracking.md) for the scripted-alarm engine internals.

docs/plans/2026-08-01-continuous-historization-live-gate.md

@@ -248,8 +248,15 @@ because gating it would also mean a failover window with no capture at all.
 ## Follow-ups
 
 - **#491 CLOSED** 2026-08-04 — both legs proven.
-- **#537** — the recorder has no Primary gate, so both pair nodes write duplicate live values.
-- **#538** — one permanently-failing tag re-writes every healthy tag on every drain pass and the
-  outbox grows unbounded (the all-or-nothing ack above).
+- **#537 CLOSED** — the recorder now gates its **capture** (not its drain — the outbox is node-local and
+  replicates to nobody, unlike the alarm queue) on `PrimaryGatePolicy.ShouldRecordHistorizedValues`, read
+  per value through `IHistorizationRoleView`. It stands down only for a Primary that records the same
+  refs — a partner in its own per-cluster mesh — and a node with no cluster-scoped role never stands down.
+  See `docs/Historian.md` §"Only the recording Primary captures".
+- **#538 CLOSED** — the drain now abandons a tag's queued entries after `MaxWriteAttempts` consecutive
+  failures, counted **only across passes where some other tag wrote successfully**, so a whole-backend
+  outage still retains everything. The all-or-nothing ack is unchanged. `Capacity` also gained a non-zero
+  default (500,000) as the disk backstop for the total-outage case. See `docs/Historian.md` §"A poison tag
+  no longer pins the outbox".
 - Still open from before this gate: `EnsureTags` fires only for newly-*added* historized tags
   (`2026-06-27-historian-gateway-integration-issues.md` Issue 5) — re-confirmed live here.