fix(historian): gate historization capture on the Primary, quarantine a poison tag (#537, #538)

Both defects were found by #491's live gate against a real HistorianGateway and filed rather than
fixed at the time, because each one's fix is a design decision rather than a patch.

#537 — both nodes of a redundant pair captured every value, so the historian received duplicate
live samples for every historized tag (measured: the gateway's write counter advanced twice per
value and halved when one central was stopped).

The recorder now reads a capture gate per value, fed from the new
PrimaryGatePolicy.ShouldRecordHistorizedValues through IHistorizationRoleView — a third verdict on
the same RedundancyRoleView object, registered unconditionally on driver nodes rather than
alongside the alarm section's IRedundancyRoleView (which only exists when that section is enabled;
piggybacking is the #511 registration accident).

Two things are deliberate and easy to get backwards:

  * It gates the CAPTURE, not the drain — the opposite of the alarm sink. That queue is in LocalDb
    and replicates, so the Secondary already holds the Primary's rows. The historization outbox is
    a node-local FasterLog replicating to nobody, so a drain-only gate would leave the standby
    appending forever to a queue no one empties: unbounded disk, then a flood of stale duplicates
    the moment it is promoted. Entries a node already captured are still drained by that node.

  * The peer it stands down for must RECORD THE SAME REFS, not merely hold the same rows — a
    Primary in this node's own per-cluster mesh. A node carrying no cluster-scoped role cannot
    establish that and never stands down: under a fleet-wide election the elected Primary is
    usually running a different cluster's configuration, and deferring to it would end this
    cluster's historization silently, with the drain reporting success on an empty outbox forever.
    That is the scope mismatch that once suspended the alarm drain on every node of the rig.

An unknown role captures, matching the alarm drain — a duplicate sample is recoverable, a hole in
the trend is not.

#538 — the drain writes every per-tag group and then acks all-or-nothing (deliberate: RemoveAsync
truncates THROUGH an id, so partial acking could drop older unwritten entries). One permanently
failing tag therefore pinned the FIFO head while every healthy tag in the batch was re-submitted
to the historian on every pass, indefinitely — duplicate samples and an outbox that never
truncated, observed live at 8.3 MB from a single unprovisionable tag.

The drain now tracks consecutive failures per historian tag and abandons that tag's entries once
it exceeds MaxWriteAttempts (new, default 5), so the head advances. The counter only advances on a
pass where some OTHER tag wrote successfully: that is what separates "this tag is poison" from
"the backend is down". A count-only quarantine looks identical on the poison-tag case and silently
discards the whole backlog during an ordinary gateway restart, turning a duplicate-samples bug
into a lost-history one. Abandonment is also deferred until every failing tag is over budget, so
nothing is discarded on a pass that could not have advanced the head anyway. Each abandonment logs
a Warning naming the tag; the total rides RecorderStatus.AbandonedEntries.

Capacity also gains a non-zero default (500,000 entries) as the disk backstop for the one case the
quarantine deliberately cannot resolve — a historian refusing every write, where retaining is
correct. Set 0 to restore unbounded retention.

Every new test was falsified by reverting the production change it guards, and each one reddens
alone:

  * drop the backend-reachable evidence -> only A_backend_that_fails_every_tag_abandons_nothing
  * never abandon -> only A_permanently_failing_tag_is_abandoned_so_healthy_tags_stop_being_rewritten
  * latch the capture gate at construction -> only The_capture_gate_is_re_read_per_value...
  * treat any Primary as a scope peer -> only the two scope-mismatch cases

Full suite green. (Four OpcUaClient.Browser failures during the run were the opc-plc fixture being
down on the shared Docker host; brought it back up and they pass.)

Claude-Session: https://claude.ai/code/session_015p7wGqy3YpZNCpDzTpGMKo

src/Server/ZB.MOM.WW.OtOpcUa.Runtime/Drivers/DriverHostActor.cs

@@ -286,6 +286,22 @@ public sealed class DriverHostActor : ReceiveActor, IWithTimers
     /// </remarks>
     private readonly IDriverPrimaryRoleSource? _primaryRoleSource;
 
+    /// <summary>
+    ///     Gitea #537: where the continuous-historization <b>capture</b> verdict is published for the
+    ///     recorder, which taps the mux fan-out and so needs the decision off its own mailbox. Null on
+    ///     nodes that wire no redundancy manager.
+    /// </summary>
+    private readonly IHistorizationRoleView? _historizationRoleView;
+
+    /// <summary>
+    ///     Tests whether a node named in a redundancy snapshot serves this node's address space — i.e.
+    ///     is a per-cluster mesh partner recording the very same historized refs. Null (the default)
+    ///     means the caller could not establish it, which never stands this node down. See
+    ///     <see cref="PrimaryGatePolicy.ShouldRecordHistorizedValues"/> for why proving the sharing
+    ///     matters rather than trusting "somebody is Primary".
+    /// </summary>
+    private readonly Func<CommonsNodeId, bool>? _sharesHistorizationScopeWithLocal;
+
     /// <summary>
     ///     Host of this node's LocalDb replication partner — the only node that holds a copy of this
     ///     node's alarm queue, and therefore the only node it may ever stand down in favour of.
@@ -422,7 +438,9 @@ public sealed class DriverHostActor : ReceiveActor, IWithTimers
         IDeploymentArtifactFetcher? artifactFetcher = null,
         IAlarmStateStore? alarmStateStore = null,
         ITelemetryLocalHub? telemetryHub = null,
-        IDriverPrimaryRoleSource? primaryRoleSource = null) =>
+        IDriverPrimaryRoleSource? primaryRoleSource = null,
+        IHistorizationRoleView? historizationRoleView = null,
+        Func<CommonsNodeId, bool>? sharesHistorizationScopeWithLocal = null) =>
         // WARNING: this forwarding list is POSITIONAL, and Props.Create compiles it into an
         // expression tree. Six IActorRef? parameters and several interface-typed ones mean a
         // mis-ordered argument is usually type-compatible and therefore compiles clean, then binds
@@ -433,7 +451,8 @@ public sealed class DriverHostActor : ReceiveActor, IWithTimers
             healthPublisher, virtualTagEvaluator, historyWriter, virtualTagHostOverride,
             loggerFactory, scriptRootLogger, scriptedAlarmHostOverride, invokerFactory, driverMemberCountProvider,
             deploymentArtifactCache, redundancyRoleView, replicationPeerHost, fetchAndCacheMode, artifactFetcher,
-            alarmStateStore, telemetryHub, primaryRoleSource));
+            alarmStateStore, telemetryHub, primaryRoleSource, historizationRoleView,
+            sharesHistorizationScopeWithLocal));
 
     /// <summary>Initializes a new DriverHostActor with the specified dependencies.</summary>
     /// <param name="dbFactory">Database context factory for configuration database access.</param>
@@ -480,6 +499,15 @@ public sealed class DriverHostActor : ReceiveActor, IWithTimers
     /// <paramref name="redundancyRoleView"/> even though the production object implements both: a test that
     /// supplies only a role-view double must not silently also satisfy this, or its primary-role assertions
     /// pass without the publish ever happening. Null on nodes that wire no redundancy manager.</param>
+    /// <param name="historizationRoleView">Gitea #537: where the continuous-historization <i>capture</i>
+    /// verdict is published, for the recorder that taps the mux fan-out. A third distinct parameter for
+    /// the same reason <paramref name="primaryRoleSource"/> is a second — the production object
+    /// implements all three, and a test double supplied for one must not silently satisfy the others.
+    /// Null on nodes that wire no redundancy manager.</param>
+    /// <param name="sharesHistorizationScopeWithLocal">Tests whether a node named in a redundancy
+    /// snapshot serves this node's address space and therefore records the same historized refs. Null
+    /// (the default) means the caller could not establish it, and this node never stands down — see
+    /// <see cref="PrimaryGatePolicy.ShouldRecordHistorizedValues"/>.</param>
     public DriverHostActor(
         IDbContextFactory<OtOpcUaConfigDbContext>? dbFactory,
         CommonsNodeId localNode,
@@ -504,8 +532,12 @@ public sealed class DriverHostActor : ReceiveActor, IWithTimers
         IDeploymentArtifactFetcher? artifactFetcher = null,
         IAlarmStateStore? alarmStateStore = null,
         ITelemetryLocalHub? telemetryHub = null,
-        IDriverPrimaryRoleSource? primaryRoleSource = null)
+        IDriverPrimaryRoleSource? primaryRoleSource = null,
+        IHistorizationRoleView? historizationRoleView = null,
+        Func<CommonsNodeId, bool>? sharesHistorizationScopeWithLocal = null)
     {
+        _historizationRoleView = historizationRoleView;
+        _sharesHistorizationScopeWithLocal = sharesHistorizationScopeWithLocal;
         _telemetryHub = telemetryHub;
         _deploymentArtifactCache = deploymentArtifactCache;
         _redundancyRoleView = redundancyRoleView;
@@ -1431,6 +1463,22 @@ public sealed class DriverHostActor : ReceiveActor, IWithTimers
         _redundancyRoleView?.Publish(
             PrimaryGatePolicy.ShouldDrainAlarmHistory(_localRole, queueSharingPeerIsPrimary: peerIsPrimary));
 
+        // Gitea #537: the continuous-historization CAPTURE verdict. Same role rule as the drain gate
+        // above, but a different peer question — historization is not pair-local, it is fleet-wide by
+        // historian tag name, so the only node worth standing down for is one recording the SAME refs:
+        // a mesh partner serving this node's address space. With no way to establish that (a legacy
+        // node carrying no cluster-scoped role, or a test harness), the predicate is null and this
+        // node keeps capturing. Duplicate samples are recoverable; a hole in the trend is not.
+        var historizationPeerIsPrimary =
+            _sharesHistorizationScopeWithLocal is { } sharesScope
+            && msg.Nodes.Any(n => n.Role == RedundancyRole.Primary
+                                  && n.NodeId != _localNode
+                                  && sharesScope(n.NodeId));
+
+        _historizationRoleView?.PublishHistorizationRole(
+            PrimaryGatePolicy.ShouldRecordHistorizedValues(
+                _localRole, addressSpaceSharingPeerIsPrimary: historizationPeerIsPrimary));
+
         // The DATA-PLANE verdict, deliberately — the same one HandleRouteNodeWrite gates on, not the
         // drain verdict published immediately above. A driver reading this claims an exclusive identity
         // on a shared external system (Gitea #511: the Sparkplug host id), where two claimants is the

src/Server/ZB.MOM.WW.OtOpcUa.Runtime/Drivers/PrimaryGatePolicy.cs

@@ -71,4 +71,56 @@ public static class PrimaryGatePolicy
             RedundancyRole.Secondary or RedundancyRole.Detached => !queueSharingPeerIsPrimary,
             _ => true,   // unknown role ⇒ drain; a duplicate row beats a silent gap
         };
+
+    /// <summary>
+    ///     Decide whether this node should <b>capture</b> continuous-historization values into its
+    ///     durable outbox (Gitea #537).
+    /// </summary>
+    /// <remarks>
+    ///     <para>
+    ///         <b>Same rule as <see cref="ShouldDrainAlarmHistory"/>, deliberately — and for the same
+    ///         reason.</b> The historian is append-only and the error costs are asymmetric: a false
+    ///         allow costs a duplicate sample, a false deny costs a permanent hole in the trend. So an
+    ///         unknown role captures, and only an explicit Secondary/Detached stands down.
+    ///     </para>
+    ///     <para>
+    ///         <b>But it gates the CAPTURE, not the drain — the opposite of the alarm sink.</b> That
+    ///         choice follows from where the queue lives. The alarm queue is in LocalDb and
+    ///         <i>replicates</i>, so the Secondary already holds the Primary's rows and gating the
+    ///         drain loses nothing. The historization outbox is a node-local FasterLog that replicates
+    ///         to nobody: gating only its drain would leave the standby appending forever to a queue no
+    ///         one empties (unbounded disk, and a flood of stale duplicates the moment it is promoted),
+    ///         while gating the capture leaves it holding nothing. Values already captured are still
+    ///         drained by the node that captured them, so a demotion loses nothing that was recorded.
+    ///     </para>
+    ///     <para>
+    ///         The residual cost is a capture gap across a failover, bounded by how long the newly
+    ///         promoted node takes to see its first snapshot. That is inherent to any non-duplicating
+    ///         design; the alternative — both nodes capturing — is what #537 was filed about.
+    ///     </para>
+    /// </remarks>
+    /// <param name="localRole">This node's last-known redundancy role, or <c>null</c> when unknown.</param>
+    /// <param name="addressSpaceSharingPeerIsPrimary">
+    ///     Whether the node holding <see cref="RedundancyRole.Primary"/> in the same snapshot is one
+    ///     that serves <b>this node's address space</b>, and therefore captures the very same historized
+    ///     refs — its per-cluster mesh partner.
+    ///     <para>
+    ///         The scoping matters for the same reason it does on the alarm queue, and the mistake is
+    ///         easier to make here because it looks safe: the redundancy election reaches only as far as
+    ///         the caller can prove, whereas historization is fleet-wide by tag name. A node that stands
+    ///         down in favour of a Primary running some <i>other</i> cluster's configuration hands its
+    ///         tags to a node that never records them, and nothing says so — the drain simply reports
+    ///         success on an empty outbox forever. Callers that cannot establish the sharing pass
+    ///         <c>false</c>, which never stands down; duplicates are recoverable, a silent gap is not.
+    ///     </para>
+    /// </param>
+    /// <returns><c>true</c> to capture; <c>false</c> only when a node recording these same tags is doing it.</returns>
+    public static bool ShouldRecordHistorizedValues(
+        RedundancyRole? localRole, bool addressSpaceSharingPeerIsPrimary) =>
+        localRole switch
+        {
+            RedundancyRole.Primary => true,
+            RedundancyRole.Secondary or RedundancyRole.Detached => !addressSpaceSharingPeerIsPrimary,
+            _ => true,   // unknown role ⇒ capture; a duplicate sample beats a hole in the trend
+        };
 }