Files
lmxopcua/src/Server/ZB.MOM.WW.OtOpcUa.Runtime/Redundancy/IRedundancyRoleView.cs
T
Joseph Doherty e7fbe97759 fix(historian): gate historization capture on the Primary, quarantine a poison tag (#537, #538)
Both defects were found by #491's live gate against a real HistorianGateway and filed rather than
fixed at the time, because each one's fix is a design decision rather than a patch.

#537 — both nodes of a redundant pair captured every value, so the historian received duplicate
live samples for every historized tag (measured: the gateway's write counter advanced twice per
value and halved when one central was stopped).

The recorder now reads a capture gate per value, fed from the new
PrimaryGatePolicy.ShouldRecordHistorizedValues through IHistorizationRoleView — a third verdict on
the same RedundancyRoleView object, registered unconditionally on driver nodes rather than
alongside the alarm section's IRedundancyRoleView (which only exists when that section is enabled;
piggybacking is the #511 registration accident).

Two things are deliberate and easy to get backwards:

  * It gates the CAPTURE, not the drain — the opposite of the alarm sink. That queue is in LocalDb
    and replicates, so the Secondary already holds the Primary's rows. The historization outbox is
    a node-local FasterLog replicating to nobody, so a drain-only gate would leave the standby
    appending forever to a queue no one empties: unbounded disk, then a flood of stale duplicates
    the moment it is promoted. Entries a node already captured are still drained by that node.

  * The peer it stands down for must RECORD THE SAME REFS, not merely hold the same rows — a
    Primary in this node's own per-cluster mesh. A node carrying no cluster-scoped role cannot
    establish that and never stands down: under a fleet-wide election the elected Primary is
    usually running a different cluster's configuration, and deferring to it would end this
    cluster's historization silently, with the drain reporting success on an empty outbox forever.
    That is the scope mismatch that once suspended the alarm drain on every node of the rig.

An unknown role captures, matching the alarm drain — a duplicate sample is recoverable, a hole in
the trend is not.

#538 — the drain writes every per-tag group and then acks all-or-nothing (deliberate: RemoveAsync
truncates THROUGH an id, so partial acking could drop older unwritten entries). One permanently
failing tag therefore pinned the FIFO head while every healthy tag in the batch was re-submitted
to the historian on every pass, indefinitely — duplicate samples and an outbox that never
truncated, observed live at 8.3 MB from a single unprovisionable tag.

The drain now tracks consecutive failures per historian tag and abandons that tag's entries once
it exceeds MaxWriteAttempts (new, default 5), so the head advances. The counter only advances on a
pass where some OTHER tag wrote successfully: that is what separates "this tag is poison" from
"the backend is down". A count-only quarantine looks identical on the poison-tag case and silently
discards the whole backlog during an ordinary gateway restart, turning a duplicate-samples bug
into a lost-history one. Abandonment is also deferred until every failing tag is over budget, so
nothing is discarded on a pass that could not have advanced the head anyway. Each abandonment logs
a Warning naming the tag; the total rides RecorderStatus.AbandonedEntries.

Capacity also gains a non-zero default (500,000 entries) as the disk backstop for the one case the
quarantine deliberately cannot resolve — a historian refusing every write, where retaining is
correct. Set 0 to restore unbounded retention.

Every new test was falsified by reverting the production change it guards, and each one reddens
alone:

  * drop the backend-reachable evidence -> only A_backend_that_fails_every_tag_abandons_nothing
  * never abandon -> only A_permanently_failing_tag_is_abandoned_so_healthy_tags_stop_being_rewritten
  * latch the capture gate at construction -> only The_capture_gate_is_re_read_per_value...
  * treat any Primary as a scope peer -> only the two scope-mismatch cases

Full suite green. (Four OpcUaClient.Browser failures during the run were the opc-plc fixture being
down on the shared Docker host; brought it back up and they pass.)

Claude-Session: https://claude.ai/code/session_015p7wGqy3YpZNCpDzTpGMKo
2026-08-04 12:12:16 -04:00

134 lines
7.2 KiB
C#

using ZB.MOM.WW.OtOpcUa.Core.Abstractions;
using ZB.MOM.WW.OtOpcUa.Runtime.Drivers;
namespace ZB.MOM.WW.OtOpcUa.Runtime.Redundancy;
/// <summary>
/// A singleton snapshot of the alarm-history drain decision, so code that lives outside an actor
/// can ask "should this node be draining the alarm queue right now?".
/// </summary>
/// <remarks>
/// <para>
/// Exists for the alarm store-and-forward drain. That drain runs on a timer owned by the
/// sink, not on an actor mailbox, so it cannot receive <c>RedundancyStateChanged</c>
/// directly — but it must be role-scoped, because the queue it drains replicates to the
/// pair peer and two draining nodes would deliver every event twice.
/// </para>
/// <para>
/// Deliberately publishes the <i>decision</i> rather than the role that produced it, so
/// <see cref="PrimaryGatePolicy"/> stays the single place that decides.
/// </para>
/// <para>
/// <b>This is not the device-write gate.</b> It is fed by
/// <see cref="PrimaryGatePolicy.ShouldDrainAlarmHistory"/>, which opens on an unknown role
/// where <see cref="PrimaryGatePolicy.ShouldServiceAsPrimary"/> closes. The naming is
/// explicit about the consumer for that reason: an earlier revision published the
/// write-gate verdict here, and on a cluster with several driver nodes but no configured
/// redundancy it suspended the drain everywhere, so alarm history accumulated on every node
/// and left none.
/// </para>
/// </remarks>
public interface IRedundancyRoleView
{
/// <summary>Whether this node should be draining the alarm store-and-forward queue right now.</summary>
bool ShouldDrainAlarmHistory { get; }
/// <summary>Records a fresh decision. Called by <c>DriverHostActor</c> on every redundancy snapshot.</summary>
/// <param name="shouldDrainAlarmHistory">
/// The decision <see cref="PrimaryGatePolicy.ShouldDrainAlarmHistory"/> produced.
/// </param>
void Publish(bool shouldDrainAlarmHistory);
}
/// <summary>
/// A singleton snapshot of the continuous-historization <b>capture</b> decision (Gitea #537), so
/// the recorder can ask "should this node be recording historized values right now?".
/// </summary>
/// <remarks>
/// <para>
/// A separate interface from <see cref="IRedundancyRoleView"/> even though one object
/// implements both, because the two are registered on different terms:
/// <c>IRedundancyRoleView</c> is registered by <c>AddAlarmHistorian</c> and so exists only
/// where the alarm section is enabled, while this one is registered unconditionally on every
/// driver-role node. Piggybacking on the alarm registration would silently drop the
/// historization gate on any pair that runs historization without alarm history — the same
/// registration accident Gitea #511 documents on the neighbouring verdict.
/// </para>
/// <para>
/// Publishes the <i>decision</i> rather than the role, so
/// <see cref="PrimaryGatePolicy.ShouldRecordHistorizedValues"/> stays the single place that
/// decides. Note it is <b>not</b> the alarm drain verdict: the two policies agree on the role
/// rule but disagree on what makes a peer relevant (a queue-sharing replication partner vs. a
/// node serving the same address space), and the wrong peer notion in either direction is a
/// silent gap.
/// </para>
/// </remarks>
public interface IHistorizationRoleView
{
/// <summary>Whether this node should be capturing continuous-historization values right now.</summary>
bool ShouldRecordHistorizedValues { get; }
/// <summary>Records a fresh decision. Called by <c>DriverHostActor</c> on every redundancy snapshot.</summary>
/// <param name="shouldRecordHistorizedValues">
/// The decision <see cref="PrimaryGatePolicy.ShouldRecordHistorizedValues"/> produced.
/// </param>
void PublishHistorizationRole(bool shouldRecordHistorizedValues);
}
/// <summary>
/// Thread-safe <see cref="IRedundancyRoleView"/> + <see cref="IDriverPrimaryRoleSource"/> +
/// <see cref="IHistorizationRoleView"/>, each backed by its own volatile field.
/// </summary>
/// <remarks>
/// <para>
/// <b>Two verdicts, one object, on purpose.</b> Both are published from the same
/// <c>DriverHostActor</c> snapshot handler and both are read off-actor, so a second class
/// would duplicate the publish plumbing to hold a second bool. They are kept as distinct
/// <i>properties</i> rather than collapsed into one because they are genuinely different
/// decisions from different policies — see <see cref="IRedundancyRoleView"/>'s remarks on why
/// the drain gate must open on an unknown role where the data-plane gate closes. Collapsing
/// them is the bug that live gate already found once.
/// </para>
/// </remarks>
public sealed class RedundancyRoleView : IRedundancyRoleView, IDriverPrimaryRoleSource, IHistorizationRoleView
{
// Seeded with the decision for "role unknown" rather than a bare false. An unpublished view and a
// node whose role nothing ever reports are genuinely the same situation, and they must behave the
// same: a deployment that runs no redundancy at all never publishes here, and defaulting closed
// would silently stop its alarm history forever.
private volatile bool _shouldDrainAlarmHistory =
PrimaryGatePolicy.ShouldDrainAlarmHistory(localRole: null, queueSharingPeerIsPrimary: false);
// Seeded from the same policy that will later publish here, for the unknown-role/no-peer case:
// a node nothing has reported a role for, on a cluster with no driver peer, is a single-node
// deployment and may claim. The moment a snapshot arrives this is replaced by the real verdict,
// and a multi-driver cluster's unknown role denies.
private volatile bool _shouldActAsPrimary =
PrimaryGatePolicy.ShouldServiceAsPrimary(localRole: null, driverMemberCount: 1);
// Seeded for the unknown-role case from the same policy that will later publish here: a node
// nothing has reported a role for captures, exactly as an unpaired or single-node deployment must.
// Defaulting closed would leave a deployment that never publishes a role with no trend data at all.
private volatile bool _shouldRecordHistorizedValues =
PrimaryGatePolicy.ShouldRecordHistorizedValues(localRole: null, addressSpaceSharingPeerIsPrimary: false);
/// <inheritdoc/>
public bool ShouldDrainAlarmHistory => _shouldDrainAlarmHistory;
/// <inheritdoc/>
public bool ShouldRecordHistorizedValues => _shouldRecordHistorizedValues;
/// <inheritdoc/>
public bool ShouldActAsPrimary => _shouldActAsPrimary;
/// <inheritdoc/>
public void Publish(bool shouldDrainAlarmHistory) => _shouldDrainAlarmHistory = shouldDrainAlarmHistory;
/// <inheritdoc/>
public void PublishPrimaryRole(bool shouldActAsPrimary) => _shouldActAsPrimary = shouldActAsPrimary;
/// <inheritdoc/>
public void PublishHistorizationRole(bool shouldRecordHistorizedValues)
=> _shouldRecordHistorizedValues = shouldRecordHistorizedValues;
}