e7fbe97759
Both defects were found by #491's live gate against a real HistorianGateway and filed rather than fixed at the time, because each one's fix is a design decision rather than a patch. #537 — both nodes of a redundant pair captured every value, so the historian received duplicate live samples for every historized tag (measured: the gateway's write counter advanced twice per value and halved when one central was stopped). The recorder now reads a capture gate per value, fed from the new PrimaryGatePolicy.ShouldRecordHistorizedValues through IHistorizationRoleView — a third verdict on the same RedundancyRoleView object, registered unconditionally on driver nodes rather than alongside the alarm section's IRedundancyRoleView (which only exists when that section is enabled; piggybacking is the #511 registration accident). Two things are deliberate and easy to get backwards: * It gates the CAPTURE, not the drain — the opposite of the alarm sink. That queue is in LocalDb and replicates, so the Secondary already holds the Primary's rows. The historization outbox is a node-local FasterLog replicating to nobody, so a drain-only gate would leave the standby appending forever to a queue no one empties: unbounded disk, then a flood of stale duplicates the moment it is promoted. Entries a node already captured are still drained by that node. * The peer it stands down for must RECORD THE SAME REFS, not merely hold the same rows — a Primary in this node's own per-cluster mesh. A node carrying no cluster-scoped role cannot establish that and never stands down: under a fleet-wide election the elected Primary is usually running a different cluster's configuration, and deferring to it would end this cluster's historization silently, with the drain reporting success on an empty outbox forever. That is the scope mismatch that once suspended the alarm drain on every node of the rig. An unknown role captures, matching the alarm drain — a duplicate sample is recoverable, a hole in the trend is not. #538 — the drain writes every per-tag group and then acks all-or-nothing (deliberate: RemoveAsync truncates THROUGH an id, so partial acking could drop older unwritten entries). One permanently failing tag therefore pinned the FIFO head while every healthy tag in the batch was re-submitted to the historian on every pass, indefinitely — duplicate samples and an outbox that never truncated, observed live at 8.3 MB from a single unprovisionable tag. The drain now tracks consecutive failures per historian tag and abandons that tag's entries once it exceeds MaxWriteAttempts (new, default 5), so the head advances. The counter only advances on a pass where some OTHER tag wrote successfully: that is what separates "this tag is poison" from "the backend is down". A count-only quarantine looks identical on the poison-tag case and silently discards the whole backlog during an ordinary gateway restart, turning a duplicate-samples bug into a lost-history one. Abandonment is also deferred until every failing tag is over budget, so nothing is discarded on a pass that could not have advanced the head anyway. Each abandonment logs a Warning naming the tag; the total rides RecorderStatus.AbandonedEntries. Capacity also gains a non-zero default (500,000 entries) as the disk backstop for the one case the quarantine deliberately cannot resolve — a historian refusing every write, where retaining is correct. Set 0 to restore unbounded retention. Every new test was falsified by reverting the production change it guards, and each one reddens alone: * drop the backend-reachable evidence -> only A_backend_that_fails_every_tag_abandons_nothing * never abandon -> only A_permanently_failing_tag_is_abandoned_so_healthy_tags_stop_being_rewritten * latch the capture gate at construction -> only The_capture_gate_is_re_read_per_value... * treat any Primary as a scope peer -> only the two scope-mismatch cases Full suite green. (Four OpcUaClient.Browser failures during the run were the opc-plc fixture being down on the shared Docker host; brought it back up and they pass.) Claude-Session: https://claude.ai/code/session_015p7wGqy3YpZNCpDzTpGMKo
192 lines
10 KiB
C#
192 lines
10 KiB
C#
using Shouldly;
|
|
using Xunit;
|
|
using ZB.MOM.WW.OtOpcUa.Commons.Messages.Redundancy;
|
|
using ZB.MOM.WW.OtOpcUa.Runtime.Drivers;
|
|
|
|
namespace ZB.MOM.WW.OtOpcUa.Runtime.Tests.Drivers;
|
|
|
|
/// <summary>
|
|
/// archreview 03/S4 — the single Primary data-plane gate decision. A KNOWN role wins outright; an
|
|
/// UNKNOWN role (no snapshot yet, or the snapshot never mentioned this node) is resolved by cluster
|
|
/// membership: a single-driver cluster stays default-ALLOW (boot-window / single-node posture), a
|
|
/// multi-driver cluster is default-DENY (a real Primary peer exists — don't touch the shared field
|
|
/// device until the snapshot proves this node is Primary).
|
|
/// </summary>
|
|
public sealed class PrimaryGatePolicyTests
|
|
{
|
|
[Theory]
|
|
// Primary → always service, regardless of member count.
|
|
[InlineData(RedundancyRole.Primary, 0, true)]
|
|
[InlineData(RedundancyRole.Primary, 1, true)]
|
|
[InlineData(RedundancyRole.Primary, 2, true)]
|
|
// Secondary / Detached → never service, regardless of member count.
|
|
[InlineData(RedundancyRole.Secondary, 0, false)]
|
|
[InlineData(RedundancyRole.Secondary, 2, false)]
|
|
[InlineData(RedundancyRole.Detached, 0, false)]
|
|
[InlineData(RedundancyRole.Detached, 2, false)]
|
|
public void Known_role_wins_regardless_of_member_count(RedundancyRole role, int members, bool expected)
|
|
=> PrimaryGatePolicy.ShouldServiceAsPrimary(role, members).ShouldBe(expected);
|
|
|
|
[Theory]
|
|
// Unknown role: allow only when no driver peer exists (count <= 1).
|
|
[InlineData(0, true)]
|
|
[InlineData(1, true)]
|
|
[InlineData(2, false)]
|
|
[InlineData(3, false)]
|
|
public void Unknown_role_resolves_by_membership(int members, bool expected)
|
|
=> PrimaryGatePolicy.ShouldServiceAsPrimary(null, members).ShouldBe(expected);
|
|
}
|
|
|
|
/// <summary>
|
|
/// The alarm-history drain gate, which deliberately does <b>not</b> mirror the device-write gate.
|
|
/// </summary>
|
|
/// <remarks>
|
|
/// <para>
|
|
/// <b>Why a second policy is correct here.</b> The write gate protects a shared field device:
|
|
/// two nodes driving one PLC is dangerous and irreversible, so an unknown role with a peer
|
|
/// present must deny. The drain writes to an append-only historian instead, and Phase 2's
|
|
/// delivery contract is already <i>at-least-once across a failover, by design</i> — identical
|
|
/// events even collapse to one row, because ids are a hash of the payload.
|
|
/// </para>
|
|
/// <para>
|
|
/// So the two error costs are wildly asymmetric: a false allow costs a duplicate history row,
|
|
/// while a false deny silently stops the alarm audit trail and eventually evicts it at the
|
|
/// capacity wall. This gate therefore keys on the role <b>alone</b> and opens when the role is
|
|
/// unknown.
|
|
/// </para>
|
|
/// <para>
|
|
/// <b>Found by the LocalDb Phase 2 live gate.</b> The rig runs four driver nodes in one Akka
|
|
/// cluster with no <c>Redundancy</c> section, so every node's role was unknown and the
|
|
/// cluster-wide driver count was 4 — and every node, including the two unpaired site-b nodes
|
|
/// that have no peer and no replication at all, logged "Historian drain suspended". Nothing
|
|
/// drained anywhere. Before Phase 2 that deployment drained fine, because the drain was ungated.
|
|
/// </para>
|
|
/// </remarks>
|
|
public sealed class AlarmHistoryDrainGatePolicyTests
|
|
{
|
|
[Theory]
|
|
// A Primary always drains its own queue.
|
|
[InlineData(RedundancyRole.Primary, true, true)]
|
|
[InlineData(RedundancyRole.Primary, false, true)]
|
|
// A Secondary stands down ONLY when a Primary actually exists to stand up.
|
|
[InlineData(RedundancyRole.Secondary, true, false)]
|
|
[InlineData(RedundancyRole.Detached, true, false)]
|
|
[InlineData(RedundancyRole.Secondary, false, true)]
|
|
[InlineData(RedundancyRole.Detached, false, true)]
|
|
public void A_known_role_stands_down_only_for_a_primary_that_exists(
|
|
RedundancyRole role, bool queueSharingPeerIsPrimary, bool expected)
|
|
=> PrimaryGatePolicy.ShouldDrainAlarmHistory(role, queueSharingPeerIsPrimary).ShouldBe(expected);
|
|
|
|
/// <summary>
|
|
/// Unknown role ⇒ drain, whoever else is out there. The first case the live gate broke on, and
|
|
/// the one that separates this policy from <see cref="PrimaryGatePolicy.ShouldServiceAsPrimary"/>.
|
|
/// </summary>
|
|
[Theory]
|
|
[InlineData(true)]
|
|
[InlineData(false)]
|
|
public void An_unknown_role_drains(bool queueSharingPeerIsPrimary)
|
|
=> PrimaryGatePolicy.ShouldDrainAlarmHistory(null, queueSharingPeerIsPrimary).ShouldBeTrue();
|
|
|
|
/// <summary>
|
|
/// A Secondary whose Primary is NOT its queue-sharing peer keeps draining — the deepest of the
|
|
/// live gate's findings.
|
|
/// </summary>
|
|
/// <remarks>
|
|
/// <c>RedundancyStateActor</c> derives roles from Akka's cluster-wide <c>RoleLeader("driver")</c>,
|
|
/// but the alarm queue is pair-local. On the docker-dev rig the elected Primary is a CENTRAL node
|
|
/// — it carries the driver Akka role, replicates nobody's LocalDb, and does not even run the
|
|
/// alarm historian — so under a plain "a Primary exists ⇒ stand down" rule every site node
|
|
/// suspended its drain in favour of a node that could not possibly deliver its events. The
|
|
/// buffer then grows on every node until the capacity wall evicts the oldest: silent, permanent
|
|
/// loss of the audit trail the queue exists to protect.
|
|
/// </remarks>
|
|
[Fact]
|
|
public void A_secondary_whose_peer_is_not_the_primary_keeps_draining()
|
|
{
|
|
// A Primary exists somewhere in the cluster, but it does not hold this node's rows.
|
|
PrimaryGatePolicy.ShouldDrainAlarmHistory(RedundancyRole.Secondary, queueSharingPeerIsPrimary: false)
|
|
.ShouldBeTrue("a Secondary must not defer to a Primary that cannot deliver its events");
|
|
}
|
|
|
|
/// <summary>
|
|
/// The divergence, pinned explicitly: with a peer present and no role known, the write gate
|
|
/// denies and the drain gate allows. If someone later "harmonises" the two, this fails.
|
|
/// </summary>
|
|
[Fact]
|
|
public void The_two_gates_deliberately_disagree_when_the_role_is_unknown_and_a_peer_exists()
|
|
{
|
|
PrimaryGatePolicy.ShouldServiceAsPrimary(localRole: null, driverMemberCount: 4).ShouldBeFalse();
|
|
PrimaryGatePolicy.ShouldDrainAlarmHistory(localRole: null, queueSharingPeerIsPrimary: true).ShouldBeTrue();
|
|
}
|
|
}
|
|
|
|
/// <summary>
|
|
/// Gitea #537 — the continuous-historization <b>capture</b> gate.
|
|
/// </summary>
|
|
/// <remarks>
|
|
/// <para>
|
|
/// Same role rule as the alarm drain, and for the same reason: the historian is append-only, so
|
|
/// a false allow costs a duplicate sample while a false deny costs a permanent hole in the
|
|
/// trend. What differs is the <i>peer</i> question — the alarm queue is pair-local and
|
|
/// replicated, so the peer that matters is whoever holds a copy of these rows; historization is
|
|
/// fleet-wide by tag name, so the peer that matters is whoever records these same refs.
|
|
/// </para>
|
|
/// <para>
|
|
/// It also gates the opposite half of the pipeline. The alarm sink gates its DRAIN, because the
|
|
/// Secondary already holds a replica of the Primary's queue. The historization outbox is
|
|
/// node-local and replicates to nobody, so gating only the drain would leave the standby
|
|
/// appending forever to a queue no one empties. See
|
|
/// <c>PrimaryGatePolicy.ShouldRecordHistorizedValues</c>.
|
|
/// </para>
|
|
/// </remarks>
|
|
public sealed class HistorizationCaptureGatePolicyTests
|
|
{
|
|
[Theory]
|
|
[InlineData(RedundancyRole.Primary, false, true)]
|
|
[InlineData(RedundancyRole.Primary, true, true)] // Primary captures even if another claims it
|
|
[InlineData(RedundancyRole.Secondary, true, false)] // its mesh partner is recording these refs
|
|
[InlineData(RedundancyRole.Detached, true, false)]
|
|
public void Known_role_decides_against_the_peer(
|
|
RedundancyRole role, bool peerIsPrimary, bool expected)
|
|
=> PrimaryGatePolicy.ShouldRecordHistorizedValues(role, peerIsPrimary).ShouldBe(expected);
|
|
|
|
/// <summary>
|
|
/// A Secondary whose Primary does not record these refs keeps capturing.
|
|
/// </summary>
|
|
/// <remarks>
|
|
/// This is the case that makes the peer argument load-bearing rather than decorative. Under a
|
|
/// fleet-wide election the elected Primary is usually running some other cluster's
|
|
/// configuration entirely, and it will never record this node's tags. A bare "somebody is
|
|
/// Primary ⇒ stand down" rule ends this cluster's historization outright, and reports nothing
|
|
/// wrong: the drain simply succeeds on an empty outbox forever. It is the same scope mismatch
|
|
/// that once suspended the alarm drain on every node of the docker-dev rig at once.
|
|
/// </remarks>
|
|
[Fact]
|
|
public void A_secondary_whose_primary_records_other_tags_keeps_capturing()
|
|
=> PrimaryGatePolicy
|
|
.ShouldRecordHistorizedValues(RedundancyRole.Secondary, addressSpaceSharingPeerIsPrimary: false)
|
|
.ShouldBeTrue("a Secondary must not defer to a Primary that records none of its tags");
|
|
|
|
/// <summary>
|
|
/// Unknown role captures — a deployment that never publishes a role must still historize.
|
|
/// </summary>
|
|
[Theory]
|
|
[InlineData(false)]
|
|
[InlineData(true)]
|
|
public void Unknown_role_captures_whatever_the_peer_says(bool peerIsPrimary)
|
|
=> PrimaryGatePolicy.ShouldRecordHistorizedValues(null, peerIsPrimary).ShouldBeTrue();
|
|
|
|
/// <summary>
|
|
/// Pinned against a future "harmonisation": with no role known and a peer present, the
|
|
/// device-write gate denies while this one allows.
|
|
/// </summary>
|
|
[Fact]
|
|
public void It_diverges_from_the_write_gate_when_the_role_is_unknown()
|
|
{
|
|
PrimaryGatePolicy.ShouldServiceAsPrimary(localRole: null, driverMemberCount: 4).ShouldBeFalse();
|
|
PrimaryGatePolicy
|
|
.ShouldRecordHistorizedValues(localRole: null, addressSpaceSharingPeerIsPrimary: true)
|
|
.ShouldBeTrue();
|
|
}
|
|
}
|