e7fbe97759
Both defects were found by #491's live gate against a real HistorianGateway and filed rather than fixed at the time, because each one's fix is a design decision rather than a patch. #537 — both nodes of a redundant pair captured every value, so the historian received duplicate live samples for every historized tag (measured: the gateway's write counter advanced twice per value and halved when one central was stopped). The recorder now reads a capture gate per value, fed from the new PrimaryGatePolicy.ShouldRecordHistorizedValues through IHistorizationRoleView — a third verdict on the same RedundancyRoleView object, registered unconditionally on driver nodes rather than alongside the alarm section's IRedundancyRoleView (which only exists when that section is enabled; piggybacking is the #511 registration accident). Two things are deliberate and easy to get backwards: * It gates the CAPTURE, not the drain — the opposite of the alarm sink. That queue is in LocalDb and replicates, so the Secondary already holds the Primary's rows. The historization outbox is a node-local FasterLog replicating to nobody, so a drain-only gate would leave the standby appending forever to a queue no one empties: unbounded disk, then a flood of stale duplicates the moment it is promoted. Entries a node already captured are still drained by that node. * The peer it stands down for must RECORD THE SAME REFS, not merely hold the same rows — a Primary in this node's own per-cluster mesh. A node carrying no cluster-scoped role cannot establish that and never stands down: under a fleet-wide election the elected Primary is usually running a different cluster's configuration, and deferring to it would end this cluster's historization silently, with the drain reporting success on an empty outbox forever. That is the scope mismatch that once suspended the alarm drain on every node of the rig. An unknown role captures, matching the alarm drain — a duplicate sample is recoverable, a hole in the trend is not. #538 — the drain writes every per-tag group and then acks all-or-nothing (deliberate: RemoveAsync truncates THROUGH an id, so partial acking could drop older unwritten entries). One permanently failing tag therefore pinned the FIFO head while every healthy tag in the batch was re-submitted to the historian on every pass, indefinitely — duplicate samples and an outbox that never truncated, observed live at 8.3 MB from a single unprovisionable tag. The drain now tracks consecutive failures per historian tag and abandons that tag's entries once it exceeds MaxWriteAttempts (new, default 5), so the head advances. The counter only advances on a pass where some OTHER tag wrote successfully: that is what separates "this tag is poison" from "the backend is down". A count-only quarantine looks identical on the poison-tag case and silently discards the whole backlog during an ordinary gateway restart, turning a duplicate-samples bug into a lost-history one. Abandonment is also deferred until every failing tag is over budget, so nothing is discarded on a pass that could not have advanced the head anyway. Each abandonment logs a Warning naming the tag; the total rides RecorderStatus.AbandonedEntries. Capacity also gains a non-zero default (500,000 entries) as the disk backstop for the one case the quarantine deliberately cannot resolve — a historian refusing every write, where retaining is correct. Set 0 to restore unbounded retention. Every new test was falsified by reverting the production change it guards, and each one reddens alone: * drop the backend-reachable evidence -> only A_backend_that_fails_every_tag_abandons_nothing * never abandon -> only A_permanently_failing_tag_is_abandoned_so_healthy_tags_stop_being_rewritten * latch the capture gate at construction -> only The_capture_gate_is_re_read_per_value... * treat any Primary as a scope peer -> only the two scope-mismatch cases Full suite green. (Four OpcUaClient.Browser failures during the run were the opc-plc fixture being down on the shared Docker host; brought it back up and they pass.) Claude-Session: https://claude.ai/code/session_015p7wGqy3YpZNCpDzTpGMKo
185 lines
8.1 KiB
C#
185 lines
8.1 KiB
C#
using Akka.Actor;
|
|
using Shouldly;
|
|
using Xunit;
|
|
using ZB.MOM.WW.OtOpcUa.Commons.Messages.Redundancy;
|
|
using ZB.MOM.WW.OtOpcUa.Commons.Types;
|
|
using ZB.MOM.WW.OtOpcUa.Runtime.Drivers;
|
|
using ZB.MOM.WW.OtOpcUa.Runtime.Redundancy;
|
|
using ZB.MOM.WW.OtOpcUa.Runtime.Tests.Harness;
|
|
|
|
namespace ZB.MOM.WW.OtOpcUa.Runtime.Tests.Drivers;
|
|
|
|
/// <summary>
|
|
/// Gitea #537 — the bridge that carries the continuous-historization <b>capture</b> verdict out of
|
|
/// the actor and into <see cref="IHistorizationRoleView"/>, where the recorder reads it per value.
|
|
/// </summary>
|
|
/// <remarks>
|
|
/// <para>
|
|
/// Both nodes of a redundant pair serve the same address space and see the same dependency-mux
|
|
/// fan-out, so an ungated recorder writes every sample twice. Measured on the docker-dev central
|
|
/// pair against a real gateway: the write counter advanced twice per value, and halved when one
|
|
/// node was stopped.
|
|
/// </para>
|
|
/// <para>
|
|
/// These cases share the role rule with <see cref="DriverHostActorRoleViewTests"/> but differ on
|
|
/// which peer counts. The alarm queue replicates, so the relevant peer is whoever holds a copy
|
|
/// of this node's rows; historization is fleet-wide by historian tag name, so the relevant peer
|
|
/// is whoever records this node's refs — a partner in its own per-cluster mesh.
|
|
/// </para>
|
|
/// </remarks>
|
|
public sealed class DriverHostActorHistorizationRoleViewTests : RuntimeActorTestBase
|
|
{
|
|
private static readonly NodeId TestNode = NodeId.Parse("driver-histview-test");
|
|
private static readonly NodeId MeshPeer = NodeId.Parse("mesh-peer:4053");
|
|
private static readonly NodeId ForeignNode = NodeId.Parse("other-cluster:4053");
|
|
private static readonly TimeSpan Timeout = TimeSpan.FromSeconds(5);
|
|
|
|
/// <summary>
|
|
/// Before any snapshot arrives the view must read open, not closed.
|
|
/// </summary>
|
|
/// <remarks>
|
|
/// A deployment that runs no redundancy never publishes here at all. Defaulting closed would
|
|
/// leave it with no trend data whatsoever and nothing to say why.
|
|
/// </remarks>
|
|
[Fact]
|
|
public void An_unpublished_view_captures()
|
|
=> new RedundancyRoleView().ShouldRecordHistorizedValues.ShouldBeTrue();
|
|
|
|
/// <summary>A Secondary stands down for a Primary in its own mesh — the duplicate-write case.</summary>
|
|
[Fact]
|
|
public void Secondary_stands_down_for_a_primary_in_its_own_mesh()
|
|
{
|
|
var view = new RecordingHistorizationView();
|
|
SpawnHost(view, sharesScopeWith: MeshPeer).Tell(Snapshot(
|
|
(TestNode, RedundancyRole.Secondary), (MeshPeer, RedundancyRole.Primary)));
|
|
|
|
AwaitAssert(() => view.Published.ShouldBe([false]), duration: Timeout);
|
|
}
|
|
|
|
/// <summary>
|
|
/// A Secondary whose Primary is in ANOTHER cluster keeps capturing.
|
|
/// </summary>
|
|
/// <remarks>
|
|
/// That node runs a different configuration and records none of these tags, so standing down for
|
|
/// it would end this cluster's historization silently — the drain would simply report success on
|
|
/// an empty outbox forever. Same scope mismatch that once suspended the alarm drain fleet-wide.
|
|
/// </remarks>
|
|
[Fact]
|
|
public void Secondary_keeps_capturing_when_the_primary_records_other_tags()
|
|
{
|
|
var view = new RecordingHistorizationView();
|
|
SpawnHost(view, sharesScopeWith: MeshPeer).Tell(Snapshot(
|
|
(TestNode, RedundancyRole.Secondary), (ForeignNode, RedundancyRole.Primary)));
|
|
|
|
AwaitAssert(() => view.Published.ShouldBe([true]), duration: Timeout);
|
|
}
|
|
|
|
/// <summary>
|
|
/// With NO way to establish which peers share this node's address space, it never stands down.
|
|
/// </summary>
|
|
/// <remarks>
|
|
/// The predicate is null on a node carrying no cluster-scoped role — the legacy, pre-Phase-6
|
|
/// shape, where the redundancy election is fleet-wide and the elected Primary usually belongs to
|
|
/// some other cluster. Duplicates are recoverable; a silent gap is not, so unproven means
|
|
/// capture. Delete the null check in the wiring and this is the case that goes red.
|
|
/// </remarks>
|
|
[Fact]
|
|
public void An_unprovable_scope_never_stands_down()
|
|
{
|
|
var view = new RecordingHistorizationView();
|
|
SpawnHost(view, sharesScopeWith: null).Tell(Snapshot(
|
|
(TestNode, RedundancyRole.Secondary), (MeshPeer, RedundancyRole.Primary)));
|
|
|
|
AwaitAssert(() => view.Published.ShouldBe([true]), duration: Timeout);
|
|
}
|
|
|
|
/// <summary>Primary captures.</summary>
|
|
[Fact]
|
|
public void Primary_captures()
|
|
{
|
|
var view = new RecordingHistorizationView();
|
|
SpawnHost(view, sharesScopeWith: MeshPeer).Tell(Snapshot(
|
|
(TestNode, RedundancyRole.Primary), (MeshPeer, RedundancyRole.Secondary)));
|
|
|
|
// Published, not current — `true` is also the seed, so asserting the property would pass before
|
|
// the actor had processed anything.
|
|
AwaitAssert(() => view.Published.ShouldBe([true]), duration: Timeout);
|
|
}
|
|
|
|
/// <summary>
|
|
/// Unknown role captures — the snapshot never names this node, the documented identity-mismatch
|
|
/// shape.
|
|
/// </summary>
|
|
[Fact]
|
|
public void Unknown_role_captures()
|
|
{
|
|
var view = new RecordingHistorizationView();
|
|
SpawnHost(view, sharesScopeWith: MeshPeer).Tell(Snapshot(
|
|
(MeshPeer, RedundancyRole.Primary), (ForeignNode, RedundancyRole.Secondary)));
|
|
|
|
AwaitAssert(() => view.Published.ShouldBe([true]), duration: Timeout);
|
|
}
|
|
|
|
/// <summary>A promotion must MOVE the view, not merely leave it at its open seed.</summary>
|
|
[Fact]
|
|
public void A_promotion_reopens_a_closed_view()
|
|
{
|
|
var view = new RecordingHistorizationView();
|
|
IActorRef host = SpawnHost(view, sharesScopeWith: MeshPeer);
|
|
|
|
host.Tell(Snapshot((TestNode, RedundancyRole.Secondary), (MeshPeer, RedundancyRole.Primary)));
|
|
AwaitAssert(() => view.Published.ShouldBe([false]), duration: Timeout);
|
|
|
|
host.Tell(Snapshot((TestNode, RedundancyRole.Primary), (MeshPeer, RedundancyRole.Secondary)));
|
|
AwaitAssert(() => view.Published.ShouldBe([false, true]), duration: Timeout);
|
|
}
|
|
|
|
// ---------------- helpers ----------------
|
|
|
|
/// <summary>
|
|
/// A view that remembers every published decision, so a test asserts on the actor's ACTION
|
|
/// rather than on a state its open seed already satisfies.
|
|
/// </summary>
|
|
private sealed class RecordingHistorizationView : IHistorizationRoleView
|
|
{
|
|
private readonly List<bool> _published = [];
|
|
private readonly Lock _gate = new();
|
|
private readonly RedundancyRoleView _inner = new();
|
|
|
|
/// <summary>Every value published so far, in order.</summary>
|
|
public IReadOnlyList<bool> Published
|
|
{
|
|
get { lock (_gate) return _published.ToArray(); }
|
|
}
|
|
|
|
public bool ShouldRecordHistorizedValues => _inner.ShouldRecordHistorizedValues;
|
|
|
|
public void PublishHistorizationRole(bool shouldRecordHistorizedValues)
|
|
{
|
|
lock (_gate) _published.Add(shouldRecordHistorizedValues);
|
|
_inner.PublishHistorizationRole(shouldRecordHistorizedValues);
|
|
}
|
|
}
|
|
|
|
private IActorRef SpawnHost(IHistorizationRoleView view, NodeId? sharesScopeWith) =>
|
|
Sys.ActorOf(DriverHostActor.Props(
|
|
NewInMemoryDbFactory(),
|
|
TestNode,
|
|
ackRouter: null,
|
|
driverMemberCountProvider: () => 2,
|
|
historizationRoleView: view,
|
|
sharesHistorizationScopeWithLocal:
|
|
sharesScopeWith is { } peer ? node => node == peer : null));
|
|
|
|
private static RedundancyStateChanged Snapshot(params (NodeId Node, RedundancyRole Role)[] nodes) =>
|
|
new(
|
|
nodes.Select(n => new NodeRedundancyState(
|
|
n.Node,
|
|
n.Role,
|
|
IsClusterLeader: n.Role == RedundancyRole.Primary,
|
|
IsDriverPrimary: n.Role == RedundancyRole.Primary,
|
|
AsOfUtc: DateTime.UtcNow))
|
|
.ToList(),
|
|
CorrelationId.NewId());
|
|
}
|