Files
lmxopcua/tests/Server/ZB.MOM.WW.OtOpcUa.Runtime.Tests/Historian/HistorianAdapterActorTests.cs
Joseph Doherty 154171f48c test(runtime): fix the intermittent Runtime.Tests failure — 4 ordering/logic defects + the presence-budget class (#500)
The reported symptom was "Runtime.Tests occasionally reports Failed: 1" with no test
name. Instrumenting 30 full-assembly runs reproduced it at 13% (4 runs) and showed it
was never ONE flaky test: five failures across three distinct tests in that batch, and
five more distinct tests over the verification rounds that followed.

Two hypotheses were tested and DISPROVED by measurement before anything was changed:

- Cluster-formation timeout. Every test in this assembly forms a real single-node Akka
  cluster over TCP in RuntimeActorTestBase's constructor and waits 5 s for MemberStatus.Up
  — 219 formations per run. Instrumented: idle max 960 ms, under-load max 1470 ms, zero
  timeouts. A 3.4x margin; not the cause.
- Ephemeral-port exhaustion from that bind churn. TIME_WAIT measured at ~0 throughout.
  Not the cause.

FOUR GENUINE ORDERING/LOGIC DEFECTS (none is a timeout):

1. VirtualTagHostActorTests.ApplyVirtualTags_respawns_child_when_plan_changes_in_place
   The test loops to accept RegisterInterest/UnregisterInterest in EITHER order — and
   then asserts on mux.LastSender, which DEPENDS on that order. Under the [Register,
   Unregister] interleaving LastSender is the DYING child and the assertion fails. Now
   captures the sender of the Register message as it arrives. Fully deterministic; no
   timing involved. (Swept the other four LastSender sites: all drain the Unregister
   first, so their ordering is fixed. Only this one was unsafe.)

2. DriverHostActorWriteRoutingTests.Primary_routes_write_via_uns_nodeid_to_driver_by_rawpath
   ApplyAck marks the end of the APPLY, not the point a write can succeed: the child is
   still in Connecting, which deliberately fast-fails writes ("driver not connected"),
   and the NodeId->driver reverse map has not been pushed. Now retries until accepted.
   This does not weaken the assertion — every rejection branch replies WITHOUT reaching
   the driver, so Writes.Count.ShouldBe(1) still means exactly what it did.

3. HistorianAdapterActorTests.Redundancy_snapshot_without_local_node_...
   One 500 ms constant served both presence budgets (AwaitAssert) and absence windows
   (ExpectNoMsg). Different quantities that happened to share a number, so the presence
   budget could not be raised without slowing every absence check. Split into
   AssertTimeout (5 s) and Settle (500 ms).

4. ContinuousHistorizationRecorderTests.Retry_after_writer_failure_eventually_acks
   Waited on `outbox == 0`, which is ALSO true before the first append — so the poll
   could win the race against the recorder and return having observed the INITIAL state,
   after which the CallCount>=2 check outside the block failed against a recorder that
   had not run. Both conditions are now polled together with the retry count as the
   discriminator. (The preceding test in the same file documents this exact trap.)

THE PRESENCE-BUDGET CLASS:

After those four, three consecutive 30-run rounds each still failed once — on a
DIFFERENT test, in a DIFFERENT file, every time (OpcUaPublishActorApplyFailureTests 2 s,
OpcUaPublishActorRebuildTests 2 s, OpcUaPublishActorTests 500 ms). That is a
distribution, not a defect, and fixing it one test at a time was the wrong method.

RuntimeActorTestBase now carries a documented PresenceBudget (15 s) and 57 sub-3 s
AwaitAssert/AwaitCondition durations across four files route through it.

Why this is safe rather than sloppy: a presence budget is an upper bound before giving
up, not a wait — these helpers poll and return the instant the condition holds. Raising
one costs nothing on the happy path and CANNOT make a genuinely failing assertion pass;
it only changes how quickly a real breakage reports. Absence windows are the opposite
(the elapsed time IS the assertion) and were deliberately left untouched with their own
short, individually-calibrated literals.

ScriptedAlarmHostActorTests is a special case, diagnosed rather than merely raised: its
8 s budget was sized for message passing but actually waits on a REAL Roslyn compile
(ApplyScriptedAlarms -> ScriptedAlarmEngine.LoadAsync compiles every predicate). Cold
script compiles are the slowest and most variable thing in the assembly, so that class
gets 30 s with the reason recorded.

Verification: 30 consecutive full-assembly runs, 30 passed / 0 failed, against a 13%
baseline. Full solution builds; all 41 unit-test projects green.

NOT fixed here, filed as #501: DriverHostActorNativeAlarmAckRoutingTests:90 and :116 use
AwaitAssert for an ABSENCE assertion, so they return instantly and prove nothing. They
are excluded from the budget change on purpose — a bigger number does not fix them, and
rewriting them to settle-then-assert may legitimately turn them red.
2026-07-25 20:20:06 -04:00

350 lines
15 KiB
C#
Raw Permalink Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
using Akka.Actor;
using Shouldly;
using Xunit;
using ZB.MOM.WW.OtOpcUa.Commons.Messages.Alerts;
using ZB.MOM.WW.OtOpcUa.Commons.Messages.Redundancy;
using ZB.MOM.WW.OtOpcUa.Commons.Types;
using ZB.MOM.WW.OtOpcUa.Core.Abstractions;
using ZB.MOM.WW.OtOpcUa.Core.AlarmHistorian;
using ZB.MOM.WW.OtOpcUa.Runtime.Historian;
using ZB.MOM.WW.OtOpcUa.Runtime.Tests.Harness;
namespace ZB.MOM.WW.OtOpcUa.Runtime.Tests.Historian;
/// <summary>
/// TestKit coverage for <see cref="HistorianAdapterActor"/>'s Primary-only historization gate.
/// The actor caches this node's <see cref="RedundancyRole"/> from the <c>redundancy-state</c>
/// topic and SKIPS the sink enqueue when the local node is Secondary/Detached so a future
/// per-node feeder writes exactly once across the warm-redundant pair. Unknown/null role
/// default-writes (single-node deploys + the boot window must never silently drop historization).
/// </summary>
public sealed class HistorianAdapterActorTests : RuntimeActorTestBase
{
/// <summary>The local node id the gating tests construct the adapter with.</summary>
private static readonly NodeId LocalNode = new("node-A");
/// <summary>
/// The window an <b>absence</b> assertion waits before concluding nothing arrived
/// (<c>ExpectNoMsg</c>). Its length is a calibration decision — long enough that a message which
/// was going to arrive would have — so it is deliberately NOT generous.
/// </summary>
private static readonly TimeSpan Settle = TimeSpan.FromMilliseconds(500);
/// <summary>
/// The budget a <b>presence</b> assertion may take to become true (<c>AwaitAssert</c>).
/// </summary>
/// <remarks>
/// Separate from <see cref="Settle"/> on purpose, though they once shared its 500 ms. The two are
/// different quantities that merely had the same number: a presence budget is an upper bound before
/// giving up, and <c>AwaitAssert</c> returns the instant the condition holds, so a generous value
/// costs nothing in the passing case and can never make a genuinely failing assertion pass. An
/// absence window is the opposite — every millisecond is spent on every run. Conflating them meant
/// the enqueue assertions could only be given more headroom by slowing every <c>ExpectNoMsg</c> in
/// the class, so they kept a 500 ms budget that a fully parallel assembly run occasionally missed.
/// </remarks>
private static readonly TimeSpan AssertTimeout = TimeSpan.FromSeconds(5);
/// <summary>Thread-safe fake sink that records every <see cref="EnqueueAsync"/> call.</summary>
private sealed class RecordingSink : IAlarmHistorianSink
{
private readonly object _lock = new();
private readonly List<AlarmHistorianEvent> _events = new();
/// <summary>The number of <see cref="EnqueueAsync"/> calls observed so far.</summary>
public int EnqueueCount { get { lock (_lock) { return _events.Count; } } }
/// <summary>A snapshot of every event enqueued so far (in arrival order).</summary>
public IReadOnlyList<AlarmHistorianEvent> Events
{
get { lock (_lock) { return _events.ToArray(); } }
}
/// <inheritdoc />
public Task EnqueueAsync(AlarmHistorianEvent evt, CancellationToken cancellationToken)
{
lock (_lock)
{
_events.Add(evt);
}
return Task.CompletedTask;
}
/// <inheritdoc />
public HistorianSinkStatus GetStatus() => new(
QueueDepth: 0,
DeadLetterDepth: 0,
LastDrainUtc: null,
LastSuccessUtc: null,
LastError: null,
DrainState: HistorianDrainState.Idle);
}
/// <summary>Builds a minimal <see cref="AlarmHistorianEvent"/> for the gate tests.</summary>
private static AlarmHistorianEvent SampleEvent() => new(
AlarmId: "alm-1",
EquipmentPath: "Area/Line/Equip",
AlarmName: "HiHi",
AlarmTypeName: "LimitAlarm",
Severity: AlarmSeverity.High,
EventKind: "Activated",
Message: "level high",
User: "system",
Comment: null,
TimestampUtc: DateTime.UtcNow);
/// <summary>Tell <paramref name="actor"/> a <see cref="RedundancyStateChanged"/> snapshot marking
/// <see cref="LocalNode"/> with <paramref name="role"/> so the gate observes the local role.</summary>
private static void TellRedundancyRole(IActorRef actor, RedundancyRole role) =>
actor.Tell(new RedundancyStateChanged(
new[]
{
new NodeRedundancyState(
NodeId: LocalNode,
Role: role,
IsClusterLeader: role == RedundancyRole.Primary,
IsDriverPrimary: role == RedundancyRole.Primary,
AsOfUtc: DateTime.UtcNow),
},
CorrelationId.NewId()));
/// <summary>Default-write (T1): before any redundancy snapshot — the boot window and the steady
/// state for single-node deploys — the adapter MUST historize. Constructed WITH a localNode but
/// no snapshot sent, so the cached role is unknown ⇒ default-write.</summary>
[Fact]
public void Default_before_redundancy_state_historizes()
{
var sink = new RecordingSink();
var actor = Sys.ActorOf(HistorianAdapterActor.Props(sink, LocalNode));
actor.Tell(SampleEvent());
AwaitAssert(() => sink.EnqueueCount.ShouldBe(1), AssertTimeout);
}
/// <summary>Secondary suppression (T2): when the cached local role is Secondary, the adapter MUST
/// NOT enqueue to the durable sink (the Primary writes the single copy).</summary>
[Fact]
public void Secondary_node_does_not_historize()
{
var sink = new RecordingSink();
var actor = Sys.ActorOf(HistorianAdapterActor.Props(sink, LocalNode));
TellRedundancyRole(actor, RedundancyRole.Secondary);
actor.Tell(SampleEvent());
// Give the (suppressed) fire-and-forget a stable window, then assert nothing landed.
ExpectNoMsg(Settle);
sink.EnqueueCount.ShouldBe(0);
}
/// <summary>Detached suppression (T3): a Detached node likewise MUST NOT historize.</summary>
[Fact]
public void Detached_node_does_not_historize()
{
var sink = new RecordingSink();
var actor = Sys.ActorOf(HistorianAdapterActor.Props(sink, LocalNode));
TellRedundancyRole(actor, RedundancyRole.Detached);
actor.Tell(SampleEvent());
ExpectNoMsg(Settle);
sink.EnqueueCount.ShouldBe(0);
}
/// <summary>Primary writes (T4): when the cached local role is Primary, the adapter historizes as
/// normal (this is the single copy the durable sink sees).</summary>
[Fact]
public void Primary_node_historizes()
{
var sink = new RecordingSink();
var actor = Sys.ActorOf(HistorianAdapterActor.Props(sink, LocalNode));
TellRedundancyRole(actor, RedundancyRole.Primary);
actor.Tell(SampleEvent());
AwaitAssert(() => sink.EnqueueCount.ShouldBe(1), AssertTimeout);
}
/// <summary>Absent-node default-historize (T5): a snapshot that mentions only a DIFFERENT node
/// must NOT update the local cached role — the actor's own node is absent, so the role stays
/// null/unknown and the default-historize path must fire. Partial/stale snapshots MUST NOT
/// silently suppress historization for nodes not yet observed.</summary>
[Fact]
public void Redundancy_snapshot_without_local_node_leaves_role_unknown_and_historizes()
{
var sink = new RecordingSink();
var actor = Sys.ActorOf(HistorianAdapterActor.Props(sink, LocalNode));
// Send a snapshot that only describes a peer node — the local node is absent.
actor.Tell(new RedundancyStateChanged(
new[]
{
new NodeRedundancyState(
NodeId: new NodeId("some-other-node"),
Role: RedundancyRole.Secondary,
IsClusterLeader: false,
IsDriverPrimary: false,
AsOfUtc: DateTime.UtcNow),
},
CorrelationId.NewId()));
actor.Tell(SampleEvent());
// Local role is still unknown ⇒ default-historize path: sink must record exactly one enqueue.
AwaitAssert(() => sink.EnqueueCount.ShouldBe(1), AssertTimeout);
}
/// <summary>Builds an <see cref="AlarmTransitionEvent"/> (the shape published on the <c>alerts</c>
/// DPS topic) for the translate tests, with overridable severity / type / comment / kind.
/// <paramref name="historizeToAveva"/> is <c>bool?</c> so tests can pass <c>null</c> to simulate the
/// rolling-restart / cross-version case (missing field → CLR default null).</summary>
private static AlarmTransitionEvent SampleTransition(
int severity = 750,
string alarmTypeName = "LimitAlarm",
string? comment = "note",
string transitionKind = "Activated",
bool? historizeToAveva = true) => new(
AlarmId: "alm-9",
EquipmentPath: "Area/Line/Equip",
AlarmName: "HiHi",
TransitionKind: transitionKind,
Severity: severity,
Message: "level high",
User: "operator1",
TimestampUtc: DateTime.UtcNow,
AlarmTypeName: alarmTypeName,
Comment: comment,
HistorizeToAveva: historizeToAveva);
/// <summary>Alerts translate (T6): an <see cref="AlarmTransitionEvent"/> off the <c>alerts</c> topic
/// is translated to an <see cref="AlarmHistorianEvent"/> and historized by default (unknown role).
/// The translation must carry AlarmId, AlarmTypeName, EventKind (← TransitionKind), Severity bucket,
/// and Comment through faithfully.</summary>
[Fact]
public void Alerts_transition_is_historized_by_default()
{
var sink = new RecordingSink();
var actor = Sys.ActorOf(HistorianAdapterActor.Props(sink));
actor.Tell(SampleTransition());
AwaitAssert(
() =>
{
sink.EnqueueCount.ShouldBe(1);
var e = sink.Events.ShouldHaveSingleItem();
e.AlarmId.ShouldBe("alm-9");
e.AlarmTypeName.ShouldBe("LimitAlarm");
e.EventKind.ShouldBe("Activated");
e.Severity.ShouldBe(AlarmSeverity.High);
e.Comment.ShouldBe("note");
},
AssertTimeout);
}
/// <summary>Secondary suppression for alerts (T7): a Secondary node must NOT historize a transition
/// off the <c>alerts</c> topic — the Primary writes the single copy (DistributedPubSub fans the
/// single publish to BOTH nodes' historian adapters, so the gate is what makes it exactly-once).</summary>
[Fact]
public void Secondary_node_does_not_historize_alerts_transition()
{
var sink = new RecordingSink();
var actor = Sys.ActorOf(HistorianAdapterActor.Props(sink, LocalNode));
TellRedundancyRole(actor, RedundancyRole.Secondary);
actor.Tell(SampleTransition());
ExpectNoMsg(Settle);
sink.EnqueueCount.ShouldBe(0);
}
/// <summary>Primary writes alerts (T8): a Primary node historizes a transition off the
/// <c>alerts</c> topic (the single copy the durable sink sees).</summary>
[Fact]
public void Primary_node_historizes_alerts_transition()
{
var sink = new RecordingSink();
var actor = Sys.ActorOf(HistorianAdapterActor.Props(sink, LocalNode));
TellRedundancyRole(actor, RedundancyRole.Primary);
actor.Tell(SampleTransition());
AwaitAssert(() => sink.EnqueueCount.ShouldBe(1), AssertTimeout);
}
/// <summary>Per-alarm opt-out (T8b): a Primary node must NOT historize a transition whose
/// <c>HistorizeToAveva</c> is <c>false</c> — that flag is a per-alarm opt-out of DURABLE
/// historization only. The live <c>alerts</c> fan-out already happened upstream (the publish is NOT
/// gated on this flag), so only the durable sink write is suppressed.</summary>
[Fact]
public void Primary_node_does_not_historize_when_opted_out()
{
var sink = new RecordingSink();
var actor = Sys.ActorOf(HistorianAdapterActor.Props(sink, LocalNode));
TellRedundancyRole(actor, RedundancyRole.Primary);
actor.Tell(SampleTransition(historizeToAveva: false));
ExpectNoMsg(Settle);
sink.EnqueueCount.ShouldBe(0);
}
/// <summary>Rolling-restart default-on (T8c): when <c>HistorizeToAveva</c> is <c>null</c> — the shape
/// a cross-version / rolling-restart deserialize produces (old-format message missing the field maps to
/// the CLR default <c>null</c> for <c>bool?</c>) — a Primary node MUST historize. <c>null</c> is the
/// safe default-on posture: no audit row is dropped at a handover, matching the <c>AlarmTypeName</c>
/// null-coalesce precedent in the same <c>HistorianAdapterActor.Translate</c>.</summary>
[Fact]
public void Primary_historizes_when_flag_is_null()
{
var sink = new RecordingSink();
var actor = Sys.ActorOf(HistorianAdapterActor.Props(sink, LocalNode));
TellRedundancyRole(actor, RedundancyRole.Primary);
actor.Tell(SampleTransition(historizeToAveva: null));
AwaitAssert(() => sink.EnqueueCount.ShouldBe(1), AssertTimeout);
}
/// <summary>Severity buckets (T9): the OPC UA 11000 numeric severity on the transition maps onto
/// the coarse <see cref="AlarmSeverity"/> at the same ceilings <c>ScriptedAlarmHostActor.SeverityToInt</c>
/// emits (Low≤250, Medium≤500, High≤750, Critical otherwise). Driven end-to-end through the enqueue.</summary>
[Theory]
[InlineData(250, AlarmSeverity.Low)]
[InlineData(251, AlarmSeverity.Medium)]
[InlineData(500, AlarmSeverity.Medium)]
[InlineData(750, AlarmSeverity.High)]
[InlineData(751, AlarmSeverity.Critical)]
[InlineData(1000, AlarmSeverity.Critical)]
public void Alerts_transition_severity_buckets(int severity, AlarmSeverity expected)
{
var sink = new RecordingSink();
var actor = Sys.ActorOf(HistorianAdapterActor.Props(sink));
actor.Tell(SampleTransition(severity: severity));
AwaitAssert(
() => sink.Events.ShouldHaveSingleItem().Severity.ShouldBe(expected),
AssertTimeout);
}
/// <summary>Rolling-restart null default (T10): an old-format transition deserialized by Akka's JSON
/// serializer applies the CLR default (null) to <c>AlarmTypeName</c> rather than the record's
/// "AlarmCondition" call-site default. The translation must null-coalesce that back to
/// "AlarmCondition" so the historian never stores a null alarm type. Forced here by constructing the
/// transition with <c>AlarmTypeName: null!</c> (simulating the post-deserialization shape).</summary>
[Fact]
public void Alerts_transition_with_missing_AlarmTypeName_defaults()
{
var sink = new RecordingSink();
var actor = Sys.ActorOf(HistorianAdapterActor.Props(sink));
actor.Tell(SampleTransition(alarmTypeName: null!));
AwaitAssert(
() => sink.Events.ShouldHaveSingleItem().AlarmTypeName.ShouldBe("AlarmCondition"),
AssertTimeout);
}
}