154171f48c
The reported symptom was "Runtime.Tests occasionally reports Failed: 1" with no test
name. Instrumenting 30 full-assembly runs reproduced it at 13% (4 runs) and showed it
was never ONE flaky test: five failures across three distinct tests in that batch, and
five more distinct tests over the verification rounds that followed.
Two hypotheses were tested and DISPROVED by measurement before anything was changed:
- Cluster-formation timeout. Every test in this assembly forms a real single-node Akka
cluster over TCP in RuntimeActorTestBase's constructor and waits 5 s for MemberStatus.Up
— 219 formations per run. Instrumented: idle max 960 ms, under-load max 1470 ms, zero
timeouts. A 3.4x margin; not the cause.
- Ephemeral-port exhaustion from that bind churn. TIME_WAIT measured at ~0 throughout.
Not the cause.
FOUR GENUINE ORDERING/LOGIC DEFECTS (none is a timeout):
1. VirtualTagHostActorTests.ApplyVirtualTags_respawns_child_when_plan_changes_in_place
The test loops to accept RegisterInterest/UnregisterInterest in EITHER order — and
then asserts on mux.LastSender, which DEPENDS on that order. Under the [Register,
Unregister] interleaving LastSender is the DYING child and the assertion fails. Now
captures the sender of the Register message as it arrives. Fully deterministic; no
timing involved. (Swept the other four LastSender sites: all drain the Unregister
first, so their ordering is fixed. Only this one was unsafe.)
2. DriverHostActorWriteRoutingTests.Primary_routes_write_via_uns_nodeid_to_driver_by_rawpath
ApplyAck marks the end of the APPLY, not the point a write can succeed: the child is
still in Connecting, which deliberately fast-fails writes ("driver not connected"),
and the NodeId->driver reverse map has not been pushed. Now retries until accepted.
This does not weaken the assertion — every rejection branch replies WITHOUT reaching
the driver, so Writes.Count.ShouldBe(1) still means exactly what it did.
3. HistorianAdapterActorTests.Redundancy_snapshot_without_local_node_...
One 500 ms constant served both presence budgets (AwaitAssert) and absence windows
(ExpectNoMsg). Different quantities that happened to share a number, so the presence
budget could not be raised without slowing every absence check. Split into
AssertTimeout (5 s) and Settle (500 ms).
4. ContinuousHistorizationRecorderTests.Retry_after_writer_failure_eventually_acks
Waited on `outbox == 0`, which is ALSO true before the first append — so the poll
could win the race against the recorder and return having observed the INITIAL state,
after which the CallCount>=2 check outside the block failed against a recorder that
had not run. Both conditions are now polled together with the retry count as the
discriminator. (The preceding test in the same file documents this exact trap.)
THE PRESENCE-BUDGET CLASS:
After those four, three consecutive 30-run rounds each still failed once — on a
DIFFERENT test, in a DIFFERENT file, every time (OpcUaPublishActorApplyFailureTests 2 s,
OpcUaPublishActorRebuildTests 2 s, OpcUaPublishActorTests 500 ms). That is a
distribution, not a defect, and fixing it one test at a time was the wrong method.
RuntimeActorTestBase now carries a documented PresenceBudget (15 s) and 57 sub-3 s
AwaitAssert/AwaitCondition durations across four files route through it.
Why this is safe rather than sloppy: a presence budget is an upper bound before giving
up, not a wait — these helpers poll and return the instant the condition holds. Raising
one costs nothing on the happy path and CANNOT make a genuinely failing assertion pass;
it only changes how quickly a real breakage reports. Absence windows are the opposite
(the elapsed time IS the assertion) and were deliberately left untouched with their own
short, individually-calibrated literals.
ScriptedAlarmHostActorTests is a special case, diagnosed rather than merely raised: its
8 s budget was sized for message passing but actually waits on a REAL Roslyn compile
(ApplyScriptedAlarms -> ScriptedAlarmEngine.LoadAsync compiles every predicate). Cold
script compiles are the slowest and most variable thing in the assembly, so that class
gets 30 s with the reason recorded.
Verification: 30 consecutive full-assembly runs, 30 passed / 0 failed, against a 13%
baseline. Full solution builds; all 41 unit-test projects green.
NOT fixed here, filed as #501: DriverHostActorNativeAlarmAckRoutingTests:90 and :116 use
AwaitAssert for an ABSENCE assertion, so they return instantly and prove nothing. They
are excluded from the budget change on purpose — a bigger number does not fix them, and
rewriting them to settle-then-assert may legitimately turn them red.
690 lines
37 KiB
C#
690 lines
37 KiB
C#
using System.Collections.Concurrent;
|
|
using System.Text.Json;
|
|
using Akka.Actor;
|
|
using Microsoft.EntityFrameworkCore;
|
|
using Microsoft.Extensions.Logging.Abstractions;
|
|
using Shouldly;
|
|
using Xunit;
|
|
using ZB.MOM.WW.OtOpcUa.Commons.OpcUa;
|
|
using ZB.MOM.WW.OtOpcUa.Commons.Types;
|
|
using ZB.MOM.WW.OtOpcUa.Configuration;
|
|
using ZB.MOM.WW.OtOpcUa.Configuration.Entities;
|
|
using ZB.MOM.WW.OtOpcUa.Configuration.Enums;
|
|
using ZB.MOM.WW.OtOpcUa.OpcUaServer;
|
|
using ZB.MOM.WW.OtOpcUa.Runtime.OpcUa;
|
|
using ZB.MOM.WW.OtOpcUa.Runtime.Tests.Harness;
|
|
|
|
namespace ZB.MOM.WW.OtOpcUa.Runtime.Tests.OpcUa;
|
|
|
|
public sealed class OpcUaPublishActorRebuildTests : RuntimeActorTestBase
|
|
{
|
|
/// <summary>R2-07 — RebuildAddressSpace with dbFactory loads the artifact, composes, and applies. Two
|
|
/// brand-new equipment are a PureAdd: the Materialise passes create the equipment folders WITHOUT a full
|
|
/// rebuild (existing subscriptions survive), and the added parents are announced via NodeAdded. (Was:
|
|
/// pre-R2-07 this asserted one full rebuild.)</summary>
|
|
[Fact]
|
|
public void RebuildAddressSpace_with_dbFactory_loads_artifact_composes_and_applies()
|
|
{
|
|
var db = NewInMemoryDbFactory();
|
|
var sink = new RecordingSink();
|
|
var applier = new AddressSpaceApplier(sink, NullLogger<AddressSpaceApplier>.Instance);
|
|
|
|
SeedDeployment(db, equipmentIds: new[] { "eq-1", "eq-2" }, driverIds: new[] { "drv-1" });
|
|
|
|
var actor = Sys.ActorOf(OpcUaPublishActor.PropsForTests(
|
|
sink: sink,
|
|
dbFactory: db,
|
|
applier: applier));
|
|
|
|
actor.Tell(new OpcUaPublishActor.RebuildAddressSpace(CorrelationId.NewId()));
|
|
|
|
AwaitAssert(() =>
|
|
{
|
|
// PureAdd: the equipment folders are materialised, no full rebuild, and the added parent
|
|
// (the shared line-1) is announced exactly once.
|
|
sink.Calls.ShouldContain("EF:eq-1");
|
|
sink.Calls.ShouldContain("EF:eq-2");
|
|
sink.Calls.ShouldContain("NA:line-1");
|
|
}, duration: PresenceBudget);
|
|
sink.RebuildCalls.ShouldBe(0);
|
|
}
|
|
|
|
/// <summary>Tests that rebuild with no artifact is idempotent no-op.</summary>
|
|
[Fact]
|
|
public void Rebuild_with_no_artifact_is_idempotent_no_op()
|
|
{
|
|
var db = NewInMemoryDbFactory();
|
|
var sink = new RecordingSink();
|
|
var applier = new AddressSpaceApplier(sink, NullLogger<AddressSpaceApplier>.Instance);
|
|
|
|
// No deployment seeded — LoadLatestArtifact returns empty blob.
|
|
var actor = Sys.ActorOf(OpcUaPublishActor.PropsForTests(
|
|
sink: sink,
|
|
dbFactory: db,
|
|
applier: applier));
|
|
|
|
actor.Tell(new OpcUaPublishActor.RebuildAddressSpace(CorrelationId.NewId()));
|
|
Thread.Sleep(200);
|
|
|
|
sink.RebuildCalls.ShouldBe(0);
|
|
}
|
|
|
|
/// <summary>Tests that a second rebuild with the same artifact is an empty-plan no-op. The first deploy
|
|
/// (one new equipment) is a PureAdd — no full rebuild, but it DOES materialise the folder; the second
|
|
/// deploy of the identical composition diffs to an empty plan and the actor short-circuits, adding no
|
|
/// further sink calls.</summary>
|
|
[Fact]
|
|
public void Second_rebuild_with_same_artifact_is_empty_plan_no_op()
|
|
{
|
|
var db = NewInMemoryDbFactory();
|
|
var sink = new RecordingSink();
|
|
var applier = new AddressSpaceApplier(sink, NullLogger<AddressSpaceApplier>.Instance);
|
|
SeedDeployment(db, equipmentIds: new[] { "eq-1" }, driverIds: Array.Empty<string>());
|
|
|
|
var actor = Sys.ActorOf(OpcUaPublishActor.PropsForTests(
|
|
sink: sink, dbFactory: db, applier: applier));
|
|
|
|
actor.Tell(new OpcUaPublishActor.RebuildAddressSpace(CorrelationId.NewId()));
|
|
AwaitAssert(() => sink.Calls.ShouldContain("EF:eq-1"), duration: PresenceBudget);
|
|
sink.RebuildCalls.ShouldBe(0); // PureAdd — no full rebuild
|
|
var callsAfterFirst = sink.Calls.Count;
|
|
|
|
actor.Tell(new OpcUaPublishActor.RebuildAddressSpace(CorrelationId.NewId()));
|
|
Thread.Sleep(200);
|
|
// Same composition ⇒ plan IsEmpty ⇒ applier + materialise passes not run again.
|
|
sink.Calls.Count.ShouldBe(callsAfterFirst);
|
|
sink.RebuildCalls.ShouldBe(0);
|
|
}
|
|
|
|
/// <summary>Tests that rebuild without dbFactory falls back to raw sink rebuild.</summary>
|
|
[Fact]
|
|
public void Rebuild_without_dbFactory_falls_back_to_raw_sink_rebuild()
|
|
{
|
|
// Pre-#109 behavior: no dbFactory wired ⇒ RebuildAddressSpace calls _sink.RebuildAddressSpace
|
|
// directly. The dev/Mac path before the full integration is bound.
|
|
var sink = new RecordingSink();
|
|
var actor = Sys.ActorOf(OpcUaPublishActor.PropsForTests(sink: sink));
|
|
|
|
actor.Tell(new OpcUaPublishActor.RebuildAddressSpace(CorrelationId.NewId()));
|
|
|
|
AwaitAssert(() => sink.RebuildCalls.ShouldBe(1), duration: PresenceBudget);
|
|
}
|
|
|
|
/// <summary>
|
|
/// mesh-phase4 — the wipe-prevention test. A driver-only node has a NULL dbFactory (ConfigDb is
|
|
/// admin-only now) but a WIRED applier. A RebuildAddressSpace carrying the artifact bytes in hand
|
|
/// (from the fetch/cache) must drive the REAL diff-and-apply (the equipment folder is materialised)
|
|
/// and must NOT take the raw-sink <c>RebuildAddressSpace()</c> fallback, which wipes the address
|
|
/// space without re-materialising. Before the guard split, a null dbFactory routed straight to that
|
|
/// fallback and the applier never ran — a FetchAndCache node deployed green but served EMPTY.
|
|
/// </summary>
|
|
[Fact]
|
|
public void Rebuild_with_null_factory_but_wired_applier_diff_applies_from_in_hand_bytes()
|
|
{
|
|
var sink = new RecordingSink();
|
|
var applier = new AddressSpaceApplier(sink, NullLogger<AddressSpaceApplier>.Instance);
|
|
var artifact = BuildNamedEquipmentArtifact(("eq-1", "Pump-1"));
|
|
|
|
// Driver-only node: null dbFactory, wired applier.
|
|
var actor = Sys.ActorOf(OpcUaPublishActor.PropsForTests(
|
|
sink: sink, dbFactory: null, applier: applier));
|
|
|
|
actor.Tell(new OpcUaPublishActor.RebuildAddressSpace(CorrelationId.NewId(), Artifact: artifact));
|
|
|
|
// The in-hand bytes drove the real diff-and-apply — the equipment folder was materialised…
|
|
AwaitAssert(() => sink.Calls.ShouldContain("EF:eq-1"), duration: PresenceBudget);
|
|
// …and the address-space-wiping raw-sink fallback was NOT taken.
|
|
sink.RebuildCalls.ShouldBe(0);
|
|
}
|
|
|
|
/// <summary>
|
|
/// mesh-phase4 / #485 negative — a driver-only node (null dbFactory, wired applier) that gets a
|
|
/// RebuildAddressSpace with NO artifact bytes has no way to obtain a config (there is no ConfigDb to
|
|
/// read). The artifact source yields <c>Array.Empty<byte>()</c>, which the #485 zero-length
|
|
/// guard treats as "no answer": the rebuild is abandoned WITHOUT touching the served address space —
|
|
/// the applier never runs and the raw-sink wipe fallback is never taken. Last-known-good stands.
|
|
/// </summary>
|
|
[Fact]
|
|
public void Rebuild_with_null_factory_and_no_artifact_abandons_without_wiping()
|
|
{
|
|
var sink = new RecordingSink();
|
|
var applier = new AddressSpaceApplier(sink, NullLogger<AddressSpaceApplier>.Instance);
|
|
|
|
var actor = Sys.ActorOf(OpcUaPublishActor.PropsForTests(
|
|
sink: sink, dbFactory: null, applier: applier));
|
|
|
|
actor.Tell(new OpcUaPublishActor.RebuildAddressSpace(CorrelationId.NewId()));
|
|
Thread.Sleep(200);
|
|
|
|
sink.RebuildCalls.ShouldBe(0); // the raw-sink wipe fallback was NOT taken
|
|
sink.Calls.ShouldBeEmpty(); // the applier's diff-and-apply never ran
|
|
}
|
|
|
|
/// <summary>
|
|
/// mesh-phase4 regression (dev seam) — a node with NO applier at all still falls back to the raw-sink
|
|
/// rebuild, and now the fallback is keyed on the applier ALONE: even WITH a dbFactory wired, a null
|
|
/// applier takes the F10b/dev fallback. Proves the guard split did not change the no-applier seam.
|
|
/// </summary>
|
|
[Fact]
|
|
public void Rebuild_with_null_applier_falls_back_to_raw_sink_even_with_dbFactory()
|
|
{
|
|
var db = NewInMemoryDbFactory();
|
|
var sink = new RecordingSink();
|
|
|
|
var actor = Sys.ActorOf(OpcUaPublishActor.PropsForTests(
|
|
sink: sink, dbFactory: db, applier: null));
|
|
|
|
actor.Tell(new OpcUaPublishActor.RebuildAddressSpace(CorrelationId.NewId()));
|
|
|
|
AwaitAssert(() => sink.RebuildCalls.ShouldBe(1), duration: PresenceBudget);
|
|
}
|
|
|
|
/// <summary>
|
|
/// mesh-phase4 regression (Direct mode) — a DB-backed node (dbFactory wired, applier wired) with no
|
|
/// in-hand bytes but a DeploymentId still loads the artifact from the ConfigDb via <c>LoadArtifact</c>
|
|
/// and drives the applier. The guard split must leave the Direct path byte-identical.
|
|
/// </summary>
|
|
[Fact]
|
|
public void Rebuild_direct_mode_with_dbFactory_and_deploymentId_still_loads_and_applies()
|
|
{
|
|
var db = NewInMemoryDbFactory();
|
|
var sink = new RecordingSink();
|
|
var applier = new AddressSpaceApplier(sink, NullLogger<AddressSpaceApplier>.Instance);
|
|
var dep = SeedNamedEquipmentDeployment(db, ("eq-1", "Pump-1"));
|
|
|
|
var actor = Sys.ActorOf(OpcUaPublishActor.PropsForTests(
|
|
sink: sink, dbFactory: db, applier: applier));
|
|
|
|
actor.Tell(new OpcUaPublishActor.RebuildAddressSpace(CorrelationId.NewId(), new DeploymentId(dep)));
|
|
|
|
AwaitAssert(() => sink.Calls.ShouldContain("EF:eq-1"), duration: PresenceBudget);
|
|
sink.RebuildCalls.ShouldBe(0);
|
|
}
|
|
|
|
/// <summary>
|
|
/// Wiring proof for per-ClusterId scoping (Task 4): a multi-cluster artifact must
|
|
/// materialise ONLY the local node's cluster slice. Mirrors the multi-cluster artifact
|
|
/// shape exercised in <c>DeploymentArtifactTests</c> (MAIN + SITE-A, one Galaxy driver +
|
|
/// one equipment tag each — Galaxy points are ordinary equipment tags now). The scoped
|
|
/// rebuild for the SITE-A node must surface the SITE-A tag (<c>t-sa</c> → folder-scoped
|
|
/// variable <c>eq-sa/F/S1</c>) and NOT MAIN's (<c>t-main</c> → <c>eq-main/F/M1</c>); the
|
|
/// mirror holds for the MAIN node. Without the production scoping edit, the unscoped parse
|
|
/// would materialise BOTH variables on every node.
|
|
/// </summary>
|
|
[Fact(Skip = ZB.MOM.WW.OtOpcUa.Runtime.Tests.Drivers.DarkAddressSpaceReasons.EquipmentTagsDarkBatch4)]
|
|
public void Rebuild_materialises_only_the_nodes_cluster()
|
|
{
|
|
// --- SITE-A node: only the SITE-A tag's variable, never MAIN's. ---
|
|
var dbA = NewInMemoryDbFactory();
|
|
var sinkA = new RecordingSink();
|
|
var applierA = new AddressSpaceApplier(sinkA, NullLogger<AddressSpaceApplier>.Instance);
|
|
SeedMultiClusterDeployment(dbA);
|
|
|
|
var siteActor = Sys.ActorOf(OpcUaPublishActor.PropsForTests(
|
|
sink: sinkA,
|
|
dbFactory: dbA,
|
|
applier: applierA,
|
|
localNode: NodeId.Parse("site-a-1:4053")));
|
|
|
|
siteActor.Tell(new OpcUaPublishActor.RebuildAddressSpace(CorrelationId.NewId()));
|
|
|
|
// PureAdd (equipment + tag) ⇒ no full rebuild; the materialise passes still run the cluster slice.
|
|
// t-sa (EquipmentId "eq-sa", FolderPath "F", Name "S1") → folder-scoped variable "eq-sa/F/S1".
|
|
AwaitAssert(() => sinkA.Calls.ShouldContain("EV:eq-sa/F/S1"), duration: PresenceBudget);
|
|
sinkA.RebuildCalls.ShouldBe(0);
|
|
// t-main (MAIN cluster) must NOT leak onto the SITE-A node.
|
|
sinkA.Calls.ShouldNotContain("EV:eq-main/F/M1");
|
|
|
|
// --- MAIN node: the mirror — only MAIN's tag's variable, never SITE-A's. ---
|
|
var dbM = NewInMemoryDbFactory();
|
|
var sinkM = new RecordingSink();
|
|
var applierM = new AddressSpaceApplier(sinkM, NullLogger<AddressSpaceApplier>.Instance);
|
|
SeedMultiClusterDeployment(dbM);
|
|
|
|
var mainActor = Sys.ActorOf(OpcUaPublishActor.PropsForTests(
|
|
sink: sinkM,
|
|
dbFactory: dbM,
|
|
applier: applierM,
|
|
localNode: NodeId.Parse("central-1:4053")));
|
|
|
|
mainActor.Tell(new OpcUaPublishActor.RebuildAddressSpace(CorrelationId.NewId()));
|
|
|
|
AwaitAssert(() => sinkM.Calls.ShouldContain("EV:eq-main/F/M1"), duration: PresenceBudget);
|
|
sinkM.RebuildCalls.ShouldBe(0);
|
|
sinkM.Calls.ShouldNotContain("EV:eq-sa/F/S1");
|
|
}
|
|
|
|
/// <summary>
|
|
/// Seal a 2-cluster deployment (MAIN + SITE-A) whose artifact mirrors the multi-cluster
|
|
/// shape the composer emits: a <c>Clusters</c> + <c>Nodes</c> map, one Equipment namespace +
|
|
/// Galaxy driver + equipment tag per cluster (Galaxy points are ordinary equipment tags now).
|
|
/// Used by <see cref="Rebuild_materialises_only_the_nodes_cluster"/>.
|
|
/// </summary>
|
|
private static void SeedMultiClusterDeployment(IDbContextFactory<OtOpcUaConfigDbContext> dbFactory)
|
|
{
|
|
var artifact = JsonSerializer.SerializeToUtf8Bytes(new
|
|
{
|
|
Clusters = new[] { new { ClusterId = "MAIN" }, new { ClusterId = "SITE-A" } },
|
|
Nodes = new[]
|
|
{
|
|
new { NodeId = "central-1:4053", ClusterId = "MAIN" },
|
|
new { NodeId = "site-a-1:4053", ClusterId = "SITE-A" },
|
|
},
|
|
UnsAreas = new[]
|
|
{
|
|
new { UnsAreaId = "area-main", ClusterId = "MAIN", Name = "main-area" },
|
|
new { UnsAreaId = "area-sa", ClusterId = "SITE-A", Name = "sa-area" },
|
|
},
|
|
UnsLines = new[]
|
|
{
|
|
new { UnsLineId = "line-main", UnsAreaId = "area-main", Name = "main-line" },
|
|
new { UnsLineId = "line-sa", UnsAreaId = "area-sa", Name = "sa-line" },
|
|
},
|
|
Equipment = new[]
|
|
{
|
|
new { EquipmentId = "eq-main", DriverInstanceId = "main-galaxy", UnsLineId = "line-main", Name = "eq-main", MachineCode = "EQ-MAIN" },
|
|
new { EquipmentId = "eq-sa", DriverInstanceId = "sa-galaxy", UnsLineId = "line-sa", Name = "eq-sa", MachineCode = "EQ-SA" },
|
|
},
|
|
DriverInstances = new[]
|
|
{
|
|
new { DriverInstanceId = "main-galaxy", DriverType = "GalaxyMxGateway", DriverConfig = "{}", ClusterId = "MAIN", NamespaceId = "main-ns" },
|
|
new { DriverInstanceId = "sa-galaxy", DriverType = "GalaxyMxGateway", DriverConfig = "{}", ClusterId = "SITE-A", NamespaceId = "sa-ns" },
|
|
},
|
|
Namespaces = new[]
|
|
{
|
|
new { NamespaceId = "main-ns", ClusterId = "MAIN", Kind = 0 }, // NamespaceKind.Equipment
|
|
new { NamespaceId = "sa-ns", ClusterId = "SITE-A", Kind = 0 },
|
|
},
|
|
Tags = new[]
|
|
{
|
|
new { TagId = "t-main", DriverInstanceId = "main-galaxy", EquipmentId = (string?)"eq-main", Name = "M1", FolderPath = "F", DataType = "Boolean", TagConfig = "{}" },
|
|
new { TagId = "t-sa", DriverInstanceId = "sa-galaxy", EquipmentId = (string?)"eq-sa", Name = "S1", FolderPath = "F", DataType = "Boolean", TagConfig = "{}" },
|
|
},
|
|
ScriptedAlarms = Array.Empty<object>(),
|
|
});
|
|
|
|
using var ctx = dbFactory.CreateDbContext();
|
|
ctx.Deployments.Add(new Deployment
|
|
{
|
|
DeploymentId = Guid.NewGuid(),
|
|
RevisionHash = new string('b', 64),
|
|
Status = DeploymentStatus.Sealed,
|
|
CreatedBy = "test",
|
|
SealedAtUtc = DateTime.UtcNow,
|
|
ArtifactBlob = artifact,
|
|
});
|
|
ctx.SaveChanges();
|
|
}
|
|
|
|
/// <summary>R2-07 T4b — a PureAdd deploy (here a single new equipment) does NOT rebuild, and the
|
|
/// NodeAdded announcement is raised AFTER the Materialise passes have created the node (so the node
|
|
/// exists when a re-browsing client arrives). Proven via the ordered <c>Calls</c> queue: the "NA:"
|
|
/// announcement index is strictly after the "EF:" materialise index.</summary>
|
|
[Fact]
|
|
public void Pure_add_announces_node_after_materialising_and_never_rebuilds()
|
|
{
|
|
var db = NewInMemoryDbFactory();
|
|
var sink = new RecordingSink();
|
|
var applier = new AddressSpaceApplier(sink, NullLogger<AddressSpaceApplier>.Instance);
|
|
SeedDeployment(db, equipmentIds: new[] { "eq-1" }, driverIds: Array.Empty<string>());
|
|
|
|
var actor = Sys.ActorOf(OpcUaPublishActor.PropsForTests(sink: sink, dbFactory: db, applier: applier));
|
|
actor.Tell(new OpcUaPublishActor.RebuildAddressSpace(CorrelationId.NewId()));
|
|
|
|
AwaitAssert(() => sink.Calls.ShouldContain("NA:line-1"), duration: PresenceBudget);
|
|
sink.RebuildCalls.ShouldBe(0);
|
|
|
|
var calls = sink.Calls.ToList();
|
|
var materialiseIdx = calls.IndexOf("EF:eq-1");
|
|
var announceIdx = calls.IndexOf("NA:line-1");
|
|
materialiseIdx.ShouldBeGreaterThanOrEqualTo(0);
|
|
announceIdx.ShouldBeGreaterThan(materialiseIdx); // announced AFTER the node was materialised
|
|
}
|
|
|
|
/// <summary>R2-07 T4b — when a deploy forces a full rebuild (here a renamed equipment ⇒ ChangedEquipment
|
|
/// ⇒ Rebuild) even though it ALSO adds an equipment, the post-materialise announce hook is SKIPPED (the
|
|
/// rebuild made subscriptions moot). Proven by capturing the NodeAdded count after the first pure-add
|
|
/// deploy and asserting the rebuild-forcing second deploy adds none.</summary>
|
|
[Fact]
|
|
public void Rebuild_kind_deploy_skips_the_node_added_announce()
|
|
{
|
|
var db = NewInMemoryDbFactory();
|
|
var sink = new RecordingSink();
|
|
var applier = new AddressSpaceApplier(sink, NullLogger<AddressSpaceApplier>.Instance);
|
|
|
|
var dep1 = SeedNamedEquipmentDeployment(db, ("eq-1", "Pump-1"));
|
|
var actor = Sys.ActorOf(OpcUaPublishActor.PropsForTests(sink: sink, dbFactory: db, applier: applier));
|
|
|
|
actor.Tell(new OpcUaPublishActor.RebuildAddressSpace(CorrelationId.NewId(), new DeploymentId(dep1)));
|
|
AwaitAssert(() => sink.Calls.Count(c => c.StartsWith("NA:")).ShouldBe(1), duration: PresenceBudget);
|
|
sink.RebuildCalls.ShouldBe(0);
|
|
var naAfterFirst = sink.Calls.Count(c => c.StartsWith("NA:"));
|
|
|
|
// Second deploy: eq-1 RENAMED (ChangedEquipment ⇒ Rebuild) AND a new eq-2 (an add). The rebuild
|
|
// fires, so the announce hook is skipped despite eq-2 being added.
|
|
var dep2 = SeedNamedEquipmentDeployment(db, ("eq-1", "Pump-1-RENAMED"), ("eq-2", "Pump-2"));
|
|
actor.Tell(new OpcUaPublishActor.RebuildAddressSpace(CorrelationId.NewId(), new DeploymentId(dep2)));
|
|
|
|
AwaitAssert(() => sink.RebuildCalls.ShouldBe(1), duration: PresenceBudget);
|
|
// No new NodeAdded announcement was raised for the rebuild-kind deploy.
|
|
sink.Calls.Count(c => c.StartsWith("NA:")).ShouldBe(naAfterFirst);
|
|
}
|
|
|
|
/// <summary>
|
|
/// Issue #485 — a transient ConfigDb error while loading a deploy's artifact must NOT empty the
|
|
/// served address space. The loader's own log already claims "rebuild becomes no-op", but an
|
|
/// unreadable artifact used to become an EMPTY composition, which the planner then diffed against
|
|
/// the live one as a PureRemove and tore every node down. A blip on the way to the database is not
|
|
/// evidence that the operator deployed an empty configuration: the last-known-good address space
|
|
/// must stand until a real artifact says otherwise.
|
|
/// </summary>
|
|
[Fact]
|
|
public void Rebuild_keeps_the_last_known_good_address_space_when_the_artifact_load_fails()
|
|
{
|
|
var db = new FlakyDbFactory(NewInMemoryDbFactory());
|
|
var sink = new RecordingSink();
|
|
var applier = new AddressSpaceApplier(sink, NullLogger<AddressSpaceApplier>.Instance);
|
|
var dep1 = SeedNamedEquipmentDeployment(db, ("eq-1", "Pump-1"), ("eq-2", "Pump-2"));
|
|
|
|
var actor = Sys.ActorOf(OpcUaPublishActor.PropsForTests(sink: sink, dbFactory: db, applier: applier));
|
|
actor.Tell(new OpcUaPublishActor.RebuildAddressSpace(CorrelationId.NewId(), new DeploymentId(dep1)));
|
|
AwaitAssert(() => sink.Calls.ShouldContain("EF:eq-2"), duration: PresenceBudget);
|
|
var callsAfterFirst = sink.Calls.Count;
|
|
|
|
// The next deploy arrives while the ConfigDb is briefly unreachable.
|
|
db.Fail = true;
|
|
actor.Tell(new OpcUaPublishActor.RebuildAddressSpace(CorrelationId.NewId(), new DeploymentId(Guid.NewGuid())));
|
|
Thread.Sleep(200);
|
|
|
|
sink.Calls.ShouldNotContain(c => c.StartsWith("RE:")); // no equipment subtree torn down
|
|
sink.RebuildCalls.ShouldBe(0);
|
|
sink.Calls.Count.ShouldBe(callsAfterFirst); // the failed load touched nothing at all
|
|
|
|
// …and the actor still remembers WHAT is applied: once the database is back, re-applying the same
|
|
// composition diffs to an empty plan. A clobbered _lastApplied would re-materialise everything.
|
|
db.Fail = false;
|
|
actor.Tell(new OpcUaPublishActor.RebuildAddressSpace(CorrelationId.NewId(), new DeploymentId(dep1)));
|
|
Thread.Sleep(200);
|
|
sink.Calls.Count.ShouldBe(callsAfterFirst);
|
|
}
|
|
|
|
/// <summary>
|
|
/// Positive control for <see cref="Rebuild_keeps_the_last_known_good_address_space_when_the_artifact_load_fails"/>:
|
|
/// the guard suppresses teardown ONLY when the artifact could not be read. A real deploy that
|
|
/// genuinely drops an equipment still tears its subtree down, so the removal path is not simply dead.
|
|
/// </summary>
|
|
[Fact]
|
|
public void Rebuild_still_removes_equipment_a_readable_artifact_genuinely_dropped()
|
|
{
|
|
var db = new FlakyDbFactory(NewInMemoryDbFactory());
|
|
var sink = new RecordingSink();
|
|
var applier = new AddressSpaceApplier(sink, NullLogger<AddressSpaceApplier>.Instance);
|
|
var dep1 = SeedNamedEquipmentDeployment(db, ("eq-1", "Pump-1"), ("eq-2", "Pump-2"));
|
|
|
|
var actor = Sys.ActorOf(OpcUaPublishActor.PropsForTests(sink: sink, dbFactory: db, applier: applier));
|
|
actor.Tell(new OpcUaPublishActor.RebuildAddressSpace(CorrelationId.NewId(), new DeploymentId(dep1)));
|
|
AwaitAssert(() => sink.Calls.ShouldContain("EF:eq-2"), duration: PresenceBudget);
|
|
|
|
// eq-2 really is gone from the next artifact — that IS a configuration change, so it must apply.
|
|
var dep2 = SeedNamedEquipmentDeployment(db, ("eq-1", "Pump-1"));
|
|
actor.Tell(new OpcUaPublishActor.RebuildAddressSpace(CorrelationId.NewId(), new DeploymentId(dep2)));
|
|
|
|
AwaitAssert(() => sink.Calls.ShouldContain("RE:eq-2"), duration: PresenceBudget);
|
|
}
|
|
|
|
/// <summary>An <see cref="IDbContextFactory{TContext}"/> whose <c>CreateDbContext</c> can be made to
|
|
/// throw the way a real transient ConfigDb outage does, so the artifact load fails exactly where issue
|
|
/// #485 observed it failing.</summary>
|
|
private sealed class FlakyDbFactory(IDbContextFactory<OtOpcUaConfigDbContext> inner)
|
|
: IDbContextFactory<OtOpcUaConfigDbContext>
|
|
{
|
|
/// <summary>Gets or sets a value indicating whether the next context creation should throw.</summary>
|
|
public bool Fail { get; set; }
|
|
|
|
/// <summary>Creates a context, or throws when <see cref="Fail"/> is set.</summary>
|
|
/// <returns>A new context.</returns>
|
|
public OtOpcUaConfigDbContext CreateDbContext() => Fail
|
|
? throw new InvalidOperationException(
|
|
"An error occurred using the connection to database 'OtOpcUa' on server 'sql,1433'.")
|
|
: inner.CreateDbContext();
|
|
}
|
|
|
|
/// <summary>Build the artifact bytes for the given (equipmentId, name) rows under a shared line —
|
|
/// WITHOUT sealing them into a deployment. This is what a driver-only / FetchAndCache node carries
|
|
/// in <c>RebuildAddressSpace.Artifact</c> (no ConfigDb read), so tests can drive the in-hand-bytes
|
|
/// path directly.</summary>
|
|
private static byte[] BuildNamedEquipmentArtifact(params (string Id, string Name)[] equipment) =>
|
|
JsonSerializer.SerializeToUtf8Bytes(new
|
|
{
|
|
UnsAreas = new[] { new { UnsAreaId = "area-1", ClusterId = "c1", Name = "Area" } },
|
|
UnsLines = new[] { new { UnsLineId = "line-1", UnsAreaId = "area-1", Name = "Line" } },
|
|
Equipment = equipment.Select(e => new
|
|
{
|
|
EquipmentId = e.Id,
|
|
DriverInstanceId = (string?)null,
|
|
UnsLineId = "line-1",
|
|
Name = e.Name,
|
|
MachineCode = e.Id.ToUpperInvariant(),
|
|
}).ToArray(),
|
|
DriverInstances = Array.Empty<object>(),
|
|
Namespaces = Array.Empty<object>(),
|
|
Tags = Array.Empty<object>(),
|
|
ScriptedAlarms = Array.Empty<object>(),
|
|
});
|
|
|
|
/// <summary>Seal a deployment carrying the given (equipmentId, name) rows under a shared line, returning
|
|
/// the new DeploymentId so a test can target THAT artifact. Renaming an equipment across two such
|
|
/// deployments produces a ChangedEquipment delta (Rebuild), while adding a row is a PureAdd.</summary>
|
|
private static Guid SeedNamedEquipmentDeployment(
|
|
IDbContextFactory<OtOpcUaConfigDbContext> dbFactory,
|
|
params (string Id, string Name)[] equipment)
|
|
{
|
|
var artifact = BuildNamedEquipmentArtifact(equipment);
|
|
|
|
var id = Guid.NewGuid();
|
|
using var ctx = dbFactory.CreateDbContext();
|
|
ctx.Deployments.Add(new Deployment
|
|
{
|
|
DeploymentId = id,
|
|
RevisionHash = new string('d', 64),
|
|
Status = DeploymentStatus.Sealed,
|
|
CreatedBy = "test",
|
|
SealedAtUtc = DateTime.UtcNow,
|
|
ArtifactBlob = artifact,
|
|
});
|
|
ctx.SaveChanges();
|
|
return id;
|
|
}
|
|
|
|
private static void SeedDeployment(
|
|
IDbContextFactory<OtOpcUaConfigDbContext> dbFactory,
|
|
string[] equipmentIds,
|
|
string[] driverIds)
|
|
{
|
|
var artifact = JsonSerializer.SerializeToUtf8Bytes(new
|
|
{
|
|
Equipment = equipmentIds.Select(id => new
|
|
{
|
|
EquipmentId = id,
|
|
MachineCode = id.ToUpperInvariant(),
|
|
UnsLineId = "line-1",
|
|
Name = id,
|
|
}).ToArray(),
|
|
DriverInstances = driverIds.Select(id => new
|
|
{
|
|
DriverInstanceId = id,
|
|
DriverType = "Modbus",
|
|
Enabled = true,
|
|
DriverConfig = "{}",
|
|
}).ToArray(),
|
|
ScriptedAlarms = Array.Empty<object>(),
|
|
});
|
|
|
|
using var ctx = dbFactory.CreateDbContext();
|
|
ctx.Deployments.Add(new Deployment
|
|
{
|
|
DeploymentId = Guid.NewGuid(),
|
|
RevisionHash = new string('a', 64),
|
|
Status = DeploymentStatus.Sealed,
|
|
CreatedBy = "test",
|
|
SealedAtUtc = DateTime.UtcNow,
|
|
ArtifactBlob = artifact,
|
|
});
|
|
ctx.SaveChanges();
|
|
}
|
|
|
|
/// <summary>OpcUaServer-001 — a deploy whose ONLY change is a UNS Area rename (the area Name differs;
|
|
/// no equipment/driver/alarm/tag delta) must NOT short-circuit at the actor's IsEmpty gate: the second
|
|
/// RebuildAddressSpace drives the in-place folder display-name refresh (a surgical
|
|
/// <c>UpdateFolderDisplayName</c> call) without a full rebuild. Proves the planner→applier→sink chain
|
|
/// reaches the apply path through the actor for a rename-only redeploy.</summary>
|
|
[Fact]
|
|
public void Rebuild_with_area_rename_only_updates_folder_in_place_without_rebuild()
|
|
{
|
|
var db = NewInMemoryDbFactory();
|
|
var sink = new RecordingSink();
|
|
var applier = new AddressSpaceApplier(sink, NullLogger<AddressSpaceApplier>.Instance);
|
|
var dep1 = SeedAreaDeployment(db, areaName: "Plant North");
|
|
|
|
var actor = Sys.ActorOf(OpcUaPublishActor.PropsForTests(
|
|
sink: sink, dbFactory: db, applier: applier));
|
|
|
|
// First deploy: the area folder is materialised with the OLD name. R2-07 — this is now a PureAdd
|
|
// (area + line + equipment all added), so NO full rebuild; the folder is materialised directly.
|
|
actor.Tell(new OpcUaPublishActor.RebuildAddressSpace(CorrelationId.NewId(), new DeploymentId(dep1)));
|
|
AwaitAssert(() => sink.Calls.ShouldContain("EF:area-1"), duration: PresenceBudget);
|
|
sink.RebuildCalls.ShouldBe(0);
|
|
|
|
// Second deploy: ONLY the area Name changed — a rename. The actor must reach the apply path and
|
|
// drive a surgical in-place folder rename (NOT a rebuild, NOT a no-op).
|
|
var dep2 = SeedAreaDeployment(db, areaName: "Plant South");
|
|
actor.Tell(new OpcUaPublishActor.RebuildAddressSpace(CorrelationId.NewId(), new DeploymentId(dep2)));
|
|
|
|
AwaitAssert(() =>
|
|
{
|
|
sink.FolderRenameCalls.ShouldContain(("area-1", "Plant South"));
|
|
}, duration: PresenceBudget);
|
|
sink.RebuildCalls.ShouldBe(0); // the rename did NOT force a full rebuild
|
|
}
|
|
|
|
/// <summary>Seal a deployment whose area carries <paramref name="areaName"/>, with a line + one
|
|
/// equipment under it. The equipment makes the FIRST deploy a structural rebuild (so the area folder is
|
|
/// materialised); a later redeploy that changes ONLY the area Name is then a pure rename. Returns the
|
|
/// new DeploymentId so the test can target THAT artifact (apply-time, not-yet-sealed semantics).</summary>
|
|
private static Guid SeedAreaDeployment(IDbContextFactory<OtOpcUaConfigDbContext> dbFactory, string areaName)
|
|
{
|
|
var artifact = JsonSerializer.SerializeToUtf8Bytes(new
|
|
{
|
|
UnsAreas = new[] { new { UnsAreaId = "area-1", ClusterId = "c1", Name = areaName } },
|
|
UnsLines = new[] { new { UnsLineId = "line-1", UnsAreaId = "area-1", Name = "Cell A" } },
|
|
Equipment = new[] { new { EquipmentId = "eq-1", DriverInstanceId = (string?)null, UnsLineId = "line-1", Name = "Pump-1", MachineCode = "EQ-1" } },
|
|
DriverInstances = Array.Empty<object>(),
|
|
Namespaces = Array.Empty<object>(),
|
|
Tags = Array.Empty<object>(),
|
|
ScriptedAlarms = Array.Empty<object>(),
|
|
});
|
|
|
|
var id = Guid.NewGuid();
|
|
using var ctx = dbFactory.CreateDbContext();
|
|
ctx.Deployments.Add(new Deployment
|
|
{
|
|
DeploymentId = id,
|
|
RevisionHash = new string('c', 64),
|
|
Status = DeploymentStatus.Sealed,
|
|
CreatedBy = "test",
|
|
SealedAtUtc = DateTime.UtcNow,
|
|
ArtifactBlob = artifact,
|
|
});
|
|
ctx.SaveChanges();
|
|
return id;
|
|
}
|
|
|
|
private sealed class RecordingSink : IOpcUaAddressSpaceSink, ISurgicalAddressSpaceSink
|
|
{
|
|
/// <summary>Gets the list of recorded sink calls.</summary>
|
|
public ConcurrentQueue<string> Calls { get; } = new();
|
|
/// <summary>Gets or sets the count of rebuild address space calls.</summary>
|
|
public int RebuildCalls;
|
|
/// <summary>Gets the queue of surgical in-place folder display-name update calls (OpcUaServer-001).</summary>
|
|
public ConcurrentQueue<(string FolderNodeId, string DisplayName)> FolderRenameQueue { get; } = new();
|
|
/// <summary>Gets the list of recorded surgical folder display-name update calls.</summary>
|
|
public List<(string FolderNodeId, string DisplayName)> FolderRenameCalls => FolderRenameQueue.ToList();
|
|
/// <summary>Records a value write call.</summary>
|
|
/// <param name="nodeId">The OPC UA node ID.</param>
|
|
/// <param name="value">The value to write.</param>
|
|
/// <param name="quality">The OPC UA quality code.</param>
|
|
/// <param name="ts">The timestamp of the write.</param>
|
|
public void WriteValue(string nodeId, object? value, OpcUaQuality quality, DateTime ts, AddressSpaceRealm realm = AddressSpaceRealm.Uns)
|
|
=> Calls.Enqueue($"WV:{nodeId}");
|
|
/// <summary>Records an alarm condition write call.</summary>
|
|
/// <param name="alarmNodeId">The alarm node ID.</param>
|
|
/// <param name="state">The full condition state snapshot.</param>
|
|
/// <param name="ts">The timestamp of the state change.</param>
|
|
public void WriteAlarmQuality(string alarmNodeId, OpcUaQuality quality, DateTime sourceTimestampUtc, AddressSpaceRealm realm) { }
|
|
public void WriteAlarmCondition(string alarmNodeId, AlarmConditionSnapshot state, DateTime ts, AddressSpaceRealm realm = AddressSpaceRealm.Uns)
|
|
=> Calls.Enqueue($"WA:{alarmNodeId}");
|
|
/// <summary>Records a materialise-alarm-condition call.</summary>
|
|
/// <param name="alarmNodeId">The alarm node ID (== ScriptedAlarmId).</param>
|
|
/// <param name="equipmentNodeId">The equipment folder node ID.</param>
|
|
/// <param name="displayName">The condition display name.</param>
|
|
/// <param name="alarmType">The domain alarm type.</param>
|
|
/// <param name="severity">The domain severity.</param>
|
|
public void MaterialiseAlarmCondition(string alarmNodeId, string equipmentNodeId, string displayName, string alarmType, int severity, AddressSpaceRealm realm, bool isNative = false)
|
|
=> Calls.Enqueue($"MA:{alarmNodeId}");
|
|
/// <summary>Records a folder ensure call.</summary>
|
|
/// <param name="folderNodeId">The folder node ID.</param>
|
|
/// <param name="parentNodeId">The parent node ID, or null if this is a root folder.</param>
|
|
/// <param name="displayName">The display name of the folder.</param>
|
|
public void EnsureFolder(string folderNodeId, string? parentNodeId, string displayName, AddressSpaceRealm realm = AddressSpaceRealm.Uns)
|
|
=> Calls.Enqueue($"EF:{folderNodeId}");
|
|
/// <summary>Records a variable ensure call.</summary>
|
|
/// <param name="variableNodeId">The variable node ID.</param>
|
|
/// <param name="parentFolderNodeId">The parent folder node ID, or null if this is a root variable.</param>
|
|
/// <param name="displayName">The display name of the variable.</param>
|
|
/// <param name="dataType">The OPC UA built-in type name.</param>
|
|
/// <param name="writable">Whether the node is created read/write.</param>
|
|
/// <param name="historianTagname">The resolved historian tagname (null ⇒ not historized).</param>
|
|
public void EnsureVariable(string variableNodeId, string? parentFolderNodeId, string displayName, string dataType, bool writable, AddressSpaceRealm realm, string? historianTagname = null, bool isArray = false, uint? arrayLength = null)
|
|
=> Calls.Enqueue($"EV:{variableNodeId}");
|
|
/// <summary>Records a rebuild address space call.</summary>
|
|
public void RebuildAddressSpace() => Interlocked.Increment(ref RebuildCalls);
|
|
/// <summary>Records a NodeAdded model-change announcement.</summary>
|
|
/// <param name="affectedNodeId">The node under which discovered nodes were added.</param>
|
|
public void RaiseNodesAddedModelChange(string affectedNodeId, AddressSpaceRealm realm = AddressSpaceRealm.Uns) => Calls.Enqueue($"NA:{affectedNodeId}");
|
|
public void WireAlarmNotifiers(string alarmNodeId, AddressSpaceRealm alarmRealm, IReadOnlyList<string> notifierFolderNodeIds, AddressSpaceRealm notifierFolderRealm) { }
|
|
public void AddReference(string sourceNodeId, AddressSpaceRealm sourceRealm, string targetNodeId, AddressSpaceRealm targetRealm, string referenceType = "Organizes") { }
|
|
/// <summary>Records a surgical in-place tag-attribute update (always succeeds in this recording sink).</summary>
|
|
public bool UpdateTagAttributes(string variableNodeId, bool writable, string? historianTagname, string dataType, bool isArray, uint? arrayLength, AddressSpaceRealm realm = AddressSpaceRealm.Uns)
|
|
{
|
|
Calls.Enqueue($"UT:{variableNodeId}");
|
|
return true;
|
|
}
|
|
/// <summary>Records a surgical in-place folder display-name update (always succeeds in this recording sink).</summary>
|
|
/// <param name="folderNodeId">The folder node ID to update in place.</param>
|
|
/// <param name="displayName">The new display name to apply.</param>
|
|
public bool UpdateFolderDisplayName(string folderNodeId, string displayName, AddressSpaceRealm realm = AddressSpaceRealm.Uns)
|
|
{
|
|
FolderRenameQueue.Enqueue((folderNodeId, displayName));
|
|
return true;
|
|
}
|
|
/// <summary>Records a surgical variable-node removal (always succeeds in this recording sink).</summary>
|
|
public bool RemoveVariableNode(string variableNodeId, AddressSpaceRealm realm = AddressSpaceRealm.Uns)
|
|
{
|
|
Calls.Enqueue($"RV:{variableNodeId}");
|
|
return true;
|
|
}
|
|
/// <summary>Records a surgical alarm-condition-node removal (always succeeds in this recording sink).</summary>
|
|
public bool RemoveAlarmConditionNode(string alarmNodeId, AddressSpaceRealm realm = AddressSpaceRealm.Uns)
|
|
{
|
|
Calls.Enqueue($"RA:{alarmNodeId}");
|
|
return true;
|
|
}
|
|
/// <summary>Records a surgical equipment-subtree removal (always succeeds in this recording sink).</summary>
|
|
public bool RemoveEquipmentSubtree(string equipmentNodeId, AddressSpaceRealm realm = AddressSpaceRealm.Uns)
|
|
{
|
|
Calls.Enqueue($"RE:{equipmentNodeId}");
|
|
return true;
|
|
}
|
|
}
|
|
}
|