fix(v3-batch4-wp4): alarm fan-out hardening — event-delivery test, resolution-failure meter, teardown guards (Wave C review M1/M2/M3/L1/L3)

M1 — over-the-wire event-delivery proof (new NativeAlarmMultiNotifierEventDeliveryTests
in OpcUaServer.IntegrationTests): boots the real server, wires one condition to two
equipment folders, fires ONE transition, and asserts a Server-object subscriber gets
EXACTLY ONE event (the shared-InstanceStateSnapshot queue dedup), plus overlapping
Server + equipment-folder subscribers each get exactly one copy. Captures via the
subscription FastEventCallback keyed by ClientHandle (the per-item Notification/DequeueEvents
path delivers nothing for these conditions in the SDK client — the working capture is the
fast callback).

M2 — resolution-failure signal + path-agreement tripwire (AddressSpaceApplier): when a
native alarm HAS ReferencingEquipmentPaths but NONE resolve to an equipment folder, count
it as a failed node (degraded-apply meter) + LogWarning instead of a silent skip; documented
the DisplayName==Name coupling as a binding invariant. Tests:
Composer_referencing_equipment_paths_resolve_back_to_equipment_ids_in_the_applier (real
composer output → applier inversion) + ..._unresolvable_referencing_equipment_counts_as_failed.

M3 — WireAlarmNotifiers is now a RECONCILE (unwires a folder dropped from the supplied set,
bidirectionally) so a future surgical ChangedRawTags path can't leave a de-referenced
equipment receiving the alarm; binding guard comment added on the classifier's
ChangedRawTags→Rebuild branch. Test: WireAlarmNotifiers_reconciles_a_dropped_folder_out_of_the_notifier_set.

L1 — MaterialiseAlarmCondition's kind-swap drop-and-recreate now calls
UnwireAlarmNotifiers(conditionKey) before discarding the old instance (teardown symmetry).

L3 — BuildEquipmentIdByFolderPath LogWarnings on a duplicate Area/Line/Name collision
(defense-in-depth; UNS uniqueness prevents it).

Claude-Session: https://claude.ai/code/session_01LVneM3eh1UtJxEisFXgmox
This commit is contained in:
Joseph Doherty
2026-07-16 12:52:34 -04:00
parent 90e52a4415
commit 3cf3576c75
6 changed files with 465 additions and 8 deletions
@@ -0,0 +1,282 @@
using System.Net;
using System.Net.Sockets;
using Microsoft.Extensions.Logging.Abstractions;
using Opc.Ua;
using Opc.Ua.Client;
using Shouldly;
using ZB.MOM.WW.OtOpcUa.Commons.OpcUa;
using Xunit;
namespace ZB.MOM.WW.OtOpcUa.OpcUaServer.IntegrationTests;
/// <summary>
/// v3 Batch 4 (B4-WP4) — the HEADLINE over-the-wire proof for the multi-notifier native alarm (Wave C
/// review M1): a SINGLE native transition on a condition wired to MULTIPLE notifier roots (its raw
/// device-folder parent + N referencing equipment folders) delivers <b>exactly one</b> event copy to a
/// Server-object event subscriber — the Part 9 shared-<c>InstanceStateSnapshot</c> dedup
/// (<c>MonitoredItem.QueueEvent</c> → <c>IsEventContainedInQueue</c>), NOT one copy per root. This is the
/// regression guard the design's "duplicate-report fallback rejected" decision rests on, and it is exactly
/// the multi-root topology (2 equipment folders → 2 notifier paths up to Server) where a per-root
/// re-report would surface as double delivery.
///
/// <para>Boots the real <see cref="OtOpcUaSdkServer"/> exactly as <c>SubscriptionSurvivalTests</c> does,
/// materialises the raw condition + two referencing equipment folders + the notifier wiring through the
/// production <see cref="SdkAddressSpaceSink"/>, opens a real client subscription with event monitored
/// items (captured via the subscription <see cref="Subscription.FastEventCallback"/>, keyed by
/// <c>ClientHandle</c>), fires ONE transition via <see cref="SdkAddressSpaceSink.WriteAlarmCondition"/>,
/// and counts the delivered events. Heavy in-process server+client integration — runs in the serial
/// integration pass, not the lightweight unit sweep.</para>
/// </summary>
public sealed class NativeAlarmMultiNotifierEventDeliveryTests
{
private const string ServerUri = "urn:OtOpcUa.MultiNotifierEventDelivery";
// The raw device-oriented topology (ns=Raw) + two referencing equipment folders (ns=UNS).
private const string RawDeviceFolder = "Plant/Modbus/dev1";
private const string RawAlarmPath = "Plant/Modbus/dev1/temp_hi";
private const string Equip1 = "EQ-filling-line1-station1";
private const string Equip2 = "EQ-filling-line2-station2";
private const string AlarmMessage = "over-temperature (multi-notifier test)";
// Index of the Message select clause in the event filter (see BuildEventFilter): [0]=EventId, [1]=Message.
private const int MessageFieldIndex = 1;
/// <summary>One native transition on a condition wired to two equipment folders delivers EXACTLY ONE event
/// to a Server-object subscriber (not one per notifier root) — the shared-snapshot queue dedup.</summary>
[Fact]
public async Task Single_transition_delivers_exactly_one_event_at_the_Server_object()
{
var pkiRoot = Path.Combine(Path.GetTempPath(), $"otopcua-multinotif-evt-{Guid.NewGuid():N}");
var port = AllocateFreePort();
var ct = TestContext.Current.CancellationToken;
try
{
var (server, host) = await BootServerAsync(port, pkiRoot + "-srv", ServerUri, ct);
await using var _ = host;
var sink = new SdkAddressSpaceSink(server.NodeManager!);
WireSingleConditionToTwoEquipment(sink);
using var session = await OpenSessionAsync($"opc.tcp://127.0.0.1:{port}/OtOpcUa", ct);
var collector = new EventCollector();
var subscription = new Subscription(session.DefaultSubscription) { PublishingInterval = 100 };
subscription.FastEventCallback = collector.OnEvents;
session.AddSubscription(subscription);
await subscription.CreateAsync(ct);
var serverItem = AddEventItem(subscription, ObjectIds.Server);
await subscription.ApplyChangesAsync(ct);
// Fire ONE genuine transition (inactive+acked → active+unacked). No ConditionRefresh is issued, so
// the ONLY event is this transition — a per-root re-report would show up as a second copy.
sink.WriteAlarmCondition(RawAlarmPath, ActiveSnapshot(), DateTime.UtcNow, AddressSpaceRealm.Raw);
await WaitUntilAsync(() => collector.CountFor(serverItem.ClientHandle) >= 1, TimeSpan.FromSeconds(5));
// Settle past the publishing interval so any (erroneous) duplicate would have arrived.
await Task.Delay(TimeSpan.FromMilliseconds(700), ct);
collector.CountFor(serverItem.ClientHandle).ShouldBe(1,
"the single condition must deliver exactly ONE event to the Server object despite two notifier " +
"roots (2 equipment folders) — a per-root re-report would deliver 2.");
await subscription.DeleteAsync(true, ct);
}
finally
{
SafeDelete(pkiRoot + "-srv");
}
}
/// <summary>Overlapping subscriptions: event monitored items at the Server object AND an equipment folder
/// each receive EXACTLY ONE copy of the one transition — the multi-root topology does not multiply delivery
/// to any single subscriber, and the equipment-folder subscriber (ns=UNS) sees the native alarm without
/// touching ns=Raw.</summary>
[Fact]
public async Task Overlapping_folder_and_server_subscriptions_each_receive_exactly_one_copy()
{
var pkiRoot = Path.Combine(Path.GetTempPath(), $"otopcua-multinotif-ovl-{Guid.NewGuid():N}");
var port = AllocateFreePort();
var ct = TestContext.Current.CancellationToken;
try
{
var (server, host) = await BootServerAsync(port, pkiRoot + "-srv", ServerUri + ".Overlap", ct);
await using var _ = host;
var sink = new SdkAddressSpaceSink(server.NodeManager!);
WireSingleConditionToTwoEquipment(sink);
using var session = await OpenSessionAsync($"opc.tcp://127.0.0.1:{port}/OtOpcUa", ct);
var unsNs = (ushort)session.NamespaceUris.GetIndex(V3NodeIds.UnsNamespaceUri);
unsNs.ShouldBeGreaterThan((ushort)0);
var collector = new EventCollector();
var subscription = new Subscription(session.DefaultSubscription) { PublishingInterval = 100 };
subscription.FastEventCallback = collector.OnEvents;
session.AddSubscription(subscription);
await subscription.CreateAsync(ct);
var serverItem = AddEventItem(subscription, ObjectIds.Server);
var equipItem = AddEventItem(subscription, new NodeId(Equip1, unsNs));
await subscription.ApplyChangesAsync(ct);
sink.WriteAlarmCondition(RawAlarmPath, ActiveSnapshot(), DateTime.UtcNow, AddressSpaceRealm.Raw);
await WaitUntilAsync(() =>
collector.CountFor(serverItem.ClientHandle) >= 1 &&
collector.CountFor(equipItem.ClientHandle) >= 1,
TimeSpan.FromSeconds(5));
await Task.Delay(TimeSpan.FromMilliseconds(700), ct);
// Each subscriber gets exactly one copy — the shared snapshot is deduped per monitored item; the
// multi-root topology never multiplies delivery.
collector.CountFor(serverItem.ClientHandle).ShouldBe(1, "Server-object subscriber");
collector.CountFor(equipItem.ClientHandle).ShouldBe(1, "equipment-folder subscriber (ns=UNS) sees the native alarm");
await subscription.DeleteAsync(true, ct);
}
finally
{
SafeDelete(pkiRoot + "-srv");
}
}
/// <summary>Materialise the raw device folder + the single native condition at its RawPath + two referencing
/// equipment folders (UNS), then wire the condition as an event notifier of both equipment folders.</summary>
private static void WireSingleConditionToTwoEquipment(SdkAddressSpaceSink sink)
{
sink.EnsureFolder(RawDeviceFolder, parentNodeId: null, displayName: "dev1", realm: AddressSpaceRealm.Raw);
sink.MaterialiseAlarmCondition(RawAlarmPath, RawDeviceFolder, "temp_hi", "OffNormalAlarm", 700, AddressSpaceRealm.Raw, isNative: true);
sink.EnsureFolder(Equip1, parentNodeId: null, displayName: "station1", realm: AddressSpaceRealm.Uns);
sink.EnsureFolder(Equip2, parentNodeId: null, displayName: "station2", realm: AddressSpaceRealm.Uns);
sink.WireAlarmNotifiers(RawAlarmPath, AddressSpaceRealm.Raw, new[] { Equip1, Equip2 }, AddressSpaceRealm.Uns);
}
private static AlarmConditionSnapshot ActiveSnapshot() =>
new(Active: true, Acknowledged: false, Confirmed: true, Enabled: true,
Shelving: AlarmShelvingKind.Unshelved, Severity: 700, Message: AlarmMessage);
private static MonitoredItem AddEventItem(Subscription subscription, NodeId source)
{
var item = new MonitoredItem(subscription.DefaultItem)
{
StartNodeId = source,
AttributeId = Attributes.EventNotifier,
Filter = BuildEventFilter(),
QueueSize = 100,
SamplingInterval = 0,
};
subscription.AddItem(item);
return item;
}
/// <summary>An EventFilter selecting [0]=EventId (dedup identity) and [1]=Message (our discriminator).</summary>
private static EventFilter BuildEventFilter()
{
var filter = new EventFilter();
filter.AddSelectClause(ObjectTypes.BaseEventType, BrowseNames.EventId);
filter.AddSelectClause(ObjectTypes.BaseEventType, BrowseNames.Message);
return filter;
}
/// <summary>Collects events delivered to a subscription's <see cref="Subscription.FastEventCallback"/>,
/// tallying per monitored-item ClientHandle and filtering to our unique alarm Message (so unrelated server
/// events / refresh brackets never count).</summary>
private sealed class EventCollector
{
private readonly object _gate = new();
private readonly Dictionary<uint, int> _byHandle = new();
public void OnEvents(Subscription subscription, EventNotificationList notification, IList<string> stringTable)
{
lock (_gate)
{
foreach (var e in notification.Events)
{
if (e.EventFields.Count > MessageFieldIndex &&
e.EventFields[MessageFieldIndex].Value is LocalizedText lt &&
lt.Text == AlarmMessage)
{
_byHandle[e.ClientHandle] = _byHandle.GetValueOrDefault(e.ClientHandle) + 1;
}
}
}
}
public int CountFor(uint clientHandle)
{
lock (_gate) return _byHandle.GetValueOrDefault(clientHandle);
}
}
private static async Task<(OtOpcUaSdkServer Server, OpcUaApplicationHost Host)> BootServerAsync(
int port, string pkiRoot, string appUri, CancellationToken ct)
{
var options = new OpcUaApplicationHostOptions
{
ApplicationName = appUri,
ApplicationUri = appUri,
OpcUaPort = port,
PublicHostname = "127.0.0.1",
PkiStoreRoot = pkiRoot,
EnabledSecurityProfiles = new List<OpcUaSecurityProfile> { OpcUaSecurityProfile.None },
AutoAcceptUntrustedClientCertificates = true,
};
var server = new OtOpcUaSdkServer();
var host = new OpcUaApplicationHost(options, NullLogger<OpcUaApplicationHost>.Instance);
await host.StartAsync(server, ct);
return (server, host);
}
private static async Task<ISession> OpenSessionAsync(string endpointUrl, CancellationToken ct)
{
var appConfig = new ApplicationConfiguration
{
ApplicationName = "OtOpcUa.MultiNotifierEventDeliveryClient",
ApplicationUri = $"urn:OtOpcUa.MultiNotifierEventDeliveryClient.{Guid.NewGuid():N}",
ApplicationType = ApplicationType.Client,
SecurityConfiguration = TestClientSecurity.Build(TestClientSecurity.AllocatePkiRoot()),
ClientConfiguration = new ClientConfiguration { DefaultSessionTimeout = 60_000 },
};
await appConfig.ValidateAsync(ApplicationType.Client, ct);
appConfig.CertificateValidator.CertificateValidation += (_, e) => e.Accept = true;
var endpoint = await CoreClientUtils.SelectEndpointAsync(
appConfig, endpointUrl, false, DefaultTelemetry.Create(_ => { }), ct);
var endpointConfiguration = EndpointConfiguration.Create(appConfig);
var configuredEndpoint = new ConfiguredEndpoint(null, endpoint, endpointConfiguration);
return await new DefaultSessionFactory(DefaultTelemetry.Create(_ => { })).CreateAsync(
appConfig, configuredEndpoint, updateBeforeConnect: false, checkDomain: false,
sessionName: "MultiNotifierEventDeliveryTests", sessionTimeout: 60_000,
identity: new UserIdentity(new AnonymousIdentityToken()), preferredLocales: null, ct: ct);
}
private static async Task WaitUntilAsync(Func<bool> condition, TimeSpan timeout)
{
var deadline = DateTime.UtcNow + timeout;
while (DateTime.UtcNow < deadline)
{
if (condition()) return;
await Task.Delay(50);
}
condition().ShouldBeTrue("condition not met within timeout");
}
private static int AllocateFreePort()
{
var listener = new TcpListener(IPAddress.Loopback, 0);
listener.Start();
var port = ((IPEndPoint)listener.LocalEndpoint).Port;
listener.Stop();
return port;
}
private static void SafeDelete(string dir)
{
if (Directory.Exists(dir))
{
try { Directory.Delete(dir, recursive: true); }
catch { /* best-effort */ }
}
}
}
@@ -2,6 +2,8 @@ using Microsoft.Extensions.Logging.Abstractions;
using Shouldly;
using Xunit;
using ZB.MOM.WW.OtOpcUa.Commons.OpcUa;
using ZB.MOM.WW.OtOpcUa.Configuration.Entities;
using ZB.MOM.WW.OtOpcUa.Configuration.Enums;
using ZB.MOM.WW.OtOpcUa.OpcUaServer;
namespace ZB.MOM.WW.OtOpcUa.OpcUaServer.Tests;
@@ -152,6 +154,88 @@ public sealed class AddressSpaceApplierRawUnsTests
w.FolderNodeIds.ShouldBe(new[] { "EQ-1", "EQ-2" }, ignoreOrder: true);
}
/// <summary>
/// Wave C review M2 — composer↔applier PATH-AGREEMENT (the tripwire for the latent DisplayName==Name
/// coupling). The composer builds a native alarm's <c>ReferencingEquipmentPaths</c> from the
/// Area/Line/Equipment <c>Name</c>s; the applier's <c>BuildEquipmentIdByFolderPath</c> inverts those
/// paths back to the <c>EquipmentId</c> via the projected DisplayName. Feed a REAL composer output
/// (not a hand-built plan) through <see cref="AddressSpaceApplier.MaterialiseRawSubtree"/> and assert
/// the notifier wired to exactly the backing equipment's id — proving the two path constructions agree
/// end-to-end. If a future editable Area/Line display-name feature made <c>DisplayName != Name</c>, this
/// inversion (and thus the native-alarm fan-out) would break with no deploy error; this test trips first.
/// </summary>
[Fact]
public void Composer_referencing_equipment_paths_resolve_back_to_equipment_ids_in_the_applier()
{
// Real config: a raw alarm tag (Speed) referenced by one equipment (eq-1 under Area "filling" /
// Line "line-1" / Equipment "station-1").
var folder = new RawFolder { RawFolderId = "raw-plant", ClusterId = "c1", Name = "Plant" };
var driver = new DriverInstance
{
DriverInstanceId = "drv-modbus", ClusterId = "c1", RawFolderId = "raw-plant",
Name = "Modbus1", DriverType = "Modbus", DriverConfig = "{}",
};
var device = new Device { DeviceId = "dev-1", DriverInstanceId = "drv-modbus", Name = "PLC-A", DeviceConfig = "{}" };
var speed = new Tag
{
TagId = "t-speed", DeviceId = "dev-1", Name = "Speed", DataType = "Float",
AccessLevel = TagAccessLevel.Read,
TagConfig = "{\"alarm\":{\"alarmType\":\"OffNormalAlarm\",\"severity\":700}}",
};
var area = new UnsArea { UnsAreaId = "area-1", ClusterId = "c1", Name = "filling" };
var line = new UnsLine { UnsLineId = "line-1", UnsAreaId = "area-1", Name = "line-1" };
var equip = new Equipment { EquipmentId = "eq-1", UnsLineId = "line-1", Name = "station-1", MachineCode = "STATION_001" };
var refSpeed = new UnsTagReference { UnsTagReferenceId = "ref-speed", EquipmentId = "eq-1", TagId = "t-speed" };
var composition = AddressSpaceComposer.Compose(
new[] { area }, new[] { line }, new[] { equip },
new[] { driver }, Array.Empty<ScriptedAlarm>(),
unsTagReferences: new[] { refSpeed },
tags: new[] { speed },
rawFolders: new[] { folder },
devices: new[] { device },
tagGroups: Array.Empty<TagGroup>());
// The composer emitted the alarm tag's referencing-equipment path from the Area/Line/Equipment Names.
var alarmTag = composition.RawTags.Single(t => t.Alarm is not null);
alarmTag.ReferencingEquipmentPaths.ShouldBe(new[] { V3NodeIds.Uns("filling", "line-1", "station-1") });
// The applier inverts that path back to the EquipmentId folder and wires the notifier there.
var sink = new RealmRecordingSink();
NewApplier(sink).MaterialiseRawSubtree(composition);
var w = sink.NotifierWirings.ShouldHaveSingleItem();
w.AlarmNodeId.ShouldBe(alarmTag.NodeId);
w.FolderNodeIds.ShouldBe(new[] { "eq-1" }); // resolved path → EquipmentId
}
/// <summary>Wave C review M2 — when a native alarm HAS referencing equipment paths but NONE resolve to an
/// equipment folder (broken ancestry / the latent coupling breaking), the applier surfaces it as a failed
/// node (so the degraded-apply meter fires) instead of silently skipping — no notifier is wired, but the
/// condition itself is still materialised.</summary>
[Fact]
public void MaterialiseRawSubtree_alarm_with_unresolvable_referencing_equipment_counts_as_failed()
{
var sink = new RealmRecordingSink();
// No equipment nodes in the composition → the referencing path can't resolve to an EquipmentId folder.
var composition = new AddressSpaceComposition(
Array.Empty<EquipmentNode>(), Array.Empty<DriverInstancePlan>(), Array.Empty<ScriptedAlarmPlan>())
{
RawTags = new[]
{
new RawTagPlan("t1", "Plant/A/dev/OverTemp", "Plant/A/dev", "drv", "OverTemp", "Boolean",
Writable: false, Alarm: new EquipmentTagAlarmInfo("OffNormalAlarm", 700),
ReferencingEquipmentPaths: new[] { "filling/line1/ghost" }),
},
};
var failed = NewApplier(sink).MaterialiseRawSubtree(composition);
failed.ShouldBeGreaterThanOrEqualTo(1); // resolution failure surfaced (not a silent skip)
sink.NotifierWirings.ShouldBeEmpty(); // nothing wired
sink.Conditions.ShouldHaveSingleItem(); // the condition itself still materialised
}
[Fact]
public void MaterialiseUnsReferences_creates_uns_variable_and_organizes_edge_inheriting_historian()
{
@@ -177,6 +177,28 @@ public sealed class NodeManagerMultiNotifierAlarmTests : IDisposable
await host.DisposeAsync();
}
/// <summary>Wave C review M3 — <see cref="OtOpcUaNodeManager.WireAlarmNotifiers"/> is a RECONCILE: re-wiring
/// with a SHRUNK folder set unwires (bidirectionally) the dropped equipment folder, so a de-referenced
/// equipment stops receiving the alarm. (Guards a future surgical ChangedRawTags path; today the shrink
/// arrives as a full rebuild.)</summary>
[Trait("Category", "Unit")]
[Fact]
public async Task WireAlarmNotifiers_reconciles_a_dropped_folder_out_of_the_notifier_set()
{
var (host, nm) = await BootWithTwoEquipmentAsync();
nm.WireAlarmNotifiers(RawAlarm, AddressSpaceRealm.Raw, new[] { Equip1, Equip2 }, AddressSpaceRealm.Uns);
nm.AlarmNotifierCount(RawAlarm, AddressSpaceRealm.Raw).ShouldBe(2);
// Re-wire with only Equip1 (Equip2 de-referenced) — reconcile unwires Equip2 bidirectionally.
nm.WireAlarmNotifiers(RawAlarm, AddressSpaceRealm.Raw, new[] { Equip1 }, AddressSpaceRealm.Uns);
nm.AlarmNotifierCount(RawAlarm, AddressSpaceRealm.Raw).ShouldBe(1);
nm.FolderNotifierCount(Equip1, AddressSpaceRealm.Uns).ShouldBe(1);
nm.FolderNotifierCount(Equip2, AddressSpaceRealm.Uns).ShouldBe(0); // no dangling inverse to the condition
await host.DisposeAsync();
}
private async Task<(OpcUaApplicationHost Host, OtOpcUaSdkServer Server)> BootAsync()
{
var host = new OpcUaApplicationHost(