dohertj2/lmxopcua
1
0
Fork 0
Code Issues 25 Pull Requests Actions 1 Packages Projects Releases Wiki Activity

fix(alarms): surface a refused native alarm ack synchronously, revert one that fails later (lmxopcua#760)

Browse Source 
The native-ack path is fire-and-forget after the Part 9 Acknowledge returns, so
a Galaxy ack refused for lack of a credential (#749), or rejected by MXAccess,
left the client with Good and the condition showing Acknowledged while Galaxy
was never acked. Both options from the issue, layered:

1. Synchronous refusal for what the driver can decide without I/O. New optional
   IAlarmAcknowledgePreflight (Core.Abstractions); GalaxyDriver refuses an
   anonymous session when the driver has no AlarmAck fallback account.
   DriverHostActor publishes a condition NodeId -> owning-driver snapshot into
   NativeAlarmAckPreflightView (lock-free, volatile); the node manager's new
   NativeAlarmAckPreflight seam, consulted after the authz and Primary gates,
   returns BadUserAccessDenied with the reason - nothing routed, applied or
   reported. A keyed session is not pre-checked (its identity lives in the
   Secrets store, which must not be read under the SDK lock).

2. Revert for everything else. RouteAlarmAck carries the condition NodeId;
   DriverInstanceActor replies AlarmAckFailed on a refusal, a backend failure,
   a driver without an alarm source, or an ack dropped while connecting /
   reconnecting. DriverHostActor re-projects the condition unacknowledged via
   NativeAlarmProjector.TryRevertAcknowledge, but only while the local ack is
   still pending - any driver transition projected since then wins. A disabled
   or unspawned driver is not reverted (unchanged).

GatewayGalaxyAlarmAcknowledger now throws DriverOperationRefusedException on a
non-zero native hresult (wrong password, no ack permission) instead of only
warning, so that case reverts too; as a refusal it neither retries nor trips
the breaker.

Tests (each fails with its fix mutated out): projector revert/supersede,
DriverHostActor refused + failed ack -> unacked AlarmStateUpdate (successful
ack control), preflight view snapshot, node-manager preflight unit test,
over-the-wire ConditionMethodEventTests (BadUserAccessDenied, no route, no
event; positive control), Galaxy preflight + hresult refusal.
Docs: Galaxy.md, AlarmTracking.md, ScriptedAlarms.md, .claude/rules/alarms.md.

Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
This commit is contained in:
Joseph Doherty
2026-11-09 13:12:17 -05:00
parent 0217d36cf4
commit 8e8d4cdf13
21 changed files with 737 additions and 61 deletions
Download Patch File Download Diff File Expand all files Collapse all files
tests/Server/ZB.MOM.WW.OtOpcUa.OpcUaServer.IntegrationTests/ConditionMethodEventTests.cs
+57
View File
@@ -493,6 +493,63 @@ public sealed class ConditionMethodEventTests
finally { await rig.DisposeAsync(); }
}
/// <summary>
/// lmxopcua#760 — an OPC UA Acknowledge of a NATIVE condition that the owning driver's synchronous pre-check
/// refuses (e.g. an anonymous session on a Galaxy driver with no fallback ack account) fails ON THE WIRE with
/// <c>BadUserAccessDenied</c> carrying the driver's reason: nothing is routed, applied or reported. Once the
/// pre-check has no objection the same call is accepted (the positive control).
/// </summary>
[Fact]
public async Task A_native_acknowledge_the_driver_would_refuse_is_refused_synchronously()
{
var ct = TestContext.Current.CancellationToken;
var rig = await StartAsync("CondNativeAckPreflight", encrypted: false, ct);
try
{
rig.Enforce(true, [Key()], ClusterGrant(KeyPrincipal, "acl-key", All));
rig.MaterialiseNative();
var current = PreState("Acknowledge", native: true) with { Message = "base" };
WireSimulatedOwners(rig, native: true, () => current, s => current = s);
var routed = 0;
var driverAck = rig.Nm.NativeAlarmAckRouter!;
rig.Nm.NativeAlarmAckRouter = ack => { Interlocked.Increment(ref routed); driverAck(ack); };
string? refusal = "no Galaxy ack account is configured for this session";
rig.Nm.NativeAlarmAckPreflight = _ => refusal;
using var session = await rig.ConnectAsync(anonymous: false, ct);
var events = new RecordedEvents();
var subscription = await RecordedEvents.SubscribeAsync(session, events, ct);
var item = RecordedEvents.AddEventItem(subscription, ObjectIds.Server);
await subscription.ApplyChangesAsync(ct);
rig.Project(native: true, current);
await RecordedEvents.WaitUntilAsync(() => events.Has(item, "base"));
var condition = ConditionNodeId(session, native: true);
var refused = await Should.ThrowAsync<ServiceResultException>(
CallAsync(session, condition, "Acknowledge", events.LatestEventId(item, "base"), ct));
refused.StatusCode.ShouldBe(StatusCodes.BadUserAccessDenied);
routed.ShouldBe(0, "a refused ack reaches no driver");
rig.Nm.TryGetAlarmCondition(RawAlarmPath, AddressSpaceRealm.Raw)!.AckedState.Id.Value
.ShouldBeFalse("a refused ack applies nothing");
// Positive control: with nothing to object to, the same call is accepted, routed and reported.
refusal = null;
await CallAsync(session, condition, "Acknowledge", events.LatestEventId(item, "base"), ct);
await rig.DrainProjectionsAsync();
routed.ShouldBe(1);
rig.Project(native: true, current = current with { Message = "sentinel" });
await RecordedEvents.WaitUntilAsync(() => events.Has(item, "sentinel"));
var baseEvents = events.ConditionEvents(item, "base");
baseEvents.Count.ShouldBe(2, "the base transition plus the accepted ack's event; the refusal reported nothing");
baseEvents[1][RecordedEvents.AckedField].Value.ShouldBe(true);
await subscription.DeleteAsync(true, ct);
}
finally { await rig.DisposeAsync(); }
}
/// <summary>
/// Acknowledging an acknowledged condition WITHOUT a comment reports nothing, and does not flip a scripted
/// condition's Confirmed out of step with its engine. With a comment it reports exactly one event carrying
tests/Server/ZB.MOM.WW.OtOpcUa.OpcUaServer.Tests/AlarmCommandRouterTests.cs
+41
View File
@@ -450,6 +450,47 @@ public sealed class AlarmCommandRouterTests : IDisposable
await host.DisposeAsync();
}
/// <summary>
/// lmxopcua#760 — when the owning driver's synchronous pre-check refuses the acknowledge, OnAcknowledge
/// returns <c>BadUserAccessDenied</c> carrying the driver's reason, and nothing is routed (so the SDK applies
/// no state change and no event is reported). The pre-check sees the condition NodeId and the session.
/// </summary>
[Fact]
public async Task Native_OnAcknowledge_refused_by_preflight_returns_denied_with_reason_and_routes_nothing()
{
var (host, server) = await BootAsync();
var nm = server.NodeManager!;
var native = new List<NativeAlarmAck>();
var checkedAcks = new List<NativeAlarmAck>();
nm.NativeAlarmAckRouter = native.Add;
nm.NativeAlarmAckPreflight = ack =>
{
checkedAcks.Add(ack);
return "no Galaxy ack account for an anonymous session";
};
nm.EnsureFolder("eq-nak5", parentNodeId: null, displayName: "Equipment NAK5", realm: AddressSpaceRealm.Uns);
nm.MaterialiseAlarmCondition("alm-nak5", "eq-nak5", "HighTemp", "OffNormalAlarm", severity: 700, isNative: true, realm: AddressSpaceRealm.Uns);
var condition = nm.TryGetAlarmCondition("alm-nak5");
condition.ShouldNotBeNull();
var ctx = SessionContext(server, "quinn", OpcUaDataPlaneRoles.AlarmAck);
var result = condition!.OnAcknowledge!(ctx, condition, EventIdBytes(), new LocalizedText("ack"));
result.StatusCode.Code.ShouldBe(StatusCodes.BadUserAccessDenied);
result.LocalizedText.Text.ShouldBe("no Galaxy ack account for an anonymous session");
native.ShouldBeEmpty();
checkedAcks.ShouldHaveSingleItem().ConditionNodeId.ShouldBe("alm-nak5");
// Positive control: the same ack goes through once the pre-check has nothing to object to.
nm.NativeAlarmAckPreflight = _ => null;
condition.OnAcknowledge!(ctx, condition, EventIdBytes(), new LocalizedText("ack")).ShouldBe(ServiceResult.Good);
native.ShouldHaveSingleItem();
await host.DisposeAsync();
}
/// <summary>H6c — a SCRIPTED condition's Acknowledge still routes through the scripted
/// <see cref="OtOpcUaNodeManager.AlarmCommandRouter"/> (Operation == "Acknowledge"); the native
/// router is NOT invoked.</summary>
tests/Server/ZB.MOM.WW.OtOpcUa.Runtime.Tests/Drivers/DriverHostActorNativeAlarmAckRoutingTests.cs
+145 -4
View File
@@ -1,6 +1,7 @@
using System.Collections.Concurrent;
using System.Text.Json;
using Akka.Actor;
using Akka.TestKit;
using Microsoft.EntityFrameworkCore;
using Shouldly;
using Xunit;
@@ -8,12 +9,14 @@ using ZB.MOM.WW.OtOpcUa.Commons.Engines;
using ZB.MOM.WW.OtOpcUa.Commons.Messages.Deploy;
using ZB.MOM.WW.OtOpcUa.Commons.Messages.Fleet;
using ZB.MOM.WW.OtOpcUa.Commons.Messages.Redundancy;
using ZB.MOM.WW.OtOpcUa.Commons.OpcUa;
using ZB.MOM.WW.OtOpcUa.Commons.Types;
using ZB.MOM.WW.OtOpcUa.Configuration;
using ZB.MOM.WW.OtOpcUa.Configuration.Entities;
using ZB.MOM.WW.OtOpcUa.Configuration.Enums;
using ZB.MOM.WW.OtOpcUa.Core.Abstractions;
using ZB.MOM.WW.OtOpcUa.Runtime.Drivers;
using ZB.MOM.WW.OtOpcUa.Runtime.OpcUa;
using ZB.MOM.WW.OtOpcUa.Runtime.Tests.Harness;
namespace ZB.MOM.WW.OtOpcUa.Runtime.Tests.Drivers;
@@ -144,17 +147,140 @@ public sealed class DriverHostActorNativeAlarmAckRoutingTests : RuntimeActorTest
recorder.Acks.Count.ShouldBe(1);
}
/// <summary>
/// lmxopcua#760 — the driver REFUSES the acknowledge (no Galaxy ack credential): the host reverts the
/// condition the node manager optimistically acknowledged, re-projecting it Active + UNacknowledged with
/// its message and severity intact, so the address space stops claiming an ack Galaxy never received.
/// </summary>
[Fact]
public void A_refused_ack_reverts_the_condition_to_unacknowledged()
{
var db = NewInMemoryDbFactory();
var recorder = new RecordingAlarmDriverFactory("GalaxyMxGateway")
{
AckFault = new DriverOperationRefusedException("no Galaxy ack credential"),
};
var deploymentId = SeedV3AlarmDeployment(db, RevA, Driver: "drv-1", Tag: "temp_hi");
var publish = this.CreateCleanOutcomePublishProbe();
var actor = SpawnHostAndApply(db, deploymentId, recorder, publish.Ref);
RaiseAndExpectUnacked(actor, publish);
actor.Tell(new DriverHostActor.RouteNativeAlarmAck(AlarmRawPath, "cmt", "alice"));
var reverted = NextAlarmUpdate(publish);
reverted.AlarmNodeId.ShouldBe(AlarmRawPath);
reverted.Realm.ShouldBe(AddressSpaceRealm.Raw);
reverted.State.Active.ShouldBeTrue();
reverted.State.Acknowledged.ShouldBeFalse();
reverted.State.Message.ShouldBe("temperature high");
reverted.State.Severity.ShouldBe((ushort)700);
recorder.Acks.Count.ShouldBe(1, "the ack did reach the driver, which refused it");
}
/// <summary>
/// lmxopcua#760 — any backend failure (here: the gateway rejecting the ack) reverts the same way, not just
/// a deliberate refusal.
/// </summary>
[Fact]
public void A_failed_ack_reverts_the_condition_to_unacknowledged()
{
var db = NewInMemoryDbFactory();
var recorder = new RecordingAlarmDriverFactory("GalaxyMxGateway")
{
AckFault = new InvalidOperationException("gateway hresult 0x80070005"),
};
var deploymentId = SeedV3AlarmDeployment(db, RevA, Driver: "drv-1", Tag: "temp_hi");
var publish = this.CreateCleanOutcomePublishProbe();
var actor = SpawnHostAndApply(db, deploymentId, recorder, publish.Ref);
RaiseAndExpectUnacked(actor, publish);
actor.Tell(new DriverHostActor.RouteNativeAlarmAck(AlarmRawPath, "cmt", "alice"));
NextAlarmUpdate(publish).State.Acknowledged.ShouldBeFalse();
}
/// <summary>
/// lmxopcua#760 — a SUCCESSFUL ack reverts nothing. Positive control: the next driver Clear projects
/// inactive + ACKED, which it only can when the local ack stayed in the projector (a revert would have
/// left it unacked) — and it must be the very next condition update, so no revert slipped in between.
/// </summary>
[Fact]
public void A_successful_ack_is_not_reverted()
{
var db = NewInMemoryDbFactory();
var recorder = new RecordingAlarmDriverFactory("GalaxyMxGateway");
var deploymentId = SeedV3AlarmDeployment(db, RevA, Driver: "drv-1", Tag: "temp_hi");
var publish = this.CreateCleanOutcomePublishProbe();
var actor = SpawnHostAndApply(db, deploymentId, recorder, publish.Ref);
RaiseAndExpectUnacked(actor, publish);
actor.Tell(new DriverHostActor.RouteNativeAlarmAck(AlarmRawPath, "cmt", "alice"));
AwaitAssert(() => recorder.Acks.Count.ShouldBe(1), duration: Timeout);
actor.Tell(new DriverInstanceActor.AttributeAlarmPublished("drv-1", Evt(AlarmTransitionKind.Clear)));
var next = NextAlarmUpdate(publish);
next.State.Active.ShouldBeFalse();
next.State.Acknowledged.ShouldBeTrue("the successful ack must have survived — no revert");
}
/// <summary>
/// lmxopcua#760 — the host publishes the condition → owning-driver pre-check snapshot the OPC UA node
/// manager uses to refuse an acknowledge synchronously: the driver's own verdict for the condition it
/// owns, nothing for an unknown condition.
/// </summary>
[Fact]
public void The_ack_preflight_view_answers_with_the_owning_drivers_verdict()
{
var db = NewInMemoryDbFactory();
var recorder = new RecordingAlarmDriverFactory("GalaxyMxGateway")
{
PreflightRefusal = key => key is null ? "anonymous sessions cannot acknowledge" : null,
};
var deploymentId = SeedV3AlarmDeployment(db, RevA, Driver: "drv-1", Tag: "temp_hi");
var view = new NativeAlarmAckPreflightView();
SpawnHostAndApply(db, deploymentId, recorder, ackPreflightView: view);
AwaitAssert(() => view.Check(AlarmRawPath, apiKeyId: null).ShouldBe("anonymous sessions cannot acknowledge"),
duration: Timeout);
view.Check(AlarmRawPath, apiKeyId: "0a1b2c").ShouldBeNull();
view.Check("Plant/gw/dev1/does-not-exist", apiKeyId: null).ShouldBeNull();
}
private static AlarmEventArgs Evt(AlarmTransitionKind kind) => new(
new StubAlarmHandleForTests(), SourceNodeId: "Temp", ConditionId: AlarmRawPath, AlarmType: "OffNormalAlarm",
Message: "temperature high", Severity: AlarmSeverity.High, SourceTimestampUtc: DateTime.UtcNow, Kind: kind);
private void RaiseAndExpectUnacked(IActorRef actor, TestProbe publish)
{
actor.Tell(new DriverInstanceActor.AttributeAlarmPublished("drv-1", Evt(AlarmTransitionKind.Raise)));
var raised = NextAlarmUpdate(publish);
raised.State.Active.ShouldBeTrue();
raised.State.Acknowledged.ShouldBeFalse();
}
private static OpcUaPublishActor.AlarmStateUpdate NextAlarmUpdate(TestProbe publish) =>
publish.FishForMessage<OpcUaPublishActor.AlarmStateUpdate>(_ => true, Timeout);
private sealed class StubAlarmHandleForTests : IAlarmSubscriptionHandle
{
public string DiagnosticId => "test-alarm-sub";
}
/// <summary>Spawns the host with the recording alarm-driver factory, dispatches the deployment, and waits
/// for the Applied ACK so the apply (and thus the inverse-map build in PushDesiredSubscriptions) has
/// completed before the test routes an ack.</summary>
private IActorRef SpawnHostAndApply(
IDbContextFactory<OtOpcUaConfigDbContext> db, DeploymentId deploymentId, IDriverFactory factory)
IDbContextFactory<OtOpcUaConfigDbContext> db, DeploymentId deploymentId, IDriverFactory factory,
IActorRef? opcUaPublishActor = null, NativeAlarmAckPreflightView? ackPreflightView = null)
{
var coordinator = CreateTestProbe();
var actor = Sys.ActorOf(DriverHostActor.Props(
db, TestNode, coordinator.Ref,
driverFactory: factory,
localRoles: new HashSet<string> { "driver" }));
localRoles: new HashSet<string> { "driver" },
opcUaPublishActor: opcUaPublishActor,
ackPreflightView: ackPreflightView));
actor.Tell(new DispatchDeployment(deploymentId, RevA, CorrelationId.NewId()));
coordinator.ExpectMsg<ApplyAck>(Timeout).Outcome.ShouldBe(ApplyAckOutcome.Applied);
@@ -248,6 +374,12 @@ public sealed class DriverHostActorNativeAlarmAckRoutingTests : RuntimeActorTest
/// <summary>True once the driver child has reached Connected and opened its alarm subscription.</summary>
public bool AlarmsSubscribed => _driver.AlarmsSubscribed;
/// <summary>When set, every acknowledge is recorded and then fails with this exception.</summary>
public Exception? AckFault { init => _driver.AckFault = value; }
/// <summary>The driver's synchronous pre-check verdict (lmxopcua#760); null ⇒ always proceed.</summary>
public Func<string?, string?>? PreflightRefusal { init => _driver.PreflightRefusal = value; }
/// <inheritdoc />
public IDriver? TryCreate(string driverType, string driverInstanceId, string driverConfigJson)
{
@@ -261,8 +393,17 @@ public sealed class DriverHostActorNativeAlarmAckRoutingTests : RuntimeActorTest
}
/// <summary>An <see cref="IDriver"/> + <see cref="IAlarmSource"/> that records every acknowledge.</summary>
private sealed class RecordingAlarmDriver : IDriver, IAlarmSource
private sealed class RecordingAlarmDriver : IDriver, IAlarmSource, IAlarmAcknowledgePreflight
{
/// <summary>When set, <see cref="AcknowledgeAsync"/> records the request and then throws this.</summary>
public Exception? AckFault { get; set; }
/// <summary>The <see cref="GetAcknowledgeRefusal"/> verdict; null ⇒ always proceed.</summary>
public Func<string?, string?>? PreflightRefusal { get; set; }
/// <inheritdoc />
public string? GetAcknowledgeRefusal(string? apiKeyId) => PreflightRefusal?.Invoke(apiKeyId);
private readonly ConcurrentQueue<AlarmAcknowledgeRequest> _acks = new();
/// <inheritdoc />
public string DriverInstanceId { get; private set; } = string.Empty;
@@ -303,7 +444,7 @@ public sealed class DriverHostActorNativeAlarmAckRoutingTests : RuntimeActorTest
IReadOnlyList<AlarmAcknowledgeRequest> acknowledgements, CancellationToken cancellationToken)
{
foreach (var a in acknowledgements) _acks.Enqueue(a);
return Task.CompletedTask;
return AckFault is { } fault ? Task.FromException(fault) : Task.CompletedTask;
}
/// <inheritdoc />
tests/Server/ZB.MOM.WW.OtOpcUa.Runtime.Tests/Drivers/NativeAlarmProjectorTests.cs
+56
View File
@@ -530,6 +530,62 @@ public class NativeAlarmProjectorTests
snap.Active.ShouldBeTrue();
}
/// <summary>
/// lmxopcua#760 — an acknowledge the driver refused reverts the condition: the snapshot keeps the active
/// state, message, severity and source quality and is unacknowledged, and the next driver Clear projects
/// from the reverted (unacked) prior rather than the optimistic ack.
/// </summary>
[Fact]
public void TryRevertAcknowledge_reverts_a_pending_local_ack()
{
var sut = new NativeAlarmProjector();
sut.Project("n1", Evt(AlarmTransitionKind.Raise, AlarmSeverity.Critical, "boom"));
sut.MarkAcknowledged("n1");
sut.TryRevertAcknowledge("n1", out var reverted).ShouldBeTrue();
reverted.ShouldNotBeNull();
reverted.Active.ShouldBeTrue();
reverted.Acknowledged.ShouldBeFalse();
reverted.Severity.ShouldBe((ushort)900);
reverted.Message.ShouldBe("boom");
reverted.StatusCode.ShouldBe(OpcUaStatus.Good);
sut.Project("n1", Evt(AlarmTransitionKind.Clear)).Acknowledged.ShouldBeFalse(
"the reverted ack must not come back on the next Clear");
}
/// <summary>lmxopcua#760 — a revert happens at most once per local ack, and never without one.</summary>
[Fact]
public void TryRevertAcknowledge_without_a_pending_ack_does_nothing()
{
var sut = new NativeAlarmProjector();
sut.TryRevertAcknowledge("untracked", out var none).ShouldBeFalse();
none.ShouldBeNull();
sut.Project("n1", Evt(AlarmTransitionKind.Raise));
sut.TryRevertAcknowledge("n1", out _).ShouldBeFalse("nothing was acknowledged locally");
sut.MarkAcknowledged("n1");
sut.TryRevertAcknowledge("n1", out _).ShouldBeTrue();
sut.TryRevertAcknowledge("n1", out _).ShouldBeFalse("a second failure report must not revert twice");
}
/// <summary>
/// lmxopcua#760 — once a driver transition for the condition has been projected after the local ack, that
/// transition is authoritative: a late failure report does not rewrite it.
/// </summary>
[Fact]
public void A_driver_transition_after_the_ack_supersedes_a_late_failure()
{
var sut = new NativeAlarmProjector();
sut.Project("n1", Evt(AlarmTransitionKind.Raise));
sut.MarkAcknowledged("n1");
sut.Project("n1", Evt(AlarmTransitionKind.Acknowledge)).Acknowledged.ShouldBeTrue();
sut.TryRevertAcknowledge("n1", out var snapshot).ShouldBeFalse();
snapshot.ShouldBeNull();
}
private static AlarmEventArgs Evt(
AlarmTransitionKind kind,
AlarmSeverity sev = AlarmSeverity.High,