fix(alarms): surface a refused native alarm ack synchronously, revert one that fails later (lmxopcua#760)

The native-ack path is fire-and-forget after the Part 9 Acknowledge returns, so
a Galaxy ack refused for lack of a credential (#749), or rejected by MXAccess,
left the client with Good and the condition showing Acknowledged while Galaxy
was never acked. Both options from the issue, layered:

1. Synchronous refusal for what the driver can decide without I/O. New optional
   IAlarmAcknowledgePreflight (Core.Abstractions); GalaxyDriver refuses an
   anonymous session when the driver has no AlarmAck fallback account.
   DriverHostActor publishes a condition NodeId -> owning-driver snapshot into
   NativeAlarmAckPreflightView (lock-free, volatile); the node manager's new
   NativeAlarmAckPreflight seam, consulted after the authz and Primary gates,
   returns BadUserAccessDenied with the reason - nothing routed, applied or
   reported. A keyed session is not pre-checked (its identity lives in the
   Secrets store, which must not be read under the SDK lock).

2. Revert for everything else. RouteAlarmAck carries the condition NodeId;
   DriverInstanceActor replies AlarmAckFailed on a refusal, a backend failure,
   a driver without an alarm source, or an ack dropped while connecting /
   reconnecting. DriverHostActor re-projects the condition unacknowledged via
   NativeAlarmProjector.TryRevertAcknowledge, but only while the local ack is
   still pending - any driver transition projected since then wins. A disabled
   or unspawned driver is not reverted (unchanged).

GatewayGalaxyAlarmAcknowledger now throws DriverOperationRefusedException on a
non-zero native hresult (wrong password, no ack permission) instead of only
warning, so that case reverts too; as a refusal it neither retries nor trips
the breaker.

Tests (each fails with its fix mutated out): projector revert/supersede,
DriverHostActor refused + failed ack -> unacked AlarmStateUpdate (successful
ack control), preflight view snapshot, node-manager preflight unit test,
over-the-wire ConditionMethodEventTests (BadUserAccessDenied, no route, no
event; positive control), Galaxy preflight + hresult refusal.
Docs: Galaxy.md, AlarmTracking.md, ScriptedAlarms.md, .claude/rules/alarms.md.

Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
This commit is contained in:
Joseph Doherty
2026-11-09 13:12:17 -05:00
parent 0217d36cf4
commit 8e8d4cdf13
21 changed files with 737 additions and 61 deletions
@@ -1,9 +1,10 @@
namespace ZB.MOM.WW.OtOpcUa.Core.Abstractions;
/// <summary>
/// Thrown by a driver that deliberately refuses an operation before contacting its backend — a
/// configuration / authorization verdict (e.g. no Galaxy ack credential is available for the
/// caller, lmxopcua#749), not a transport or device fault.
/// Thrown when an operation is refused as a configuration / authorization verdict rather than failing
/// on a transport or device fault: by the driver itself before contacting its backend (e.g. no Galaxy
/// ack credential is available for the caller, lmxopcua#749), or by a backend that answered and
/// rejected it (e.g. MXAccess rejecting the operator account on a Galaxy acknowledge, lmxopcua#760).
/// </summary>
/// <remarks>
/// The resilience pipeline neither retries it nor counts it toward the circuit breaker: a refusal
@@ -0,0 +1,24 @@
namespace ZB.MOM.WW.OtOpcUa.Core.Abstractions;
/// <summary>
/// Optional companion to <see cref="IAlarmSource"/>: lets the OPC UA Part 9 <c>Acknowledge</c> method
/// refuse, with a synchronous status, an acknowledge the driver already knows it would refuse
/// (lmxopcua#760). The native ack path is otherwise fire-and-forget after the method returns, so without
/// this the client sees <c>Good</c> for an acknowledge that never reached the upstream alarm system.
/// </summary>
/// <remarks>
/// Called on an OPC UA server thread, under the SDK node-manager lock: an implementation must answer from
/// state it already holds — no I/O, no secret lookups, no blocking. It is a best-effort pre-check for
/// static misconfiguration only; an acknowledge it lets through can still fail later (that path reverts
/// the condition to unacknowledged instead).
/// </remarks>
public interface IAlarmAcknowledgePreflight
{
/// <summary>
/// Returns why an acknowledge from a session that authenticated with <paramref name="apiKeyId"/> would
/// be refused, or <c>null</c> when it would proceed or the driver cannot tell without I/O.
/// </summary>
/// <param name="apiKeyId">The session's API key id, or null for an anonymous session.</param>
/// <returns>A refusal reason safe to show the OPC UA client (never a secret), or null.</returns>
string? GetAcknowledgeRefusal(string? apiKeyId);
}
@@ -38,7 +38,7 @@ namespace ZB.MOM.WW.OtOpcUa.Driver.Galaxy;
/// </para>
/// </remarks>
public sealed class GalaxyDriver
: IDriver, ITagDiscovery, IReadable, IWritable, ISubscribable, IRediscoverable, IHostConnectivityProbe, IAlarmSource, ISkippedTagReporter, IDisposable, IAsyncDisposable
: IDriver, ITagDiscovery, IReadable, IWritable, ISubscribable, IRediscoverable, IHostConnectivityProbe, IAlarmSource, IAlarmAcknowledgePreflight, ISkippedTagReporter, IDisposable, IAsyncDisposable
{
/// <summary>Authored tags this driver could not dial — a TagConfig with no usable
/// <c>attributeRef</c> has no MXAccess address to bind.</summary>
@@ -1966,6 +1966,21 @@ public sealed class GalaxyDriver
}
}
/// <inheritdoc />
/// <remarks>
/// Only the case decidable without I/O (lmxopcua#760): an anonymous session (no API key, so no key-mapped
/// Galaxy identity can exist) on a driver with no <c>AlarmAck.FallbackUser</c>. A keyed session is let
/// through — whether the key has an identity lives in the Secrets store, which must not be read under the
/// SDK lock; <see cref="AcknowledgeAsync"/> still refuses it, and the host reverts the condition.
/// </remarks>
public string? GetAcknowledgeRefusal(string? apiKeyId)
{
if (!string.IsNullOrWhiteSpace(apiKeyId) || (_options.AlarmAck?.HasFallback ?? false)) return null;
return $"the session is anonymous (no API key) and Galaxy driver '{_driverInstanceId}' has no " +
"AlarmAck.FallbackUser configured, so the acknowledge cannot be sent to Galaxy. Connect with an API key " +
"that has a Galaxy ack identity (AdminUI /api-keys) or configure the driver's fallback ack account.";
}
private static string PrincipalOf(AlarmAcknowledgeRequest ack) =>
string.IsNullOrWhiteSpace(ack.OperatorUser) ? "anonymous" : ack.OperatorUser;
@@ -1,6 +1,7 @@
using Microsoft.Extensions.Logging;
using ZB.MOM.WW.MxGateway.Client;
using ZB.MOM.WW.MxGateway.Contracts.Proto;
using ZB.MOM.WW.OtOpcUa.Core.Abstractions;
namespace ZB.MOM.WW.OtOpcUa.Driver.Galaxy.Runtime;
@@ -26,8 +27,12 @@ namespace ZB.MOM.WW.OtOpcUa.Driver.Galaxy.Runtime;
/// A non-OK <see cref="ProtocolStatus"/> means the gateway never reached MXAccess (transport /
/// dispatch / validation failure) and is surfaced as a thrown exception. A non-zero native ack
/// return code (<c>hresult</c> — now the <c>AuthenticateUser</c> HRESULT or the AckMsg write status)
/// means MXAccess rejected the ack; that is logged as a warning rather than thrown so a transient
/// MXAccess hiccup doesn't block the operator workflow — the operator can retry.
/// means MXAccess rejected the ack (a wrong password, an account without ack permission, a failed
/// AckMsg write). It is logged and thrown as a <see cref="DriverOperationRefusedException"/>
/// (lmxopcua#760): the host then reverts the OPC UA condition to unacknowledged instead of leaving it
/// claiming an ack Galaxy never recorded, and the operator can retry. A refusal, not a fault — the
/// gateway answered — so it neither retries nor counts toward the resilience breaker (a mis-keyed
/// account must not open the breaker for every correctly-configured caller).
/// </para>
/// </remarks>
internal sealed class GatewayGalaxyAlarmAcknowledger : IGalaxyAlarmAcknowledger
@@ -98,7 +103,8 @@ internal sealed class GatewayGalaxyAlarmAcknowledger : IGalaxyAlarmAcknowledger
}
/// <summary>
/// Applies the reply semantics: non-OK protocol status throws; a non-zero native code warns.
/// Applies the reply semantics: non-OK protocol status throws; a non-zero native code warns and throws a
/// <see cref="DriverOperationRefusedException"/> (lmxopcua#760).
/// Every message is scrubbed of the password.
/// </summary>
/// <param name="reply">The gateway reply.</param>
@@ -125,6 +131,9 @@ internal sealed class GatewayGalaxyAlarmAcknowledger : IGalaxyAlarmAcknowledger
logger.LogWarning(
"Galaxy AcknowledgeAlarm for {AlarmRef} as {GalaxyUser} returned native ack failure code {Hresult}.",
alarmFullReference, credential.DisplayName, reply.Hresult);
throw new DriverOperationRefusedException(
$"MXAccess rejected the Galaxy acknowledge of '{alarmFullReference}' as '{credential.DisplayName}' "
+ $"(native code 0x{reply.Hresult:X8}); check the account's password and Galaxy acknowledge permission.");
}
}
@@ -49,6 +49,7 @@ public sealed class OtOpcUaServerHostedService : IHostedService, IAsyncDisposabl
private readonly IDriverPrimaryRoleSource? _primaryRoleSource;
private readonly HistorianUpdaterHolder _historianUpdater;
private readonly IHistoryUpdateRoleView? _historyUpdateRoleView;
private readonly NativeAlarmAckPreflightView? _ackPreflightView;
private readonly OpcUaEndpointState _endpointState;
private readonly bool _rethrowStartFailure;
@@ -113,6 +114,8 @@ public sealed class OtOpcUaServerHostedService : IHostedService, IAsyncDisposabl
/// to serve this endpoint, so a failed SDK start RETHROWS there (the host's fatal handler exits non-zero and
/// the supervisor restarts the process); a fused node keeps its AdminUI up and reports the failure through
/// the health probe instead. Null (harnesses) behaves as fused.</param>
/// <param name="ackPreflightView">The native-ack pre-check snapshot <c>DriverHostActor</c> publishes
/// (lmxopcua#760). Optional: registered on every driver-role node; null wires no pre-check.</param>
public OtOpcUaServerHostedService(
IOptions<OpcUaApplicationHostOptions> options,
DeferredAddressSpaceSink deferredSink,
@@ -131,7 +134,8 @@ public sealed class OtOpcUaServerHostedService : IHostedService, IAsyncDisposabl
HistorianUpdaterHolder? historianUpdater = null,
IHistoryUpdateRoleView? historyUpdateRoleView = null,
OpcUaEndpointState? endpointState = null,
EffectiveRoles? roles = null)
EffectiveRoles? roles = null,
NativeAlarmAckPreflightView? ackPreflightView = null)
{
ArgumentNullException.ThrowIfNull(aclOptions);
ArgumentNullException.ThrowIfNull(time);
@@ -156,6 +160,7 @@ public sealed class OtOpcUaServerHostedService : IHostedService, IAsyncDisposabl
_primaryRoleSource = primaryRoleSource;
_historianUpdater = historianUpdater ?? HistorianUpdaterHolder.None;
_historyUpdateRoleView = historyUpdateRoleView;
_ackPreflightView = ackPreflightView;
_endpointState = endpointState ?? new OpcUaEndpointState();
_rethrowStartFailure = roles is { HasDriver: true, HasAdmin: false };
_logger = loggerFactory.CreateLogger<OtOpcUaServerHostedService>();
@@ -279,6 +284,14 @@ public sealed class OtOpcUaServerHostedService : IHostedService, IAsyncDisposabl
}
});
// lmxopcua#760: refuse, with a synchronous BadUserAccessDenied, a native ack the owning driver already
// knows it would refuse (e.g. anonymous session + Galaxy driver without a fallback ack account). A
// volatile snapshot read — safe under the SDK Lock.
if (_ackPreflightView is { } ackPreflightView)
{
_server.SetNativeAlarmAckPreflight(ack => ackPreflightView.Check(ack.ConditionNodeId, ack.ApiKeyId));
}
// A native ack is accepted only on the Primary. The verdict is the one DriverHostActor
// publishes and gates the route above on, so a Secondary refuses (BadInvalidState) instead of applying an
// ack the driver will never see and then reverting it. A volatile property read — safe under the SDK Lock.
@@ -544,6 +557,7 @@ public sealed class OtOpcUaServerHostedService : IHostedService, IAsyncDisposabl
// Clear the native-alarm-ack router so a late ack doesn't Tell a stopping DriverHostActor.
_server?.SetNativeAlarmAckRouter(null);
_server?.SetNativeAlarmAckPrimaryGate(null);
_server?.SetNativeAlarmAckPreflight(null);
_server?.SetHistoryUpdatePrimaryGate(null);
// Restore the Null historian so a late HistoryRead doesn't hit a disposed read client.
_server?.SetHistorianDataSource(null);
@@ -255,6 +255,15 @@ public sealed class OtOpcUaNodeManager : CustomNodeManager2
/// </summary>
public Func<bool>? NativeAlarmAckPrimaryGate { get; set; }
/// <summary>
/// Synchronous pre-check for an OPC UA Acknowledge of a NATIVE condition (lmxopcua#760): returns why the
/// owning driver would refuse it, or null to proceed. Consulted after the authorization and Primary gates;
/// a non-null answer refuses the method with <c>BadUserAccessDenied</c> carrying the reason, and nothing is
/// routed, applied or reported. Runs under the manager's <c>Lock</c>, so it must be a lock-free read of
/// state the host already holds. Null (the default) skips the check.
/// </summary>
public Func<NativeAlarmAck, string?>? NativeAlarmAckPreflight { get; set; }
/// <summary>
/// Whether this node may serve an OPC UA HistoryUpdate right now: the redundancy data-plane
/// verdict (<c>PrimaryGatePolicy.ShouldServiceAsPrimary</c>, as published to <c>IDriverPrimaryRoleSource</c>) —
@@ -2123,7 +2132,8 @@ public sealed class OtOpcUaNodeManager : CustomNodeManager2
/// <param name="comment">The acknowledge comment text, or <c>null</c> when none was supplied.</param>
/// <returns><c>ServiceResult.Good</c> when allowed (the SDK then applies state; the method path reports the event);
/// <c>BadUserAccessDenied</c> when the gate vetoes, <c>BadInvalidState</c> when this node is not the redundancy
/// Primary (<see cref="NativeAlarmAckPrimaryGate"/>) — in both cases no route, no state mutation.</returns>
/// Primary (<see cref="NativeAlarmAckPrimaryGate"/>), <c>BadUserAccessDenied</c> with the driver's reason when
/// <see cref="NativeAlarmAckPreflight"/> refuses it — in every case no route, no state mutation.</returns>
private ServiceResult HandleNativeAlarmAck(ISystemContext context, ConditionState condition, LocalizedText? comment)
{
// Resolve + gate the SAME way HandleAlarmCommand does so native and scripted acks share one authz
@@ -2145,12 +2155,22 @@ public sealed class OtOpcUaNodeManager : CustomNodeManager2
// Same condition-node-id extraction HandleAlarmCommand uses (condition.NodeId.Identifier?.ToString()):
// for a native condition this is the folder-scoped NodeId string the DriverHostActor inverse map
// (_alarmNodeIdByDriverRef value set) keys by, so the next task can resolve it back to (driver, ref).
// Non-blocking by contract (host wires a fire-and-forget dispatch); safe to call under Lock.
NativeAlarmAckRouter?.Invoke(new NativeAlarmAck(
var ack = new NativeAlarmAck(
ConditionNodeId: condition.NodeId.Identifier?.ToString() ?? string.Empty,
Comment: comment?.Text,
OperatorUser: AlarmOperatorName(identity!),
ApiKeyId: identity!.ApiKeyId));
ApiKeyId: identity!.ApiKeyId);
// lmxopcua#760 — a driver that already knows it would refuse this acknowledge (e.g. an anonymous session
// on a Galaxy driver with no fallback ack account) refuses it HERE, synchronously, rather than the client
// getting Good for an ack the upstream alarm system never sees. Lock-free read; nothing routed or applied.
if (NativeAlarmAckPreflight?.Invoke(ack) is { } refusal)
{
return new ServiceResult(StatusCodes.BadUserAccessDenied, new LocalizedText(refusal));
}
// Non-blocking by contract (host wires a fire-and-forget dispatch); safe to call under Lock.
NativeAlarmAckRouter?.Invoke(ack);
// Good ⇒ the SDK applies the node-state change; WireMethodTransitionReporting then reports the one
// condition event (the SDK reports none of its own).
@@ -83,6 +83,20 @@ public sealed class OtOpcUaSdkServer : StandardServer
return true;
}
/// <summary>
/// Wire the synchronous native-ack pre-check (<see cref="OtOpcUaNodeManager.NativeAlarmAckPreflight"/>,
/// lmxopcua#760). Passing <c>null</c> clears it. No-op (returns <c>false</c>) when the node manager has not
/// been created yet (mirrors <see cref="SetNativeAlarmAckRouter"/>).
/// </summary>
/// <param name="preflight">A lock-free read returning the refusal reason, or null to proceed.</param>
/// <returns><c>true</c> when set on a live node manager; <c>false</c> when no node manager exists yet.</returns>
public bool SetNativeAlarmAckPreflight(Func<NativeAlarmAck, string?>? preflight)
{
if (_otOpcUaNodeManager is null) return false;
_otOpcUaNodeManager.NativeAlarmAckPreflight = preflight;
return true;
}
/// <summary>
/// Wire the redundancy verdict that decides whether this node serves an OPC UA HistoryUpdate
/// (<see cref="OtOpcUaNodeManager.HistoryUpdatePrimaryGate"/>); on any other node every authorized node of a
@@ -476,6 +476,9 @@ public sealed class DriverHostActor : ReceiveActor, IWithTimers, IWithUnboundedS
/// <summary>Where the HistoryUpdate verdict is published. Null on nodes that wire no redundancy manager.</summary>
private readonly IHistoryUpdateRoleView? _historyUpdateRoleView;
/// <summary>Where the synchronous native-ack pre-check snapshot is published (lmxopcua#760); null ⇒ none.</summary>
private readonly NativeAlarmAckPreflightView? _ackPreflightView;
/// <summary>Whether any redundancy snapshot has reached this host — the HistoryUpdate gate stays
/// closed until one has (see <see cref="PrimaryGatePolicy.ShouldServeHistoryUpdate"/>).</summary>
private bool _redundancySnapshotReceived;
@@ -617,6 +620,9 @@ public sealed class DriverHostActor : ReceiveActor, IWithTimers, IWithUnboundedS
private sealed record ChildEntry(IActorRef Actor, DriverInstanceSpec Spec, bool Stubbed)
{
/// <summary>The driver's synchronous native-ack pre-check, when it offers one (lmxopcua#760).</summary>
public IAlarmAcknowledgePreflight? AckPreflight { get; init; }
/// <summary>When this child was spawned — the diagnostics' last-change time until it publishes health.</summary>
public DateTime SpawnedUtc { get; } = DateTime.UtcNow;
@@ -749,7 +755,8 @@ public sealed class DriverHostActor : ReceiveActor, IWithTimers, IWithUnboundedS
TimeSpan? applyOutcomeTimeout = null,
bool captureWhilePrimaryUnreachable = true,
IHistoryUpdateRoleView? historyUpdateRoleView = null,
ScriptingOptions? scripting = null) =>
ScriptingOptions? scripting = null,
NativeAlarmAckPreflightView? ackPreflightView = null) =>
// WARNING: this forwarding list is POSITIONAL, and Props.Create compiles it into an
// expression tree. Six IActorRef? parameters and several interface-typed ones mean a
// mis-ordered argument is usually type-compatible and therefore compiles clean, then binds
@@ -763,7 +770,7 @@ public sealed class DriverHostActor : ReceiveActor, IWithTimers, IWithUnboundedS
alarmStateStore, telemetryHub, primaryRoleSource, historizationRoleView,
sharesHistorizationScopeWithLocal, applyWedgeTimeout, reprovisionBackoff, commsRecoveryAuditDelay,
sharesAlarmQueueWithLocal, applyOutcomeTimeout, captureWhilePrimaryUnreachable, historyUpdateRoleView,
scripting));
scripting, ackPreflightView));
/// <summary>Initializes a new DriverHostActor with the specified dependencies.</summary>
/// <param name="dbFactory">Database context factory for configuration database access.</param>
@@ -852,6 +859,8 @@ public sealed class DriverHostActor : ReceiveActor, IWithTimers, IWithUnboundedS
/// redundancy manager.</param>
/// <param name="scripting">The node-wide <c>Scripting</c> run-guard options handed to the scripted-alarm
/// engine. Null ⇒ the compiled defaults.</param>
/// <param name="ackPreflightView">Where the condition → owning-driver native-ack pre-check snapshot is published
/// for the node manager's synchronous refusal (lmxopcua#760). Null ⇒ nothing published (no pre-check).</param>
public DriverHostActor(
IDbContextFactory<OtOpcUaConfigDbContext>? dbFactory,
CommonsNodeId localNode,
@@ -885,8 +894,10 @@ public sealed class DriverHostActor : ReceiveActor, IWithTimers, IWithUnboundedS
TimeSpan? applyOutcomeTimeout = null,
bool captureWhilePrimaryUnreachable = true,
IHistoryUpdateRoleView? historyUpdateRoleView = null,
ScriptingOptions? scripting = null)
ScriptingOptions? scripting = null,
NativeAlarmAckPreflightView? ackPreflightView = null)
{
_ackPreflightView = ackPreflightView;
_scripting = scripting;
_captureWhilePrimaryUnreachable = captureWhilePrimaryUnreachable;
_historyUpdateRoleView = historyUpdateRoleView;
@@ -1545,6 +1556,7 @@ public sealed class DriverHostActor : ReceiveActor, IWithTimers, IWithUnboundedS
Receive<ReconnectDriver>(HandleReconnectDriver);
Receive<RouteNodeWrite>(HandleRouteNodeWrite);
Receive<RouteNativeAlarmAck>(HandleRouteNativeAlarmAck);
Receive<DriverInstanceActor.AlarmAckFailed>(HandleAlarmAckFailed);
Receive<RedundancyStateChanged>(OnRedundancyStateChanged);
Receive<Akka.Cluster.ClusterEvent.IMemberEvent>(OnDriverMembershipChanged);
Receive<Akka.Cluster.ClusterEvent.ReachabilityEvent>(OnReachabilityChanged);
@@ -1609,6 +1621,7 @@ public sealed class DriverHostActor : ReceiveActor, IWithTimers, IWithUnboundedS
Receive<ReconnectDriver>(HandleReconnectDriver);
Receive<RouteNodeWrite>(HandleRouteNodeWrite);
Receive<RouteNativeAlarmAck>(HandleRouteNativeAlarmAck);
Receive<DriverInstanceActor.AlarmAckFailed>(HandleAlarmAckFailed);
Receive<RedundancyStateChanged>(OnRedundancyStateChanged);
Receive<Akka.Cluster.ClusterEvent.IMemberEvent>(OnDriverMembershipChanged);
Receive<Akka.Cluster.ClusterEvent.ReachabilityEvent>(OnReachabilityChanged);
@@ -2260,9 +2273,9 @@ public sealed class DriverHostActor : ReceiveActor, IWithTimers, IWithUnboundedS
/// <summary>
/// Routes an inbound native-condition acknowledge (the host Tells this from the OPC UA node-manager
/// side when a client Acknowledges a NATIVE Part 9 condition) to the owning driver child. Mirrors
/// <see cref="HandleRouteNodeWrite"/>'s order + gating, but the ack is fire-and-forget (no reply):
/// the OPC UA Part 9 acknowledge already committed the local condition state and the driver's
/// <see cref="IAlarmSource.AcknowledgeAsync"/> returns no per-condition status.
/// <see cref="HandleRouteNodeWrite"/>'s order + gating, but the ack is fire-and-forget (no reply to the
/// OPC UA client): the Part 9 acknowledge already committed the local condition state. A refused or failed
/// ack comes back as <see cref="DriverInstanceActor.AlarmAckFailed"/> and reverts it (lmxopcua#760).
/// <list type="number">
/// <item>PRIMARY gate FIRST — reuses the same <see cref="_localRole"/> redundancy signal as the
/// inbound-write gate. Only the Primary pushes the ack to the shared upstream alarm system
@@ -2316,13 +2329,68 @@ public sealed class DriverHostActor : ReceiveActor, IWithTimers, IWithUnboundedS
{
_log.Debug("DriverHost {Node}: driver {Driver} not running for alarm ack of {Cond} — dropped",
_localNode, target.DriverInstanceId, msg.ConditionNodeId);
// Deliberately NOT reverted (lmxopcua#760 covers what a driver reports): a disabled or not-yet-spawned
// driver produces no transitions either, so the local ack stands, as before.
return;
}
// Fire-and-forget: the OPC UA Part 9 ack already committed the local condition state, and the
// driver's AcknowledgeAsync surfaces no per-condition status, so there is nothing to reply. The
// driver correlates on ConditionId (= the authored alarm FullName the inverse map keyed on).
entry.Actor.Tell(new DriverInstanceActor.RouteAlarmAck(target.RawPath, msg.Comment, msg.OperatorUser, msg.ApiKeyId));
// The OPC UA Part 9 ack already committed the local condition state; success sends nothing back. A
// refusal or failure comes back as DriverInstanceActor.AlarmAckFailed (keyed by the condition NodeId
// carried here) and reverts the node (HandleAlarmAckFailed). The driver correlates on ConditionId
// (= the authored alarm FullName the inverse map keyed on).
entry.Actor.Tell(new DriverInstanceActor.RouteAlarmAck(
target.RawPath, msg.Comment, msg.OperatorUser, msg.ApiKeyId, ConditionNodeId: msg.ConditionNodeId));
}
/// <summary>
/// A native acknowledge the driver refused or failed to deliver upstream (lmxopcua#760): revert the
/// condition to unacknowledged so the address space stops claiming an ack the alarm system never saw.
/// Mirrors the optimistic-write self-correction — the client got <c>Good</c> on the method call, and the
/// revert reaches it as a fresh condition event (Acked=false, Retain=true).
/// </summary>
/// <param name="msg">The failure report from the driver child.</param>
private void HandleAlarmAckFailed(DriverInstanceActor.AlarmAckFailed msg) =>
RevertNativeAlarmAck(msg.ConditionNodeId, msg.Reason);
/// <summary>
/// Re-projects <paramref name="conditionNodeId"/> as unacknowledged when its local acknowledge is still
/// pending in the projector. A driver transition projected since the ack is authoritative, so a late
/// failure after it changes nothing.
/// </summary>
/// <param name="conditionNodeId">The bare condition NodeId (Raw realm).</param>
/// <param name="reason">Why the acknowledge did not reach the upstream alarm system.</param>
private void RevertNativeAlarmAck(string conditionNodeId, string reason)
{
if (!_nativeAlarmProjector.TryRevertAcknowledge(conditionNodeId, out var snapshot) || snapshot is null)
{
_log.Info("DriverHost {Node}: native-alarm acknowledge of {Cond} did not reach the alarm system ({Reason}); " +
"the condition has since been re-stated by its driver, so it is left as projected",
_localNode, conditionNodeId, reason);
return;
}
_log.Warning("DriverHost {Node}: native-alarm acknowledge of {Cond} did not reach the alarm system ({Reason}); " +
"reverting the condition to unacknowledged", _localNode, conditionNodeId, reason);
// Native conditions live in the Raw realm at their bare NodeId (the apply pass's single raw condition).
_opcUaPublishActor?.Tell(new ZB.MOM.WW.OtOpcUa.Runtime.OpcUa.OpcUaPublishActor.AlarmStateUpdate(
conditionNodeId, snapshot, DateTime.UtcNow, AddressSpaceRealm.Raw));
}
/// <summary>
/// Publishes the condition NodeId → owning-driver native-ack pre-check snapshot (lmxopcua#760). Called
/// whenever the condition map or the set of running children changes; a no-op without a view.
/// </summary>
private void PublishAckPreflight()
{
if (_ackPreflightView is null) return;
var entries = new List<KeyValuePair<string, IAlarmAcknowledgePreflight>>();
foreach (var (conditionNodeId, target) in _driverRefByAlarmNodeId)
{
if (_children.TryGetValue(target.DriverInstanceId, out var child) && child.AckPreflight is { } preflight)
entries.Add(new(conditionNodeId, preflight));
}
_ackPreflightView.Publish(entries);
}
/// <summary>Whether this node should service a Primary-only data-plane operation right now — the single
@@ -2539,6 +2607,9 @@ public sealed class DriverHostActor : ReceiveActor, IWithTimers, IWithUnboundedS
Receive<RouteNativeAlarmAck>(msg =>
_log.Debug("DriverHost {Node}: dropping native-alarm ack for {Node2} while Stale (config DB unreachable)",
_localNode, msg.ConditionNodeId));
// An ack routed before the host went Stale can still come back refused; the revert needs only the
// projector + publish actor, both of which survive Stale.
Receive<DriverInstanceActor.AlarmAckFailed>(HandleAlarmAckFailed);
// A child connectivity transition while the host is Stale has no live address space to annotate — drop
// it (the post-recovery rebuild re-materialises conditions, and the child re-announces on its next entry).
Receive<DriverInstanceActor.ConnectivityChanged>(_ => { });
@@ -3654,6 +3725,7 @@ public sealed class DriverHostActor : ReceiveActor, IWithTimers, IWithUnboundedS
// Acked=true — on any deploy, rebuild or not. The node manager carries the NODE's state across a rebuild,
// so the two stay in step. Keyed by the bare condition NodeId, exactly as ForwardNativeAlarm projects.
_nativeAlarmProjector.RetainOnly(_driverRefByAlarmNodeId.Keys.ToHashSet(StringComparer.Ordinal));
PublishAckPreflight();
// Cache the per-driver payload as THE authority, in the same clear-and-repopulate pass as the routing
// maps above so a rename/removal can never leave a stale entry behind. Keyed over the UNION of the two
@@ -3812,7 +3884,11 @@ public sealed class DriverHostActor : ReceiveActor, IWithTimers, IWithUnboundedS
child.Tell(new DriverInstanceActor.InitializeRequested(spec.DriverConfig));
}
_children[spec.DriverInstanceId] = new ChildEntry(child, spec, stub);
_children[spec.DriverInstanceId] = new ChildEntry(child, spec, stub)
{
AckPreflight = stub ? null : driver as IAlarmAcknowledgePreflight,
};
PublishAckPreflight();
// EVERY spawn hands the child its desired set — deploy reconcile, operator Restart (03/S16), and the
// supervision reprovision. Sending it HERE rather than at each call site is the point: a child that
@@ -3870,6 +3946,7 @@ public sealed class DriverHostActor : ReceiveActor, IWithTimers, IWithUnboundedS
RetireChild(entry.Actor);
Context.Stop(entry.Actor);
_children.Remove(driverInstanceId);
PublishAckPreflight();
EvictRetainedValues(driverInstanceId);
// The instance left the plan (StopChild is only reached via plan.ToStop), so its cached
// Health/Resilience snapshots must not replay to new telemetry subscribers. Release the shared
@@ -78,7 +78,20 @@ public sealed class DriverInstanceActor : ReceiveActor, IWithTimers
/// <param name="OperatorUser">The authenticated principal performing the acknowledge.</param>
/// <param name="ApiKeyId">Public id of the API key the session authenticated with; null for anonymous. Not a
/// secret — the Galaxy driver uses it to find the key's Galaxy ack identity (lmxopcua#749).</param>
public sealed record RouteAlarmAck(string ConditionId, string? Comment, string OperatorUser, string? ApiKeyId = null);
/// <param name="ConditionNodeId">The OPC UA condition NodeId the client acknowledged. When set, an
/// acknowledge that does not reach the upstream alarm system (refused, failed, or dropped because the driver
/// is not connected) is reported back to the sender as an <see cref="AlarmAckFailed"/> so the host can revert
/// the locally-applied acknowledge (lmxopcua#760). Null ⇒ no report.</param>
public sealed record RouteAlarmAck(
string ConditionId, string? Comment, string OperatorUser, string? ApiKeyId = null, string? ConditionNodeId = null);
/// <summary>
/// Reply to a <see cref="RouteAlarmAck"/> carrying a <see cref="RouteAlarmAck.ConditionNodeId"/> whose
/// acknowledge never reached the upstream alarm system (lmxopcua#760). Success sends nothing.
/// </summary>
/// <param name="ConditionNodeId">The OPC UA condition NodeId the client acknowledged.</param>
/// <param name="Reason">Why the acknowledge did not go through; never contains a secret.</param>
public sealed record AlarmAckFailed(string ConditionNodeId, string Reason);
public sealed record Subscribe(IReadOnlyList<string> FullReferences, TimeSpan PublishingInterval);
/// <summary>
/// Sets the set of references this driver should keep subscribed for the lifetime of the
@@ -609,10 +622,13 @@ public sealed class DriverInstanceActor : ReceiveActor, IWithTimers
Sender.Tell(new WriteAttributeResult(false, "driver not connected")));
// A write launched while Connected can finish after the transition out of it; its slot still frees.
Receive<WriteCompleted>(OnWriteCompleted);
// An ack arriving while still connecting can't reach the upstream alarm system; drop it (the ack is
// fire-and-forget — no reply to surface — and the OPC UA condition state already committed locally).
Receive<RouteAlarmAck>(_ =>
_log.Debug("DriverInstance {Id}: alarm ack arrived during connect — dropped (driver not connected)", _driverInstanceId));
// An ack arriving while still connecting can't reach the upstream alarm system; drop it and report it
// failed so the host reverts the locally-committed acknowledge (lmxopcua#760).
Receive<RouteAlarmAck>(msg =>
{
_log.Debug("DriverInstance {Id}: alarm ack arrived during connect — dropped (driver not connected)", _driverInstanceId);
ReportAckFailed(Sender, msg, "the driver is not connected");
});
Receive<InitializeSucceeded>(msg =>
{
if (msg.Generation != _initGeneration) return;
@@ -913,10 +929,13 @@ public sealed class DriverInstanceActor : ReceiveActor, IWithTimers
Receive<WriteAttribute>(_ =>
Sender.Tell(new WriteAttributeResult(false, "driver not connected")));
Receive<WriteCompleted>(OnWriteCompleted);
// An ack arriving while reconnecting can't reach the upstream alarm system; drop it (fire-and-forget,
// no reply — the OPC UA condition state already committed locally on the Part 9 ack).
Receive<RouteAlarmAck>(_ =>
_log.Debug("DriverInstance {Id}: alarm ack arrived during reconnect — dropped (driver not connected)", _driverInstanceId));
// An ack arriving while reconnecting can't reach the upstream alarm system; drop it and report it
// failed so the host reverts the locally-committed acknowledge (lmxopcua#760).
Receive<RouteAlarmAck>(msg =>
{
_log.Debug("DriverInstance {Id}: alarm ack arrived during reconnect — dropped (driver not connected)", _driverInstanceId);
ReportAckFailed(Sender, msg, "the driver is not connected");
});
Receive<InitializeSucceeded>(msg =>
{
if (msg.Generation != _initGeneration) return;
@@ -1250,15 +1269,19 @@ public sealed class DriverInstanceActor : ReceiveActor, IWithTimers
/// correlates on <see cref="AlarmAcknowledgeRequest.ConditionId"/> (= the authored alarm
/// full-reference); <see cref="AlarmAcknowledgeRequest.SourceNodeId"/> carries the same reference (the
/// driver's ack path keys on ConditionId). Bounded to 5s so a hung backend can't pin this actor.
/// Fire-and-forget — the OPC UA Part 9 ack already committed the local condition state and
/// <see cref="IAlarmSource.AcknowledgeAsync"/> returns no per-condition status — so there is no reply;
/// a failure is logged and dropped (the local condition stays Acknowledged regardless).
/// Success sends no reply (the OPC UA Part 9 ack already committed the local condition state). A refusal
/// or failure is logged and, when the message carries a condition NodeId, reported to the sender as an
/// <see cref="AlarmAckFailed"/> so the host reverts the condition to unacknowledged (lmxopcua#760) —
/// otherwise the node showed Acknowledged for an alarm the upstream system never acknowledged.
/// </summary>
private async Task HandleAcknowledgeAsync(RouteAlarmAck msg)
{
// Captured before any await: the failure report goes back to whoever routed the ack.
var replyTo = Sender;
if (_driver is not IAlarmSource src)
{
_log.Warning("DriverInstance {Id}: alarm ack dropped — driver does not implement IAlarmSource", _driverInstanceId);
ReportAckFailed(replyTo, msg, "the driver has no alarm source");
return;
}
@@ -1289,14 +1312,25 @@ public sealed class DriverInstanceActor : ReceiveActor, IWithTimers
// configuration verdict, not a backend fault, so no stack trace; the message names the fix.
_log.Warning("DriverInstance {Id}: native-alarm acknowledge of {Cond} by {User} refused — {Reason}",
_driverInstanceId, msg.ConditionId, msg.OperatorUser, ex.Message);
ReportAckFailed(replyTo, msg, ex.Message);
}
catch (Exception ex)
{
_log.Warning(ex, "DriverInstance {Id}: native-alarm acknowledge of {Cond} failed",
_driverInstanceId, msg.ConditionId);
// The exception TYPE only: a backend message is not ours to vouch for (it could echo a credential).
ReportAckFailed(replyTo, msg, $"the upstream acknowledge failed ({ex.GetType().Name})");
}
}
/// <summary>Tells <paramref name="replyTo"/> an <see cref="AlarmAckFailed"/> when the ack names a condition
/// NodeId (lmxopcua#760); a no-op for an ack routed without one.</summary>
private static void ReportAckFailed(IActorRef replyTo, RouteAlarmAck msg, string reason)
{
if (string.IsNullOrEmpty(msg.ConditionNodeId) || replyTo.IsNobody()) return;
replyTo.Tell(new AlarmAckFailed(msg.ConditionNodeId, reason));
}
private async Task HandleSubscribeAsync(Subscribe msg)
{
// Capture Sender/Self BEFORE any await. The re-subscribe path below awaits
@@ -0,0 +1,50 @@
using System.Collections.Immutable;
using ZB.MOM.WW.OtOpcUa.Core.Abstractions;
namespace ZB.MOM.WW.OtOpcUa.Runtime.Drivers;
/// <summary>
/// The lock-free snapshot the OPC UA node manager consults to refuse, synchronously, a native-condition
/// Acknowledge the owning driver would refuse anyway (lmxopcua#760). <see cref="DriverHostActor"/> owns
/// the condition → driver map and the driver objects, and publishes a fresh immutable snapshot whenever
/// either changes; <see cref="Check"/> is a single volatile read, safe under the SDK node-manager lock.
/// </summary>
/// <remarks>
/// Racy by design against a change published a moment later — it covers static misconfiguration only. An
/// ack it lets through can still be refused or fail at the driver; that path reverts the condition to
/// unacknowledged (<see cref="DriverHostActor"/>'s <c>NativeAlarmAckFailed</c> handling).
/// </remarks>
public sealed class NativeAlarmAckPreflightView
{
private volatile ImmutableDictionary<string, IAlarmAcknowledgePreflight> _byConditionNodeId =
ImmutableDictionary<string, IAlarmAcknowledgePreflight>.Empty.WithComparers(StringComparer.Ordinal);
/// <summary>
/// Returns why the driver owning <paramref name="conditionNodeId"/> would refuse an acknowledge from a
/// session authenticated with <paramref name="apiKeyId"/>, or null to proceed (unknown condition, a
/// driver without a pre-check, or one that cannot tell without I/O).
/// </summary>
/// <param name="conditionNodeId">The bare condition NodeId identifier the node manager routes by.</param>
/// <param name="apiKeyId">The session's API key id, or null for anonymous.</param>
/// <returns>The refusal reason, or null.</returns>
public string? Check(string conditionNodeId, string? apiKeyId)
{
if (string.IsNullOrEmpty(conditionNodeId)) return null;
if (!_byConditionNodeId.TryGetValue(conditionNodeId, out var preflight)) return null;
try
{
return preflight.GetAcknowledgeRefusal(apiKeyId);
}
catch
{
// A buggy pre-check must never break the Part 9 method path; let the ack proceed (the driver's own
// refusal + the host's revert still apply).
return null;
}
}
/// <summary>Replaces the snapshot. Called by <see cref="DriverHostActor"/> only.</summary>
/// <param name="byConditionNodeId">Condition NodeId → the owning driver's pre-check.</param>
internal void Publish(IEnumerable<KeyValuePair<string, IAlarmAcknowledgePreflight>> byConditionNodeId) =>
_byConditionNodeId = ImmutableDictionary.CreateRange(StringComparer.Ordinal, byConditionNodeId);
}
@@ -12,9 +12,16 @@ namespace ZB.MOM.WW.OtOpcUa.Runtime.Drivers;
/// </summary>
public sealed class NativeAlarmProjector
{
private readonly Dictionary<string, (bool Active, bool Acked, string? Message, ushort Severity)> _prior =
private readonly Dictionary<string, (bool Active, bool Acked, string? Message, ushort Severity, uint StatusCode)> _prior =
new(StringComparer.Ordinal);
/// <summary>
/// Conditions a client acknowledged locally (<see cref="MarkAcknowledged"/>) whose acknowledge has not yet
/// been confirmed or superseded by a driver transition — the only ones <see cref="TryRevertAcknowledge"/>
/// may un-acknowledge (lmxopcua#760).
/// </summary>
private readonly HashSet<string> _pendingLocalAck = new(StringComparer.Ordinal);
/// <summary>Project an alarm transition onto the full condition snapshot for <paramref name="nodeId"/>.</summary>
/// <param name="nodeId">The materialised condition node's id (the projection's state key).</param>
/// <param name="e">The native alarm transition.</param>
@@ -71,8 +78,11 @@ public sealed class NativeAlarmProjector
/// <returns>The full Part 9 condition snapshot to write to the node.</returns>
public AlarmConditionSnapshot Project(string nodeId, AlarmEventArgs e, out IReadOnlyList<AlarmTransitionKind> historyKinds)
{
// Any driver transition supersedes a pending local acknowledge: from here the projected state is the
// driver's word, so a late ack failure must not rewrite it.
_pendingLocalAck.Remove(nodeId);
var prev = _prior.TryGetValue(nodeId, out var p)
? p : (Active: false, Acked: true, Message: (string?)null, Severity: (ushort)0);
? p : (Active: false, Acked: true, Message: (string?)null, Severity: (ushort)0, StatusCode: OpcUaStatus.Good);
// A source with a finer scale than the four buckets supplies the exact Part 9 severity.
var sev = e.OpcUaSeverity is { } exact ? (ushort)Math.Clamp(exact, 1, 1000) : MapSeverity(e.Severity);
// Only a statement saying the SAME thing the condition already shows is a re-statement.
@@ -143,14 +153,15 @@ public sealed class NativeAlarmProjector
break;
}
_prior[nodeId] = (active, acked, e.Message, sev);
var statusCode = e.QualityStatusCode ?? OpcUaStatus.Good;
_prior[nodeId] = (active, acked, e.Message, sev, statusCode);
return new AlarmConditionSnapshot(
Active: active, Acknowledged: acked, Confirmed: true, Enabled: true,
Shelving: AlarmShelvingKind.Unshelved, Severity: sev, Message: e.Message,
// A per-condition source quality (MTConnect Unavailable) when the driver reports one. Absent
// (every earlier source) ⇒ Good, as before — driver-wide comms loss stays the host's
// out-of-band WriteAlarmQuality annotation, not something the projector infers.
StatusCode: e.QualityStatusCode ?? OpcUaStatus.Good);
StatusCode: statusCode);
}
/// <summary>
@@ -161,11 +172,39 @@ public sealed class NativeAlarmProjector
/// <param name="nodeId">The acknowledged condition node id.</param>
public void MarkAcknowledged(string nodeId)
{
if (_prior.TryGetValue(nodeId, out var p)) _prior[nodeId] = p with { Acked = true };
if (!_prior.TryGetValue(nodeId, out var p)) return;
// Only an ack that CHANGED the projected state is pending; re-acking an acked condition has nothing to undo.
if (!p.Acked) _pendingLocalAck.Add(nodeId);
_prior[nodeId] = p with { Acked = true };
}
/// <summary>
/// Undo a client acknowledgement the driver refused or failed to deliver upstream (lmxopcua#760), returning
/// the unacknowledged snapshot to re-project onto the node. Only an acknowledge still pending since
/// <see cref="MarkAcknowledged"/> is reverted: once any driver transition for the condition has been
/// projected, that transition is authoritative and a late failure changes nothing.
/// </summary>
/// <param name="nodeId">The condition node id whose acknowledge failed.</param>
/// <param name="snapshot">The unacknowledged snapshot to project; null when nothing was reverted.</param>
/// <returns>True when the condition was reverted to unacknowledged.</returns>
public bool TryRevertAcknowledge(string nodeId, out AlarmConditionSnapshot? snapshot)
{
snapshot = null;
if (!_pendingLocalAck.Remove(nodeId) || !_prior.TryGetValue(nodeId, out var p)) return false;
_prior[nodeId] = p with { Acked = false };
snapshot = new AlarmConditionSnapshot(
Active: p.Active, Acknowledged: false, Confirmed: true, Enabled: true,
Shelving: AlarmShelvingKind.Unshelved, Severity: p.Severity, Message: p.Message ?? string.Empty,
StatusCode: p.StatusCode);
return true;
}
/// <summary>Clears all tracked per-node state.</summary>
public void Clear() => _prior.Clear();
public void Clear()
{
_prior.Clear();
_pendingLocalAck.Clear();
}
/// <summary>
/// Drop the tracked state of every condition NOT in <paramref name="deployedNodeIds"/>, keeping
@@ -180,7 +219,10 @@ public sealed class NativeAlarmProjector
{
ArgumentNullException.ThrowIfNull(deployedNodeIds);
foreach (var nodeId in _prior.Keys.Where(k => !deployedNodeIds.Contains(k)).ToList())
{
_prior.Remove(nodeId);
_pendingLocalAck.Remove(nodeId);
}
}
private static ushort MapSeverity(AlarmSeverity s) => s switch
@@ -104,6 +104,10 @@ public static class ServiceCollectionExtensions
// redundancy manager, wires no HistoryUpdate gate at all.
services.TryAddSingleton<IHistoryUpdateRoleView>(sp => sp.GetRequiredService<RedundancyRoleView>());
// The condition → owning-driver native-ack pre-check snapshot (lmxopcua#760): DriverHostActor publishes
// it, the OPC UA node manager reads it to refuse an acknowledge the driver would refuse anyway.
services.TryAddSingleton<NativeAlarmAckPreflightView>();
// The node→central ack transport under MeshTransport:Mode = Grpc (mesh gRPC command plane).
// Registered unconditionally but resolved only by the Grpc arm of
// WithOtOpcUaRuntimeActors — the mode is not knowable here (this runs before configuration is
@@ -654,7 +658,9 @@ public static class ServiceCollectionExtensions
historyUpdateRoleView: resolver.GetService<IHistoryUpdateRoleView>(),
// The node-wide Scripting section (bound + clamped in Program.cs) for the scripted-alarm
// engine. Unbound in test graphs ⇒ null ⇒ the compiled defaults.
scripting: resolver.GetService<ScriptingOptions>()),
scripting: resolver.GetService<ScriptingOptions>(),
// Where the native-ack pre-check snapshot is published (lmxopcua#760).
ackPreflightView: resolver.GetService<NativeAlarmAckPreflightView>()),
DriverHostActorName);
registry.Register<DriverHostActorKey>(driverHost);