dohertj2/lmxopcua
1
0
Fork 0
Code Issues 20 Pull Requests Actions 2 Packages Projects Releases Wiki Activity

fix(alarms): surface a refused native alarm ack synchronously, revert one that fails later (lmxopcua#760)

Browse Source 
The native-ack path is fire-and-forget after the Part 9 Acknowledge returns, so
a Galaxy ack refused for lack of a credential (#749), or rejected by MXAccess,
left the client with Good and the condition showing Acknowledged while Galaxy
was never acked. Both options from the issue, layered:

1. Synchronous refusal for what the driver can decide without I/O. New optional
   IAlarmAcknowledgePreflight (Core.Abstractions); GalaxyDriver refuses an
   anonymous session when the driver has no AlarmAck fallback account.
   DriverHostActor publishes a condition NodeId -> owning-driver snapshot into
   NativeAlarmAckPreflightView (lock-free, volatile); the node manager's new
   NativeAlarmAckPreflight seam, consulted after the authz and Primary gates,
   returns BadUserAccessDenied with the reason - nothing routed, applied or
   reported. A keyed session is not pre-checked (its identity lives in the
   Secrets store, which must not be read under the SDK lock).

2. Revert for everything else. RouteAlarmAck carries the condition NodeId;
   DriverInstanceActor replies AlarmAckFailed on a refusal, a backend failure,
   a driver without an alarm source, or an ack dropped while connecting /
   reconnecting. DriverHostActor re-projects the condition unacknowledged via
   NativeAlarmProjector.TryRevertAcknowledge, but only while the local ack is
   still pending - any driver transition projected since then wins. A disabled
   or unspawned driver is not reverted (unchanged).

GatewayGalaxyAlarmAcknowledger now throws DriverOperationRefusedException on a
non-zero native hresult (wrong password, no ack permission) instead of only
warning, so that case reverts too; as a refusal it neither retries nor trips
the breaker.

Tests (each fails with its fix mutated out): projector revert/supersede,
DriverHostActor refused + failed ack -> unacked AlarmStateUpdate (successful
ack control), preflight view snapshot, node-manager preflight unit test,
over-the-wire ConditionMethodEventTests (BadUserAccessDenied, no route, no
event; positive control), Galaxy preflight + hresult refusal.
Docs: Galaxy.md, AlarmTracking.md, ScriptedAlarms.md, .claude/rules/alarms.md.

Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
This commit is contained in:
Joseph Doherty
2026-11-09 13:12:17 -05:00
parent 0217d36cf4
commit 8e8d4cdf13
21 changed files with 737 additions and 61 deletions
Download Patch File Download Diff File Expand all files Collapse all files
src/Server/ZB.MOM.WW.OtOpcUa.Runtime/Drivers/DriverHostActor.cs
+88 -11
View File
@@ -476,6 +476,9 @@ public sealed class DriverHostActor : ReceiveActor, IWithTimers, IWithUnboundedS
/// <summary>Where the HistoryUpdate verdict is published. Null on nodes that wire no redundancy manager.</summary>
private readonly IHistoryUpdateRoleView? _historyUpdateRoleView;
/// <summary>Where the synchronous native-ack pre-check snapshot is published (lmxopcua#760); null ⇒ none.</summary>
private readonly NativeAlarmAckPreflightView? _ackPreflightView;
/// <summary>Whether any redundancy snapshot has reached this host — the HistoryUpdate gate stays
/// closed until one has (see <see cref="PrimaryGatePolicy.ShouldServeHistoryUpdate"/>).</summary>
private bool _redundancySnapshotReceived;
@@ -617,6 +620,9 @@ public sealed class DriverHostActor : ReceiveActor, IWithTimers, IWithUnboundedS
private sealed record ChildEntry(IActorRef Actor, DriverInstanceSpec Spec, bool Stubbed)
{
/// <summary>The driver's synchronous native-ack pre-check, when it offers one (lmxopcua#760).</summary>
public IAlarmAcknowledgePreflight? AckPreflight { get; init; }
/// <summary>When this child was spawned — the diagnostics' last-change time until it publishes health.</summary>
public DateTime SpawnedUtc { get; } = DateTime.UtcNow;
@@ -749,7 +755,8 @@ public sealed class DriverHostActor : ReceiveActor, IWithTimers, IWithUnboundedS
TimeSpan? applyOutcomeTimeout = null,
bool captureWhilePrimaryUnreachable = true,
IHistoryUpdateRoleView? historyUpdateRoleView = null,
ScriptingOptions? scripting = null) =>
ScriptingOptions? scripting = null,
NativeAlarmAckPreflightView? ackPreflightView = null) =>
// WARNING: this forwarding list is POSITIONAL, and Props.Create compiles it into an
// expression tree. Six IActorRef? parameters and several interface-typed ones mean a
// mis-ordered argument is usually type-compatible and therefore compiles clean, then binds
@@ -763,7 +770,7 @@ public sealed class DriverHostActor : ReceiveActor, IWithTimers, IWithUnboundedS
alarmStateStore, telemetryHub, primaryRoleSource, historizationRoleView,
sharesHistorizationScopeWithLocal, applyWedgeTimeout, reprovisionBackoff, commsRecoveryAuditDelay,
sharesAlarmQueueWithLocal, applyOutcomeTimeout, captureWhilePrimaryUnreachable, historyUpdateRoleView,
scripting));
scripting, ackPreflightView));
/// <summary>Initializes a new DriverHostActor with the specified dependencies.</summary>
/// <param name="dbFactory">Database context factory for configuration database access.</param>
@@ -852,6 +859,8 @@ public sealed class DriverHostActor : ReceiveActor, IWithTimers, IWithUnboundedS
/// redundancy manager.</param>
/// <param name="scripting">The node-wide <c>Scripting</c> run-guard options handed to the scripted-alarm
/// engine. Null ⇒ the compiled defaults.</param>
/// <param name="ackPreflightView">Where the condition → owning-driver native-ack pre-check snapshot is published
/// for the node manager's synchronous refusal (lmxopcua#760). Null ⇒ nothing published (no pre-check).</param>
public DriverHostActor(
IDbContextFactory<OtOpcUaConfigDbContext>? dbFactory,
CommonsNodeId localNode,
@@ -885,8 +894,10 @@ public sealed class DriverHostActor : ReceiveActor, IWithTimers, IWithUnboundedS
TimeSpan? applyOutcomeTimeout = null,
bool captureWhilePrimaryUnreachable = true,
IHistoryUpdateRoleView? historyUpdateRoleView = null,
ScriptingOptions? scripting = null)
ScriptingOptions? scripting = null,
NativeAlarmAckPreflightView? ackPreflightView = null)
{
_ackPreflightView = ackPreflightView;
_scripting = scripting;
_captureWhilePrimaryUnreachable = captureWhilePrimaryUnreachable;
_historyUpdateRoleView = historyUpdateRoleView;
@@ -1545,6 +1556,7 @@ public sealed class DriverHostActor : ReceiveActor, IWithTimers, IWithUnboundedS
Receive<ReconnectDriver>(HandleReconnectDriver);
Receive<RouteNodeWrite>(HandleRouteNodeWrite);
Receive<RouteNativeAlarmAck>(HandleRouteNativeAlarmAck);
Receive<DriverInstanceActor.AlarmAckFailed>(HandleAlarmAckFailed);
Receive<RedundancyStateChanged>(OnRedundancyStateChanged);
Receive<Akka.Cluster.ClusterEvent.IMemberEvent>(OnDriverMembershipChanged);
Receive<Akka.Cluster.ClusterEvent.ReachabilityEvent>(OnReachabilityChanged);
@@ -1609,6 +1621,7 @@ public sealed class DriverHostActor : ReceiveActor, IWithTimers, IWithUnboundedS
Receive<ReconnectDriver>(HandleReconnectDriver);
Receive<RouteNodeWrite>(HandleRouteNodeWrite);
Receive<RouteNativeAlarmAck>(HandleRouteNativeAlarmAck);
Receive<DriverInstanceActor.AlarmAckFailed>(HandleAlarmAckFailed);
Receive<RedundancyStateChanged>(OnRedundancyStateChanged);
Receive<Akka.Cluster.ClusterEvent.IMemberEvent>(OnDriverMembershipChanged);
Receive<Akka.Cluster.ClusterEvent.ReachabilityEvent>(OnReachabilityChanged);
@@ -2260,9 +2273,9 @@ public sealed class DriverHostActor : ReceiveActor, IWithTimers, IWithUnboundedS
/// <summary>
/// Routes an inbound native-condition acknowledge (the host Tells this from the OPC UA node-manager
/// side when a client Acknowledges a NATIVE Part 9 condition) to the owning driver child. Mirrors
/// <see cref="HandleRouteNodeWrite"/>'s order + gating, but the ack is fire-and-forget (no reply):
/// the OPC UA Part 9 acknowledge already committed the local condition state and the driver's
/// <see cref="IAlarmSource.AcknowledgeAsync"/> returns no per-condition status.
/// <see cref="HandleRouteNodeWrite"/>'s order + gating, but the ack is fire-and-forget (no reply to the
/// OPC UA client): the Part 9 acknowledge already committed the local condition state. A refused or failed
/// ack comes back as <see cref="DriverInstanceActor.AlarmAckFailed"/> and reverts it (lmxopcua#760).
/// <list type="number">
/// <item>PRIMARY gate FIRST — reuses the same <see cref="_localRole"/> redundancy signal as the
/// inbound-write gate. Only the Primary pushes the ack to the shared upstream alarm system
@@ -2316,13 +2329,68 @@ public sealed class DriverHostActor : ReceiveActor, IWithTimers, IWithUnboundedS
{
_log.Debug("DriverHost {Node}: driver {Driver} not running for alarm ack of {Cond} — dropped",
_localNode, target.DriverInstanceId, msg.ConditionNodeId);
// Deliberately NOT reverted (lmxopcua#760 covers what a driver reports): a disabled or not-yet-spawned
// driver produces no transitions either, so the local ack stands, as before.
return;
}
// Fire-and-forget: the OPC UA Part 9 ack already committed the local condition state, and the
// driver's AcknowledgeAsync surfaces no per-condition status, so there is nothing to reply. The
// driver correlates on ConditionId (= the authored alarm FullName the inverse map keyed on).
entry.Actor.Tell(new DriverInstanceActor.RouteAlarmAck(target.RawPath, msg.Comment, msg.OperatorUser, msg.ApiKeyId));
// The OPC UA Part 9 ack already committed the local condition state; success sends nothing back. A
// refusal or failure comes back as DriverInstanceActor.AlarmAckFailed (keyed by the condition NodeId
// carried here) and reverts the node (HandleAlarmAckFailed). The driver correlates on ConditionId
// (= the authored alarm FullName the inverse map keyed on).
entry.Actor.Tell(new DriverInstanceActor.RouteAlarmAck(
target.RawPath, msg.Comment, msg.OperatorUser, msg.ApiKeyId, ConditionNodeId: msg.ConditionNodeId));
}
/// <summary>
/// A native acknowledge the driver refused or failed to deliver upstream (lmxopcua#760): revert the
/// condition to unacknowledged so the address space stops claiming an ack the alarm system never saw.
/// Mirrors the optimistic-write self-correction — the client got <c>Good</c> on the method call, and the
/// revert reaches it as a fresh condition event (Acked=false, Retain=true).
/// </summary>
/// <param name="msg">The failure report from the driver child.</param>
private void HandleAlarmAckFailed(DriverInstanceActor.AlarmAckFailed msg) =>
RevertNativeAlarmAck(msg.ConditionNodeId, msg.Reason);
/// <summary>
/// Re-projects <paramref name="conditionNodeId"/> as unacknowledged when its local acknowledge is still
/// pending in the projector. A driver transition projected since the ack is authoritative, so a late
/// failure after it changes nothing.
/// </summary>
/// <param name="conditionNodeId">The bare condition NodeId (Raw realm).</param>
/// <param name="reason">Why the acknowledge did not reach the upstream alarm system.</param>
private void RevertNativeAlarmAck(string conditionNodeId, string reason)
{
if (!_nativeAlarmProjector.TryRevertAcknowledge(conditionNodeId, out var snapshot) || snapshot is null)
{
_log.Info("DriverHost {Node}: native-alarm acknowledge of {Cond} did not reach the alarm system ({Reason}); " +
"the condition has since been re-stated by its driver, so it is left as projected",
_localNode, conditionNodeId, reason);
return;
}
_log.Warning("DriverHost {Node}: native-alarm acknowledge of {Cond} did not reach the alarm system ({Reason}); " +
"reverting the condition to unacknowledged", _localNode, conditionNodeId, reason);
// Native conditions live in the Raw realm at their bare NodeId (the apply pass's single raw condition).
_opcUaPublishActor?.Tell(new ZB.MOM.WW.OtOpcUa.Runtime.OpcUa.OpcUaPublishActor.AlarmStateUpdate(
conditionNodeId, snapshot, DateTime.UtcNow, AddressSpaceRealm.Raw));
}
/// <summary>
/// Publishes the condition NodeId → owning-driver native-ack pre-check snapshot (lmxopcua#760). Called
/// whenever the condition map or the set of running children changes; a no-op without a view.
/// </summary>
private void PublishAckPreflight()
{
if (_ackPreflightView is null) return;
var entries = new List<KeyValuePair<string, IAlarmAcknowledgePreflight>>();
foreach (var (conditionNodeId, target) in _driverRefByAlarmNodeId)
{
if (_children.TryGetValue(target.DriverInstanceId, out var child) && child.AckPreflight is { } preflight)
entries.Add(new(conditionNodeId, preflight));
}
_ackPreflightView.Publish(entries);
}
/// <summary>Whether this node should service a Primary-only data-plane operation right now — the single
@@ -2539,6 +2607,9 @@ public sealed class DriverHostActor : ReceiveActor, IWithTimers, IWithUnboundedS
Receive<RouteNativeAlarmAck>(msg =>
_log.Debug("DriverHost {Node}: dropping native-alarm ack for {Node2} while Stale (config DB unreachable)",
_localNode, msg.ConditionNodeId));
// An ack routed before the host went Stale can still come back refused; the revert needs only the
// projector + publish actor, both of which survive Stale.
Receive<DriverInstanceActor.AlarmAckFailed>(HandleAlarmAckFailed);
// A child connectivity transition while the host is Stale has no live address space to annotate — drop
// it (the post-recovery rebuild re-materialises conditions, and the child re-announces on its next entry).
Receive<DriverInstanceActor.ConnectivityChanged>(_ => { });
@@ -3654,6 +3725,7 @@ public sealed class DriverHostActor : ReceiveActor, IWithTimers, IWithUnboundedS
// Acked=true — on any deploy, rebuild or not. The node manager carries the NODE's state across a rebuild,
// so the two stay in step. Keyed by the bare condition NodeId, exactly as ForwardNativeAlarm projects.
_nativeAlarmProjector.RetainOnly(_driverRefByAlarmNodeId.Keys.ToHashSet(StringComparer.Ordinal));
PublishAckPreflight();
// Cache the per-driver payload as THE authority, in the same clear-and-repopulate pass as the routing
// maps above so a rename/removal can never leave a stale entry behind. Keyed over the UNION of the two
@@ -3812,7 +3884,11 @@ public sealed class DriverHostActor : ReceiveActor, IWithTimers, IWithUnboundedS
child.Tell(new DriverInstanceActor.InitializeRequested(spec.DriverConfig));
}
_children[spec.DriverInstanceId] = new ChildEntry(child, spec, stub);
_children[spec.DriverInstanceId] = new ChildEntry(child, spec, stub)
{
AckPreflight = stub ? null : driver as IAlarmAcknowledgePreflight,
};
PublishAckPreflight();
// EVERY spawn hands the child its desired set — deploy reconcile, operator Restart (03/S16), and the
// supervision reprovision. Sending it HERE rather than at each call site is the point: a child that
@@ -3870,6 +3946,7 @@ public sealed class DriverHostActor : ReceiveActor, IWithTimers, IWithUnboundedS
RetireChild(entry.Actor);
Context.Stop(entry.Actor);
_children.Remove(driverInstanceId);
PublishAckPreflight();
EvictRetainedValues(driverInstanceId);
// The instance left the plan (StopChild is only reached via plan.ToStop), so its cached
// Health/Resilience snapshots must not replay to new telemetry subscribers. Release the shared