fix(alarms): native ack refused off-Primary, commented re-ack reported, real UnshelveTime (#597)

- Native-condition Acknowledge on a non-Primary node returns BadInvalidState (nothing
  routed/applied/reported) via NativeAlarmAckPrimaryGate, fed from
  IDriverPrimaryRoleSource by OtOpcUaServerHostedService.
- A re-ack carrying a comment reports one comment event (Acked/Confirmed unchanged).
- TimedShelve applies its requested duration and the scripted projection carries
  UnshelveAtUtc, so ShelvingState/UnshelveTime counts down instead of ~24.8 days.
- Native re-ack still routed to the driver (deliberate: documented retry path), pinned.
- AdminUI /alerts client ack/shelve path test; docs/AlarmTracking.md decisions incl.
  accepted SDK EventsMonitored residual (item 6).
- Also #599/03/S20: RemoveEquipmentSubtree severs inverse Organizes edges.

Claude-Session: https://claude.ai/code/session_01Ef2ozxS7QPanGT8uoATbTe
This commit is contained in:
Joseph Doherty
2026-10-08 07:12:29 -04:00
parent 0bd3ec00e4
commit bd63810645
10 changed files with 459 additions and 51 deletions
+66 -20
View File
@@ -292,8 +292,9 @@ branches on native-ness and routes through a dedicated path — separate
from the scripted `AlarmCommandRouter`:
1. **`OtOpcUaNodeManager.HandleNativeAlarmAck`** — gates on the caller's
`AlarmAck` role (fails closed: no role → `BadUserAccessDenied`), then
dispatches a `NativeAlarmAck(ConditionNodeId, Comment, OperatorUser)`
`AlarmAck` role (fails closed: no role → `BadUserAccessDenied`), then on the
node's redundancy role (not the Primary → `BadInvalidState`, see
§"Acknowledge on a non-Primary node" below), then dispatches a `NativeAlarmAck(ConditionNodeId, Comment, OperatorUser)`
to the `NativeAlarmAckRouter` seam (fire-and-forget, non-blocking under
the node-manager Lock). `OperatorUser` carries the authenticated
session principal's display name.
@@ -331,20 +332,38 @@ projection's Retain rule, `Active || !Acked`) after the SDK handler, before the
one ack event. ScadaBridge maps `ConfirmedState` into its alarm state, so this
is client-visible.
> **Secondary-node ack flap.** On a node that is not the redundancy Primary,
> an OPC UA Acknowledge is applied to the local condition and reported (one
> event, Acked), but `DriverHostActor`'s Primary gate drops the route upstream.
> The driver never acknowledges, so its next transition for that alarm is
> projected with `Acked = false` and the node reverts — with its own event. A
> client acking against the Secondary sees acked → unacked. Ack against the
> Primary (the higher ServiceLevel) for the ack to stick.
> **Acknowledge on a non-Primary node is refused** (Gitea #597). On a node
> that is not the redundancy Primary, an OPC UA Acknowledge of a **native**
> condition returns **`BadInvalidState`** ("not the redundancy Primary;
> acknowledge on the Primary") — nothing is routed, applied or reported. The
> verdict is the same data-plane one the device-write gate and
> `DriverHostActor`'s ack route use (`PrimaryGatePolicy.ShouldServiceAsPrimary`,
> published to `IDriverPrimaryRoleSource` on every redundancy snapshot and
> wired onto the node manager by `OtOpcUaServerHostedService`). It is checked
> after authorization, so a caller who may not ack learns nothing about the
> node's role.
>
> This replaces a documented flap: the Secondary used to apply and report the
> ack locally while `DriverHostActor` dropped the route upstream, so the driver
> never acknowledged, its next transition projected `Acked = false`, and the
> client saw acked → unacked. An honest refusal is client-visible — a client
> (ScadaBridge included) that acks against the Secondary now gets a Bad result
> instead of a Good that later reverts — so ack against the Primary (the higher
> ServiceLevel). Scripted conditions are **not** gated: their engine runs on
> both nodes, so an ack on the Secondary is real. Residual: the published
> verdict changes only on a redundancy snapshot, so in the boot window of a
> multi-node cluster (no snapshot yet) the node still accepts, while
> `DriverHostActor`'s live check may drop the route — the old flap, limited to
> that window.
> **Best-effort during reconnect.** If a native ack arrives while the owning
> driver is `Connecting`/`Reconnecting` (or on a non-Primary node), it is
> dropped with a debug log and **not** retried upstream — the OPC UA Part 9
> `AlarmConditionState` was already committed locally at step 1, so the rare
> outcome is "acked locally but not propagated to AVEVA." Re-issue the ack once
> the driver is reconnected if upstream propagation is required.
> driver is `Connecting`/`Reconnecting` (or, in the boot window above, on a
> node whose role is not yet known), it is dropped with a debug log and **not**
> retried upstream — the OPC UA Part 9 `AlarmConditionState` was already
> committed locally at step 1, so the rare outcome is "acked locally but not
> propagated to AVEVA." Re-issue the ack once the driver is reconnected if
> upstream propagation is required — a re-ack of a node already showing Acked
> is still routed to the driver for exactly this reason.
### Legacy sub-attribute path
@@ -412,8 +431,9 @@ reports nothing.
`AreEventsMonitored` bookkeeping, which is imprecise for conditions
materialised under an already-subscribed folder.
- **The SDK's shelve methods never apply shelving state** (they only call
`OnShelve`), so the wrapper applies it — with the same
`(shelved, oneShot, 0)` mapping the projection uses — before reporting.
`OnShelve`), so the wrapper applies it — with the same `(shelved, oneShot)`
mapping the projection uses, and a TimedShelve with its requested duration —
before reporting.
- **The re-projection does not duplicate it.** The engine's (or driver's)
answer to the routed command reaches `WriteAlarmCondition`'s delta-gate,
which compares against the node's current state — already the new state —
@@ -425,7 +445,16 @@ reports nothing.
nothing. The SDK accepts acknowledging an acknowledged condition and sets
`Confirmed = false` again, so a re-ack also restores the pre-call Confirmed
and Retain — silent, and a scripted node never drifts from its engine.
AddComment always reports (the comment is the change).
AddComment always reports (the comment is the change), and so does a re-ack
**carrying a comment** (Gitea #597): the SDK writes that comment to the node,
so it is reported once, with Acked/Confirmed unchanged. Before #597 the
comment landed on the node but no subscriber ever saw it.
- **A native re-ack is still routed to the driver** (deliberate, #597). The
route is fire-and-forget, so a node showing Acked does not prove the
upstream system acknowledged — an ack dropped during a reconnect leaves
exactly that state — and re-acking is the documented retry (§"Best-effort
during reconnect"). The cost is a duplicate ack upstream for an alarm that
was already acknowledged there.
- **Native conditions: Acknowledge and AddComment only** — see §"Native alarm
acknowledge → AVEVA" above. The other methods are refused
(`BadNotSupported`, no event), and a native ack keeps the condition
@@ -433,9 +462,26 @@ reports nothing.
- **TimedShelve works (scripted).** `Create()` left `MaxTimeShelved` at 0, and
the SDK rejects any shelving time above it, so every client TimedShelve
failed with `BadShelvingTimeOutOfRange`. It is now `int.MaxValue` ms (≈24.8
days). The node applies the shelve with no SDK timer of its own; the engine
owns the expiry and its Unshelved re-projection reports the one expiry
event.
days). The engine owns the expiry and its Unshelved re-projection reports
the one expiry event.
- **`UnshelveTime` is the real remaining time** (Gitea #597). Both the method
path and the projection used to apply a timed shelve with `shelvingTime 0`,
which the SDK reads as `MaxTimeShelved` — so every timed shelve advertised
≈24.8 days remaining. The method now applies the requested duration, and the
engine's projection carries its expiry (`AlarmConditionSnapshot.UnshelveAtUtc`),
so `ShelvingState/UnshelveTime` counts down to the engine's expiry whichever
path shelved it (the AdminUI `/alerts` timed shelve included). This arms the
SDK's own unshelve timer at the real instant; it changes no node state and
only routes an `Unshelve` (user `system`) back to the engine — a no-op if the
engine's 5 s expiry sweep got there first, otherwise it simply lands the
expiry on time.
- **Accepted, not fixed — the SDK's `EventsMonitored` unit for late conditions**
(Gitea #597 item 6, from the #586 review). The SDK's `AreEventsMonitored`
counter never reaches a condition materialised under a folder that already had
an event subscriber. Our own reporting does not depend on it (above); the one
thing it still gates is the SDK's method **audit** event, and only while the
server is auditing — which no configuration in this repo enables. Closed as
accepted.
- Method **audit** events (`AuditConditionAcknowledgeEventType`, …) are still
the SDK's, delivered only while the server is auditing over an encrypted
channel.
+1 -1
View File
@@ -358,7 +358,7 @@ On allow, the handler publishes a `Commons.OpcUa.AlarmCommand` (containing comma
### Delta-gate de-duplication
`WriteAlarmCondition` fires a Part 9 condition event only when the incoming state differs from the node's current live state. For an OPC UA method call (Acknowledge, Confirm, AddComment, the shelve methods, Enable/Disable) the SDK applies the new state to the node, and the node manager then reports **the one** condition event for it from the method path (Gitea #587 — the SDK reports none of its own, since `AutoReportStateChanges` stays false; the shelve methods' state is applied by the node manager, because the SDK's never applies it; a call that changed nothing — e.g. a re-ack — reports nothing). Native (driver-fed) conditions support only Acknowledge and AddComment; their Confirm, shelve and Enable/Disable methods return `BadNotSupported`. The engine's later re-projection of the same transition finds the node already in that state and is suppressed, so each accepted call yields exactly one event. See `docs/AlarmTracking.md` §"One condition event per method call".
`WriteAlarmCondition` fires a Part 9 condition event only when the incoming state differs from the node's current live state. For an OPC UA method call (Acknowledge, Confirm, AddComment, the shelve methods, Enable/Disable) the SDK applies the new state to the node, and the node manager then reports **the one** condition event for it from the method path (Gitea #587 — the SDK reports none of its own, since `AutoReportStateChanges` stays false; the shelve methods' state is applied by the node manager, because the SDK's never applies it; a call that changed nothing — e.g. an uncommented re-ack — reports nothing; a re-ack carrying a comment reports that comment once, Gitea #597). Native (driver-fed) conditions support only Acknowledge and AddComment; their Confirm, shelve and Enable/Disable methods return `BadNotSupported`, and their Acknowledge is refused with `BadInvalidState` on a node that is not the redundancy Primary (Gitea #597). A timed shelve's `UnshelveTime` counts down to the engine's expiry (the projection carries `UnshelveAtUtc`). The engine's later re-projection of the same transition finds the node already in that state and is suppressed, so each accepted call yields exactly one event. See `docs/AlarmTracking.md` §"One condition event per method call".
### AdminUI path
@@ -24,6 +24,12 @@ namespace ZB.MOM.WW.OtOpcUa.Commons.OpcUa;
/// Carries the full sub-code since #519, so a comms-lost condition reports
/// <see cref="OpcUaStatus.BadNoCommunication"/> rather than a bare <see cref="OpcUaStatus.Bad"/>.
/// </param>
/// <param name="UnshelveAtUtc">
/// For <see cref="AlarmShelvingKind.Timed"/>, the UTC instant the engine expires the shelve; otherwise
/// <c>null</c>. Carried so the node's Part 9 <c>ShelvingState.UnshelveTime</c> reports the real remaining
/// time instead of the SDK's ~24.8-day ceiling (Gitea #597). A timed snapshot without it (a caller that
/// predates the field) falls back to that ceiling — the engine still owns the expiry either way.
/// </param>
public sealed record AlarmConditionSnapshot(
bool Active,
bool Acknowledged,
@@ -32,7 +38,8 @@ public sealed record AlarmConditionSnapshot(
AlarmShelvingKind Shelving,
ushort Severity,
string Message,
uint StatusCode = OpcUaStatus.Good);
uint StatusCode = OpcUaStatus.Good,
DateTime? UnshelveAtUtc = null);
/// <summary>
/// Commons-local mirror of the Core <c>ShelvingKind</c> enum so this assembly carries no
@@ -44,6 +44,7 @@ public sealed class OtOpcUaServerHostedService : IHostedService, IAsyncDisposabl
private readonly ActorRegistry _actorRegistry;
private readonly ILoggerFactory _loggerFactory;
private readonly ILogger<OtOpcUaServerHostedService> _logger;
private readonly IDriverPrimaryRoleSource? _primaryRoleSource;
private OpcUaApplicationHost? _appHost;
private OtOpcUaSdkServer? _server;
@@ -88,6 +89,11 @@ public sealed class OtOpcUaServerHostedService : IHostedService, IAsyncDisposabl
/// <param name="time">The shared clock — the one instance the authorizer and the API-key store bound
/// also read. Its <c>CreateTimer</c> arms the revalidation tick.</param>
/// <param name="loggerFactory">The logger factory for creating loggers.</param>
/// <param name="primaryRoleSource">Gitea #597 — the data-plane Primary verdict <c>DriverHostActor</c>
/// publishes on every redundancy snapshot (the same one its native-ack route and device-write gates apply).
/// Wired onto the node manager so a native-condition Acknowledge on a non-Primary node is refused rather than
/// applied and then reverted. Optional: a node with no driver role registers none, and null means "allow" —
/// the single-node posture <c>UnmanagedDriverPrimaryRoleSource</c> also answers with.</param>
public OtOpcUaServerHostedService(
IOptions<OpcUaApplicationHostOptions> options,
DeferredAddressSpaceSink deferredSink,
@@ -101,7 +107,8 @@ public sealed class OtOpcUaServerHostedService : IHostedService, IAsyncDisposabl
DataPlaneAuthorizer? dataPlaneAuthorizer,
IOptions<AclOptions> aclOptions,
TimeProvider time,
ILoggerFactory loggerFactory)
ILoggerFactory loggerFactory,
IDriverPrimaryRoleSource? primaryRoleSource = null)
{
ArgumentNullException.ThrowIfNull(aclOptions);
ArgumentNullException.ThrowIfNull(time);
@@ -123,6 +130,7 @@ public sealed class OtOpcUaServerHostedService : IHostedService, IAsyncDisposabl
_actorSystemAccessor = actorSystemAccessor;
_actorRegistry = actorRegistry;
_loggerFactory = loggerFactory;
_primaryRoleSource = primaryRoleSource;
_logger = loggerFactory.CreateLogger<OtOpcUaServerHostedService>();
}
@@ -225,6 +233,21 @@ public sealed class OtOpcUaServerHostedService : IHostedService, IAsyncDisposabl
}
});
// Gitea #597: a native ack is accepted only on the Primary. The verdict is the one DriverHostActor
// publishes and gates the route above on, so a Secondary refuses (BadInvalidState) instead of applying an
// ack the driver will never see and then reverting it. A volatile property read — safe under the SDK Lock.
if (_primaryRoleSource is { } primaryRoleSource)
{
_server.SetNativeAlarmAckPrimaryGate(() => primaryRoleSource.ShouldActAsPrimary);
}
else
{
// Every driver node registers one (AddOtOpcUaRuntime), so this is a wiring regression, not a mode:
// say so rather than let a Secondary silently go back to accepting acks it will revert.
_logger.LogWarning(
"OtOpcUaServerHostedService: no IDriverPrimaryRoleSource registered — native-alarm acknowledges are accepted on every node regardless of redundancy role");
}
// Wire the reverse-path inbound operator-write gateway: a client write to a writable equipment-tag
// node that passes the node manager's WriteOperate gate routes the write to the owning driver child
// (RouteNodeWrite → NodeWriteResult) via the local DriverHostActor. The node manager calls the
@@ -342,6 +365,7 @@ public sealed class OtOpcUaServerHostedService : IHostedService, IAsyncDisposabl
_server?.SetNodeWriteGateway(null);
// Clear the native-alarm-ack router so a late ack doesn't Tell a stopping DriverHostActor.
_server?.SetNativeAlarmAckRouter(null);
_server?.SetNativeAlarmAckPrimaryGate(null);
// Restore the Null historian so a late HistoryRead doesn't hit a disposed read client.
_server?.SetHistorianDataSource(null);
// Restore the empty authority so a late session activation cannot authorize against a stopping node.
@@ -205,6 +205,26 @@ public sealed class OtOpcUaNodeManager : CustomNodeManager2
/// </summary>
public Action<NativeAlarmAck>? NativeAlarmAckRouter { get; set; }
/// <summary>
/// Gitea #597 — whether this node may accept an OPC UA Acknowledge of a NATIVE (driver-fed) condition
/// right now: the redundancy data-plane verdict (<c>PrimaryGatePolicy.ShouldServiceAsPrimary</c>, as
/// published to <c>IDriverPrimaryRoleSource</c>) — the same verdict <c>DriverHostActor</c> gates the
/// upstream ack route and device writes on.
/// <para>
/// <c>false</c> ⇒ the Acknowledge is REFUSED with <c>BadInvalidState</c> before anything is routed or
/// applied. Without this the Secondary applied and reported the ack locally while
/// <c>DriverHostActor</c> dropped the route upstream, so the driver never acknowledged and its next
/// transition reverted the node — the client saw acked → unacked. Scripted conditions are not gated:
/// their engine runs on both nodes, so an ack against the Secondary is real.
/// </para>
/// <para>
/// Read under <c>Lock</c> on the SDK's Call thread, so it MUST be a cheap, non-blocking read (the
/// host passes a volatile-backed property). Null (the default) allows — the single-node posture, the
/// same default <c>UnmanagedDriverPrimaryRoleSource</c> answers with.
/// </para>
/// </summary>
public Func<bool>? NativeAlarmAckPrimaryGate { get; set; }
private volatile IOpcUaNodeWriteGateway _nodeWriteGateway = NullOpcUaNodeWriteGateway.Instance;
/// <summary>
@@ -849,13 +869,16 @@ public sealed class OtOpcUaNodeManager : CustomNodeManager2
if (condition.ShelvingState is not null)
{
// SetShelvingState(shelved, oneShot, shelvingTime): map our 3-way kind onto the SDK's
// (shelved, oneShot) flag pair. Timed shelving's expiry is owned by the engine, not the
// SDK timer, so we pass shelvingTime=0 (no SDK-managed auto-unshelve).
// (shelved, oneShot) flag pair. Timed shelving's expiry is owned by the engine; the time
// passed here is the engine's remaining time, so the node's UnshelveTime counts down to the
// real expiry (Gitea #597 — a 0 here reads as MaxTimeShelved, ~24.8 days). The SDK timer it
// arms only routes an Unshelve back to the engine (OnTimedUnshelve), a no-op once the
// engine's own expiry has run.
condition.SetShelvingState(
SystemContext,
shelved: state.Shelving != AlarmShelvingKind.Unshelved,
oneShot: state.Shelving == AlarmShelvingKind.OneShot,
shelvingTime: 0);
shelvingTime: state.Shelving == AlarmShelvingKind.Timed ? RemainingShelveMs(state.UnshelveAtUtc) : 0);
}
condition.SetSeverity(SystemContext, MapSeverity(state.Severity));
condition.Message.Value = new LocalizedText(state.Message);
@@ -909,6 +932,22 @@ public sealed class OtOpcUaNodeManager : CustomNodeManager2
}
}
/// <summary>
/// Gitea #597 — the <c>shelvingTime</c> (ms) to hand <c>SetShelvingState</c> for a timed shelve expiring at
/// <paramref name="unshelveAtUtc"/>. <c>0</c> when the expiry is unknown (the SDK then uses
/// <c>MaxTimeShelved</c>, the pre-#597 behaviour); otherwise the remaining time, clamped to at least 1 ms
/// (an already-passed expiry must not read as "unknown" — the engine's own expiry is about to land) and to
/// at most <c>int.MaxValue</c> (the SDK timer's range).
/// </summary>
/// <param name="unshelveAtUtc">The engine's expiry instant, or null.</param>
/// <returns>The shelving time in milliseconds.</returns>
internal static double RemainingShelveMs(DateTime? unshelveAtUtc)
{
if (unshelveAtUtc is not { } at) return 0;
var remaining = (at - DateTime.UtcNow).TotalMilliseconds;
return Math.Clamp(Math.Ceiling(remaining), 1, int.MaxValue);
}
/// <summary>
/// #477 Layer 2 — apply a source-data quality annotation to a materialised condition, OUT OF BAND
/// from any alarm transition. Used by the driver-connectivity path (comms lost → BadNoCommunication,
@@ -1545,9 +1584,10 @@ public sealed class OtOpcUaNodeManager : CustomNodeManager2
/// </para>
/// <para>
/// <b>Shelving is applied here.</b> The SDK's shelve methods never change <c>ShelvingState</c>
/// themselves, so an accepted shelve call applies it — with the same <c>(shelved, oneShot, 0)</c>
/// mapping <see cref="WriteAlarmCondition"/> projects — before reporting. The engine's re-projection of
/// the shelve then finds no delta. Every other method's state the SDK has already applied.
/// themselves, so an accepted shelve call applies it — with the same <c>(shelved, oneShot)</c> mapping
/// <see cref="WriteAlarmCondition"/> projects, and a TimedShelve with its requested duration so
/// <c>UnshelveTime</c> is real (Gitea #597) — before reporting. The engine's re-projection of the shelve
/// then finds no delta. Every other method's state the SDK has already applied.
/// </para>
/// <para>
/// <b>Exactly once.</b> The whole call runs under <c>Lock</c>. The method handlers hand the command
@@ -1569,7 +1609,10 @@ public sealed class OtOpcUaNodeManager : CustomNodeManager2
/// the SDK handler, and nothing is reported when it did not move. The SDK accepts acknowledging an
/// acknowledged condition and sets <c>Confirmed = false</c> again; a re-ack therefore also restores the
/// pre-call Confirmed and Retain, so a scripted node never drifts from its engine (which ignores a
/// re-ack) and the call is silent. AddComment always reports: adding a comment is itself the change.
/// re-ack) and the call is silent — unless it carried a comment, which the SDK writes and which is then
/// reported once like AddComment (Gitea #597). AddComment always reports: adding a comment is itself
/// the change. A NATIVE re-ack is still routed to the driver: the route is fire-and-forget, so a node
/// showing Acked does not prove the upstream system acknowledged, and re-acking is the documented retry.
/// </para>
/// <para>
/// A rejected call (bad <see cref="ServiceResult"/>, e.g. a denied gate, or a refused method on a native
@@ -1582,8 +1625,12 @@ public sealed class OtOpcUaNodeManager : CustomNodeManager2
if (alarm.Acknowledge?.OnCall is { } acknowledge)
{
// A commented acknowledge always reports (Gitea #597): the SDK writes the comment even on a re-ack,
// and a new comment is a Part 9 change in its own right — exactly as AddComment always reports. An
// uncommented re-ack still changes nothing and stays silent.
alarm.Acknowledge.OnCall = (context, m, objectId, eventId, comment) =>
ReportAfterMethod(() => acknowledge(context, m, objectId, eventId, comment), apply: null, isAcknowledge: true);
ReportAfterMethod(() => acknowledge(context, m, objectId, eventId, comment), apply: null, isAcknowledge: true,
alwaysReport: !LocalizedText.IsNullOrEmpty(comment));
}
WrapComment(alarm.Confirm, alwaysReport: false);
@@ -1597,11 +1644,13 @@ public sealed class OtOpcUaNodeManager : CustomNodeManager2
WrapGeneric(shelving.Unshelve, () => alarm.SetShelvingState(SystemContext, shelved: false, oneShot: false, shelvingTime: 0));
if (shelving.TimedShelve is { OnCall: { } timed } timedShelve)
{
// shelvingTime 0, as WriteAlarmCondition projects it: the scripted engine owns the expiry and
// re-projects Unshelved when it passes (the SDK's own timer only routes an Unshelve back to it).
// The requested duration, so UnshelveTime counts down to the real expiry (Gitea #597 — 0 reads as
// MaxTimeShelved, ~24.8 days). The scripted engine still owns the expiry and re-projects Unshelved
// when it passes (with its own remaining time meanwhile — WriteAlarmCondition); the SDK timer this
// arms only routes an Unshelve back to it (OnTimedUnshelve), a no-op once the engine has expired it.
timedShelve.OnCall = (context, method, objectId, shelvingTime) => ReportAfterMethod(
() => timed(context, method, objectId, shelvingTime),
() => alarm.SetShelvingState(SystemContext, shelved: true, oneShot: false, shelvingTime: 0));
() => alarm.SetShelvingState(SystemContext, shelved: true, oneShot: false, shelvingTime: shelvingTime));
}
}
@@ -1729,7 +1778,8 @@ public sealed class OtOpcUaNodeManager : CustomNodeManager2
/// folder-scoped condition node id the driver-bound router resolves back to a driver ref.</param>
/// <param name="comment">The acknowledge comment text, or <c>null</c> when none was supplied.</param>
/// <returns><c>ServiceResult.Good</c> when allowed (the SDK then applies state; the method path reports the event);
/// <c>BadUserAccessDenied</c> when the gate vetoes (no route, no state mutation).</returns>
/// <c>BadUserAccessDenied</c> when the gate vetoes, <c>BadInvalidState</c> when this node is not the redundancy
/// Primary (<see cref="NativeAlarmAckPrimaryGate"/>, Gitea #597) — in both cases no route, no state mutation.</returns>
private ServiceResult HandleNativeAlarmAck(ISystemContext context, ConditionState condition, LocalizedText? comment)
{
// Resolve + gate the SAME way HandleAlarmCommand does so native and scripted acks share one authz
@@ -1738,6 +1788,16 @@ public sealed class OtOpcUaNodeManager : CustomNodeManager2
var gate = EvaluateAlarmCommandGate(identity, condition, "Acknowledge");
if (gate is not null) return gate;
// Gitea #597 — PRIMARY GATE, after authorization (a caller who may not ack learns nothing about the
// node's role). Only the Primary routes a native ack upstream (DriverHostActor applies the same verdict
// to the route), so on any other node the ack is refused outright rather than applied locally and then
// reverted by the driver's next transition. Nothing is routed, applied or reported.
if (NativeAlarmAckPrimaryGate is { } primary && !primary())
{
return new ServiceResult(StatusCodes.BadInvalidState,
new LocalizedText("This server is not the redundancy Primary; acknowledge the alarm on the Primary (the server with the higher ServiceLevel)."));
}
// Same condition-node-id extraction HandleAlarmCommand uses (condition.NodeId.Identifier?.ToString()):
// for a native condition this is the folder-scoped NodeId string the DriverHostActor inverse map
// (_alarmNodeIdByDriverRef value set) keys by, so the next task can resolve it back to (driver, ref).
@@ -3046,6 +3106,10 @@ public sealed class OtOpcUaNodeManager : CustomNodeManager2
{
if (_variables.TryRemove(id, out var v))
{
// Gitea #599 (archreview 03/S20): sever the inverse half of each cross-tree Organizes edge
// first, exactly as RemoveVariableNode does — a UNS reference variable in scope points at a
// RAW node that survives this UNS-scoped removal and would otherwise keep a dangling edge.
SeverOutgoingReferences(v);
v.Parent?.RemoveChild(v);
PredefinedNodes?.Remove(v.NodeId);
}
@@ -60,6 +60,22 @@ public sealed class OtOpcUaSdkServer : StandardServer
return true;
}
/// <summary>
/// Gitea #597 — wire the redundancy verdict that decides whether this node accepts an OPC UA Acknowledge
/// of a NATIVE condition (<see cref="OtOpcUaNodeManager.NativeAlarmAckPrimaryGate"/>); on any other node
/// the ack is refused with <c>BadInvalidState</c> instead of being applied locally and reverted by the
/// driver's next transition. Passing <c>null</c> clears it (allow). No-op (returns <c>false</c>) when the
/// node manager has not been created yet (mirrors <see cref="SetNativeAlarmAckRouter"/>).
/// </summary>
/// <param name="isPrimary">A cheap, non-blocking read of the data-plane Primary verdict; null to allow.</param>
/// <returns><c>true</c> when set on a live node manager; <c>false</c> when no node manager exists yet.</returns>
public bool SetNativeAlarmAckPrimaryGate(Func<bool>? isPrimary)
{
if (_otOpcUaNodeManager is null) return false;
_otOpcUaNodeManager.NativeAlarmAckPrimaryGate = isPrimary;
return true;
}
/// <summary>
/// Wire the reverse-path gateway for inbound operator writes to writable equipment-tag nodes onto the
/// created <see cref="OtOpcUaNodeManager"/>. The host calls this after start with an
@@ -739,7 +739,9 @@ public sealed class ScriptedAlarmHostActor : ReceiveActor
// #478 — the condition's status is the worst status across the script's input tags at evaluation
// time (carried on the event by the engine). A transition fired while an input is Uncertain projects
// Uncertain here so the full-snapshot write doesn't clobber quality back to Good.
StatusCode: worstInputStatusCode);
StatusCode: worstInputStatusCode,
// Gitea #597 — the engine's own expiry, so the node's UnshelveTime counts down to it.
UnshelveAtUtc: condition.Shelving.Kind == ShelvingKind.Timed ? condition.Shelving.UnshelveAtUtc : null);
/// <summary>Maps the Core <see cref="ShelvingKind"/> onto the Commons <see cref="AlarmShelvingKind"/>
/// mirror (the Commons assembly can't see the Core enum).</summary>
@@ -0,0 +1,81 @@
using Akka.Hosting;
using Akka.TestKit.Xunit2;
using Shouldly;
using Xunit;
using ZB.MOM.WW.OtOpcUa.AdminUI.Clients;
using ZB.MOM.WW.OtOpcUa.AdminUI.Tests.Acls;
using ZB.MOM.WW.OtOpcUa.Commons.Messages.Admin;
using ZB.MOM.WW.OtOpcUa.Commons.Messages.Mesh;
using ZB.MOM.WW.OtOpcUa.Commons.OpcUa;
using ZB.MOM.WW.OtOpcUa.ControlPlane;
using ZB.MOM.WW.OtOpcUa.ControlPlane.AdminOperations;
using ZB.MOM.WW.OtOpcUa.Core.Abstractions;
namespace ZB.MOM.WW.OtOpcUa.AdminUI.Tests;
/// <summary>
/// Gitea #597 — the AdminUI <c>/alerts</c> acknowledge / shelve path, from the page's client call down to
/// the command the cluster receives: <see cref="AdminOperationsClient"/> (what <c>Alerts.razor</c>'s
/// per-row buttons call) Asks the real <see cref="AdminOperationsActor"/> registered under
/// <see cref="AdminOperationsActorKey"/>, which maps the request to an <see cref="AlarmCommand"/> and hands
/// it to the mesh transport. Until now only the actor half had a test; the client seam the page goes
/// through had none.
/// </summary>
/// <remarks>
/// The razor click handler itself is not driven here — this repo has no bUnit. What it adds on top of the
/// client call is the operator name and <c>comment: null</c>, both reproduced below exactly as the page
/// passes them.
/// </remarks>
public sealed class AlertsAckPathTests : TestKit
{
private (AdminOperationsClient Client, Akka.TestKit.TestProbe Mesh) Wire()
{
var mesh = CreateTestProbe("mesh-router");
var actor = Sys.ActorOf(AdminOperationsActor.Props(
AclScopeTestDb.Factory($"alerts-ack-{Guid.NewGuid():N}"),
coordinator: CreateTestProbe("coordinator").Ref,
probes: Enumerable.Empty<IDriverProbe>(),
meshRouter: mesh.Ref));
var registry = new ActorRegistry();
registry.Register<AdminOperationsActorKey>(actor);
return (new AdminOperationsClient(registry), mesh);
}
/// <summary>The page's Acknowledge reaches the cluster as an <c>Acknowledge</c> AlarmCommand on the
/// <c>alarm-commands</c> topic carrying the alarm id and the operator, and the page is told it was sent.</summary>
[Fact]
public async Task Acknowledge_from_the_alerts_page_dispatches_an_Acknowledge_command_and_reports_sent()
{
var (client, mesh) = Wire();
var result = await client.AcknowledgeAlarmAsync("alarm-42", "operator-jo", comment: null, CancellationToken.None);
result.Ok.ShouldBeTrue();
result.Message.ShouldBeNull();
var sent = mesh.ExpectMsg<MeshCommand>(TimeSpan.FromSeconds(5));
sent.Topic.ShouldBe(AlarmCommandsTopic.Name);
var command = sent.Message.ShouldBeOfType<AlarmCommand>();
command.AlarmId.ShouldBe("alarm-42");
command.Operation.ShouldBe("Acknowledge");
command.User.ShouldBe("operator-jo");
command.Comment.ShouldBeNull();
command.UnshelveAtUtc.ShouldBeNull();
}
/// <summary>The page's timed shelve carries its expiry through to the command — the instant the engine
/// projects as the node's <c>UnshelveTime</c>.</summary>
[Fact]
public async Task Timed_shelve_from_the_alerts_page_carries_its_expiry_to_the_command()
{
var (client, mesh) = Wire();
var until = new DateTime(2026, 10, 8, 12, 30, 0, DateTimeKind.Utc);
var result = await client.ShelveAlarmAsync(
"alarm-7", "operator-kim", ShelveKind.Timed, unshelveAtUtc: until, comment: null, CancellationToken.None);
result.Ok.ShouldBeTrue();
var command = mesh.ExpectMsg<MeshCommand>(TimeSpan.FromSeconds(5)).Message.ShouldBeOfType<AlarmCommand>();
command.Operation.ShouldBe("TimedShelve");
command.UnshelveAtUtc.ShouldBe(until);
}
}
@@ -266,10 +266,11 @@ public sealed class ConditionMethodEventTests
/// reports exactly one unshelve event after the one shelve event.
/// </summary>
/// <remarks>
/// The method path applies the shelve with <c>shelvingTime 0</c>, as the projection does, so the expiry is
/// the engine's (computed from the command's <c>UnshelveAtUtc</c>), never the SDK timer's — a node timer
/// would only route an Unshelve back to the engine anyway. The stand-in engine here schedules its expiry
/// from the real <c>UnshelveAtUtc</c> the method handler computed.
/// The expiry is the engine's (computed from the command's <c>UnshelveAtUtc</c>). Since Gitea #597 the node
/// is shelved for the real duration, so the SDK's own timer fires at the same instant — but it only routes
/// an Unshelve back to the engine (ignored by this stand-in, a no-op on an unshelved engine alarm) and
/// changes no node state, so it adds no event. The stand-in engine here schedules its expiry from the real
/// <c>UnshelveAtUtc</c> the method handler computed.
/// </remarks>
[Fact]
public async Task A_scripted_timed_shelve_expires_with_exactly_one_unshelve_event()
@@ -326,17 +327,157 @@ public sealed class ConditionMethodEventTests
}
/// <summary>
/// Acknowledging an acknowledged condition reports nothing, and does not flip a scripted condition's
/// Confirmed out of step with its engine.
/// A scripted condition's events carry the real remaining shelve time in
/// <c>ShelvingState/UnshelveTime</c> — after a client TimedShelve, after the engine's re-projection of it,
/// and after an engine-driven timed shelve (the AdminUI <c>/alerts</c> path, which never calls the method).
/// </summary>
/// <remarks>
/// The SDK accepts a re-ack and sets Confirmed false again; the engine ignores it. The scripted leg
/// acknowledges and confirms first, so the flip would be visible on the sentinel.
/// Gitea #597: both paths applied the shelve with <c>shelvingTime 0</c>, which the SDK reads as "the
/// longest shelve allowed" — <c>MaxTimeShelved</c>, <c>int.MaxValue</c> ms — so every timed shelve
/// advertised ~24.8 days remaining while the engine expired it on time. Read through the event select
/// clause rather than a Read of the child node: the condition's children carry their type's
/// instance-declaration NodeIds (<c>i=9189</c> for UnshelveTime), so a direct Read reaches the type, not
/// this condition.
/// </remarks>
[Fact]
public async Task A_scripted_timed_shelve_reports_the_real_remaining_unshelve_time()
{
var ct = TestContext.Current.CancellationToken;
var rig = await StartAsync("CondUnshelveTime", encrypted: false, ct);
try
{
rig.Enforce(true, [Key()], ClusterGrant(KeyPrincipal, "acl-key", All));
rig.MaterialiseScripted();
var current = PreState("TimedShelve", native: false) with { Message = "base" };
rig.Nm.AlarmCommandRouter = cmd =>
{
if (cmd.Operation != "TimedShelve") return;
// The engine's answer carries its own expiry — what ScriptedAlarmHostActor projects.
current = current with { Shelving = AlarmShelvingKind.Timed, UnshelveAtUtc = cmd.UnshelveAtUtc };
rig.ProjectLater(native: false, current);
};
using var session = await rig.ConnectAsync(anonymous: false, ct);
var events = new RecordedEvents();
var subscription = await RecordedEvents.SubscribeAsync(session, events, ct);
var item = RecordedEvents.AddEventItem(subscription, ObjectIds.Server);
await subscription.ApplyChangesAsync(ct);
double RemainingMs(string message) =>
(double)events.ConditionEvents(item, message)[^1][RecordedEvents.UnshelveTimeField].Value;
rig.Project(false, current);
await RecordedEvents.WaitUntilAsync(() => events.Has(item, "base"));
RemainingMs("base").ShouldBe(0d, "unshelved");
// The method's one event carries the shelve's remaining time …
await CallAsync(session, ConditionNodeId(session, false), "TimedShelve", null, ct, shelvingTimeMs: 60_000);
await rig.DrainProjectionsAsync();
await RecordedEvents.WaitUntilAsync(() => events.ConditionEvents(item, "base").Count == 2);
RemainingMs("base").ShouldBeInRange(50_000, 60_000, "the TimedShelve method event");
// … and so does a later transition, after the engine's re-projection re-applied the shelve.
rig.Project(false, current = current with { Message = "after-reprojection" });
await RecordedEvents.WaitUntilAsync(() => events.Has(item, "after-reprojection"));
RemainingMs("after-reprojection").ShouldBeInRange(50_000, 60_000, "after the engine's re-projection");
// Engine-driven (the AdminUI path): unshelve, then a timed shelve that reaches the node only as a
// projection, never through the method.
rig.Project(false, current = current with { Shelving = AlarmShelvingKind.Unshelved, UnshelveAtUtc = null, Message = "unshelved" });
await RecordedEvents.WaitUntilAsync(() => events.Has(item, "unshelved"));
RemainingMs("unshelved").ShouldBe(0d);
rig.Project(false, current with
{
Shelving = AlarmShelvingKind.Timed, UnshelveAtUtc = DateTime.UtcNow.AddSeconds(30), Message = "projected",
});
await RecordedEvents.WaitUntilAsync(() => events.Has(item, "projected"));
RemainingMs("projected").ShouldBeInRange(20_000, 30_000, "a projected timed shelve");
await subscription.DeleteAsync(true, ct);
}
finally { await rig.DisposeAsync(); }
}
/// <summary>
/// On a node that is not the redundancy Primary, an OPC UA Acknowledge of a NATIVE condition is refused
/// with <c>BadInvalidState</c>: nothing is routed to the driver, nothing is applied, nothing is reported.
/// On the Primary the same call is accepted (the positive control).
/// </summary>
/// <remarks>
/// Gitea #597 decision. The ack used to be applied locally and reported, while <c>DriverHostActor</c>'s
/// Primary gate dropped the route upstream — so the driver never acknowledged, its next transition
/// projected <c>Acked = false</c>, and a client acking against the Secondary saw acked → unacked. An
/// honest refusal replaces an ack that visibly sticks and then reverts. Scripted conditions are
/// unaffected: their engine runs on both nodes, so the Secondary's ack is real.
/// </remarks>
[Fact]
public async Task A_native_acknowledge_on_a_non_primary_node_is_refused()
{
var ct = TestContext.Current.CancellationToken;
var rig = await StartAsync("CondNativeAckSecondary", encrypted: false, ct);
try
{
rig.Enforce(true, [Key()], ClusterGrant(KeyPrincipal, "acl-key", All));
rig.MaterialiseNative();
var current = PreState("Acknowledge", native: true) with { Message = "base" };
WireSimulatedOwners(rig, native: true, () => current, s => current = s);
var routed = 0;
var driverAck = rig.Nm.NativeAlarmAckRouter!;
rig.Nm.NativeAlarmAckRouter = ack => { Interlocked.Increment(ref routed); driverAck(ack); };
var primary = false;
rig.Nm.NativeAlarmAckPrimaryGate = () => primary;
using var session = await rig.ConnectAsync(anonymous: false, ct);
var events = new RecordedEvents();
var subscription = await RecordedEvents.SubscribeAsync(session, events, ct);
var item = RecordedEvents.AddEventItem(subscription, ObjectIds.Server);
await subscription.ApplyChangesAsync(ct);
rig.Project(native: true, current);
await RecordedEvents.WaitUntilAsync(() => events.Has(item, "base"));
var condition = ConditionNodeId(session, native: true);
var refused = await Should.ThrowAsync<ServiceResultException>(
CallAsync(session, condition, "Acknowledge", events.LatestEventId(item, "base"), ct));
refused.StatusCode.ShouldBe(StatusCodes.BadInvalidState);
routed.ShouldBe(0, "a refused ack reaches no driver");
rig.Nm.TryGetAlarmCondition(RawAlarmPath, AddressSpaceRealm.Raw)!.AckedState.Id.Value
.ShouldBeFalse("a refused ack applies nothing");
// Positive control: the same call on the Primary is accepted, routed and reported.
primary = true;
await CallAsync(session, condition, "Acknowledge", events.LatestEventId(item, "base"), ct);
await rig.DrainProjectionsAsync();
routed.ShouldBe(1);
rig.Project(native: true, current = current with { Message = "sentinel" });
await RecordedEvents.WaitUntilAsync(() => events.Has(item, "sentinel"));
var baseEvents = events.ConditionEvents(item, "base");
baseEvents.Count.ShouldBe(2, "the base transition plus the Primary's one ack event; the refusal reported nothing");
baseEvents[1][RecordedEvents.AckedField].Value.ShouldBe(true);
await subscription.DeleteAsync(true, ct);
}
finally { await rig.DisposeAsync(); }
}
/// <summary>
/// Acknowledging an acknowledged condition WITHOUT a comment reports nothing, and does not flip a scripted
/// condition's Confirmed out of step with its engine. With a comment it reports exactly one event carrying
/// that comment — the comment is a Part 9 change in its own right, as AddComment is (Gitea #597) — and
/// still leaves Acked/Confirmed alone.
/// </summary>
/// <remarks>
/// The SDK accepts a re-ack, writes its comment, and sets Confirmed false again; the engine ignores it.
/// The scripted leg acknowledges and confirms first, so the flip would be visible on the sentinel. Before
/// #597 a commented re-ack wrote the comment to the node but reported nothing, so no subscriber ever saw
/// it.
/// </remarks>
[Theory]
[InlineData(false)]
[InlineData(true)]
public async Task A_re_acknowledge_reports_nothing_and_changes_nothing(bool native)
[InlineData(false, null)]
[InlineData(true, null)]
[InlineData(false, "again")]
[InlineData(true, "again")]
public async Task A_re_acknowledge_reports_only_its_comment_and_changes_nothing_else(bool native, string? reAckComment)
{
var ct = TestContext.Current.CancellationToken;
var rig = await StartAsync($"CondReAck{native}", encrypted: false, ct);
@@ -347,6 +488,9 @@ public sealed class ConditionMethodEventTests
rig.MaterialiseScripted();
var current = PreState("Acknowledge", native) with { Message = "base" };
WireSimulatedOwners(rig, native, () => current, s => current = s);
var routedAcks = 0;
var driverAck = rig.Nm.NativeAlarmAckRouter!;
rig.Nm.NativeAlarmAckRouter = ack => { Interlocked.Increment(ref routedAcks); driverAck(ack); };
using var session = await rig.ConnectAsync(anonymous: false, ct);
var events = new RecordedEvents();
@@ -370,13 +514,32 @@ public sealed class ConditionMethodEventTests
}
await RecordedEvents.WaitUntilAsync(() => events.ConditionEvents(item, "base").Count == expected);
await CallAsync(session, condition, "Acknowledge", events.LatestEventId(item, "base"), ct);
var routedBefore = routedAcks;
await CallAsync(session, condition, "Acknowledge", events.LatestEventId(item, "base"), ct, ackComment: reAckComment);
await rig.DrainProjectionsAsync();
rig.Project(native, current = current with { Message = "sentinel" });
await RecordedEvents.WaitUntilAsync(() => events.Has(item, "sentinel"));
events.ConditionEvents(item, "base").Count.ShouldBe(expected, "the re-ack must report nothing");
var baseEvents = events.ConditionEvents(item, "base");
if (reAckComment is null)
{
baseEvents.Count.ShouldBe(expected, "an uncommented re-ack must report nothing");
}
else
{
baseEvents.Count.ShouldBe(expected + 1, "a commented re-ack reports exactly one comment event");
var commented = baseEvents[^1];
(commented[RecordedEvents.CommentField].Value as LocalizedText)?.Text.ShouldBe(reAckComment);
commented[RecordedEvents.AckedField].Value.ShouldBe(true);
commented[RecordedEvents.ConfirmedField].Value.ShouldBe(true, "the commented re-ack must not unconfirm the node");
}
// A native re-ack is still routed to the driver: the route is fire-and-forget, so a node showing
// Acked does not prove the upstream system acknowledged, and re-acking is the documented retry
// (docs/AlarmTracking.md, "Best-effort during reconnect"). Pinned so it is changed deliberately.
if (native) (routedAcks - routedBefore).ShouldBe(1);
var sentinel = events.ConditionEvents(item, "sentinel");
sentinel.Count.ShouldBe(1);
sentinel[0][RecordedEvents.AckedField].Value.ShouldBe(true);
@@ -467,11 +630,13 @@ public sealed class ConditionMethodEventTests
/// condition's method declarations — the shelve methods included); throws on a bad result.
/// </summary>
private static async Task CallAsync(
ISession session, NodeId condition, string method, byte[]? eventId, CancellationToken ct, double shelvingTimeMs = 60_000)
ISession session, NodeId condition, string method, byte[]? eventId, CancellationToken ct, double shelvingTimeMs = 60_000,
string? ackComment = "ack")
{
var (methodId, args) = method switch
{
"Acknowledge" => (MethodIds.AcknowledgeableConditionType_Acknowledge, new object[] { eventId!, new LocalizedText("ack") }),
"Acknowledge" => (MethodIds.AcknowledgeableConditionType_Acknowledge,
new object[] { eventId!, ackComment is null ? LocalizedText.Null : new LocalizedText(ackComment) }),
"Confirm" => (MethodIds.AcknowledgeableConditionType_Confirm, [eventId!, new LocalizedText("confirm")]),
"AddComment" => (MethodIds.ConditionType_AddComment, [eventId!, new LocalizedText("note")]),
"Enable" => (MethodIds.ConditionType_Enable, []),
@@ -292,6 +292,7 @@ internal sealed class RecordedEvents
public const int CommentField = 7;
public const int ChangesField = 8;
public const int SourceNodeField = 9;
public const int UnshelveTimeField = 10;
private readonly object _gate = new();
private readonly List<(uint Handle, VariantCollection Fields)> _events = [];
@@ -349,6 +350,8 @@ internal sealed class RecordedEvents
filter.AddSelectClause(ObjectTypeIds.ConditionType, BrowseNames.Comment);
filter.AddSelectClause(ObjectTypeIds.GeneralModelChangeEventType, BrowseNames.Changes);
filter.AddSelectClause(ObjectTypes.BaseEventType, BrowseNames.SourceNode);
filter.SelectClauses.Add(new SimpleAttributeOperand(
ObjectTypeIds.AlarmConditionType, [BrowseNames.ShelvingState, BrowseNames.UnshelveTime]));
var item = new MonitoredItem(subscription.DefaultItem)
{