Merge #869 Galaxy native alarm ack: unconfirmed gateway OK waits for Galaxy's acknowledged transition, reverts on timeout

Claude-Session: https://claude.ai/code/session_01GT67AD99zQQqpLCVm3p7xk
This commit is contained in:
Joseph Doherty
2026-11-11 13:30:40 -05:00
23 changed files with 1438 additions and 60 deletions
+6 -1
View File
@@ -42,10 +42,15 @@ yet re-listed. Only an identity written straight into the store by the `secret`
refused until the next refresh, which the refusal itself requests. (2) anything else (a keyed session the cache cannot vouch for, MXAccess refused until the next refresh, which the refusal itself requests. (2) anything else (a keyed session the cache cannot vouch for, MXAccess
non-zero `hresult`, driver still connecting/reconnecting) comes back as `DriverInstanceActor.AlarmAckFailed` — or, for a non-zero `hresult`, driver still connecting/reconnecting) comes back as `DriverInstanceActor.AlarmAckFailed` — or, for a
disabled/unspawned/stubbed driver, is decided in the host itself (#765) — and `DriverHostActor` re-projects the condition unacknowledged — only while the ack is still pending in disabled/unspawned/stubbed driver, is decided in the host itself (#765) — and `DriverHostActor` re-projects the condition unacknowledged — only while the ack is still pending in
`NativeAlarmProjector` (any driver transition since then wins). A host that came up **Stale** cannot route any `NativeAlarmProjector` (a driver transition since then that STATES the ack wins; a Clear / same-message Raise only carries it through and leaves it revertible, #869; the revert is keyed to that ack's `MarkAcknowledged` sequence, echoed on `RouteAlarmAck`/`AlarmAckFailed`, so another ack's failure never undoes it). A host that came up **Stale** cannot route any
native ack, so it marks the whole `NativeAlarmAckPreflightView` unavailable and every native ack is refused native ack, so it marks the whole `NativeAlarmAckPreflightView` unavailable and every native ack is refused
synchronously until `BecomeSteady` (#765). A test that routes an ack and expects it to STAND must start the owning synchronously until `BecomeSteady` (#765). A test that routes an ack and expects it to STAND must start the owning
driver (`AckingAlarmDriver` in Runtime.Tests). driver (`AckingAlarmDriver` in Runtime.Tests).
**A returned `AcknowledgeAsync` is not proof either (#869):** a driver implementing `IAlarmAcknowledgeConfirmation`
(Galaxy — the gateway's OK also covers "AckMsg write sent, unconfirmed") is asked to confirm AFTER it, off the
`DriverInstanceActor` mailbox and outside the resilience pipeline; a confirmation failure is the same
`AlarmAckFailed` revert. Never await that confirmation inside the ack handler (`ReceiveAsync` suspends the mailbox)
or inside the invoker (Tier-A 5 s timeout + breaker).
Behaviours that were wrong or missing and are now fixed — keep them: the SDK's `OnTimedUnshelve` is a Behaviours that were wrong or missing and are now fixed — keep them: the SDK's `OnTimedUnshelve` is a
**no-op** (the engine's `AutoUnshelve` owns timed-shelve expiry, and `UnshelveTime` reports the real **no-op** (the engine's `AutoUnshelve` owns timed-shelve expiry, and `UnshelveTime` reports the real
+16 -4
View File
@@ -474,7 +474,12 @@ from the scripted `AlarmCommandRouter`:
condition event). An ack whose owning driver is not running (disabled, condition event). An ack whose owning driver is not running (disabled,
not yet spawned) or is a stub on this node is reverted the same way by not yet spawned) or is a stub on this node is reverted the same way by
the host itself (#765). Only an ack still pending in `NativeAlarmProjector` is the host itself (#765). Only an ack still pending in `NativeAlarmProjector` is
reverted: a driver transition projected since then is authoritative. A reverted: a driver transition projected since then that states the ack
(Acknowledge, ActiveAcknowledged, or an un-acking Raise/Retrigger/
InactiveUnacknowledged) is authoritative; one that only carries the ack
through (a Clear, a same-message Raise) leaves it revertible (#869) — and
only by a failure of THAT acknowledge: the revert is keyed to the sequence
`MarkAcknowledged` issued, echoed on `RouteAlarmAck` / `AlarmAckFailed`. A
refusal the driver can decide without I/O (Galaxy, with no fallback ack refusal the driver can decide without I/O (Galaxy, with no fallback ack
account: an anonymous session, or a keyed session whose key the cached account: an anonymous session, or a keyed session whose key the cached
`GalaxyAckIdentityDirectory` knows has no ack identity — #765) is instead `GalaxyAckIdentityDirectory` knows has no ack identity — #765) is instead
@@ -483,9 +488,16 @@ from the scripted `AlarmCommandRouter`:
before anything is applied — as is every native ack while the driver host before anything is applied — as is every native ack while the driver host
is Stale (configuration database unreachable at boot), when no ack can be is Stale (configuration database unreachable at boot), when no ack can be
routed or reverted (#765). Galaxy value writes have a separate correlated routed or reverted (#765). Galaxy value writes have a separate correlated
write-outcome path. An awaited write-outcome path. **An OK ack reply is not proof (#869):** the gateway
ack RPC does not prove the ack reached AVEVA event history: the H6 live also answers OK when the `AckMsg` write was only *sent* (its diagnostic
verification remains [operator-gated](plans/2026-06-15-stillpending-phase-3-opcua-standards-design.md). says so). The driver then waits — off the actor mailbox, outside the
resilience pipeline, bounded by `AlarmAck.ConfirmTimeoutMs` (default 10 s)
— for Galaxy's own acknowledged transition on the alarm feed
(`IAlarmAcknowledgeConfirmation`); without one the ack is reverted exactly
as above and the `BadTimeout` verdict plus the gateway diagnostic are
logged. See [`drivers/Galaxy.md` §Ack confirmation](drivers/Galaxy.md#ack-confirmation-869).
The H6 live verification of a confirmed ack reaching AVEVA event history
remains [operator-gated](plans/2026-06-15-stillpending-phase-3-opcua-standards-design.md).
Only the **Acknowledge** is routed to the driver. **A native condition Only the **Acknowledge** is routed to the driver. **A native condition
supports Acknowledge and AddComment only** (Gitea #587 review): `Confirm`, supports Acknowledge and AddComment only** (Gitea #587 review): `Confirm`,
+47 -3
View File
@@ -85,7 +85,7 @@ Project root files:
- **`MxAccess`** — `ClientName` (must be unique per OtOpcUa instance — redundancy pairs enforce uniqueness at install time), `PublishingIntervalMs` (forwarded as `buffered_update_interval_ms` on subscribe), `WriteUserId` for ArchestrA secured-write, `EventPumpChannelCapacity` (default 50_000 — one second of headroom at 50k tags / 1Hz; tune via the `galaxy.events.dropped` metric). - **`MxAccess`** — `ClientName` (must be unique per OtOpcUa instance — redundancy pairs enforce uniqueness at install time), `PublishingIntervalMs` (forwarded as `buffered_update_interval_ms` on subscribe), `WriteUserId` for ArchestrA secured-write, `EventPumpChannelCapacity` (default 50_000 — one second of headroom at 50k tags / 1Hz; tune via the `galaxy.events.dropped` metric).
- **`Repository`** — `DiscoverPageSize`, `WatchDeployEvents`. - **`Repository`** — `DiscoverPageSize`, `WatchDeployEvents`.
- **`Reconnect`** — `InitialBackoffMs`, `MaxBackoffMs`, `ReplayOnSessionLost` (default `true`: after a gateway session is recreated, re-issue `SubscribeBulk` for every tracked subscription; `false` leaves them stranded — Bad + Degraded, see [Reconnect + Replay](#reconnect--replay)). - **`Reconnect`** — `InitialBackoffMs`, `MaxBackoffMs`, `ReplayOnSessionLost` (default `true`: after a gateway session is recreated, re-issue `SubscribeBulk` for every tracked subscription; `false` leaves them stranded — Bad + Degraded, see [Reconnect + Replay](#reconnect--replay)).
- **`AlarmAck`** (optional, lmxopcua#749) — `FallbackUser`, `FallbackDomain`, `FallbackPasswordSecretRef`. See [Alarm acknowledge credential](#alarm-acknowledge-credential). - **`AlarmAck`** (optional, lmxopcua#749) — `FallbackUser`, `FallbackDomain`, `FallbackPasswordSecretRef`. See [Alarm acknowledge credential](#alarm-acknowledge-credential). `ConfirmTimeoutMs` (default `10000`, max `120000`, `0` = do not wait, negative = default; #869) — see [Ack confirmation](#ack-confirmation-869).
### Alarm acknowledge credential ### Alarm acknowledge credential
@@ -129,8 +129,52 @@ Galaxy password, so `GalaxyAckCredentialResolver` picks the account per acknowle
identity is broken (a user with no password secret), or MXAccess rejecting the account (a non-zero identity is broken (a user with no password secret), or MXAccess rejecting the account (a non-zero
native `hresult`, e.g. a wrong password, now thrown as a refusal): the call returned `Good`, then native `hresult`, e.g. a wrong password, now thrown as a refusal): the call returned `Good`, then
`DriverHostActor` re-projects the condition **unacknowledged** (a fresh condition event, `DriverHostActor` re-projects the condition **unacknowledged** (a fresh condition event,
`Retain=true`) and logs the reason. A driver transition projected after the ack wins over a late `Retain=true`) and logs the reason. A driver transition projected after the ack that states the ack
failure. state wins over a late failure; a Clear only carries it through and leaves it revertible (#869).
### Ack confirmation (#869)
The gateway answers `AcknowledgeAlarm` **OK with `hresult` 0 in two cases**: the `AckMsg` write's
`OnWriteComplete` reported success (*applied* — empty `diagnostic_message`), or the write was sent and no
completion arrived inside the worker's window (*sent but unconfirmed* — `diagnostic_message` says so,
e.g. `AckMsg write to '…' sent; OnWriteComplete not observed within 5000 ms.`). The H6 live gate
(2026-11-11, VD03) caught the second case: OtOpcUa showed the condition Acknowledged while Galaxy's
`.Acked` stayed False and no ACK row was ever historized.
- `GatewayGalaxyAlarmAcknowledger.EvaluateReply` returns **Applied** for an OK reply with no diagnostic and
**Unconfirmed** for one with a diagnostic (any unexplained diagnostic counts — the safe direction), and
logs the gateway diagnostic on every branch that carries one.
- `GalaxyAckConfirmationTracker` registers a watch for the alarm **before** the RPC (the alarm feed can
deliver the confirming transition ahead of the unary reply) and, for an unconfirmed reply, waits up to
`AlarmAck.ConfirmTimeoutMs` for an `Acknowledge` / `ActiveAcknowledged` transition of that alarm on the
alarm feed (live or snapshot re-statement; the reference match is case-insensitive). Whoever acked it,
Galaxy now shows it acknowledged — that is the truth the condition must match.
- The wait is the optional `IAlarmAcknowledgeConfirmation` capability. `DriverInstanceActor` calls it after
`AcknowledgeAsync` returned, **off its mailbox** (a detached task) and **outside the resilience pipeline**
(a wait on an upstream event: it must not be cut by the Tier-A 5 s `AlarmAcknowledge` timeout nor count
toward the breaker). The OPC UA method thread and the node-manager `Lock` are never involved — the method
already returned when the fire-and-forget route started.
- **On timeout** the confirmation throws `DriverOperationRefusedException` (`… not confirm … (BadTimeout;
gateway: <diagnostic>)`), the driver logs a Warning with the diagnostic, and the actor reports
`AlarmAckFailed`: the host re-projects the condition **unacknowledged** — the same #760 revert path.
**What the OPC UA client sees:** the `Acknowledge` call has already returned `Good` (it cannot be changed
after the fact, exactly as for every other #760 revert); the failure reaches the client as a fresh
condition event with `AckedState = false`, and the `BadTimeout` verdict is recorded in the driver and host
Warning logs. A confirming transition that arrives after the revert simply re-acknowledges the condition.
A confirmation in flight when the driver actor stops is abandoned and reverted the same way.
- **Limits to know:**
- A gateway **reconnect** detaches the feed; an ack in flight across it is confirmed only if the new feed's
snapshot re-statement (`ActiveAcknowledged`) arrives inside the window, otherwise it reverts (fails safe).
- Acking an alarm that already **returned to normal** (`UNACK_RTN`) is confirmed by the worker's
`Acknowledge` (`ACK_RTN`) transition; if the gateway reconciles it as a `Clear` instead, nothing confirms
and the correctly-applied ack reverts to unacked. Unverified on hardware — include it in the next live gate.
- The RPC itself still runs under the actor's 5 s ack bound and the Tier-A `AlarmAcknowledge` timeout (5 s).
The gateway worker's write-completion window (`DefaultWriteCompletionTimeout`, 1.5 s) plus `AuthenticateUser` must stay
well under that, or the unconfirmed reply is never seen and the ack reverts as a timeout that also counts
toward the breaker.
- `ConfirmTimeoutMs = 0` restores the pre-#869 behaviour (an unconfirmed ack counts as acknowledged, with a
Warning) — an escape hatch only. Why the gateway-side ack did not stick on VD03 is tracked upstream
(mxaccessgw#226).
The comment written to `AckMsg` is suffixed with the OPC UA identity — The comment written to `AckMsg` is suffixed with the OPC UA identity —
`"<comment> [OPC UA: <principal>, key <keyId>]"` or `"[OPC UA: anonymous]"` — so the client is not `"<comment> [OPC UA: <principal>, key <keyId>]"` or `"[OPC UA: anonymous]"` — so the client is not
@@ -0,0 +1,33 @@
namespace ZB.MOM.WW.OtOpcUa.Core.Abstractions;
/// <summary>
/// Optional companion to <see cref="IAlarmSource"/> for a driver whose upstream can ACCEPT an acknowledge
/// without CONFIRMING it was applied (lmxopcua#869). mxaccessgw answers its <c>AcknowledgeAlarm</c> RPC OK
/// both when the <c>AckMsg</c> write was applied and when it was only sent with no completion observed; the
/// truth arrives later, as the alarm's own acknowledged transition on the alarm stream.
/// </summary>
/// <remarks>
/// <para>
/// The runtime calls <see cref="ConfirmAcknowledgedAsync"/> after <see cref="IAlarmSource.AcknowledgeAsync"/>
/// returned normally, with the SAME request instances, OUTSIDE the resilience pipeline (it is a wait on an
/// upstream event, not I/O, so it must neither be cut by the capability timeout nor count toward the
/// breaker) and OFF the driver actor's mailbox (the wait is seconds long).
/// </para>
/// <para>
/// It completes normally when every request is confirmed, or never needed confirming (the upstream
/// reported it applied). It throws — normally a <see cref="DriverOperationRefusedException"/> naming why —
/// when an acknowledge is not confirmed within the driver's own bounded window; the runtime then reverts
/// the condition to unacknowledged exactly as it does for an <see cref="IAlarmSource.AcknowledgeAsync"/>
/// failure (lmxopcua#760). The window is the driver's to own and bound.
/// </para>
/// </remarks>
public interface IAlarmAcknowledgeConfirmation
{
/// <summary>Waits for the upstream to confirm the acknowledges <see cref="IAlarmSource.AcknowledgeAsync"/> sent.</summary>
/// <param name="acknowledgements">The request instances previously passed to <see cref="IAlarmSource.AcknowledgeAsync"/>.</param>
/// <param name="cancellationToken">Abandons the wait (the caller then treats the acknowledge as unconfirmed).</param>
/// <returns>A task that completes when every acknowledge is confirmed or needed no confirmation.</returns>
Task ConfirmAcknowledgedAsync(
IReadOnlyList<AlarmAcknowledgeRequest> acknowledgements,
CancellationToken cancellationToken);
}
@@ -79,6 +79,28 @@ public sealed record GalaxyAlarmAckOptions(
string? FallbackDomain = null, string? FallbackDomain = null,
string? FallbackPasswordSecretRef = null) string? FallbackPasswordSecretRef = null)
{ {
/// <summary>Default <see cref="ConfirmTimeoutMs"/>.</summary>
public const int DefaultConfirmTimeoutMs = 10_000;
/// <summary>Upper bound for <see cref="ConfirmTimeoutMs"/>.</summary>
public const int MaxConfirmTimeoutMs = 120_000;
/// <summary>
/// How long an acknowledge the gateway SENT but could not confirm may wait for Galaxy's own acknowledged
/// transition on the alarm feed before it is treated as failed and the OPC UA condition is reverted to
/// unacknowledged (lmxopcua#869). Default 10 000 ms; <c>0</c> disables confirmation (an unconfirmed ack
/// counts as success, with a Warning — the pre-#869 behaviour); a negative value means the default, and
/// anything above 120 000 is clamped to it.
/// </summary>
[Display(Name = "Ack confirm timeout (ms)", Description = "Wait for Galaxy to confirm an unconfirmed acknowledge. 0 = do not wait.", GroupName = "Alarms")]
[Range(0, MaxConfirmTimeoutMs)]
public int ConfirmTimeoutMs { get; init; } = DefaultConfirmTimeoutMs;
/// <summary>The confirmation window: negative ⇒ the default, else clamped to <see cref="MaxConfirmTimeoutMs"/>.</summary>
[System.Text.Json.Serialization.JsonIgnore]
public TimeSpan ConfirmTimeout => TimeSpan.FromMilliseconds(
ConfirmTimeoutMs < 0 ? DefaultConfirmTimeoutMs : Math.Min(ConfirmTimeoutMs, MaxConfirmTimeoutMs));
/// <summary>True when a fallback account is configured.</summary> /// <summary>True when a fallback account is configured.</summary>
[System.Text.Json.Serialization.JsonIgnore] [System.Text.Json.Serialization.JsonIgnore]
public bool HasFallback => !string.IsNullOrWhiteSpace(FallbackUser); public bool HasFallback => !string.IsNullOrWhiteSpace(FallbackUser);
@@ -38,7 +38,7 @@ namespace ZB.MOM.WW.OtOpcUa.Driver.Galaxy;
/// </para> /// </para>
/// </remarks> /// </remarks>
public sealed class GalaxyDriver public sealed class GalaxyDriver
: IDriver, ITagDiscovery, IReadable, IWritable, ISubscribable, IRediscoverable, IHostConnectivityProbe, IAlarmSource, IAlarmAcknowledgePreflight, ISkippedTagReporter, IDriverDiagnostics, IDisposable, IAsyncDisposable : IDriver, ITagDiscovery, IReadable, IWritable, ISubscribable, IRediscoverable, IHostConnectivityProbe, IAlarmSource, IAlarmAcknowledgePreflight, IAlarmAcknowledgeConfirmation, ISkippedTagReporter, IDriverDiagnostics, IDisposable, IAsyncDisposable
{ {
// ⚠️ lmxopcua#815 warn-only compatibility (deprecated): this RUNNING driver keeps resolving a pre-#815 // ⚠️ lmxopcua#815 warn-only compatibility (deprecated): this RUNNING driver keeps resolving a pre-#815
// out-of-namespace credential reference (API key on InitializeAsync AND every gateway ReopenAsync, plus the // out-of-namespace credential reference (API key on InitializeAsync AND every gateway ReopenAsync, plus the
@@ -138,6 +138,15 @@ public sealed class GalaxyDriver
// lmxopcua#765: the cached key → has-Galaxy-identity set the synchronous ack pre-check reads (no I/O). // lmxopcua#765: the cached key → has-Galaxy-identity set the synchronous ack pre-check reads (no I/O).
// Null = no directory wired, so a keyed session is never pre-refused (the pre-#765 behaviour). // Null = no directory wired, so a keyed session is never pre-refused (the pre-#765 behaviour).
private readonly IGalaxyAckIdentityDirectory? _ackIdentityDirectory; private readonly IGalaxyAckIdentityDirectory? _ackIdentityDirectory;
// lmxopcua#869: confirms acks the gateway SENT but could not confirm, against Galaxy's own acknowledged
// transition on the alarm feed (bounded by AlarmAck.ConfirmTimeoutMs). Built in the ctor.
private readonly GalaxyAckConfirmationTracker _ackConfirmations;
/// <summary>The ack confirmation window in force (lmxopcua#869) — tests.</summary>
internal TimeSpan AckConfirmationWindow => _ackConfirmations.Window;
/// <summary>Acks still awaiting (or exempt from) confirmation (lmxopcua#869) — tests.</summary>
internal int PendingAckConfirmations => _ackConfirmations.PendingCount;
private IGalaxyAlarmFeed? _alarmFeed; private IGalaxyAlarmFeed? _alarmFeed;
private readonly Lock _alarmHandlersLock = new(); private readonly Lock _alarmHandlersLock = new();
private readonly Lock _alarmFeedLock = new(); private readonly Lock _alarmFeedLock = new();
@@ -377,6 +386,8 @@ public sealed class GalaxyDriver
_alarmFeed = alarmFeed; _alarmFeed = alarmFeed;
_ackCredentialResolver = ackCredentialResolver _ackCredentialResolver = ackCredentialResolver
?? new GalaxyAckCredentialResolver(_driverInstanceId, _options.AlarmAck, _secretResolver, _logger, legacy: _legacyCredentials); ?? new GalaxyAckCredentialResolver(_driverInstanceId, _options.AlarmAck, _secretResolver, _logger, legacy: _legacyCredentials);
_ackConfirmations = new GalaxyAckConfirmationTracker(
_options.AlarmAck?.ConfirmTimeout ?? GalaxyAckConfirmationTracker.DefaultWindow, _logger, _driverInstanceId);
// Build the v3 RawPath ⇄ attributeRef tables from the authored raw tags. Empty when no tags // Build the v3 RawPath ⇄ attributeRef tables from the authored raw tags. Empty when no tags
// are authored (live-browse path), leaving every translation an identity no-op. // are authored (live-browse path), leaving every translation an identity no-op.
@@ -2623,11 +2634,40 @@ public sealed class GalaxyDriver
_driverInstanceId, alarmFullReference, credential.DisplayName, credential.Source, _driverInstanceId, alarmFullReference, credential.DisplayName, credential.Source,
PrincipalOf(ack), ack.ApiKeyId ?? "none"); PrincipalOf(ack), ack.ApiKeyId ?? "none");
await _alarmAcknowledger.AcknowledgeAsync( // lmxopcua#869: the gateway's OK can mean "sent but unconfirmed". Watch for Galaxy's own acknowledged
alarmFullReference, comment, credential, cancellationToken).ConfigureAwait(false); // transition from BEFORE the RPC — the feed can deliver it ahead of the unary reply — and arm the
// watch when the reply is unconfirmed; ConfirmAcknowledgedAsync then awaits it, bounded.
var watch = _ackConfirmations.Begin(ack, alarmFullReference);
var armed = false;
try
{
var outcome = await _alarmAcknowledger.AcknowledgeAsync(
alarmFullReference, comment, credential, cancellationToken).ConfigureAwait(false);
if (!outcome.IsApplied)
{
_ackConfirmations.Arm(watch, outcome.Diagnostic);
armed = true;
}
}
finally
{
if (!armed) _ackConfirmations.Release(watch);
}
} }
} }
/// <inheritdoc />
/// <remarks>
/// lmxopcua#869: completes at once for an ack the gateway reported applied; for one it only SENT, waits up to
/// <c>AlarmAck.ConfirmTimeoutMs</c> (default 10 s) for Galaxy's acknowledged transition on the alarm feed and
/// throws <see cref="DriverOperationRefusedException"/> when it does not come — the host then reverts the
/// condition to unacknowledged. The feed must be running (it is whenever the host holds native conditions,
/// the only way an ack reaches this driver); without it nothing can confirm and the ack reverts.
/// </remarks>
public Task ConfirmAcknowledgedAsync(
IReadOnlyList<AlarmAcknowledgeRequest> acknowledgements, CancellationToken cancellationToken) =>
_ackConfirmations.ConfirmAsync(acknowledgements, cancellationToken);
/// <inheritdoc /> /// <inheritdoc />
/// <remarks> /// <remarks>
/// Only cases decidable without I/O, on a driver with no <c>AlarmAck.FallbackUser</c>: an anonymous session /// Only cases decidable without I/O, on a driver with no <c>AlarmAck.FallbackUser</c>: an anonymous session
@@ -2687,6 +2727,10 @@ public sealed class GalaxyDriver
/// </summary> /// </summary>
private void OnAlarmFeedTransition(object? sender, GalaxyAlarmTransition transition) private void OnAlarmFeedTransition(object? sender, GalaxyAlarmTransition transition)
{ {
// lmxopcua#869: an acknowledged transition confirms any ack the gateway sent unconfirmed — before the
// subscription gate below, which only governs fan-out to OnAlarmEvent.
_ackConfirmations.Observe(transition);
GalaxyAlarmSubscriptionHandle? handle; GalaxyAlarmSubscriptionHandle? handle;
int subCount; int subCount;
lock (_alarmHandlersLock) lock (_alarmHandlersLock)
@@ -114,7 +114,10 @@ public static class GalaxyDriverFactoryExtensions
AlarmAck = new GalaxyAlarmAckOptions( AlarmAck = new GalaxyAlarmAckOptions(
FallbackUser: NullIfBlank(dto.AlarmAck?.FallbackUser), FallbackUser: NullIfBlank(dto.AlarmAck?.FallbackUser),
FallbackDomain: NullIfBlank(dto.AlarmAck?.FallbackDomain), FallbackDomain: NullIfBlank(dto.AlarmAck?.FallbackDomain),
FallbackPasswordSecretRef: NullIfBlank(dto.AlarmAck?.FallbackPasswordSecretRef)), FallbackPasswordSecretRef: NullIfBlank(dto.AlarmAck?.FallbackPasswordSecretRef))
{
ConfirmTimeoutMs = dto.AlarmAck?.ConfirmTimeoutMs ?? GalaxyAlarmAckOptions.DefaultConfirmTimeoutMs,
},
}; };
return new GalaxyDriver( return new GalaxyDriver(
@@ -160,6 +163,8 @@ public static class GalaxyDriverFactoryExtensions
public string? FallbackDomain { get; init; } public string? FallbackDomain { get; init; }
/// <summary>Gets or sets the fallback password reference (secret:/env:/file:/dev:).</summary> /// <summary>Gets or sets the fallback password reference (secret:/env:/file:/dev:).</summary>
public string? FallbackPasswordSecretRef { get; init; } public string? FallbackPasswordSecretRef { get; init; }
/// <summary>Gets or sets how long an unconfirmed ack waits for Galaxy to confirm it (lmxopcua#869); null = default.</summary>
public int? ConfirmTimeoutMs { get; init; }
} }
/// <summary>Gateway configuration section.</summary> /// <summary>Gateway configuration section.</summary>
@@ -0,0 +1,249 @@
using System.Diagnostics;
using Microsoft.Extensions.Logging;
using ZB.MOM.WW.OtOpcUa.Core.Abstractions;
namespace ZB.MOM.WW.OtOpcUa.Driver.Galaxy.Runtime;
/// <summary>
/// Confirms Galaxy acknowledges the gateway sent but could not confirm (lmxopcua#869) against Galaxy's own
/// acknowledged transition on the session-less alarm feed.
/// </summary>
/// <remarks>
/// <para>
/// Lifecycle of one acknowledge: <see cref="Begin"/> registers a watch BEFORE the gateway RPC is sent, so a
/// confirming transition that overtakes the RPC reply is not lost (the feed and the unary reply race). The
/// RPC outcome then either releases the watch (<see cref="Release"/>: applied, or the RPC failed) or arms it
/// (<see cref="Arm"/>: sent but unconfirmed — the confirmation window starts). <see cref="ConfirmAsync"/>
/// awaits the armed watches of the same request instances, outside the resilience pipeline and off the
/// driver actor's mailbox (see <see cref="IAlarmAcknowledgeConfirmation"/>), and throws a
/// <see cref="DriverOperationRefusedException"/> when the window passes without confirmation — the host
/// then reverts the condition to unacknowledged.
/// </para>
/// <para>
/// A confirmation is an <see cref="GalaxyAlarmTransitionKind.Acknowledge"/> or
/// <see cref="GalaxyAlarmTransitionKind.ActiveAcknowledged"/> transition (live or snapshot re-statement) for
/// the same alarm reference — whoever acknowledged it, Galaxy now shows it acknowledged, which is the
/// truth the OPC UA condition must match. Galaxy references are case-insensitive, so the match is too.
/// </para>
/// <para>
/// A watch whose <see cref="ConfirmAsync"/> never comes (an <see cref="IAlarmSource.AcknowledgeAsync"/>
/// caller that does not confirm) is pruned once it is well past its window, so the set stays bounded.
/// A zero window disables confirmation: an unconfirmed ack is counted as success with a Warning — the
/// pre-#869 behaviour, kept only as an operator escape hatch.
/// </para>
/// </remarks>
internal sealed class GalaxyAckConfirmationTracker
{
/// <summary>Default confirmation window — long enough for a gateway alarm-feed round trip, short enough
/// that an operator sees the revert while still looking at the alarm.</summary>
internal static readonly TimeSpan DefaultWindow = TimeSpan.FromSeconds(10);
/// <summary>How long past its window an armed watch nobody confirms is kept before it is pruned.</summary>
private static readonly TimeSpan AbandonedGrace = TimeSpan.FromMinutes(1);
private readonly TimeSpan _window;
private readonly ILogger _logger;
private readonly string _driverInstanceId;
private readonly Lock _lock = new();
private readonly List<Watch> _watches = [];
/// <summary>Initializes a new instance of the <see cref="GalaxyAckConfirmationTracker"/> class.</summary>
/// <param name="window">How long an unconfirmed ack may wait for Galaxy's transition; zero disables confirmation.</param>
/// <param name="logger">Logger for confirmation outcomes.</param>
/// <param name="driverInstanceId">The owning driver, for log context.</param>
public GalaxyAckConfirmationTracker(TimeSpan window, ILogger logger, string driverInstanceId)
{
if (window < TimeSpan.Zero) throw new ArgumentOutOfRangeException(nameof(window), window, "must not be negative");
_window = window;
_logger = logger ?? throw new ArgumentNullException(nameof(logger));
_driverInstanceId = driverInstanceId ?? string.Empty;
}
/// <summary>The confirmation window (zero = disabled).</summary>
public TimeSpan Window => _window;
/// <summary>Number of live watches (tests / diagnostics).</summary>
internal int PendingCount
{
get { lock (_lock) return _watches.Count; }
}
/// <summary>Registers a watch for <paramref name="alarmFullReference"/> before its acknowledge is sent.</summary>
/// <param name="request">The request instance the caller will later pass to <see cref="ConfirmAsync"/>.</param>
/// <param name="alarmFullReference">The Galaxy alarm reference the ack is sent for.</param>
/// <returns>The watch, to be <see cref="Release"/>d or <see cref="Arm"/>ed once the RPC answers.</returns>
public Watch Begin(AlarmAcknowledgeRequest request, string alarmFullReference)
{
ArgumentNullException.ThrowIfNull(request);
ArgumentException.ThrowIfNullOrEmpty(alarmFullReference);
var watch = new Watch(request, alarmFullReference);
lock (_lock)
{
PruneAbandoned();
_watches.Add(watch);
}
return watch;
}
/// <summary>Drops a watch whose acknowledge needs no confirmation (applied) or was never delivered (failed).</summary>
/// <param name="watch">The watch.</param>
public void Release(Watch watch)
{
lock (_lock) _watches.Remove(watch);
}
/// <summary>
/// Starts the confirmation window for an acknowledge the gateway sent but could not confirm. A watch the feed
/// already confirmed while the RPC was in flight is released at once; with confirmation disabled the ack is
/// counted as success (logged).
/// </summary>
/// <param name="watch">The watch.</param>
/// <param name="diagnostic">The gateway's diagnostic, carried into the timeout report.</param>
public void Arm(Watch watch, string diagnostic)
{
watch.Diagnostic = diagnostic ?? string.Empty;
if (_window == TimeSpan.Zero)
{
Release(watch);
_logger.LogWarning(
"Galaxy driver {DriverInstanceId}: acknowledge of {AlarmRef} was not confirmed by the gateway ({Diagnostic}) "
+ "and ack confirmation is disabled (AlarmAck.ConfirmTimeoutMs = 0); counting it as acknowledged.",
_driverInstanceId, watch.AlarmFullReference, watch.Diagnostic);
return;
}
if (watch.IsConfirmed)
{
Release(watch);
LogConfirmed(watch);
return;
}
lock (_lock) watch.ArmedTimestamp = Stopwatch.GetTimestamp();
}
/// <summary>
/// Feed hook: confirms every watch for the transition's alarm when the transition says Galaxy shows it
/// acknowledged. Runs on the feed's dispatch thread — never blocks.
/// </summary>
/// <param name="transition">The alarm transition.</param>
public void Observe(GalaxyAlarmTransition transition)
{
if (transition.TransitionKind is not (GalaxyAlarmTransitionKind.Acknowledge or GalaxyAlarmTransitionKind.ActiveAcknowledged))
return;
if (string.IsNullOrEmpty(transition.AlarmFullReference)) return;
lock (_lock)
{
foreach (var watch in _watches)
{
if (string.Equals(watch.AlarmFullReference, transition.AlarmFullReference, StringComparison.OrdinalIgnoreCase))
watch.Confirm();
}
}
}
/// <summary>
/// Awaits confirmation of every armed watch belonging to <paramref name="requests"/> (matched by instance).
/// Requests with no armed watch (applied, or never sent) need nothing. Throws
/// <see cref="DriverOperationRefusedException"/> when any is not confirmed inside the window.
/// </summary>
/// <param name="requests">The request instances the acknowledge was sent with.</param>
/// <param name="cancellationToken">Abandons the wait.</param>
/// <returns>A task that completes when every armed watch is confirmed.</returns>
public async Task ConfirmAsync(IReadOnlyList<AlarmAcknowledgeRequest> requests, CancellationToken cancellationToken)
{
ArgumentNullException.ThrowIfNull(requests);
List<Watch> mine;
lock (_lock)
{
mine = _watches.Where(w => w.ArmedTimestamp != 0 && requests.Any(r => ReferenceEquals(r, w.Request))).ToList();
}
if (mine.Count == 0) return;
var failures = new List<string>();
try
{
foreach (var watch in mine)
{
var remaining = _window - Stopwatch.GetElapsedTime(watch.ArmedTimestamp);
try
{
if (!watch.IsConfirmed)
{
if (remaining <= TimeSpan.Zero) throw new TimeoutException();
await watch.Confirmed.WaitAsync(remaining, cancellationToken).ConfigureAwait(false);
}
LogConfirmed(watch);
}
catch (TimeoutException)
{
_logger.LogWarning(
"Galaxy driver {DriverInstanceId}: acknowledge of {AlarmRef} was NOT confirmed — Galaxy reported no "
+ "acknowledged transition within {Window} after the gateway sent it unconfirmed (gateway diagnostic: "
+ "{Diagnostic}). Treating it as failed (BadTimeout); the OPC UA condition is reverted to unacknowledged.",
_driverInstanceId, watch.AlarmFullReference, _window, watch.Diagnostic);
failures.Add(
$"Galaxy did not confirm the acknowledge of '{watch.AlarmFullReference}' within "
+ $"{_window.TotalSeconds:0.#} s (BadTimeout; gateway: {watch.Diagnostic})");
}
}
}
finally
{
lock (_lock)
{
foreach (var watch in mine) _watches.Remove(watch);
}
}
if (failures.Count > 0)
throw new DriverOperationRefusedException(string.Join("; ", failures));
}
private void LogConfirmed(Watch watch) =>
_logger.LogInformation(
"Galaxy driver {DriverInstanceId}: acknowledge of {AlarmRef} confirmed by Galaxy's acknowledged transition "
+ "(the gateway had reported it sent but unconfirmed: {Diagnostic}).",
_driverInstanceId, watch.AlarmFullReference, watch.Diagnostic);
// Caller holds _lock.
private void PruneAbandoned()
{
var limit = _window + AbandonedGrace;
_watches.RemoveAll(w => w.ArmedTimestamp != 0 && Stopwatch.GetElapsedTime(w.ArmedTimestamp) > limit);
}
/// <summary>One acknowledge awaiting (or exempt from) confirmation.</summary>
internal sealed class Watch
{
private readonly TaskCompletionSource _confirmed = new(TaskCreationOptions.RunContinuationsAsynchronously);
internal Watch(AlarmAcknowledgeRequest request, string alarmFullReference)
{
Request = request;
AlarmFullReference = alarmFullReference;
}
/// <summary>The request instance this watch belongs to.</summary>
public AlarmAcknowledgeRequest Request { get; }
/// <summary>The Galaxy alarm reference.</summary>
public string AlarmFullReference { get; }
/// <summary>The gateway's diagnostic for the unconfirmed send.</summary>
public string Diagnostic { get; set; } = string.Empty;
/// <summary><see cref="Stopwatch"/> timestamp the window started at; 0 while not armed.</summary>
public long ArmedTimestamp { get; set; }
/// <summary>Completes when Galaxy reports the alarm acknowledged.</summary>
public Task Confirmed => _confirmed.Task;
/// <summary>True once Galaxy reported the alarm acknowledged.</summary>
public bool IsConfirmed => _confirmed.Task.IsCompleted;
/// <summary>Marks the watch confirmed (idempotent).</summary>
public void Confirm() => _confirmed.TrySetResult();
}
}
@@ -34,6 +34,18 @@ namespace ZB.MOM.WW.OtOpcUa.Driver.Galaxy.Runtime;
/// gateway answered — so it neither retries nor counts toward the resilience breaker (a mis-keyed /// gateway answered — so it neither retries nor counts toward the resilience breaker (a mis-keyed
/// account must not open the breaker for every correctly-configured caller). /// account must not open the breaker for every correctly-configured caller).
/// </para> /// </para>
/// <para>
/// <b>An OK reply is not proof the ack was applied (lmxopcua#869).</b> The gateway answers OK with
/// <c>hresult</c> 0 both when the <c>AckMsg</c> write completed and when it was only SENT — its worker saw no
/// <c>OnWriteComplete</c> inside its window — and in the second case it sets <c>diagnostic_message</c>
/// ("AckMsg write to '…' sent; OnWriteComplete not observed within N ms."). An applied ack carries an empty
/// diagnostic. So an OK reply with a non-empty diagnostic is returned as
/// <see cref="GalaxyAckOutcome.Unconfirmed"/> and the driver waits for Galaxy's own acknowledged transition
/// (<see cref="GalaxyAckConfirmationTracker"/>). Any unexplained diagnostic on an OK reply is treated the
/// same way — the safe direction: at worst an applied ack waits for its (arriving) transition. The H6 live
/// gate (2026-11-11) caught an unconfirmed ack being reported Acknowledged while Galaxy never applied it.
/// The diagnostic is logged on every branch that carries one.
/// </para>
/// </remarks> /// </remarks>
internal sealed class GatewayGalaxyAlarmAcknowledger : IGalaxyAlarmAcknowledger internal sealed class GatewayGalaxyAlarmAcknowledger : IGalaxyAlarmAcknowledger
{ {
@@ -53,7 +65,7 @@ internal sealed class GatewayGalaxyAlarmAcknowledger : IGalaxyAlarmAcknowledger
} }
/// <inheritdoc /> /// <inheritdoc />
public async Task AcknowledgeAsync( public async Task<GalaxyAckOutcome> AcknowledgeAsync(
string alarmFullReference, string alarmFullReference,
string comment, string comment,
GalaxyAckCredential credential, GalaxyAckCredential credential,
@@ -77,7 +89,7 @@ internal sealed class GatewayGalaxyAlarmAcknowledger : IGalaxyAlarmAcknowledger
+ "details withheld because they echoed the operator credential."); + "details withheld because they echoed the operator credential.");
} }
EvaluateReply(reply, alarmFullReference, credential, _logger); return EvaluateReply(reply, alarmFullReference, credential, _logger);
} }
/// <summary>Maps an acknowledge onto the gateway request. Pure — pinned by unit tests.</summary> /// <summary>Maps an acknowledge onto the gateway request. Pure — pinned by unit tests.</summary>
@@ -104,23 +116,32 @@ internal sealed class GatewayGalaxyAlarmAcknowledger : IGalaxyAlarmAcknowledger
/// <summary> /// <summary>
/// Applies the reply semantics: non-OK protocol status throws; a non-zero native code warns and throws a /// Applies the reply semantics: non-OK protocol status throws; a non-zero native code warns and throws a
/// <see cref="DriverOperationRefusedException"/> (lmxopcua#760). /// <see cref="DriverOperationRefusedException"/> (lmxopcua#760); an OK reply is
/// Every message is scrubbed of the password. /// <see cref="GalaxyAckOutcome.Applied"/> when it carries no diagnostic and
/// <see cref="GalaxyAckOutcome.Unconfirmed"/> when it does (lmxopcua#869). The gateway's diagnostic is
/// logged on every branch that carries one. Every message is scrubbed of the password.
/// </summary> /// </summary>
/// <param name="reply">The gateway reply.</param> /// <param name="reply">The gateway reply.</param>
/// <param name="alarmFullReference">The alarm reference.</param> /// <param name="alarmFullReference">The alarm reference.</param>
/// <param name="credential">The credential the request carried.</param> /// <param name="credential">The credential the request carried.</param>
/// <param name="logger">The logger.</param> /// <param name="logger">The logger.</param>
internal static void EvaluateReply( /// <returns>Whether the gateway confirmed the ack was applied.</returns>
internal static GalaxyAckOutcome EvaluateReply(
AcknowledgeAlarmReply reply, string alarmFullReference, GalaxyAckCredential credential, ILogger logger) AcknowledgeAlarmReply reply, string alarmFullReference, GalaxyAckCredential credential, ILogger logger)
{ {
var diagnostic = Scrub(reply.DiagnosticMessage, credential.Password).Trim();
// Protocol status — the gateway failed before MXAccess saw the ack. This is a // Protocol status — the gateway failed before MXAccess saw the ack. This is a
// hard failure: the operator's request was not delivered at all. // hard failure: the operator's request was not delivered at all.
if (reply.ProtocolStatus is { } proto && proto.Code != ProtocolStatusCode.Ok) if (reply.ProtocolStatus is { } proto && proto.Code != ProtocolStatusCode.Ok)
{ {
var protoMessage = Scrub(proto.Message, credential.Password);
logger.LogWarning(
"Galaxy AcknowledgeAlarm for {AlarmRef} as {GalaxyUser} failed at the gateway: {ProtocolCode} {ProtocolMessage} (gateway diagnostic: {Diagnostic})",
alarmFullReference, credential.DisplayName, proto.Code, protoMessage, DiagnosticText(diagnostic));
throw new InvalidOperationException( throw new InvalidOperationException(
$"Galaxy AcknowledgeAlarm for '{alarmFullReference}' as '{credential.DisplayName}' failed at the gateway: " $"Galaxy AcknowledgeAlarm for '{alarmFullReference}' as '{credential.DisplayName}' failed at the gateway: "
+ $"{proto.Code} {Scrub(proto.Message, credential.Password)}"); + $"{proto.Code} {protoMessage}");
} }
// hresult is the authoritative native ack return code (0 = success) — the AuthenticateUser HRESULT // hresult is the authoritative native ack return code (0 = success) — the AuthenticateUser HRESULT
@@ -129,14 +150,29 @@ internal sealed class GatewayGalaxyAlarmAcknowledger : IGalaxyAlarmAcknowledger
if (reply.HasHresult && reply.Hresult != 0) if (reply.HasHresult && reply.Hresult != 0)
{ {
logger.LogWarning( logger.LogWarning(
"Galaxy AcknowledgeAlarm for {AlarmRef} as {GalaxyUser} returned native ack failure code {Hresult}.", "Galaxy AcknowledgeAlarm for {AlarmRef} as {GalaxyUser} returned native ack failure code {Hresult} (gateway diagnostic: {Diagnostic}).",
alarmFullReference, credential.DisplayName, reply.Hresult); alarmFullReference, credential.DisplayName, reply.Hresult, DiagnosticText(diagnostic));
throw new DriverOperationRefusedException( throw new DriverOperationRefusedException(
$"MXAccess rejected the Galaxy acknowledge of '{alarmFullReference}' as '{credential.DisplayName}' " $"MXAccess rejected the Galaxy acknowledge of '{alarmFullReference}' as '{credential.DisplayName}' "
+ $"(native code 0x{reply.Hresult:X8}); check the account's password and Galaxy acknowledge permission."); + $"(native code 0x{reply.Hresult:X8}); check the account's password and Galaxy acknowledge permission.");
} }
// lmxopcua#869: OK + hresult 0 means "applied OR sent-but-unconfirmed"; only the unconfirmed case carries a
// diagnostic. Not a failure yet — the driver waits for Galaxy's own acknowledged transition.
if (diagnostic.Length > 0)
{
logger.LogInformation(
"Galaxy AcknowledgeAlarm for {AlarmRef} as {GalaxyUser} was sent but not confirmed by the gateway "
+ "(gateway diagnostic: {Diagnostic}); awaiting Galaxy's acknowledged transition.",
alarmFullReference, credential.DisplayName, diagnostic);
return GalaxyAckOutcome.Unconfirmed(diagnostic);
}
return GalaxyAckOutcome.Applied;
} }
private static string DiagnosticText(string diagnostic) => diagnostic.Length > 0 ? diagnostic : "none";
/// <summary>Replaces every occurrence of <paramref name="secret"/> in <paramref name="text"/>.</summary> /// <summary>Replaces every occurrence of <paramref name="secret"/> in <paramref name="text"/>.</summary>
/// <param name="text">Text that may echo the secret.</param> /// <param name="text">Text that may echo the secret.</param>
/// <param name="secret">The secret; empty means nothing to scrub.</param> /// <param name="secret">The secret; empty means nothing to scrub.</param>
@@ -22,10 +22,32 @@ internal interface IGalaxyAlarmAcknowledger
/// Carries a password — implementations must keep it out of logs and exception text. /// Carries a password — implementations must keep it out of logs and exception text.
/// </param> /// </param>
/// <param name="cancellationToken">Cancels the gateway RPC.</param> /// <param name="cancellationToken">Cancels the gateway RPC.</param>
/// <returns>A task that represents the asynchronous operation.</returns> /// <returns>
Task AcknowledgeAsync( /// Whether the gateway reported the ack APPLIED or only SENT without confirmation (lmxopcua#869). A refusal
/// or failure throws instead.
/// </returns>
Task<GalaxyAckOutcome> AcknowledgeAsync(
string alarmFullReference, string alarmFullReference,
string comment, string comment,
GalaxyAckCredential credential, GalaxyAckCredential credential,
CancellationToken cancellationToken); CancellationToken cancellationToken);
} }
/// <summary>
/// What the gateway said about an acknowledge it accepted (lmxopcua#869). mxaccessgw answers OK both when the
/// <c>AckMsg</c> write's <c>OnWriteComplete</c> reported success (<see cref="Applied"/>) and when the write was
/// sent but no completion arrived inside the worker's window (<see cref="Unconfirmed"/>) — its contract tells
/// clients to watch the alarm stream for the real acknowledged transition in that case.
/// </summary>
/// <param name="IsApplied">True when the gateway confirmed the ack write was applied.</param>
/// <param name="Diagnostic">The gateway's diagnostic message (password-scrubbed); empty when it sent none.</param>
internal sealed record GalaxyAckOutcome(bool IsApplied, string Diagnostic)
{
/// <summary>The gateway confirmed the ack write.</summary>
public static GalaxyAckOutcome Applied { get; } = new(true, string.Empty);
/// <summary>The gateway sent the ack write but could not confirm it.</summary>
/// <param name="diagnostic">The gateway's diagnostic message.</param>
/// <returns>The outcome.</returns>
public static GalaxyAckOutcome Unconfirmed(string diagnostic) => new(false, diagnostic ?? string.Empty);
}
@@ -109,6 +109,11 @@
placeholder="secret:otopcua/galaxy/ack-fallback-password" /> placeholder="secret:otopcua/galaxy/ack-fallback-password" />
<div class="form-text">Forms: <code>secret:galaxy/NAME</code> or <code>secret:otopcua/galaxy/NAME</code> (production), <code>env:OTOPCUA_GALAXY_NAME</code>, <code>dev:VALUE</code>. A bare literal, <code>file:</code>, or a reference outside the Galaxy namespace is refused (#815).</div> <div class="form-text">Forms: <code>secret:galaxy/NAME</code> or <code>secret:otopcua/galaxy/NAME</code> (production), <code>env:OTOPCUA_GALAXY_NAME</code>, <code>dev:VALUE</code>. A bare literal, <code>file:</code>, or a reference outside the Galaxy namespace is refused (#815).</div>
</div> </div>
<div class="col-md-3">
<label class="form-label">Ack confirm timeout (ms)</label>
<InputNumber @bind-Value="_form.Galaxy.AckConfirmTimeoutMs" @bind-Value:after="EmitAsync" min="0" max="120000" class="form-control form-control-sm" />
<div class="form-text">When the gateway sends an ack but cannot confirm it, wait this long for Galaxy's acknowledged transition before reverting the condition to unacknowledged (#869). Default 10000; 0 = do not wait.</div>
</div>
</div> </div>
</div> </div>
</section> </section>
@@ -270,6 +275,7 @@
public string? AckFallbackUser { get; set; } public string? AckFallbackUser { get; set; }
public string? AckFallbackDomain { get; set; } public string? AckFallbackDomain { get; set; }
public string? AckFallbackPasswordSecretRef { get; set; } public string? AckFallbackPasswordSecretRef { get; set; }
public int AckConfirmTimeoutMs { get; set; } = GalaxyAlarmAckOptions.DefaultConfirmTimeoutMs;
public static GalaxyFormModel FromRecord(GalaxyDriverOptions r) public static GalaxyFormModel FromRecord(GalaxyDriverOptions r)
{ {
@@ -300,6 +306,7 @@
AckFallbackUser = ack.FallbackUser, AckFallbackUser = ack.FallbackUser,
AckFallbackDomain = ack.FallbackDomain, AckFallbackDomain = ack.FallbackDomain,
AckFallbackPasswordSecretRef = ack.FallbackPasswordSecretRef, AckFallbackPasswordSecretRef = ack.FallbackPasswordSecretRef,
AckConfirmTimeoutMs = ack.ConfirmTimeoutMs,
}; };
} }
@@ -329,7 +336,10 @@
AlarmAck = new GalaxyAlarmAckOptions( AlarmAck = new GalaxyAlarmAckOptions(
FallbackUser: string.IsNullOrWhiteSpace(AckFallbackUser) ? null : AckFallbackUser.Trim(), FallbackUser: string.IsNullOrWhiteSpace(AckFallbackUser) ? null : AckFallbackUser.Trim(),
FallbackDomain: string.IsNullOrWhiteSpace(AckFallbackDomain) ? null : AckFallbackDomain.Trim(), FallbackDomain: string.IsNullOrWhiteSpace(AckFallbackDomain) ? null : AckFallbackDomain.Trim(),
FallbackPasswordSecretRef: string.IsNullOrWhiteSpace(AckFallbackPasswordSecretRef) ? null : AckFallbackPasswordSecretRef.Trim()), FallbackPasswordSecretRef: string.IsNullOrWhiteSpace(AckFallbackPasswordSecretRef) ? null : AckFallbackPasswordSecretRef.Trim())
{
ConfirmTimeoutMs = AckConfirmTimeoutMs,
},
}; };
} }
} }
@@ -2413,7 +2413,7 @@ public sealed class DriverHostActor : ReceiveActor, IWithTimers, IWithUnboundedS
// projector too, or the next driver Clear projects (Active=false, Acked=prev=false) and un-acknowledges it. // projector too, or the next driver Clear projects (Active=false, Acked=prev=false) and un-acknowledges it.
// Most sources (MTConnect, FOCAS) send no Acknowledge transition back, and the projector now survives a // Most sources (MTConnect, FOCAS) send no Acknowledge transition back, and the projector now survives a
// deploy rather than resetting to its acked default. // deploy rather than resetting to its acked default.
_nativeAlarmProjector.MarkAcknowledged(msg.ConditionNodeId); var ackSequence = _nativeAlarmProjector.MarkAcknowledged(msg.ConditionNodeId);
if (!_children.TryGetValue(target.DriverInstanceId, out var entry) || entry.Stubbed) if (!_children.TryGetValue(target.DriverInstanceId, out var entry) || entry.Stubbed)
{ {
@@ -2424,7 +2424,7 @@ public sealed class DriverHostActor : ReceiveActor, IWithTimers, IWithUnboundedS
var reason = entry is null ? "the owning driver is not running" : "the owning driver is stubbed on this node"; var reason = entry is null ? "the owning driver is not running" : "the owning driver is stubbed on this node";
_log.Debug("DriverHost {Node}: driver {Driver} cannot take the alarm ack of {Cond} ({Reason}) — dropped", _log.Debug("DriverHost {Node}: driver {Driver} cannot take the alarm ack of {Cond} ({Reason}) — dropped",
_localNode, target.DriverInstanceId, msg.ConditionNodeId, reason); _localNode, target.DriverInstanceId, msg.ConditionNodeId, reason);
RevertNativeAlarmAck(msg.ConditionNodeId, reason); RevertNativeAlarmAck(msg.ConditionNodeId, reason, ackSequence);
return; return;
} }
@@ -2433,7 +2433,8 @@ public sealed class DriverHostActor : ReceiveActor, IWithTimers, IWithUnboundedS
// carried here) and reverts the node (HandleAlarmAckFailed). The driver correlates on ConditionId // carried here) and reverts the node (HandleAlarmAckFailed). The driver correlates on ConditionId
// (= the authored alarm FullName the inverse map keyed on). // (= the authored alarm FullName the inverse map keyed on).
entry.Actor.Tell(new DriverInstanceActor.RouteAlarmAck( entry.Actor.Tell(new DriverInstanceActor.RouteAlarmAck(
target.RawPath, msg.Comment, msg.OperatorUser, msg.ApiKeyId, ConditionNodeId: msg.ConditionNodeId)); target.RawPath, msg.Comment, msg.OperatorUser, msg.ApiKeyId, ConditionNodeId: msg.ConditionNodeId,
AckSequence: ackSequence));
} }
/// <summary> /// <summary>
@@ -2444,7 +2445,7 @@ public sealed class DriverHostActor : ReceiveActor, IWithTimers, IWithUnboundedS
/// </summary> /// </summary>
/// <param name="msg">The failure report from the driver child.</param> /// <param name="msg">The failure report from the driver child.</param>
private void HandleAlarmAckFailed(DriverInstanceActor.AlarmAckFailed msg) => private void HandleAlarmAckFailed(DriverInstanceActor.AlarmAckFailed msg) =>
RevertNativeAlarmAck(msg.ConditionNodeId, msg.Reason); RevertNativeAlarmAck(msg.ConditionNodeId, msg.Reason, msg.AckSequence);
/// <summary> /// <summary>
/// Re-projects <paramref name="conditionNodeId"/> as unacknowledged when its local acknowledge is still /// Re-projects <paramref name="conditionNodeId"/> as unacknowledged when its local acknowledge is still
@@ -2453,12 +2454,15 @@ public sealed class DriverHostActor : ReceiveActor, IWithTimers, IWithUnboundedS
/// </summary> /// </summary>
/// <param name="conditionNodeId">The bare condition NodeId (Raw realm).</param> /// <param name="conditionNodeId">The bare condition NodeId (Raw realm).</param>
/// <param name="reason">Why the acknowledge did not reach the upstream alarm system.</param> /// <param name="reason">Why the acknowledge did not reach the upstream alarm system.</param>
private void RevertNativeAlarmAck(string conditionNodeId, string reason) /// <param name="ackSequence">The failed acknowledge's projector sequence (lmxopcua#869).</param>
private void RevertNativeAlarmAck(string conditionNodeId, string reason, long ackSequence)
{ {
if (!_nativeAlarmProjector.TryRevertAcknowledge(conditionNodeId, out var snapshot) || snapshot is null) // Keyed to the failed ack's own sequence (lmxopcua#869): a pending ack from an earlier or later acknowledge of
// the same condition is not this failure's to undo.
if (!_nativeAlarmProjector.TryRevertAcknowledge(conditionNodeId, ackSequence, out var snapshot) || snapshot is null)
{ {
_log.Info("DriverHost {Node}: native-alarm acknowledge of {Cond} did not reach the alarm system ({Reason}); " + _log.Info("DriverHost {Node}: native-alarm acknowledge of {Cond} did not reach the alarm system ({Reason}); " +
"the condition has since been re-stated by its driver, so it is left as projected", "no acknowledge of THIS request is pending (re-stated by its driver since, superseded by another ack, or it changed nothing), so it is left as projected",
_localNode, conditionNodeId, reason); _localNode, conditionNodeId, reason);
return; return;
} }
@@ -90,8 +90,11 @@ public sealed class DriverInstanceActor : ReceiveActor, IWithTimers
/// acknowledge that does not reach the upstream alarm system (refused, failed, or dropped because the driver /// acknowledge that does not reach the upstream alarm system (refused, failed, or dropped because the driver
/// is not connected) is reported back to the sender as an <see cref="AlarmAckFailed"/> so the host can revert /// is not connected) is reported back to the sender as an <see cref="AlarmAckFailed"/> so the host can revert
/// the locally-applied acknowledge (lmxopcua#760). Null ⇒ no report.</param> /// the locally-applied acknowledge (lmxopcua#760). Null ⇒ no report.</param>
/// <param name="AckSequence">The host projector's sequence for this acknowledge (lmxopcua#869), echoed on
/// <see cref="AlarmAckFailed"/> so only a failure of THIS ack reverts it; 0 ⇒ the ack changed nothing.</param>
public sealed record RouteAlarmAck( public sealed record RouteAlarmAck(
string ConditionId, string? Comment, string OperatorUser, string? ApiKeyId = null, string? ConditionNodeId = null); string ConditionId, string? Comment, string OperatorUser, string? ApiKeyId = null, string? ConditionNodeId = null,
long AckSequence = 0);
/// <summary> /// <summary>
/// Reply to a <see cref="RouteAlarmAck"/> carrying a <see cref="RouteAlarmAck.ConditionNodeId"/> whose /// Reply to a <see cref="RouteAlarmAck"/> carrying a <see cref="RouteAlarmAck.ConditionNodeId"/> whose
@@ -99,7 +102,8 @@ public sealed class DriverInstanceActor : ReceiveActor, IWithTimers
/// </summary> /// </summary>
/// <param name="ConditionNodeId">The OPC UA condition NodeId the client acknowledged.</param> /// <param name="ConditionNodeId">The OPC UA condition NodeId the client acknowledged.</param>
/// <param name="Reason">Why the acknowledge did not go through; never contains a secret.</param> /// <param name="Reason">Why the acknowledge did not go through; never contains a secret.</param>
public sealed record AlarmAckFailed(string ConditionNodeId, string Reason); /// <param name="AckSequence">The failed acknowledge's <see cref="RouteAlarmAck.AckSequence"/> (lmxopcua#869).</param>
public sealed record AlarmAckFailed(string ConditionNodeId, string Reason, long AckSequence = 0);
public sealed record Subscribe(IReadOnlyList<string> FullReferences, TimeSpan PublishingInterval); public sealed record Subscribe(IReadOnlyList<string> FullReferences, TimeSpan PublishingInterval);
/// <summary> /// <summary>
/// Sets the set of references this driver should keep subscribed for the lifetime of the /// Sets the set of references this driver should keep subscribed for the lifetime of the
@@ -365,6 +369,10 @@ public sealed class DriverInstanceActor : ReceiveActor, IWithTimers
/// shutdown rather than let it race the teardown on a restart.</summary> /// shutdown rather than let it race the teardown on a restart.</summary>
private CancellationTokenSource? _initCts; private CancellationTokenSource? _initCts;
/// <summary>Cancelled by <see cref="PostStop"/> to abandon in-flight native-ack confirmations (lmxopcua#869);
/// never disposed — detached confirmation tasks may still hold its token.</summary>
private readonly CancellationTokenSource _ackConfirmCts = new();
/// <summary> /// <summary>
/// Timestamps of recent Faulted-state transitions; used to compute the 5-minute error count. /// Timestamps of recent Faulted-state transitions; used to compute the 5-minute error count.
/// No lock needed — every read/write site runs inside an Akka message handler, which is /// No lock needed — every read/write site runs inside an Akka message handler, which is
@@ -1367,6 +1375,21 @@ public sealed class DriverInstanceActor : ReceiveActor, IWithTimers
cts.Token); cts.Token);
_log.Info("DriverInstance {Id}: acknowledged native condition {Cond} by {User}", _log.Info("DriverInstance {Id}: acknowledged native condition {Cond} by {User}",
_driverInstanceId, msg.ConditionId, msg.OperatorUser); _driverInstanceId, msg.ConditionId, msg.OperatorUser);
// lmxopcua#869: an upstream may accept an ack without confirming it was applied (Galaxy: the gateway's
// "sent but unconfirmed" OK). The confirmation wait is seconds long, so it runs OFF this mailbox and
// OUTSIDE the resilience pipeline (a wait on an upstream event — it must neither be cut by the
// AlarmAcknowledge capability timeout nor count toward the breaker). Failure reverts like any other.
// Task.Run, not a bare call: this handler is a ReceiveAsync, so TaskScheduler.Current is the actor's
// scheduler; the confirmation must not depend on this actor's context at all — in particular not after
// PostStop, when it still has to report the abandoned ack (pinned by DriverInstanceActorAckConfirmationTests).
if (src is IAlarmAcknowledgeConfirmation confirmer)
{
var stopToken = _ackConfirmCts.Token;
var driverInstanceId = _driverInstanceId;
var log = _log;
_ = Task.Run(() => RunAckConfirmationAsync(confirmer, request, msg, replyTo, stopToken, driverInstanceId, log));
}
} }
catch (DriverOperationRefusedException ex) catch (DriverOperationRefusedException ex)
{ {
@@ -1385,12 +1408,54 @@ public sealed class DriverInstanceActor : ReceiveActor, IWithTimers
} }
} }
/// <summary>
/// Backstop over a driver's own confirmation window (lmxopcua#869): the driver bounds the wait itself; this
/// only guarantees a driver that ignores its window cannot hold a confirmation forever.
/// </summary>
internal static readonly TimeSpan AckConfirmationCeiling = TimeSpan.FromMinutes(3);
/// <summary>
/// Awaits <see cref="IAlarmAcknowledgeConfirmation.ConfirmAcknowledgedAsync"/> off the actor context and, when
/// the upstream does not confirm, reports <see cref="AlarmAckFailed"/> so the host reverts the condition to
/// unacknowledged — the same report an <see cref="IAlarmSource.AcknowledgeAsync"/> failure makes. Reads no actor
/// state: everything it needs is captured by the caller on the actor thread.
/// </summary>
private static async Task RunAckConfirmationAsync(
IAlarmAcknowledgeConfirmation confirmer, IReadOnlyList<AlarmAcknowledgeRequest> request, RouteAlarmAck msg,
IActorRef replyTo, CancellationToken stopToken, string driverInstanceId, ILoggingAdapter log)
{
try
{
await confirmer.ConfirmAcknowledgedAsync(request, stopToken)
.WaitAsync(AckConfirmationCeiling, stopToken).ConfigureAwait(false);
}
catch (DriverOperationRefusedException ex)
{
// The driver's verdict (e.g. Galaxy: no acknowledged transition inside its window) — it names why.
log.Warning("DriverInstance {Id}: native-alarm acknowledge of {Cond} by {User} was not confirmed upstream — {Reason}",
driverInstanceId, msg.ConditionId, msg.OperatorUser, ex.Message);
ReportAckFailed(replyTo, msg, ex.Message);
}
catch (OperationCanceledException) when (stopToken.IsCancellationRequested)
{
log.Info("DriverInstance {Id}: confirmation of the native-alarm acknowledge of {Cond} abandoned — the driver stopped",
driverInstanceId, msg.ConditionId);
ReportAckFailed(replyTo, msg, "the driver stopped before the upstream confirmed the acknowledge");
}
catch (Exception ex)
{
log.Warning(ex, "DriverInstance {Id}: confirmation of the native-alarm acknowledge of {Cond} failed",
driverInstanceId, msg.ConditionId);
ReportAckFailed(replyTo, msg, $"the upstream acknowledge was not confirmed ({ex.GetType().Name})");
}
}
/// <summary>Tells <paramref name="replyTo"/> an <see cref="AlarmAckFailed"/> when the ack names a condition /// <summary>Tells <paramref name="replyTo"/> an <see cref="AlarmAckFailed"/> when the ack names a condition
/// NodeId (lmxopcua#760); a no-op for an ack routed without one.</summary> /// NodeId (lmxopcua#760); a no-op for an ack routed without one.</summary>
private static void ReportAckFailed(IActorRef replyTo, RouteAlarmAck msg, string reason) private static void ReportAckFailed(IActorRef replyTo, RouteAlarmAck msg, string reason)
{ {
if (string.IsNullOrEmpty(msg.ConditionNodeId) || replyTo.IsNobody()) return; if (string.IsNullOrEmpty(msg.ConditionNodeId) || replyTo.IsNobody()) return;
replyTo.Tell(new AlarmAckFailed(msg.ConditionNodeId, reason)); replyTo.Tell(new AlarmAckFailed(msg.ConditionNodeId, reason, msg.AckSequence));
} }
private async Task HandleSubscribeAsync(Subscribe msg) private async Task HandleSubscribeAsync(Subscribe msg)
@@ -2004,6 +2069,10 @@ public sealed class DriverInstanceActor : ReceiveActor, IWithTimers
try { _initCts?.Cancel(); } try { _initCts?.Cancel(); }
catch (ObjectDisposedException) { /* the attempt already finished */ } catch (ObjectDisposedException) { /* the attempt already finished */ }
// In-flight native-ack confirmations (lmxopcua#869) are abandoned, and each reports its ack unconfirmed so
// the host reverts it — this incarnation can no longer observe the upstream's confirmation.
_ackConfirmCts.Cancel();
// BOUNDED by ONE shared budget across shutdown + dispose. PostStop runs on a dispatcher thread and this // BOUNDED by ONE shared budget across shutdown + dispose. PostStop runs on a dispatcher thread and this
// blocks it, so an unbounded wait on a driver whose shutdown hangs against a dead socket / COM proxy pinned // blocks it, so an unbounded wait on a driver whose shutdown hangs against a dead socket / COM proxy pinned
// that thread for the whole wait, and a few such stops starved the dispatcher. A shutdown that overruns is // that thread for the whole wait, and a few such stops starved the dispatcher. A shutdown that overruns is
@@ -18,9 +18,13 @@ public sealed class NativeAlarmProjector
/// <summary> /// <summary>
/// Conditions a client acknowledged locally (<see cref="MarkAcknowledged"/>) whose acknowledge has not yet /// Conditions a client acknowledged locally (<see cref="MarkAcknowledged"/>) whose acknowledge has not yet
/// been confirmed or superseded by a driver transition — the only ones <see cref="TryRevertAcknowledge"/> /// been confirmed or superseded by a driver transition — the only ones <see cref="TryRevertAcknowledge"/>
/// may un-acknowledge (lmxopcua#760). /// may un-acknowledge (lmxopcua#760) — keyed to the sequence <see cref="MarkAcknowledged"/> issued, so only a
/// failure of THAT acknowledge reverts it (lmxopcua#869).
/// </summary> /// </summary>
private readonly HashSet<string> _pendingLocalAck = new(StringComparer.Ordinal); private readonly Dictionary<string, long> _pendingLocalAck = new(StringComparer.Ordinal);
/// <summary>Last sequence issued by <see cref="MarkAcknowledged"/>; 0 is never issued.</summary>
private long _ackSequence;
/// <summary>Project an alarm transition onto the full condition snapshot for <paramref name="nodeId"/>.</summary> /// <summary>Project an alarm transition onto the full condition snapshot for <paramref name="nodeId"/>.</summary>
/// <param name="nodeId">The materialised condition node's id (the projection's state key).</param> /// <param name="nodeId">The materialised condition node's id (the projection's state key).</param>
@@ -78,9 +82,6 @@ public sealed class NativeAlarmProjector
/// <returns>The full Part 9 condition snapshot to write to the node.</returns> /// <returns>The full Part 9 condition snapshot to write to the node.</returns>
public AlarmConditionSnapshot Project(string nodeId, AlarmEventArgs e, out IReadOnlyList<AlarmTransitionKind> historyKinds) public AlarmConditionSnapshot Project(string nodeId, AlarmEventArgs e, out IReadOnlyList<AlarmTransitionKind> historyKinds)
{ {
// Any driver transition supersedes a pending local acknowledge: from here the projected state is the
// driver's word, so a late ack failure must not rewrite it.
_pendingLocalAck.Remove(nodeId);
var prev = _prior.TryGetValue(nodeId, out var p) var prev = _prior.TryGetValue(nodeId, out var p)
? p : (Active: false, Acked: true, Message: (string?)null, Severity: (ushort)0, StatusCode: OpcUaStatus.Good, Class: (AlarmConditionClass?)null); ? p : (Active: false, Acked: true, Message: (string?)null, Severity: (ushort)0, StatusCode: OpcUaStatus.Good, Class: (AlarmConditionClass?)null);
// A source with a finer scale than the four buckets supplies the exact Part 9 severity. // A source with a finer scale than the four buckets supplies the exact Part 9 severity.
@@ -153,6 +154,15 @@ public sealed class NativeAlarmProjector
break; break;
} }
// A driver transition that STATES the ack (Acknowledge / ActiveAcknowledged, or an un-acking Raise /
// Retrigger / InactiveUnacknowledged) supersedes a pending local acknowledge: from here the ack state is the
// driver's word, so a late ack failure must not rewrite it. One that merely CARRIES the prior ack through
// (a Clear, a same-message Raise, a message/severity update) says nothing about it — the optimistic local
// ack rode along, so it stays pending and a later failure still reverts it (lmxopcua#869: an operator acks,
// the value returns to normal while the unconfirmed ack is still being confirmed, Galaxy shows UNACK_RTN).
if (acked != prev.Acked || e.Kind is AlarmTransitionKind.Acknowledge or AlarmTransitionKind.ActiveAcknowledged)
_pendingLocalAck.Remove(nodeId);
var statusCode = e.QualityStatusCode ?? OpcUaStatus.Good; var statusCode = e.QualityStatusCode ?? OpcUaStatus.Good;
// A source that classifies its conditions (#839) states the class on its transitions; one that does not (or a // A source that classifies its conditions (#839) states the class on its transitions; one that does not (or a
// quality-only annotation) keeps whatever the condition last carried. // quality-only annotation) keeps whatever the condition last carried.
@@ -174,27 +184,60 @@ public sealed class NativeAlarmProjector
/// stale unacknowledged prior. An untracked condition already defaults to acknowledged. /// stale unacknowledged prior. An untracked condition already defaults to acknowledged.
/// </summary> /// </summary>
/// <param name="nodeId">The acknowledged condition node id.</param> /// <param name="nodeId">The acknowledged condition node id.</param>
public void MarkAcknowledged(string nodeId) /// <returns>
/// The sequence identifying this pending acknowledge — pass it back to
/// <see cref="TryRevertAcknowledge(string, long, out AlarmConditionSnapshot?)"/> with its failure — or 0 when the
/// ack changed nothing (untracked, or already acknowledged), so its failure has nothing to undo.
/// </returns>
public long MarkAcknowledged(string nodeId)
{ {
if (!_prior.TryGetValue(nodeId, out var p)) return; if (!_prior.TryGetValue(nodeId, out var p)) return 0;
// Only an ack that CHANGED the projected state is pending; re-acking an acked condition has nothing to undo. // Only an ack that CHANGED the projected state is pending; re-acking an acked condition has nothing to undo.
if (!p.Acked) _pendingLocalAck.Add(nodeId); long sequence = 0;
if (!p.Acked)
{
sequence = ++_ackSequence;
_pendingLocalAck[nodeId] = sequence;
}
_prior[nodeId] = p with { Acked = true }; _prior[nodeId] = p with { Acked = true };
return sequence;
} }
/// <summary> /// <summary>
/// Undo a client acknowledgement the driver refused or failed to deliver upstream (lmxopcua#760), returning /// Undo a client acknowledgement the driver refused or failed to deliver upstream (lmxopcua#760), returning
/// the unacknowledged snapshot to re-project onto the node. Only an acknowledge still pending since /// the unacknowledged snapshot to re-project onto the node. Only an acknowledge still pending since
/// <see cref="MarkAcknowledged"/> is reverted: once any driver transition for the condition has been /// <see cref="MarkAcknowledged"/> is reverted: once a driver transition that STATES the condition's ack state
/// projected, that transition is authoritative and a late failure changes nothing. /// has been projected, that transition is authoritative and a late failure changes nothing. A transition that
/// only carries the prior ack through (e.g. a Clear) leaves it pending (lmxopcua#869).
/// </summary> /// </summary>
/// <param name="nodeId">The condition node id whose acknowledge failed.</param> /// <param name="nodeId">The condition node id whose acknowledge failed.</param>
/// <param name="snapshot">The unacknowledged snapshot to project; null when nothing was reverted.</param> /// <param name="snapshot">The unacknowledged snapshot to project; null when nothing was reverted.</param>
/// <returns>True when the condition was reverted to unacknowledged.</returns> /// <returns>True when the condition was reverted to unacknowledged.</returns>
public bool TryRevertAcknowledge(string nodeId, out AlarmConditionSnapshot? snapshot) public bool TryRevertAcknowledge(string nodeId, out AlarmConditionSnapshot? snapshot) =>
TryRevert(nodeId, ackSequence: null, out snapshot);
/// <summary>
/// Like <see cref="TryRevertAcknowledge(string, out AlarmConditionSnapshot?)"/>, but only when the pending
/// acknowledge is the one <see cref="MarkAcknowledged"/> issued <paramref name="ackSequence"/> for
/// (lmxopcua#869). A pending ack can outlive its own delivery — a source that never sends an Acknowledge
/// transition back (FOCAS, MTConnect), across a Clear that only carried it through — so a failure of a LATER,
/// unrelated acknowledge (a re-ack while the driver is down) or of an EARLIER one (a Galaxy confirmation timing
/// out after the alarm re-occurred and was acked again) must not revert it.
/// </summary>
/// <param name="nodeId">The condition node id whose acknowledge failed.</param>
/// <param name="ackSequence">The sequence of the failed acknowledge; 0 never matches.</param>
/// <param name="snapshot">The unacknowledged snapshot to project; null when nothing was reverted.</param>
/// <returns>True when the condition was reverted to unacknowledged.</returns>
public bool TryRevertAcknowledge(string nodeId, long ackSequence, out AlarmConditionSnapshot? snapshot) =>
TryRevert(nodeId, ackSequence, out snapshot);
private bool TryRevert(string nodeId, long? ackSequence, out AlarmConditionSnapshot? snapshot)
{ {
snapshot = null; snapshot = null;
if (!_pendingLocalAck.Remove(nodeId) || !_prior.TryGetValue(nodeId, out var p)) return false; if (!_pendingLocalAck.TryGetValue(nodeId, out var pending)
|| (ackSequence is { } wanted && wanted != pending)
|| !_prior.TryGetValue(nodeId, out var p)) return false;
_pendingLocalAck.Remove(nodeId);
_prior[nodeId] = p with { Acked = false }; _prior[nodeId] = p with { Acked = false };
snapshot = new AlarmConditionSnapshot( snapshot = new AlarmConditionSnapshot(
Active: p.Active, Acknowledged: false, Confirmed: true, Enabled: true, Active: p.Active, Acknowledged: false, Confirmed: true, Enabled: true,
@@ -317,10 +317,10 @@ public sealed class GalaxyDriverAlarmSourceTests
/// <param name="credential">The resolved Galaxy credential.</param> /// <param name="credential">The resolved Galaxy credential.</param>
/// <param name="cancellationToken">Cancellation token.</param> /// <param name="cancellationToken">Cancellation token.</param>
/// <returns>A completed task.</returns> /// <returns>A completed task.</returns>
public Task AcknowledgeAsync(string alarmFullReference, string comment, GalaxyAckCredential credential, CancellationToken cancellationToken) public Task<GalaxyAckOutcome> AcknowledgeAsync(string alarmFullReference, string comment, GalaxyAckCredential credential, CancellationToken cancellationToken)
{ {
Calls.Add((alarmFullReference, comment, credential.User)); Calls.Add((alarmFullReference, comment, credential.User));
return Task.CompletedTask; return Task.FromResult(GalaxyAckOutcome.Applied);
} }
} }
@@ -0,0 +1,408 @@
using Microsoft.Extensions.Logging;
using Shouldly;
using Xunit;
using ZB.MOM.WW.MxGateway.Contracts.Proto;
using ZB.MOM.WW.OtOpcUa.Core.Abstractions;
using ZB.MOM.WW.OtOpcUa.Driver.Galaxy.Config;
using ZB.MOM.WW.OtOpcUa.Driver.Galaxy.Runtime;
namespace ZB.MOM.WW.OtOpcUa.Driver.Galaxy.Tests.Runtime;
/// <summary>
/// lmxopcua#869 — mxaccessgw answers <c>AcknowledgeAlarm</c> OK both when the <c>AckMsg</c> write was APPLIED and
/// when it was only SENT (no <c>OnWriteComplete</c> inside the worker's window; the reply then carries a
/// diagnostic). The H6 live gate saw an unconfirmed ack reported Acknowledged while Galaxy never applied it.
/// The driver now treats an unconfirmed OK as pending and confirms it against Galaxy's own acknowledged
/// transition on the alarm feed, bounded by <c>AlarmAck.ConfirmTimeoutMs</c>.
/// </summary>
/// <remarks>
/// The "fake gateway" is a scripted <see cref="AcknowledgeAlarmReply"/> run through the production
/// <see cref="GatewayGalaxyAlarmAcknowledger.EvaluateReply"/> — the gRPC transport seam is internal to the
/// client package, so the reply mapping is the part under test.
/// </remarks>
public sealed class GalaxyAckConfirmationTests
{
private const string AlarmRef = "Tank01.Level.HiHi";
private const string UnconfirmedDiagnostic =
"AckMsg write to 'Tank01.Level.HiHi.AckMsg' sent; OnWriteComplete not observed within 5000 ms.";
private static readonly GalaxyAckCredential Alice =
new("alice", "CORP", "alice-pw", GalaxyAckCredentialSource.ApiKey);
// ------------------------------------------------------------------ reply mapping
[Fact]
public void EvaluateReply_ok_without_diagnostic_is_applied()
{
var logger = new CapturingLogger();
var outcome = GatewayGalaxyAlarmAcknowledger.EvaluateReply(AppliedReply(), AlarmRef, Alice, logger);
outcome.IsApplied.ShouldBeTrue();
logger.Entries.ShouldBeEmpty();
}
/// <summary>The H6 reply: OK + hresult 0 + a "sent; not observed" diagnostic is UNCONFIRMED, and the
/// diagnostic is logged. Before #869 the same reply returned silently as success.</summary>
[Fact]
public void EvaluateReply_ok_with_diagnostic_is_unconfirmed_and_logs_the_diagnostic()
{
var logger = new CapturingLogger();
var outcome = GatewayGalaxyAlarmAcknowledger.EvaluateReply(UnconfirmedReply(), AlarmRef, Alice, logger);
outcome.IsApplied.ShouldBeFalse();
outcome.Diagnostic.ShouldBe(UnconfirmedDiagnostic);
logger.AllText.ShouldContain(UnconfirmedDiagnostic);
}
[Fact]
public void EvaluateReply_native_refusal_logs_the_diagnostic_and_refuses()
{
var reply = new AcknowledgeAlarmReply
{
ProtocolStatus = new ProtocolStatus { Code = ProtocolStatusCode.Ok },
Hresult = 1008,
DiagnosticMessage = "AckMsg write to 'Tank01.Level.HiHi.AckMsg' rejected: category SecurityError, detail 1008.",
};
var logger = new CapturingLogger();
Should.Throw<DriverOperationRefusedException>(
() => GatewayGalaxyAlarmAcknowledger.EvaluateReply(reply, AlarmRef, Alice, logger));
logger.AllText.ShouldContain("rejected: category SecurityError");
}
[Fact]
public void EvaluateReply_scrubs_the_password_from_the_diagnostic()
{
var reply = new AcknowledgeAlarmReply
{
ProtocolStatus = new ProtocolStatus { Code = ProtocolStatusCode.Ok },
Hresult = 0,
DiagnosticMessage = "sent as alice/alice-pw; not observed",
};
var logger = new CapturingLogger();
var outcome = GatewayGalaxyAlarmAcknowledger.EvaluateReply(reply, AlarmRef, Alice, logger);
outcome.Diagnostic.ShouldNotContain("alice-pw");
logger.AllText.ShouldNotContain("alice-pw");
}
// ------------------------------------------------------------------ driver: confirmation
/// <summary>Applied OK → acknowledged: nothing pending, the confirmation completes at once.</summary>
[Fact]
public async Task Applied_ack_is_confirmed_at_once()
{
var feed = new FakeAlarmFeed();
var gateway = new ScriptedGateway(AppliedReply());
using var driver = NewDriver(feed, gateway, confirmTimeoutMs: 30_000);
await driver.SubscribeAlarmsAsync([AlarmRef], CancellationToken.None);
var request = Ack(AlarmRef);
await driver.AcknowledgeAsync(request, CancellationToken.None);
var confirm = driver.ConfirmAcknowledgedAsync(request, CancellationToken.None);
confirm.IsCompletedSuccessfully.ShouldBeTrue("an applied ack needs no confirmation");
gateway.Calls.ShouldBe(1);
}
/// <summary>
/// Unconfirmed OK + Galaxy's acknowledged transition on the feed → acknowledged, end to end through the
/// driver's feed hook (non-matching transitions are emitted first; that they do NOT confirm is pinned
/// synchronously by <see cref="Tracker_confirms_only_on_this_alarms_acknowledged_transition"/>).
/// </summary>
[Fact]
public async Task Unconfirmed_ack_is_confirmed_by_the_acknowledged_transition()
{
var feed = new FakeAlarmFeed();
var gateway = new ScriptedGateway(UnconfirmedReply());
var logger = new CapturingLogger();
using var driver = NewDriver(feed, gateway, confirmTimeoutMs: 120_000, logger);
await driver.SubscribeAlarmsAsync([AlarmRef], CancellationToken.None);
var request = Ack(AlarmRef);
await driver.AcknowledgeAsync(request, CancellationToken.None);
var confirm = driver.ConfirmAcknowledgedAsync(request, CancellationToken.None);
feed.Emit(Transition("Other01.Level.Hi", GalaxyAlarmTransitionKind.Acknowledge));
feed.Emit(Transition(AlarmRef, GalaxyAlarmTransitionKind.Raise));
feed.Emit(Transition(AlarmRef.ToUpperInvariant(), GalaxyAlarmTransitionKind.Acknowledge));
await TestWait.ForAsync(confirm, "the acknowledged transition must confirm the pending ack");
logger.AllText.ShouldContain(UnconfirmedDiagnostic);
}
/// <summary>
/// The feed and the unary reply race: Galaxy's acknowledged transition can arrive BEFORE the RPC answers
/// "unconfirmed". The watch is registered before the RPC, so that early transition still confirms — with a
/// 1 ms window, a lost transition would time out instead.
/// </summary>
[Fact]
public async Task Acknowledged_transition_that_overtakes_the_reply_still_confirms()
{
var feed = new FakeAlarmFeed();
var gateway = new ScriptedGateway(UnconfirmedReply())
{
DuringCall = () => feed.Emit(Transition(AlarmRef, GalaxyAlarmTransitionKind.Acknowledge)),
};
using var driver = NewDriver(feed, gateway, confirmTimeoutMs: 1);
await driver.SubscribeAlarmsAsync([AlarmRef], CancellationToken.None);
var request = Ack(AlarmRef);
await driver.AcknowledgeAsync(request, CancellationToken.None);
await driver.ConfirmAcknowledgedAsync(request, CancellationToken.None);
}
/// <summary>
/// Unconfirmed OK + no acknowledged transition → the confirmation fails with a refusal naming the timeout and
/// the gateway diagnostic (the runtime reverts the condition on it), and a Warning carrying the diagnostic is
/// logged. Transitions that must not confirm (another alarm's ack, this alarm's retrigger) are emitted during
/// the wait. Before #869 this ack counted as acknowledged.
/// </summary>
[Fact]
public async Task Unconfirmed_ack_without_confirmation_fails_with_the_diagnostic()
{
var feed = new FakeAlarmFeed();
var gateway = new ScriptedGateway(UnconfirmedReply());
var logger = new CapturingLogger();
using var driver = NewDriver(feed, gateway, confirmTimeoutMs: 300, logger);
await driver.SubscribeAlarmsAsync([AlarmRef], CancellationToken.None);
var request = Ack(AlarmRef);
await driver.AcknowledgeAsync(request, CancellationToken.None);
var confirm = driver.ConfirmAcknowledgedAsync(request, CancellationToken.None);
feed.Emit(Transition("Other01.Level.Hi", GalaxyAlarmTransitionKind.Acknowledge));
feed.Emit(Transition(AlarmRef, GalaxyAlarmTransitionKind.Retrigger));
var ex = await Should.ThrowAsync<DriverOperationRefusedException>(() => confirm);
ex.Message.ShouldContain(AlarmRef);
ex.Message.ShouldContain("BadTimeout");
ex.Message.ShouldContain(UnconfirmedDiagnostic);
logger.Entries.ShouldContain(e => e.Level == LogLevel.Warning && e.Text.Contains(UnconfirmedDiagnostic));
}
/// <summary>A refused ack (non-zero native code) throws from AcknowledgeAsync — the existing revert path —
/// and leaves nothing pending to confirm.</summary>
[Fact]
public async Task Refused_ack_throws_and_leaves_nothing_pending()
{
var feed = new FakeAlarmFeed();
var gateway = new ScriptedGateway(new AcknowledgeAlarmReply
{
ProtocolStatus = new ProtocolStatus { Code = ProtocolStatusCode.Ok },
Hresult = unchecked((int)0x80070005),
});
using var driver = NewDriver(feed, gateway, confirmTimeoutMs: 30_000);
await driver.SubscribeAlarmsAsync([AlarmRef], CancellationToken.None);
var request = Ack(AlarmRef);
await Should.ThrowAsync<DriverOperationRefusedException>(
() => driver.AcknowledgeAsync(request, CancellationToken.None));
driver.ConfirmAcknowledgedAsync(request, CancellationToken.None).IsCompletedSuccessfully.ShouldBeTrue();
driver.PendingAckConfirmations.ShouldBe(0);
}
/// <summary><c>ConfirmTimeoutMs = 0</c> is the opt-out: an unconfirmed ack counts as acknowledged, with a Warning.</summary>
[Fact]
public async Task Zero_timeout_counts_an_unconfirmed_ack_as_acknowledged_with_a_warning()
{
var feed = new FakeAlarmFeed();
var gateway = new ScriptedGateway(UnconfirmedReply());
var logger = new CapturingLogger();
using var driver = NewDriver(feed, gateway, confirmTimeoutMs: 0, logger);
var request = Ack(AlarmRef);
await driver.AcknowledgeAsync(request, CancellationToken.None);
driver.ConfirmAcknowledgedAsync(request, CancellationToken.None).IsCompletedSuccessfully.ShouldBeTrue();
logger.Entries.ShouldContain(e => e.Level == LogLevel.Warning && e.Text.Contains("confirmation is disabled"));
driver.PendingAckConfirmations.ShouldBe(0);
}
/// <summary>Each confirmation is for ITS request instance: confirming one leaves no watch behind, and a later
/// ack of the same alarm is confirmed independently.</summary>
[Fact]
public async Task Confirmations_release_their_watches()
{
var feed = new FakeAlarmFeed();
var gateway = new ScriptedGateway(UnconfirmedReply());
using var driver = NewDriver(feed, gateway, confirmTimeoutMs: 120_000);
await driver.SubscribeAlarmsAsync([AlarmRef], CancellationToken.None);
for (var i = 0; i < 2; i++)
{
var request = Ack(AlarmRef);
await driver.AcknowledgeAsync(request, CancellationToken.None);
driver.PendingAckConfirmations.ShouldBe(1);
var confirm = driver.ConfirmAcknowledgedAsync(request, CancellationToken.None);
feed.Emit(Transition(AlarmRef, GalaxyAlarmTransitionKind.ActiveAcknowledged));
await TestWait.ForAsync(confirm, $"ack #{i} must be confirmed");
driver.PendingAckConfirmations.ShouldBe(0);
}
}
/// <summary>
/// Tracker matching rules, checked synchronously: only an Acknowledge / ActiveAcknowledged transition of the
/// SAME alarm (case-insensitive) confirms. Positive control: the matching transition does flip it.
/// </summary>
[Fact]
public void Tracker_confirms_only_on_this_alarms_acknowledged_transition()
{
var tracker = new GalaxyAckConfirmationTracker(TimeSpan.FromSeconds(30), new CapturingLogger(), "drv");
var watch = tracker.Begin(Ack(AlarmRef)[0], AlarmRef);
tracker.Arm(watch, UnconfirmedDiagnostic);
foreach (var kind in new[]
{
GalaxyAlarmTransitionKind.Raise, GalaxyAlarmTransitionKind.Clear,
GalaxyAlarmTransitionKind.Retrigger, GalaxyAlarmTransitionKind.Unspecified,
})
{
tracker.Observe(Transition(AlarmRef, kind));
}
tracker.Observe(Transition("Other01.Level.Hi", GalaxyAlarmTransitionKind.Acknowledge));
tracker.Observe(Transition("Other01.Level.Hi", GalaxyAlarmTransitionKind.ActiveAcknowledged));
watch.IsConfirmed.ShouldBeFalse("no transition so far says THIS alarm is acknowledged");
tracker.Observe(Transition(AlarmRef.ToLowerInvariant(), GalaxyAlarmTransitionKind.Acknowledge));
watch.IsConfirmed.ShouldBeTrue();
}
[Fact]
public void Options_default_to_ten_seconds_and_map_negative_to_the_default()
{
new GalaxyAlarmAckOptions().ConfirmTimeout.ShouldBe(TimeSpan.FromSeconds(10));
new GalaxyAlarmAckOptions { ConfirmTimeoutMs = -5 }.ConfirmTimeout.ShouldBe(TimeSpan.FromSeconds(10));
new GalaxyAlarmAckOptions { ConfirmTimeoutMs = 0 }.ConfirmTimeout.ShouldBe(TimeSpan.Zero);
new GalaxyAlarmAckOptions { ConfirmTimeoutMs = 999_999 }.ConfirmTimeout.ShouldBe(TimeSpan.FromMinutes(2));
}
[Fact]
public void Factory_binds_ConfirmTimeoutMs_and_defaults_when_absent()
{
const string Json = """
{ "gateway": { "endpoint": "https://gw:5001", "apiKeySecretRef": "dev:k" },
"mxAccess": { "clientName": "C" }, "alarmAck": { "confirmTimeoutMs": 2500 } }
""";
using var withKnob = GalaxyDriverFactoryExtensions.CreateInstance("g", Json);
withKnob.AckConfirmationWindow.ShouldBe(TimeSpan.FromMilliseconds(2500));
using var without = GalaxyDriverFactoryExtensions.CreateInstance(
"g", """{ "gateway": { "endpoint": "https://gw:5001", "apiKeySecretRef": "dev:k" }, "mxAccess": { "clientName": "C" } }""");
without.AckConfirmationWindow.ShouldBe(TimeSpan.FromSeconds(10));
}
// ------------------------------------------------------------------ helpers
private static AcknowledgeAlarmReply AppliedReply() => new()
{
ProtocolStatus = new ProtocolStatus { Code = ProtocolStatusCode.Ok },
Hresult = 0,
};
private static AcknowledgeAlarmReply UnconfirmedReply() => new()
{
ProtocolStatus = new ProtocolStatus { Code = ProtocolStatusCode.Ok },
Hresult = 0,
DiagnosticMessage = UnconfirmedDiagnostic,
};
private static AlarmAcknowledgeRequest[] Ack(string conditionId) =>
[new AlarmAcknowledgeRequest(SourceNodeId: "Tank01", ConditionId: conditionId, Comment: "c")];
private static GalaxyDriver NewDriver(
IGalaxyAlarmFeed feed, IGalaxyAlarmAcknowledger acknowledger, int confirmTimeoutMs, CapturingLogger? logger = null)
{
var options = new GalaxyDriverOptions(
new GalaxyGatewayOptions("http://localhost:5000", "literal-api-key"),
new GalaxyMxAccessOptions("AckConfirmTest"),
new GalaxyRepositoryOptions(),
new GalaxyReconnectOptions())
{
AlarmAck = new GalaxyAlarmAckOptions { ConfirmTimeoutMs = confirmTimeoutMs },
};
return new GalaxyDriver(
driverInstanceId: "drv-ack",
options: options,
hierarchySource: null,
alarmAcknowledger: acknowledger,
alarmFeed: feed,
logger: logger is null ? null : new TypedLogger<GalaxyDriver>(logger),
ackCredentialResolver: FixedAckCredentialResolver.Fallback());
}
private static GalaxyAlarmTransition Transition(string alarmRef, GalaxyAlarmTransitionKind kind) => new(
AlarmFullReference: alarmRef,
SourceObjectReference: "Tank01",
AlarmTypeName: "AnalogLimitAlarm.HiHi",
TransitionKind: kind,
SeverityBucket: AlarmSeverity.High,
OpcUaSeverity: 700,
RawMxAccessSeverity: 500,
OriginalRaiseTimestampUtc: null,
TransitionTimestampUtc: DateTime.UtcNow,
OperatorUser: string.Empty,
OperatorComment: string.Empty,
Category: "Process",
Description: "Tank 01 high-high level");
/// <summary>
/// A fake gateway: every call answers with <paramref name="reply"/>, mapped by the production
/// <see cref="GatewayGalaxyAlarmAcknowledger.EvaluateReply"/>. <see cref="DuringCall"/> runs while the RPC is
/// "in flight" (before the reply is evaluated).
/// </summary>
private sealed class ScriptedGateway(AcknowledgeAlarmReply reply) : IGalaxyAlarmAcknowledger
{
private int _calls;
public Action? DuringCall { get; init; }
public int Calls => Volatile.Read(ref _calls);
public Task<GalaxyAckOutcome> AcknowledgeAsync(
string alarmFullReference, string comment, GalaxyAckCredential credential, CancellationToken cancellationToken)
{
Interlocked.Increment(ref _calls);
DuringCall?.Invoke();
try
{
return Task.FromResult(GatewayGalaxyAlarmAcknowledger.EvaluateReply(
reply, alarmFullReference, credential, Microsoft.Extensions.Logging.Abstractions.NullLogger.Instance));
}
catch (Exception ex)
{
return Task.FromException<GalaxyAckOutcome>(ex);
}
}
}
private sealed class FakeAlarmFeed : IGalaxyAlarmFeed
{
public event EventHandler<GalaxyAlarmTransition>? OnAlarmTransition;
public void Start() { }
public void Emit(GalaxyAlarmTransition transition) => OnAlarmTransition?.Invoke(this, transition);
public ValueTask DisposeAsync() => ValueTask.CompletedTask;
}
private sealed class TypedLogger<T>(CapturingLogger inner) : ILogger<T>
{
private readonly Lock _gate = new();
public IDisposable? BeginScope<TState>(TState state) where TState : notnull => null;
public bool IsEnabled(LogLevel logLevel) => true;
public void Log<TState>(LogLevel l, EventId id, TState s, Exception? e, Func<TState, Exception?, string> f)
{
lock (_gate) inner.Log(l, id, s, e, f);
}
}
}
@@ -485,11 +485,11 @@ public sealed class GalaxyAckCredentialTests
{ {
public List<(string AlarmRef, string Comment, GalaxyAckCredential Credential)> Calls { get; } = []; public List<(string AlarmRef, string Comment, GalaxyAckCredential Credential)> Calls { get; } = [];
public Task AcknowledgeAsync( public Task<GalaxyAckOutcome> AcknowledgeAsync(
string alarmFullReference, string comment, GalaxyAckCredential credential, CancellationToken cancellationToken) string alarmFullReference, string comment, GalaxyAckCredential credential, CancellationToken cancellationToken)
{ {
Calls.Add((alarmFullReference, comment, credential)); Calls.Add((alarmFullReference, comment, credential));
return Task.CompletedTask; return Task.FromResult(GalaxyAckOutcome.Applied);
} }
} }
@@ -242,10 +242,10 @@ public sealed class GalaxyDriverRawPathKeyingTests
{ {
public List<(string AlarmRef, string Comment, string Operator)> Calls { get; } = []; public List<(string AlarmRef, string Comment, string Operator)> Calls { get; } = [];
public Task AcknowledgeAsync(string alarmFullReference, string comment, GalaxyAckCredential credential, CancellationToken cancellationToken) public Task<GalaxyAckOutcome> AcknowledgeAsync(string alarmFullReference, string comment, GalaxyAckCredential credential, CancellationToken cancellationToken)
{ {
Calls.Add((alarmFullReference, comment, credential.User)); Calls.Add((alarmFullReference, comment, credential.User));
return Task.CompletedTask; return Task.FromResult(GalaxyAckOutcome.Applied);
} }
} }
@@ -34,22 +34,22 @@ public sealed class GatewayGalaxyAlarmAcknowledgerTests
/// </summary> /// </summary>
private sealed class FakeAcknowledger : IGalaxyAlarmAcknowledger private sealed class FakeAcknowledger : IGalaxyAlarmAcknowledger
{ {
private readonly Queue<Func<Task>> _outcomes = new(); private readonly Queue<Func<Task<GalaxyAckOutcome>>> _outcomes = new();
/// <summary>Calls recorded in the order received.</summary> /// <summary>Calls recorded in the order received.</summary>
public List<(string AlarmRef, string Comment, string Operator)> Calls { get; } = []; public List<(string AlarmRef, string Comment, string Operator)> Calls { get; } = [];
/// <summary>Enqueues an outcome for the next call (completes normally or throws).</summary> /// <summary>Enqueues an outcome for the next call (completes normally or throws).</summary>
public void Enqueue(Func<Task> outcome) => _outcomes.Enqueue(outcome); public void Enqueue(Func<Task<GalaxyAckOutcome>> outcome) => _outcomes.Enqueue(outcome);
public Task AcknowledgeAsync( public Task<GalaxyAckOutcome> AcknowledgeAsync(
string alarmFullReference, string alarmFullReference,
string comment, string comment,
GalaxyAckCredential credential, GalaxyAckCredential credential,
CancellationToken cancellationToken) CancellationToken cancellationToken)
{ {
Calls.Add((alarmFullReference, comment, credential.User)); Calls.Add((alarmFullReference, comment, credential.User));
return _outcomes.Count > 0 ? _outcomes.Dequeue()() : Task.CompletedTask; return _outcomes.Count > 0 ? _outcomes.Dequeue()() : Task.FromResult(GalaxyAckOutcome.Applied);
} }
} }
@@ -77,9 +77,9 @@ public sealed class GatewayGalaxyAlarmAcknowledgerTests
{ {
var fake = new FakeAcknowledger(); var fake = new FakeAcknowledger();
// First call: simulates a transient transport blip (e.g. RpcException from gRPC). // First call: simulates a transient transport blip (e.g. RpcException from gRPC).
fake.Enqueue(() => Task.FromException(new IOException("synthetic transport fault"))); fake.Enqueue(() => Task.FromException<GalaxyAckOutcome>(new IOException("synthetic transport fault")));
// Second call: succeeds — proves no dead-state latch. // Second call: succeeds — proves no dead-state latch.
fake.Enqueue(() => Task.CompletedTask); fake.Enqueue(() => Task.FromResult(GalaxyAckOutcome.Applied));
var driver = new GalaxyDriver("g", Opts(), var driver = new GalaxyDriver("g", Opts(),
hierarchySource: null, alarmAcknowledger: fake, hierarchySource: null, alarmAcknowledger: fake,
@@ -109,8 +109,8 @@ public sealed class GatewayGalaxyAlarmAcknowledgerTests
public async Task Acknowledge_after_protocol_failure_still_issues_next_call() public async Task Acknowledge_after_protocol_failure_still_issues_next_call()
{ {
var fake = new FakeAcknowledger(); var fake = new FakeAcknowledger();
fake.Enqueue(() => Task.FromException(new InvalidOperationException("gateway overload"))); fake.Enqueue(() => Task.FromException<GalaxyAckOutcome>(new InvalidOperationException("gateway overload")));
fake.Enqueue(() => Task.CompletedTask); fake.Enqueue(() => Task.FromResult(GalaxyAckOutcome.Applied));
var driver = new GalaxyDriver("g", Opts(), var driver = new GalaxyDriver("g", Opts(),
hierarchySource: null, alarmAcknowledger: fake, hierarchySource: null, alarmAcknowledger: fake,
@@ -259,4 +259,30 @@ public sealed class GalaxyDriverPageFormSerializationTests
json.ShouldNotContain("hasFallback"); json.ShouldNotContain("hasFallback");
JsonSerializer.Deserialize<GalaxyDriverOptions>(json, _pageOpts)!.AlarmAck.ShouldBe(original.AlarmAck); JsonSerializer.Deserialize<GalaxyDriverOptions>(json, _pageOpts)!.AlarmAck.ShouldBe(original.AlarmAck);
} }
/// <summary>
/// lmxopcua#869: the ack-confirmation window must survive FromRecord/ToRecord and JSON, or saving the page
/// would silently reset an operator's setting (e.g. the 0 = do-not-wait opt-out) to the default.
/// </summary>
[Fact]
public void FormModel_RoundTrips_AlarmAckConfirmTimeout()
{
var original = new GalaxyDriverOptions(
new GalaxyGatewayOptions("https://gw:5001", "env:K"),
new GalaxyMxAccessOptions("OtOpcUa"),
new GalaxyRepositoryOptions(),
new GalaxyReconnectOptions())
{
AlarmAck = new GalaxyAlarmAckOptions { ConfirmTimeoutMs = 0 },
};
var back = GalaxyDriverForm.GalaxyFormModel.FromRecord(original).ToRecord();
back.AlarmAck.ConfirmTimeoutMs.ShouldBe(0);
var json = JsonSerializer.Serialize(back, _pageOpts);
json.ShouldContain("\"confirmTimeoutMs\":0");
json.ShouldNotContain("\"confirmTimeout\":");
JsonSerializer.Deserialize<GalaxyDriverOptions>(json, _pageOpts)!.AlarmAck.ConfirmTimeoutMs.ShouldBe(0);
new GalaxyDriverForm.GalaxyFormModel().AckConfirmTimeoutMs.ShouldBe(GalaxyAlarmAckOptions.DefaultConfirmTimeoutMs);
}
} }
@@ -217,11 +217,105 @@ public sealed class DriverHostActorNativeAlarmAckRoutingTests : RuntimeActorTest
RaiseAndExpectUnacked(actor, publish); RaiseAndExpectUnacked(actor, publish);
actor.Tell(new DriverHostActor.RouteNativeAlarmAck(AlarmRawPath, "cmt", "alice")); actor.Tell(new DriverHostActor.RouteNativeAlarmAck(AlarmRawPath, "cmt", "alice"));
AwaitAssert(() => recorder.Acks.Count.ShouldBe(1), duration: Timeout, cancellationToken: TestContext.Current.CancellationToken); AwaitAssert(() => recorder.Acks.Count.ShouldBe(1), duration: Timeout, cancellationToken: TestContext.Current.CancellationToken);
// lmxopcua#869: the driver is an IAlarmAcknowledgeConfirmation whose confirmation succeeds at once — it was
// asked, and a confirmed ack must not revert.
AwaitAssert(() => recorder.ConfirmCalls.ShouldBe(1), duration: Timeout, cancellationToken: TestContext.Current.CancellationToken);
actor.Tell(new DriverInstanceActor.AttributeAlarmPublished("drv-1", Evt(AlarmTransitionKind.Clear))); actor.Tell(new DriverInstanceActor.AttributeAlarmPublished("drv-1", Evt(AlarmTransitionKind.Clear)));
var next = NextAlarmUpdate(publish); var next = NextAlarmUpdate(publish);
next.State.Active.ShouldBeFalse(); next.State.Active.ShouldBeFalse();
next.State.Acknowledged.ShouldBeTrue("the successful ack must have survived — no revert"); next.State.Acknowledged.ShouldBeTrue("the successful ack must have survived — no revert");
// A Clear carries the ack through and so leaves a still-pending ack revertible (#869): a wrong revert landing
// after it would be the NEXT update. A carry-through update must therefore still show it acknowledged.
actor.Tell(new DriverInstanceActor.AttributeAlarmPublished("drv-1", Evt(AlarmTransitionKind.Unspecified)));
NextAlarmUpdate(publish).State.Acknowledged.ShouldBeTrue("no late revert may follow a confirmed ack");
}
/// <summary>
/// lmxopcua#869 — the common sequence: an operator acks, the alarm returns to normal (a Clear, carrying the
/// optimistic ack) while the unconfirmed ack is still being confirmed, then the confirmation fails. The ack
/// must still revert — to inactive + UNacknowledged, Galaxy's UNACK_RTN — not stay hidden behind the Clear.
/// </summary>
[Fact]
public void An_unconfirmed_ack_reverts_even_after_a_clear_inside_the_window()
{
var db = NewInMemoryDbFactory();
var gate = new TaskCompletionSource();
var recorder = new RecordingAlarmDriverFactory("GalaxyMxGateway") { Confirmation = () => gate.Task };
var deploymentId = SeedV3AlarmDeployment(db, RevA, Driver: "drv-1", Tag: "temp_hi");
var publish = this.CreateCleanOutcomePublishProbe();
var actor = SpawnHostAndApply(db, deploymentId, recorder, publish.Ref);
RaiseAndExpectUnacked(actor, publish);
actor.Tell(new DriverHostActor.RouteNativeAlarmAck(AlarmRawPath, "cmt", "alice"));
AwaitAssert(() => recorder.ConfirmCalls.ShouldBe(1), duration: Timeout, cancellationToken: TestContext.Current.CancellationToken);
actor.Tell(new DriverInstanceActor.AttributeAlarmPublished("drv-1", Evt(AlarmTransitionKind.Clear)));
var cleared = NextAlarmUpdate(publish);
cleared.State.Active.ShouldBeFalse();
cleared.State.Acknowledged.ShouldBeTrue("the optimistic ack rides the Clear");
gate.SetException(new DriverOperationRefusedException("Galaxy did not confirm the acknowledge (BadTimeout)"));
var reverted = NextAlarmUpdate(publish);
reverted.State.Active.ShouldBeFalse();
reverted.State.Acknowledged.ShouldBeFalse("the unconfirmed ack must not survive behind the Clear");
}
/// <summary>
/// lmxopcua#869 — the driver ACCEPTED the ack (AcknowledgeAsync returned normally, e.g. the gateway's "sent but
/// unconfirmed" OK) but its upstream never confirmed it: the confirmation refusal reverts the condition to
/// unacknowledged exactly like an AcknowledgeAsync failure. Before #869 nothing awaited a confirmation, so the
/// condition stayed Acknowledged although Galaxy never applied the ack.
/// </summary>
[Fact]
public void An_accepted_but_unconfirmed_ack_reverts_the_condition_to_unacknowledged()
{
var db = NewInMemoryDbFactory();
var recorder = new RecordingAlarmDriverFactory("GalaxyMxGateway")
{
Confirmation = () => Task.FromException(new DriverOperationRefusedException(
"Galaxy did not confirm the acknowledge of 'Tank.Hi' within 10 s (BadTimeout; gateway: AckMsg write sent; OnWriteComplete not observed)")),
};
var deploymentId = SeedV3AlarmDeployment(db, RevA, Driver: "drv-1", Tag: "temp_hi");
var publish = this.CreateCleanOutcomePublishProbe();
var actor = SpawnHostAndApply(db, deploymentId, recorder, publish.Ref);
RaiseAndExpectUnacked(actor, publish);
actor.Tell(new DriverHostActor.RouteNativeAlarmAck(AlarmRawPath, "cmt", "alice"));
var reverted = NextAlarmUpdate(publish);
reverted.AlarmNodeId.ShouldBe(AlarmRawPath);
reverted.State.Active.ShouldBeTrue();
reverted.State.Acknowledged.ShouldBeFalse("an ack the upstream never confirmed must not stand");
recorder.Acks.Count.ShouldBe(1);
recorder.ConfirmCalls.ShouldBe(1);
}
/// <summary>
/// lmxopcua#869 — the confirmation wait runs OFF the driver actor's mailbox: while one ack's confirmation is
/// still pending (it never completes here), a second ack is still routed to the driver. A confirmation awaited
/// inside the ack handler would suspend the mailbox and the second ack would never arrive.
/// </summary>
[Fact]
public void A_pending_confirmation_does_not_block_the_driver_actor()
{
var db = NewInMemoryDbFactory();
var never = new TaskCompletionSource();
var recorder = new RecordingAlarmDriverFactory("GalaxyMxGateway")
{
Confirmation = () => never.Task,
};
var deploymentId = SeedV3AlarmDeployment(db, RevA, Driver: "drv-1", Tag: "temp_hi");
var actor = SpawnHostAndApply(db, deploymentId, recorder);
actor.Tell(new DriverHostActor.RouteNativeAlarmAck(AlarmRawPath, "first", "alice"));
AwaitAssert(() => recorder.ConfirmCalls.ShouldBe(1), duration: Timeout, cancellationToken: TestContext.Current.CancellationToken);
actor.Tell(new DriverHostActor.RouteNativeAlarmAck(AlarmRawPath, "second", "alice"));
AwaitAssert(() => recorder.Acks.ShouldContain(a => a.Comment == "second"), duration: Timeout,
cancellationToken: TestContext.Current.CancellationToken);
never.TrySetResult();
} }
/// <summary> /// <summary>
@@ -452,6 +546,12 @@ public sealed class DriverHostActorNativeAlarmAckRoutingTests : RuntimeActorTest
/// <summary>The driver's synchronous pre-check verdict (lmxopcua#760); null ⇒ always proceed.</summary> /// <summary>The driver's synchronous pre-check verdict (lmxopcua#760); null ⇒ always proceed.</summary>
public Func<string?, string?>? PreflightRefusal { init => _driver.PreflightRefusal = value; } public Func<string?, string?>? PreflightRefusal { init => _driver.PreflightRefusal = value; }
/// <summary>lmxopcua#869: what <see cref="IAlarmAcknowledgeConfirmation.ConfirmAcknowledgedAsync"/> returns; null ⇒ confirmed at once.</summary>
public Func<Task>? Confirmation { init => _driver.Confirmation = value; }
/// <summary>How many confirmations the spawned driver was asked for.</summary>
public int ConfirmCalls => _driver.ConfirmCalls;
/// <inheritdoc /> /// <inheritdoc />
public IDriver? TryCreate(string driverType, string driverInstanceId, string driverConfigJson) public IDriver? TryCreate(string driverType, string driverInstanceId, string driverConfigJson)
{ {
@@ -465,8 +565,24 @@ public sealed class DriverHostActorNativeAlarmAckRoutingTests : RuntimeActorTest
} }
/// <summary>An <see cref="IDriver"/> + <see cref="IAlarmSource"/> that records every acknowledge.</summary> /// <summary>An <see cref="IDriver"/> + <see cref="IAlarmSource"/> that records every acknowledge.</summary>
private sealed class RecordingAlarmDriver : IDriver, IAlarmSource, IAlarmAcknowledgePreflight private sealed class RecordingAlarmDriver : IDriver, IAlarmSource, IAlarmAcknowledgePreflight, IAlarmAcknowledgeConfirmation
{ {
/// <summary>lmxopcua#869: the confirmation outcome; null ⇒ confirmed at once.</summary>
public Func<Task>? Confirmation { get; set; }
private int _confirmCalls;
/// <summary>How many confirmations were requested.</summary>
public int ConfirmCalls => Volatile.Read(ref _confirmCalls);
/// <inheritdoc />
public Task ConfirmAcknowledgedAsync(
IReadOnlyList<AlarmAcknowledgeRequest> acknowledgements, CancellationToken cancellationToken)
{
Interlocked.Increment(ref _confirmCalls);
return Confirmation?.Invoke() ?? Task.CompletedTask;
}
/// <summary>When set, <see cref="AcknowledgeAsync"/> records the request and then throws this.</summary> /// <summary>When set, <see cref="AcknowledgeAsync"/> records the request and then throws this.</summary>
public Exception? AckFault { get; set; } public Exception? AckFault { get; set; }
@@ -0,0 +1,180 @@
using System.Collections.Concurrent;
using Akka.Actor;
using Shouldly;
using Xunit;
using ZB.MOM.WW.OtOpcUa.Core.Abstractions;
using ZB.MOM.WW.OtOpcUa.Runtime.Drivers;
using ZB.MOM.WW.OtOpcUa.Runtime.Tests.Harness;
namespace ZB.MOM.WW.OtOpcUa.Runtime.Tests.Drivers;
/// <summary>
/// lmxopcua#869 — <see cref="DriverInstanceActor"/> asks an <see cref="IAlarmAcknowledgeConfirmation"/> driver to
/// confirm an ack AFTER <see cref="IAlarmSource.AcknowledgeAsync"/> returned, off its mailbox, and reports
/// <see cref="DriverInstanceActor.AlarmAckFailed"/> to the router when the confirmation fails — including one that
/// fails long after the ack handler returned, and one abandoned because the actor stopped.
/// </summary>
public sealed class DriverInstanceActorAckConfirmationTests : RuntimeActorTestBase
{
private static readonly TimeSpan Timeout = PresenceBudget;
/// <summary>A confirmation that faults asynchronously, after the ack handler returned, is reported failed.</summary>
[Fact]
public void A_confirmation_that_fails_later_is_reported_as_a_failed_ack()
{
var driver = new ConfirmingDriver();
var actor = SpawnConnected(driver);
var gate = new TaskCompletionSource();
driver.Confirmation = () => gate.Task;
var router = CreateTestProbe();
actor.Tell(Ack("n1"), router.Ref);
AwaitCondition(() => driver.ConfirmCalls == 1, Timeout, TestContext.Current.CancellationToken);
gate.SetException(new DriverOperationRefusedException("not confirmed within 10 s (BadTimeout)"));
var failed = router.ExpectMsg<DriverInstanceActor.AlarmAckFailed>(Timeout, cancellationToken: TestContext.Current.CancellationToken);
failed.ConditionNodeId.ShouldBe("n1");
failed.Reason.ShouldContain("BadTimeout");
driver.ForeignConfirmations.ShouldBe(0, "the confirmation must get the SAME request instance the ack was sent with");
}
/// <summary>
/// A confirmation still pending when the actor stops is abandoned AND reported failed, so the host reverts the
/// ack this incarnation can no longer confirm. (A confirmation continuation posted back to the stopped actor
/// would dead-letter and never report.)
/// </summary>
[Fact]
public void A_confirmation_pending_when_the_actor_stops_is_reported_as_a_failed_ack()
{
var driver = new ConfirmingDriver();
var actor = SpawnConnected(driver);
driver.Confirmation = () => new TaskCompletionSource().Task; // never completes on its own
var router = CreateTestProbe();
actor.Tell(Ack("n-stop"), router.Ref);
AwaitCondition(() => driver.ConfirmCalls == 1, Timeout, TestContext.Current.CancellationToken);
Sys.Stop(actor);
var failed = router.ExpectMsg<DriverInstanceActor.AlarmAckFailed>(Timeout, cancellationToken: TestContext.Current.CancellationToken);
failed.ConditionNodeId.ShouldBe("n-stop");
failed.Reason.ShouldContain("stopped");
}
/// <summary>
/// A confirmation that succeeds reports nothing. Positive control: a later ack whose confirmation fails is
/// reported, and it is the FIRST report the router receives — so none came for the confirmed one.
/// </summary>
[Fact]
public void A_confirmed_ack_reports_nothing()
{
var driver = new ConfirmingDriver();
var actor = SpawnConnected(driver);
var router = CreateTestProbe();
// Already complete: the actor's confirmation task finishes (and would report) right after the call that
// bumps ConfirmCalls, long before the second ack below makes its actor round trip.
driver.Confirmation = () => Task.CompletedTask;
actor.Tell(Ack("n-ok"), router.Ref);
AwaitCondition(() => driver.ConfirmCalls == 1, Timeout, TestContext.Current.CancellationToken);
driver.Confirmation = () => Task.FromException(new DriverOperationRefusedException("unconfirmed"));
actor.Tell(Ack("n-bad"), router.Ref);
router.ExpectMsg<DriverInstanceActor.AlarmAckFailed>(Timeout, cancellationToken: TestContext.Current.CancellationToken).ConditionNodeId.ShouldBe("n-bad");
}
/// <summary>
/// lmxopcua#869 review — the failure echoes the routed ack's <see cref="DriverInstanceActor.RouteAlarmAck.AckSequence"/>,
/// which is what lets the host revert only the acknowledge that failed (keyed in
/// <c>NativeAlarmProjector.TryRevertAcknowledge(nodeId, ackSequence, …)</c>).
/// </summary>
[Fact]
public void A_failed_confirmation_echoes_the_ack_sequence()
{
var driver = new ConfirmingDriver();
var actor = SpawnConnected(driver);
driver.Confirmation = () => Task.FromException(new DriverOperationRefusedException("unconfirmed"));
var router = CreateTestProbe();
actor.Tell(Ack("n-seq") with { AckSequence = 42 }, router.Ref);
router.ExpectMsg<DriverInstanceActor.AlarmAckFailed>(Timeout, cancellationToken: TestContext.Current.CancellationToken)
.AckSequence.ShouldBe(42L);
}
private static DriverInstanceActor.RouteAlarmAck Ack(string conditionNodeId) =>
new("Tank.Hi", "c", "alice", ConditionNodeId: conditionNodeId);
/// <summary>Spawns the actor as a child of the test actor and waits for its Connected announcement.</summary>
private IActorRef SpawnConnected(ConfirmingDriver driver)
{
var actor = ChildActorOf(DriverInstanceActor.Props(driver));
actor.Tell(new DriverInstanceActor.InitializeRequested("{}"));
FishForMessage<DriverInstanceActor.ConnectivityChanged>(c => c.Connected, Timeout);
return actor;
}
private sealed class ConfirmingDriver : IDriver, IAlarmSource, IAlarmAcknowledgeConfirmation
{
private int _ackCalls;
private int _confirmCalls;
private readonly ConcurrentQueue<IReadOnlyList<AlarmAcknowledgeRequest>> _ackedLists = new();
private int _foreignConfirmations;
/// <summary>Confirmations asked for a request list that was NOT the instance passed to AcknowledgeAsync —
/// the contract a driver matching by instance (Galaxy) relies on.</summary>
public int ForeignConfirmations => Volatile.Read(ref _foreignConfirmations);
public Func<Task>? Confirmation { get; set; }
public int AckCalls => Volatile.Read(ref _ackCalls);
public int ConfirmCalls => Volatile.Read(ref _confirmCalls);
public string DriverInstanceId => "drv-confirm";
public string DriverType => "Test";
public Task InitializeAsync(string driverConfigJson, CancellationToken cancellationToken) => Task.CompletedTask;
public Task ReinitializeAsync(string driverConfigJson, CancellationToken cancellationToken) => Task.CompletedTask;
public Task ShutdownAsync(CancellationToken cancellationToken) => Task.CompletedTask;
public DriverHealth GetHealth() => new(DriverState.Healthy, null, null);
public Task<IAlarmSubscriptionHandle> SubscribeAlarmsAsync(
IReadOnlyList<string> sourceNodeIds, CancellationToken cancellationToken) =>
Task.FromResult<IAlarmSubscriptionHandle>(new Handle());
public Task UnsubscribeAlarmsAsync(IAlarmSubscriptionHandle handle, CancellationToken cancellationToken) =>
Task.CompletedTask;
public Task AcknowledgeAsync(
IReadOnlyList<AlarmAcknowledgeRequest> acknowledgements, CancellationToken cancellationToken)
{
_ackedLists.Enqueue(acknowledgements);
Interlocked.Increment(ref _ackCalls);
return Task.CompletedTask;
}
public Task ConfirmAcknowledgedAsync(
IReadOnlyList<AlarmAcknowledgeRequest> acknowledgements, CancellationToken cancellationToken)
{
if (!_ackedLists.Any(l => ReferenceEquals(l, acknowledgements))) Interlocked.Increment(ref _foreignConfirmations);
Interlocked.Increment(ref _confirmCalls);
return (Confirmation?.Invoke() ?? Task.CompletedTask).WaitAsync(cancellationToken);
}
public event EventHandler<AlarmEventArgs>? OnAlarmEvent
{
add { }
remove { }
}
private sealed class Handle : IAlarmSubscriptionHandle
{
public string DiagnosticId => "h";
}
}
}
@@ -606,6 +606,56 @@ public class NativeAlarmProjectorTests
snapshot.ShouldBeNull(); snapshot.ShouldBeNull();
} }
/// <summary>
/// lmxopcua#869 — a transition that only CARRIES the optimistic ack through (a Clear: the alarm returned to
/// normal while the ack was still being confirmed upstream) does not settle it: a late failure still reverts,
/// to inactive + UNacked (Galaxy's UNACK_RTN). Positive control: an Acknowledge transition does settle it.
/// </summary>
[Fact]
public void A_carry_through_transition_keeps_the_local_ack_pending()
{
var sut = new NativeAlarmProjector();
sut.Project("n1", Evt(AlarmTransitionKind.Raise));
sut.MarkAcknowledged("n1");
sut.Project("n1", Evt(AlarmTransitionKind.Clear)).Acknowledged.ShouldBeTrue("the optimistic ack rides along");
sut.TryRevertAcknowledge("n1", out var reverted).ShouldBeTrue();
reverted!.Active.ShouldBeFalse();
reverted.Acknowledged.ShouldBeFalse();
sut.Project("n1", Evt(AlarmTransitionKind.Raise));
sut.MarkAcknowledged("n1");
sut.Project("n1", Evt(AlarmTransitionKind.Raise)).Acknowledged.ShouldBeTrue("a same-message Raise carries it too");
sut.Project("n1", Evt(AlarmTransitionKind.Acknowledge));
sut.TryRevertAcknowledge("n1", out _).ShouldBeFalse("an Acknowledge transition states the ack — authoritative");
}
/// <summary>
/// lmxopcua#869 review — a pending ack can outlive its own delivery (a source that sends no Acknowledge back,
/// across a Clear that only carried it), so a revert is keyed to the failed ack's sequence: a later re-ack that
/// changed nothing (sequence 0) or a stale earlier ack's failure does not undo it. Positive control: the
/// matching sequence does.
/// </summary>
[Fact]
public void A_revert_only_undoes_the_acknowledge_that_failed()
{
var sut = new NativeAlarmProjector();
sut.Project("n1", Evt(AlarmTransitionKind.Raise));
var first = sut.MarkAcknowledged("n1");
first.ShouldBeGreaterThan(0);
sut.Project("n1", Evt(AlarmTransitionKind.Clear)); // first ack delivered; nothing settles it
var reAck = sut.MarkAcknowledged("n1"); // a re-ack of the acked, cleared condition
reAck.ShouldBe(0L, "it changed nothing, so its failure has nothing to undo");
sut.TryRevertAcknowledge("n1", reAck, out _).ShouldBeFalse("a later re-ack's failure must not revert the first");
sut.Project("n1", Evt(AlarmTransitionKind.Raise, msg: "again")); // a new occurrence (un-acks)
var second = sut.MarkAcknowledged("n1");
sut.TryRevertAcknowledge("n1", first, out _).ShouldBeFalse("a stale earlier ack's failure must not revert the newer one");
sut.TryRevertAcknowledge("n1", second, out var reverted).ShouldBeTrue();
reverted!.Acknowledged.ShouldBeFalse();
}
private static AlarmEventArgs Evt( private static AlarmEventArgs Evt(
AlarmTransitionKind kind, AlarmTransitionKind kind,
AlarmSeverity sev = AlarmSeverity.High, AlarmSeverity sev = AlarmSeverity.High,