dohertj2/lmxopcua
1
0
Fork 0
Code Issues 5 Pull Requests Actions Packages Projects Releases Wiki Activity

fix(runtime,galaxy): #869 review round 1 — confirmation off the actor scheduler, stop/async-fault tests

Browse Source 
- Start the ack confirmation with Task.Run + ConfigureAwait(false) so it never
  depends on the ReceiveAsync handler's ActorTaskScheduler (the PostStop
  abandon-and-revert report must run after the actor stopped).
- DriverInstanceActorAckConfirmationTests: a confirmation that faults after the
  handler returned is reported; one pending at actor stop is reported; a
  confirmed ack reports nothing (positive control).
- Deterministic tracker matching test (Raise/Clear/Retrigger/other alarm do not
  confirm; case-insensitive match does).
- AdminUI min/max on the confirm-timeout field; Galaxy.md documents the
  reconnect, UNACK_RTN-as-Clear and RPC-budget limits.

Claude-Session: https://claude.ai/code/session_01GT67AD99zQQqpLCVm3p7xk
This commit is contained in:
Joseph Doherty
2026-11-11 13:11:49 -05:00
parent 4e492e6550
commit 77872b1d1c
5 changed files with 204 additions and 7 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/drivers/Galaxy.md
+10
View File
@@ -162,6 +162,16 @@ e.g. `AckMsg write to '…' sent; OnWriteComplete not observed within 5000 ms.`)
condition event with `AckedState = false`, and the `BadTimeout` verdict is recorded in the driver and host
Warning logs. A confirming transition that arrives after the revert simply re-acknowledges the condition.
A confirmation in flight when the driver actor stops is abandoned and reverted the same way.
- **Limits to know:**
- A gateway **reconnect** detaches the feed; an ack in flight across it is confirmed only if the new feed's
snapshot re-statement (`ActiveAcknowledged`) arrives inside the window, otherwise it reverts (fails safe).
- Acking an alarm that already **returned to normal** (`UNACK_RTN`) is confirmed by the worker's
`Acknowledge` (`ACK_RTN`) transition; if the gateway reconciles it as a `Clear` instead, nothing confirms
and the correctly-applied ack reverts to unacked. Unverified on hardware — include it in the next live gate.
- The RPC itself still runs under the actor's 5 s ack bound and the Tier-A `AlarmAcknowledge` timeout (5 s).
The gateway worker's write-completion window (`DefaultWriteCompletionTimeout`, 1.5 s) plus `AuthenticateUser` must stay
well under that, or the unconfirmed reply is never seen and the ack reverts as a timeout that also counts
toward the breaker.
- `ConfirmTimeoutMs = 0` restores the pre-#869 behaviour (an unconfirmed ack counts as acknowledged, with a
Warning) — an escape hatch only. Why the gateway-side ack did not stick on VD03 is tracked upstream
(mxaccessgw#226).
src/Server/ZB.MOM.WW.OtOpcUa.AdminUI/Components/Shared/Drivers/Forms/GalaxyDriverForm.razor
+1 -1
View File
@@ -111,7 +111,7 @@
</div>
<div class="col-md-3">
<label class="form-label">Ack confirm timeout (ms)</label>
<InputNumber @bind-Value="_form.Galaxy.AckConfirmTimeoutMs" @bind-Value:after="EmitAsync" class="form-control form-control-sm" />
<InputNumber @bind-Value="_form.Galaxy.AckConfirmTimeoutMs" @bind-Value:after="EmitAsync" min="0" max="120000" class="form-control form-control-sm" />
<div class="form-text">When the gateway sends an ack but cannot confirm it, wait this long for Galaxy's acknowledged transition before reverting the condition to unacknowledged (#869). Default 10000; 0 = do not wait.</div>
</div>
</div>
src/Server/ZB.MOM.WW.OtOpcUa.Runtime/Drivers/DriverInstanceActor.cs
+11 -3
View File
@@ -1376,8 +1376,16 @@ public sealed class DriverInstanceActor : ReceiveActor, IWithTimers
// "sent but unconfirmed" OK). The confirmation wait is seconds long, so it runs OFF this mailbox and
// OUTSIDE the resilience pipeline (a wait on an upstream event — it must neither be cut by the
// AlarmAcknowledge capability timeout nor count toward the breaker). Failure reverts like any other.
// Task.Run, not a bare call: this handler is a ReceiveAsync, so TaskScheduler.Current is the actor's
// scheduler; the confirmation must not depend on this actor's context at all — in particular not after
// PostStop, when it still has to report the abandoned ack (pinned by DriverInstanceActorAckConfirmationTests).
if (src is IAlarmAcknowledgeConfirmation confirmer)
_ = RunAckConfirmationAsync(confirmer, request, msg, replyTo, _ackConfirmCts.Token, _driverInstanceId, _log);
{
var stopToken = _ackConfirmCts.Token;
var driverInstanceId = _driverInstanceId;
var log = _log;
_ = Task.Run(() => RunAckConfirmationAsync(confirmer, request, msg, replyTo, stopToken, driverInstanceId, log));
}
}
catch (DriverOperationRefusedException ex)
{
@@ -1412,10 +1420,10 @@ public sealed class DriverInstanceActor : ReceiveActor, IWithTimers
IAlarmAcknowledgeConfirmation confirmer, IReadOnlyList<AlarmAcknowledgeRequest> request, RouteAlarmAck msg,
IActorRef replyTo, CancellationToken stopToken, string driverInstanceId, ILoggingAdapter log)
{
await Task.Yield();
try
{
await confirmer.ConfirmAcknowledgedAsync(request, stopToken).WaitAsync(AckConfirmationCeiling, stopToken);
await confirmer.ConfirmAcknowledgedAsync(request, stopToken)
.WaitAsync(AckConfirmationCeiling, stopToken).ConfigureAwait(false);
}
catch (DriverOperationRefusedException ex)
{
tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Galaxy.Tests/Runtime/GalaxyAckConfirmationTests.cs
+30 -3
View File
@@ -111,9 +111,9 @@ public sealed class GalaxyAckConfirmationTests
}
/// <summary>
/// Unconfirmed OK + Galaxy's acknowledged transition on the feed → acknowledged. Positive control: the
/// confirmation is still pending after transitions that must NOT confirm (another alarm's ack, this
/// alarm's raise) — it only completes on the matching one.
/// Unconfirmed OK + Galaxy's acknowledged transition on the feed → acknowledged, end to end through the
/// driver's feed hook (non-matching transitions are emitted first; that they do NOT confirm is pinned
/// synchronously by <see cref="Tracker_confirms_only_on_this_alarms_acknowledged_transition"/>).
/// </summary>
[Fact]
public async Task Unconfirmed_ack_is_confirmed_by_the_acknowledged_transition()
@@ -246,6 +246,33 @@ public sealed class GalaxyAckConfirmationTests
}
}
/// <summary>
/// Tracker matching rules, checked synchronously: only an Acknowledge / ActiveAcknowledged transition of the
/// SAME alarm (case-insensitive) confirms. Positive control: the matching transition does flip it.
/// </summary>
[Fact]
public void Tracker_confirms_only_on_this_alarms_acknowledged_transition()
{
var tracker = new GalaxyAckConfirmationTracker(TimeSpan.FromSeconds(30), new CapturingLogger(), "drv");
var watch = tracker.Begin(Ack(AlarmRef)[0], AlarmRef);
tracker.Arm(watch, UnconfirmedDiagnostic);
foreach (var kind in new[]
{
GalaxyAlarmTransitionKind.Raise, GalaxyAlarmTransitionKind.Clear,
GalaxyAlarmTransitionKind.Retrigger, GalaxyAlarmTransitionKind.Unspecified,
})
{
tracker.Observe(Transition(AlarmRef, kind));
}
tracker.Observe(Transition("Other01.Level.Hi", GalaxyAlarmTransitionKind.Acknowledge));
tracker.Observe(Transition("Other01.Level.Hi", GalaxyAlarmTransitionKind.ActiveAcknowledged));
watch.IsConfirmed.ShouldBeFalse("no transition so far says THIS alarm is acknowledged");
tracker.Observe(Transition(AlarmRef.ToLowerInvariant(), GalaxyAlarmTransitionKind.Acknowledge));
watch.IsConfirmed.ShouldBeTrue();
}
[Fact]
public void Options_default_to_ten_seconds_and_map_negative_to_the_default()
{
tests/Server/ZB.MOM.WW.OtOpcUa.Runtime.Tests/Drivers/DriverInstanceActorAckConfirmationTests.cs
+152
View File
@@ -0,0 +1,152 @@
using System.Collections.Concurrent;
using Akka.Actor;
using Shouldly;
using Xunit;
using ZB.MOM.WW.OtOpcUa.Core.Abstractions;
using ZB.MOM.WW.OtOpcUa.Runtime.Drivers;
using ZB.MOM.WW.OtOpcUa.Runtime.Tests.Harness;
namespace ZB.MOM.WW.OtOpcUa.Runtime.Tests.Drivers;
/// <summary>
/// lmxopcua#869 — <see cref="DriverInstanceActor"/> asks an <see cref="IAlarmAcknowledgeConfirmation"/> driver to
/// confirm an ack AFTER <see cref="IAlarmSource.AcknowledgeAsync"/> returned, off its mailbox, and reports
/// <see cref="DriverInstanceActor.AlarmAckFailed"/> to the router when the confirmation fails — including one that
/// fails long after the ack handler returned, and one abandoned because the actor stopped.
/// </summary>
public sealed class DriverInstanceActorAckConfirmationTests : RuntimeActorTestBase
{
private static readonly TimeSpan Timeout = PresenceBudget;
/// <summary>A confirmation that faults asynchronously, after the ack handler returned, is reported failed.</summary>
[Fact]
public void A_confirmation_that_fails_later_is_reported_as_a_failed_ack()
{
var driver = new ConfirmingDriver();
var actor = SpawnConnected(driver);
var gate = new TaskCompletionSource();
driver.Confirmation = () => gate.Task;
var router = CreateTestProbe();
actor.Tell(Ack("n1"), router.Ref);
AwaitCondition(() => driver.ConfirmCalls == 1, Timeout, TestContext.Current.CancellationToken);
gate.SetException(new DriverOperationRefusedException("not confirmed within 10 s (BadTimeout)"));
var failed = router.ExpectMsg<DriverInstanceActor.AlarmAckFailed>(Timeout, cancellationToken: TestContext.Current.CancellationToken);
failed.ConditionNodeId.ShouldBe("n1");
failed.Reason.ShouldContain("BadTimeout");
}
/// <summary>
/// A confirmation still pending when the actor stops is abandoned AND reported failed, so the host reverts the
/// ack this incarnation can no longer confirm. (A confirmation continuation posted back to the stopped actor
/// would dead-letter and never report.)
/// </summary>
[Fact]
public void A_confirmation_pending_when_the_actor_stops_is_reported_as_a_failed_ack()
{
var driver = new ConfirmingDriver();
var actor = SpawnConnected(driver);
driver.Confirmation = () => new TaskCompletionSource().Task; // never completes on its own
var router = CreateTestProbe();
actor.Tell(Ack("n-stop"), router.Ref);
AwaitCondition(() => driver.ConfirmCalls == 1, Timeout, TestContext.Current.CancellationToken);
Sys.Stop(actor);
var failed = router.ExpectMsg<DriverInstanceActor.AlarmAckFailed>(Timeout, cancellationToken: TestContext.Current.CancellationToken);
failed.ConditionNodeId.ShouldBe("n-stop");
failed.Reason.ShouldContain("stopped");
}
/// <summary>
/// A confirmation that succeeds reports nothing. Positive control: a later ack whose confirmation fails is
/// reported, and it is the FIRST report the router receives — so none came for the confirmed one.
/// </summary>
[Fact]
public void A_confirmed_ack_reports_nothing()
{
var driver = new ConfirmingDriver();
var actor = SpawnConnected(driver);
var router = CreateTestProbe();
driver.Confirmation = () => Task.Delay(20);
actor.Tell(Ack("n-ok"), router.Ref);
AwaitCondition(() => driver.ConfirmCalls == 1, Timeout, TestContext.Current.CancellationToken);
driver.Confirmation = () => Task.FromException(new DriverOperationRefusedException("unconfirmed"));
actor.Tell(Ack("n-bad"), router.Ref);
router.ExpectMsg<DriverInstanceActor.AlarmAckFailed>(Timeout, cancellationToken: TestContext.Current.CancellationToken).ConditionNodeId.ShouldBe("n-bad");
}
private static DriverInstanceActor.RouteAlarmAck Ack(string conditionNodeId) =>
new("Tank.Hi", "c", "alice", ConditionNodeId: conditionNodeId);
/// <summary>Spawns the actor as a child of the test actor and waits for its Connected announcement.</summary>
private IActorRef SpawnConnected(ConfirmingDriver driver)
{
var actor = ChildActorOf(DriverInstanceActor.Props(driver));
actor.Tell(new DriverInstanceActor.InitializeRequested("{}"));
FishForMessage<DriverInstanceActor.ConnectivityChanged>(c => c.Connected, Timeout);
return actor;
}
private sealed class ConfirmingDriver : IDriver, IAlarmSource, IAlarmAcknowledgeConfirmation
{
private int _ackCalls;
private int _confirmCalls;
private readonly ConcurrentQueue<AlarmAcknowledgeRequest> _acks = new();
public Func<Task>? Confirmation { get; set; }
public int AckCalls => Volatile.Read(ref _ackCalls);
public int ConfirmCalls => Volatile.Read(ref _confirmCalls);
public string DriverInstanceId => "drv-confirm";
public string DriverType => "Test";
public Task InitializeAsync(string driverConfigJson, CancellationToken cancellationToken) => Task.CompletedTask;
public Task ReinitializeAsync(string driverConfigJson, CancellationToken cancellationToken) => Task.CompletedTask;
public Task ShutdownAsync(CancellationToken cancellationToken) => Task.CompletedTask;
public DriverHealth GetHealth() => new(DriverState.Healthy, null, null);
public Task<IAlarmSubscriptionHandle> SubscribeAlarmsAsync(
IReadOnlyList<string> sourceNodeIds, CancellationToken cancellationToken) =>
Task.FromResult<IAlarmSubscriptionHandle>(new Handle());
public Task UnsubscribeAlarmsAsync(IAlarmSubscriptionHandle handle, CancellationToken cancellationToken) =>
Task.CompletedTask;
public Task AcknowledgeAsync(
IReadOnlyList<AlarmAcknowledgeRequest> acknowledgements, CancellationToken cancellationToken)
{
foreach (var a in acknowledgements) _acks.Enqueue(a);
Interlocked.Increment(ref _ackCalls);
return Task.CompletedTask;
}
public Task ConfirmAcknowledgedAsync(
IReadOnlyList<AlarmAcknowledgeRequest> acknowledgements, CancellationToken cancellationToken)
{
Interlocked.Increment(ref _confirmCalls);
return (Confirmation?.Invoke() ?? Task.CompletedTask).WaitAsync(cancellationToken);
}
public event EventHandler<AlarmEventArgs>? OnAlarmEvent
{
add { }
remove { }
}
private sealed class Handle : IAlarmSubscriptionHandle
{
public string DiagnosticId => "h";
}
}
}