dohertj2/lmxopcua
1
0
Fork 0
Code Issues 24 Pull Requests Actions 2 Packages Projects Releases Wiki Activity

fix(runtime): derive the write cap from the configured write timeout (lmxopcua#774)

Browse Source 
DriverInstanceActor.RunWriteAsync cancelled every write with a hard-coded,
undilated 5 s CancellationTokenSource, so writeTimeout / DefaultWriteTimeout
(then 10 s) only ever served as the pessimistic abandon bound: a configured
bound above 5 s never took effect.

The write's token is now cancelled at the configured write timeout; a write
that ignores its token is abandoned WriteAbandonGrace later (half the
timeout, at most 2 s) so a token-honouring driver still gets to answer the
cancellation itself. Both bounds run on an injectable TimeProvider
(Props writeClock, System by default).

DefaultWriteTimeout is now 5 s (cancel 5 s, abandon 7 s), not 10 s: the
host's write Ask is 8 s, and a write allowed to run past it would already
have been reported as a timeout and its node reverted while the device could
still take the value - the exact failure the off-mailbox write work fixed.
Production behaviour is therefore unchanged for the cancel (5 s) and tighter
for a token-ignoring driver (abandoned at 7 s instead of 10 s, i.e. inside
the Ask). DriverHostActor's Ask comment updated to match.

Tests (DriverInstanceActorConcurrentWriteTests, manual clock copied from
Ab.Transport.Tests into Runtime.Tests/Harness/ManualTimeProvider):
- a write held past 6 s under a 10 s write timeout succeeds;
- positive control: at 10 s the token is cancelled ("write timeout");
- a token-ignoring write is still pending at 10 s and abandoned (BadTimeout)
  after the grace.
Mutation checks: a fixed 5 s token on the same clock fails the first test;
dropping the grace fails the third (and the second, where the abandon then
races the cancellation).

342a825d re-check: its held-write tests were written around the undilated
5 s cap. Connect() now defaults the write timeout to the DILATED presence
budget, so a held, explicitly-released write no longer races any undilated
deadline; tests about the bound pass their own. AwaitWritesEntered keeps its
20 ms interval (doc updated).

Runtime.Tests: 946 passed / 3 skipped / 0 failed at AKKA_TEST_TIMEFACTOR 3
and 1. Solution Release build: 0 warnings.

Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
This commit is contained in:
Joseph Doherty
2026-11-09 15:18:37 -05:00
parent f0992394b6
commit 75be6fe12b
4 changed files with 259 additions and 27 deletions
Download Patch File Download Diff File Expand all files Collapse all files
src/Server/ZB.MOM.WW.OtOpcUa.Runtime/Drivers/DriverHostActor.cs
+3 -2
View File
@@ -2237,8 +2237,9 @@ public sealed class DriverHostActor : ReceiveActor, IWithTimers, IWithUnboundedS
return;
}
// Ask the child a bounded write — DriverInstanceActor.HandleWriteAsync already bounds the backend
// call to 5s, so the 8s Ask is a safety net for an actor that never replies. Capture Sender before
// Ask the child a bounded write — DriverInstanceActor bounds the backend call by its write timeout
// (DefaultWriteTimeout 5 s; a token-ignoring write is abandoned at 7 s), so the 8s Ask is a safety net
// for an actor that never replies. Capture Sender before
// the continuation: it runs off the actor thread, where raw Sender is unsafe to read.
var replyTo = Sender;
entry.Actor.Ask<DriverInstanceActor.WriteAttributeResult>(
src/Server/ZB.MOM.WW.OtOpcUa.Runtime/Drivers/DriverInstanceActor.cs
+51 -20
View File
@@ -229,11 +229,30 @@ public sealed class DriverInstanceActor : ReceiveActor, IWithTimers
public static readonly TimeSpan DefaultInitializeTimeout = TimeSpan.FromSeconds(30);
/// <summary>
/// Pessimistic bound on one write round-trip, on top of the 5 s token-based write cap. A driver whose
/// <see cref="IWritable.WriteAsync"/> ignores its token would otherwise hold its slot AND its tag forever,
/// piling every later write to that tag up in the queue. 2× the write cap.
/// Bound on one write round-trip. The write's cancellation token is cancelled when it elapses (the write
/// answers <c>"write timeout"</c>); a driver whose <see cref="IWritable.WriteAsync"/> ignores its token is
/// abandoned <see cref="WriteAbandonGrace"/> later (the write answers <c>BadTimeout</c> and frees its slot
/// and tag), so it cannot hold its slot AND its tag forever, piling every later write to that tag up in the
/// queue.
/// </summary>
public static readonly TimeSpan DefaultWriteTimeout = TimeSpan.FromSeconds(10);
/// <remarks>
/// 5 s, so both bounds (5 s cancel, 7 s abandon) answer inside the host's 8 s write Ask
/// (<c>DriverHostActor</c>): a write that outlives the Ask has already been reported as a timeout and its
/// node reverted upstream, so letting it run on would let the device take a value the client was told
/// failed. Until lmxopcua#774 the cancel was a hard-coded 5 s that ignored this value, and the default was
/// 10 s — a configured bound above 5 s never took effect.
/// </remarks>
public static readonly TimeSpan DefaultWriteTimeout = TimeSpan.FromSeconds(5);
/// <summary>
/// How long after a write's token is cancelled (<see cref="DefaultWriteTimeout"/>) a write that has not
/// returned is abandoned as token-ignoring: half the write timeout, at most 2 s. Long enough for a driver
/// that honours its token to unwind and answer the cancellation itself.
/// </summary>
/// <param name="writeTimeout">The write timeout in force.</param>
/// <returns>The grace between cancelling the write's token and abandoning the write.</returns>
internal static TimeSpan WriteAbandonGrace(TimeSpan writeTimeout) =>
TimeSpan.FromTicks(Math.Min(writeTimeout.Ticks / 2, TimeSpan.FromSeconds(2).Ticks));
/// <summary>Self-sent by a write task once it has replied, releasing its slot and its tag.</summary>
/// <param name="TagId">The tag the write held.</param>
@@ -289,6 +308,9 @@ public sealed class DriverInstanceActor : ReceiveActor, IWithTimers
/// <summary>See <see cref="DefaultWriteTimeout"/>.</summary>
private readonly TimeSpan _writeTimeout;
/// <summary>The clock the write bounds run on — <see cref="TimeProvider.System"/> outside tests.</summary>
private readonly TimeProvider _writeClock;
/// <summary>This actor object's identity for its own off-mailbox completions (drawn from the process-wide
/// generation source, so it is unique across supervision restarts of the same child).</summary>
private readonly int _incarnation;
@@ -434,8 +456,11 @@ public sealed class DriverInstanceActor : ReceiveActor, IWithTimers
/// <c>BadTimeout</c>; defaults to <see cref="DefaultQueuedWriteBudget"/>.</param>
/// <param name="initializeTimeout">Optional pessimistic bound on one initialise attempt; defaults to
/// <see cref="DefaultInitializeTimeout"/>.</param>
/// <param name="writeTimeout">Optional pessimistic bound on one write round-trip; defaults to
/// <param name="writeTimeout">Optional bound on one write round-trip (its token is cancelled when it elapses;
/// a token-ignoring write is abandoned <see cref="WriteAbandonGrace"/> later); defaults to
/// <see cref="DefaultWriteTimeout"/>.</param>
/// <param name="writeClock">Optional clock for the write bounds; defaults to <see cref="TimeProvider.System"/>.
/// Exists so a test can step past a bound without waiting for it.</param>
/// <returns>A <see cref="Akka.Actor.Props"/> instance configured to create the wrapped <see cref="DriverInstanceActor"/>.</returns>
public static Props Props(
IDriver driver,
@@ -450,7 +475,8 @@ public sealed class DriverInstanceActor : ReceiveActor, IWithTimers
int? maxWritesInFlight = null,
TimeSpan? queuedWriteBudget = null,
TimeSpan? initializeTimeout = null,
TimeSpan? writeTimeout = null) =>
TimeSpan? writeTimeout = null,
TimeProvider? writeClock = null) =>
Akka.Actor.Props.Create(() => new DriverInstanceActor(
driver,
reconnectInterval ?? DefaultReconnectInterval,
@@ -464,7 +490,8 @@ public sealed class DriverInstanceActor : ReceiveActor, IWithTimers
maxWritesInFlight,
queuedWriteBudget,
initializeTimeout,
writeTimeout));
writeTimeout,
writeClock));
/// <summary>
/// Returns true when the driver should boot in DEV-STUB mode based on host platform and
@@ -509,8 +536,9 @@ public sealed class DriverInstanceActor : ReceiveActor, IWithTimers
/// <see cref="DefaultQueuedWriteBudget"/> when null or non-positive.</param>
/// <param name="initializeTimeout">Pessimistic bound on one initialise attempt; defaults to
/// <see cref="DefaultInitializeTimeout"/> when null or non-positive.</param>
/// <param name="writeTimeout">Pessimistic bound on one write round-trip; defaults to
/// <param name="writeTimeout">Bound on one write round-trip; defaults to
/// <see cref="DefaultWriteTimeout"/> when null or non-positive.</param>
/// <param name="writeClock">Clock for the write bounds; defaults to <see cref="TimeProvider.System"/>.</param>
public DriverInstanceActor(
IDriver driver,
TimeSpan reconnectInterval,
@@ -524,9 +552,11 @@ public sealed class DriverInstanceActor : ReceiveActor, IWithTimers
int? maxWritesInFlight = null,
TimeSpan? queuedWriteBudget = null,
TimeSpan? initializeTimeout = null,
TimeSpan? writeTimeout = null)
TimeSpan? writeTimeout = null,
TimeProvider? writeClock = null)
{
_writeTimeout = writeTimeout is { } wt && wt > TimeSpan.Zero ? wt : DefaultWriteTimeout;
_writeClock = writeClock ?? TimeProvider.System;
_incarnation = Interlocked.Increment(ref s_initGenerationSource);
_maxWritesInFlight = maxWritesInFlight is > 0 ? maxWritesInFlight.Value : DefaultMaxWritesInFlight;
_queuedWriteBudget = queuedWriteBudget is { } qb && qb > TimeSpan.Zero ? qb : DefaultQueuedWriteBudget;
@@ -1139,21 +1169,22 @@ public sealed class DriverInstanceActor : ReceiveActor, IWithTimers
var self = Self;
var hostKey = ResolveHost(msg.TagId);
var invoker = _invoker;
_ = RunWriteAsync(writable, invoker, hostKey, msg, replyTo, self, _writeTimeout, _incarnation, _log);
_ = RunWriteAsync(writable, invoker, hostKey, msg, replyTo, self, _writeTimeout, _writeClock, _incarnation, _log);
}
private static async Task RunWriteAsync(
IWritable writable, IDriverCapabilityInvoker invoker, string hostKey,
WriteAttribute msg, IActorRef replyTo, IActorRef self, TimeSpan writeTimeout, int incarnation,
ILoggingAdapter log)
WriteAttribute msg, IActorRef replyTo, IActorRef self, TimeSpan writeTimeout, TimeProvider clock,
int incarnation, ILoggingAdapter log)
{
// Off the actor context from here on: the first await yields, and nothing below reads actor state.
await Task.Yield();
var request = new[] { new WriteRequest(msg.TagId, msg.Value) };
// Bound the write so a hung backend can't hold a slot forever — retry stays
// off by default, but a stalled call still needs an answer. Disposed below, except when the write is
// abandoned (the orphan may still be holding the token).
var cts = new CancellationTokenSource(TimeSpan.FromSeconds(5));
// off by default, but a stalled call still needs an answer. The bound is the configured write timeout
// (lmxopcua#774: it used to be a hard-coded 5 s, so a longer configured bound never took effect).
// Disposed below, except when the write is abandoned (the orphan may still be holding the token).
var cts = new CancellationTokenSource(writeTimeout, clock);
var abandoned = false;
try
{
@@ -1164,7 +1195,7 @@ public sealed class DriverInstanceActor : ReceiveActor, IWithTimers
// therefore authoritative regardless of any configured Write retry. Per-tag opt-in to retries via
// WriteIdempotentAttribute is the unshipped forward path (see WriteIdempotentAttribute). The outer
// cts is retained as a hard backstop above the (typically tighter) tier timeout.
// PESSIMISTIC bound on top of the token-based cap: a driver that ignores its token would otherwise
// PESSIMISTIC bound a grace after the token-based cap: a driver that ignores its token would otherwise
// hold the slot and the tag forever, piling every later write to that tag up in the queue.
var attempt = invoker.ExecuteWriteAsync(
hostKey,
@@ -1174,15 +1205,15 @@ public sealed class DriverInstanceActor : ReceiveActor, IWithTimers
IReadOnlyList<WriteResult> results;
try
{
results = await attempt.WaitAsync(writeTimeout);
results = await attempt.WaitAsync(writeTimeout + WriteAbandonGrace(writeTimeout), clock);
}
catch (TimeoutException)
{
abandoned = true;
log.Warning(
"DriverInstance write of {Tag} did not complete within {Bound} (the driver ignored its cancellation " +
"token); abandoned — its slot and tag are released, its eventual outcome is discarded",
msg.TagId, writeTimeout);
"DriverInstance write of {Tag} did not complete within {Bound} and ignored its cancellation token " +
"for {Grace} more; abandoned — its slot and tag are released, its eventual outcome is discarded",
msg.TagId, writeTimeout, WriteAbandonGrace(writeTimeout));
try { cts.Cancel(); } catch (ObjectDisposedException) { }
_ = attempt.ContinueWith(t => _ = t.Exception, TaskContinuationOptions.OnlyOnFaulted);
replyTo.Tell(new WriteAttributeResult(false,