fix(runtime): derive every write bound and Ask from one write timeout (lmxopcua#777)
v2-ci / unit-tests (push) Has been cancelled

Three write bounds were configured independently: the inbound-write
gateway's 10 s Ask, DriverHostActor's fixed 8 s Ask of the child, and the
child's own write timeout (5 s + up to 2 s grace) with a fixed 8 s queue
budget. An outer Ask that fires first reports the write failed and the node
reverts while the child is still driving the device, so any write timeout
above ~6 s (or a write dequeued late in its 8 s budget) let the device take
a value the client was told failed.

DriverWriteBounds (Runtime) derives them all from DriverWrites:Timeout:
  token cancel   = Timeout
  abandon        = Timeout + min(Timeout/2, 2 s)            (RunBound)
  queue budget   = RunBound
  child replies  <= queue budget + RunBound                  (ReplyBound)
  host Ask       = ReplyBound + 1 s
  gateway Ask    = host Ask + 1 s
Default Timeout is 10 s (the Tier C Write timeout, so no tier default is
capped below its own policy); host Ask 25 s, gateway Ask 26 s. Tier A/B
writes are still cancelled first by their 2 s / 4 s tier timeouts.

- DriverHostActor takes writeBounds + writeClock, spawns children with them,
  and runs its Ask deadline as a token on the write clock (Infinite Akka
  timeout), so host and child bounds share one clock.
- WriteAttribute gains IssuedAt (write-clock timestamp, stamped by the
  host): the queue budget counts mailbox time too, and a write past it is
  failed BadTimeout and never started - including on arrival. Stale queued
  writes are expired across the whole queue, not only up to a free slot.
- ActorNodeWriteGateway's default Ask = Default.GatewayAsk; the hosted
  service passes the configured one.
- Startup validation (Host DriverWriteBoundsBinding, same warn-and-clamp
  style as ScriptingOptionsBinding): unparsable / non-positive -> default,
  above 5 min -> clamped, and each tier whose default Write timeout exceeds
  the write timeout is named in a Warning (its policy would never take
  effect). The derived bounds are logged at Information. Per-instance
  ResilienceConfig Write overrides arrive with a deployment, so
  DriverCapabilityInvokerFactory warns at spawn (writeTimeoutCeiling).
- appsettings.json documents DriverWrites:Timeout.

Tests (manual clock):
- DriverHostActorWriteAskTests: a held write under a 9 s timeout survives
  an 8.5 s step and succeeds; a token-ignoring write is answered by the
  child's BadTimeout at 9 s + grace, not by the Ask; the Infinite-timeout
  Ask ends exactly at its clock deadline.
- DriverInstanceActorConcurrentWriteTests: a write issued longer ago than
  the queue budget is failed and never reaches the driver; positive control
  at exactly the budget is written.
- DriverWriteBoundsTests: every outer bound covers the inner ones for 0.5 s
  to 300 s timeouts.
- DriverWriteBoundsBindingTests: default fits every tier; 3 s timeout is
  caught at startup with Tier B and C named (not A); bad values fall back
  with a warning; over-max clamps.
- DriverCapabilityInvokerFactoryTests: spawn warns when a Write override
  exceeds the ceiling; no warning within it or without a ceiling.
Mutation checks: a fixed 8 s host Ask on the same clock fails both 9 s host
tests; stamping arrival time instead of IssuedAt fails the issued-at test;
dropping the tier check / factory check fails their tests.

Runtime.Tests 961 passed / 3 skipped at AKKA_TEST_TIMEFACTOR 3 and 1;
Host.Tests 232/232 at 3 and 1; Core.Tests 361 passed / 1 skipped.
Solution Release build: 0 warnings.

ContinuousHistorizationRecorderTypedValueTests (historian) deliberately
left for the historian lane.

Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
This commit is contained in:
Joseph Doherty
2026-11-09 15:46:04 -05:00
parent b051bdc87c
commit 97dc715dac
16 changed files with 824 additions and 57 deletions
@@ -61,7 +61,15 @@ public sealed class DriverInstanceActor : ReceiveActor, IWithTimers
/// <param name="DriverInstanceId">The driver instance that was restarted.</param>
/// <param name="Reason">The type name of the exception that tripped supervision, for the host's log.</param>
public sealed record ReprovisionRequested(string DriverInstanceId, string Reason);
public sealed record WriteAttribute(string TagId, object Value);
/// <summary>An operator write of <paramref name="Value"/> to the tag <paramref name="TagId"/>.</summary>
/// <param name="TagId">The tag's driver wire reference (RawPath).</param>
/// <param name="Value">The value to write.</param>
/// <param name="IssuedAt">When the asker issued the write, as a timestamp of the actor's write clock
/// (<see cref="TimeProvider.GetTimestamp"/>); null = when the actor receives it. <see cref="DriverHostActor"/>
/// stamps it so the time the write spends in this actor's MAILBOX counts against the queue budget too: a write
/// delayed there (behind a hanging alarm subscribe, say) and then started could otherwise still be running
/// when the host's Ask fires (lmxopcua#777).</param>
public sealed record WriteAttribute(string TagId, object Value, long? IssuedAt = null);
public sealed record WriteAttributeResult(bool Success, string? Reason);
/// <summary>
/// Sent by <see cref="DriverHostActor"/> when an OPC UA client Acknowledges a NATIVE Part 9
@@ -216,10 +224,13 @@ public sealed class DriverInstanceActor : ReceiveActor, IWithTimers
/// <summary>
/// How long a queued write may wait for a slot before it is failed <c>BadTimeout</c> instead of being
/// written. The host's write Ask is 8 s: a write older than that has already been reported as a timeout
/// (and its node reverted) upstream, so writing it late would push a stale value to the device.
/// written, when no explicit budget is given: <see cref="DriverWriteBounds.QueueBudget"/> of the write
/// timeout in force (lmxopcua#777 — it was a fixed 8 s, equal to the host's whole Ask, so a write dequeued
/// just inside it could still be running when the Ask fired). The host's Ask covers this budget PLUS a full
/// write (<see cref="DriverWriteBounds.HostAsk"/>), so a write started inside the budget is always answered
/// before the host reports it failed.
/// </summary>
public static readonly TimeSpan DefaultQueuedWriteBudget = TimeSpan.FromSeconds(8);
public static readonly TimeSpan DefaultQueuedWriteBudget = DriverWriteBounds.Default.QueueBudget;
/// <summary>
/// Pessimistic bound on one <see cref="IDriver.InitializeAsync"/> attempt. The invoker's tier timeout is
@@ -236,23 +247,19 @@ public sealed class DriverInstanceActor : ReceiveActor, IWithTimers
/// queue.
/// </summary>
/// <remarks>
/// 5 s, so both bounds (5 s cancel, 7 s abandon) answer inside the host's 8 s write Ask
/// (<c>DriverHostActor</c>): a write that outlives the Ask has already been reported as a timeout and its
/// node reverted upstream, so letting it run on would let the device take a value the client was told
/// failed. Until lmxopcua#774 the cancel was a hard-coded 5 s that ignored this value, and the default was
/// 10 s — a configured bound above 5 s never took effect.
/// <see cref="DriverWriteBounds.DefaultTimeout"/>. The host's write Ask is DERIVED from the same value
/// (<see cref="DriverWriteBounds.HostAsk"/>, lmxopcua#777), so any configured write timeout is answered
/// inside it; until then the Ask was a fixed 8 s and a write timeout above ~6 s let a write outlive it.
/// </remarks>
public static readonly TimeSpan DefaultWriteTimeout = TimeSpan.FromSeconds(5);
public static readonly TimeSpan DefaultWriteTimeout = DriverWriteBounds.DefaultTimeout;
/// <summary>
/// How long after a write's token is cancelled (<see cref="DefaultWriteTimeout"/>) a write that has not
/// returned is abandoned as token-ignoring: half the write timeout, at most 2 s. Long enough for a driver
/// that honours its token to unwind and answer the cancellation itself.
/// How long after a write's token is cancelled a write that has not returned is abandoned as
/// token-ignoring; see <see cref="DriverWriteBounds.AbandonGrace"/>.
/// </summary>
/// <param name="writeTimeout">The write timeout in force.</param>
/// <returns>The grace between cancelling the write's token and abandoning the write.</returns>
internal static TimeSpan WriteAbandonGrace(TimeSpan writeTimeout) =>
TimeSpan.FromTicks(Math.Min(writeTimeout.Ticks / 2, TimeSpan.FromSeconds(2).Ticks));
internal static TimeSpan WriteAbandonGrace(TimeSpan writeTimeout) => DriverWriteBounds.AbandonGraceFor(writeTimeout);
/// <summary>Self-sent by a write task once it has replied, releasing its slot and its tag.</summary>
/// <param name="TagId">The tag the write held.</param>
@@ -292,8 +299,8 @@ public sealed class DriverInstanceActor : ReceiveActor, IWithTimers
/// <summary>Writes waiting for a slot (or for their tag), in arrival order, stamped with arrival time.
/// Failed fast on every exit from <c>Connected</c>, so a queued write never reaches a driver that is
/// reconnecting; failed <c>BadTimeout</c> at dequeue once older than <see cref="_queuedWriteBudget"/>.</summary>
private readonly LinkedList<(WriteAttribute Write, IActorRef ReplyTo, DateTime EnqueuedUtc)> _pendingWrites = new();
/// reconnecting; failed <c>BadTimeout</c> once older than <see cref="_queuedWriteBudget"/> since issued.</summary>
private readonly LinkedList<(WriteAttribute Write, IActorRef ReplyTo, long IssuedAt)> _pendingWrites = new();
/// <summary>Tags with a write currently inside the driver. A second write to the same tag waits in the
/// queue behind it — two concurrent writes v1, v2 to one tag could otherwise land as v1.</summary>
@@ -453,7 +460,7 @@ public sealed class DriverInstanceActor : ReceiveActor, IWithTimers
/// <param name="maxWritesInFlight">Optional cap on concurrent writes against the driver; defaults to
/// <see cref="DefaultMaxWritesInFlight"/>. Writes beyond it queue in arrival order.</param>
/// <param name="queuedWriteBudget">Optional bound on how long a queued write may wait before it is failed
/// <c>BadTimeout</c>; defaults to <see cref="DefaultQueuedWriteBudget"/>.</param>
/// <c>BadTimeout</c>; defaults to <see cref="DriverWriteBounds.QueueBudget"/> of the write timeout in force.</param>
/// <param name="initializeTimeout">Optional pessimistic bound on one initialise attempt; defaults to
/// <see cref="DefaultInitializeTimeout"/>.</param>
/// <param name="writeTimeout">Optional bound on one write round-trip (its token is cancelled when it elapses;
@@ -533,7 +540,7 @@ public sealed class DriverInstanceActor : ReceiveActor, IWithTimers
/// <param name="maxWritesInFlight">Cap on concurrent writes; defaults to
/// <see cref="DefaultMaxWritesInFlight"/> when null or non-positive.</param>
/// <param name="queuedWriteBudget">Bound on a queued write's wait; defaults to
/// <see cref="DefaultQueuedWriteBudget"/> when null or non-positive.</param>
/// <see cref="DriverWriteBounds.QueueBudget"/> of the write timeout when null or non-positive.</param>
/// <param name="initializeTimeout">Pessimistic bound on one initialise attempt; defaults to
/// <see cref="DefaultInitializeTimeout"/> when null or non-positive.</param>
/// <param name="writeTimeout">Bound on one write round-trip; defaults to
@@ -559,7 +566,9 @@ public sealed class DriverInstanceActor : ReceiveActor, IWithTimers
_writeClock = writeClock ?? TimeProvider.System;
_incarnation = Interlocked.Increment(ref s_initGenerationSource);
_maxWritesInFlight = maxWritesInFlight is > 0 ? maxWritesInFlight.Value : DefaultMaxWritesInFlight;
_queuedWriteBudget = queuedWriteBudget is { } qb && qb > TimeSpan.Zero ? qb : DefaultQueuedWriteBudget;
_queuedWriteBudget = queuedWriteBudget is { } qb && qb > TimeSpan.Zero
? qb
: new DriverWriteBounds(_writeTimeout).QueueBudget;
_initializeTimeout = initializeTimeout is { } it && it > TimeSpan.Zero ? it : DefaultInitializeTimeout;
_driver = driver;
_invoker = invoker ?? NullDriverCapabilityInvoker.Instance;
@@ -646,7 +655,7 @@ public sealed class DriverInstanceActor : ReceiveActor, IWithTimers
{
Receive<InitializeRequested>(msg => InitializeAsync(msg.DriverConfigJson));
// Fast-fail writes while still connecting — without this the inbound WriteAttribute dead-letters
// and DriverHostActor.HandleRouteNodeWrite waits its full 8s Ask before reporting a generic
// and DriverHostActor.HandleRouteNodeWrite waits its full Ask before reporting a generic
// "write timeout". Synchronous Receive: Sender.Tell on the actor thread is safe.
Receive<WriteAttribute>(_ =>
Sender.Tell(new WriteAttributeResult(false, "driver not connected")));
@@ -919,7 +928,7 @@ public sealed class DriverInstanceActor : ReceiveActor, IWithTimers
/// non-empty desired set, so it costs nothing on a driver with no alarm tags.
/// <para>A persistently failing subscribe backs off: after the n-th consecutive failure the reconcile
/// skips 2^(n-1) − 1 ticks, capped at <see cref="MaxAlarmRetrySkipTicks"/>, because a HANGING subscribe
/// holds the mailbox for its full 10 s bound (writes queued behind it can miss the host's 8 s Ask). A
/// holds the mailbox for its full 10 s bound (writes delayed behind it eat into their queue budget). A
/// Connected entry or a deploy's desired-set push still retries immediately — only the timer backs off.</para>
/// </summary>
private void ReconcileAlarmSubscription()
@@ -1151,9 +1160,13 @@ public sealed class DriverInstanceActor : ReceiveActor, IWithTimers
return;
}
// Monotonic, so a wall-clock step cannot expire (or extend) a queued write.
var issuedAt = msg.IssuedAt ?? _writeClock.GetTimestamp();
if (TryExpireWrite(replyTo, issuedAt)) return;
if (_writesInFlight >= _maxWritesInFlight || _tagsInFlight.Contains(msg.TagId))
{
_pendingWrites.AddLast((msg, replyTo, DateTime.UtcNow));
_pendingWrites.AddLast((msg, replyTo, issuedAt));
return;
}
@@ -1258,21 +1271,22 @@ public sealed class DriverInstanceActor : ReceiveActor, IWithTimers
private void DrainWriteQueue()
{
if (_driver is not IWritable writable) return;
var now = DateTime.UtcNow;
var node = _pendingWrites.First;
while (node is not null && _writesInFlight < _maxWritesInFlight)
// Expire FIRST, over the whole queue, regardless of free slots: a write past its budget must be answered
// now, not whenever a slot next frees and the scan happens to reach it — the host's Ask covers the budget
// plus one write (DriverWriteBounds.HostAsk), not an unbounded wait behind a full set of slots.
for (var node = _pendingWrites.First; node is not null;)
{
var next = node.Next;
var (write, replyTo, enqueuedUtc) = node.Value;
if (now - enqueuedUtc > _queuedWriteBudget)
{
// Already reported as a timeout upstream (the host's Ask is the budget) and the node reverted —
// writing it now would push a stale value to the device. BadTimeout, never written.
_pendingWrites.Remove(node);
replyTo.Tell(new WriteAttributeResult(false,
$"StatusCode=0x800A0000 (BadTimeout: queued for {(now - enqueuedUtc).TotalSeconds:F1}s, over the {_queuedWriteBudget.TotalSeconds:F0}s write budget)"));
}
else if (!_tagsInFlight.Contains(write.TagId))
if (TryExpireWrite(node.Value.ReplyTo, node.Value.IssuedAt)) _pendingWrites.Remove(node);
node = next;
}
for (var node = _pendingWrites.First; node is not null && _writesInFlight < _maxWritesInFlight;)
{
var next = node.Next;
var (write, replyTo, _) = node.Value;
if (!_tagsInFlight.Contains(write.TagId))
{
_pendingWrites.Remove(node);
LaunchWrite(writable, write, replyTo);
@@ -1282,6 +1296,23 @@ public sealed class DriverInstanceActor : ReceiveActor, IWithTimers
}
}
/// <summary>
/// Fails a write that has waited longer than <see cref="_queuedWriteBudget"/> since it was issued (in the
/// mailbox or the queue). Only a write inside the budget may start: the host's Ask covers the budget plus one
/// full write (<see cref="DriverWriteBounds.HostAsk"/>), so a write started later could still be running
/// when the Ask reports it failed and its node is reverted — and the device could then take a value the
/// client was told failed. BadTimeout, never written.
/// </summary>
/// <returns>True when the write was expired (and answered).</returns>
private bool TryExpireWrite(IActorRef replyTo, long issuedAt)
{
var waited = _writeClock.GetElapsedTime(issuedAt);
if (waited <= _queuedWriteBudget) return false;
replyTo.Tell(new WriteAttributeResult(false,
$"StatusCode=0x800A0000 (BadTimeout: queued for {waited.TotalSeconds:F1}s, over the {_queuedWriteBudget.TotalSeconds:F0}s write budget)"));
return true;
}
/// <summary>Replies to every queued write with <paramref name="reason"/>; called on each exit from
/// <c>Connected</c>. In-flight writes are left to finish — they already hold the device and reply with its
/// real outcome.</summary>