fix(runtime): derive every write bound and Ask from one write timeout (lmxopcua#777)
v2-ci / unit-tests (push) Has been cancelled
v2-ci / unit-tests (push) Has been cancelled
Three write bounds were configured independently: the inbound-write gateway's 10 s Ask, DriverHostActor's fixed 8 s Ask of the child, and the child's own write timeout (5 s + up to 2 s grace) with a fixed 8 s queue budget. An outer Ask that fires first reports the write failed and the node reverts while the child is still driving the device, so any write timeout above ~6 s (or a write dequeued late in its 8 s budget) let the device take a value the client was told failed. DriverWriteBounds (Runtime) derives them all from DriverWrites:Timeout: token cancel = Timeout abandon = Timeout + min(Timeout/2, 2 s) (RunBound) queue budget = RunBound child replies <= queue budget + RunBound (ReplyBound) host Ask = ReplyBound + 1 s gateway Ask = host Ask + 1 s Default Timeout is 10 s (the Tier C Write timeout, so no tier default is capped below its own policy); host Ask 25 s, gateway Ask 26 s. Tier A/B writes are still cancelled first by their 2 s / 4 s tier timeouts. - DriverHostActor takes writeBounds + writeClock, spawns children with them, and runs its Ask deadline as a token on the write clock (Infinite Akka timeout), so host and child bounds share one clock. - WriteAttribute gains IssuedAt (write-clock timestamp, stamped by the host): the queue budget counts mailbox time too, and a write past it is failed BadTimeout and never started - including on arrival. Stale queued writes are expired across the whole queue, not only up to a free slot. - ActorNodeWriteGateway's default Ask = Default.GatewayAsk; the hosted service passes the configured one. - Startup validation (Host DriverWriteBoundsBinding, same warn-and-clamp style as ScriptingOptionsBinding): unparsable / non-positive -> default, above 5 min -> clamped, and each tier whose default Write timeout exceeds the write timeout is named in a Warning (its policy would never take effect). The derived bounds are logged at Information. Per-instance ResilienceConfig Write overrides arrive with a deployment, so DriverCapabilityInvokerFactory warns at spawn (writeTimeoutCeiling). - appsettings.json documents DriverWrites:Timeout. Tests (manual clock): - DriverHostActorWriteAskTests: a held write under a 9 s timeout survives an 8.5 s step and succeeds; a token-ignoring write is answered by the child's BadTimeout at 9 s + grace, not by the Ask; the Infinite-timeout Ask ends exactly at its clock deadline. - DriverInstanceActorConcurrentWriteTests: a write issued longer ago than the queue budget is failed and never reaches the driver; positive control at exactly the budget is written. - DriverWriteBoundsTests: every outer bound covers the inner ones for 0.5 s to 300 s timeouts. - DriverWriteBoundsBindingTests: default fits every tier; 3 s timeout is caught at startup with Tier B and C named (not A); bad values fall back with a warning; over-max clamps. - DriverCapabilityInvokerFactoryTests: spawn warns when a Write override exceeds the ceiling; no warning within it or without a ceiling. Mutation checks: a fixed 8 s host Ask on the same clock fails both 9 s host tests; stamping arrival time instead of IssuedAt fails the issued-at test; dropping the tier check / factory check fails their tests. Runtime.Tests 961 passed / 3 skipped at AKKA_TEST_TIMEFACTOR 3 and 1; Host.Tests 232/232 at 3 and 1; Core.Tests 361 passed / 1 skipped. Solution Release build: 0 warnings. ContinuousHistorizationRecorderTypedValueTests (historian) deliberately left for the historian lane. Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
This commit is contained in:
@@ -22,8 +22,6 @@ namespace ZB.MOM.WW.OtOpcUa.Runtime.Drivers;
|
||||
/// </summary>
|
||||
public sealed class ActorNodeWriteGateway : IOpcUaNodeWriteGateway
|
||||
{
|
||||
/// <summary>Default Ask timeout — matches the legacy inline lambda in the hosted service.</summary>
|
||||
private static readonly TimeSpan DefaultAskTimeout = TimeSpan.FromSeconds(10);
|
||||
|
||||
private readonly Func<IActorRef?> _resolveDriverHost;
|
||||
private readonly TimeSpan _askTimeout;
|
||||
@@ -34,12 +32,15 @@ public sealed class ActorNodeWriteGateway : IOpcUaNodeWriteGateway
|
||||
/// returns null until the actor has registered (StartAsync ordering — the actor registers AFTER the host
|
||||
/// wires this gateway).</param>
|
||||
/// <param name="logger">Logger for dropped/rejected/timed-out writes.</param>
|
||||
/// <param name="askTimeout">Ask timeout; defaults to 10s (the legacy lambda's value).</param>
|
||||
/// <param name="askTimeout">Ask timeout; defaults to <see cref="DriverWriteBounds.GatewayAsk"/> of
|
||||
/// <see cref="DriverWriteBounds.Default"/>. Production passes the GatewayAsk of the node's configured write
|
||||
/// bounds, so this Ask always outlasts the host's (lmxopcua#777) — an outer Ask that fires first reports the
|
||||
/// write failed, and the node reverts, while the device may still take the value.</param>
|
||||
public ActorNodeWriteGateway(Func<IActorRef?> resolveDriverHost, ILogger logger, TimeSpan? askTimeout = null)
|
||||
{
|
||||
_resolveDriverHost = resolveDriverHost ?? throw new ArgumentNullException(nameof(resolveDriverHost));
|
||||
_logger = logger ?? throw new ArgumentNullException(nameof(logger));
|
||||
_askTimeout = askTimeout ?? DefaultAskTimeout;
|
||||
_askTimeout = askTimeout ?? DriverWriteBounds.Default.GatewayAsk;
|
||||
}
|
||||
|
||||
/// <inheritdoc />
|
||||
|
||||
@@ -479,6 +479,14 @@ public sealed class DriverHostActor : ReceiveActor, IWithTimers, IWithUnboundedS
|
||||
/// <summary>Where the synchronous native-ack pre-check snapshot is published (lmxopcua#760); null ⇒ none.</summary>
|
||||
private readonly NativeAlarmAckPreflightView? _ackPreflightView;
|
||||
|
||||
/// <summary>The write bounds every child is spawned with and this host's write Ask is derived from
|
||||
/// (lmxopcua#777).</summary>
|
||||
private readonly DriverWriteBounds _writeBounds;
|
||||
|
||||
/// <summary>The clock the write bounds run on, shared with every child so a write's issue stamp
|
||||
/// (<see cref="DriverInstanceActor.WriteAttribute.IssuedAt"/>) means the same thing on both sides.</summary>
|
||||
private readonly TimeProvider _writeClock;
|
||||
|
||||
/// <summary>Whether any redundancy snapshot has reached this host — the HistoryUpdate gate stays
|
||||
/// closed until one has (see <see cref="PrimaryGatePolicy.ShouldServeHistoryUpdate"/>).</summary>
|
||||
private bool _redundancySnapshotReceived;
|
||||
@@ -721,6 +729,11 @@ public sealed class DriverHostActor : ReceiveActor, IWithTimers, IWithUnboundedS
|
||||
/// HistoryUpdate Primary gate. Null on nodes that wire no redundancy manager.</param>
|
||||
/// <param name="scripting">The node-wide <c>Scripting</c> run-guard options handed to the scripted-alarm
|
||||
/// engine. Null ⇒ the compiled defaults.</param>
|
||||
/// <param name="ackPreflightView">Where the native-ack pre-check snapshot is published. Null ⇒ none.</param>
|
||||
/// <param name="writeBounds">The write timeout every child is spawned with, and from which this host's write
|
||||
/// Ask is derived (<see cref="DriverWriteBounds.HostAsk"/>). Null ⇒ <see cref="DriverWriteBounds.Default"/>.</param>
|
||||
/// <param name="writeClock">The clock the write bounds (and the host's write Ask) run on, shared with every
|
||||
/// child. Null ⇒ <see cref="TimeProvider.System"/>.</param>
|
||||
/// <returns>The Akka.NET <see cref="Akka.Actor.Props"/> used to spawn this actor.</returns>
|
||||
public static Props Props(
|
||||
IDbContextFactory<OtOpcUaConfigDbContext>? dbFactory,
|
||||
@@ -756,7 +769,9 @@ public sealed class DriverHostActor : ReceiveActor, IWithTimers, IWithUnboundedS
|
||||
bool captureWhilePrimaryUnreachable = true,
|
||||
IHistoryUpdateRoleView? historyUpdateRoleView = null,
|
||||
ScriptingOptions? scripting = null,
|
||||
NativeAlarmAckPreflightView? ackPreflightView = null) =>
|
||||
NativeAlarmAckPreflightView? ackPreflightView = null,
|
||||
DriverWriteBounds? writeBounds = null,
|
||||
TimeProvider? writeClock = null) =>
|
||||
// WARNING: this forwarding list is POSITIONAL, and Props.Create compiles it into an
|
||||
// expression tree. Six IActorRef? parameters and several interface-typed ones mean a
|
||||
// mis-ordered argument is usually type-compatible and therefore compiles clean, then binds
|
||||
@@ -770,7 +785,7 @@ public sealed class DriverHostActor : ReceiveActor, IWithTimers, IWithUnboundedS
|
||||
alarmStateStore, telemetryHub, primaryRoleSource, historizationRoleView,
|
||||
sharesHistorizationScopeWithLocal, applyWedgeTimeout, reprovisionBackoff, commsRecoveryAuditDelay,
|
||||
sharesAlarmQueueWithLocal, applyOutcomeTimeout, captureWhilePrimaryUnreachable, historyUpdateRoleView,
|
||||
scripting, ackPreflightView));
|
||||
scripting, ackPreflightView, writeBounds, writeClock));
|
||||
|
||||
/// <summary>Initializes a new DriverHostActor with the specified dependencies.</summary>
|
||||
/// <param name="dbFactory">Database context factory for configuration database access.</param>
|
||||
@@ -861,6 +876,10 @@ public sealed class DriverHostActor : ReceiveActor, IWithTimers, IWithUnboundedS
|
||||
/// engine. Null ⇒ the compiled defaults.</param>
|
||||
/// <param name="ackPreflightView">Where the condition → owning-driver native-ack pre-check snapshot is published
|
||||
/// for the node manager's synchronous refusal (lmxopcua#760). Null ⇒ nothing published (no pre-check).</param>
|
||||
/// <param name="writeBounds">The write timeout every child is spawned with, and from which this host's write
|
||||
/// Ask is derived (<see cref="DriverWriteBounds.HostAsk"/>). Null ⇒ <see cref="DriverWriteBounds.Default"/>.</param>
|
||||
/// <param name="writeClock">The clock the write bounds (and the host's write Ask) run on, shared with every
|
||||
/// child. Null ⇒ <see cref="TimeProvider.System"/>.</param>
|
||||
public DriverHostActor(
|
||||
IDbContextFactory<OtOpcUaConfigDbContext>? dbFactory,
|
||||
CommonsNodeId localNode,
|
||||
@@ -895,8 +914,12 @@ public sealed class DriverHostActor : ReceiveActor, IWithTimers, IWithUnboundedS
|
||||
bool captureWhilePrimaryUnreachable = true,
|
||||
IHistoryUpdateRoleView? historyUpdateRoleView = null,
|
||||
ScriptingOptions? scripting = null,
|
||||
NativeAlarmAckPreflightView? ackPreflightView = null)
|
||||
NativeAlarmAckPreflightView? ackPreflightView = null,
|
||||
DriverWriteBounds? writeBounds = null,
|
||||
TimeProvider? writeClock = null)
|
||||
{
|
||||
_writeBounds = writeBounds ?? DriverWriteBounds.Default;
|
||||
_writeClock = writeClock ?? TimeProvider.System;
|
||||
_ackPreflightView = ackPreflightView;
|
||||
_scripting = scripting;
|
||||
_captureWhilePrimaryUnreachable = captureWhilePrimaryUnreachable;
|
||||
@@ -2237,17 +2260,30 @@ public sealed class DriverHostActor : ReceiveActor, IWithTimers, IWithUnboundedS
|
||||
return;
|
||||
}
|
||||
|
||||
// Ask the child a bounded write — DriverInstanceActor bounds the backend call by its write timeout
|
||||
// (DefaultWriteTimeout 5 s; a token-ignoring write is abandoned at 7 s), so the 8s Ask is a safety net
|
||||
// for an actor that never replies. Capture Sender before
|
||||
// the continuation: it runs off the actor thread, where raw Sender is unsafe to read.
|
||||
// Ask the child a bounded write. The child answers every write within DriverWriteBounds.ReplyBound of the
|
||||
// IssuedAt stamped here (queue budget + write timeout + abandon grace — it refuses to START a write that
|
||||
// could not finish inside it), and the Ask is DERIVED from the same write timeout with a margin on top
|
||||
// (lmxopcua#777). So the Ask is only a safety net for a child that never replies: it can no longer fire
|
||||
// first and report a write failed — reverting the node — while the child is still driving the device. It
|
||||
// was a fixed 8 s, and any write timeout above ~6 s let a write outlive it.
|
||||
//
|
||||
// The Ask's deadline is a token on the write clock rather than Akka's scheduler, so it runs on the same
|
||||
// clock as the child's bounds (and a test can step both together). Capture Sender before the
|
||||
// continuation: it runs off the actor thread, where raw Sender is unsafe to read.
|
||||
var replyTo = Sender;
|
||||
var askDeadline = new CancellationTokenSource(_writeBounds.HostAsk, _writeClock);
|
||||
entry.Actor.Ask<DriverInstanceActor.WriteAttributeResult>(
|
||||
new DriverInstanceActor.WriteAttribute(target.RawPath, msg.Value!), TimeSpan.FromSeconds(8))
|
||||
new DriverInstanceActor.WriteAttribute(target.RawPath, msg.Value!, _writeClock.GetTimestamp()),
|
||||
System.Threading.Timeout.InfiniteTimeSpan,
|
||||
askDeadline.Token)
|
||||
.ContinueWith(
|
||||
t => t.IsCompletedSuccessfully
|
||||
? new NodeWriteResult(t.Result.Success, t.Result.Reason)
|
||||
: new NodeWriteResult(false, "write timeout"),
|
||||
t =>
|
||||
{
|
||||
askDeadline.Dispose();
|
||||
return t.IsCompletedSuccessfully
|
||||
? new NodeWriteResult(t.Result.Success, t.Result.Reason)
|
||||
: new NodeWriteResult(false, "write timeout");
|
||||
},
|
||||
CancellationToken.None,
|
||||
TaskContinuationOptions.None,
|
||||
TaskScheduler.Default)
|
||||
@@ -3890,7 +3926,9 @@ public sealed class DriverHostActor : ReceiveActor, IWithTimers, IWithUnboundedS
|
||||
driver!,
|
||||
healthPublisher: _healthPublisher,
|
||||
clusterId: clusterId,
|
||||
invoker: invoker),
|
||||
invoker: invoker,
|
||||
writeTimeout: _writeBounds.Timeout,
|
||||
writeClock: _writeClock),
|
||||
actorName);
|
||||
child.Tell(new DriverInstanceActor.InitializeRequested(spec.DriverConfig));
|
||||
}
|
||||
|
||||
@@ -61,7 +61,15 @@ public sealed class DriverInstanceActor : ReceiveActor, IWithTimers
|
||||
/// <param name="DriverInstanceId">The driver instance that was restarted.</param>
|
||||
/// <param name="Reason">The type name of the exception that tripped supervision, for the host's log.</param>
|
||||
public sealed record ReprovisionRequested(string DriverInstanceId, string Reason);
|
||||
public sealed record WriteAttribute(string TagId, object Value);
|
||||
/// <summary>An operator write of <paramref name="Value"/> to the tag <paramref name="TagId"/>.</summary>
|
||||
/// <param name="TagId">The tag's driver wire reference (RawPath).</param>
|
||||
/// <param name="Value">The value to write.</param>
|
||||
/// <param name="IssuedAt">When the asker issued the write, as a timestamp of the actor's write clock
|
||||
/// (<see cref="TimeProvider.GetTimestamp"/>); null = when the actor receives it. <see cref="DriverHostActor"/>
|
||||
/// stamps it so the time the write spends in this actor's MAILBOX counts against the queue budget too: a write
|
||||
/// delayed there (behind a hanging alarm subscribe, say) and then started could otherwise still be running
|
||||
/// when the host's Ask fires (lmxopcua#777).</param>
|
||||
public sealed record WriteAttribute(string TagId, object Value, long? IssuedAt = null);
|
||||
public sealed record WriteAttributeResult(bool Success, string? Reason);
|
||||
/// <summary>
|
||||
/// Sent by <see cref="DriverHostActor"/> when an OPC UA client Acknowledges a NATIVE Part 9
|
||||
@@ -216,10 +224,13 @@ public sealed class DriverInstanceActor : ReceiveActor, IWithTimers
|
||||
|
||||
/// <summary>
|
||||
/// How long a queued write may wait for a slot before it is failed <c>BadTimeout</c> instead of being
|
||||
/// written. The host's write Ask is 8 s: a write older than that has already been reported as a timeout
|
||||
/// (and its node reverted) upstream, so writing it late would push a stale value to the device.
|
||||
/// written, when no explicit budget is given: <see cref="DriverWriteBounds.QueueBudget"/> of the write
|
||||
/// timeout in force (lmxopcua#777 — it was a fixed 8 s, equal to the host's whole Ask, so a write dequeued
|
||||
/// just inside it could still be running when the Ask fired). The host's Ask covers this budget PLUS a full
|
||||
/// write (<see cref="DriverWriteBounds.HostAsk"/>), so a write started inside the budget is always answered
|
||||
/// before the host reports it failed.
|
||||
/// </summary>
|
||||
public static readonly TimeSpan DefaultQueuedWriteBudget = TimeSpan.FromSeconds(8);
|
||||
public static readonly TimeSpan DefaultQueuedWriteBudget = DriverWriteBounds.Default.QueueBudget;
|
||||
|
||||
/// <summary>
|
||||
/// Pessimistic bound on one <see cref="IDriver.InitializeAsync"/> attempt. The invoker's tier timeout is
|
||||
@@ -236,23 +247,19 @@ public sealed class DriverInstanceActor : ReceiveActor, IWithTimers
|
||||
/// queue.
|
||||
/// </summary>
|
||||
/// <remarks>
|
||||
/// 5 s, so both bounds (5 s cancel, 7 s abandon) answer inside the host's 8 s write Ask
|
||||
/// (<c>DriverHostActor</c>): a write that outlives the Ask has already been reported as a timeout and its
|
||||
/// node reverted upstream, so letting it run on would let the device take a value the client was told
|
||||
/// failed. Until lmxopcua#774 the cancel was a hard-coded 5 s that ignored this value, and the default was
|
||||
/// 10 s — a configured bound above 5 s never took effect.
|
||||
/// <see cref="DriverWriteBounds.DefaultTimeout"/>. The host's write Ask is DERIVED from the same value
|
||||
/// (<see cref="DriverWriteBounds.HostAsk"/>, lmxopcua#777), so any configured write timeout is answered
|
||||
/// inside it; until then the Ask was a fixed 8 s and a write timeout above ~6 s let a write outlive it.
|
||||
/// </remarks>
|
||||
public static readonly TimeSpan DefaultWriteTimeout = TimeSpan.FromSeconds(5);
|
||||
public static readonly TimeSpan DefaultWriteTimeout = DriverWriteBounds.DefaultTimeout;
|
||||
|
||||
/// <summary>
|
||||
/// How long after a write's token is cancelled (<see cref="DefaultWriteTimeout"/>) a write that has not
|
||||
/// returned is abandoned as token-ignoring: half the write timeout, at most 2 s. Long enough for a driver
|
||||
/// that honours its token to unwind and answer the cancellation itself.
|
||||
/// How long after a write's token is cancelled a write that has not returned is abandoned as
|
||||
/// token-ignoring; see <see cref="DriverWriteBounds.AbandonGrace"/>.
|
||||
/// </summary>
|
||||
/// <param name="writeTimeout">The write timeout in force.</param>
|
||||
/// <returns>The grace between cancelling the write's token and abandoning the write.</returns>
|
||||
internal static TimeSpan WriteAbandonGrace(TimeSpan writeTimeout) =>
|
||||
TimeSpan.FromTicks(Math.Min(writeTimeout.Ticks / 2, TimeSpan.FromSeconds(2).Ticks));
|
||||
internal static TimeSpan WriteAbandonGrace(TimeSpan writeTimeout) => DriverWriteBounds.AbandonGraceFor(writeTimeout);
|
||||
|
||||
/// <summary>Self-sent by a write task once it has replied, releasing its slot and its tag.</summary>
|
||||
/// <param name="TagId">The tag the write held.</param>
|
||||
@@ -292,8 +299,8 @@ public sealed class DriverInstanceActor : ReceiveActor, IWithTimers
|
||||
|
||||
/// <summary>Writes waiting for a slot (or for their tag), in arrival order, stamped with arrival time.
|
||||
/// Failed fast on every exit from <c>Connected</c>, so a queued write never reaches a driver that is
|
||||
/// reconnecting; failed <c>BadTimeout</c> at dequeue once older than <see cref="_queuedWriteBudget"/>.</summary>
|
||||
private readonly LinkedList<(WriteAttribute Write, IActorRef ReplyTo, DateTime EnqueuedUtc)> _pendingWrites = new();
|
||||
/// reconnecting; failed <c>BadTimeout</c> once older than <see cref="_queuedWriteBudget"/> since issued.</summary>
|
||||
private readonly LinkedList<(WriteAttribute Write, IActorRef ReplyTo, long IssuedAt)> _pendingWrites = new();
|
||||
|
||||
/// <summary>Tags with a write currently inside the driver. A second write to the same tag waits in the
|
||||
/// queue behind it — two concurrent writes v1, v2 to one tag could otherwise land as v1.</summary>
|
||||
@@ -453,7 +460,7 @@ public sealed class DriverInstanceActor : ReceiveActor, IWithTimers
|
||||
/// <param name="maxWritesInFlight">Optional cap on concurrent writes against the driver; defaults to
|
||||
/// <see cref="DefaultMaxWritesInFlight"/>. Writes beyond it queue in arrival order.</param>
|
||||
/// <param name="queuedWriteBudget">Optional bound on how long a queued write may wait before it is failed
|
||||
/// <c>BadTimeout</c>; defaults to <see cref="DefaultQueuedWriteBudget"/>.</param>
|
||||
/// <c>BadTimeout</c>; defaults to <see cref="DriverWriteBounds.QueueBudget"/> of the write timeout in force.</param>
|
||||
/// <param name="initializeTimeout">Optional pessimistic bound on one initialise attempt; defaults to
|
||||
/// <see cref="DefaultInitializeTimeout"/>.</param>
|
||||
/// <param name="writeTimeout">Optional bound on one write round-trip (its token is cancelled when it elapses;
|
||||
@@ -533,7 +540,7 @@ public sealed class DriverInstanceActor : ReceiveActor, IWithTimers
|
||||
/// <param name="maxWritesInFlight">Cap on concurrent writes; defaults to
|
||||
/// <see cref="DefaultMaxWritesInFlight"/> when null or non-positive.</param>
|
||||
/// <param name="queuedWriteBudget">Bound on a queued write's wait; defaults to
|
||||
/// <see cref="DefaultQueuedWriteBudget"/> when null or non-positive.</param>
|
||||
/// <see cref="DriverWriteBounds.QueueBudget"/> of the write timeout when null or non-positive.</param>
|
||||
/// <param name="initializeTimeout">Pessimistic bound on one initialise attempt; defaults to
|
||||
/// <see cref="DefaultInitializeTimeout"/> when null or non-positive.</param>
|
||||
/// <param name="writeTimeout">Bound on one write round-trip; defaults to
|
||||
@@ -559,7 +566,9 @@ public sealed class DriverInstanceActor : ReceiveActor, IWithTimers
|
||||
_writeClock = writeClock ?? TimeProvider.System;
|
||||
_incarnation = Interlocked.Increment(ref s_initGenerationSource);
|
||||
_maxWritesInFlight = maxWritesInFlight is > 0 ? maxWritesInFlight.Value : DefaultMaxWritesInFlight;
|
||||
_queuedWriteBudget = queuedWriteBudget is { } qb && qb > TimeSpan.Zero ? qb : DefaultQueuedWriteBudget;
|
||||
_queuedWriteBudget = queuedWriteBudget is { } qb && qb > TimeSpan.Zero
|
||||
? qb
|
||||
: new DriverWriteBounds(_writeTimeout).QueueBudget;
|
||||
_initializeTimeout = initializeTimeout is { } it && it > TimeSpan.Zero ? it : DefaultInitializeTimeout;
|
||||
_driver = driver;
|
||||
_invoker = invoker ?? NullDriverCapabilityInvoker.Instance;
|
||||
@@ -646,7 +655,7 @@ public sealed class DriverInstanceActor : ReceiveActor, IWithTimers
|
||||
{
|
||||
Receive<InitializeRequested>(msg => InitializeAsync(msg.DriverConfigJson));
|
||||
// Fast-fail writes while still connecting — without this the inbound WriteAttribute dead-letters
|
||||
// and DriverHostActor.HandleRouteNodeWrite waits its full 8s Ask before reporting a generic
|
||||
// and DriverHostActor.HandleRouteNodeWrite waits its full Ask before reporting a generic
|
||||
// "write timeout". Synchronous Receive: Sender.Tell on the actor thread is safe.
|
||||
Receive<WriteAttribute>(_ =>
|
||||
Sender.Tell(new WriteAttributeResult(false, "driver not connected")));
|
||||
@@ -919,7 +928,7 @@ public sealed class DriverInstanceActor : ReceiveActor, IWithTimers
|
||||
/// non-empty desired set, so it costs nothing on a driver with no alarm tags.
|
||||
/// <para>A persistently failing subscribe backs off: after the n-th consecutive failure the reconcile
|
||||
/// skips 2^(n-1) − 1 ticks, capped at <see cref="MaxAlarmRetrySkipTicks"/>, because a HANGING subscribe
|
||||
/// holds the mailbox for its full 10 s bound (writes queued behind it can miss the host's 8 s Ask). A
|
||||
/// holds the mailbox for its full 10 s bound (writes delayed behind it eat into their queue budget). A
|
||||
/// Connected entry or a deploy's desired-set push still retries immediately — only the timer backs off.</para>
|
||||
/// </summary>
|
||||
private void ReconcileAlarmSubscription()
|
||||
@@ -1151,9 +1160,13 @@ public sealed class DriverInstanceActor : ReceiveActor, IWithTimers
|
||||
return;
|
||||
}
|
||||
|
||||
// Monotonic, so a wall-clock step cannot expire (or extend) a queued write.
|
||||
var issuedAt = msg.IssuedAt ?? _writeClock.GetTimestamp();
|
||||
if (TryExpireWrite(replyTo, issuedAt)) return;
|
||||
|
||||
if (_writesInFlight >= _maxWritesInFlight || _tagsInFlight.Contains(msg.TagId))
|
||||
{
|
||||
_pendingWrites.AddLast((msg, replyTo, DateTime.UtcNow));
|
||||
_pendingWrites.AddLast((msg, replyTo, issuedAt));
|
||||
return;
|
||||
}
|
||||
|
||||
@@ -1258,21 +1271,22 @@ public sealed class DriverInstanceActor : ReceiveActor, IWithTimers
|
||||
private void DrainWriteQueue()
|
||||
{
|
||||
if (_driver is not IWritable writable) return;
|
||||
var now = DateTime.UtcNow;
|
||||
var node = _pendingWrites.First;
|
||||
while (node is not null && _writesInFlight < _maxWritesInFlight)
|
||||
|
||||
// Expire FIRST, over the whole queue, regardless of free slots: a write past its budget must be answered
|
||||
// now, not whenever a slot next frees and the scan happens to reach it — the host's Ask covers the budget
|
||||
// plus one write (DriverWriteBounds.HostAsk), not an unbounded wait behind a full set of slots.
|
||||
for (var node = _pendingWrites.First; node is not null;)
|
||||
{
|
||||
var next = node.Next;
|
||||
var (write, replyTo, enqueuedUtc) = node.Value;
|
||||
if (now - enqueuedUtc > _queuedWriteBudget)
|
||||
{
|
||||
// Already reported as a timeout upstream (the host's Ask is the budget) and the node reverted —
|
||||
// writing it now would push a stale value to the device. BadTimeout, never written.
|
||||
_pendingWrites.Remove(node);
|
||||
replyTo.Tell(new WriteAttributeResult(false,
|
||||
$"StatusCode=0x800A0000 (BadTimeout: queued for {(now - enqueuedUtc).TotalSeconds:F1}s, over the {_queuedWriteBudget.TotalSeconds:F0}s write budget)"));
|
||||
}
|
||||
else if (!_tagsInFlight.Contains(write.TagId))
|
||||
if (TryExpireWrite(node.Value.ReplyTo, node.Value.IssuedAt)) _pendingWrites.Remove(node);
|
||||
node = next;
|
||||
}
|
||||
|
||||
for (var node = _pendingWrites.First; node is not null && _writesInFlight < _maxWritesInFlight;)
|
||||
{
|
||||
var next = node.Next;
|
||||
var (write, replyTo, _) = node.Value;
|
||||
if (!_tagsInFlight.Contains(write.TagId))
|
||||
{
|
||||
_pendingWrites.Remove(node);
|
||||
LaunchWrite(writable, write, replyTo);
|
||||
@@ -1282,6 +1296,23 @@ public sealed class DriverInstanceActor : ReceiveActor, IWithTimers
|
||||
}
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Fails a write that has waited longer than <see cref="_queuedWriteBudget"/> since it was issued (in the
|
||||
/// mailbox or the queue). Only a write inside the budget may start: the host's Ask covers the budget plus one
|
||||
/// full write (<see cref="DriverWriteBounds.HostAsk"/>), so a write started later could still be running
|
||||
/// when the Ask reports it failed and its node is reverted — and the device could then take a value the
|
||||
/// client was told failed. BadTimeout, never written.
|
||||
/// </summary>
|
||||
/// <returns>True when the write was expired (and answered).</returns>
|
||||
private bool TryExpireWrite(IActorRef replyTo, long issuedAt)
|
||||
{
|
||||
var waited = _writeClock.GetElapsedTime(issuedAt);
|
||||
if (waited <= _queuedWriteBudget) return false;
|
||||
replyTo.Tell(new WriteAttributeResult(false,
|
||||
$"StatusCode=0x800A0000 (BadTimeout: queued for {waited.TotalSeconds:F1}s, over the {_queuedWriteBudget.TotalSeconds:F0}s write budget)"));
|
||||
return true;
|
||||
}
|
||||
|
||||
/// <summary>Replies to every queued write with <paramref name="reason"/>; called on each exit from
|
||||
/// <c>Connected</c>. In-flight writes are left to finish — they already hold the device and reply with its
|
||||
/// real outcome.</summary>
|
||||
|
||||
@@ -0,0 +1,105 @@
|
||||
namespace ZB.MOM.WW.OtOpcUa.Runtime.Drivers;
|
||||
|
||||
/// <summary>
|
||||
/// Every bound on an operator write's path, derived from ONE configured value: the write timeout
|
||||
/// (<c>DriverWrites:Timeout</c>). lmxopcua#777.
|
||||
/// </summary>
|
||||
/// <remarks>
|
||||
/// <para>
|
||||
/// A write passes three bounds on its way to the device, and they used to be configured independently:
|
||||
/// the gateway's Ask of <see cref="DriverHostActor"/> (10 s), the host's Ask of the
|
||||
/// <see cref="DriverInstanceActor"/> child (a fixed 8 s), and the child's own write timeout plus the
|
||||
/// queued-write budget (5 s + up to 2 s, and 8 s in the queue). An outer Ask that fires first reports the
|
||||
/// write failed and the node manager reverts the node, while the child is still driving the device and the
|
||||
/// device may still take the value — the client is told "failed" for a write that applied. Raising the
|
||||
/// write timeout above ~6 s silently produced exactly that.
|
||||
/// </para>
|
||||
/// <para>
|
||||
/// Here every bound is derived from <see cref="Timeout"/>, innermost first, so each outer bound covers
|
||||
/// the worst case of everything inside it:
|
||||
/// <list type="number">
|
||||
/// <item><see cref="Timeout"/> — the write's cancellation token fires.</item>
|
||||
/// <item><see cref="RunBound"/> = timeout + <see cref="AbandonGrace"/> — a write that ignored its
|
||||
/// token is abandoned (answers <c>BadTimeout</c>).</item>
|
||||
/// <item><see cref="QueueBudget"/> = <see cref="RunBound"/> — the longest a write may wait for a
|
||||
/// slot (or for an earlier write to the same tag) before it is failed instead of started.</item>
|
||||
/// <item><see cref="ReplyBound"/> = queue + run — the child answers every write within this of its
|
||||
/// arrival.</item>
|
||||
/// <item><see cref="HostAsk"/> = reply + <see cref="ReplyMargin"/>, and
|
||||
/// <see cref="GatewayAsk"/> = host Ask + margin.</item>
|
||||
/// </list>
|
||||
/// The Asks are safety nets for an actor that never replies, not a second write timeout.
|
||||
/// </para>
|
||||
/// <para>
|
||||
/// The Phase 6.1 resilience tier's per-capability Write timeout (<c>DriverResilienceOptions</c>) runs
|
||||
/// INSIDE <see cref="Timeout"/>: the actor's token is the outer cap, so a tier Write timeout above
|
||||
/// <see cref="Timeout"/> never takes effect. The Host warns about that at startup (tier defaults) and at
|
||||
/// each driver spawn (per-instance <c>ResilienceConfig</c> overrides).
|
||||
/// </para>
|
||||
/// </remarks>
|
||||
public sealed record DriverWriteBounds
|
||||
{
|
||||
/// <summary>The configuration section the write timeout is bound from.</summary>
|
||||
public const string SectionName = "DriverWrites";
|
||||
|
||||
/// <summary>
|
||||
/// Default write timeout. 10 s — the longest resilience-tier Write timeout (Tier C), so no tier default
|
||||
/// is capped below its own policy. Tier A/B writes are cancelled by their tighter tier timeout first.
|
||||
/// </summary>
|
||||
public static readonly TimeSpan DefaultTimeout = TimeSpan.FromSeconds(10);
|
||||
|
||||
/// <summary>Largest accepted write timeout; a larger configured value is clamped (with a startup warning).</summary>
|
||||
public static readonly TimeSpan MaxTimeout = TimeSpan.FromMinutes(5);
|
||||
|
||||
/// <summary>
|
||||
/// Headroom each Ask keeps over the bound it wraps, for mailbox and dispatch latency between the reply being
|
||||
/// sent and the Ask observing it.
|
||||
/// </summary>
|
||||
public static readonly TimeSpan ReplyMargin = TimeSpan.FromSeconds(1);
|
||||
|
||||
/// <summary>The write bounds derived from <see cref="DefaultTimeout"/>.</summary>
|
||||
public static DriverWriteBounds Default { get; } = new(DefaultTimeout);
|
||||
|
||||
/// <summary>Creates the bounds for a write timeout.</summary>
|
||||
/// <param name="timeout">The write timeout; must be positive.</param>
|
||||
/// <exception cref="ArgumentOutOfRangeException">The timeout is zero or negative.</exception>
|
||||
public DriverWriteBounds(TimeSpan timeout)
|
||||
{
|
||||
ArgumentOutOfRangeException.ThrowIfLessThanOrEqual(timeout, TimeSpan.Zero);
|
||||
Timeout = timeout;
|
||||
}
|
||||
|
||||
/// <summary>When a write's cancellation token is cancelled (the write answers <c>"write timeout"</c>).</summary>
|
||||
public TimeSpan Timeout { get; }
|
||||
|
||||
/// <summary>
|
||||
/// How long after <see cref="Timeout"/> a write that has not returned is abandoned as token-ignoring: half
|
||||
/// the timeout, at most 2 s. Long enough for a driver that honours its token to answer the cancellation
|
||||
/// itself.
|
||||
/// </summary>
|
||||
public TimeSpan AbandonGrace => AbandonGraceFor(Timeout);
|
||||
|
||||
/// <summary>The latest a running write is answered after it starts: <see cref="Timeout"/> + <see cref="AbandonGrace"/>.</summary>
|
||||
public TimeSpan RunBound => Timeout + AbandonGrace;
|
||||
|
||||
/// <summary>
|
||||
/// The longest a write may wait for a slot before it is failed <c>BadTimeout</c> instead of being written:
|
||||
/// one <see cref="RunBound"/>, i.e. long enough for an earlier write to the same tag to finish.
|
||||
/// </summary>
|
||||
public TimeSpan QueueBudget => RunBound;
|
||||
|
||||
/// <summary>The latest the driver child answers a write after it arrives: <see cref="QueueBudget"/> + <see cref="RunBound"/>.</summary>
|
||||
public TimeSpan ReplyBound => QueueBudget + RunBound;
|
||||
|
||||
/// <summary><see cref="DriverHostActor"/>'s Ask of the child: <see cref="ReplyBound"/> + <see cref="ReplyMargin"/>.</summary>
|
||||
public TimeSpan HostAsk => ReplyBound + ReplyMargin;
|
||||
|
||||
/// <summary><see cref="ActorNodeWriteGateway"/>'s Ask of the host: <see cref="HostAsk"/> + <see cref="ReplyMargin"/>.</summary>
|
||||
public TimeSpan GatewayAsk => HostAsk + ReplyMargin;
|
||||
|
||||
/// <summary>See <see cref="AbandonGrace"/>.</summary>
|
||||
/// <param name="timeout">The write timeout in force.</param>
|
||||
/// <returns>The grace between cancelling the write's token and abandoning the write.</returns>
|
||||
public static TimeSpan AbandonGraceFor(TimeSpan timeout) =>
|
||||
TimeSpan.FromTicks(Math.Min(timeout.Ticks / 2, TimeSpan.FromSeconds(2).Ticks));
|
||||
}
|
||||
Reference in New Issue
Block a user