fix(runtime): derive every write bound and Ask from one write timeout (lmxopcua#777)
v2-ci / unit-tests (push) Has been cancelled
v2-ci / unit-tests (push) Has been cancelled
Three write bounds were configured independently: the inbound-write gateway's 10 s Ask, DriverHostActor's fixed 8 s Ask of the child, and the child's own write timeout (5 s + up to 2 s grace) with a fixed 8 s queue budget. An outer Ask that fires first reports the write failed and the node reverts while the child is still driving the device, so any write timeout above ~6 s (or a write dequeued late in its 8 s budget) let the device take a value the client was told failed. DriverWriteBounds (Runtime) derives them all from DriverWrites:Timeout: token cancel = Timeout abandon = Timeout + min(Timeout/2, 2 s) (RunBound) queue budget = RunBound child replies <= queue budget + RunBound (ReplyBound) host Ask = ReplyBound + 1 s gateway Ask = host Ask + 1 s Default Timeout is 10 s (the Tier C Write timeout, so no tier default is capped below its own policy); host Ask 25 s, gateway Ask 26 s. Tier A/B writes are still cancelled first by their 2 s / 4 s tier timeouts. - DriverHostActor takes writeBounds + writeClock, spawns children with them, and runs its Ask deadline as a token on the write clock (Infinite Akka timeout), so host and child bounds share one clock. - WriteAttribute gains IssuedAt (write-clock timestamp, stamped by the host): the queue budget counts mailbox time too, and a write past it is failed BadTimeout and never started - including on arrival. Stale queued writes are expired across the whole queue, not only up to a free slot. - ActorNodeWriteGateway's default Ask = Default.GatewayAsk; the hosted service passes the configured one. - Startup validation (Host DriverWriteBoundsBinding, same warn-and-clamp style as ScriptingOptionsBinding): unparsable / non-positive -> default, above 5 min -> clamped, and each tier whose default Write timeout exceeds the write timeout is named in a Warning (its policy would never take effect). The derived bounds are logged at Information. Per-instance ResilienceConfig Write overrides arrive with a deployment, so DriverCapabilityInvokerFactory warns at spawn (writeTimeoutCeiling). - appsettings.json documents DriverWrites:Timeout. Tests (manual clock): - DriverHostActorWriteAskTests: a held write under a 9 s timeout survives an 8.5 s step and succeeds; a token-ignoring write is answered by the child's BadTimeout at 9 s + grace, not by the Ask; the Infinite-timeout Ask ends exactly at its clock deadline. - DriverInstanceActorConcurrentWriteTests: a write issued longer ago than the queue budget is failed and never reaches the driver; positive control at exactly the budget is written. - DriverWriteBoundsTests: every outer bound covers the inner ones for 0.5 s to 300 s timeouts. - DriverWriteBoundsBindingTests: default fits every tier; 3 s timeout is caught at startup with Tier B and C named (not A); bad values fall back with a warning; over-max clamps. - DriverCapabilityInvokerFactoryTests: spawn warns when a Write override exceeds the ceiling; no warning within it or without a ceiling. Mutation checks: a fixed 8 s host Ask on the same clock fails both 9 s host tests; stamping arrival time instead of IssuedAt fails the issued-at test; dropping the tier check / factory check fails their tests. Runtime.Tests 961 passed / 3 skipped at AKKA_TEST_TIMEFACTOR 3 and 1; Host.Tests 232/232 at 3 and 1; Core.Tests 361 passed / 1 skipped. Solution Release build: 0 warnings. ContinuousHistorizationRecorderTypedValueTests (historian) deliberately left for the historian lane. Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
This commit is contained in:
@@ -479,6 +479,14 @@ public sealed class DriverHostActor : ReceiveActor, IWithTimers, IWithUnboundedS
|
||||
/// <summary>Where the synchronous native-ack pre-check snapshot is published (lmxopcua#760); null ⇒ none.</summary>
|
||||
private readonly NativeAlarmAckPreflightView? _ackPreflightView;
|
||||
|
||||
/// <summary>The write bounds every child is spawned with and this host's write Ask is derived from
|
||||
/// (lmxopcua#777).</summary>
|
||||
private readonly DriverWriteBounds _writeBounds;
|
||||
|
||||
/// <summary>The clock the write bounds run on, shared with every child so a write's issue stamp
|
||||
/// (<see cref="DriverInstanceActor.WriteAttribute.IssuedAt"/>) means the same thing on both sides.</summary>
|
||||
private readonly TimeProvider _writeClock;
|
||||
|
||||
/// <summary>Whether any redundancy snapshot has reached this host — the HistoryUpdate gate stays
|
||||
/// closed until one has (see <see cref="PrimaryGatePolicy.ShouldServeHistoryUpdate"/>).</summary>
|
||||
private bool _redundancySnapshotReceived;
|
||||
@@ -721,6 +729,11 @@ public sealed class DriverHostActor : ReceiveActor, IWithTimers, IWithUnboundedS
|
||||
/// HistoryUpdate Primary gate. Null on nodes that wire no redundancy manager.</param>
|
||||
/// <param name="scripting">The node-wide <c>Scripting</c> run-guard options handed to the scripted-alarm
|
||||
/// engine. Null ⇒ the compiled defaults.</param>
|
||||
/// <param name="ackPreflightView">Where the native-ack pre-check snapshot is published. Null ⇒ none.</param>
|
||||
/// <param name="writeBounds">The write timeout every child is spawned with, and from which this host's write
|
||||
/// Ask is derived (<see cref="DriverWriteBounds.HostAsk"/>). Null ⇒ <see cref="DriverWriteBounds.Default"/>.</param>
|
||||
/// <param name="writeClock">The clock the write bounds (and the host's write Ask) run on, shared with every
|
||||
/// child. Null ⇒ <see cref="TimeProvider.System"/>.</param>
|
||||
/// <returns>The Akka.NET <see cref="Akka.Actor.Props"/> used to spawn this actor.</returns>
|
||||
public static Props Props(
|
||||
IDbContextFactory<OtOpcUaConfigDbContext>? dbFactory,
|
||||
@@ -756,7 +769,9 @@ public sealed class DriverHostActor : ReceiveActor, IWithTimers, IWithUnboundedS
|
||||
bool captureWhilePrimaryUnreachable = true,
|
||||
IHistoryUpdateRoleView? historyUpdateRoleView = null,
|
||||
ScriptingOptions? scripting = null,
|
||||
NativeAlarmAckPreflightView? ackPreflightView = null) =>
|
||||
NativeAlarmAckPreflightView? ackPreflightView = null,
|
||||
DriverWriteBounds? writeBounds = null,
|
||||
TimeProvider? writeClock = null) =>
|
||||
// WARNING: this forwarding list is POSITIONAL, and Props.Create compiles it into an
|
||||
// expression tree. Six IActorRef? parameters and several interface-typed ones mean a
|
||||
// mis-ordered argument is usually type-compatible and therefore compiles clean, then binds
|
||||
@@ -770,7 +785,7 @@ public sealed class DriverHostActor : ReceiveActor, IWithTimers, IWithUnboundedS
|
||||
alarmStateStore, telemetryHub, primaryRoleSource, historizationRoleView,
|
||||
sharesHistorizationScopeWithLocal, applyWedgeTimeout, reprovisionBackoff, commsRecoveryAuditDelay,
|
||||
sharesAlarmQueueWithLocal, applyOutcomeTimeout, captureWhilePrimaryUnreachable, historyUpdateRoleView,
|
||||
scripting, ackPreflightView));
|
||||
scripting, ackPreflightView, writeBounds, writeClock));
|
||||
|
||||
/// <summary>Initializes a new DriverHostActor with the specified dependencies.</summary>
|
||||
/// <param name="dbFactory">Database context factory for configuration database access.</param>
|
||||
@@ -861,6 +876,10 @@ public sealed class DriverHostActor : ReceiveActor, IWithTimers, IWithUnboundedS
|
||||
/// engine. Null ⇒ the compiled defaults.</param>
|
||||
/// <param name="ackPreflightView">Where the condition → owning-driver native-ack pre-check snapshot is published
|
||||
/// for the node manager's synchronous refusal (lmxopcua#760). Null ⇒ nothing published (no pre-check).</param>
|
||||
/// <param name="writeBounds">The write timeout every child is spawned with, and from which this host's write
|
||||
/// Ask is derived (<see cref="DriverWriteBounds.HostAsk"/>). Null ⇒ <see cref="DriverWriteBounds.Default"/>.</param>
|
||||
/// <param name="writeClock">The clock the write bounds (and the host's write Ask) run on, shared with every
|
||||
/// child. Null ⇒ <see cref="TimeProvider.System"/>.</param>
|
||||
public DriverHostActor(
|
||||
IDbContextFactory<OtOpcUaConfigDbContext>? dbFactory,
|
||||
CommonsNodeId localNode,
|
||||
@@ -895,8 +914,12 @@ public sealed class DriverHostActor : ReceiveActor, IWithTimers, IWithUnboundedS
|
||||
bool captureWhilePrimaryUnreachable = true,
|
||||
IHistoryUpdateRoleView? historyUpdateRoleView = null,
|
||||
ScriptingOptions? scripting = null,
|
||||
NativeAlarmAckPreflightView? ackPreflightView = null)
|
||||
NativeAlarmAckPreflightView? ackPreflightView = null,
|
||||
DriverWriteBounds? writeBounds = null,
|
||||
TimeProvider? writeClock = null)
|
||||
{
|
||||
_writeBounds = writeBounds ?? DriverWriteBounds.Default;
|
||||
_writeClock = writeClock ?? TimeProvider.System;
|
||||
_ackPreflightView = ackPreflightView;
|
||||
_scripting = scripting;
|
||||
_captureWhilePrimaryUnreachable = captureWhilePrimaryUnreachable;
|
||||
@@ -2237,17 +2260,30 @@ public sealed class DriverHostActor : ReceiveActor, IWithTimers, IWithUnboundedS
|
||||
return;
|
||||
}
|
||||
|
||||
// Ask the child a bounded write — DriverInstanceActor bounds the backend call by its write timeout
|
||||
// (DefaultWriteTimeout 5 s; a token-ignoring write is abandoned at 7 s), so the 8s Ask is a safety net
|
||||
// for an actor that never replies. Capture Sender before
|
||||
// the continuation: it runs off the actor thread, where raw Sender is unsafe to read.
|
||||
// Ask the child a bounded write. The child answers every write within DriverWriteBounds.ReplyBound of the
|
||||
// IssuedAt stamped here (queue budget + write timeout + abandon grace — it refuses to START a write that
|
||||
// could not finish inside it), and the Ask is DERIVED from the same write timeout with a margin on top
|
||||
// (lmxopcua#777). So the Ask is only a safety net for a child that never replies: it can no longer fire
|
||||
// first and report a write failed — reverting the node — while the child is still driving the device. It
|
||||
// was a fixed 8 s, and any write timeout above ~6 s let a write outlive it.
|
||||
//
|
||||
// The Ask's deadline is a token on the write clock rather than Akka's scheduler, so it runs on the same
|
||||
// clock as the child's bounds (and a test can step both together). Capture Sender before the
|
||||
// continuation: it runs off the actor thread, where raw Sender is unsafe to read.
|
||||
var replyTo = Sender;
|
||||
var askDeadline = new CancellationTokenSource(_writeBounds.HostAsk, _writeClock);
|
||||
entry.Actor.Ask<DriverInstanceActor.WriteAttributeResult>(
|
||||
new DriverInstanceActor.WriteAttribute(target.RawPath, msg.Value!), TimeSpan.FromSeconds(8))
|
||||
new DriverInstanceActor.WriteAttribute(target.RawPath, msg.Value!, _writeClock.GetTimestamp()),
|
||||
System.Threading.Timeout.InfiniteTimeSpan,
|
||||
askDeadline.Token)
|
||||
.ContinueWith(
|
||||
t => t.IsCompletedSuccessfully
|
||||
? new NodeWriteResult(t.Result.Success, t.Result.Reason)
|
||||
: new NodeWriteResult(false, "write timeout"),
|
||||
t =>
|
||||
{
|
||||
askDeadline.Dispose();
|
||||
return t.IsCompletedSuccessfully
|
||||
? new NodeWriteResult(t.Result.Success, t.Result.Reason)
|
||||
: new NodeWriteResult(false, "write timeout");
|
||||
},
|
||||
CancellationToken.None,
|
||||
TaskContinuationOptions.None,
|
||||
TaskScheduler.Default)
|
||||
@@ -3890,7 +3926,9 @@ public sealed class DriverHostActor : ReceiveActor, IWithTimers, IWithUnboundedS
|
||||
driver!,
|
||||
healthPublisher: _healthPublisher,
|
||||
clusterId: clusterId,
|
||||
invoker: invoker),
|
||||
invoker: invoker,
|
||||
writeTimeout: _writeBounds.Timeout,
|
||||
writeClock: _writeClock),
|
||||
actorName);
|
||||
child.Tell(new DriverInstanceActor.InitializeRequested(spec.DriverConfig));
|
||||
}
|
||||
|
||||
Reference in New Issue
Block a user