fix(runtime): derive every write bound and Ask from one write timeout (lmxopcua#777)
v2-ci / unit-tests (push) Has been cancelled

Three write bounds were configured independently: the inbound-write
gateway's 10 s Ask, DriverHostActor's fixed 8 s Ask of the child, and the
child's own write timeout (5 s + up to 2 s grace) with a fixed 8 s queue
budget. An outer Ask that fires first reports the write failed and the node
reverts while the child is still driving the device, so any write timeout
above ~6 s (or a write dequeued late in its 8 s budget) let the device take
a value the client was told failed.

DriverWriteBounds (Runtime) derives them all from DriverWrites:Timeout:
  token cancel   = Timeout
  abandon        = Timeout + min(Timeout/2, 2 s)            (RunBound)
  queue budget   = RunBound
  child replies  <= queue budget + RunBound                  (ReplyBound)
  host Ask       = ReplyBound + 1 s
  gateway Ask    = host Ask + 1 s
Default Timeout is 10 s (the Tier C Write timeout, so no tier default is
capped below its own policy); host Ask 25 s, gateway Ask 26 s. Tier A/B
writes are still cancelled first by their 2 s / 4 s tier timeouts.

- DriverHostActor takes writeBounds + writeClock, spawns children with them,
  and runs its Ask deadline as a token on the write clock (Infinite Akka
  timeout), so host and child bounds share one clock.
- WriteAttribute gains IssuedAt (write-clock timestamp, stamped by the
  host): the queue budget counts mailbox time too, and a write past it is
  failed BadTimeout and never started - including on arrival. Stale queued
  writes are expired across the whole queue, not only up to a free slot.
- ActorNodeWriteGateway's default Ask = Default.GatewayAsk; the hosted
  service passes the configured one.
- Startup validation (Host DriverWriteBoundsBinding, same warn-and-clamp
  style as ScriptingOptionsBinding): unparsable / non-positive -> default,
  above 5 min -> clamped, and each tier whose default Write timeout exceeds
  the write timeout is named in a Warning (its policy would never take
  effect). The derived bounds are logged at Information. Per-instance
  ResilienceConfig Write overrides arrive with a deployment, so
  DriverCapabilityInvokerFactory warns at spawn (writeTimeoutCeiling).
- appsettings.json documents DriverWrites:Timeout.

Tests (manual clock):
- DriverHostActorWriteAskTests: a held write under a 9 s timeout survives
  an 8.5 s step and succeeds; a token-ignoring write is answered by the
  child's BadTimeout at 9 s + grace, not by the Ask; the Infinite-timeout
  Ask ends exactly at its clock deadline.
- DriverInstanceActorConcurrentWriteTests: a write issued longer ago than
  the queue budget is failed and never reaches the driver; positive control
  at exactly the budget is written.
- DriverWriteBoundsTests: every outer bound covers the inner ones for 0.5 s
  to 300 s timeouts.
- DriverWriteBoundsBindingTests: default fits every tier; 3 s timeout is
  caught at startup with Tier B and C named (not A); bad values fall back
  with a warning; over-max clamps.
- DriverCapabilityInvokerFactoryTests: spawn warns when a Write override
  exceeds the ceiling; no warning within it or without a ceiling.
Mutation checks: a fixed 8 s host Ask on the same clock fails both 9 s host
tests; stamping arrival time instead of IssuedAt fails the issued-at test;
dropping the tier check / factory check fails their tests.

Runtime.Tests 961 passed / 3 skipped at AKKA_TEST_TIMEFACTOR 3 and 1;
Host.Tests 232/232 at 3 and 1; Core.Tests 361 passed / 1 skipped.
Solution Release build: 0 warnings.

ContinuousHistorizationRecorderTypedValueTests (historian) deliberately
left for the historian lane.

Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
This commit is contained in:
Joseph Doherty
2026-11-09 15:46:04 -05:00
parent b051bdc87c
commit 97dc715dac
16 changed files with 824 additions and 57 deletions
@@ -307,6 +307,49 @@ public sealed class DriverInstanceActorConcurrentWriteTests : RuntimeActorTestBa
private void AwaitWriteBoundsArmed(ManualTimeProvider clock) =>
AwaitCondition(() => clock.ArmedTimers == 2, PresenceBudget, TimeSpan.FromMilliseconds(20));
/// <summary>
/// lmxopcua#777: the queue budget runs from when the HOST issued the write
/// (<see cref="DriverInstanceActor.WriteAttribute.IssuedAt"/>), not from when this actor got round to it. A
/// write that sat in the mailbox past the budget and was then started could still be running when the host's
/// Ask fired — reported failed, node reverted, device still taking the value. It is failed BadTimeout and
/// never written. (Mutation-checked: stamping the arrival time instead of IssuedAt writes it.)
/// </summary>
[Fact]
public async Task A_write_issued_longer_ago_than_the_queue_budget_is_failed_not_started()
{
var clock = new ManualTimeProvider();
var bounds = new DriverWriteBounds(TimeSpan.FromSeconds(10));
var driver = new SlowWritableDriver();
var (actor, _) = Connect(driver, writeTimeout: bounds.Timeout, writeClock: clock);
var issuedAt = clock.GetTimestamp();
clock.Advance(bounds.QueueBudget + TimeSpan.FromMilliseconds(1));
var result = await actor.Ask<DriverInstanceActor.WriteAttributeResult>(
new DriverInstanceActor.WriteAttribute("tag-a", 1, issuedAt), PresenceBudget);
result.Success.ShouldBeFalse();
result.Reason!.ShouldContain("800A0000", customMessage: "BadTimeout");
driver.WritesEntered.ShouldBe(0, "a write past its budget must never reach the device");
}
/// <summary>Positive control for the test above: a write issued just INSIDE the budget is written.</summary>
[Fact]
public async Task A_write_issued_within_the_queue_budget_is_written()
{
var clock = new ManualTimeProvider();
var bounds = new DriverWriteBounds(TimeSpan.FromSeconds(10));
var driver = new SlowWritableDriver();
var (actor, _) = Connect(driver, writeTimeout: bounds.Timeout, writeClock: clock);
var issuedAt = clock.GetTimestamp();
clock.Advance(bounds.QueueBudget);
var result = await actor.Ask<DriverInstanceActor.WriteAttributeResult>(
new DriverInstanceActor.WriteAttribute("tag-a", 1, issuedAt), PresenceBudget);
result.Success.ShouldBeTrue(result.Reason);
driver.Writes.Select(w => w.FullReference).ShouldBe(new[] { "tag-a" });
}
[Fact]
public async Task Queued_writes_fail_fast_when_the_driver_leaves_Connected()
{