Central pin 1.1.0 -> 3.2.2; AdminUI/ControlPlane/Runtime/Host.Tests no longer carry VersionOverride.
The three driver IntegrationTests fixtures' Fact/Theory attributes gain the source-info ctor (xUnit3003).
No xUnit1051 fallout remained (already cleaned). Test counts unchanged.
Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
Akka.TestKit.Xunit2 -> Akka.TestKit.Xunit 1.5.62 (xunit.v3 TestKit; needs xunit.v3 3.2.2, set via
VersionOverride on the four projects). Removes the xunit 2.9.3 pin. IAsyncLifetime returns ValueTask,
SqlServerFactAttribute gains the source-info ctor, and xUnit1051 call sites pass
TestContext.Current.CancellationToken. Test counts unchanged.
Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
- NodeManagerLockTimeBenchmark: wall-clock budget enforced only with OTOPCUA_BENCH=1; default run keeps the state-correctness asserts and logs timings.
- OpcUaClientDriverWire host-status: signal (semaphore) wait that only accepts a non-Running state observed after the server is disposed.
- AbCip PollLatencyComparison: retry once on a fixture connection reset; p95 criterion enforced only with AB_LAT_ENFORCE=1.
- ScriptRunGuardTests: orphan scripts held by a ctx.Released signal instead of a 250 ms wall-clock spin that could finish inside the 60 ms budget under load; stuck-orphan wall-clock rule disabled in the tests that pin the strike counting.
- DependencyMuxReplayTests: split the inside-grace positive control (1 h grace) from the release test (control asserted before unregister), removing the 200 ms grace race.
Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
Proven broken first: VirtualTagToVirtualTagFlowTests (real mux + host + Roslyn
evaluator) failed on master - VT A evaluated, VT B reading A by its key never did.
VT outputs only reached their OPC UA node; the mux carried driver values only.
- VirtualTagHostActor also sends each real result (not re-asserts) and a disabled
tag's BadOutOfService to the mux as VirtualTagOutput, keyed by the VT's UNS
NodeId identifier {EquipmentId}/{Name}. Publish path unchanged (single source).
- DependencyMuxActor pins VT outputs (retained with no subscriber, no grace,
immune to SourceRemoved) and replays them; VirtualTagOutputsRemoved releases
keys that leave the deployed set.
- DraftValidator: blocking VirtualTagDependencyCycle (A->B->A, self-read, loops
through Calculation tags, {{equip}} refs resolved); disabled VTs are not nodes.
EquipReferenceUnresolved names the key form when {{equip}}/<name> is a VT.
- Runtime backstop: the host re-detects VT cycles per apply and does not route
outputs of tags on one.
- AdminUI ScriptTagCatalog lists VTs by key (the bare leaf name never resolved).
- docs/VirtualTags.md documents the routing, key, retention and cycle rules.
Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
Revises 963b0d66, which dropped a disabled VirtualTag from the runtime plan so a
bound client got BadNodeIdUnknown. Now the plan carries Enabled (part of value
equality): AddressSpaceComposer.Compose and DeploymentArtifact.BuildEquipmentVirtualTagPlans
keep the row with Enabled=false (byte-parity), the applier materialises the node as
normal, and VirtualTagHostActor spawns no VirtualTagActor for it but publishes
BadOutOfService (new OpcUaStatus constant) with a null value stamped at deploy, on
every apply. Enabled flips are node-irrelevant to the change classifier; enabled->disabled
stops the child, disabled->enabled resumes evaluation. Validator exemption and name
uniqueness unchanged. Scripted alarms keep their not-materialised convention.
Tests updated; new wire test DisabledVirtualTagWireTests. docs/VirtualTags.md updated.
Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
VirtualTag.Enabled was authorable but ignored by both runtime-plan seams.
AddressSpaceComposer.Compose and DeploymentArtifact.BuildEquipmentVirtualTagPlans
now drop disabled rows (byte-parity), so no node is materialised and no
VirtualTagActor is spawned - the same outcome as a disabled scripted alarm.
DraftValidator exempts a disabled VT from VirtualTagNoEffectiveTrigger.
Calculation tags have no Enabled flag, so no equivalent gap. docs/VirtualTags.md updated.
Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
Second P1 slice of docs/plans/2026-11-09-out-of-process-script-runner-design.md; still opt-in behind
Scripting:Runner=OutOfProcess (InProcess unchanged).
- Compiled-PE cache (design §7, §14 #8): Scripting:Worker:PeCacheDir holds <dir>/pe/<id>.pe entries authenticated
with HMAC-SHA256 under a per-node random key (<dir>/pe-cache.key, created by the host, 0600/0700 on Unix), handed to
the worker in Hello. Entry id = SHA-256(compile fingerprint + kind + source); the fingerprint covers the Roslyn
version, the Core.Scripting build (analyzer, wrapper, stack guard), the sandbox reference set and the runtime. Edited,
truncated, other-key or renamed entries are rejected, deleted and recompiled; the worker never loads unverified bytes.
Pruned by recency at startup (30 d / 50 000). ScriptEvaluator splits into CompileToImage + FromImage. Measured on
40 scripts, restart to every first result: macOS 820 ms no cache / 840 ms empty / 85 ms warm;
windev (Windows) 9.5 s / 9.4 s / 1.0 s.
- Quarantine race: WorkerConnection runs exit attribution synchronously BEFORE completing the evaluations/loads it held
(and before Exited), and the shard passes it straight to the runner — a kill strike is in place before the killed
key's slot is released.
- Supervision: a dead worker restarts by itself after its backoff; Unavailable outcomes carry ScriptRunOutcome.Recovery
(-> VirtualTagEvalResult.RetryWhen) and the VT actor / Calculation driver re-evaluate once against current inputs when
it completes. Isolation worker stops after Scripting:Worker:IsolationIdleTimeout (10 min). Keys compiling when a
worker dies are suspects too (isolation finds a poison compile and quarantines it). Startup orphan sweep by launch
marker: kills only a process whose marker's host is gone and whose pid + start time + image match.
- Crash-loop circuit (Scripting:Worker:CircuitOpenAfter, 2 min): runner Unavailable, one Error, evaluations answered
at once while shards keep retrying at the cap.
- Observability (§10): otopcua.script.runner.state, worker.up/inflight/queue_depth/heap_bytes (shard),
ipc.roundtrip + eval.duration (kind), compile{cache}, isolation.suspects, exits reason=idle. `script-runner` health
check on the ready tier, at worst Degraded (§14 #12: degraded but still ready).
Left on #780: self-contained worker publish, installer (account, pipe ACL, firewall), docker-dev live drill,
scripted-alarm seam, the §4.5 per-seam batch submit, the §8 perf harness.
Tests: SupervisionTests, PeCacheStoreTests/PeCacheWorkerTests (real workers), ScriptRunnerHealthCheckTests, VT + Calc
recovery seam tests; 14 mutations each turned their test red.
Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
Child_is_respawned_after_unexpected_termination watched the child die, then re-applied once. The host prunes _children only when its own Terminated arrives, so a re-apply that lands first still sees vt-1 as present, spawns nothing, and the test times out at 15 s (caught 1 in 5 full Runtime.Tests runs on 2026-11-09). Re-apply inside AwaitAssert until the new RegisterInterest arrives; the assertions are unchanged. 30/30 isolated runs pass.
Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
Choice: implement the timer (option a). The field was designed and promised as a
working trigger: the entity doc says ChangeTriggered and TimerIntervalMs together
realize the trigger model, the DB check constraint requires ChangeTriggered OR a
timer >= 50, the AdminUI virtual-tag modal exposes "TimerIntervalMs (optional)" and
UnsTreeService rejects a tag with neither, and the 2026-06-08 design specified
"ChangeTriggered OR TimerIntervalMs>=50". Only the runtime never honoured it (the
standalone TimerTriggerScheduler was deleted in #603), so a timer-only tag sat at
BadWaitingForInitialData forever.
- EquipmentVirtualTagPlan carries ChangeTriggered/TimerIntervalMs (in equality, so
an edit respawns the child); both the composer and the artifact decode fill them.
- VirtualTagActor schedules a repeating tick (PreStart, cancelled in PostStop,
clamped to the 50 ms floor). ChangeTriggered=false makes the timer the only
trigger. A tick during an in-flight async run (#779) is dropped, not stashed, so
slow scripts never build a backlog of runs.
- DraftValidator VirtualTagNoEffectiveTrigger (hard error, same severity as
CalculationNoEffectiveTrigger): no timer and no change trigger with a literal
ctx.GetTag read.
Tests: VirtualTagActorTimerTests, DraftValidatorVirtualTagTriggerTests, artifact
decode test. Mutation-checked: no scheduling, stashing ticks, no cancel in
PostStop, and removing the validator call each fail a test.
Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
The Lane H2 remediation made Periodic(100 ms) the options-class default, but
appsettings.json, docker-dev and Program.cs's bad-value fallback all used
PerEntry, so effective behaviour never changed and the docs contradicted the
shipped config. The options default is now PerEntry; Periodic stays opt-in
(<= CommitIntervalMs of crash loss in exchange for throughput). Program.cs now
uses ContinuousHistorizationOptions.ResolveCommitMode() so the fallback shares
the single DefaultCommitMode constant. docs/Historian.md updated.
Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
VirtualTagActor called the blocking IVirtualTagEvaluator.Evaluate, and
RoslynVirtualTagEvaluator blocked on the run guard with
.GetAwaiter().GetResult(), so every script run parked a dispatcher thread for
up to its 2 s budget (a first-use Roslyn compile did too). This is the #750 P0
refactor for virtual tags.
- IVirtualTagEvaluator.EvaluateAsync (default: a completed task over the
snapshot overload, so the null evaluator and fakes are unchanged).
- RoslynVirtualTagEvaluator implements it without blocking: passthrough and
compile failures complete synchronously, a first-use compile goes to the
pool, the run is awaited. The blocking overload now delegates to it.
- VirtualTagActor: a completed task is applied in the same turn, as before;
an incomplete one switches to an "evaluating" behaviour that stashes every
dependency/reassert message until a private completion message arrives,
then applies the result and unstashes. Each trigger is still evaluated
against its own inputs, results are applied/published in trigger order and
the dedup/Bad-transition state sees the same sequence. The dependency map is
not touched while a run reads it.
Calculation and scripted-alarm serialisation is NOT relaxed: neither blocks a
thread, and running independent scripts in parallel is not semantics-neutral
(pool consumption bound, cross-tag publish order, timer/pump exclusion for
Calc; state-machine/persist boundary, transition timestamps and the
store-failure path for alarms). Recorded in docs/VirtualTags.md and the #750
design's P0 entry.
Tests: VirtualTagActorAsyncEvaluationTests — on a one-thread dispatcher a slow
script no longer starves a sibling actor and the slow actor answers Identify
(fails when the actor calls the blocking Evaluate); triggers during a run are
evaluated afterwards in order against their own inputs; a reassert during a
run is applied after it; a faulted evaluation takes the evaluator-threw path
(the ordering/reassert tests fail with the stash removed).
RoslynVirtualTagEvaluatorAsyncTests — a slow script and a cold compile return
an incomplete task (fail without the override); cheap outcomes and timeouts
match the blocking overload.
Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
DriverHostActor built ScriptedAlarmEngine without a scriptTimeout, so every
production alarm predicate fell back to TimedScriptEvaluator.DefaultTimeout
(250 ms) while virtual tags and Calculation tags ran on 2 s. History shows no
decision behind it: 250 ms is the generic Phase-7 A.4 default, the VT
evaluator moved to 2 s in F8b (219d10a2) and the alarm engine was simply never
given a value.
- New node-wide Scripting:RunTimeout (default 2 s, clamped to (0, 60 s],
non-positive -> default, each clamp a startup Warning) on ScriptingOptions.
- The VT evaluator and the alarm engine default to it; DriverHostActor now
builds the engine through CreateScriptedAlarmEngine, which passes it
explicitly. Calculation keeps its per-driver runTimeoutMs.
- Docs: ScriptEditor (config table), ScriptedAlarms, VirtualTags.
Tests: ScriptedAlarmEngineBudgetTests (a 600 ms predicate now completes and
activates; default and configured budgets), ScriptedAlarmEngineBudgetWiringTests
(the host-built engine), ScriptingOptionsBindingTests (bind + clamps). The
engine and wiring tests fail with the old 250 ms default restored.
Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
Three write bounds were configured independently: the inbound-write
gateway's 10 s Ask, DriverHostActor's fixed 8 s Ask of the child, and the
child's own write timeout (5 s + up to 2 s grace) with a fixed 8 s queue
budget. An outer Ask that fires first reports the write failed and the node
reverts while the child is still driving the device, so any write timeout
above ~6 s (or a write dequeued late in its 8 s budget) let the device take
a value the client was told failed.
DriverWriteBounds (Runtime) derives them all from DriverWrites:Timeout:
token cancel = Timeout
abandon = Timeout + min(Timeout/2, 2 s) (RunBound)
queue budget = RunBound
child replies <= queue budget + RunBound (ReplyBound)
host Ask = ReplyBound + 1 s
gateway Ask = host Ask + 1 s
Default Timeout is 10 s (the Tier C Write timeout, so no tier default is
capped below its own policy); host Ask 25 s, gateway Ask 26 s. Tier A/B
writes are still cancelled first by their 2 s / 4 s tier timeouts.
- DriverHostActor takes writeBounds + writeClock, spawns children with them,
and runs its Ask deadline as a token on the write clock (Infinite Akka
timeout), so host and child bounds share one clock.
- WriteAttribute gains IssuedAt (write-clock timestamp, stamped by the
host): the queue budget counts mailbox time too, and a write past it is
failed BadTimeout and never started - including on arrival. Stale queued
writes are expired across the whole queue, not only up to a free slot.
- ActorNodeWriteGateway's default Ask = Default.GatewayAsk; the hosted
service passes the configured one.
- Startup validation (Host DriverWriteBoundsBinding, same warn-and-clamp
style as ScriptingOptionsBinding): unparsable / non-positive -> default,
above 5 min -> clamped, and each tier whose default Write timeout exceeds
the write timeout is named in a Warning (its policy would never take
effect). The derived bounds are logged at Information. Per-instance
ResilienceConfig Write overrides arrive with a deployment, so
DriverCapabilityInvokerFactory warns at spawn (writeTimeoutCeiling).
- appsettings.json documents DriverWrites:Timeout.
Tests (manual clock):
- DriverHostActorWriteAskTests: a held write under a 9 s timeout survives
an 8.5 s step and succeeds; a token-ignoring write is answered by the
child's BadTimeout at 9 s + grace, not by the Ask; the Infinite-timeout
Ask ends exactly at its clock deadline.
- DriverInstanceActorConcurrentWriteTests: a write issued longer ago than
the queue budget is failed and never reaches the driver; positive control
at exactly the budget is written.
- DriverWriteBoundsTests: every outer bound covers the inner ones for 0.5 s
to 300 s timeouts.
- DriverWriteBoundsBindingTests: default fits every tier; 3 s timeout is
caught at startup with Tier B and C named (not A); bad values fall back
with a warning; over-max clamps.
- DriverCapabilityInvokerFactoryTests: spawn warns when a Write override
exceeds the ceiling; no warning within it or without a ceiling.
Mutation checks: a fixed 8 s host Ask on the same clock fails both 9 s host
tests; stamping arrival time instead of IssuedAt fails the issued-at test;
dropping the tier check / factory check fails their tests.
Runtime.Tests 961 passed / 3 skipped at AKKA_TEST_TIMEFACTOR 3 and 1;
Host.Tests 232/232 at 3 and 1; Core.Tests 361 passed / 1 skipped.
Solution Release build: 0 warnings.
ContinuousHistorizationRecorderTypedValueTests (historian) deliberately
left for the historian lane.
Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
otopcua.historization.recorder.pending_appends was written only from the
actor thread: when a burst was accepted (just after the increment, so
never 0) and on GetStatus, which nothing in production sends. Once ingest
stopped the gauge froze at the depth the last burst saw, so after a
backpressure episode it kept reporting a queue near the 64-burst cap that
had long drained.
The append loop now publishes the depth after every decrement
(ReportPendingAppends). The mailbox-depth gauge stays actor-thread only;
its description now says when it is sampled. New test (non-parallel
collection, the gauge is process-wide): the gauge reads 2 with one append
stuck and one queued, and 0 after the queue drains with no status
request. With the loop's report removed it fails (stays 2).
Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
The recorder now appends off its mailbox (9fda2919), so two separate Tells
are two bursts the append loop may commit and drain one at a time. The
#538 tests told Good.Tag and Bad.Tag separately and relied on both landing
in one drain pass; when the loop drained Good.Tag alone, Bad.Tag was left
alone at the head, every later pass failed with no sibling success (the
outage path), and nothing was ever abandoned. Run in isolation the
recorder test class failed 4 of 4 times (QueuedDepth 1, expected 0).
The five tests now send the pair as one DependencyValueBatch, and the
in-memory outbox appends a batch under one lock so a concurrent peek sees
it all-or-nothing, as a group commit is. 6 of 6 isolated runs green.
Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
HistorianAdapterActor fired `_ = EnqueueAsync(...)` per alarm transition
with no in-flight cap, so an alarm storm (a native source re-raising
hundreds of conditions on reconnect) spawned hundreds of concurrent SQLite
inserts that serialised on the file lock and held pool threads. It also kept
a Receive<AlarmHistorianEvent> with no production sender: a transition told
directly AND published on `alerts` would have been historized twice.
Transitions now go to a bounded channel (10000) drained by one loop, so the
sink sees one enqueue at a time in arrival order; a full queue holds the
mailbox (backpressure, nothing dropped); PostStop flushes the queue within
the shared 5 s stop budget. The direct receive is deleted. The gate tests
drive transitions instead; a directly-told AlarmHistorianEvent is shown to
be unhandled against a positive control.
Rebased from the unpushed remed/lane-H2 branch (originally authored 2026-10-22
in session_01Ub9hGmYLQvn9XETzw7CMsT) onto current master.
Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
The continuous-historization recorder awaited every outbox append (an fsync
under PerEntry) on its mailbox, so ingestion was capped at 1/fsync_latency
bursts per second while the mux kept telling into an unbounded, unmetered
mailbox. GetStatus also suspended the mailbox on an unbounded outbox count.
- Gated ingest bursts go to a bounded channel (64 bursts) drained by one
ordered append loop, which commits every burst waiting behind a slow
append in a single AppendBatchAsync (FIFO kept). Only a full queue holds
the mailbox (backpressure, bounded memory). Counters are Interlocked; the
loop nudges the drain. PostStop flushes queued bursts within the shared
5 s stop budget.
- GetStatus reads counters on the actor thread and pipes a count bounded by
a 10 s timeout (-1 when it does not answer); it no longer suspends the
mailbox. RecorderStatus gains PendingAppends, MailboxDepth and
AppendBackpressureWaits; two gauges publish pending appends and mailbox
depth.
- ContinuousHistorization:CommitMode defaults to Periodic (100 ms). The
Host's appsettings.json, its unparseable-value fallback and docker-dev
still say PerEntry: filed as a Host follow-up (outside this lane).
Rebased from the unpushed remed/lane-H2 branch (originally authored 2026-10-22
in session_01Ub9hGmYLQvn9XETzw7CMsT) onto current master.
Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
The QuestDB live writer waited for WAL visibility after every tag group, and
the wait polled wal_tables() every 10 ms. With DrainConcurrency = 8 that was
up to eight pollers at ~100 queries/s each against a database whose WAL
apply was already behind.
New Runtime seam IHistorianPassWriter (WriteLiveValuesInPassAsync +
FlushAsync). The recorder writes every group of a pass through it and calls
FlushAsync once after all groups report, before the ack, whenever any group
wrote (an un-acked pass is redelivered and its insert-if-absent check must
see this pass's rows). A flush failure does not fail the pass. The QuestDB
writer implements it: in-pass writes only mark a flush as owed; FlushAsync
waits once. The poll now backs off 10 -> 20 -> ... -> 200 ms, never past the
timeout. Direct WriteLiveValuesAsync callers still wait per call.
Also updates the QuestDB cancelled-write integration test to the rethrow
contract from the previous commit.
Rebased from the unpushed remed/lane-H2 branch (originally authored 2026-10-22
in session_01Ub9hGmYLQvn9XETzw7CMsT) onto current master.
Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
Akka TestKit's AwaitCondition without an interval polls every max/10 of the
DILATED budget. Measured here: a condition that turned true at 2 s under a
15 s budget returned after 3003 ms at timefactor 1 (3 polls) and 4502 ms at
timefactor 3 (2 polls). AwaitAssert / AwaitAssertAsync are unaffected (a
fixed 100 ms: 21 polls in ~2 s at both factors).
RuntimeActorTestBase and ControlPlaneActorTestBase now declare
AwaitCondition overloads that hide TestKit's and default the interval to
PollInterval (20 ms), so every subclass call without an interval binds to
them (measured after: returns 5-22 ms after the condition holds, at both
factors). PresenceBudget's doc no longer claims the helpers return "the
instant the condition holds" - it states the per-helper poll behaviour.
Bare-TestKit classes with AwaitCondition calls get an explicit interval:
TelemetryDialDeadlineTests / TelemetryDialSupervisorTests (next to their own
undilated 5 s / 8 s WaitForCount deadlines and a 1 s dial deadline) and
OpcUaEndpointStepDownActorTests (RuntimeActorTestBase.PollInterval, now
public like PresenceBudget). ContinuousHistorizationRecorderTypedValueTests'
AwaitConditionAsync was left alone (historian code is being changed in
parallel).
Suite results (Release, 0 warnings):
- Runtime.Tests 946/3 skipped/0 failed at timefactor 3 (1 m 45 s, was
2 m 57 s) and 1 (1 m 18 s, was 2 m 57 s);
- ControlPlane.Tests 319/0 failed at 3 (1 m 51 s) and 1 (1 m 11 s);
- AdminUI.Tests Telemetry 15/0 failed at 3 and 1.
Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
DriverInstanceActor.RunWriteAsync cancelled every write with a hard-coded,
undilated 5 s CancellationTokenSource, so writeTimeout / DefaultWriteTimeout
(then 10 s) only ever served as the pessimistic abandon bound: a configured
bound above 5 s never took effect.
The write's token is now cancelled at the configured write timeout; a write
that ignores its token is abandoned WriteAbandonGrace later (half the
timeout, at most 2 s) so a token-honouring driver still gets to answer the
cancellation itself. Both bounds run on an injectable TimeProvider
(Props writeClock, System by default).
DefaultWriteTimeout is now 5 s (cancel 5 s, abandon 7 s), not 10 s: the
host's write Ask is 8 s, and a write allowed to run past it would already
have been reported as a timeout and its node reverted while the device could
still take the value - the exact failure the off-mailbox write work fixed.
Production behaviour is therefore unchanged for the cancel (5 s) and tighter
for a token-ignoring driver (abandoned at 7 s instead of 10 s, i.e. inside
the Ask). DriverHostActor's Ask comment updated to match.
Tests (DriverInstanceActorConcurrentWriteTests, manual clock copied from
Ab.Transport.Tests into Runtime.Tests/Harness/ManualTimeProvider):
- a write held past 6 s under a 10 s write timeout succeeds;
- positive control: at 10 s the token is cancelled ("write timeout");
- a token-ignoring write is still pending at 10 s and abandoned (BadTimeout)
after the grace.
Mutation checks: a fixed 5 s token on the same clock fails the first test;
dropping the grace fails the third (and the second, where the abandon then
races the cancellation).
342a825d re-check: its held-write tests were written around the undilated
5 s cap. Connect() now defaults the write timeout to the DILATED presence
budget, so a held, explicitly-released write no longer races any undilated
deadline; tests about the bound pass their own. AwaitWritesEntered keeps its
20 ms interval (doc updated).
Runtime.Tests: 946 passed / 3 skipped / 0 failed at AKKA_TEST_TIMEFACTOR 3
and 1. Solution Release build: 0 warnings.
Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
A_queued_write_older_than_the_budget_is_failed_BadTimeout_instead_of_written_late
failed hosted runs 2215 (7ec5b95d) and 2260 (8ef5adc6) at
"(await inFlight).Success should be True but was False", after 7 s each time.
It is not load: it fails on an idle machine with AKKA_TEST_TIMEFACTOR=3, the
value v2-ci sets, and passes with 1.
Akka's AwaitCondition with no interval polls every max/10 of the dilated
budget: PresenceBudget 15 s x 3 = 45 s, so a 4.5 s sleep after a first poll
that runs before the pool has entered the write. The held write is bounded by
DriverInstanceActor.RunWriteAsync's fixed, undilated 5 s
CancellationTokenSource, so 4.5 s of polling plus the test's 600 ms settle
cancelled it ("write timeout") before ReleaseWrites(). The sibling tests that
hold a write took 7 s for the same reason and passed only by margin.
All eight AwaitCondition calls in the class now go through
AwaitWritesEntered, which polls every 20 ms. The budget (PresenceBudget)
and every assertion are unchanged.
Local, AKKA_TEST_TIMEFACTOR=3: before 6/7 (this test failing, 7 s); after
7/7 in three runs, test 3 s.
Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
Residuals of #760:
1. An ack routed to a disabled / not-yet-spawned driver, or to a stubbed child
(driver does not run on this node, or its factory rejected the config), was
dropped and left the condition Acknowledged. DriverHostActor now reverts it
through the same NativeAlarmProjector.TryRevertAcknowledge path as a refused
ack.
2. A native ack arriving while the host is Stale was dropped before the projector
recorded it, so it could never be reverted. Stale now marks the whole
NativeAlarmAckPreflightView unavailable (SetHostUnavailable), so the node
manager refuses every native ack synchronously; BecomeSteady clears it.
3. (keyed sessions without a Galaxy identity) is not addressed.
Tests: DriverHostActorNativeAlarmTests' client-ack test and the FOCAS respawn
theory now start the host's own driver (AckingAlarmDriver; a factory building a
real FocasDriver per spawn over per-instance fake CNCs, with a real config-change
respawn) - both fail against the fix in their old shape. New: ack to a disabled
driver reverts, ack to a stubbed driver reverts, a Stale host refuses through the
pre-check view - all three fail with the fix mutated out. Runtime.Tests 943/0/3.
Docs: AlarmTracking.md, .claude/rules/alarms.md.
Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
AbLegacyDriver stamped LastSuccessfulRead (init, read, write), the read's sample
timestamps and host-state change times from raw DateTime.UtcNow; only the backoff
path read the injectable Clock. All of them now read Clock(), so
NativePcccBridgeStatusHealthTests drives a manual clock (init at T0, read at T1)
and asserts LastSuccessfulRead == T1 exactly instead of spin-waiting for the wall
clock to pass the init stamp (#767). Without the driver change the 6 tests fail
(init stamp is wall-clock, not T0).
Directory.Build.props sets TreatWarningsAsErrors for every project since #766, so
the 75 per-csproj copies (64 src + 11 tests) are removed and the props comment
says it is now the only place. tools/mem-probe keeps its explicit opt-out.
Release build: 0 warnings, 0 errors; a planted CS0168 still fails the build in a
src project and in a test project that used to carry its own copy.
Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
The native-ack path is fire-and-forget after the Part 9 Acknowledge returns, so
a Galaxy ack refused for lack of a credential (#749), or rejected by MXAccess,
left the client with Good and the condition showing Acknowledged while Galaxy
was never acked. Both options from the issue, layered:
1. Synchronous refusal for what the driver can decide without I/O. New optional
IAlarmAcknowledgePreflight (Core.Abstractions); GalaxyDriver refuses an
anonymous session when the driver has no AlarmAck fallback account.
DriverHostActor publishes a condition NodeId -> owning-driver snapshot into
NativeAlarmAckPreflightView (lock-free, volatile); the node manager's new
NativeAlarmAckPreflight seam, consulted after the authz and Primary gates,
returns BadUserAccessDenied with the reason - nothing routed, applied or
reported. A keyed session is not pre-checked (its identity lives in the
Secrets store, which must not be read under the SDK lock).
2. Revert for everything else. RouteAlarmAck carries the condition NodeId;
DriverInstanceActor replies AlarmAckFailed on a refusal, a backend failure,
a driver without an alarm source, or an ack dropped while connecting /
reconnecting. DriverHostActor re-projects the condition unacknowledged via
NativeAlarmProjector.TryRevertAcknowledge, but only while the local ack is
still pending - any driver transition projected since then wins. A disabled
or unspawned driver is not reverted (unchanged).
GatewayGalaxyAlarmAcknowledger now throws DriverOperationRefusedException on a
non-zero native hresult (wrong password, no ack permission) instead of only
warning, so that case reverts too; as a refusal it neither retries nor trips
the breaker.
Tests (each fails with its fix mutated out): projector revert/supersede,
DriverHostActor refused + failed ack -> unacked AlarmStateUpdate (successful
ack control), preflight view snapshot, node-manager preflight unit test,
over-the-wire ConditionMethodEventTests (BadUserAccessDenied, no route, no
event; positive control), Galaxy preflight + hresult refusal.
Docs: Galaxy.md, AlarmTracking.md, ScriptedAlarms.md, .claude/rules/alarms.md.
Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
IOpcUaEndpointState (review-remediation lane C3) recorded whether the SDK started, but only the
health check read it: a fused node whose endpoint failed (port 4840 taken, ...) still advertised
250/240 and stayed driver Primary, so the Primary-gated data plane ran on a node no client could
reach while the reachable peer refused writes as Secondary.
- Move IOpcUaEndpointState/OpcUaEndpointState/OpcUaEndpointStatus from Host to Commons.OpcUa so
the Runtime can consume it.
- ServiceLevelCalculator: new NodeHealthInputs.EndpointListening (default true). false -> 0,
ahead of every other input including the +10 Primary bonus.
- OpcUaPublishActor: takes the endpoint state, feeds EndpointListening (Failed only; NotStarted /
Stopped are boot/shutdown windows) on the DB-backed and DB-less paths, and the legacy role-only
seam returns 0 for a failed endpoint too.
- New OpcUaEndpointStepDownActor (one per driver node): a Primary (ClusterActiveNode.SelfIsActive,
the election's own primitive) with a Failed endpoint gracefully Leaves its mesh - the manual
failover mechanism - so the peer becomes the oldest Up driver member, hosts the singletons and is
Primary. The election itself is not bent. Guarded on a peer having recently reported its own
endpoint listening (EndpointReport on the new opcua-endpoint-state DPS topic), so a pair with
both endpoints failed does not bounce the role and restart nodes forever, and a lone node never
shuts itself down.
- Runtime now references ZB.MOM.WW.Health.Akka (ClusterActiveNode).
- Docs: Redundancy.md tier table + new "Endpoint start failure: Primary step-down" section;
cluster-topology rule; HealthEndpoints / OpcUaEndpointHealthCheck comments (F-C1 landed).
Tests: ServiceLevelCalculatorTests (+5), OpcUaPublishActorEndpointStateTests (7: DB-less,
DB-backed, legacy seam, NotStarted/Stopped do not demote, a later failure demotes on the next
tick), OpcUaEndpointStepDownActorTests (3, real two-node mesh: holds while the peer is not
listening then leaves and the peer becomes Primary; failed Secondary and healthy Primary never
leave). Each was mutation-checked: removing the calculator/legacy guard, the Leave, the
peer-listening guard or the Primary guard fails the corresponding tests.
mxaccessgw 488649c+ acknowledges by AuthenticateUser as the operator and
writing <alarm>.AckMsg, so AcknowledgeAlarmRequest now needs a real Galaxy
operator_user + operator_domain + operator_password. The OPC UA data plane is
API-key/anonymous and carries no Galaxy password, so the driver resolves the
account per ack:
1. the session's API key's Galaxy ack identity, set on AdminUI /apikeys and
held ONLY in the encrypted Secrets store
(otopcua/apikeys/<keyId>/galaxy-ack-{user,password}); a key with a user but
no password secret is refused, never silently downgraded;
2. else the driver instance's AlarmAck fallback account (DriverConfig
FallbackUser/FallbackDomain/FallbackPasswordSecretRef; secret:/${secret:}/
env:/file:/dev: only — a literal is refused);
3. else DriverOperationRefusedException: nothing is sent, never an empty
operator_user. Refusals are excluded from the resilience retry/breaker.
- Pins ZB.MOM.WW.MxGateway.Client/Contracts 0.2.1 -> 0.3.0 (vendored into
nuget-packages/ from the Gitea feed, per the local-mxgw source mapping).
- The session ApiKeyId (not a secret) rides NativeAlarmAck ->
RouteNativeAlarmAck -> RouteAlarmAck -> AlarmAcknowledgeRequest; the password
is resolved in the driver and never travels through actor messages.
- The OPC UA principal + key id are appended to the AckMsg comment
("[OPC UA: <principal>, key <id>]") and logged, so a shared fallback
account does not lose who acknowledged.
- Password never logged: GalaxyAckCredential.ToString omits it, the
acknowledger scrubs it from every gateway message it echoes and withholds
any exception that echoes it; audit rows carry only the key id.
- AdminUI: /apikeys "Ack identity" editor (set/remove; delete-key clears it),
Galaxy driver form gains the fallback section.
- Synchronous OPC UA status for a refusal is follow-up #760 (the native ack
path is fire-and-forget after the Part 9 method returns).
Tests: request mapping (password populated), password absent from logs and
exception text (mutation-checked), key-mapped / fallback / refused paths at
resolver and driver level, breaker non-trip, key-id propagation through node
manager and DriverHostActor, AdminUI service + form round-trip.
Closes#749
I-1: a node whose in-memory trail is one short of the stored trail (it missed
a command the peer applied, whose entry then replicated in) silently dropped
its own next entry — in failover, the survivor's newest ack — because the
positional "past MAX(seq)" rule treated it as already stored.
SaveAsync now reconciles the last 16 in-memory entries against the stored
rows from that position on, by content (Kind/User/Text, timestamps within a
5-minute skew window; matched rows are consumed so repeats count). The peer's
copy of the same command is recognised and not re-appended; an unmatched
entry is inserted at its own ordinal when free, otherwise appended at
MAX(seq)+1 with a Warning — kept, never silent. Bounded read, constant write.
m1: MigrateInlineComments copies only — it no longer UPDATEs the condition
row (a fresh-HLC re-stamp could beat a newer peer ack under LWW); the next
save clears comments_json. m2: it returns unparseable alarm ids and OnReady
logs a Warning per id (+ an Information tally). m3/m4: documented in the
store remarks (Remove vs lagging peer; in-memory index == durable ordinal).
Tests: store-level failover sequence (missed command -> survivor's ack and
next entry persisted once, re-save writes 0 rows, one Warning) with the
dual-write positive control (peer copy present -> 0 rows, no Warning) and a
repeated-identical-comment case; a loopback pair failover test (link down,
survivor acks, rejoin -> both entries on both peers, held); migration
no-row-rewrite, partial fill and unparseable reporting; OnReady Warning.
Both failover tests fail against the round-0 store.
Refs #748
Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
Every Part 9 transition used to re-serialize the alarm's whole comment trail
into alarm_condition_state.comments_json and replicate it to the pair peer:
O(n) per transition, O(n^2) over an alarm's life (measured: one ack at 10k
comments shipped 1,080,553 oplog bytes; now ~1.6 KB, the same as at 10).
- New replicated table alarm_condition_comment, PK (scripted_alarm_id, seq),
seq = the entry's ordinal in the trail. Ordinal (not content) keying makes
the two pair nodes' copies of one logical entry (each stamped by its own
clock, since both nodes apply every command) collapse to one row under LWW.
- LocalDbAlarmConditionStateStore.SaveAsync appends only entries past the
stored MAX(seq) (INSERT OR IGNORE, append-only) and upserts a fixed-size
condition row with comments_json = "[]", in one transaction. Remove drops
the trail with the row. Load merges the table with any inline trail.
- comments_json column kept (always "[]") so the condition table's shape and
replication digest are unchanged and old-shape rows still load exactly.
- AlarmConditionStateSchema.MigrateInlineComments: idempotent per-row move of
inline trails, run by LocalDbSetup.OnReady after RegisterReplicated so the
moved entries replicate. Comment table joins ReplicatedTables (now five).
Tests: constant transition payload at 10 vs 10k comments (+ instrument
positive control), old-shape back-compat load, migration idempotency /
partial re-run / lazy move, OnReady boot migration captured in the oplog,
and two loopback pair tests (A->B and B->A trail convergence; the shared
entry converges to one copy, with a distinct-alarm positive control).
Refs #748
Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
I-1: a PUT/GET-disabled CPU (every read denied) stayed Faulted with
LastSuccessfulRead frozen, so the comms-loss watchdog reconnected and
re-subscribed it every 90 s forever. Both access-denied arms now also record
an answer: Faulted is kept (worst state wins), the stamp advances, and the
sticky write fault still clears only on a real success. Pinned end to end in
Runtime.Tests: the real S7Driver under the real DriverInstanceActor watchdog
is not reconnected across 2x the threshold; the control (every read timing
out) is.
Minors:
- m-1: injectable pass clock (UtcNow); "LastSuccessfulRead advanced" is now
asserted strictly, including the write-refusal case.
- m-2: the editor uses S7TagShapeRules.IsByteAnchored instead of its own list.
- m-3: a release that races an init publish also cancels the probe loop the
publish started after the release took its snapshot.
- m-4: a retained poll failing during a lifecycle operation writes no health.
- m-5: a multi-var chunk that fell back memoises its members refused alone.
- m-7: access-denied needs S7.Net's exact wording and no transport fault in
the chain.
- m-8: sync teardown clears the write-side Degraded owner; a sync Dispose
racing a running shutdown is tested.
Refs #742
Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
N3 (critical): the node manager carries only NATIVE conditions' state over a
full rebuild; scripted conditions come back fresh and rely on the engine's
re-assert, which since the off-thread plan is handled in the window and lands
on the discarded instance. On a rebuild the condition replay scope is now
AddedConditions plus every enabled scripted condition in the new composition
(native kept conditions stay carried-over, not replayed). The test sink again
models the real rebuild: it clears non-native condition state only.
N4: a surgical in-place shape edit (DataType / IsArray / ArrayLength) resets the
raw node and its UNS mirror to BadWaitingForInitialData without ensuring them,
so the window's value was not replayed and the two NodeIds disagreed. The
applier records re-shaped nodes (MaterialisationRecording.Reshaped); their
deferred value replays only once fitted to the new shape (VariableShape.TryFit:
as is, or converted invariantly; never an old-typed value - WriteValues assigns
the value verbatim with no type check), otherwise it is dropped at Debug and the
node keeps waiting. Status-only entries are not replayed onto a re-shaped node.
Minors: m1/m2 documented on _replayScope (degraded-path only); m3 corrected the
false carry-over premise in the scope doc (the ScriptedAlarmHostActor ordering
remark is outside this lane - F8); m4 the replay Debug log reports alarm-state
and alarm-quality counts again.
Refs #732
Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
Review I-2. Under factory Props a supervision restart of
ScriptedAlarmHostActor builds an empty engine, and nothing re-sent
ApplyScriptedAlarms. Until the next deploy the host had no mux interest
(inputs stopped), and its ownership filter silently dropped every
operator ack and shelve.
The host now tells its parent ScriptedAlarmHostRestarted from
PostRestart. DriverHostActor keeps the last plans it applied and
re-sends ApplyScriptedAlarms when the signal comes from the current
host, in Steady, Applying and Stale. This mirrors how a driver child's
ReprovisionRequested is answered. The OnAlarmsLoaded comment, which
assumed a re-apply that never happened, now says where it comes from.
DriverHostActorScriptedAlarmRestartTests deploys one scripted alarm
through a real host, crashes the alarm host, does NOT re-apply by hand,
and asserts that interest is re-registered, a dependency push activates
the alarm, and an operator ack reaches the engine. It is red without
the signal (times out waiting for the re-registration).
Refs #731#734
Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
N1: a full alarm-state snapshot also sets the condition's Quality, so it now
supersedes an earlier quality-only write for the same id in the deferred
buffer; a quality arriving after the state is kept and replays after it.
N2 / M1: the replay is not inert on a node the apply KEPT - an SDK-applied
Acknowledge/Confirm/Shelve or an optimistic client Write in the window would
be rolled back (with spurious Part 9 events). The replay is now scoped to
nodes the apply created: variables the applier ensured during this apply
(AddressSpaceApplier.BeginRecordingEnsuredVariables) or every variable after a
full rebuild; conditions in the applied composition but not the previous one
(also on a rebuild, which carries a kept condition's state over). Option (b),
a sink that reports whether a live write landed, needs the Commons
DeferredAddressSpaceSink seam outside this lane (follow-up F7).
M2: the benchmark remark now matches its Category=Benchmark trait.
M3: TestHooks and the seam-carrying constructor are internal; PropsForTests
produces the actor through an IIndirectActorProducer.
M4: pinned replay-nothing-and-drop-the-buffer after a failed plan.
M5: PreRestart answers every queued rebuild requester (degraded).
Refs #732
Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
Review I-1. Context.Stop is asynchronous, so a removed child kept
publishing through the rest of the apply handler and its own stop.
Those bursts were forwarded after SourceRemoved and re-retained the
removed driver's last Good in the mux, where nothing would ever clear
it while a replay subscriber held the ref (a removed driver never
publishes again, and its PostStop Bad fan resolves no refs once the
index is rebuilt without it).
StopChild and HandleRestartDriver now retire the child (_retiredChildren
+ Context.Watch) before stopping it. ForwardBatchToMux drops a burst
whose Sender is retired, and Terminated (handled in Steady, Applying and
Stale) removes the entry. Akka delivers Terminated after the watched
child's last message, so the set is bounded and a live incarnation is
never filtered. ConnectivityChanged from a retired child is
deliberately not filtered, because the respawn path relies on its Bad
fan.
Tests (review m-3 folded in): a stale burst spoofed as the removed child,
held alive in PostStop by a new RecordingSubscribableDriver.ShutdownGate,
no longer re-retains (red with the guard disabled). New cases cover a
config-change respawn and an operator Restart, each with the new
driver's value replaying as the positive control. The arrange step now
waits for the real subscription set and fishes for both drivers'
values at once (two sequential fishes could discard the other burst).
Refs #731#734
Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
It had no production sender: every driver data change is already a
batch, and the comms-loss fan was converted earlier. DependencyMuxActor
drops its per-value receive, DriverInstanceActor drops the record, and
DependencyMuxActorTests' 13 sends move to the ValuePublishes.Single
one-item batch (same semantics through the batch path). XML docs and
the living docs (DriverLifecycle, VirtualTags, ScriptEditor) are
reworded; historical plan docs are left as written.
Refs #731#734
Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
DriverHostActor now tells the mux SourceRemoved(driverInstanceId)
wherever a driver object ends: StopChild (deploy removal and the stop
half of a config-change respawn) and an operator Restart. It is not
sent on a supervision restart, which re-uses the same driver. Before
this nothing sent it, so a removed driver's last Good stayed in the
mux and replayed to the next replay-opting subscriber as current.
DriverHostActorSourceRemovedTests drives a real mux with two real
driver children: after a deploy drops DI-1 only DI-2's value replays
(positive control: both replay before the removal), and a supervision
restart of DI-1 leaves its value replayable.
Residual (documented on EvictRetainedValues): a burst the old child
enqueued just before it stopped can still be forwarded after the
eviction; the next publish, a comms-loss fan or the grace sweep
bounds it.
Refs #731#734
Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
C1 (critical): B6 computes the plan off the pinned thread while B1 made writes
non-creating, so a value/status/alarm write for a node the in-flight plan adds
was skipped, and change-driven drivers never republish it (BadWaitingForInitialData
indefinitely; on a full rebuild every re-subscribe value landed on discarded
nodes). OpcUaPublishActor now applies writes immediately AND remembers the
latest per (realm, id) while a plan is in flight or queued (value+status
compose; alarm state/quality coalesce to the latest, which the node's delta
gate makes inert on an existing condition), replays it after each apply before
answering, and drops it once the pipeline is idle. Bounded (250k keys by
default; beyond it new keys are not held, logged once).
I2: the data-plane-authority parse runs in its own try inside the plan task; a
throw is logged and never degrades the address-space apply.
I3: the SDK's DeleteMonitoredItems releases an item's component-cache
reference on a Good delete; a tombstoned item's reference was already
released, so the delete took one from the re-created node's entry. Released
items are remembered and the orphan delete takes one back first. Event items
never held a reference (EventManager + SubscribeToEvents), so the three
event-item releases are gone.
I4: direct tests for the HistoryUpdate override's per-request updater check
and the HistoryReadEvents override's source guard (both reached by restoring
the bit after a reset - the B7 race window); fixed the false remark.
Minors: the MaterialiseAlarmCondition kind-swap tombstones the dropped
instance; a bytes-less rebuild arriving mid-plan queues behind it (replies in
order); a PlanReady/PlanFailed from a previous actor incarnation is ignored and
its requester answered degraded; the tie-cluster end bound of an uncapped read
fits against the backend ceiling (defence in depth - guarded by a sweep test);
the lock-time benchmark stays in the default run, headroom noted.
Refs #732
Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
- ScriptLogRateLimiter.LimitPerWindow. The ScriptLogTopicSink and
VirtualTagActor summaries now print the configured limit, not
DefaultLimitPerWindow. Tests assert "limit N/window" for a non-default
N.
- CalculationDriver.Dispose cancels, then takes _evalGate (bounded 2 s)
and disposes the gate and _evalCts. The pump and timer ticks enter the
gate through TryEnterEvalGateAsync, which stops cleanly once the
driver is disposed. Test: Dispose with a 6 s runaway run in flight
returns promptly, disposes both, and a late trigger is harmless.
- ScriptedAlarmEngine captures _shutdownToken once and disposes
_shutdownCts in Dispose when every drained task has completed. A task
stuck past the grace leaves the CTS to the GC. Lifecycle tests assert
the disposal.
- DraftValidator: CalculationNoEffectiveTrigger is skipped for a
change-triggered tag whose scriptId does not resolve. That case is
already CalculationScriptMissing or CalculationScriptNotFound, and
there are no reads to count. Both existing tests now assert the
absence (red before the change).
- docs/VirtualTags.md: System.Threading is denied as a whole namespace,
with an allow-list; the type list is now accurate (Unsafe,
RuntimeHelpers, and the ForbiddenMembers on System.Type); the resource
limits section covers the run guard and the allocation budget.
docs/ScriptEditor.md: accurate deny-list and a new "Runtime safety"
section covering timeout, refusal, quarantine after 3 timeouts,
allocation budget, lifting, and rate-limit summaries that appear only
on the source's next entry after a burst then quiet.
Refs #734
Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
The fixture wedged EVERY write on one shared token-ignoring release gate, so
the later same-tag write was also hung, and the 300 ms pessimistic bound
abandoned it while the test was still polling before ReleaseWrites(). Its
reason was "write did not complete within" (the bound), not "queued for" (the
queue budget), and it had already entered the driver, so the tag had been
released. The release path is correct. Only the first write now hangs, and
later writes complete at once.
The abandon reason formatted the bound with TotalSeconds:F0, which printed a
300 ms bound as "0s". It now uses milliseconds, and the test asserts it.
Refs #731
Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
DependencyMuxActor retained the last value of every ref ever registered,
and never removed the entries. The continuous-historization recorder
registers every historized ref, so on a historizing node the map grew
with the historized address space and never shrank. The class remark
claimed a bound that did not hold.
- Retain only refs that at least one subscriber registered with
ReplayRetained: true names (VT actors, the Calculation mux adapter, the
scripted-alarm host). A non-opting subscriber is still delivered to but
retains nothing.
- When a ref's last replay-opting subscriber leaves, its value is kept
for a grace (default 2 min, Props(retentionGrace)) so a restart
(old adapter gone, new one not yet registered) still replays. A
periodic sweep then drops it along with the entry. Entries with no
subscriber and nothing retained are removed.
- New SourceRemoved(DriverInstanceId) evicts the values a removed driver
produced, so its last Good is never replayed. The mux records the
publishing driver per retained ref. No production sender exists yet:
DriverHostActor is outside this lane (follow-up).
- Remark rewritten. Tests: retention bounded to replay-opting refs,
release after grace (with an in-grace replay as the positive control),
SourceRemoved eviction, and the restart-replay scenario still green.
Refs #734
Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
Joseph Doherty