HistorianAdapterActor fired `_ = EnqueueAsync(...)` per alarm transition
with no in-flight cap, so an alarm storm (a native source re-raising
hundreds of conditions on reconnect) spawned hundreds of concurrent SQLite
inserts that serialised on the file lock and held pool threads. It also kept
a Receive<AlarmHistorianEvent> with no production sender: a transition told
directly AND published on `alerts` would have been historized twice.
Transitions now go to a bounded channel (10000) drained by one loop, so the
sink sees one enqueue at a time in arrival order; a full queue holds the
mailbox (backpressure, nothing dropped); PostStop flushes the queue within
the shared 5 s stop budget. The direct receive is deleted. The gate tests
drive transitions instead; a directly-told AlarmHistorianEvent is shown to
be unhandled against a positive control.
Rebased from the unpushed remed/lane-H2 branch (originally authored 2026-10-22
in session_01Ub9hGmYLQvn9XETzw7CMsT) onto current master.
Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
The continuous-historization recorder awaited every outbox append (an fsync
under PerEntry) on its mailbox, so ingestion was capped at 1/fsync_latency
bursts per second while the mux kept telling into an unbounded, unmetered
mailbox. GetStatus also suspended the mailbox on an unbounded outbox count.
- Gated ingest bursts go to a bounded channel (64 bursts) drained by one
ordered append loop, which commits every burst waiting behind a slow
append in a single AppendBatchAsync (FIFO kept). Only a full queue holds
the mailbox (backpressure, bounded memory). Counters are Interlocked; the
loop nudges the drain. PostStop flushes queued bursts within the shared
5 s stop budget.
- GetStatus reads counters on the actor thread and pipes a count bounded by
a 10 s timeout (-1 when it does not answer); it no longer suspends the
mailbox. RecorderStatus gains PendingAppends, MailboxDepth and
AppendBackpressureWaits; two gauges publish pending appends and mailbox
depth.
- ContinuousHistorization:CommitMode defaults to Periodic (100 ms). The
Host's appsettings.json, its unparseable-value fallback and docker-dev
still say PerEntry: filed as a Host follow-up (outside this lane).
Rebased from the unpushed remed/lane-H2 branch (originally authored 2026-10-22
in session_01Ub9hGmYLQvn9XETzw7CMsT) onto current master.
Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
The QuestDB live writer waited for WAL visibility after every tag group, and
the wait polled wal_tables() every 10 ms. With DrainConcurrency = 8 that was
up to eight pollers at ~100 queries/s each against a database whose WAL
apply was already behind.
New Runtime seam IHistorianPassWriter (WriteLiveValuesInPassAsync +
FlushAsync). The recorder writes every group of a pass through it and calls
FlushAsync once after all groups report, before the ack, whenever any group
wrote (an un-acked pass is redelivered and its insert-if-absent check must
see this pass's rows). A flush failure does not fail the pass. The QuestDB
writer implements it: in-pass writes only mark a flush as owed; FlushAsync
waits once. The poll now backs off 10 -> 20 -> ... -> 200 ms, never past the
timeout. Direct WriteLiveValuesAsync callers still wait per call.
Also updates the QuestDB cancelled-write integration test to the rethrow
contract from the previous commit.
Rebased from the unpushed remed/lane-H2 branch (originally authored 2026-10-22
in session_01Ub9hGmYLQvn9XETzw7CMsT) onto current master.
Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
Akka TestKit's AwaitCondition without an interval polls every max/10 of the
DILATED budget. Measured here: a condition that turned true at 2 s under a
15 s budget returned after 3003 ms at timefactor 1 (3 polls) and 4502 ms at
timefactor 3 (2 polls). AwaitAssert / AwaitAssertAsync are unaffected (a
fixed 100 ms: 21 polls in ~2 s at both factors).
RuntimeActorTestBase and ControlPlaneActorTestBase now declare
AwaitCondition overloads that hide TestKit's and default the interval to
PollInterval (20 ms), so every subclass call without an interval binds to
them (measured after: returns 5-22 ms after the condition holds, at both
factors). PresenceBudget's doc no longer claims the helpers return "the
instant the condition holds" - it states the per-helper poll behaviour.
Bare-TestKit classes with AwaitCondition calls get an explicit interval:
TelemetryDialDeadlineTests / TelemetryDialSupervisorTests (next to their own
undilated 5 s / 8 s WaitForCount deadlines and a 1 s dial deadline) and
OpcUaEndpointStepDownActorTests (RuntimeActorTestBase.PollInterval, now
public like PresenceBudget). ContinuousHistorizationRecorderTypedValueTests'
AwaitConditionAsync was left alone (historian code is being changed in
parallel).
Suite results (Release, 0 warnings):
- Runtime.Tests 946/3 skipped/0 failed at timefactor 3 (1 m 45 s, was
2 m 57 s) and 1 (1 m 18 s, was 2 m 57 s);
- ControlPlane.Tests 319/0 failed at 3 (1 m 51 s) and 1 (1 m 11 s);
- AdminUI.Tests Telemetry 15/0 failed at 3 and 1.
Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
DriverInstanceActor.RunWriteAsync cancelled every write with a hard-coded,
undilated 5 s CancellationTokenSource, so writeTimeout / DefaultWriteTimeout
(then 10 s) only ever served as the pessimistic abandon bound: a configured
bound above 5 s never took effect.
The write's token is now cancelled at the configured write timeout; a write
that ignores its token is abandoned WriteAbandonGrace later (half the
timeout, at most 2 s) so a token-honouring driver still gets to answer the
cancellation itself. Both bounds run on an injectable TimeProvider
(Props writeClock, System by default).
DefaultWriteTimeout is now 5 s (cancel 5 s, abandon 7 s), not 10 s: the
host's write Ask is 8 s, and a write allowed to run past it would already
have been reported as a timeout and its node reverted while the device could
still take the value - the exact failure the off-mailbox write work fixed.
Production behaviour is therefore unchanged for the cancel (5 s) and tighter
for a token-ignoring driver (abandoned at 7 s instead of 10 s, i.e. inside
the Ask). DriverHostActor's Ask comment updated to match.
Tests (DriverInstanceActorConcurrentWriteTests, manual clock copied from
Ab.Transport.Tests into Runtime.Tests/Harness/ManualTimeProvider):
- a write held past 6 s under a 10 s write timeout succeeds;
- positive control: at 10 s the token is cancelled ("write timeout");
- a token-ignoring write is still pending at 10 s and abandoned (BadTimeout)
after the grace.
Mutation checks: a fixed 5 s token on the same clock fails the first test;
dropping the grace fails the third (and the second, where the abandon then
races the cancellation).
342a825d re-check: its held-write tests were written around the undilated
5 s cap. Connect() now defaults the write timeout to the DILATED presence
budget, so a held, explicitly-released write no longer races any undilated
deadline; tests about the bound pass their own. AwaitWritesEntered keeps its
20 ms interval (doc updated).
Runtime.Tests: 946 passed / 3 skipped / 0 failed at AKKA_TEST_TIMEFACTOR 3
and 1. Solution Release build: 0 warnings.
Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
A_queued_write_older_than_the_budget_is_failed_BadTimeout_instead_of_written_late
failed hosted runs 2215 (7ec5b95d) and 2260 (8ef5adc6) at
"(await inFlight).Success should be True but was False", after 7 s each time.
It is not load: it fails on an idle machine with AKKA_TEST_TIMEFACTOR=3, the
value v2-ci sets, and passes with 1.
Akka's AwaitCondition with no interval polls every max/10 of the dilated
budget: PresenceBudget 15 s x 3 = 45 s, so a 4.5 s sleep after a first poll
that runs before the pool has entered the write. The held write is bounded by
DriverInstanceActor.RunWriteAsync's fixed, undilated 5 s
CancellationTokenSource, so 4.5 s of polling plus the test's 600 ms settle
cancelled it ("write timeout") before ReleaseWrites(). The sibling tests that
hold a write took 7 s for the same reason and passed only by margin.
All eight AwaitCondition calls in the class now go through
AwaitWritesEntered, which polls every 20 ms. The budget (PresenceBudget)
and every assertion are unchanged.
Local, AKKA_TEST_TIMEFACTOR=3: before 6/7 (this test failing, 7 s); after
7/7 in three runs, test 3 s.
Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
Residuals of #760:
1. An ack routed to a disabled / not-yet-spawned driver, or to a stubbed child
(driver does not run on this node, or its factory rejected the config), was
dropped and left the condition Acknowledged. DriverHostActor now reverts it
through the same NativeAlarmProjector.TryRevertAcknowledge path as a refused
ack.
2. A native ack arriving while the host is Stale was dropped before the projector
recorded it, so it could never be reverted. Stale now marks the whole
NativeAlarmAckPreflightView unavailable (SetHostUnavailable), so the node
manager refuses every native ack synchronously; BecomeSteady clears it.
3. (keyed sessions without a Galaxy identity) is not addressed.
Tests: DriverHostActorNativeAlarmTests' client-ack test and the FOCAS respawn
theory now start the host's own driver (AckingAlarmDriver; a factory building a
real FocasDriver per spawn over per-instance fake CNCs, with a real config-change
respawn) - both fail against the fix in their old shape. New: ack to a disabled
driver reverts, ack to a stubbed driver reverts, a Stale host refuses through the
pre-check view - all three fail with the fix mutated out. Runtime.Tests 943/0/3.
Docs: AlarmTracking.md, .claude/rules/alarms.md.
Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
AbLegacyDriver stamped LastSuccessfulRead (init, read, write), the read's sample
timestamps and host-state change times from raw DateTime.UtcNow; only the backoff
path read the injectable Clock. All of them now read Clock(), so
NativePcccBridgeStatusHealthTests drives a manual clock (init at T0, read at T1)
and asserts LastSuccessfulRead == T1 exactly instead of spin-waiting for the wall
clock to pass the init stamp (#767). Without the driver change the 6 tests fail
(init stamp is wall-clock, not T0).
Directory.Build.props sets TreatWarningsAsErrors for every project since #766, so
the 75 per-csproj copies (64 src + 11 tests) are removed and the props comment
says it is now the only place. tools/mem-probe keeps its explicit opt-out.
Release build: 0 warnings, 0 errors; a planted CS0168 still fails the build in a
src project and in a test project that used to carry its own copy.
Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
The native-ack path is fire-and-forget after the Part 9 Acknowledge returns, so
a Galaxy ack refused for lack of a credential (#749), or rejected by MXAccess,
left the client with Good and the condition showing Acknowledged while Galaxy
was never acked. Both options from the issue, layered:
1. Synchronous refusal for what the driver can decide without I/O. New optional
IAlarmAcknowledgePreflight (Core.Abstractions); GalaxyDriver refuses an
anonymous session when the driver has no AlarmAck fallback account.
DriverHostActor publishes a condition NodeId -> owning-driver snapshot into
NativeAlarmAckPreflightView (lock-free, volatile); the node manager's new
NativeAlarmAckPreflight seam, consulted after the authz and Primary gates,
returns BadUserAccessDenied with the reason - nothing routed, applied or
reported. A keyed session is not pre-checked (its identity lives in the
Secrets store, which must not be read under the SDK lock).
2. Revert for everything else. RouteAlarmAck carries the condition NodeId;
DriverInstanceActor replies AlarmAckFailed on a refusal, a backend failure,
a driver without an alarm source, or an ack dropped while connecting /
reconnecting. DriverHostActor re-projects the condition unacknowledged via
NativeAlarmProjector.TryRevertAcknowledge, but only while the local ack is
still pending - any driver transition projected since then wins. A disabled
or unspawned driver is not reverted (unchanged).
GatewayGalaxyAlarmAcknowledger now throws DriverOperationRefusedException on a
non-zero native hresult (wrong password, no ack permission) instead of only
warning, so that case reverts too; as a refusal it neither retries nor trips
the breaker.
Tests (each fails with its fix mutated out): projector revert/supersede,
DriverHostActor refused + failed ack -> unacked AlarmStateUpdate (successful
ack control), preflight view snapshot, node-manager preflight unit test,
over-the-wire ConditionMethodEventTests (BadUserAccessDenied, no route, no
event; positive control), Galaxy preflight + hresult refusal.
Docs: Galaxy.md, AlarmTracking.md, ScriptedAlarms.md, .claude/rules/alarms.md.
Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
IOpcUaEndpointState (review-remediation lane C3) recorded whether the SDK started, but only the
health check read it: a fused node whose endpoint failed (port 4840 taken, ...) still advertised
250/240 and stayed driver Primary, so the Primary-gated data plane ran on a node no client could
reach while the reachable peer refused writes as Secondary.
- Move IOpcUaEndpointState/OpcUaEndpointState/OpcUaEndpointStatus from Host to Commons.OpcUa so
the Runtime can consume it.
- ServiceLevelCalculator: new NodeHealthInputs.EndpointListening (default true). false -> 0,
ahead of every other input including the +10 Primary bonus.
- OpcUaPublishActor: takes the endpoint state, feeds EndpointListening (Failed only; NotStarted /
Stopped are boot/shutdown windows) on the DB-backed and DB-less paths, and the legacy role-only
seam returns 0 for a failed endpoint too.
- New OpcUaEndpointStepDownActor (one per driver node): a Primary (ClusterActiveNode.SelfIsActive,
the election's own primitive) with a Failed endpoint gracefully Leaves its mesh - the manual
failover mechanism - so the peer becomes the oldest Up driver member, hosts the singletons and is
Primary. The election itself is not bent. Guarded on a peer having recently reported its own
endpoint listening (EndpointReport on the new opcua-endpoint-state DPS topic), so a pair with
both endpoints failed does not bounce the role and restart nodes forever, and a lone node never
shuts itself down.
- Runtime now references ZB.MOM.WW.Health.Akka (ClusterActiveNode).
- Docs: Redundancy.md tier table + new "Endpoint start failure: Primary step-down" section;
cluster-topology rule; HealthEndpoints / OpcUaEndpointHealthCheck comments (F-C1 landed).
Tests: ServiceLevelCalculatorTests (+5), OpcUaPublishActorEndpointStateTests (7: DB-less,
DB-backed, legacy seam, NotStarted/Stopped do not demote, a later failure demotes on the next
tick), OpcUaEndpointStepDownActorTests (3, real two-node mesh: holds while the peer is not
listening then leaves and the peer becomes Primary; failed Secondary and healthy Primary never
leave). Each was mutation-checked: removing the calculator/legacy guard, the Leave, the
peer-listening guard or the Primary guard fails the corresponding tests.
mxaccessgw 488649c+ acknowledges by AuthenticateUser as the operator and
writing <alarm>.AckMsg, so AcknowledgeAlarmRequest now needs a real Galaxy
operator_user + operator_domain + operator_password. The OPC UA data plane is
API-key/anonymous and carries no Galaxy password, so the driver resolves the
account per ack:
1. the session's API key's Galaxy ack identity, set on AdminUI /apikeys and
held ONLY in the encrypted Secrets store
(otopcua/apikeys/<keyId>/galaxy-ack-{user,password}); a key with a user but
no password secret is refused, never silently downgraded;
2. else the driver instance's AlarmAck fallback account (DriverConfig
FallbackUser/FallbackDomain/FallbackPasswordSecretRef; secret:/${secret:}/
env:/file:/dev: only — a literal is refused);
3. else DriverOperationRefusedException: nothing is sent, never an empty
operator_user. Refusals are excluded from the resilience retry/breaker.
- Pins ZB.MOM.WW.MxGateway.Client/Contracts 0.2.1 -> 0.3.0 (vendored into
nuget-packages/ from the Gitea feed, per the local-mxgw source mapping).
- The session ApiKeyId (not a secret) rides NativeAlarmAck ->
RouteNativeAlarmAck -> RouteAlarmAck -> AlarmAcknowledgeRequest; the password
is resolved in the driver and never travels through actor messages.
- The OPC UA principal + key id are appended to the AckMsg comment
("[OPC UA: <principal>, key <id>]") and logged, so a shared fallback
account does not lose who acknowledged.
- Password never logged: GalaxyAckCredential.ToString omits it, the
acknowledger scrubs it from every gateway message it echoes and withholds
any exception that echoes it; audit rows carry only the key id.
- AdminUI: /apikeys "Ack identity" editor (set/remove; delete-key clears it),
Galaxy driver form gains the fallback section.
- Synchronous OPC UA status for a refusal is follow-up #760 (the native ack
path is fire-and-forget after the Part 9 method returns).
Tests: request mapping (password populated), password absent from logs and
exception text (mutation-checked), key-mapped / fallback / refused paths at
resolver and driver level, breaker non-trip, key-id propagation through node
manager and DriverHostActor, AdminUI service + form round-trip.
Closes#749
I-1: a node whose in-memory trail is one short of the stored trail (it missed
a command the peer applied, whose entry then replicated in) silently dropped
its own next entry — in failover, the survivor's newest ack — because the
positional "past MAX(seq)" rule treated it as already stored.
SaveAsync now reconciles the last 16 in-memory entries against the stored
rows from that position on, by content (Kind/User/Text, timestamps within a
5-minute skew window; matched rows are consumed so repeats count). The peer's
copy of the same command is recognised and not re-appended; an unmatched
entry is inserted at its own ordinal when free, otherwise appended at
MAX(seq)+1 with a Warning — kept, never silent. Bounded read, constant write.
m1: MigrateInlineComments copies only — it no longer UPDATEs the condition
row (a fresh-HLC re-stamp could beat a newer peer ack under LWW); the next
save clears comments_json. m2: it returns unparseable alarm ids and OnReady
logs a Warning per id (+ an Information tally). m3/m4: documented in the
store remarks (Remove vs lagging peer; in-memory index == durable ordinal).
Tests: store-level failover sequence (missed command -> survivor's ack and
next entry persisted once, re-save writes 0 rows, one Warning) with the
dual-write positive control (peer copy present -> 0 rows, no Warning) and a
repeated-identical-comment case; a loopback pair failover test (link down,
survivor acks, rejoin -> both entries on both peers, held); migration
no-row-rewrite, partial fill and unparseable reporting; OnReady Warning.
Both failover tests fail against the round-0 store.
Refs #748
Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
Every Part 9 transition used to re-serialize the alarm's whole comment trail
into alarm_condition_state.comments_json and replicate it to the pair peer:
O(n) per transition, O(n^2) over an alarm's life (measured: one ack at 10k
comments shipped 1,080,553 oplog bytes; now ~1.6 KB, the same as at 10).
- New replicated table alarm_condition_comment, PK (scripted_alarm_id, seq),
seq = the entry's ordinal in the trail. Ordinal (not content) keying makes
the two pair nodes' copies of one logical entry (each stamped by its own
clock, since both nodes apply every command) collapse to one row under LWW.
- LocalDbAlarmConditionStateStore.SaveAsync appends only entries past the
stored MAX(seq) (INSERT OR IGNORE, append-only) and upserts a fixed-size
condition row with comments_json = "[]", in one transaction. Remove drops
the trail with the row. Load merges the table with any inline trail.
- comments_json column kept (always "[]") so the condition table's shape and
replication digest are unchanged and old-shape rows still load exactly.
- AlarmConditionStateSchema.MigrateInlineComments: idempotent per-row move of
inline trails, run by LocalDbSetup.OnReady after RegisterReplicated so the
moved entries replicate. Comment table joins ReplicatedTables (now five).
Tests: constant transition payload at 10 vs 10k comments (+ instrument
positive control), old-shape back-compat load, migration idempotency /
partial re-run / lazy move, OnReady boot migration captured in the oplog,
and two loopback pair tests (A->B and B->A trail convergence; the shared
entry converges to one copy, with a distinct-alarm positive control).
Refs #748
Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
I-1: a PUT/GET-disabled CPU (every read denied) stayed Faulted with
LastSuccessfulRead frozen, so the comms-loss watchdog reconnected and
re-subscribed it every 90 s forever. Both access-denied arms now also record
an answer: Faulted is kept (worst state wins), the stamp advances, and the
sticky write fault still clears only on a real success. Pinned end to end in
Runtime.Tests: the real S7Driver under the real DriverInstanceActor watchdog
is not reconnected across 2x the threshold; the control (every read timing
out) is.
Minors:
- m-1: injectable pass clock (UtcNow); "LastSuccessfulRead advanced" is now
asserted strictly, including the write-refusal case.
- m-2: the editor uses S7TagShapeRules.IsByteAnchored instead of its own list.
- m-3: a release that races an init publish also cancels the probe loop the
publish started after the release took its snapshot.
- m-4: a retained poll failing during a lifecycle operation writes no health.
- m-5: a multi-var chunk that fell back memoises its members refused alone.
- m-7: access-denied needs S7.Net's exact wording and no transport fault in
the chain.
- m-8: sync teardown clears the write-side Degraded owner; a sync Dispose
racing a running shutdown is tested.
Refs #742
Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
N3 (critical): the node manager carries only NATIVE conditions' state over a
full rebuild; scripted conditions come back fresh and rely on the engine's
re-assert, which since the off-thread plan is handled in the window and lands
on the discarded instance. On a rebuild the condition replay scope is now
AddedConditions plus every enabled scripted condition in the new composition
(native kept conditions stay carried-over, not replayed). The test sink again
models the real rebuild: it clears non-native condition state only.
N4: a surgical in-place shape edit (DataType / IsArray / ArrayLength) resets the
raw node and its UNS mirror to BadWaitingForInitialData without ensuring them,
so the window's value was not replayed and the two NodeIds disagreed. The
applier records re-shaped nodes (MaterialisationRecording.Reshaped); their
deferred value replays only once fitted to the new shape (VariableShape.TryFit:
as is, or converted invariantly; never an old-typed value - WriteValues assigns
the value verbatim with no type check), otherwise it is dropped at Debug and the
node keeps waiting. Status-only entries are not replayed onto a re-shaped node.
Minors: m1/m2 documented on _replayScope (degraded-path only); m3 corrected the
false carry-over premise in the scope doc (the ScriptedAlarmHostActor ordering
remark is outside this lane - F8); m4 the replay Debug log reports alarm-state
and alarm-quality counts again.
Refs #732
Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
Review I-2. Under factory Props a supervision restart of
ScriptedAlarmHostActor builds an empty engine, and nothing re-sent
ApplyScriptedAlarms. Until the next deploy the host had no mux interest
(inputs stopped), and its ownership filter silently dropped every
operator ack and shelve.
The host now tells its parent ScriptedAlarmHostRestarted from
PostRestart. DriverHostActor keeps the last plans it applied and
re-sends ApplyScriptedAlarms when the signal comes from the current
host, in Steady, Applying and Stale. This mirrors how a driver child's
ReprovisionRequested is answered. The OnAlarmsLoaded comment, which
assumed a re-apply that never happened, now says where it comes from.
DriverHostActorScriptedAlarmRestartTests deploys one scripted alarm
through a real host, crashes the alarm host, does NOT re-apply by hand,
and asserts that interest is re-registered, a dependency push activates
the alarm, and an operator ack reaches the engine. It is red without
the signal (times out waiting for the re-registration).
Refs #731#734
Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
N1: a full alarm-state snapshot also sets the condition's Quality, so it now
supersedes an earlier quality-only write for the same id in the deferred
buffer; a quality arriving after the state is kept and replays after it.
N2 / M1: the replay is not inert on a node the apply KEPT - an SDK-applied
Acknowledge/Confirm/Shelve or an optimistic client Write in the window would
be rolled back (with spurious Part 9 events). The replay is now scoped to
nodes the apply created: variables the applier ensured during this apply
(AddressSpaceApplier.BeginRecordingEnsuredVariables) or every variable after a
full rebuild; conditions in the applied composition but not the previous one
(also on a rebuild, which carries a kept condition's state over). Option (b),
a sink that reports whether a live write landed, needs the Commons
DeferredAddressSpaceSink seam outside this lane (follow-up F7).
M2: the benchmark remark now matches its Category=Benchmark trait.
M3: TestHooks and the seam-carrying constructor are internal; PropsForTests
produces the actor through an IIndirectActorProducer.
M4: pinned replay-nothing-and-drop-the-buffer after a failed plan.
M5: PreRestart answers every queued rebuild requester (degraded).
Refs #732
Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
Review I-1. Context.Stop is asynchronous, so a removed child kept
publishing through the rest of the apply handler and its own stop.
Those bursts were forwarded after SourceRemoved and re-retained the
removed driver's last Good in the mux, where nothing would ever clear
it while a replay subscriber held the ref (a removed driver never
publishes again, and its PostStop Bad fan resolves no refs once the
index is rebuilt without it).
StopChild and HandleRestartDriver now retire the child (_retiredChildren
+ Context.Watch) before stopping it. ForwardBatchToMux drops a burst
whose Sender is retired, and Terminated (handled in Steady, Applying and
Stale) removes the entry. Akka delivers Terminated after the watched
child's last message, so the set is bounded and a live incarnation is
never filtered. ConnectivityChanged from a retired child is
deliberately not filtered, because the respawn path relies on its Bad
fan.
Tests (review m-3 folded in): a stale burst spoofed as the removed child,
held alive in PostStop by a new RecordingSubscribableDriver.ShutdownGate,
no longer re-retains (red with the guard disabled). New cases cover a
config-change respawn and an operator Restart, each with the new
driver's value replaying as the positive control. The arrange step now
waits for the real subscription set and fishes for both drivers'
values at once (two sequential fishes could discard the other burst).
Refs #731#734
Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
It had no production sender: every driver data change is already a
batch, and the comms-loss fan was converted earlier. DependencyMuxActor
drops its per-value receive, DriverInstanceActor drops the record, and
DependencyMuxActorTests' 13 sends move to the ValuePublishes.Single
one-item batch (same semantics through the batch path). XML docs and
the living docs (DriverLifecycle, VirtualTags, ScriptEditor) are
reworded; historical plan docs are left as written.
Refs #731#734
Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
DriverHostActor now tells the mux SourceRemoved(driverInstanceId)
wherever a driver object ends: StopChild (deploy removal and the stop
half of a config-change respawn) and an operator Restart. It is not
sent on a supervision restart, which re-uses the same driver. Before
this nothing sent it, so a removed driver's last Good stayed in the
mux and replayed to the next replay-opting subscriber as current.
DriverHostActorSourceRemovedTests drives a real mux with two real
driver children: after a deploy drops DI-1 only DI-2's value replays
(positive control: both replay before the removal), and a supervision
restart of DI-1 leaves its value replayable.
Residual (documented on EvictRetainedValues): a burst the old child
enqueued just before it stopped can still be forwarded after the
eviction; the next publish, a comms-loss fan or the grace sweep
bounds it.
Refs #731#734
Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
C1 (critical): B6 computes the plan off the pinned thread while B1 made writes
non-creating, so a value/status/alarm write for a node the in-flight plan adds
was skipped, and change-driven drivers never republish it (BadWaitingForInitialData
indefinitely; on a full rebuild every re-subscribe value landed on discarded
nodes). OpcUaPublishActor now applies writes immediately AND remembers the
latest per (realm, id) while a plan is in flight or queued (value+status
compose; alarm state/quality coalesce to the latest, which the node's delta
gate makes inert on an existing condition), replays it after each apply before
answering, and drops it once the pipeline is idle. Bounded (250k keys by
default; beyond it new keys are not held, logged once).
I2: the data-plane-authority parse runs in its own try inside the plan task; a
throw is logged and never degrades the address-space apply.
I3: the SDK's DeleteMonitoredItems releases an item's component-cache
reference on a Good delete; a tombstoned item's reference was already
released, so the delete took one from the re-created node's entry. Released
items are remembered and the orphan delete takes one back first. Event items
never held a reference (EventManager + SubscribeToEvents), so the three
event-item releases are gone.
I4: direct tests for the HistoryUpdate override's per-request updater check
and the HistoryReadEvents override's source guard (both reached by restoring
the bit after a reset - the B7 race window); fixed the false remark.
Minors: the MaterialiseAlarmCondition kind-swap tombstones the dropped
instance; a bytes-less rebuild arriving mid-plan queues behind it (replies in
order); a PlanReady/PlanFailed from a previous actor incarnation is ignored and
its requester answered degraded; the tie-cluster end bound of an uncapped read
fits against the backend ceiling (defence in depth - guarded by a sweep test);
the lock-time benchmark stays in the default run, headroom noted.
Refs #732
Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
- ScriptLogRateLimiter.LimitPerWindow. The ScriptLogTopicSink and
VirtualTagActor summaries now print the configured limit, not
DefaultLimitPerWindow. Tests assert "limit N/window" for a non-default
N.
- CalculationDriver.Dispose cancels, then takes _evalGate (bounded 2 s)
and disposes the gate and _evalCts. The pump and timer ticks enter the
gate through TryEnterEvalGateAsync, which stops cleanly once the
driver is disposed. Test: Dispose with a 6 s runaway run in flight
returns promptly, disposes both, and a late trigger is harmless.
- ScriptedAlarmEngine captures _shutdownToken once and disposes
_shutdownCts in Dispose when every drained task has completed. A task
stuck past the grace leaves the CTS to the GC. Lifecycle tests assert
the disposal.
- DraftValidator: CalculationNoEffectiveTrigger is skipped for a
change-triggered tag whose scriptId does not resolve. That case is
already CalculationScriptMissing or CalculationScriptNotFound, and
there are no reads to count. Both existing tests now assert the
absence (red before the change).
- docs/VirtualTags.md: System.Threading is denied as a whole namespace,
with an allow-list; the type list is now accurate (Unsafe,
RuntimeHelpers, and the ForbiddenMembers on System.Type); the resource
limits section covers the run guard and the allocation budget.
docs/ScriptEditor.md: accurate deny-list and a new "Runtime safety"
section covering timeout, refusal, quarantine after 3 timeouts,
allocation budget, lifting, and rate-limit summaries that appear only
on the source's next entry after a burst then quiet.
Refs #734
Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
The fixture wedged EVERY write on one shared token-ignoring release gate, so
the later same-tag write was also hung, and the 300 ms pessimistic bound
abandoned it while the test was still polling before ReleaseWrites(). Its
reason was "write did not complete within" (the bound), not "queued for" (the
queue budget), and it had already entered the driver, so the tag had been
released. The release path is correct. Only the first write now hangs, and
later writes complete at once.
The abandon reason formatted the bound with TotalSeconds:F0, which printed a
300 ms bound as "0s". It now uses milliseconds, and the test asserts it.
Refs #731
Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
DependencyMuxActor retained the last value of every ref ever registered,
and never removed the entries. The continuous-historization recorder
registers every historized ref, so on a historizing node the map grew
with the historized address space and never shrank. The class remark
claimed a bound that did not hold.
- Retain only refs that at least one subscriber registered with
ReplayRetained: true names (VT actors, the Calculation mux adapter, the
scripted-alarm host). A non-opting subscriber is still delivered to but
retains nothing.
- When a ref's last replay-opting subscriber leaves, its value is kept
for a grace (default 2 min, Props(retentionGrace)) so a restart
(old adapter gone, new one not yet registered) still replays. A
periodic sweep then drops it along with the entry. Entries with no
subscriber and nothing retained are removed.
- New SourceRemoved(DriverInstanceId) evicts the values a removed driver
produced, so its last Good is never replayed. The mux records the
publishing driver per retained ref. No production sender exists yet:
DriverHostActor is outside this lane (follow-up).
- Remark rewritten. Tests: retention bounded to replay-opting refs,
release after grace (with an in-grace replay as the positive control),
SourceRemoved eviction, and the restart-replay scenario still green.
Refs #734
Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
Review 02 P5. OpcUaPublishActor ran the whole apply — DeploymentArtifact
.Open, ParseComposition, AddressSpacePlanner.Compute and the sink
materialisation — on the opcua-synchronized-dispatcher, so a large
artifact serialised every live value batch (and, because the node
manager lock is the SDK's, every client Publish) behind the parse.
StartRebuild now runs Open + ParseDataPlaneAuthority + ParseComposition
+ Compute in Task.Run and pipes a PlanReady (or PlanFailed) back to Self;
only ApplyPlan — the sink passes — runs on the actor thread. Rebuilds are
applied strictly in arrival order (a plan is a diff against _lastApplied,
which only the apply advances): one in flight, later ones queue in
_pendingRebuilds while value/status batches keep flowing. The apply is
closed (CompleteAddressSpaceApply) and the RebuildCompleted reply sent on
every path, PlanFailed included. The bytes-less ConfigDb fallback
(LoadArtifact / LoadLatestArtifact) is deleted: it had no production
producer and would have blocked the pinned thread on central SQL; the
dbFactory Props/ctor parameter is kept for wiring compatibility (F4).
Tests: OpcUaPublishActorOffThreadPlanTests — a value batch sent right
after a 50k-tag rebuild lands before that rebuild's first materialise
call (positive control: all 50k tags still materialise); back-to-back
rebuilds apply in arrival order (the second diffs against the first);
unreadable bytes answer degraded with nothing torn down. The rebuild /
apply-failure suites now hand the actor the artifact bytes every
production producer carries (SealArtifact records them; Rebuild(dep)).
Refs #732
Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
- RunWriteAsync wraps the invoker call in WaitAsync(writeTimeout) (Props,
default 10 s = 2x the 5 s write cap): a token-ignoring hung write now
answers BadTimeout, releases its slot and tag, and its orphan outcome
is observed and discarded - later writes to that tag reach the device.
- WriteCompleted carries the launching incarnation (from the process-wide
generation source); a late completion from a previous incarnation after
a supervision restart is ignored instead of freeing the new
incarnation's slot/tag.
- Test: a hung token-ignoring tag-a write is abandoned and a later tag-a
write reaches the device while the abandoned one is still inside it.
Refs #731
Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
- Writes: a second write to a tag already inside the driver waits in the
queue behind it (_tagsInFlight) while other tags still run under the
cap of 4; queued writes are stamped and failed BadTimeout at dequeue
once older than the 8 s host Ask budget (Props queuedWriteBudget)
instead of being written late.
- Initialise attempts are bounded pessimistically (initializeTimeout,
default 30 s = 2x Tier C) on top of the invoker's token-only timeout:
an overrun is logged as abandoned, reported InitializeFailed so the
retry loop continues, and its late result is dropped by generation.
Generations come from a process-wide Interlocked counter so a prior
incarnation's late result cannot collide (F6).
- PostStop shares one shutdownTimeout budget across shutdown + dispose.
- TelemetryDialSupervisor cancels the establishment deadline on
StreamStopped; FleetDiagnosticsClient builds the ActorSelection inside
the try (a malformed node degrades to "unavailable"); the diagnostics
page labels the column "Last health publish".
- Tests: same-tag serialisation with a distinct-tag positive control;
stale queued write BadTimeout; token-ignoring hung init -> later
attempt connects; stopwatch assertion demoted to a ceiling.
Refs #731
Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
- R02:H1 ScriptedAlarmHostActor gains a factory Props
(Func<(upstream, engine)>) so every incarnation builds and owns its own
engine; a supervision restart no longer re-creates the actor around a
disposed engine. The legacy instance-taking Props stays source-compatible
and is now restart-safe in the weaker sense (PreRestart marks the stop as
a restart, PostStop then detaches instead of disposing). DriverHostActor
still spawns through the legacy overload (not owned by this lane — F3).
- R02:H3 OnAlarmCommand runs every engine op under a 10 s token linked to
the actor's lifetime (DefaultCommandTimeout / Props commandTimeout) and
logs a Warning on timeout instead of suspending the mailbox on a stalled
store.
- R02:L3 DependencyMuxTagUpstreamSource.RetainOnly(loaded dep-ref union),
called after every successful load.
- R02:P6 VirtualTagActor's per-failure ScriptLogEntry is rate-limited per
tag (ScriptLogRateLimiter, 20/min + one summary); the Bad publish and the
metric are untouched.
- R07:H6 DependencyMuxActor retains the last (value, status, ts) per
ever-registered ref (through unregisters) and replays them in one batch to
a RegisterInterest(ReplayRetained: true) — VirtualTagActor, the Calculation
mux adapter and the alarm host opt in; the historization recorder does not
(a replay there would record a stale sample per deploy). CalculationDriver
records inputs regardless of _running so a replay landing before
InitializeAsync still satisfies the change-gate.
- R07:P4 CalculationEvaluator negative-caches compile failures by source
hash (one Roslyn attempt + one Warning per source until redeploy); the
driver's script-log entry fires on the Good→Bad transition or a change of
reason, matching the once-per-transition Bad publish.
- R07:H8 array results dedup structurally.
- R07:L2 ShutdownAsync drops the compiled scripts (collectible ALCs).
- R07:U5 DraftValidator rule CalculationNoEffectiveTrigger: a tag with no
timer whose change trigger is off or whose script has no literal
ctx.GetTag read can never evaluate.
Refs #734
Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
- NodeManagerHistoryReadRawCapFuzzTests: 10k seeds (env OTOPCUA_PAGING_FUZZ_ITERATIONS)
through a real SDK node manager against wire-faithful fakes of SqlServer / QuestDB /
Gateway that enforce the backend half of the cap; random series with ties, random
windows (empty, on/past a tie, between samples, forward/reverse/end-only, bounds),
N in {0,1,2,small,huge}, cap in {off,3..1000}, tie budget {3,8,65536}. Pages must
concatenate to the cap-off one-shot answer (checked against the series), terminate
within values + 2 (+ one empty step per oversized boundary cluster), never exceed
the cap, never ask the backend for 0 / over-cap; refusals only past the budget.
- HistoryPaging.AdvertiseMaxReturnDataValues writes HistoryServerCapabilities/
MaxReturnDataValues (i=11273), read back through the SDK master node manager.
- MinMaxRawValuesPerRead raised to 3: at 2 a resumed page has no room (value + its
boundary tie + lookahead) and every value cost an empty page + CP (seed 15).
- RawPageLimit rows pin the default-cap arithmetic (9 999 first page, 9 998 resumed).
Fuzz findings recorded as follow-ups: a bounded read whose data fits one page
returns cap + 1 (end bound fitted against N, not the cap; node manager, F2);
the Gateway walk-back's cost on a sparse series under a tiny tie budget is
pre-existing (seed 320: 82 reads uncapped, 93 capped).
Refs #736
Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
DriverInstanceDiagnostics.Name carries the string driver-instance id
(what /hosts, telemetry and the artifact key on) and DriverInstanceId
the DB row Guid; the wire record has no slot for the authored display
name. The write-cap test asserts entry order per slot pair instead of
completion order of two concurrent writes.
Refs #731
Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
DriverHostActor wraps the health publisher it hands its children in a
ChildHealthLedger (last snapshot per driver id, forwarded to the real
publisher), so GetDiagnostics now answers with each child's row id,
authored name, last-published state, device counts (host statuses when
the driver probes hosts, else one device) and last-change time — instead
of Guid.Empty / "Spawned" / 0 / now. Entries are forgotten on StopChild
and operator Restart.
IFleetDiagnosticsClient gains its first consumer: the new
/fleet/diagnostics page (node selector + ?node= query, revision,
running-from-cache notice, per-driver table). FleetDiagnosticsClient
logs the degrade-to-empty path at Warning.
Review ref: 02/U1.
Refs #731
Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
- DriverHostActor.StopChild removes the driver's _lastReprovisionUtc stamp
(and its deferred-reprovision timer): the map grew per removed id and a
re-added driver inherited a backoff it never earned (02/L2).
- Stale now registers explicit drops for AttributeValueBatchPublished,
AttributeAlarmPublished, ArtifactForApply and ApplyWedged instead of
letting them hit Unhandled (02/H5).
- DbHealthProbeActor runs its SELECT on the thread pool and pipes the
outcome back (single-flight), with a 2 s command timeout on relational
providers, so a hung SQL Server no longer pins the actor thread and the
cached GetStatus answer stays reachable (02/P7).
- TelemetryLocalHub eviction tombstones are timestamped and swept past a
10 min TTL on the next Evict; a TimeProvider ctor is the test seam (02/L4).
- Tests: ValuePublishes.Single helper replaces the per-value
AttributeValuePublished in CalculationDependencyFlowTests (02/U4, test
half; the production receive is owned by Lane E).
- R05/8 audit: PollGroupEngine.Unsubscribe has no production caller; every
driver already uses UnsubscribeAsync (no change).
Refs #731
Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
WriteAttribute was a ReceiveAsync, suspending the child's mailbox for the
whole device round-trip (up to 5 s): writes serialised, the second of two
operator writes to a slow device missed the host's 8 s Ask and was
reverted although the device took it, and every data change queued behind
the write. Writes are now a synchronous Receive that launches the device
call on the thread pool (sender captured, reply from the task, slot freed
by a self-sent WriteCompleted) under a per-driver cap (default 4; excess
writes queue in order). Queued writes fail fast on every exit from
Connected. Subscribe/unsubscribe stay ReceiveAsync.
Review ref: 02/P2.
Refs #731
Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
Review R08:1.1: an uncapped HistoryRead-Raw (NumValuesPerNode = 0) read a
tag's whole window into memory on every backend (SQL TOP(int.MaxValue),
QuestDB LIMIT, Gateway max_values = int.MaxValue + a second snapshot copy).
New ServerHistorian:MaxRawValuesPerRead (default 10 000; <= 0 unbounded with
a Validate() warning; 1 raised to 2). Each backend data source takes it,
reports it as IHistorianDataSource.MaxRawValuesPerRead (the Gateway: min
with a strict gateway's learned ceiling), so the node manager's existing
RawPageLimit paging serves N = 0 and N > cap as pages of at most the cap
with a continuation point. Independently, a backend raw read asking for 0
or more than the cap reads cap + 1 rows and is REFUSED
(HistoryReadLimitExceededException => BadTooManyOperations) when the window
holds more - never answered truncated, since the server's paging reads a
short answer as the last page. Bound through the Host registry, the
SqlServer/QuestDb providers and GatewayHistorian.CreateDataSource.
Tests: options (default/binding/warnings), provider binding, Gateway unit +
through a real SDK node manager (uncapped -> 999/998/998/605 + CPs), and a
conformance case for the SQL backends.
Refs #736
Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
The Reconnecting retry timer started a new ShutdownAsync + InitializeAsync
pair on every tick whether or not the previous attempt had finished; the
generation token only discarded the older RESULT, so with an Initialize
bound longer than the cadence the teardown ran underneath an attempt still
inside the driver (the dispose-before-overwrite race). An _initInFlight
gate, cleared by the piped InitializeSucceeded/Failed, skips a tick while
an attempt is running; the pre-reconnect teardown is bounded by the
shutdown timeout; PostStop cancels an in-flight attempt before shutting
the driver down.
Review ref: 02/H2.
Refs #731
Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
DriverInstanceActor.PostStop now bounds IDriver.ShutdownAsync by a
shutdownTimeout (default 5 s, logged on overrun) and then disposes the
driver (IAsyncDisposable preferred, else IDisposable) — but only on a
real stop. PreRestart flags the restart so the PostStop it triggers
skips the dispose, because a supervision restart re-uses the same
driver object (Props closure) and the host re-initialises it.
DiscoveryDriverBrowser disposes its throwaway capture/probe driver
after the bounded shutdown, including after a failed or timed-out one.
Review refs: 02/L1, 02/P4, 06/G-1 (host context), 07/L2.
Refs #731
Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
Item 1: IHistorianDataSource.MaxRawValuesPerRead (Gateway: learned ceiling - 16 = L; 0 elsewhere, guard-tested).
RawPageLimit plans every raw page on top of #692's lookahead: with L > 0 the read (page P + boundary ties k
+ lookahead) is <= L, P = min(N, L - k - 1) (uncapped: L - k - 1); no room => straight to the tie-cluster
over-fetch (never an uncapped or underflowing request). One re-plan + re-read when the read taught a new
limit or was refused for crowding pre-window rows (HistoryReadLimitExceededException.PreWindowRows): the
page shrinks by them. Over-fetch bound min(MaxTieClusterOverfetch, L - 1); slice size never 0.
Review fixes: F1 a window starting at/before the epoch counts AVEVA's synthetic row as pre-window (a
ceiling-truncated read was reported complete); F2 the page path disables ReverseRawReader's own re-run
(cold end-only ceiling 20/N=3000 cost 3,023 reads, now <= 40); F3 crowded look-back re-read around, not
refused; F4 the #687 crowding re-read room at least doubles (up to 8 re-reads), so a cluster of hundreds
just before a resume point no longer fails on a default gateway; a crowding refusal that leaves no room
is BadTooManyOperations, never a tick-by-tick cursor.
Item 2: ServerHistorian:EventReadHorizon (default 00:00:00 = exact, unchanged). When set, every fresh
HistoryReadEvents window's earlier bound is raised to now - horizon (both directions, all backends).
Lossy by design; documented.
Item 3 (native TimeAverage/Total re-check) stays blocked on historiangw#27.
1. The at-or-after bound read on a forward-only backend (Gateway) ends at
now + ReverseRawReader.LookAhead instead of DateTime.MaxValue (the gateway
only checks start <= end, so a year-9999 end reached AVEVA); a bound beyond
the horizon is not found without a read. SqlServer/QuestDB keep MaxValue:
they bind it safely and may hold samples further ahead. The Gateway fake now
refuses far-future ends, so the missing-end-bound test is no longer vacuous.
3. No data and no bound found (whole read empty) => GoodNoData, no values —
the pre-bounds answer (6.5.3.2: success only if one or both bounds exist).
A one-value page whose start bound is missing first checks there is data or
a found end bound.
4. ServerHistorian:HonorReturnBounds (default true); false ignores ReturnBounds
as before #691, including the modified-read Bad_InvalidArgument.
Joseph Doherty