Item 5: DraftValidator now emits SqlQueryNotDeclared (always blocking) for a Query tag whose
query is absent from the merged driver config's queries map, via the driver's own tag parse.
Item 3: StartDeployment appends a non-blocking warning to the accepted message when the Tag
entity's AccessLevel and the Sql blob's writable flag disagree.
Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
Reproduced first, out of process (ScriptRecursionContainmentTests re-launches
the xunit v3 test assembly to run one recursive script through
ScriptEvaluator + ScriptRunGuard, so the runner never executes one): before
this change every shape — local function, mutual local functions,
self-referencing Func<>, anonymous self-application, recursion through LINQ,
an expression-bodied void lambda, an anonymous method — ended the child with
exit 134 (SIGABRT, "Stack overflow."). A StackOverflowException cannot be
caught, so in production one script takes the whole node down.
Fix: ScriptEvaluator.Compile now rewrites the validated wrapper
(ScriptStackGuardInjector, after ForbiddenTypeAnalyzer) so the script body and
every lambda, anonymous method and local function starts with
ScriptStackGuard.Check(), which throws ScriptStackExhaustedException when
RuntimeHelpers.TryEnsureSufficientExecutionStack reports the stack nearly
spent. The script then fails like any throwing script (Bad, script log) and
the process lives.
Why a runtime check, not an analyzer rule: recursion is not decidable from
syntax (anonymous self-application never names itself), and bounded recursion
must keep working. Expression-bodied lambdas/local functions become blocks
(`return e;` or `e;`, chosen from the bound symbol, so the same overload
binds); expression-tree and async lambdas are left alone. Limits (documented
on ScriptStackGuard and in docs/VirtualTags.md): depth depends on the thread's
stack size; a recursion that never re-enters script code is not seen; a script
may catch the exception, which only unwinds it.
Tests: ScriptRecursionContainmentTests (7 shapes, child process each; all 7
fail with exit 134 when the rewrite is disabled), ScriptStackGuardInjectionTests
(bounded recursion and every rewritten lambda/local-function shape keep their
value; the expression-tree case fails if those lambdas are rewritten).
Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
VirtualTagActor called the blocking IVirtualTagEvaluator.Evaluate, and
RoslynVirtualTagEvaluator blocked on the run guard with
.GetAwaiter().GetResult(), so every script run parked a dispatcher thread for
up to its 2 s budget (a first-use Roslyn compile did too). This is the #750 P0
refactor for virtual tags.
- IVirtualTagEvaluator.EvaluateAsync (default: a completed task over the
snapshot overload, so the null evaluator and fakes are unchanged).
- RoslynVirtualTagEvaluator implements it without blocking: passthrough and
compile failures complete synchronously, a first-use compile goes to the
pool, the run is awaited. The blocking overload now delegates to it.
- VirtualTagActor: a completed task is applied in the same turn, as before;
an incomplete one switches to an "evaluating" behaviour that stashes every
dependency/reassert message until a private completion message arrives,
then applies the result and unstashes. Each trigger is still evaluated
against its own inputs, results are applied/published in trigger order and
the dedup/Bad-transition state sees the same sequence. The dependency map is
not touched while a run reads it.
Calculation and scripted-alarm serialisation is NOT relaxed: neither blocks a
thread, and running independent scripts in parallel is not semantics-neutral
(pool consumption bound, cross-tag publish order, timer/pump exclusion for
Calc; state-machine/persist boundary, transition timestamps and the
store-failure path for alarms). Recorded in docs/VirtualTags.md and the #750
design's P0 entry.
Tests: VirtualTagActorAsyncEvaluationTests — on a one-thread dispatcher a slow
script no longer starves a sibling actor and the slow actor answers Identify
(fails when the actor calls the blocking Evaluate); triggers during a run are
evaluated afterwards in order against their own inputs; a reassert during a
run is applied after it; a faulted evaluation takes the evaluator-threw path
(the ordering/reassert tests fail with the stash removed).
RoslynVirtualTagEvaluatorAsyncTests — a slow script and a cold compile return
an incomplete task (fail without the override); cheap outcomes and timeouts
match the blocking overload.
Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
DriverHostActor built ScriptedAlarmEngine without a scriptTimeout, so every
production alarm predicate fell back to TimedScriptEvaluator.DefaultTimeout
(250 ms) while virtual tags and Calculation tags ran on 2 s. History shows no
decision behind it: 250 ms is the generic Phase-7 A.4 default, the VT
evaluator moved to 2 s in F8b (219d10a2) and the alarm engine was simply never
given a value.
- New node-wide Scripting:RunTimeout (default 2 s, clamped to (0, 60 s],
non-positive -> default, each clamp a startup Warning) on ScriptingOptions.
- The VT evaluator and the alarm engine default to it; DriverHostActor now
builds the engine through CreateScriptedAlarmEngine, which passes it
explicitly. Calculation keeps its per-driver runTimeoutMs.
- Docs: ScriptEditor (config table), ScriptedAlarms, VirtualTags.
Tests: ScriptedAlarmEngineBudgetTests (a 600 ms predicate now completes and
activates; default and configured budgets), ScriptedAlarmEngineBudgetWiringTests
(the host-built engine), ScriptingOptionsBindingTests (bind + clamps). The
engine and wiring tests fail with the old 250 ms default restored.
Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
Three write bounds were configured independently: the inbound-write
gateway's 10 s Ask, DriverHostActor's fixed 8 s Ask of the child, and the
child's own write timeout (5 s + up to 2 s grace) with a fixed 8 s queue
budget. An outer Ask that fires first reports the write failed and the node
reverts while the child is still driving the device, so any write timeout
above ~6 s (or a write dequeued late in its 8 s budget) let the device take
a value the client was told failed.
DriverWriteBounds (Runtime) derives them all from DriverWrites:Timeout:
token cancel = Timeout
abandon = Timeout + min(Timeout/2, 2 s) (RunBound)
queue budget = RunBound
child replies <= queue budget + RunBound (ReplyBound)
host Ask = ReplyBound + 1 s
gateway Ask = host Ask + 1 s
Default Timeout is 10 s (the Tier C Write timeout, so no tier default is
capped below its own policy); host Ask 25 s, gateway Ask 26 s. Tier A/B
writes are still cancelled first by their 2 s / 4 s tier timeouts.
- DriverHostActor takes writeBounds + writeClock, spawns children with them,
and runs its Ask deadline as a token on the write clock (Infinite Akka
timeout), so host and child bounds share one clock.
- WriteAttribute gains IssuedAt (write-clock timestamp, stamped by the
host): the queue budget counts mailbox time too, and a write past it is
failed BadTimeout and never started - including on arrival. Stale queued
writes are expired across the whole queue, not only up to a free slot.
- ActorNodeWriteGateway's default Ask = Default.GatewayAsk; the hosted
service passes the configured one.
- Startup validation (Host DriverWriteBoundsBinding, same warn-and-clamp
style as ScriptingOptionsBinding): unparsable / non-positive -> default,
above 5 min -> clamped, and each tier whose default Write timeout exceeds
the write timeout is named in a Warning (its policy would never take
effect). The derived bounds are logged at Information. Per-instance
ResilienceConfig Write overrides arrive with a deployment, so
DriverCapabilityInvokerFactory warns at spawn (writeTimeoutCeiling).
- appsettings.json documents DriverWrites:Timeout.
Tests (manual clock):
- DriverHostActorWriteAskTests: a held write under a 9 s timeout survives
an 8.5 s step and succeeds; a token-ignoring write is answered by the
child's BadTimeout at 9 s + grace, not by the Ask; the Infinite-timeout
Ask ends exactly at its clock deadline.
- DriverInstanceActorConcurrentWriteTests: a write issued longer ago than
the queue budget is failed and never reaches the driver; positive control
at exactly the budget is written.
- DriverWriteBoundsTests: every outer bound covers the inner ones for 0.5 s
to 300 s timeouts.
- DriverWriteBoundsBindingTests: default fits every tier; 3 s timeout is
caught at startup with Tier B and C named (not A); bad values fall back
with a warning; over-max clamps.
- DriverCapabilityInvokerFactoryTests: spawn warns when a Write override
exceeds the ceiling; no warning within it or without a ceiling.
Mutation checks: a fixed 8 s host Ask on the same clock fails both 9 s host
tests; stamping arrival time instead of IssuedAt fails the issued-at test;
dropping the tier check / factory check fails their tests.
Runtime.Tests 961 passed / 3 skipped at AKKA_TEST_TIMEFACTOR 3 and 1;
Host.Tests 232/232 at 3 and 1; Core.Tests 361 passed / 1 skipped.
Solution Release build: 0 warnings.
ContinuousHistorizationRecorderTypedValueTests (historian) deliberately
left for the historian lane.
Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
HistoryReadEvents projected only EventId / SourceName / Time / ReceiveTime /
Message / Severity: a select clause asking for EventType, ConditionName or
SourceNode (what most generic alarm viewers ask for) got nulls, and the
event filter's where-clause (an OfType) was ignored.
- HistoricalEvent gains optional EventTypeName, ConditionName and AlarmId.
SqlServer, QuestDB and Gateway event reads fill them from what their
alarm writers already store (AlarmTypeName / AlarmName / AlarmId; the
gateway event Type and its AlarmName / AlarmId properties).
- The node manager maps the type name to its Part 9 event type (unknown or
non-event => AlarmConditionType), projects ConditionName, and resolves
SourceNode to the materialised condition the alarm id names (the live
events' SourceNode), else null.
- The where-clause is evaluated per event with the SDK content filter
through an IFilterTarget (OfType via the type tree; other operators
against the projected fields), after the page's cursor is taken, so
paging is unchanged.
Tested in the node manager, over a real client session (the Client.CLI has
no `historyread --events` mode yet; filed as a follow-up), and by alarm
writer -> event read round trips on SqlServer and QuestDB.
Rebased from the unpushed remed/lane-H2 branch (originally authored 2026-10-22
in session_01Ub9hGmYLQvn9XETzw7CMsT) onto current master.
Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
AbLegacyDriver stamped LastSuccessfulRead (init, read, write), the read's sample
timestamps and host-state change times from raw DateTime.UtcNow; only the backoff
path read the injectable Clock. All of them now read Clock(), so
NativePcccBridgeStatusHealthTests drives a manual clock (init at T0, read at T1)
and asserts LastSuccessfulRead == T1 exactly instead of spin-waiting for the wall
clock to pass the init stamp (#767). Without the driver change the 6 tests fail
(init stamp is wall-clock, not T0).
Directory.Build.props sets TreatWarningsAsErrors for every project since #766, so
the 75 per-csproj copies (64 src + 11 tests) are removed and the props comment
says it is now the only place. tools/mem-probe keeps its explicit opt-out.
Release build: 0 warnings, 0 errors; a planted CS0168 still fails the build in a
src project and in a test project that used to carry its own copy.
Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
The native-ack path is fire-and-forget after the Part 9 Acknowledge returns, so
a Galaxy ack refused for lack of a credential (#749), or rejected by MXAccess,
left the client with Good and the condition showing Acknowledged while Galaxy
was never acked. Both options from the issue, layered:
1. Synchronous refusal for what the driver can decide without I/O. New optional
IAlarmAcknowledgePreflight (Core.Abstractions); GalaxyDriver refuses an
anonymous session when the driver has no AlarmAck fallback account.
DriverHostActor publishes a condition NodeId -> owning-driver snapshot into
NativeAlarmAckPreflightView (lock-free, volatile); the node manager's new
NativeAlarmAckPreflight seam, consulted after the authz and Primary gates,
returns BadUserAccessDenied with the reason - nothing routed, applied or
reported. A keyed session is not pre-checked (its identity lives in the
Secrets store, which must not be read under the SDK lock).
2. Revert for everything else. RouteAlarmAck carries the condition NodeId;
DriverInstanceActor replies AlarmAckFailed on a refusal, a backend failure,
a driver without an alarm source, or an ack dropped while connecting /
reconnecting. DriverHostActor re-projects the condition unacknowledged via
NativeAlarmProjector.TryRevertAcknowledge, but only while the local ack is
still pending - any driver transition projected since then wins. A disabled
or unspawned driver is not reverted (unchanged).
GatewayGalaxyAlarmAcknowledger now throws DriverOperationRefusedException on a
non-zero native hresult (wrong password, no ack permission) instead of only
warning, so that case reverts too; as a refusal it neither retries nor trips
the breaker.
Tests (each fails with its fix mutated out): projector revert/supersede,
DriverHostActor refused + failed ack -> unacked AlarmStateUpdate (successful
ack control), preflight view snapshot, node-manager preflight unit test,
over-the-wire ConditionMethodEventTests (BadUserAccessDenied, no route, no
event; positive control), Galaxy preflight + hresult refusal.
Docs: Galaxy.md, AlarmTracking.md, ScriptedAlarms.md, .claude/rules/alarms.md.
Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
IOpcUaEndpointState (review-remediation lane C3) recorded whether the SDK started, but only the
health check read it: a fused node whose endpoint failed (port 4840 taken, ...) still advertised
250/240 and stayed driver Primary, so the Primary-gated data plane ran on a node no client could
reach while the reachable peer refused writes as Secondary.
- Move IOpcUaEndpointState/OpcUaEndpointState/OpcUaEndpointStatus from Host to Commons.OpcUa so
the Runtime can consume it.
- ServiceLevelCalculator: new NodeHealthInputs.EndpointListening (default true). false -> 0,
ahead of every other input including the +10 Primary bonus.
- OpcUaPublishActor: takes the endpoint state, feeds EndpointListening (Failed only; NotStarted /
Stopped are boot/shutdown windows) on the DB-backed and DB-less paths, and the legacy role-only
seam returns 0 for a failed endpoint too.
- New OpcUaEndpointStepDownActor (one per driver node): a Primary (ClusterActiveNode.SelfIsActive,
the election's own primitive) with a Failed endpoint gracefully Leaves its mesh - the manual
failover mechanism - so the peer becomes the oldest Up driver member, hosts the singletons and is
Primary. The election itself is not bent. Guarded on a peer having recently reported its own
endpoint listening (EndpointReport on the new opcua-endpoint-state DPS topic), so a pair with
both endpoints failed does not bounce the role and restart nodes forever, and a lone node never
shuts itself down.
- Runtime now references ZB.MOM.WW.Health.Akka (ClusterActiveNode).
- Docs: Redundancy.md tier table + new "Endpoint start failure: Primary step-down" section;
cluster-topology rule; HealthEndpoints / OpcUaEndpointHealthCheck comments (F-C1 landed).
Tests: ServiceLevelCalculatorTests (+5), OpcUaPublishActorEndpointStateTests (7: DB-less,
DB-backed, legacy seam, NotStarted/Stopped do not demote, a later failure demotes on the next
tick), OpcUaEndpointStepDownActorTests (3, real two-node mesh: holds while the peer is not
listening then leaves and the peer becomes Primary; failed Secondary and healthy Primary never
leave). Each was mutation-checked: removing the calculator/legacy guard, the Leave, the
peer-listening guard or the Primary guard fails the corresponding tests.
mxaccessgw 488649c+ acknowledges by AuthenticateUser as the operator and
writing <alarm>.AckMsg, so AcknowledgeAlarmRequest now needs a real Galaxy
operator_user + operator_domain + operator_password. The OPC UA data plane is
API-key/anonymous and carries no Galaxy password, so the driver resolves the
account per ack:
1. the session's API key's Galaxy ack identity, set on AdminUI /apikeys and
held ONLY in the encrypted Secrets store
(otopcua/apikeys/<keyId>/galaxy-ack-{user,password}); a key with a user but
no password secret is refused, never silently downgraded;
2. else the driver instance's AlarmAck fallback account (DriverConfig
FallbackUser/FallbackDomain/FallbackPasswordSecretRef; secret:/${secret:}/
env:/file:/dev: only — a literal is refused);
3. else DriverOperationRefusedException: nothing is sent, never an empty
operator_user. Refusals are excluded from the resilience retry/breaker.
- Pins ZB.MOM.WW.MxGateway.Client/Contracts 0.2.1 -> 0.3.0 (vendored into
nuget-packages/ from the Gitea feed, per the local-mxgw source mapping).
- The session ApiKeyId (not a secret) rides NativeAlarmAck ->
RouteNativeAlarmAck -> RouteAlarmAck -> AlarmAcknowledgeRequest; the password
is resolved in the driver and never travels through actor messages.
- The OPC UA principal + key id are appended to the AckMsg comment
("[OPC UA: <principal>, key <id>]") and logged, so a shared fallback
account does not lose who acknowledged.
- Password never logged: GalaxyAckCredential.ToString omits it, the
acknowledger scrubs it from every gateway message it echoes and withholds
any exception that echoes it; audit rows carry only the key id.
- AdminUI: /apikeys "Ack identity" editor (set/remove; delete-key clears it),
Galaxy driver form gains the fallback section.
- Synchronous OPC UA status for a refusal is follow-up #760 (the native ack
path is fire-and-forget after the Part 9 method returns).
Tests: request mapping (password populated), password absent from logs and
exception text (mutation-checked), key-mapped / fallback / refused paths at
resolver and driver level, breaker non-trip, key-id propagation through node
manager and DriverHostActor, AdminUI service + form round-trip.
Closes#749
Review r0 of Integration-W3:
- I1: the effective FC16 cap is min(configured maxRegistersPerWrite, 123). The
driver trusts a larger configured cap and sends an over-ceiling FC16 (past 127
registers its byte-count byte wraps), which every conformant device refuses, so
such an array is never advertised writable.
- I2: a BitInRegister[] write reads its span first in one unchunked FC03, so it is
also bounded by min(configured maxRegistersPerRead, 125) (absent/0 = 125).
- m2: every span (registers and coils) must end at or before address 65535.
- m1: the String-node wording says it converts only when the text parses, which
the server cannot promise (DateTime never converts).
- m4: the parity device now enforces FC01/02 2000, FC03/04 125 (or a smaller
device read limit), FC15 1968, FC16 123, FC23 121/125, the FC15/FC16 byte-count
field (exception 03) and the address-space end (exception 02).
Tests: parity rows for each ceiling/cap/span boundary against the real write
path, and Core.Abstractions row tests. The configured-cap test that asserted a
cap of 200 lifts the ceiling now asserts the opposite.
Refs #741
Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
The Modbus lane (#741 U3) gave the driver String[] and BitInRegister[] holding-register array encoders, and
moved address resolution to ModbusTagAddressResolution (keys first, a bit region coerced to Bool, strict key
reads), but DriverArrayWriteSupport still listed both array shapes read-only (follow-up F1), so a client
write was BadNotWritable before it reached the driver.
Modbus row now mirrors the driver end to end:
- Keys shape only (region + address both present). A string-only addressString tag is resolved by the
driver with its own family through a parser Core.Abstractions cannot reference, so it stays read-only
(pinned: the conservative direction).
- Strict reads as the resolver: address an integer 0-65535; a present region / dataType / byteOrder /
stringByteOrder string must name a member.
- Coils: any dataType (coerced to Bool), <= 1968 coils (FC15).
- HoldingRegisters (FC16, never Bool): numeric, String ((stringLength+1)/2 registers, stringLength >= 1),
BitInRegister (1 register, bitIndex 0-15, span ending <= 65535), all within maxRegistersPerWrite.
FC23 is opt-in and never changes the answer.
- Elements convert (ScadaBridge int[] -> Int16 stays writable); only a String / DateTime node over a
numeric or bit element is left out, since such an element never converts.
Tests: the parity test's PendingF1 exemption and its guard are gone; the parity matrix is now region x
element type x node type against the real WriteAsync with a spec-conformant device (FC15 > 1968 refused),
plus shape edges at every cap boundary (numeric, String, BitInRegister, coil), strict key reads, FC23
on/off, string-only, the ScadaBridge int[] shape and U3 wire bytes. Core.Abstractions row tests and a
composer test (String[] and BitInRegister[] writable; over-cap, no-stringLength and string-only read-only).
Docs: table XML doc, Uns.md, modbus-addressing.md, drivers/Modbus.md, data-plane rules.
Refs #741
Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
I-1: a node whose in-memory trail is one short of the stored trail (it missed
a command the peer applied, whose entry then replicated in) silently dropped
its own next entry — in failover, the survivor's newest ack — because the
positional "past MAX(seq)" rule treated it as already stored.
SaveAsync now reconciles the last 16 in-memory entries against the stored
rows from that position on, by content (Kind/User/Text, timestamps within a
5-minute skew window; matched rows are consumed so repeats count). The peer's
copy of the same command is recognised and not re-appended; an unmatched
entry is inserted at its own ordinal when free, otherwise appended at
MAX(seq)+1 with a Warning — kept, never silent. Bounded read, constant write.
m1: MigrateInlineComments copies only — it no longer UPDATEs the condition
row (a fresh-HLC re-stamp could beat a newer peer ack under LWW); the next
save clears comments_json. m2: it returns unparseable alarm ids and OnReady
logs a Warning per id (+ an Information tally). m3/m4: documented in the
store remarks (Remove vs lagging peer; in-memory index == durable ordinal).
Tests: store-level failover sequence (missed command -> survivor's ack and
next entry persisted once, re-save writes 0 rows, one Warning) with the
dual-write positive control (peer copy present -> 0 rows, no Warning) and a
repeated-identical-comment case; a loopback pair failover test (link down,
survivor acks, rejoin -> both entries on both peers, held); migration
no-row-rewrite, partial fill and unparseable reporting; OnReady Warning.
Both failover tests fail against the round-0 store.
Refs #748
Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
Every Part 9 transition used to re-serialize the alarm's whole comment trail
into alarm_condition_state.comments_json and replicate it to the pair peer:
O(n) per transition, O(n^2) over an alarm's life (measured: one ack at 10k
comments shipped 1,080,553 oplog bytes; now ~1.6 KB, the same as at 10).
- New replicated table alarm_condition_comment, PK (scripted_alarm_id, seq),
seq = the entry's ordinal in the trail. Ordinal (not content) keying makes
the two pair nodes' copies of one logical entry (each stamped by its own
clock, since both nodes apply every command) collapse to one row under LWW.
- LocalDbAlarmConditionStateStore.SaveAsync appends only entries past the
stored MAX(seq) (INSERT OR IGNORE, append-only) and upserts a fixed-size
condition row with comments_json = "[]", in one transaction. Remove drops
the trail with the row. Load merges the table with any inline trail.
- comments_json column kept (always "[]") so the condition table's shape and
replication digest are unchanged and old-shape rows still load exactly.
- AlarmConditionStateSchema.MigrateInlineComments: idempotent per-row move of
inline trails, run by LocalDbSetup.OnReady after RegisterReplicated so the
moved entries replicate. Comment table joins ReplicatedTables (now five).
Tests: constant transition payload at 10 vs 10k comments (+ instrument
positive control), old-shape back-compat load, migration idempotency /
partial re-run / lazy move, OnReady boot migration captured in the oplog,
and two loopback pair tests (A->B and B->A trail convergence; the shared
entry converges to one copy, with a distinct-alarm positive control).
Refs #748
Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
The Wave-2 lanes gave AbCip, AbLegacy and TwinCAT real whole-array write
encoders (review 07 U10/U9), but the capability table still listed all three
as scalar-only, so every array node materialised read-only and a client write
was BadNotWritable before it reached the driver.
Rows added, each listing only what WriteAsync encodes end to end:
- AbCip: SInt/USInt/Int/UInt/DInt/UDInt/Real/Dt/LInt/ULInt/LReal, declared
length, not a safetyTag. BOOL/STRING/Structure arrays stay read-only.
- AbLegacy: Int/AnalogInt on N/B/A, Long on L, Float on F (file-width match),
plain <file><n>:<word> address, <= 256 elements; not /bit, .sub, indirect,
S:/I:/O:, nor native PLC-5.
- TwinCAT: every codec element type, declared length; not a BOOL bit-in-word
path.
Elements convert in all three encoders, so the rows never key on an exact
node DataType (ScadaBridge int[] -> Int16 stays writable); they leave out
only node types whose elements never convert (DateTime/String over numeric,
Boolean on AbLegacy, STRING under DateTime on TwinCAT: culture-dependent).
Tests: driver-level parity for each driver (every element type x node type
plus shape cases, against the real write path), the AbLegacy parity tests
un-skipped, Core.Abstractions row tests, a composer test (three drivers
writable, refused shapes read-only), and the existing AbCip negative cases
moved to a BOOL array. Docs: Uns.md table, per-driver docs, data-plane rules.
Refs #737#738#739
Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
Review m-6. The XML doc said the token is cancelled on timeout only.
ScriptRunGuard links it to the caller's token
(CreateLinkedTokenSource(ct)), so it is also cancelled on shutdown or
when the evaluating actor stops mid-run. The doc now says so.
Refs #731#734
Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
It had no production sender: every driver data change is already a
batch, and the comms-loss fan was converted earlier. DependencyMuxActor
drops its per-value receive, DriverInstanceActor drops the record, and
DependencyMuxActorTests' 13 sends move to the ValuePublishes.Single
one-item batch (same semantics through the batch path). XML docs and
the living docs (DriverLifecycle, VirtualTags, ScriptEditor) are
reworded; historical plan docs are left as written.
Refs #731#734
Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
ScriptContext.CancellationToken is backed by an AsyncLocal that
ScriptEvaluator.RunAsync scopes around the compiled delegate, so it is
exactly the token ScriptRunGuard cancels on timeout. A polling script
can now end its own abandoned run and free the slot. AsyncLocal rather
than a field because the scripted-alarm engine reuses one context per
alarm: an orphan keeps seeing its own cancelled token. The scoping
entry point is internal (IVT to Core.Scripting) so a script cannot
replace its own token.
Editor parity: ScriptAnalysisService dot-completion now walks the
receiver's base chain (instance members only, stopping before
System.Object). Before this, a member declared only on ScriptContext
compiled and published but never completed. DiagnoseTests pins the
member as allow-listed; CompletionTests pins the completion.
Refs #731#734
Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
Program.cs binds ScriptingOptions from Scripting:* once, clamps every
numeric knob through ScriptingOptions.Sanitized (logging a Warning per
adjustment) and registers the one instance. It now reaches all three
script seams: RoslynVirtualTagEvaluator, the scripted-alarm engine
built in DriverHostActor (new trailing Props parameter, wired in
WithOtOpcUaRuntimeActors) and every Calculation driver (node options
as the base, per-driver config keys override one knob on a copy).
ScriptRunGuard sanitizes again in its ctor and computes the stuck
threshold with a saturating multiply, so an absurd
StuckOrphanQuarantineFactor can no longer throw OverflowException on
a refused run.
Refs #731#734
Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
Ruling refinement on I1. Only timeouts are strikes, so a never-returning
while(true), which times out once and is then only refused, was never
quarantined, and that defeated E1. A refusal now quarantines immediately
when the orphan it was refused behind has been executing for longer than
ScriptingOptions.StuckOrphanQuarantineFactor x the script's timeout
(default 10, so 20 s at the 2 s default; 0 or less disables it). It uses
the same quarantine path: one Error, otopcua.script.quarantined with
reason=stuck, Bad, lifted on source change or redeploy. A young orphan
(a short overrun) is still a plain refusal.
The guard stamps the run's start (Stopwatch timestamp) when the pool
picks it up and clears it when the run really finishes.
Tests: the 30-trigger while(true) regression (paced) now ends
quarantined with reason "stuck", still with exactly one orphan and a
healthy sibling. Positive control: a 2x overrun is refused, not
quarantined, and evaluates again once the orphan completes. A factor of
0 disables the rule. Docs (ScriptEditor "Runtime safety", VirtualTags)
and the seam remarks are updated.
Refs #734
Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
The one-snapshot refactor re-sorted every collection in memory with an ordinal comparer, while the old
composer's SQL ORDER BY used the database's case-insensitive collation; mixed-case keys therefore re-hashed
an unchanged configuration (spurious drift fleet-wide, a "new" revision on a no-op redeploy). The snapshot
factory now reads every composed table ORDER BY its natural key server-side and Compose serialises in the
order given. Pinned byte-for-byte against a verbatim replica of the old composer on the real SQL Server.
Refs #733
Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
- ScriptLogRateLimiter.LimitPerWindow. The ScriptLogTopicSink and
VirtualTagActor summaries now print the configured limit, not
DefaultLimitPerWindow. Tests assert "limit N/window" for a non-default
N.
- CalculationDriver.Dispose cancels, then takes _evalGate (bounded 2 s)
and disposes the gate and _evalCts. The pump and timer ticks enter the
gate through TryEnterEvalGateAsync, which stops cleanly once the
driver is disposed. Test: Dispose with a 6 s runaway run in flight
returns promptly, disposes both, and a late trigger is harmless.
- ScriptedAlarmEngine captures _shutdownToken once and disposes
_shutdownCts in Dispose when every drained task has completed. A task
stuck past the grace leaves the CTS to the GC. Lifecycle tests assert
the disposal.
- DraftValidator: CalculationNoEffectiveTrigger is skipped for a
change-triggered tag whose scriptId does not resolve. That case is
already CalculationScriptMissing or CalculationScriptNotFound, and
there are no reads to count. Both existing tests now assert the
absence (red before the change).
- docs/VirtualTags.md: System.Threading is denied as a whole namespace,
with an allow-list; the type list is now accurate (Unsafe,
RuntimeHelpers, and the ForbiddenMembers on System.Type); the resource
limits section covers the run guard and the allocation budget.
docs/ScriptEditor.md: accurate deny-list and a new "Runtime safety"
section covering timeout, refusal, quarantine after 3 timeouts,
allocation budget, lifting, and rate-limit summaries that appear only
on the source's next entry after a burst then quiet.
Refs #734
Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
- I1 ScriptRunGuard: a run refused because the previous one is still
executing is a Bad publish plus otopcua.script.run_refused, and no longer
a quarantine strike. Only a timeout (and an allocation breach) counts;
a refusal neither adds to nor resets the timeout streak; a completed run
(value or throw) resets it. On a 1 Hz trigger with a 2 s budget a single
4 s overrun is one timeout plus two refusals and must not quarantine.
Tests updated across the guard, the alarm engine, the Calculation
evaluator and the VT evaluator, plus
Refusals_neither_count_toward_nor_reset_the_timeout_streak.
- I3 ScriptedAlarmEngine: the quality bucket is tracked only after the
predicate completes (or in the inputs-not-ready path), so the tracked
bucket is the last one annotated. A refused or quarantined predicate
used to re-track the input bucket Good before the attempt and then flip
it Bad, emitting QualityChanged on every push. The run-guard tests now
count emissions (exactly 1 across N pushes, refused and quarantined
cases), with a healthy sibling's dispatch count as the positive control
that every push was processed. Both were red before the fix.
Refs #734
Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
Every deploy stored the whole gzip'd fleet snapshot in Deployment.ArtifactBlob forever (R03:1.2).
DeploymentRetentionActor (admin singleton, proxy-less) sweeps every Deployments:Retention:
IntervalMinutes (60): terminal rows (Sealed / PartiallyFailed / TimedOut) older than the newest
KeepSealed (50) and not referenced by a NodeDeploymentState started/applied within ReferencedDays
(30) get their blob EMPTIED — never deleted, never NULLed: the column stays NOT NULL and every
reader already treats zero bytes as "no artifact" (#485), so no schema migration and no reader
change. In-flight rows are never touched; MaxRowsPerSweep (200) bounds one sweep.
Refs #733
Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
- R02:H1 ScriptedAlarmHostActor gains a factory Props
(Func<(upstream, engine)>) so every incarnation builds and owns its own
engine; a supervision restart no longer re-creates the actor around a
disposed engine. The legacy instance-taking Props stays source-compatible
and is now restart-safe in the weaker sense (PreRestart marks the stop as
a restart, PostStop then detaches instead of disposing). DriverHostActor
still spawns through the legacy overload (not owned by this lane — F3).
- R02:H3 OnAlarmCommand runs every engine op under a 10 s token linked to
the actor's lifetime (DefaultCommandTimeout / Props commandTimeout) and
logs a Warning on timeout instead of suspending the mailbox on a stalled
store.
- R02:L3 DependencyMuxTagUpstreamSource.RetainOnly(loaded dep-ref union),
called after every successful load.
- R02:P6 VirtualTagActor's per-failure ScriptLogEntry is rate-limited per
tag (ScriptLogRateLimiter, 20/min + one summary); the Bad publish and the
metric are untouched.
- R07:H6 DependencyMuxActor retains the last (value, status, ts) per
ever-registered ref (through unregisters) and replays them in one batch to
a RegisterInterest(ReplayRetained: true) — VirtualTagActor, the Calculation
mux adapter and the alarm host opt in; the historization recorder does not
(a replay there would record a stale sample per deploy). CalculationDriver
records inputs regardless of _running so a replay landing before
InitializeAsync still satisfies the change-gate.
- R07:P4 CalculationEvaluator negative-caches compile failures by source
hash (one Roslyn attempt + one Warning per source until redeploy); the
driver's script-log entry fires on the Good→Bad transition or a change of
reason, matching the once-per-transition Bad publish.
- R07:H8 array results dedup structurally.
- R07:L2 ShutdownAsync drops the compiled scripts (collectible ALCs).
- R07:U5 DraftValidator rule CalculationNoEffectiveTrigger: a tag with no
timer whose change trigger is off or whose script has no literal
ctx.GetTag read can never evaluate.
Refs #734
Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
- R05:1/12 ScriptedAlarmEngine: _valueCache is cleared in LoadAsync (it is
re-seeded for the referenced paths) so retired tags no longer accrete across
deploys; OnUpstreamChange no longer writes the cache after Dispose.
- R05:3 ScriptedAlarmEngine.Dispose drains in-flight work for at most 5 s
(DisposeDrainTimeout), then cancels it through a shutdown token every
DrainDirtyAsync/ShelvingCheckAsync now runs on, logs a Warning and gives
one second of grace — a stalled IAlarmStateStore.SaveAsync can no longer
hang host shutdown. A save that completes inside the bound is still drained
(positive control test).
- R05:6 ScriptedAlarmSource logs a throwing OnAlarmEvent subscriber at
Warning (naming the alarm) instead of swallowing it.
- R05:7 LocalDbStoreAndForwardSink keeps a cached dead-letter counter in step
with every dead-letter/revive/purge path and resyncs it with the queued
counter on the same cadence; GetStatus no longer runs a COUNT(*) scan.
- R05:9 ScriptLogRateLimiter (20 per source per minute, fixed window, one
summary entry naming the dropped count when the window rolls) behind
ScriptLogTopicSink (key = script|vtag|alarm) and ScriptLogCompanionSink
(key = script name).
Refs #734
Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
R05:10 — the namespace deny-list covered System.Threading.Tasks but not
System.Threading, so SemaphoreSlim / Monitor / SpinWait / Mutex /
EventWaitHandle could park a script's pool thread past its budget (or touch a
system-global kernel object); System.Type.GetType(string) / GetMethod* /
InvokeMember and System.Runtime.CompilerServices.RuntimeHelpers resolved and
reach a forbidden type without naming it syntactically.
ForbiddenTypeAnalyzer now denies the whole System.Threading namespace with a
type-granular allow-list (CancellationToken, Interlocked, Volatile — the
sync-free primitives), adds a member deny-list for System.Type's reflection
entry points (GetType/GetMethod(s)/InvokeMember/GetProperty(ies)/GetField(s)/
GetMember(s)/GetConstructor(s)/GetEvent(s)/GetDefaultMembers; instance
object.GetType() stays allowed) and puts RuntimeHelpers on the type deny-list.
Thread/ThreadPool/Timer leave the type list — the prefix rule reaches them.
R05:11 — the allocation budget landed with E1's ScriptRunGuard
(Scripting:MaxAllocBytesPerRun, default 64 MB): sampled around each run,
breach quarantines. It is observation after the fact; a pre-emptive OOM guard
still needs an out-of-process runner (follow-up issue in the lane report).
The Monaco editor shares the analyzer, so DiagnoseTests pins every new form
as an OTSCRIPT_FORBIDDEN marker and every allow-listed form as clean.
Refs #734
Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
A timed-out script is abandoned, not stopped, and every seam re-ran it on the
next trigger, leaking one pool thread per trigger (R03:1.1, R05:2, R07:L1).
A shared ScriptRunGuard, keyed per script instance, now:
- refuses a run while that script's previous orphan is still executing
(Bad "previous evaluation still running" + otopcua.script.run_refused);
- quarantines after 3 consecutive non-completions or an allocation-budget
breach (sampled with GC.GetAllocatedBytesForCurrentThread; default 64 MB):
no further dispatch, Bad quality, one Error, otopcua.script.quarantined,
cleared on source change / redeploy / reset;
- starts the budget clock when the pool picks the run up, so starvation is
no longer charged to the script (R07:P5), and cancels the evaluator token
on timeout.
Wired into RoslynVirtualTagEvaluator, CalculationEvaluator (new async
EvaluateAsync; the driver's pump and timer ticks now await the run behind a
SemaphoreSlim gate) and ScriptedAlarmEngine (a non-completing predicate
holds state and annotates the condition BadInternalError). ScriptingOptions
(Scripting:MaxAllocBytesPerRun, Scripting:QuarantineAfterConsecutiveTimeouts)
carries the knobs; the Calculation driver config accepts the same two keys.
Tests: guard unit suite (incl. the 30-trigger runaway leaving one orphan with
a healthy sibling as positive control), per-seam refusal/quarantine/reset
tests. The S12 race test now uses one vtag id per loop: twelve concurrent
evaluations of ONE id is a shape production never produces and the guard
refuses by design.
Refs #734
Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
CallTimeoutSeconds was bound, defaulted and documented but consumed
nowhere. TelemetryDialSupervisor now arms a per-generation deadline on
every dial: a dial that has delivered no event by the deadline is
cancelled and treated as a transport failure (reconnect with a failure
streak). A live node replays its snapshots on subscribe, so this bites
only a dial that never connects (a black-holed host, where the socket
connect has no timeout). An established stream is deliberately not
bounded — the wire has no heartbeat, keepalive pings own its liveness.
Review ref: 02/U2.
Refs #731
Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
StartDeployment read the whole config DB twice with no transaction — DraftSnapshotFactory for
the validator, then ConfigComposer for the artifact, plus clusters/nodes and Script.SourceCode
again in between. An AdminUI save landing between the two shipped an artifact the DraftValidator
never saw.
- DraftSnapshot carries ServerClusters, ClusterNodes and ApiKeys; DraftSnapshotFactory reads
every table inside one SNAPSHOT transaction on a relational provider (no-op on InMemory or
under an ambient transaction).
- ConfigComposer.Compose(DraftSnapshot) is pure over the snapshot (ordinal-sorted keys);
SnapshotAndFlattenAsync(db) delegates to it, byte-identical.
- AdminOperationsActor validates topology, estimates compile cost and composes the artifact
from the one snapshot it holds.
- Migration EnableSnapshotIsolation: ALTER DATABASE CURRENT SET ALLOW_SNAPSHOT_ISOLATION ON
(suppressTransaction; no model change).
Tests: the SQL Server snapshot read ignores a row committed mid-read (positive control: a later
read sees it); an edit injected between the Equipment and Tags reads is absent from the shipped
artifact and rejected by the next deploy; Compose(snapshot) == SnapshotAndFlattenAsync(db).
Refs #733
Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
DriverInstanceActor.PostStop now bounds IDriver.ShutdownAsync by a
shutdownTimeout (default 5 s, logged on overrun) and then disposes the
driver (IAsyncDisposable preferred, else IDisposable) — but only on a
real stop. PreRestart flags the restart so the PostStop it triggers
skips the dispose, because a supervision restart re-uses the same
driver object (Props closure) and the host re-initialises it.
DiscoveryDriverBrowser disposes its throwaway capture/probe driver
after the bounded shutdown, including after a failed or timed-out one.
Review refs: 02/L1, 02/P4, 06/G-1 (host context), 07/L2.
Refs #731
Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
Clears CommentChecker findings in src/ (3,280 -> 209, the remainder being false
positives like PLC-5/IEEE-754/M0.0) by adding missing summaries/params/returns,
normalising inheritdoc usage, and removing Gitea/review tracking IDs while keeping
the rationale they annotated. Comment-only: comment-stripped sources are identical
to the parent commit.
Item 1: IHistorianDataSource.MaxRawValuesPerRead (Gateway: learned ceiling - 16 = L; 0 elsewhere, guard-tested).
RawPageLimit plans every raw page on top of #692's lookahead: with L > 0 the read (page P + boundary ties k
+ lookahead) is <= L, P = min(N, L - k - 1) (uncapped: L - k - 1); no room => straight to the tie-cluster
over-fetch (never an uncapped or underflowing request). One re-plan + re-read when the read taught a new
limit or was refused for crowding pre-window rows (HistoryReadLimitExceededException.PreWindowRows): the
page shrinks by them. Over-fetch bound min(MaxTieClusterOverfetch, L - 1); slice size never 0.
Review fixes: F1 a window starting at/before the epoch counts AVEVA's synthetic row as pre-window (a
ceiling-truncated read was reported complete); F2 the page path disables ReverseRawReader's own re-run
(cold end-only ceiling 20/N=3000 cost 3,023 reads, now <= 40); F3 crowded look-back re-read around, not
refused; F4 the #687 crowding re-read room at least doubles (up to 8 re-reads), so a cluster of hundreds
just before a resume point no longer fails on a default gateway; a crowding refusal that leaves no room
is BadTooManyOperations, never a tick-by-tick cursor.
Item 2: ServerHistorian:EventReadHorizon (default 00:00:00 = exact, unchanged). When set, every fresh
HistoryReadEvents window's earlier bound is raised to now - horizon (both directions, all backends).
Lossy by design; documented.
Item 3 (native TimeAverage/Total re-check) stays blocked on historiangw#27.
Review follow-up: on the Gateway every value is a Double (AVEVA stores a Boolean
as a 0/1 Int2), so the raw-evaluated AtTime interpolated a Boolean node to 0.5.
- IHistorianDataSource gains a default-interface overload
ReadAtTimeAsync(fullReference, timestamps, TypeCode? valueType, ct) that
forwards to the existing one (SqlServer/QuestDB/Null/fakes unchanged).
- OtOpcUaNodeManager.HistoryReadAtTime passes the node's scalar DataType
(AtTimeValueType: built-in type -> TypeCode; arrays/other types => null);
Raw and UNS NodeIds alike.
- GatewayAtTimeReader converts each raw row to the hinted kind BEFORE the shared
AtTimeEvaluator runs (Boolean = value != 0 => stepped; integers => Int64 =>
interpolated, rounded half away from zero, returned as the node's integer
type; floats Double), so the evaluator's own type branch decides.
- AtTimeEvaluator boxed every interpolated Int64 as a Double (`cond ? (long)x :
v` widens to double); its "keeping its type" test passed because Shouldly
ShouldBe compares numbers across types. Fixed (affects SqlServer/QuestDB too)
and pinned with ShouldBeOfType.
The null source has no history in either direction; reporting false
made the server walk an empty reverse events read back through three
empty forward reads. It now answers the reverse overload with the empty
result itself (guard test unchanged: it still declares both members).
A reverse or end-only capped events read on a backend without
SupportsReverseEvents (Gateway, Null) was refused
BadHistoryOperationUnsupported (explicit reverse) or answered with the
OLDEST N ascending (end-only, #675). It is now served newest-first by
the same ReverseWalk as raw reads: page-sized ms-aligned segments,
never answered from a read the source flagged truncated, ties sorted by
EventId (the gateway orders by event time only - the fake shuffles
them per call), one extra event walked to signal 'more', uncapped = one
forward read reversed, and the width + proven-empty stretch carried in
the events cursor. The oversized-cluster over-fetch keeps exactly the
events at T in the same order. A tie cluster over the budget is
BadTooManyOperations. Docs note the gateway's EventReadMaxRows must be
unlimited (it truncates silently).
#669 turned an end-only capped HistoryRead-Raw into a reverse read (start > end)
for every backend; the gateway rejects start > end (InvalidArgument), so on the
default Gateway backend every end-only capped and every explicit reverse raw read
failed the node and was counted as a backend health failure.
- IHistorianDataSource.SupportsReverseRaw (default false; SqlServer/QuestDB/Null
true, Gateway false; guard test pins that every production backend declares it).
- OtOpcUaNodeManager sends a reverse window only to a backend that declares it;
otherwise ReverseRawReader walks (lo, hi] back in bounded forward segment reads
(1 min, x4 while sparse; each <= MaxTieClusterOverfetch + 1 rows; a full read is
re-read narrower, never answered truncated; filtered on sample timestamps, which
also drops AVEVA's fabricated query-start rows; stops at the Unix epoch). A
one-timestamp tie cluster over the bound => BadTooManyOperations.
- ServeRawPagedAsync maps HistoryReadLimitExceededException to BadTooManyOperations.
- GatewayHistorianDataSource.ReadRawAsync refuses start > end unsent, without
recording a health failure.
ServerCapabilities/AggregateFunctions (i=2997) is Part 5's list of every
aggregate the server supports, and live AggregateFilter subscriptions still
compute all 37 SDK aggregates, so it keeps the SDK's full set; only i=11201
follows the backend. A null SupportedAggregates is treated as the interface
default (all) by both the advertisement and HistoryRead-Processed (it used to
NRE inside the batch). Docs: i=2997/i=11201 split and the boot window.
Co-Authored-By: Claude <noreply@anthropic.com>
IHistorianDataSource.SupportedAggregates (default: every HistoryAggregateType)
is the one authority for both ServerCapabilities/AggregateFunctions (i=2997) and
HistoryServerCapabilities/AggregateFunctions (i=11201) and for the up-front
BadAggregateNotSupported in HistoryRead-Processed. Gateway declares
AggregateModeMapper.Served (4), SqlServer/QuestDB AggregateFamilies.Served (17),
Null none. Only the folder references are trimmed; the SDK's calculator
factories stay so live AggregateFilter subscriptions keep working.
Co-Authored-By: Claude <noreply@anthropic.com>
Joseph Doherty