Child_is_respawned_after_unexpected_termination watched the child die, then re-applied once. The host prunes _children only when its own Terminated arrives, so a re-apply that lands first still sees vt-1 as present, spawns nothing, and the test times out at 15 s (caught 1 in 5 full Runtime.Tests runs on 2026-11-09). Re-apply inside AwaitAssert until the new RegisterInterest arrives; the assertions are unchanged. 30/30 isolated runs pass.
Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
(a) ScriptRunGuard logs its quarantine Error through the seam's script root
logger, which carries no ScriptName, so ScriptLogCompanionSink mirrored it to
the main log as "[Script] unknown emitted Error ..." and rate-limited every
quarantine on the node under one shared "unknown" key. The guard now binds the
script instance key as ScriptName (ForContext) on the quarantine Error and the
subscriber-threw Warning, so the mirror names the script and limits each one
separately. Test drives two real quarantines through the companion sink with a
1-per-window limiter: both mirrored, each named, none "unknown". Mutation-
checked: dropping the ForContext fails it.
(b) Evaluated a quarantine-specific OPC UA status (BadOutOfService) and kept
BadInternalError: it is the documented code for any script-attributable fault
on virtual tags, Calculation tags and scripted-alarm quality (and the
out-of-process runner design keeps it), BadOutOfService reads as a deliberately
disabled point, a new code is a ScadaBridge-visible contract change, and the
once-per-Good->Bad dedup means a timeout-driven quarantine would rarely show a
different code anyway. Documented in docs/VirtualTags.md ("Quarantine: what a
client sees"), with pointers from Calculation.md, ScriptedAlarms.md and the
OpcUaStatus.BadInternalError doc comment, plus where a quarantine IS visible.
Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
In the normal timeout branch (some acks landed) the deployment row was set to
TimedOut with a null FailureReason, so GET /api/deployments/{id} could not say
which node to fix; only the central log named it. The branch now persists a
reason naming the silent nodes ("Timed out waiting for acks (n/m landed). No ack
from: ..."), bounded to 20 named nodes plus a count of the rest, and still
backstopped by the 2048-char Truncate.
Tests: the reason names the silent node and not the acked one; on a 25-node
silence it names 20, counts 5 and fits the column. Mutation-checked: dropping
the SetFailureReason fails both, dropping the bound fails the second.
Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
The docker-dev centrals logged "Mesh PSK secret ot-mesh-psk-<cluster> ... is
absent (or blank) in the store; the key is REVOKED" for every cluster on every
contact refresh, while every gRPC dispatch and ack authenticated.
Investigated: not an auth bypass. The rig supplies each cluster's key through
MeshTransport:ClusterPsks (central side, equal to that cluster's nodes'
MeshTransport:GrpcPsk) and seeds nothing in the store. TryGetPsk/Matches consult
the override first on every call, so enforcement was on; only RefreshAsync read
the store for those clusters regardless and logged a revocation that never took
effect. RefreshAsync now skips clusters covered by a non-blank override, so the
REVOKED warning is only logged for a cluster that genuinely has no key.
Test pins it with a positive control (a cluster with no override is still read,
warned about and refused) and enforcement on the override key (wrong token
rejected). Mutation-checked: removing the skip fails the test.
Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
Choice: implement the timer (option a). The field was designed and promised as a
working trigger: the entity doc says ChangeTriggered and TimerIntervalMs together
realize the trigger model, the DB check constraint requires ChangeTriggered OR a
timer >= 50, the AdminUI virtual-tag modal exposes "TimerIntervalMs (optional)" and
UnsTreeService rejects a tag with neither, and the 2026-06-08 design specified
"ChangeTriggered OR TimerIntervalMs>=50". Only the runtime never honoured it (the
standalone TimerTriggerScheduler was deleted in #603), so a timer-only tag sat at
BadWaitingForInitialData forever.
- EquipmentVirtualTagPlan carries ChangeTriggered/TimerIntervalMs (in equality, so
an edit respawns the child); both the composer and the artifact decode fill them.
- VirtualTagActor schedules a repeating tick (PreStart, cancelled in PostStop,
clamped to the 50 ms floor). ChangeTriggered=false makes the timer the only
trigger. A tick during an in-flight async run (#779) is dropped, not stashed, so
slow scripts never build a backlog of runs.
- DraftValidator VirtualTagNoEffectiveTrigger (hard error, same severity as
CalculationNoEffectiveTrigger): no timer and no change trigger with a literal
ctx.GetTag read.
Tests: VirtualTagActorTimerTests, DraftValidatorVirtualTagTriggerTests, artifact
decode test. Mutation-checked: no scheduling, stashing ticks, no cancel in
PostStop, and removing the validator call each fail a test.
Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
Gates A (#731), B (#732), C (#733) and E (#734) run on the local docker-dev
rig against origin/master 1b9c36f3 on 2026-11-09: all four PASS. Records the
procedure, the measured thread/fd/RSS, subscription and deploy-status
numbers per gate, plus six non-blocking observations.
Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
Catalogue of what breaks moving from 1.5.378.106 to 2.0.1: restore
break (MEL 10.0.12), 42 -> 103 compile errors in the first two waves,
root-cause groups with API mappings, PeerAddress replacement for the
reflection-based remote endpoint resolver, effort and risks.
Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
The Lane H2 remediation made Periodic(100 ms) the options-class default, but
appsettings.json, docker-dev and Program.cs's bad-value fallback all used
PerEntry, so effective behaviour never changed and the docs contradicted the
shipped config. The options default is now PerEntry; Periodic stays opt-in
(<= CommitIntervalMs of crash loss in exchange for throughput). Program.cs now
uses ContinuousHistorizationOptions.ResolveCommitMode() so the fallback shares
the single DefaultCommitMode constant. docs/Historian.md updated.
Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
Auth 0.4.1 (BindMode stays ServiceAccount; empty password now refused in the library too; Username canonicalised from UserNameAttribute). No API breaks, no test changes needed.
Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
The 1500 ms pre-control and 500 ms trailing sleeps are gone. ConditionRefresh
completes server-side before SubscribeAlarmsAsync returns, so refresh events
queue ahead of the control Acknowledge; a trailing message-change barrier event
is projected and awaited (bounded 10 s poll) so any stray event must arrive
before it. The assertion is now the exact sequence [Acknowledge, Unspecified],
no weaker than before.
Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
The simulator now packs Int32/Boolean/Float/Double/String arrays per the
Sparkplug B spec (independent of the driver codec) and seeds Profile, Flags
and Curve in the standalone catalog. SparkplugLiveTests gains a live test that
asserts typed int[]/bool[]/float[] values from the birth, a changed vector
(including a different Boolean length) from alias-only DATA, the read path,
and BadTypeMismatch for an array metric bound without isArray. Run against a
throwaway local Mosquitto; mutation-checked (scalar binding / seed removed).
Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
Item 5: DraftValidator now emits SqlQueryNotDeclared (always blocking) for a Query tag whose
query is absent from the merged driver config's queries map, via the driver's own tag parse.
Item 3: StartDeployment appends a non-blocking warning to the accepted message when the Tag
entity's AccessLevel and the Sql blob's writable flag disagree.
Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
The docs said script authoring was Admin-gated (VirtualTags.md "Known
resource limits") and that a Designer "cannot publish" (security.md roles
table). The code admits Designer to both: Scripts, ScriptEdit, EquipmentPage
(virtual tags, scripted alarms), /api/script-analysis/* and Deployments all
carry AdminUiPolicies.ConfigEditor = RequireRole("Administrator",
"Designer"). That is a recorded decision, not drift: AdminUI-001 (98b27fc1)
loosened the analysis backend from FleetAdmin to Administrator,Designer to
match the page gate, and R2-05 converged the same pages on ConfigEditor
"with the same semantics". So the docs follow the code:
- VirtualTags.md: the primary control is ConfigEditor (Administrator or
Designer), which also gates deploy.
- security.md roles table: Designer = the whole ConfigEditor surface incl.
scripts and deploy; Administrator = Designer + the FleetAdmin-only surface
(role grants, API keys, certificate actions, manual failover) — the old
"publish, cluster/node CRUD" were ConfigEditor all along.
- 2026-06-09 Monaco plan: its FleetAdmin gate is marked superseded.
Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
Reproduced first, out of process (ScriptRecursionContainmentTests re-launches
the xunit v3 test assembly to run one recursive script through
ScriptEvaluator + ScriptRunGuard, so the runner never executes one): before
this change every shape — local function, mutual local functions,
self-referencing Func<>, anonymous self-application, recursion through LINQ,
an expression-bodied void lambda, an anonymous method — ended the child with
exit 134 (SIGABRT, "Stack overflow."). A StackOverflowException cannot be
caught, so in production one script takes the whole node down.
Fix: ScriptEvaluator.Compile now rewrites the validated wrapper
(ScriptStackGuardInjector, after ForbiddenTypeAnalyzer) so the script body and
every lambda, anonymous method and local function starts with
ScriptStackGuard.Check(), which throws ScriptStackExhaustedException when
RuntimeHelpers.TryEnsureSufficientExecutionStack reports the stack nearly
spent. The script then fails like any throwing script (Bad, script log) and
the process lives.
Why a runtime check, not an analyzer rule: recursion is not decidable from
syntax (anonymous self-application never names itself), and bounded recursion
must keep working. Expression-bodied lambdas/local functions become blocks
(`return e;` or `e;`, chosen from the bound symbol, so the same overload
binds); expression-tree and async lambdas are left alone. Limits (documented
on ScriptStackGuard and in docs/VirtualTags.md): depth depends on the thread's
stack size; a recursion that never re-enters script code is not seen; a script
may catch the exception, which only unwinds it.
Tests: ScriptRecursionContainmentTests (7 shapes, child process each; all 7
fail with exit 134 when the rewrite is disabled), ScriptStackGuardInjectionTests
(bounded recursion and every rewritten lambda/local-function shape keep their
value; the expression-tree case fails if those lambdas are rewritten).
Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
VirtualTagActor called the blocking IVirtualTagEvaluator.Evaluate, and
RoslynVirtualTagEvaluator blocked on the run guard with
.GetAwaiter().GetResult(), so every script run parked a dispatcher thread for
up to its 2 s budget (a first-use Roslyn compile did too). This is the #750 P0
refactor for virtual tags.
- IVirtualTagEvaluator.EvaluateAsync (default: a completed task over the
snapshot overload, so the null evaluator and fakes are unchanged).
- RoslynVirtualTagEvaluator implements it without blocking: passthrough and
compile failures complete synchronously, a first-use compile goes to the
pool, the run is awaited. The blocking overload now delegates to it.
- VirtualTagActor: a completed task is applied in the same turn, as before;
an incomplete one switches to an "evaluating" behaviour that stashes every
dependency/reassert message until a private completion message arrives,
then applies the result and unstashes. Each trigger is still evaluated
against its own inputs, results are applied/published in trigger order and
the dedup/Bad-transition state sees the same sequence. The dependency map is
not touched while a run reads it.
Calculation and scripted-alarm serialisation is NOT relaxed: neither blocks a
thread, and running independent scripts in parallel is not semantics-neutral
(pool consumption bound, cross-tag publish order, timer/pump exclusion for
Calc; state-machine/persist boundary, transition timestamps and the
store-failure path for alarms). Recorded in docs/VirtualTags.md and the #750
design's P0 entry.
Tests: VirtualTagActorAsyncEvaluationTests — on a one-thread dispatcher a slow
script no longer starves a sibling actor and the slow actor answers Identify
(fails when the actor calls the blocking Evaluate); triggers during a run are
evaluated afterwards in order against their own inputs; a reassert during a
run is applied after it; a faulted evaluation takes the evaluator-threw path
(the ordering/reassert tests fail with the stash removed).
RoslynVirtualTagEvaluatorAsyncTests — a slow script and a cold compile return
an incomplete task (fail without the override); cheap outcomes and timeouts
match the blocking overload.
Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
DriverHostActor built ScriptedAlarmEngine without a scriptTimeout, so every
production alarm predicate fell back to TimedScriptEvaluator.DefaultTimeout
(250 ms) while virtual tags and Calculation tags ran on 2 s. History shows no
decision behind it: 250 ms is the generic Phase-7 A.4 default, the VT
evaluator moved to 2 s in F8b (219d10a2) and the alarm engine was simply never
given a value.
- New node-wide Scripting:RunTimeout (default 2 s, clamped to (0, 60 s],
non-positive -> default, each clamp a startup Warning) on ScriptingOptions.
- The VT evaluator and the alarm engine default to it; DriverHostActor now
builds the engine through CreateScriptedAlarmEngine, which passes it
explicitly. Calculation keeps its per-driver runTimeoutMs.
- Docs: ScriptEditor (config table), ScriptedAlarms, VirtualTags.
Tests: ScriptedAlarmEngineBudgetTests (a 600 ms predicate now completes and
activates; default and configured budgets), ScriptedAlarmEngineBudgetWiringTests
(the host-built engine), ScriptingOptionsBindingTests (bind + clamps). The
engine and wiring tests fail with the old 250 ms default restored.
Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
allowWrites is no longer inert: SqlDriver implements IWritable, default off.
- Gates (outermost first): node Writable + WriteOperate (server), driver
allowWrites, the tag's own TagConfig "writable": true (strict JSON boolean;
anything else refuses the tag, and the deploy gate says why), and a model that
names exactly one row — KeyValue, or WideRow by where-pair. topByTimestamp and
Query are BadNotWritable.
- SqlWritePlanner: the only statement a write can become is
UPDATE <table> SET <value col> = @v WHERE <key col> = @k, built from the
catalog-gate-canonical identifiers (dialect-quoted) with value and key bound;
value-before-key marker order so positional ODBC binds correctly. No authored
write SQL, INSERT, UPSERT or procedure.
- Executed in a transaction that commits only when exactly one row was touched:
0 rows -> BadNoData (never inserts), >1 -> rolled back, BadConfigurationError,
unknown count -> rolled back, BadNotSupported. Bounded by operationTimeout;
provider errors -> BadCommunicationError; health untouched.
- Discovery advertises Operate only where a write could succeed.
- AdminUI: driver form's Allow writes switch is live; tag editor gains a
per-tag write switch; model validates writable strictly (parity corpus).
- Factory no longer warns "allowWrites ignored"; it logs whether writes are on.
- Tests: SqlWriteTests (planner SQL/bind order/injection/coercion, every gate,
readback, absent key, two-row rollback, gate-rejected tag, discovery);
MySqlReadTests write legs against InnoDB (keyed row only, two-row rollback,
server-side grant refusal, default-off). Seed adds a setpoints table and the
one UPDATE grant.
(lmxopcua#753)
Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
SqlProvider.Odbc is now constructed; every enum member is a working backend.
- OdbcDialect: System.Data.Odbc 10.0.14 (no vulnerable packages), ANSI double-
quote quoting, positional `?` markers, INFORMATION_SCHEMA catalog (system
schemas of the common backends excluded by name — the live gate caught
psqlODBC listing pg_catalog), FETCH FIRST n ROWS ONLY limits, a conservative
1000-parameter chunk. Positional binding is safe because every caller adds
parameters in marker order (planner IN list, catalog schema-then-table).
- Named queries on a positional provider are checked by `?` count against the
declared params (literals/comments ignored), since there is no name to match.
- Supported backends are those whose SQL accepts that profile (PostgreSQL via
psqlODBC proven, Snowflake-like); SQL Server/MySQL/DB2 should use their native
provider. The host needs the ODBC driver manager + vendor driver.
- Reserved-provider controls in factory/probe/browser tests now use an
undeclared ordinal (99); the AdminUI form offers Odbc and a test pins the
dropdown theory to SqlDialects.Supported.
- Tests: OdbcDialectTests (unit, any host). OdbcReadTests + Docker/odbc/
run-odbc-gate.sh run the built suite in a throwaway Linux SDK container with
unixODBC + psqlODBC against a throwaway seeded PostgreSQL: KV IN-list
positional binding, where-pair, topByTimestamp via FETCH FIRST, a two-
parameter named query, and the browse walk. macOS has no vendor ODBC driver,
so there the suite skips.
(lmxopcua#753)
Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
SqlProvider.MySql is now a constructed backend instead of a reserved name.
- MySqlDialect (Driver.Sql.Dialect): MySqlConnector 2.6.2 (MIT, fully managed,
no vulnerable packages), backtick quoting with MySQL's 64-char / no-trailing-
space rules, information_schema catalog SQL with the database as the schema
(system databases excluded, aliases pinned), trailing LIMIT, a type map that
also reads the reader-side spellings (INT UNSIGNED -> UInt32, BOOL -> Boolean).
- SqlDialects: the one provider -> dialect map. The factory, the Test Connect
probe and the browse picker each carried their own switch (and drifted once);
they now all read this, and their refusals list the supported/reserved sets
from it.
- AdminUI driver form offers MySql.
- Tests: MySqlDialectTests (golden quoting/catalog/type map/planner SQL); the
provider-gate tests in factory, probe, browser and form now treat MySql as
supported and Odbc as the reserved control. MySqlReadTests + Docker/mysql
(throwaway mysql:8.4, SELECT-only login): liveness, KV/wide-row/topByTimestamp,
INT keys vs authored text, case folding against case-sensitive table names,
catalog rejection, browse walk, factory+probe, named query with a bound
parameter, and a writing stored function whose effect the rolled-back
transaction discards.
(lmxopcua#753)
Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
SqlTagModel.Query (design §5.4) is now served end to end instead of refused.
The driver config declares `queries` (name -> sql, params, maxRows); a tag names
the query, the result `column` it reads and optionally a `rowKey` picking its row.
- Parser/deploy gate: query + column required, rowKey all-or-nothing; Inspect
explains each refusal. Table is not read for this model.
- SqlReadOnlyStatement: single SELECT/WITH only, no data/schema-changing keyword
outside literals/comments (refuses SELECT INTO, FOR UPDATE, DML CTEs). Shared by
the factory and the AdminUI driver form (SqlNamedQueryRules.Describe), so the
form refuses what the driver refuses. One bad entry fails the driver.
- Parameters are always bound; the factory refuses a declared parameter whose
dialect marker the statement never uses.
- Reader: one execution per query per poll, inside a transaction that is always
rolled back; result columns matched exact-then-unique-CI; no row -> BadNoData,
ambiguous row / missing column / over maxRows -> BadConfigurationError.
- Catalog gate passes Query tags through (nothing is quoted into SQL); a
Query-only driver skips the catalog load. A tag naming an undeclared query is
skipped with the reason on /hosts.
- AdminUI: typed editor offers Query (query/column/rowKey); driver form edits
`queries` as JSON and keeps the last valid set on a typo.
(lmxopcua#753)
Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
Design only. Documents why today's in-process ScriptRunGuard can abandon but
not stop a runaway script (file:line evidence), and proposes a sharded
worker pool with an isolation worker under a ScriptRunnerActor, a
named-pipe + protobuf frame protocol carrying full input snapshots,
soft/hard deadlines with process kill, kill attribution and quarantine,
worker sandboxing (scrubbed env, pinned assemblies, Job Object, heap
limit), an HMAC'd content-addressed PE cache, performance budgets,
failure-quality semantics, a flag-gated InProcess/Shadow/OutOfProcess
rollout, a test plan, open decisions and an assumptions register.
Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
v2-integration.yml runs the Sql, Historian.SqlServer, Historian.QuestDb and OpcUaClient
integration projects against throwaway SQL Server/PostgreSQL/QuestDB/opc-plc containers
started by scripts/ci/integration-fixtures.sh, with a MIN_EXECUTED=440 floor. v2-e2e.yml is
manual/weekly, probes the shared fleet and skips cleanly when unreachable. v2-ci.yml is
functionally unchanged. Unverified on a hosted runner until first run.
Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
Item 3 of #765. A keyed (API-key) session whose key had no Galaxy ack identity
got Good and then a revert, because whether the key has an identity lives in
the Secrets store, which must not be read under the SDK lock.
GalaxyAckIdentityDirectory (Driver.Galaxy) keeps a node-wide set of the key ids
that have a galaxy-ack-user secret. It lists this node's Secrets store
(metadata only, nothing decrypted) and is read lock-free.
GalaxyDriver.GetAcknowledgeRefusal (IAlarmAcknowledgePreflight, published
through NativeAlarmAckPreflightView as in #760) now refuses a keyed session
whose key the directory reports Absent, when the driver has no fallback ack
account. A refusal also asks for a refresh.
Fails open: before the first refresh, once the snapshot is older than
StaleAfter, and from a signalled identity-secret change until the next
refresh, the lookup answers Unknown and the ack takes the old Good-then-revert
path.
Refresh: GalaxyAckIdentityRefreshService runs on driver nodes, every
Drivers:Galaxy:AckIdentityCache:RefreshIntervalSeconds (default 30;
StaleAfterSeconds 0 = three intervals; validated at start). It also refreshes
early on secret changes. The Secrets library has no change event, so
SecretsRegistration decorates ISecretCacheInvalidator, which is called by
AdminUI /api-keys, /admin/secrets and the replication appliers. API-key
changes need no event: a revoked key is already denied before the pre-check.
Tests: Galaxy.Tests GalaxyAckIdentityDirectoryTests (16) and Host.Tests
GalaxyAckIdentityCacheWiringTests (9). Seven separate mutations each turn at
least one test red: pre-check ignores the directory, factory drops it, no
invalidator decoration, no staleness check, no change-pending check, no
refresh on refusal, loop never wakes early.
Docs: Galaxy.md, AlarmTracking.md, Configuration.md, .claude/rules/alarms.md.
Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
Run 959 (job 3552) was silent from 13:39 for 3h12m and was failed by the server
reaper with no cause. Pre-#707 runs 935/938/941/942/947/959 all ended at 189-193 min;
no run since 76bdca90 exceeds 82 min (healthy ~53). Adds timeout-minutes 120 and
blame-hang (20m, no dump) to the whole-solution test step.
Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
Item 1 of #776 (the B2 over-the-wire ask): Client.CLI had no way to issue a
HistoryReadEvents. `historyread --events` now sends ReadEventDetails to an
event notifier and prints one line per event in select-clause order.
--fields picks the select clauses (default EventId, EventType, Time,
SourceName, Severity, Message, ConditionName (on ConditionType), SourceNode);
--of-type adds an OfType where-clause. --events is exclusive with --modified /
--aggregate, and --fields / --of-type require it.
Client.Shared gains IOpcUaClientService/ISessionAdapter.HistoryReadEventsAsync
(default-implemented as NotSupported for other implementations) backed by the
real session adapter through HistoryReadPaging, so continuation points are
followed up to --max and a Bad node status is surfaced, not printed as empty.
Tests: CLI tests over the fake service (default filter, --fields/--of-type
filter, option conflicts); an over-the-wire test drives the shared client's
event read against the in-process server — the default fields come back
non-null, OfType filters, an invalid where-clause is BadEventFilterInvalid.
The other #776 items are decisions and are left open.
Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
QuestDB modified-history reads dropped every still-pending trail row that was
not the latest at its source time, so V0->V1 (applied marker never written)
followed by V1->V2 hid the V0->V1 change for good. A pending row below the
latest is now served when the NEXT trail row at that time recorded a different
prior value (kind, value, status or the prior's server_ts) - the next change saw
what this one wrote; an orphan of a change that never landed leaves the same
prior for the next row and stays hidden. QuestDB window functions reject
VARCHAR, so the next row's prior comes from a LEFT JOIN on (ts, lead(id)).
The updater's comment and marker-failure log no longer claim modified reads
"still serve it" unconditionally.
Regression tests (QuestDB 10.0.1 throwaway container): marker lost then a
replace / delete at the same time; an orphan followed by a marker-lost retry
then a later change (orphan still dropped); the random marker-mix fuzz now
varies the recorded priors. All four fail on the old filter.
NITs:
- SqlServerHistorianValueWriter: a SqlException raised while the token is
cancelled (SqlClient's "Operation cancelled by user") is rethrown as
OperationCanceledException instead of booked as a failed write. Integration
test holds a TABLOCKX on Sample and cancels mid-INSERT; it returned false before.
- SqlServerHistorianDataUpdater.DeleteRawBatches is an Interlocked counter.
- HistoryReadEvents: an event filter whose SDK validation throws (unknown
operator) failed with the generic BadUnexpectedError; the node manager now
re-throws any failed event-filter validation as BadEventFilterInvalid.
Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
Three write bounds were configured independently: the inbound-write
gateway's 10 s Ask, DriverHostActor's fixed 8 s Ask of the child, and the
child's own write timeout (5 s + up to 2 s grace) with a fixed 8 s queue
budget. An outer Ask that fires first reports the write failed and the node
reverts while the child is still driving the device, so any write timeout
above ~6 s (or a write dequeued late in its 8 s budget) let the device take
a value the client was told failed.
DriverWriteBounds (Runtime) derives them all from DriverWrites:Timeout:
token cancel = Timeout
abandon = Timeout + min(Timeout/2, 2 s) (RunBound)
queue budget = RunBound
child replies <= queue budget + RunBound (ReplyBound)
host Ask = ReplyBound + 1 s
gateway Ask = host Ask + 1 s
Default Timeout is 10 s (the Tier C Write timeout, so no tier default is
capped below its own policy); host Ask 25 s, gateway Ask 26 s. Tier A/B
writes are still cancelled first by their 2 s / 4 s tier timeouts.
- DriverHostActor takes writeBounds + writeClock, spawns children with them,
and runs its Ask deadline as a token on the write clock (Infinite Akka
timeout), so host and child bounds share one clock.
- WriteAttribute gains IssuedAt (write-clock timestamp, stamped by the
host): the queue budget counts mailbox time too, and a write past it is
failed BadTimeout and never started - including on arrival. Stale queued
writes are expired across the whole queue, not only up to a free slot.
- ActorNodeWriteGateway's default Ask = Default.GatewayAsk; the hosted
service passes the configured one.
- Startup validation (Host DriverWriteBoundsBinding, same warn-and-clamp
style as ScriptingOptionsBinding): unparsable / non-positive -> default,
above 5 min -> clamped, and each tier whose default Write timeout exceeds
the write timeout is named in a Warning (its policy would never take
effect). The derived bounds are logged at Information. Per-instance
ResilienceConfig Write overrides arrive with a deployment, so
DriverCapabilityInvokerFactory warns at spawn (writeTimeoutCeiling).
- appsettings.json documents DriverWrites:Timeout.
Tests (manual clock):
- DriverHostActorWriteAskTests: a held write under a 9 s timeout survives
an 8.5 s step and succeeds; a token-ignoring write is answered by the
child's BadTimeout at 9 s + grace, not by the Ask; the Infinite-timeout
Ask ends exactly at its clock deadline.
- DriverInstanceActorConcurrentWriteTests: a write issued longer ago than
the queue budget is failed and never reaches the driver; positive control
at exactly the budget is written.
- DriverWriteBoundsTests: every outer bound covers the inner ones for 0.5 s
to 300 s timeouts.
- DriverWriteBoundsBindingTests: default fits every tier; 3 s timeout is
caught at startup with Tier B and C named (not A); bad values fall back
with a warning; over-max clamps.
- DriverCapabilityInvokerFactoryTests: spawn warns when a Write override
exceeds the ceiling; no warning within it or without a ceiling.
Mutation checks: a fixed 8 s host Ask on the same clock fails both 9 s host
tests; stamping arrival time instead of IssuedAt fails the issued-at test;
dropping the tier check / factory check fails their tests.
Runtime.Tests 961 passed / 3 skipped at AKKA_TEST_TIMEFACTOR 3 and 1;
Host.Tests 232/232 at 3 and 1; Core.Tests 361 passed / 1 skipped.
Solution Release build: 0 warnings.
ContinuousHistorizationRecorderTypedValueTests (historian) deliberately
left for the historian lane.
Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
otopcua.historization.recorder.pending_appends was written only from the
actor thread: when a burst was accepted (just after the increment, so
never 0) and on GetStatus, which nothing in production sends. Once ingest
stopped the gauge froze at the depth the last burst saw, so after a
backpressure episode it kept reporting a queue near the 64-burst cap that
had long drained.
The append loop now publishes the depth after every decrement
(ReportPendingAppends). The mailbox-depth gauge stays actor-thread only;
its description now says when it is sampled. New test (non-parallel
collection, the gauge is process-wide): the gauge reads 2 with one append
stuck and one queued, and 0 after the queue drains with no status
request. With the loop's report removed it fails (stays 2).
Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
The recorder now appends off its mailbox (9fda2919), so two separate Tells
are two bursts the append loop may commit and drain one at a time. The
#538 tests told Good.Tag and Bad.Tag separately and relied on both landing
in one drain pass; when the loop drained Good.Tag alone, Bad.Tag was left
alone at the head, every later pass failed with no sibling success (the
outage path), and nothing was ever abandoned. Run in isolation the
recorder test class failed 4 of 4 times (QueuedDepth 1, expected 0).
The five tests now send the pair as one DependencyValueBatch, and the
in-memory outbox appends a batch under one lock so a concurrent peek sees
it all-or-nothing, as a group commit is. 6 of 6 isolated runs green.
Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
HistorianBackendRegistryTests (#682) still expected the Gateway data source
to advertise the two AVEVA-native aggregates. Lane H2 (R08:2.1) refuses
them by default until historiangw#27, and the lane branch never updated
this Host test, so it failed after the rebase. The default case now expects
the ten raw-evaluated aggregates, and a new case pins that
ServerHistorian:Gateway:EnableNativeTimeWeighted=true, resolved through the
Host's registry, brings TimeAverage and Total back.
Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
The batched-alarm-history test came from the lane-H2 branch, which predates
the xUnit1051 sweep on master (lmxopcua#725). Two no-token calls broke the
Release build under TreatWarningsAsErrors after the rebase.
Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
Both off-mailbox loops read _lifetime.Token per operation. PostStop disposes
that source after the flush budget, and a disposed source's Token throws, so
an operation started past the budget would have logged a spurious failure.
Each loop now captures the token when it starts. Also rewrites the
recorder's stale "capture Self before the await" note for the enqueue path.
Rebased from the unpushed remed/lane-H2 branch (originally authored 2026-10-22
in session_01Ub9hGmYLQvn9XETzw7CMsT) onto current master.
Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
HistoryReadEvents projected only EventId / SourceName / Time / ReceiveTime /
Message / Severity: a select clause asking for EventType, ConditionName or
SourceNode (what most generic alarm viewers ask for) got nulls, and the
event filter's where-clause (an OfType) was ignored.
- HistoricalEvent gains optional EventTypeName, ConditionName and AlarmId.
SqlServer, QuestDB and Gateway event reads fill them from what their
alarm writers already store (AlarmTypeName / AlarmName / AlarmId; the
gateway event Type and its AlarmName / AlarmId properties).
- The node manager maps the type name to its Part 9 event type (unknown or
non-event => AlarmConditionType), projects ConditionName, and resolves
SourceNode to the materialised condition the alarm id names (the live
events' SourceNode), else null.
- The where-clause is evaluated per event with the SDK content filter
through an IFilterTarget (OfType via the type tree; other operators
against the projected fields), after the page's cursor is taken, so
paging is unchanged.
Tested in the node manager, over a real client session (the Client.CLI has
no `historyread --events` mode yet; filed as a follow-up), and by alarm
writer -> event read round trips on SqlServer and QuestDB.
Rebased from the unpushed remed/lane-H2 branch (originally authored 2026-10-22
in session_01Ub9hGmYLQvn9XETzw7CMsT) onto current master.
Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
The Gateway backend's gaps against SqlServer/QuestDb were documented
piecemeal. The backend-selection table in docs/Historian.md now also covers
the aggregate set (and the native time-weighted opt-in), emulated reverse
and end-only reads, the client-side events cap, retention and the health
probe.
Continuous historization on Gateway silently loses sample quality (stored
Good), comms-loss outage markers and non-numeric values. Building its live
writer, which only happens when ContinuousHistorization is enabled, now
logs one startup Warning that names the gaps and the alternatives.
The review asked for the matrix in the AdminUI backend picker, but no such
picker exists: the backend is configuration-only. Filed as a follow-up.
Rebased from the unpushed remed/lane-H2 branch (originally authored 2026-10-22
in session_01Ub9hGmYLQvn9XETzw7CMsT) onto current master.
Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
HistorianAdapterActor fired `_ = EnqueueAsync(...)` per alarm transition
with no in-flight cap, so an alarm storm (a native source re-raising
hundreds of conditions on reconnect) spawned hundreds of concurrent SQLite
inserts that serialised on the file lock and held pool threads. It also kept
a Receive<AlarmHistorianEvent> with no production sender: a transition told
directly AND published on `alerts` would have been historized twice.
Transitions now go to a bounded channel (10000) drained by one loop, so the
sink sees one enqueue at a time in arrival order; a full queue holds the
mailbox (backpressure, nothing dropped); PostStop flushes the queue within
the shared 5 s stop budget. The direct receive is deleted. The gate tests
drive transitions instead; a directly-told AlarmHistorianEvent is shown to
be unhandled against a positive control.
Rebased from the unpushed remed/lane-H2 branch (originally authored 2026-10-22
in session_01Ub9hGmYLQvn9XETzw7CMsT) onto current master.
Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
The continuous-historization recorder awaited every outbox append (an fsync
under PerEntry) on its mailbox, so ingestion was capped at 1/fsync_latency
bursts per second while the mux kept telling into an unbounded, unmetered
mailbox. GetStatus also suspended the mailbox on an unbounded outbox count.
- Gated ingest bursts go to a bounded channel (64 bursts) drained by one
ordered append loop, which commits every burst waiting behind a slow
append in a single AppendBatchAsync (FIFO kept). Only a full queue holds
the mailbox (backpressure, bounded memory). Counters are Interlocked; the
loop nudges the drain. PostStop flushes queued bursts within the shared
5 s stop budget.
- GetStatus reads counters on the actor thread and pipes a count bounded by
a 10 s timeout (-1 when it does not answer); it no longer suspends the
mailbox. RecorderStatus gains PendingAppends, MailboxDepth and
AppendBackpressureWaits; two gauges publish pending appends and mailbox
depth.
- ContinuousHistorization:CommitMode defaults to Periodic (100 ms). The
Host's appsettings.json, its unparseable-value fallback and docker-dev
still say PerEntry: filed as a Host follow-up (outside this lane).
Rebased from the unpushed remed/lane-H2 branch (originally authored 2026-10-22
in session_01Ub9hGmYLQvn9XETzw7CMsT) onto current master.
Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
FasterLogHistorizationOutbox.PeekBatchAsync held the state lock across the
FasterLog scan. Once a backlog's head ages out of FasterLog's in-memory
window (any multi-minute historian outage), GetNext reads pages from disk
synchronously, and the recorder's appends (which index under the same lock)
blocked on an Akka dispatcher thread once per drain pass, exactly during
outage recovery.
The peek now snapshots the head under the lock, scans unlocked, and re-takes
the lock only when it met undecodable records, to reconcile them with the
index in O(records scanned) and drop one still at the head (re-checked,
since an ack or drop-oldest may have moved it). A test seam pauses the scan
and proves an append completes meanwhile; wrapping the scan in the lock
again fails it.
Rebased from the unpushed remed/lane-H2 branch (originally authored 2026-10-22
in session_01Ub9hGmYLQvn9XETzw7CMsT) onto current master.
Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
The applied-marker reconciliation changed the modified read's query shape (a
window function over each source time, with the resume position applied
outside it), which is a paging change. A random trail (1-4 modifications per
time; applied, orphaned and landed-without-marker rows; times with no
sample) is paged at caps 1..50 in both directions through the cursor and
must concatenate to the unpaged read, which must equal an independent oracle
of the marker rule. Falsified: dropping the latest-row rule fails it.
Rebased from the unpushed remed/lane-H2 branch (originally authored 2026-10-22
in session_01Ub9hGmYLQvn9XETzw7CMsT) onto current master.
Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
QuestDB has no multi-table transaction, so a HistoryUpdate writes its trail
row before the sample change. A failed or crashed change left an orphan
trail row that modified reads served as a change that happened, and a trail
INSERT re-sent after a client timeout duplicated (the table had no key).
- sample_modified gains `pending` and DEDUP UPSERT KEYS(ts, tag,
modified_id). The trail row is written pending; after the change lands
the same row is re-upserted pending = false (the applied marker).
- A transient trail INSERT failure is re-sent once with the same ids; a
first attempt that did apply converges on the key.
- ReadModifiedAsync serves a pending row only when it is the latest trail
row at its time and the sample there was written at or after the
modification (a crash between change and marker). The orphan of a change
that never landed, typically followed by the client's retry, is dropped.
ASOF JOIN over the window's samples of one tag; the resume filter sits
outside the window function so it never cuts a time's partition.
- The bootstrapper migrates an older trail (ADD COLUMN pending, DEDUP
ENABLE) and waits for it to apply (bounded 10 s); old rows read
pending = false and stay visible. Until applied, or with
AutoCreateSchema=false, the old format is used.
Rebased from the unpushed remed/lane-H2 branch (originally authored 2026-10-22
in session_01Ub9hGmYLQvn9XETzw7CMsT) onto current master.
Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
DeleteRaw deleted an unbounded range in one statement and transaction with
OUTPUT INTO the modification trail. Over years of a 1 Hz tag that is tens of
millions of rows deleted and audited at once: lock escalation on Sample
(blocking the live writer), transaction-log growth, and a 30 s
CommandTimeout that rolled the whole thing back.
It now runs DELETE TOP (4000) batches, each its own transaction with its
trail rows in the same statement (every deleted row is still audited),
cancellable between batches. A failure part-way leaves the committed batches
deleted and reports the failure; re-issuing completes it (documented, same
as QuestDB's paged tombstoning). The existing atomicity tests still hold per
batch.
Rebased from the unpushed remed/lane-H2 branch (originally authored 2026-10-22
in session_01Ub9hGmYLQvn9XETzw7CMsT) onto current master.
Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
Event reads filtered on source and every modified-value read filtered on tag
scanned that symbol column in every partition of the window; only
sample.tag carried an index (#683).
Both columns are now SYMBOL INDEX in the DDL, and the bootstrapper's index
migration is generalised over QuestDbTables.SymbolIndexes (sample.tag,
event.source, sample_modified.tag): a missing index is added with the same
idempotent ALTER ... ADD INDEX, Warning-only when AutoCreateSchema is false
or the ALTER is refused. Integration tests: new tables carry all three; a
pre-index table set gains them on boot and event reads answer the same.
This reverses #683's documented choice to leave the two unindexed;
docs/Historian.md carries the reasoning.
Rebased from the unpushed remed/lane-H2 branch (originally authored 2026-10-22
in session_01Ub9hGmYLQvn9XETzw7CMsT) onto current master.
Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
The SqlServer purge read every TagId and issued at least one DELETE TOP per
table per tag on every pass, even when nothing was eligible: 50000 tags made
100000 statements per hourly purge, serialised on one connection under the
applock.
One pre-filter query per table (EXISTS seek on the clustered key) now picks
the tags holding a row older than the cutoff; only those get deletes. A pass
with nothing eligible sends just the Event delete. The mid-command
cancellation test now accepts the pre-filter SELECT as the blocked command,
since it is the first statement to touch Sample.
Rebased from the unpushed remed/lane-H2 branch (originally authored 2026-10-22
in session_01Ub9hGmYLQvn9XETzw7CMsT) onto current master.
Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
All three alarm-history writers inserted one event per round trip, serially
(SqlServer also opened a pooled connection per event), so a 10000-event
alarm storm drained at 10000 x RTT while the backlog sat in LocalDb.
- Sql.Common.AlarmBatchWrite: one statement per chunk; a chunk that fails
PermanentFail-class is re-sent event by event so only the poison event
dead-letters; a transient failure retries the chunk; a cancellation never
dead-letters.
- SqlServer: multi-row insert-if-absent over a VALUES set (150 events per
statement, under the 2100-parameter limit), duplicate ids in a batch
written once; a duplicate-key race isolates per event, where it is Ack.
No new table type, so no schema migration (the review suggested a TVP;
the VALUES form gets the same one-round-trip-per-chunk without one).
- QuestDb: multi-row INSERTs of 500 events; pre-1970 events still
dead-letter unsent.
- Gateway: no batch RPC exists, so sends are pipelined, at most 8 in
flight, outcomes kept in input order.
Rebased from the unpushed remed/lane-H2 branch (originally authored 2026-10-22
in session_01Ub9hGmYLQvn9XETzw7CMsT) onto current master.
Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
The QuestDB live writer waited for WAL visibility after every tag group, and
the wait polled wal_tables() every 10 ms. With DrainConcurrency = 8 that was
up to eight pollers at ~100 queries/s each against a database whose WAL
apply was already behind.
New Runtime seam IHistorianPassWriter (WriteLiveValuesInPassAsync +
FlushAsync). The recorder writes every group of a pass through it and calls
FlushAsync once after all groups report, before the ack, whenever any group
wrote (an un-acked pass is redelivered and its insert-if-absent check must
see this pass's rows). A flush failure does not fail the pass. The QuestDB
writer implements it: in-pass writes only mark a flush as owed; FlushAsync
waits once. The poll now backs off 10 -> 20 -> ... -> 200 ms, never past the
timeout. Direct WriteLiveValuesAsync callers still wait per call.
Also updates the QuestDB cancelled-write integration test to the rethrow
contract from the previous commit.
Rebased from the unpushed remed/lane-H2 branch (originally authored 2026-10-22
in session_01Ub9hGmYLQvn9XETzw7CMsT) onto current master.
Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
Every IHistorianValueWriter caught Exception and returned false, including
OperationCanceledException from the drain's own token. The recorder's guard
("a cancelled pass is not a tag failure") was therefore dead code: at shutdown
mid-pass each cancelled group could be booked as a consecutive failure and,
with a sibling that already wrote, tip a healthy tag toward quarantine.
SqlServer, QuestDb and Gateway writers now rethrow a cancellation of the
caller's token; a cancellation-shaped error on a live token is still an
ordinary failed write.
The Gateway writer relabelled a Local-kind timestamp as UTC instead of
converting it, so the same value could land at two different times on two
backends. Stamped samples now go through LiveWritePreparer (Local -> UTC,
non-finite doubles not sent, duplicate timestamps keep the first); unstamped
samples are still left for the gateway to stamp, in batch order.
Rebased from the unpushed remed/lane-H2 branch (originally authored 2026-10-22
in session_01Ub9hGmYLQvn9XETzw7CMsT) onto current master.
Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
Joseph Doherty