Central pin 1.1.0 -> 3.2.2; AdminUI/ControlPlane/Runtime/Host.Tests no longer carry VersionOverride.
The three driver IntegrationTests fixtures' Fact/Theory attributes gain the source-info ctor (xUnit3003).
No xUnit1051 fallout remained (already cleaned). Test counts unchanged.
Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
- IScriptRunner.RunBatch: one EvalBatch frame per worker for every evaluation ready to run; per-item, streamed results.
The default (InProcessScriptRunner) is RunAsync per item. Used by the Calculation pump (dequeue all, split at a
repeated tag, publish in dequeue order), the VirtualTag seam (ScriptBatchCoalescer, no timer) and the scripted-alarm
engine (whole dirty set). In process the pump and the engine stay one at a time, unchanged.
- Exact per-item deadlines: the worker hands work to an idle evaluation thread or defers it (EvalResults
deferred_eval_ids); the host stops that run's clock and restarts T then G when it starts (started_eval_ids). A run
queued behind a runaway is neither timed out nor struck and is a victim, not the culprit, at the kill.
- Scripted-alarm seam behind Scripting:Runner:Seams; unset Seams now means VirtualTag + Calculation, so alarms run out of
process only when listed. Load still compiles each predicate (to PE bytes only) so a broken one fails the load; passes
batch and apply in the serial order under the gate. Script faults freeze the condition BadInternalError; runner faults
freeze it BadResourceUnavailable and re-evaluate on recovery; a dead worker never clears an alarm. IScriptRunnerProvider
carries the runner to DriverHostActor.
- Perf harness: batch throughput (RunBatch vs RunAsync x n) and alarm-pass latency (in process / batched / serial);
design §12.3 records macOS + windev. On windev the serial out-of-process alarm pass misses 250 ms from ~500 alarms;
batched p99 <= 53 ms at 1,000.
Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
- NodeManagerLockTimeBenchmark: wall-clock budget enforced only with OTOPCUA_BENCH=1; default run keeps the state-correctness asserts and logs timings.
- OpcUaClientDriverWire host-status: signal (semaphore) wait that only accepts a non-Running state observed after the server is disposed.
- AbCip PollLatencyComparison: retry once on a fixture connection reset; p95 criterion enforced only with AB_LAT_ENFORCE=1.
- ScriptRunGuardTests: orphan scripts held by a ctx.Released signal instead of a 250 ms wall-clock spin that could finish inside the 60 ms budget under load; stuck-orphan wall-clock rule disabled in the tests that pin the strike counting.
- DependencyMuxReplayTests: split the inside-grace positive control (1 h grace) from the release test (control asserted before unregister), removing the 200 ms grace race.
Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
Unsubscribe paths now delete through the existing DeleteSubscriptionAsync seam so a test can hold the delete open >grace. Mutation: dropping the post-delete restamp fails both slow-delete tests.
Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
Second P1 slice of docs/plans/2026-11-09-out-of-process-script-runner-design.md; still opt-in behind
Scripting:Runner=OutOfProcess (InProcess unchanged).
- Compiled-PE cache (design §7, §14 #8): Scripting:Worker:PeCacheDir holds <dir>/pe/<id>.pe entries authenticated
with HMAC-SHA256 under a per-node random key (<dir>/pe-cache.key, created by the host, 0600/0700 on Unix), handed to
the worker in Hello. Entry id = SHA-256(compile fingerprint + kind + source); the fingerprint covers the Roslyn
version, the Core.Scripting build (analyzer, wrapper, stack guard), the sandbox reference set and the runtime. Edited,
truncated, other-key or renamed entries are rejected, deleted and recompiled; the worker never loads unverified bytes.
Pruned by recency at startup (30 d / 50 000). ScriptEvaluator splits into CompileToImage + FromImage. Measured on
40 scripts, restart to every first result: macOS 820 ms no cache / 840 ms empty / 85 ms warm;
windev (Windows) 9.5 s / 9.4 s / 1.0 s.
- Quarantine race: WorkerConnection runs exit attribution synchronously BEFORE completing the evaluations/loads it held
(and before Exited), and the shard passes it straight to the runner — a kill strike is in place before the killed
key's slot is released.
- Supervision: a dead worker restarts by itself after its backoff; Unavailable outcomes carry ScriptRunOutcome.Recovery
(-> VirtualTagEvalResult.RetryWhen) and the VT actor / Calculation driver re-evaluate once against current inputs when
it completes. Isolation worker stops after Scripting:Worker:IsolationIdleTimeout (10 min). Keys compiling when a
worker dies are suspects too (isolation finds a poison compile and quarantines it). Startup orphan sweep by launch
marker: kills only a process whose marker's host is gone and whose pid + start time + image match.
- Crash-loop circuit (Scripting:Worker:CircuitOpenAfter, 2 min): runner Unavailable, one Error, evaluations answered
at once while shards keep retrying at the cap.
- Observability (§10): otopcua.script.runner.state, worker.up/inflight/queue_depth/heap_bytes (shard),
ipc.roundtrip + eval.duration (kind), compile{cache}, isolation.suspects, exits reason=idle. `script-runner` health
check on the ready tier, at worst Degraded (§14 #12: degraded but still ready).
Left on #780: self-contained worker publish, installer (account, pipe ACL, firewall), docker-dev live drill,
scripted-alarm seam, the §4.5 per-seam batch submit, the §8 perf harness.
Tests: SupervisionTests, PeCacheStoreTests/PeCacheWorkerTests (real workers), ScriptRunnerHealthCheckTests, VT + Calc
recovery seam tests; 14 mutations each turned their test red.
Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
Item 2: a tag's optional `write` block picks the statement the driver builds from its own
catalog-validated identifiers (no authored SQL; values always bound):
- Update (default, unchanged back-compat UPDATE ... WHERE key; 0 rows = BadNoData)
- Insert (key+value[+stamp]; append for a WideRow topByTimestamp tag)
- Upsert: SQL Server MERGE WITH (HOLDLOCK), PostgreSQL INSERT ... ON CONFLICT, MySQL
ON DUPLICATE KEY UPDATE (1-or-2 rows), Oracle MERGE from DUAL
- Procedure: named stored procedure resolved against the routine catalog by the catalog gate
(INFORMATION_SCHEMA.ROUTINES / ALL_PROCEDURES), value/key/timestamp params by name (MySQL positional)
- stampTimestamp sets the tag's timestamp column to the write time (UTC)
Odbc serves Update/Insert only (shared SqlWriteSupport table). Constraint failures map to
BadEntryExists (unique) / BadOutOfRange (other integrity, data exception); everything else,
permission denied included, stays BadCommunicationError. WriteIdempotent stays false.
The parser refuses malformed blocks (strict keys/enums/param charset); DraftValidator emits
blocking SqlWriteBlockInvalid / SqlWriteModeUnsupported from the driver's own parse; the AdminUI
editor surfaces the same message.
Items 1 + 4 (docs/drivers/Sql.md): ODBC stays the ANSI SQL:2008 profile (native providers for
SQL Server/MySQL, DB2 unsupported) with an ops note on driver manager + vendor driver on every
Odbc and AdminUI host; read-only drivers require a SELECT-only login, writable drivers a login
limited to their write targets.
Tests: SqlWriteModeTests (parse, every dialect's SQL, gate, status map, SQLite end-to-end),
DraftValidatorSqlWriteBlockTests, AdminUI round-trip; live SqlWriteModesLiveTests on SQL Server,
PostgreSQL, MySQL, Oracle (seeds gain write_targets + set_target) and an Odbc write leg.
Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
The simulator now packs Int32/Boolean/Float/Double/String arrays per the
Sparkplug B spec (independent of the driver codec) and seeds Profile, Flags
and Curve in the standalone catalog. SparkplugLiveTests gains a live test that
asserts typed int[]/bool[]/float[] values from the birth, a changed vector
(including a different Boolean length) from alias-only DATA, the read path,
and BadTypeMismatch for an array metric bound without isArray. Run against a
throwaway local Mosquitto; mutation-checked (scalar binding / seed removed).
Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
allowWrites is no longer inert: SqlDriver implements IWritable, default off.
- Gates (outermost first): node Writable + WriteOperate (server), driver
allowWrites, the tag's own TagConfig "writable": true (strict JSON boolean;
anything else refuses the tag, and the deploy gate says why), and a model that
names exactly one row — KeyValue, or WideRow by where-pair. topByTimestamp and
Query are BadNotWritable.
- SqlWritePlanner: the only statement a write can become is
UPDATE <table> SET <value col> = @v WHERE <key col> = @k, built from the
catalog-gate-canonical identifiers (dialect-quoted) with value and key bound;
value-before-key marker order so positional ODBC binds correctly. No authored
write SQL, INSERT, UPSERT or procedure.
- Executed in a transaction that commits only when exactly one row was touched:
0 rows -> BadNoData (never inserts), >1 -> rolled back, BadConfigurationError,
unknown count -> rolled back, BadNotSupported. Bounded by operationTimeout;
provider errors -> BadCommunicationError; health untouched.
- Discovery advertises Operate only where a write could succeed.
- AdminUI: driver form's Allow writes switch is live; tag editor gains a
per-tag write switch; model validates writable strictly (parity corpus).
- Factory no longer warns "allowWrites ignored"; it logs whether writes are on.
- Tests: SqlWriteTests (planner SQL/bind order/injection/coercion, every gate,
readback, absent key, two-row rollback, gate-rejected tag, discovery);
MySqlReadTests write legs against InnoDB (keyed row only, two-row rollback,
server-side grant refusal, default-off). Seed adds a setpoints table and the
one UPDATE grant.
(lmxopcua#753)
Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
SqlProvider.Odbc is now constructed; every enum member is a working backend.
- OdbcDialect: System.Data.Odbc 10.0.14 (no vulnerable packages), ANSI double-
quote quoting, positional `?` markers, INFORMATION_SCHEMA catalog (system
schemas of the common backends excluded by name — the live gate caught
psqlODBC listing pg_catalog), FETCH FIRST n ROWS ONLY limits, a conservative
1000-parameter chunk. Positional binding is safe because every caller adds
parameters in marker order (planner IN list, catalog schema-then-table).
- Named queries on a positional provider are checked by `?` count against the
declared params (literals/comments ignored), since there is no name to match.
- Supported backends are those whose SQL accepts that profile (PostgreSQL via
psqlODBC proven, Snowflake-like); SQL Server/MySQL/DB2 should use their native
provider. The host needs the ODBC driver manager + vendor driver.
- Reserved-provider controls in factory/probe/browser tests now use an
undeclared ordinal (99); the AdminUI form offers Odbc and a test pins the
dropdown theory to SqlDialects.Supported.
- Tests: OdbcDialectTests (unit, any host). OdbcReadTests + Docker/odbc/
run-odbc-gate.sh run the built suite in a throwaway Linux SDK container with
unixODBC + psqlODBC against a throwaway seeded PostgreSQL: KV IN-list
positional binding, where-pair, topByTimestamp via FETCH FIRST, a two-
parameter named query, and the browse walk. macOS has no vendor ODBC driver,
so there the suite skips.
(lmxopcua#753)
Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
SqlProvider.MySql is now a constructed backend instead of a reserved name.
- MySqlDialect (Driver.Sql.Dialect): MySqlConnector 2.6.2 (MIT, fully managed,
no vulnerable packages), backtick quoting with MySQL's 64-char / no-trailing-
space rules, information_schema catalog SQL with the database as the schema
(system databases excluded, aliases pinned), trailing LIMIT, a type map that
also reads the reader-side spellings (INT UNSIGNED -> UInt32, BOOL -> Boolean).
- SqlDialects: the one provider -> dialect map. The factory, the Test Connect
probe and the browse picker each carried their own switch (and drifted once);
they now all read this, and their refusals list the supported/reserved sets
from it.
- AdminUI driver form offers MySql.
- Tests: MySqlDialectTests (golden quoting/catalog/type map/planner SQL); the
provider-gate tests in factory, probe, browser and form now treat MySql as
supported and Odbc as the reserved control. MySqlReadTests + Docker/mysql
(throwaway mysql:8.4, SELECT-only login): liveness, KV/wide-row/topByTimestamp,
INT keys vs authored text, case folding against case-sensitive table names,
catalog rejection, browse walk, factory+probe, named query with a bound
parameter, and a writing stored function whose effect the rolled-back
transaction discards.
(lmxopcua#753)
Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
SqlTagModel.Query (design §5.4) is now served end to end instead of refused.
The driver config declares `queries` (name -> sql, params, maxRows); a tag names
the query, the result `column` it reads and optionally a `rowKey` picking its row.
- Parser/deploy gate: query + column required, rowKey all-or-nothing; Inspect
explains each refusal. Table is not read for this model.
- SqlReadOnlyStatement: single SELECT/WITH only, no data/schema-changing keyword
outside literals/comments (refuses SELECT INTO, FOR UPDATE, DML CTEs). Shared by
the factory and the AdminUI driver form (SqlNamedQueryRules.Describe), so the
form refuses what the driver refuses. One bad entry fails the driver.
- Parameters are always bound; the factory refuses a declared parameter whose
dialect marker the statement never uses.
- Reader: one execution per query per poll, inside a transaction that is always
rolled back; result columns matched exact-then-unique-CI; no row -> BadNoData,
ambiguous row / missing column / over maxRows -> BadConfigurationError.
- Catalog gate passes Query tags through (nothing is quoted into SQL); a
Query-only driver skips the catalog load. A tag naming an undeclared query is
skipped with the reason on /hosts.
- AdminUI: typed editor offers Query (query/column/rowKey); driver form edits
`queries` as JSON and keeps the last valid set on a typo.
(lmxopcua#753)
Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
Item 3 of #765. A keyed (API-key) session whose key had no Galaxy ack identity
got Good and then a revert, because whether the key has an identity lives in
the Secrets store, which must not be read under the SDK lock.
GalaxyAckIdentityDirectory (Driver.Galaxy) keeps a node-wide set of the key ids
that have a galaxy-ack-user secret. It lists this node's Secrets store
(metadata only, nothing decrypted) and is read lock-free.
GalaxyDriver.GetAcknowledgeRefusal (IAlarmAcknowledgePreflight, published
through NativeAlarmAckPreflightView as in #760) now refuses a keyed session
whose key the directory reports Absent, when the driver has no fallback ack
account. A refusal also asks for a refresh.
Fails open: before the first refresh, once the snapshot is older than
StaleAfter, and from a signalled identity-secret change until the next
refresh, the lookup answers Unknown and the ack takes the old Good-then-revert
path.
Refresh: GalaxyAckIdentityRefreshService runs on driver nodes, every
Drivers:Galaxy:AckIdentityCache:RefreshIntervalSeconds (default 30;
StaleAfterSeconds 0 = three intervals; validated at start). It also refreshes
early on secret changes. The Secrets library has no change event, so
SecretsRegistration decorates ISecretCacheInvalidator, which is called by
AdminUI /api-keys, /admin/secrets and the replication appliers. API-key
changes need no event: a revoked key is already denied before the pre-check.
Tests: Galaxy.Tests GalaxyAckIdentityDirectoryTests (16) and Host.Tests
GalaxyAckIdentityCacheWiringTests (9). Seven separate mutations each turn at
least one test red: pre-check ignores the directory, factory drops it, no
invalidator decoration, no staleness check, no change-pending check, no
refresh on refusal, loop never wakes early.
Docs: Galaxy.md, AlarmTracking.md, Configuration.md, .claude/rules/alarms.md.
Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
QuestDB modified-history reads dropped every still-pending trail row that was
not the latest at its source time, so V0->V1 (applied marker never written)
followed by V1->V2 hid the V0->V1 change for good. A pending row below the
latest is now served when the NEXT trail row at that time recorded a different
prior value (kind, value, status or the prior's server_ts) - the next change saw
what this one wrote; an orphan of a change that never landed leaves the same
prior for the next row and stays hidden. QuestDB window functions reject
VARCHAR, so the next row's prior comes from a LEFT JOIN on (ts, lead(id)).
The updater's comment and marker-failure log no longer claim modified reads
"still serve it" unconditionally.
Regression tests (QuestDB 10.0.1 throwaway container): marker lost then a
replace / delete at the same time; an orphan followed by a marker-lost retry
then a later change (orphan still dropped); the random marker-mix fuzz now
varies the recorded priors. All four fail on the old filter.
NITs:
- SqlServerHistorianValueWriter: a SqlException raised while the token is
cancelled (SqlClient's "Operation cancelled by user") is rethrown as
OperationCanceledException instead of booked as a failed write. Integration
test holds a TABLOCKX on Sample and cancels mid-INSERT; it returned false before.
- SqlServerHistorianDataUpdater.DeleteRawBatches is an Interlocked counter.
- HistoryReadEvents: an event filter whose SDK validation throws (unknown
operator) failed with the generic BadUnexpectedError; the node manager now
re-throws any failed event-filter validation as BadEventFilterInvalid.
Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
The batched-alarm-history test came from the lane-H2 branch, which predates
the xUnit1051 sweep on master (lmxopcua#725). Two no-token calls broke the
Release build under TreatWarningsAsErrors after the rebase.
Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
HistoryReadEvents projected only EventId / SourceName / Time / ReceiveTime /
Message / Severity: a select clause asking for EventType, ConditionName or
SourceNode (what most generic alarm viewers ask for) got nulls, and the
event filter's where-clause (an OfType) was ignored.
- HistoricalEvent gains optional EventTypeName, ConditionName and AlarmId.
SqlServer, QuestDB and Gateway event reads fill them from what their
alarm writers already store (AlarmTypeName / AlarmName / AlarmId; the
gateway event Type and its AlarmName / AlarmId properties).
- The node manager maps the type name to its Part 9 event type (unknown or
non-event => AlarmConditionType), projects ConditionName, and resolves
SourceNode to the materialised condition the alarm id names (the live
events' SourceNode), else null.
- The where-clause is evaluated per event with the SDK content filter
through an IFilterTarget (OfType via the type tree; other operators
against the projected fields), after the page's cursor is taken, so
paging is unchanged.
Tested in the node manager, over a real client session (the Client.CLI has
no `historyread --events` mode yet; filed as a follow-up), and by alarm
writer -> event read round trips on SqlServer and QuestDB.
Rebased from the unpushed remed/lane-H2 branch (originally authored 2026-10-22
in session_01Ub9hGmYLQvn9XETzw7CMsT) onto current master.
Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
The Gateway backend's gaps against SqlServer/QuestDb were documented
piecemeal. The backend-selection table in docs/Historian.md now also covers
the aggregate set (and the native time-weighted opt-in), emulated reverse
and end-only reads, the client-side events cap, retention and the health
probe.
Continuous historization on Gateway silently loses sample quality (stored
Good), comms-loss outage markers and non-numeric values. Building its live
writer, which only happens when ContinuousHistorization is enabled, now
logs one startup Warning that names the gaps and the alternatives.
The review asked for the matrix in the AdminUI backend picker, but no such
picker exists: the backend is configuration-only. Filed as a follow-up.
Rebased from the unpushed remed/lane-H2 branch (originally authored 2026-10-22
in session_01Ub9hGmYLQvn9XETzw7CMsT) onto current master.
Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
FasterLogHistorizationOutbox.PeekBatchAsync held the state lock across the
FasterLog scan. Once a backlog's head ages out of FasterLog's in-memory
window (any multi-minute historian outage), GetNext reads pages from disk
synchronously, and the recorder's appends (which index under the same lock)
blocked on an Akka dispatcher thread once per drain pass, exactly during
outage recovery.
The peek now snapshots the head under the lock, scans unlocked, and re-takes
the lock only when it met undecodable records, to reconcile them with the
index in O(records scanned) and drop one still at the head (re-checked,
since an ack or drop-oldest may have moved it). A test seam pauses the scan
and proves an append completes meanwhile; wrapping the scan in the lock
again fails it.
Rebased from the unpushed remed/lane-H2 branch (originally authored 2026-10-22
in session_01Ub9hGmYLQvn9XETzw7CMsT) onto current master.
Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
The applied-marker reconciliation changed the modified read's query shape (a
window function over each source time, with the resume position applied
outside it), which is a paging change. A random trail (1-4 modifications per
time; applied, orphaned and landed-without-marker rows; times with no
sample) is paged at caps 1..50 in both directions through the cursor and
must concatenate to the unpaged read, which must equal an independent oracle
of the marker rule. Falsified: dropping the latest-row rule fails it.
Rebased from the unpushed remed/lane-H2 branch (originally authored 2026-10-22
in session_01Ub9hGmYLQvn9XETzw7CMsT) onto current master.
Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
QuestDB has no multi-table transaction, so a HistoryUpdate writes its trail
row before the sample change. A failed or crashed change left an orphan
trail row that modified reads served as a change that happened, and a trail
INSERT re-sent after a client timeout duplicated (the table had no key).
- sample_modified gains `pending` and DEDUP UPSERT KEYS(ts, tag,
modified_id). The trail row is written pending; after the change lands
the same row is re-upserted pending = false (the applied marker).
- A transient trail INSERT failure is re-sent once with the same ids; a
first attempt that did apply converges on the key.
- ReadModifiedAsync serves a pending row only when it is the latest trail
row at its time and the sample there was written at or after the
modification (a crash between change and marker). The orphan of a change
that never landed, typically followed by the client's retry, is dropped.
ASOF JOIN over the window's samples of one tag; the resume filter sits
outside the window function so it never cuts a time's partition.
- The bootstrapper migrates an older trail (ADD COLUMN pending, DEDUP
ENABLE) and waits for it to apply (bounded 10 s); old rows read
pending = false and stay visible. Until applied, or with
AutoCreateSchema=false, the old format is used.
Rebased from the unpushed remed/lane-H2 branch (originally authored 2026-10-22
in session_01Ub9hGmYLQvn9XETzw7CMsT) onto current master.
Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
DeleteRaw deleted an unbounded range in one statement and transaction with
OUTPUT INTO the modification trail. Over years of a 1 Hz tag that is tens of
millions of rows deleted and audited at once: lock escalation on Sample
(blocking the live writer), transaction-log growth, and a 30 s
CommandTimeout that rolled the whole thing back.
It now runs DELETE TOP (4000) batches, each its own transaction with its
trail rows in the same statement (every deleted row is still audited),
cancellable between batches. A failure part-way leaves the committed batches
deleted and reports the failure; re-issuing completes it (documented, same
as QuestDB's paged tombstoning). The existing atomicity tests still hold per
batch.
Rebased from the unpushed remed/lane-H2 branch (originally authored 2026-10-22
in session_01Ub9hGmYLQvn9XETzw7CMsT) onto current master.
Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
Event reads filtered on source and every modified-value read filtered on tag
scanned that symbol column in every partition of the window; only
sample.tag carried an index (#683).
Both columns are now SYMBOL INDEX in the DDL, and the bootstrapper's index
migration is generalised over QuestDbTables.SymbolIndexes (sample.tag,
event.source, sample_modified.tag): a missing index is added with the same
idempotent ALTER ... ADD INDEX, Warning-only when AutoCreateSchema is false
or the ALTER is refused. Integration tests: new tables carry all three; a
pre-index table set gains them on boot and event reads answer the same.
This reverses #683's documented choice to leave the two unindexed;
docs/Historian.md carries the reasoning.
Rebased from the unpushed remed/lane-H2 branch (originally authored 2026-10-22
in session_01Ub9hGmYLQvn9XETzw7CMsT) onto current master.
Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
The SqlServer purge read every TagId and issued at least one DELETE TOP per
table per tag on every pass, even when nothing was eligible: 50000 tags made
100000 statements per hourly purge, serialised on one connection under the
applock.
One pre-filter query per table (EXISTS seek on the clustered key) now picks
the tags holding a row older than the cutoff; only those get deletes. A pass
with nothing eligible sends just the Event delete. The mid-command
cancellation test now accepts the pre-filter SELECT as the blocked command,
since it is the first statement to touch Sample.
Rebased from the unpushed remed/lane-H2 branch (originally authored 2026-10-22
in session_01Ub9hGmYLQvn9XETzw7CMsT) onto current master.
Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
All three alarm-history writers inserted one event per round trip, serially
(SqlServer also opened a pooled connection per event), so a 10000-event
alarm storm drained at 10000 x RTT while the backlog sat in LocalDb.
- Sql.Common.AlarmBatchWrite: one statement per chunk; a chunk that fails
PermanentFail-class is re-sent event by event so only the poison event
dead-letters; a transient failure retries the chunk; a cancellation never
dead-letters.
- SqlServer: multi-row insert-if-absent over a VALUES set (150 events per
statement, under the 2100-parameter limit), duplicate ids in a batch
written once; a duplicate-key race isolates per event, where it is Ack.
No new table type, so no schema migration (the review suggested a TVP;
the VALUES form gets the same one-round-trip-per-chunk without one).
- QuestDb: multi-row INSERTs of 500 events; pre-1970 events still
dead-letter unsent.
- Gateway: no batch RPC exists, so sends are pipelined, at most 8 in
flight, outcomes kept in input order.
Rebased from the unpushed remed/lane-H2 branch (originally authored 2026-10-22
in session_01Ub9hGmYLQvn9XETzw7CMsT) onto current master.
Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
The QuestDB live writer waited for WAL visibility after every tag group, and
the wait polled wal_tables() every 10 ms. With DrainConcurrency = 8 that was
up to eight pollers at ~100 queries/s each against a database whose WAL
apply was already behind.
New Runtime seam IHistorianPassWriter (WriteLiveValuesInPassAsync +
FlushAsync). The recorder writes every group of a pass through it and calls
FlushAsync once after all groups report, before the ack, whenever any group
wrote (an un-acked pass is redelivered and its insert-if-absent check must
see this pass's rows). A flush failure does not fail the pass. The QuestDB
writer implements it: in-pass writes only mark a flush as owed; FlushAsync
waits once. The poll now backs off 10 -> 20 -> ... -> 200 ms, never past the
timeout. Direct WriteLiveValuesAsync callers still wait per call.
Also updates the QuestDB cancelled-write integration test to the rethrow
contract from the previous commit.
Rebased from the unpushed remed/lane-H2 branch (originally authored 2026-10-22
in session_01Ub9hGmYLQvn9XETzw7CMsT) onto current master.
Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
Every IHistorianValueWriter caught Exception and returned false, including
OperationCanceledException from the drain's own token. The recorder's guard
("a cancelled pass is not a tag failure") was therefore dead code: at shutdown
mid-pass each cancelled group could be booked as a consecutive failure and,
with a sibling that already wrote, tip a healthy tag toward quarantine.
SqlServer, QuestDb and Gateway writers now rethrow a cancellation of the
caller's token; a cancellation-shaped error on a live token is still an
ordinary failed write.
The Gateway writer relabelled a Local-kind timestamp as UTC instead of
converting it, so the same value could land at two different times on two
backends. Stamped samples now go through LiveWritePreparer (Local -> UTC,
non-finite doubles not sent, duplicate timestamps keep the first); unstamped
samples are still left for the gateway to stamp, in batch order.
Rebased from the unpushed remed/lane-H2 branch (originally authored 2026-10-22
in session_01Ub9hGmYLQvn9XETzw7CMsT) onto current master.
Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
Every Gateway HistoryRead was preceded by a BrowseTagNames catalog stream
keyed by a glob filter (a name with '*' or '?' streamed every match), and the
4096-entry presence cache evicted an arbitrary entry (Keys.First()) when full,
so a sweep over more tags thrashed it.
The presence check is now one exact unary GetTagMetadata (null = absent).
Only the deployed gateway's known misparse of a missing tag (Internal /
Unknown / InvalidDataException) falls back to the exact-name catalog check;
any other failure propagates and never becomes absence. The positive TTL is 5
minutes (was 60 s); a full cache drops expired entries first, else the one
closest to expiry. Wire tests cover the working, the misparsing and the
refusing gateway.
Rebased from the unpushed remed/lane-H2 branch (originally authored 2026-10-22
in session_01Ub9hGmYLQvn9XETzw7CMsT) onto current master.
Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
The Gateway historian's ProcessConnectionOpen / EventConnectionOpen were
derived from read outcomes only, so a freshly booted node with a healthy
gateway reported the historian closed until a client read, and a node that
only writes (continuous historization) never flipped it.
The data source now runs a ProbeAsync loop (ServerHistorian:Gateway:
ProbeInterval, default 30 s, first probe at start-up; 0 = off). Each probe,
bounded at 10 s, is recorded as the latest connectivity observation; the
latest observation (probe or read) is what the snapshot reports. Probes never
touch the query/failure counters. DisposeAsync stops the loop before
disposing the client.
Rebased from the unpushed remed/lane-H2 branch (originally authored 2026-10-22
in session_01Ub9hGmYLQvn9XETzw7CMsT) onto current master.
Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
AVEVA's native TimeAverage / Total are answered with Good-status wrong values
while the gateway sends the interval in the wrong unit (historiangw#27): one
whole-span value on the earliest interval, the rest absent. Nothing on the
wire marks the answer as degraded.
Both are now refused BadAggregateNotSupported (unsent, not counted as a
backend failure, not advertised in AggregateFunctions) unless
ServerHistorian:Gateway:EnableNativeTimeWeighted is true. The default
Gateway set is the ten raw-evaluated aggregates
(GatewayHistorianDataSource.RawEvaluatedAggregates); ServedAggregates stays
the 12 the backend can serve with the flag on. The native-path tests opt in.
Also documents the ServerHistorian:Gateway sub-section (EventDrainBudget from
the previous commit) and the events drain in docs/Historian.md.
Rebased from the unpushed remed/lane-H2 branch (originally authored 2026-10-22
in session_01Ub9hGmYLQvn9XETzw7CMsT) onto current master.
Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
A capped Gateway HistoryReadEvents read broke out of the gRPC stream as soon
as the page was full. Abandoning a gateway stream cancels the RPC, which the
gateway treats as a connection fault: it evicts its pooled historian session,
so the next read reconnects and concurrent reads on that session fail.
The read now keeps consuming the stream past the cap without collecting it,
bounded by ServerHistorian:Gateway:EventDrainBudget (default 10000 rows; 0 =
the old abandon-at-cap). Rows of a foreign source never count toward the
budget before the "more events" signal is seen, so an abandoned stream never
answers a page as complete. A paged-vs-unpaged fuzz (random tie series,
foreign rows, caps 1..500, budgets 0..10000) pins the page contents and the
drain; it found that abandon-before-signal case.
ReadEventsAsync and the native AtTime arm now rethrow a caller cancellation
instead of recording it as a backend failure.
Rebased from the unpushed remed/lane-H2 branch (originally authored 2026-10-22
in session_01Ub9hGmYLQvn9XETzw7CMsT) onto current master.
Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
RegisterNotificationAsync published a refused registration into
NativeRegistrations under ConnectGate with NextRetryUtc = MinValue; the
SubscribeAsync caller booked the refusal backoff (NoteDeadFailure refused:
true) only after the gate was released and after its WARN. A probe tick in
that window took the gate, saw the registration as due, and retried it at
once - the CI flake in A_rearm_during_an_in_flight_retry_is_not_lost
("AttemptsFor(\"MAIN.T\") should be 1 but was 2").
RegisterNotificationAsync now books the failure (refusal backoff, or the
comms-class every-tick retry) before it publishes the dead registration, still
under the gate. The caller no longer books the refusal a second time (that
would double the backoff); its comms-class booking is idempotent and stays.
Tests: TwinCATCapturingLogger gains an OnWarning hook. A logger that holds
the refusal WARN for ten probe intervals forces the window open, so
A_rearm_during_an_in_flight_retry_is_not_lost exercises it on every run, and
the new A_refused_registration_is_published_with_its_backoff_already_set
pins the backoff (booked once, not doubled).
Pass rate of A_rearm_during_an_in_flight_retry_is_not_lost:
- before, unmodified test: 40/40 runs, plus 200/200 in-process repeats under
CPU load (the natural window is a few instructions wide; the flake is rare);
- before, window held open: 0/32 (deterministic failure, "should be 1 but
was 2");
- after, window held open: 40/40.
Full TwinCAT suite 663/663 at AKKA_TEST_TIMEFACTOR 1 and 3.
Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
Hosted run 2243 (dc0a24f2, job 3882) failed in
ModbusWritePduFramingTests.FC16_pdu_is_exactly_header_plus_byte_count with
"Sequence contains no matching element" at transport.Sent.Single(fc == 0x10),
one line after the write had already returned status Good: the driver sent the
FC16 PDU, but the record lost it.
The fixture left Probe at its default (Enabled = true), so InitializeAsync
started ProbeLoopAsync on a pool thread, which sends an FC03 at once. Both that
send and the write's send did List<byte[]>.Add on the capturing transport
concurrently; List is not thread-safe and one Add can overwrite the other.
- Disable the probe in this fixture, as the sibling capturing-transport
fixtures (ModbusProtocolOptionsTests, ModbusCoalescingDefaultsTests) do.
- Record PDUs in a ConcurrentQueue so a background send can never drop one.
- Dispose the driver (await using) so no loop outlives the test.
Test-only; not related to the #729/#727 changes (2243 predates them, and the
run that includes them, 2260, passed Driver.Modbus.Tests 1055/1055).
Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
CSV: IsArray with blank/0 ArrayLength is the variable-length shape browse-commit and the composer produce, so its own export now re-imports.
Plain MQTT: MqttTagDefinitionFactory.Inspect (the driver's parse, consulted by the authoring inspector) now advises when isArray is set on a Plain-shaped tag; Sparkplug-shaped blobs stay quiet.
Live Sparkplug array test and SDK reflection items remain open.
Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
The SDK keeps sending publish requests on a session that holds no subscription,
and the server answers each with an anonymous (id 0) BadNoSubscription. With
nothing tracked there is nothing to lose, but every answer was counted on
publish_error, so the counter climbed on any idle driver. HandlePublishError now
returns before counting for an anonymous BadNoSubscription while neither the data
nor the alarm subscription set tracks anything. Any other PublishError, and an
anonymous one while something is tracked, is counted and handled as before.
Not addressed from #764: the Republish/state probe for a delete slower than the
5 s grace, and the #728 wire test's fixed delays.
Tests: OpcUaClientIdlePublishErrorTests (idle session: not counted, with a
BadTooManyPublishRequests positive control; tracked subscription: still counted).
The idle test fails without the fix. Driver.OpcUaClient.Tests 295/0.
Docs: drivers/OpcUaClient.md.
Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
R06:M-5 made Sparkplug *Array leaves unbindable because the ingestor refused
them. The ingestor now decodes the packed bytes_value of all 13 array types
(new SparkplugArrayCodec: LE fixed widths, int64-ms DateTime, uint32-count +
MSB-first BooleanArray, NUL-terminated UTF-8 StringArray; golden vectors are
the spec's examples, reproduced byte-for-byte by SparkplugNet 1.3.10) and
publishes a typed 1-D array through the scalar coercion, all-or-nothing.
Shape uses the fleet's one array model: the tag's platform isArray/arrayLength
keys (ValueRank=1, ArrayDimensions=[arrayLength] or [0] = variable length).
The driver reads them with TagConfigIntent's exact semantics and never rejects
on them, so composer, driver, authoring model and deploy inspector agree
(MqttArrayShapeParityTests). Shape mismatch (array metric on a scalar tag, or
the reverse) is BadTypeMismatch - a String tag no longer gets the packed
buffer as base64; a malformed body or a length differing from a declared
arrayLength is BadDecodingError (MTConnect #627 verdict). Template/DataSet
members of an array type take the same path. Arrays stay read-only (encoder
refuses them; MQTT is not in DriverArrayWriteSupport).
Browser: array leaves are bindable again and state isArray in AddressFields;
the commit mapper writes "isArray": true (no arrayLength). Discovery reports
IsArray/ArrayDim. Docs: Mqtt.md "Array metrics", design table, drivers rule.
Tests: codec (golden + malformed), ingest through the real codec + Dispatch
(the project's MiniBroker never forwards a PUBLISH, so there is no in-process
broker harness), browse, discovery, commit mapper, parity corpus. Mutating the
fix back out fails 18 MQTT + 9 AdminUI tests.
Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
AbLegacyDriver stamped LastSuccessfulRead (init, read, write), the read's sample
timestamps and host-state change times from raw DateTime.UtcNow; only the backoff
path read the injectable Clock. All of them now read Clock(), so
NativePcccBridgeStatusHealthTests drives a manual clock (init at T0, read at T1)
and asserts LastSuccessfulRead == T1 exactly instead of spin-waiting for the wall
clock to pass the init stamp (#767). Without the driver change the 6 tests fail
(init stamp is wall-clock, not T0).
Directory.Build.props sets TreatWarningsAsErrors for every project since #766, so
the 75 per-csproj copies (64 src + 11 tests) are removed and the props comment
says it is now the only place. tools/mem-probe keeps its explicit opt-out.
Release build: 0 warnings, 0 errors; a planted CS0168 still fails the build in a
src project and in a test project that used to carry its own copy.
Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
Awaited deadline oracles get the runner token; sync-over-async tests made async;
nullable child models asserted via ShouldNotBeNull() return values; obsolete SDK
ctors replaced with the telemetry overloads; dead locals/fixture and the redundant
DI PackageReference removed. No suppressions.
Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
The native-ack path is fire-and-forget after the Part 9 Acknowledge returns, so
a Galaxy ack refused for lack of a credential (#749), or rejected by MXAccess,
left the client with Good and the condition showing Acknowledged while Galaxy
was never acked. Both options from the issue, layered:
1. Synchronous refusal for what the driver can decide without I/O. New optional
IAlarmAcknowledgePreflight (Core.Abstractions); GalaxyDriver refuses an
anonymous session when the driver has no AlarmAck fallback account.
DriverHostActor publishes a condition NodeId -> owning-driver snapshot into
NativeAlarmAckPreflightView (lock-free, volatile); the node manager's new
NativeAlarmAckPreflight seam, consulted after the authz and Primary gates,
returns BadUserAccessDenied with the reason - nothing routed, applied or
reported. A keyed session is not pre-checked (its identity lives in the
Secrets store, which must not be read under the SDK lock).
2. Revert for everything else. RouteAlarmAck carries the condition NodeId;
DriverInstanceActor replies AlarmAckFailed on a refusal, a backend failure,
a driver without an alarm source, or an ack dropped while connecting /
reconnecting. DriverHostActor re-projects the condition unacknowledged via
NativeAlarmProjector.TryRevertAcknowledge, but only while the local ack is
still pending - any driver transition projected since then wins. A disabled
or unspawned driver is not reverted (unchanged).
GatewayGalaxyAlarmAcknowledger now throws DriverOperationRefusedException on a
non-zero native hresult (wrong password, no ack permission) instead of only
warning, so that case reverts too; as a refusal it neither retries nor trips
the breaker.
Tests (each fails with its fix mutated out): projector revert/supersede,
DriverHostActor refused + failed ack -> unacked AlarmStateUpdate (successful
ack control), preflight view snapshot, node-manager preflight unit test,
over-the-wire ConditionMethodEventTests (BadUserAccessDenied, no route, no
event; positive control), Galaxy preflight + hresult refusal.
Docs: Galaxy.md, AlarmTracking.md, ScriptedAlarms.md, .claude/rules/alarms.md.
Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
HandlePublishError treated any anonymous BadNoSubscription (subscription id 0)
as loss of a tracked subscription whenever anything was tracked. A publish the
SDK queued while the session briefly held no subscription (the driver had just
deleted its last one) is answered BadNoSubscription and can be handled after the
replacement is registered, so a routine unsubscribe->resubscribe declared the new
subscription lost: Degraded health, a BadNoCommunication quality-only annotation
on every alarm condition and ConnectionLost (host re-initialise). That annotation
is the extra Unspecified event the #728 ConditionRefresh wire test saw.
- A BadNoSubscription that names a subscription id now counts only if the id is
tracked (same rule as BadSubscriptionIdInvalid).
- An anonymous BadNoSubscription within StaleNoSubscriptionGrace (5 s) of a
delete that left the driver tracking nothing is ignored (Debug log). The window
is stamped on removal and restarted when the delete completes; failed-subscribe
removes stamp it too; it resets with the session. A delete that leaves another
subscription tracked opens no window. A genuine loss inside the window is still
caught by the PublishingStopped heuristic; only the immediate ConnectionLost is
forgone.
- OpcUaClientStaleNoSubscriptionTests: stale error after alarm and data
resubscribe (fail without the fix), id-attributed stale error, plus genuine-loss
controls (after the grace, no recent delete, non-last delete, session release),
each mutation-checked.
- The ConditionRefresh wire test now records Kind, AlarmType, message, severity,
quality, restatement and timestamp of every post-resubscribe event on failure.
Wire test locally: 1/10 passing before (all 9 failures the 0x80310000
ConditionType quality-only annotation), 12/12 after.
Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
QuestDB speaks plain PG-wire; Npgsql's default GSS Encryption Mode=Prefer made
Linux central boots log 'Cannot load library libgssapi_krb5.so.2'. An explicit
'GSS Encryption Mode' in the connection string is kept. Not re-verified on a
Linux host here; the Postgres Sql driver is left alone (Kerberos may be wanted).
Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
Reinitialize_clears_lastPublished_and_lastWritten_caches now registers
ShutdownAsync through the shared FixtureCleanup helper before initialize, so an
initialize, seeding/polling-observation or assertion failure still stops the
probe loop and poll engine and disposes the transport; the primary failure is
preserved and a secondary cleanup failure is reported. The in-body shutdown
still precedes the cache-clearing assertions; the trailing cleanup is safe
because shutdown is idempotent. Default probe settings, the real subscription
path and the bounded deadline are unchanged.
New Cache_reset_fixture_failure_after_subscribe_still_shuts_the_driver_down
fails the body after the subscription is live and asserts the sentinel is kept,
the transport is disposed exactly once, caches are cleared and no further poll
or probe sends land (fails if the cleanup is removed). No production change.
The probe and re-probe worker delegates closed over the CTS and read cts.Token
when they ran. Task.Run's scheduling token only suppresses a still-queued task;
a worker the pool had committed to but not yet invoked could outlive the bounded
join in StopBackgroundLoopsAsync, after which the CTS was disposed and the
delayed cts.Token read threw ObjectDisposedException (faulting the loop task).
Snapshot the token value before Task.Run and close over it; a token value stays
valid after its source is disposed and still reports cancellation, so a late
worker exits cleanly on its own generation's cancelled token. Document the
bounded-join / transport-lifetime limits on StopBackgroundLoopsAsync.
Regression tests park the worker deterministically in TPL's synchronous
TaskStarted event (after the start check, before the delegate runs) via an
EventListener correlated to the one TaskScheduled on the test's armed flow:
shutdown (probe + reprobe) and replacement-generation identity. All three fail
with ObjectDisposedException without the fix.
Joseph Doherty