Commit Graph

309 Commits

Author SHA1 Message Date
Joseph Doherty 0e96dd08a2 fix(scripting): deny System.AppContext and other process-wide mutators in the script sandbox (lmxopcua#792)
v2-ci / unit-tests (push) Waiting to run
ForbiddenTypeAnalyzer now denies System.AppContext, System.Console, GCSettings, ProfileOptimization,
EncodingProvider/CodePagesEncodingProvider, TypeDescriptor and CryptoConfig as types, and the ambient
CultureInfo.{DefaultThreadCurrentCulture,DefaultThreadCurrentUICulture,CurrentCulture,CurrentUICulture}
and TimeZoneInfo.ClearCachedData members (member deny-list now covers properties and member-access reads).
Shadow-mode tests diverge via culture-sensitive string.Compare (worker is InvariantGlobalization) instead of
AppContext.BaseDirectory, with a premise test. Mutation-proved: 21 new cases fail with the analyzer reverted.

Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
2026-11-10 07:48:39 -05:00
Joseph Doherty 03686eb5d7 feat(scripting): Scripting:Runner=Shadow — in-process results published, each compiled run mirrored to the ScriptWorker pool and compared (lmxopcua#780 P2)
- ShadowScriptRunner (Core.Scripting): IScriptShadow.Mirror copies the inputs and queues the shadow run on the
  pool; compares value (structural), CLR type, status and failure class (compile / budget / threw+message /
  unsupported_type); runner-fault shadow outcomes (Unavailable, worker busy) are not compared. Bounded:
  Scripting:Shadow:MaxInFlight (256) and one per script; drops counted (behind | key_busy).
- Metrics otopcua.script.shadow.{compared,mismatch{script},dropped,inflight}; one rate-limited Warning per
  script and reason with a diff summary (Scripting:Shadow:MismatchLogsPerMinute, re-armed at the seam's deploy).
- The VirtualTag, Calculation and scripted-alarm seams recognise the shadow and keep their in-process path
  byte-for-byte (no coalescer, one-at-a-time pump, in-process engine), mirroring after each compiled run.
- ScriptRunnerSelection builds the shadow over an OutOfProcessScriptRunner whose lines carry
  ScriptRunnerRole=Shadow; Status() is null under Shadow so readiness never sees the shadow pool; the
  script-runner health entry stays Healthy and reports the shadow counters as data.
- docker-dev/docker-compose.script-runner-shadow.yml; docs/VirtualTags.md "Shadow mode".
- Tests: comparison table, per-script metric + once-per-reason log, rate limit, hung/blocking/throwing shadow
  never reaches the caller, MaxInFlight drop accounting, input copy; real-worker divergent + unsupported-type
  scripts counted against exactly those scripts, a worker-only runaway killed/quarantined there with every
  published value intact, alarm transitions identical to in process across a kill -9 of the shadow worker.
  Mutation-proved (16 mutants, 15 red; the one survivor added an assertion).

Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
2026-11-10 07:23:06 -05:00
Joseph Doherty 0796928504 feat(scripting): batch submit, exact per-item deadlines, scripted-alarm seam on the out-of-process runner (lmxopcua#780)
v2-ci / unit-tests (push) Waiting to run
- IScriptRunner.RunBatch: one EvalBatch frame per worker for every evaluation ready to run; per-item, streamed results.
  The default (InProcessScriptRunner) is RunAsync per item. Used by the Calculation pump (dequeue all, split at a
  repeated tag, publish in dequeue order), the VirtualTag seam (ScriptBatchCoalescer, no timer) and the scripted-alarm
  engine (whole dirty set). In process the pump and the engine stay one at a time, unchanged.
- Exact per-item deadlines: the worker hands work to an idle evaluation thread or defers it (EvalResults
  deferred_eval_ids); the host stops that run's clock and restarts T then G when it starts (started_eval_ids). A run
  queued behind a runaway is neither timed out nor struck and is a victim, not the culprit, at the kill.
- Scripted-alarm seam behind Scripting:Runner:Seams; unset Seams now means VirtualTag + Calculation, so alarms run out of
  process only when listed. Load still compiles each predicate (to PE bytes only) so a broken one fails the load; passes
  batch and apply in the serial order under the gate. Script faults freeze the condition BadInternalError; runner faults
  freeze it BadResourceUnavailable and re-evaluate on recovery; a dead worker never clears an alarm. IScriptRunnerProvider
  carries the runner to DriverHostActor.
- Perf harness: batch throughput (RunBatch vs RunAsync x n) and alarm-pass latency (in process / batched / serial);
  design §12.3 records macOS + windev. On windev the serial out-of-process alarm pass misses 250 ms from ~500 alarms;
  batched p99 <= 53 ms at 1,000.

Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
2026-11-10 06:37:36 -05:00
Joseph Doherty 11a5c1d4e2 test: deflake load-sensitive tests (lmxopcua#769)
v2-ci / unit-tests (push) Has been cancelled
- NodeManagerLockTimeBenchmark: wall-clock budget enforced only with OTOPCUA_BENCH=1; default run keeps the state-correctness asserts and logs timings.
- OpcUaClientDriverWire host-status: signal (semaphore) wait that only accepts a non-Running state observed after the server is disposed.
- AbCip PollLatencyComparison: retry once on a fixture connection reset; p95 criterion enforced only with AB_LAT_ENFORCE=1.
- ScriptRunGuardTests: orphan scripts held by a ctx.Released signal instead of a 250 ms wall-clock spin that could finish inside the 60 ms budget under load; stuck-orphan wall-clock rule disabled in the tests that pin the strike counting.
- DependencyMuxReplayTests: split the inside-grace positive control (1 h grace) from the release test (control asserted before unregister), removing the 200 ms grace race.

Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
2026-11-10 05:39:22 -05:00
Joseph Doherty 000d7f5591 fix(virtualtags): route VirtualTag outputs through the dependency mux so VT->VT reads work (lmxopcua#791)
v2-ci / unit-tests (push) Waiting to run
Proven broken first: VirtualTagToVirtualTagFlowTests (real mux + host + Roslyn
evaluator) failed on master - VT A evaluated, VT B reading A by its key never did.
VT outputs only reached their OPC UA node; the mux carried driver values only.

- VirtualTagHostActor also sends each real result (not re-asserts) and a disabled
  tag's BadOutOfService to the mux as VirtualTagOutput, keyed by the VT's UNS
  NodeId identifier {EquipmentId}/{Name}. Publish path unchanged (single source).
- DependencyMuxActor pins VT outputs (retained with no subscriber, no grace,
  immune to SourceRemoved) and replays them; VirtualTagOutputsRemoved releases
  keys that leave the deployed set.
- DraftValidator: blocking VirtualTagDependencyCycle (A->B->A, self-read, loops
  through Calculation tags, {{equip}} refs resolved); disabled VTs are not nodes.
  EquipReferenceUnresolved names the key form when {{equip}}/<name> is a VT.
- Runtime backstop: the host re-detects VT cycles per apply and does not route
  outputs of tags on one.
- AdminUI ScriptTagCatalog lists VTs by key (the bare leaf name never resolved).
- docs/VirtualTags.md documents the routing, key, retention and cycle rules.

Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
2026-11-10 05:17:00 -05:00
Joseph Doherty 963b0d66fc fix(virtualtags): a disabled VirtualTag is no longer deployed or evaluated (lmxopcua#787)
v2-ci / unit-tests (push) Waiting to run
VirtualTag.Enabled was authorable but ignored by both runtime-plan seams.
AddressSpaceComposer.Compose and DeploymentArtifact.BuildEquipmentVirtualTagPlans
now drop disabled rows (byte-parity), so no node is materialised and no
VirtualTagActor is spawned - the same outcome as a disabled scripted alarm.
DraftValidator exempts a disabled VT from VirtualTagNoEffectiveTrigger.
Calculation tags have no Enabled flag, so no equivalent gap. docs/VirtualTags.md updated.

Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
2026-11-10 03:15:22 -05:00
Joseph Doherty f5c30ed0ef feat(sql): authored write block — Update/Insert/Upsert/Procedure + timestamp stamp; ODBC scope + login docs (lmxopcua#781)
v2-ci / unit-tests (push) Has been cancelled
Item 2: a tag's optional `write` block picks the statement the driver builds from its own
catalog-validated identifiers (no authored SQL; values always bound):
- Update (default, unchanged back-compat UPDATE ... WHERE key; 0 rows = BadNoData)
- Insert (key+value[+stamp]; append for a WideRow topByTimestamp tag)
- Upsert: SQL Server MERGE WITH (HOLDLOCK), PostgreSQL INSERT ... ON CONFLICT, MySQL
  ON DUPLICATE KEY UPDATE (1-or-2 rows), Oracle MERGE from DUAL
- Procedure: named stored procedure resolved against the routine catalog by the catalog gate
  (INFORMATION_SCHEMA.ROUTINES / ALL_PROCEDURES), value/key/timestamp params by name (MySQL positional)
- stampTimestamp sets the tag's timestamp column to the write time (UTC)
Odbc serves Update/Insert only (shared SqlWriteSupport table). Constraint failures map to
BadEntryExists (unique) / BadOutOfRange (other integrity, data exception); everything else,
permission denied included, stays BadCommunicationError. WriteIdempotent stays false.
The parser refuses malformed blocks (strict keys/enums/param charset); DraftValidator emits
blocking SqlWriteBlockInvalid / SqlWriteModeUnsupported from the driver's own parse; the AdminUI
editor surfaces the same message.

Items 1 + 4 (docs/drivers/Sql.md): ODBC stays the ANSI SQL:2008 profile (native providers for
SQL Server/MySQL, DB2 unsupported) with an ops note on driver manager + vendor driver on every
Odbc and AdminUI host; read-only drivers require a SELECT-only login, writable drivers a login
limited to their write targets.

Tests: SqlWriteModeTests (parse, every dialect's SQL, gate, status map, SQLite end-to-end),
DraftValidatorSqlWriteBlockTests, AdminUI round-trip; live SqlWriteModesLiveTests on SQL Server,
PostgreSQL, MySQL, Oracle (seeds gain write_targets + set_target) and an Odbc write leg.

Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
2026-11-09 22:59:09 -05:00
Joseph Doherty 850c5405d0 fix(scripting): name the script on the mirrored quarantine line; keep BadInternalError (lmxopcua#784)
v2-ci / unit-tests (push) Has been cancelled
(a) ScriptRunGuard logs its quarantine Error through the seam's script root
logger, which carries no ScriptName, so ScriptLogCompanionSink mirrored it to
the main log as "[Script] unknown emitted Error ..." and rate-limited every
quarantine on the node under one shared "unknown" key. The guard now binds the
script instance key as ScriptName (ForContext) on the quarantine Error and the
subscriber-threw Warning, so the mirror names the script and limits each one
separately. Test drives two real quarantines through the companion sink with a
1-per-window limiter: both mirrored, each named, none "unknown". Mutation-
checked: dropping the ForContext fails it.

(b) Evaluated a quarantine-specific OPC UA status (BadOutOfService) and kept
BadInternalError: it is the documented code for any script-attributable fault
on virtual tags, Calculation tags and scripted-alarm quality (and the
out-of-process runner design keeps it), BadOutOfService reads as a deliberately
disabled point, a new code is a ScadaBridge-visible contract change, and the
once-per-Good->Bad dedup means a timeout-driven quarantine would rarely show a
different code anyway. Documented in docs/VirtualTags.md ("Quarantine: what a
client sees"), with pointers from Calculation.md, ScriptedAlarms.md and the
OpcUaStatus.BadInternalError doc comment, plus where a quarantine IS visible.

Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
2026-11-09 21:30:17 -05:00
Joseph Doherty d9b95dcf56 fix(virtualtags): consume VirtualTag.TimerIntervalMs; add VirtualTagNoEffectiveTrigger gate (lmxopcua#783)
v2-ci / unit-tests (push) Waiting to run
Choice: implement the timer (option a). The field was designed and promised as a
working trigger: the entity doc says ChangeTriggered and TimerIntervalMs together
realize the trigger model, the DB check constraint requires ChangeTriggered OR a
timer >= 50, the AdminUI virtual-tag modal exposes "TimerIntervalMs (optional)" and
UnsTreeService rejects a tag with neither, and the 2026-06-08 design specified
"ChangeTriggered OR TimerIntervalMs>=50". Only the runtime never honoured it (the
standalone TimerTriggerScheduler was deleted in #603), so a timer-only tag sat at
BadWaitingForInitialData forever.

- EquipmentVirtualTagPlan carries ChangeTriggered/TimerIntervalMs (in equality, so
  an edit respawns the child); both the composer and the artifact decode fill them.
- VirtualTagActor schedules a repeating tick (PreStart, cancelled in PostStop,
  clamped to the 50 ms floor). ChangeTriggered=false makes the timer the only
  trigger. A tick during an in-flight async run (#779) is dropped, not stashed, so
  slow scripts never build a backlog of runs.
- DraftValidator VirtualTagNoEffectiveTrigger (hard error, same severity as
  CalculationNoEffectiveTrigger): no timer and no change trigger with a literal
  ctx.GetTag read.

Tests: VirtualTagActorTimerTests, DraftValidatorVirtualTagTriggerTests, artifact
decode test. Mutation-checked: no scheduling, stashing ticks, no cancel in
PostStop, and removing the validator call each fail a test.

Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
2026-11-09 21:27:59 -05:00
Joseph Doherty 241318e7e5 feat(deploy): reject Sql Query tags naming undeclared queries; warn on Writable/writable disagreement (lmxopcua#781)
v2-ci / unit-tests (push) Has been cancelled
Item 5: DraftValidator now emits SqlQueryNotDeclared (always blocking) for a Query tag whose
query is absent from the merged driver config's queries map, via the driver's own tag parse.
Item 3: StartDeployment appends a non-blocking warning to the accepted message when the Tag
entity's AccessLevel and the Sql blob's writable flag disagree.

Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
2026-11-09 18:08:54 -05:00
Joseph Doherty 9e8b5b0ea7 fix(scripting): stop unbounded script recursion from killing the server (lmxopcua#779)
Reproduced first, out of process (ScriptRecursionContainmentTests re-launches
the xunit v3 test assembly to run one recursive script through
ScriptEvaluator + ScriptRunGuard, so the runner never executes one): before
this change every shape — local function, mutual local functions,
self-referencing Func<>, anonymous self-application, recursion through LINQ,
an expression-bodied void lambda, an anonymous method — ended the child with
exit 134 (SIGABRT, "Stack overflow."). A StackOverflowException cannot be
caught, so in production one script takes the whole node down.

Fix: ScriptEvaluator.Compile now rewrites the validated wrapper
(ScriptStackGuardInjector, after ForbiddenTypeAnalyzer) so the script body and
every lambda, anonymous method and local function starts with
ScriptStackGuard.Check(), which throws ScriptStackExhaustedException when
RuntimeHelpers.TryEnsureSufficientExecutionStack reports the stack nearly
spent. The script then fails like any throwing script (Bad, script log) and
the process lives.

Why a runtime check, not an analyzer rule: recursion is not decidable from
syntax (anonymous self-application never names itself), and bounded recursion
must keep working. Expression-bodied lambdas/local functions become blocks
(`return e;` or `e;`, chosen from the bound symbol, so the same overload
binds); expression-tree and async lambdas are left alone. Limits (documented
on ScriptStackGuard and in docs/VirtualTags.md): depth depends on the thread's
stack size; a recursion that never re-enters script code is not seen; a script
may catch the exception, which only unwinds it.

Tests: ScriptRecursionContainmentTests (7 shapes, child process each; all 7
fail with exit 134 when the rewrite is disabled), ScriptStackGuardInjectionTests
(bounded recursion and every rewritten lambda/local-function shape keep their
value; the expression-tree case fails if those lambdas are rewritten).

Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
2026-11-09 17:44:35 -05:00
Joseph Doherty d683b83156 fix(scripting): give scripted-alarm predicates the configured run budget, not 250 ms (lmxopcua#779)
DriverHostActor built ScriptedAlarmEngine without a scriptTimeout, so every
production alarm predicate fell back to TimedScriptEvaluator.DefaultTimeout
(250 ms) while virtual tags and Calculation tags ran on 2 s. History shows no
decision behind it: 250 ms is the generic Phase-7 A.4 default, the VT
evaluator moved to 2 s in F8b (219d10a2) and the alarm engine was simply never
given a value.

- New node-wide Scripting:RunTimeout (default 2 s, clamped to (0, 60 s],
  non-positive -> default, each clamp a startup Warning) on ScriptingOptions.
- The VT evaluator and the alarm engine default to it; DriverHostActor now
  builds the engine through CreateScriptedAlarmEngine, which passes it
  explicitly. Calculation keeps its per-driver runTimeoutMs.
- Docs: ScriptEditor (config table), ScriptedAlarms, VirtualTags.

Tests: ScriptedAlarmEngineBudgetTests (a 600 ms predicate now completes and
activates; default and configured budgets), ScriptedAlarmEngineBudgetWiringTests
(the host-built engine), ScriptingOptionsBindingTests (bind + clamps). The
engine and wiring tests fail with the old 250 ms default restored.

Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
2026-11-09 17:44:35 -05:00
Joseph Doherty 97dc715dac fix(runtime): derive every write bound and Ask from one write timeout (lmxopcua#777)
v2-ci / unit-tests (push) Has been cancelled
Three write bounds were configured independently: the inbound-write
gateway's 10 s Ask, DriverHostActor's fixed 8 s Ask of the child, and the
child's own write timeout (5 s + up to 2 s grace) with a fixed 8 s queue
budget. An outer Ask that fires first reports the write failed and the node
reverts while the child is still driving the device, so any write timeout
above ~6 s (or a write dequeued late in its 8 s budget) let the device take
a value the client was told failed.

DriverWriteBounds (Runtime) derives them all from DriverWrites:Timeout:
  token cancel   = Timeout
  abandon        = Timeout + min(Timeout/2, 2 s)            (RunBound)
  queue budget   = RunBound
  child replies  <= queue budget + RunBound                  (ReplyBound)
  host Ask       = ReplyBound + 1 s
  gateway Ask    = host Ask + 1 s
Default Timeout is 10 s (the Tier C Write timeout, so no tier default is
capped below its own policy); host Ask 25 s, gateway Ask 26 s. Tier A/B
writes are still cancelled first by their 2 s / 4 s tier timeouts.

- DriverHostActor takes writeBounds + writeClock, spawns children with them,
  and runs its Ask deadline as a token on the write clock (Infinite Akka
  timeout), so host and child bounds share one clock.
- WriteAttribute gains IssuedAt (write-clock timestamp, stamped by the
  host): the queue budget counts mailbox time too, and a write past it is
  failed BadTimeout and never started - including on arrival. Stale queued
  writes are expired across the whole queue, not only up to a free slot.
- ActorNodeWriteGateway's default Ask = Default.GatewayAsk; the hosted
  service passes the configured one.
- Startup validation (Host DriverWriteBoundsBinding, same warn-and-clamp
  style as ScriptingOptionsBinding): unparsable / non-positive -> default,
  above 5 min -> clamped, and each tier whose default Write timeout exceeds
  the write timeout is named in a Warning (its policy would never take
  effect). The derived bounds are logged at Information. Per-instance
  ResilienceConfig Write overrides arrive with a deployment, so
  DriverCapabilityInvokerFactory warns at spawn (writeTimeoutCeiling).
- appsettings.json documents DriverWrites:Timeout.

Tests (manual clock):
- DriverHostActorWriteAskTests: a held write under a 9 s timeout survives
  an 8.5 s step and succeeds; a token-ignoring write is answered by the
  child's BadTimeout at 9 s + grace, not by the Ask; the Infinite-timeout
  Ask ends exactly at its clock deadline.
- DriverInstanceActorConcurrentWriteTests: a write issued longer ago than
  the queue budget is failed and never reaches the driver; positive control
  at exactly the budget is written.
- DriverWriteBoundsTests: every outer bound covers the inner ones for 0.5 s
  to 300 s timeouts.
- DriverWriteBoundsBindingTests: default fits every tier; 3 s timeout is
  caught at startup with Tier B and C named (not A); bad values fall back
  with a warning; over-max clamps.
- DriverCapabilityInvokerFactoryTests: spawn warns when a Write override
  exceeds the ceiling; no warning within it or without a ceiling.
Mutation checks: a fixed 8 s host Ask on the same clock fails both 9 s host
tests; stamping arrival time instead of IssuedAt fails the issued-at test;
dropping the tier check / factory check fails their tests.

Runtime.Tests 961 passed / 3 skipped at AKKA_TEST_TIMEFACTOR 3 and 1;
Host.Tests 232/232 at 3 and 1; Core.Tests 361 passed / 1 skipped.
Solution Release build: 0 warnings.

ContinuousHistorizationRecorderTypedValueTests (historian) deliberately
left for the historian lane.

Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
2026-11-09 15:46:30 -05:00
Joseph Doherty 919009c5ef fix(ablegacy): stamp health from the injectable Clock; drop the redundant per-project TreatWarningsAsErrors (lmxopcua#768)
AbLegacyDriver stamped LastSuccessfulRead (init, read, write), the read's sample
timestamps and host-state change times from raw DateTime.UtcNow; only the backoff
path read the injectable Clock. All of them now read Clock(), so
NativePcccBridgeStatusHealthTests drives a manual clock (init at T0, read at T1)
and asserts LastSuccessfulRead == T1 exactly instead of spin-waiting for the wall
clock to pass the init stamp (#767). Without the driver change the 6 tests fail
(init stamp is wall-clock, not T0).

Directory.Build.props sets TreatWarningsAsErrors for every project since #766, so
the 75 per-csproj copies (64 src + 11 tests) are removed and the props comment
says it is now the only place. tools/mem-probe keeps its explicit opt-out.
Release build: 0 warnings, 0 errors; a planted CS0168 still fails the build in a
src project and in a test project that used to carry its own copy.

Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
2026-11-09 13:40:07 -05:00
Joseph Doherty b8202e1147 test: remaining xUnit1051/1031/1030, CS8602/8604, CS0618, CS0219, CS0649, NU1510 fixes (lmxopcua#725)
v2-ci / unit-tests (push) Has been cancelled
Awaited deadline oracles get the runner token; sync-over-async tests made async;
nullable child models asserted via ShouldNotBeNull() return values; obsolete SDK
ctors replaced with the telemetry overloads; dead locals/fixture and the redundant
DI PackageReference removed. No suppressions.

Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
2026-11-09 13:22:34 -05:00
Joseph Doherty 1d7a4b7a53 test: pass TestContext.Current.CancellationToken at xUnit1051 call sites (analyzer code fix, 451 sites) (lmxopcua#725)
Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
2026-11-09 13:19:34 -05:00
Joseph Doherty acfb4fbaca fix(redundancy): a failed OPC UA endpoint drops ServiceLevel to 0 and steps the Primary down (lmxopcua#762)
v2-ci / unit-tests (push) Has been cancelled
IOpcUaEndpointState (review-remediation lane C3) recorded whether the SDK started, but only the
health check read it: a fused node whose endpoint failed (port 4840 taken, ...) still advertised
250/240 and stayed driver Primary, so the Primary-gated data plane ran on a node no client could
reach while the reachable peer refused writes as Secondary.

- Move IOpcUaEndpointState/OpcUaEndpointState/OpcUaEndpointStatus from Host to Commons.OpcUa so
  the Runtime can consume it.
- ServiceLevelCalculator: new NodeHealthInputs.EndpointListening (default true). false -> 0,
  ahead of every other input including the +10 Primary bonus.
- OpcUaPublishActor: takes the endpoint state, feeds EndpointListening (Failed only; NotStarted /
  Stopped are boot/shutdown windows) on the DB-backed and DB-less paths, and the legacy role-only
  seam returns 0 for a failed endpoint too.
- New OpcUaEndpointStepDownActor (one per driver node): a Primary (ClusterActiveNode.SelfIsActive,
  the election's own primitive) with a Failed endpoint gracefully Leaves its mesh - the manual
  failover mechanism - so the peer becomes the oldest Up driver member, hosts the singletons and is
  Primary. The election itself is not bent. Guarded on a peer having recently reported its own
  endpoint listening (EndpointReport on the new opcua-endpoint-state DPS topic), so a pair with
  both endpoints failed does not bounce the role and restart nodes forever, and a lone node never
  shuts itself down.
- Runtime now references ZB.MOM.WW.Health.Akka (ClusterActiveNode).
- Docs: Redundancy.md tier table + new "Endpoint start failure: Primary step-down" section;
  cluster-topology rule; HealthEndpoints / OpcUaEndpointHealthCheck comments (F-C1 landed).

Tests: ServiceLevelCalculatorTests (+5), OpcUaPublishActorEndpointStateTests (7: DB-less,
DB-backed, legacy seam, NotStarted/Stopped do not demote, a later failure demotes on the next
tick), OpcUaEndpointStepDownActorTests (3, real two-node mesh: holds while the peer is not
listening then leaves and the peer becomes Primary; failed Secondary and healthy Primary never
leave). Each was mutation-checked: removing the calculator/legacy guard, the Leave, the
peer-listening guard or the Primary guard fails the corresponding tests.
2026-11-09 12:16:37 -05:00
Joseph Doherty dc0a24f224 feat(galaxy): operator-authenticated alarm ack — send Galaxy credential to mxaccessgw (#749)
v2-ci / unit-tests (push) Failing after 1h8m43s
mxaccessgw 488649c+ acknowledges by AuthenticateUser as the operator and
writing <alarm>.AckMsg, so AcknowledgeAlarmRequest now needs a real Galaxy
operator_user + operator_domain + operator_password. The OPC UA data plane is
API-key/anonymous and carries no Galaxy password, so the driver resolves the
account per ack:

1. the session's API key's Galaxy ack identity, set on AdminUI /apikeys and
   held ONLY in the encrypted Secrets store
   (otopcua/apikeys/<keyId>/galaxy-ack-{user,password}); a key with a user but
   no password secret is refused, never silently downgraded;
2. else the driver instance's AlarmAck fallback account (DriverConfig
   FallbackUser/FallbackDomain/FallbackPasswordSecretRef; secret:/${secret:}/
   env:/file:/dev: only — a literal is refused);
3. else DriverOperationRefusedException: nothing is sent, never an empty
   operator_user. Refusals are excluded from the resilience retry/breaker.

- Pins ZB.MOM.WW.MxGateway.Client/Contracts 0.2.1 -> 0.3.0 (vendored into
  nuget-packages/ from the Gitea feed, per the local-mxgw source mapping).
- The session ApiKeyId (not a secret) rides NativeAlarmAck ->
  RouteNativeAlarmAck -> RouteAlarmAck -> AlarmAcknowledgeRequest; the password
  is resolved in the driver and never travels through actor messages.
- The OPC UA principal + key id are appended to the AckMsg comment
  ("[OPC UA: <principal>, key <id>]") and logged, so a shared fallback
  account does not lose who acknowledged.
- Password never logged: GalaxyAckCredential.ToString omits it, the
  acknowledger scrubs it from every gateway message it echoes and withholds
  any exception that echoes it; audit rows carry only the key id.
- AdminUI: /apikeys "Ack identity" editor (set/remove; delete-key clears it),
  Galaxy driver form gains the fallback section.
- Synchronous OPC UA status for a refusal is follow-up #760 (the native ack
  path is fire-and-forget after the Part 9 method returns).

Tests: request mapping (password populated), password absent from logs and
exception text (mutation-checked), key-mapped / fallback / refused paths at
resolver and driver level, breaker non-trip, key-id propagation through node
manager and DriverHostActor, AdminUI service + form round-trip.

Closes #749
2026-11-06 16:23:00 -05:00
Joseph Doherty cf62007678 fix(arrays): Modbus row honours the FC16/FC03 protocol ceilings, the RMW read cap and the address-space end
Review r0 of Integration-W3:
- I1: the effective FC16 cap is min(configured maxRegistersPerWrite, 123). The
  driver trusts a larger configured cap and sends an over-ceiling FC16 (past 127
  registers its byte-count byte wraps), which every conformant device refuses, so
  such an array is never advertised writable.
- I2: a BitInRegister[] write reads its span first in one unchunked FC03, so it is
  also bounded by min(configured maxRegistersPerRead, 125) (absent/0 = 125).
- m2: every span (registers and coils) must end at or before address 65535.
- m1: the String-node wording says it converts only when the text parses, which
  the server cannot promise (DateTime never converts).
- m4: the parity device now enforces FC01/02 2000, FC03/04 125 (or a smaller
  device read limit), FC15 1968, FC16 123, FC23 121/125, the FC15/FC16 byte-count
  field (exception 03) and the address-space end (exception 02).

Tests: parity rows for each ceiling/cap/span boundary against the real write
path, and Core.Abstractions row tests. The configured-cap test that asserted a
cap of 200 lifts the ceiling now asserts the opposite.

Refs #741

Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 17:32:57 -04:00
Joseph Doherty 8699614d2f Merge commit '82380d2b' into remed/integration-w3 2026-10-22 17:21:38 -04:00
Joseph Doherty 5533de6281 feat(arrays): Modbus String[]/BitInRegister[] array writes reachable; table follows the resolver
The Modbus lane (#741 U3) gave the driver String[] and BitInRegister[] holding-register array encoders, and
moved address resolution to ModbusTagAddressResolution (keys first, a bit region coerced to Bool, strict key
reads), but DriverArrayWriteSupport still listed both array shapes read-only (follow-up F1), so a client
write was BadNotWritable before it reached the driver.

Modbus row now mirrors the driver end to end:
- Keys shape only (region + address both present). A string-only addressString tag is resolved by the
  driver with its own family through a parser Core.Abstractions cannot reference, so it stays read-only
  (pinned: the conservative direction).
- Strict reads as the resolver: address an integer 0-65535; a present region / dataType / byteOrder /
  stringByteOrder string must name a member.
- Coils: any dataType (coerced to Bool), <= 1968 coils (FC15).
- HoldingRegisters (FC16, never Bool): numeric, String ((stringLength+1)/2 registers, stringLength >= 1),
  BitInRegister (1 register, bitIndex 0-15, span ending <= 65535), all within maxRegistersPerWrite.
  FC23 is opt-in and never changes the answer.
- Elements convert (ScadaBridge int[] -> Int16 stays writable); only a String / DateTime node over a
  numeric or bit element is left out, since such an element never converts.

Tests: the parity test's PendingF1 exemption and its guard are gone; the parity matrix is now region x
element type x node type against the real WriteAsync with a spec-conformant device (FC15 > 1968 refused),
plus shape edges at every cap boundary (numeric, String, BitInRegister, coil), strict key reads, FC23
on/off, string-only, the ScadaBridge int[] shape and U3 wire bytes. Core.Abstractions row tests and a
composer test (String[] and BitInRegister[] writable; over-cap, no-stringLength and string-only read-only).
Docs: table XML doc, Uns.md, modbus-addressing.md, drivers/Modbus.md, data-plane rules.

Refs #741

Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 17:10:00 -04:00
Joseph Doherty 13f1357ad2 Merge master d520f77b into remed/lane-E5
Refs #748
Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 16:46:30 -04:00
Joseph Doherty 846c8d6335 fix(alarms): move scripted-alarm comment trail to its own append-only table (R05:4)
Every Part 9 transition used to re-serialize the alarm's whole comment trail
into alarm_condition_state.comments_json and replicate it to the pair peer:
O(n) per transition, O(n^2) over an alarm's life (measured: one ack at 10k
comments shipped 1,080,553 oplog bytes; now ~1.6 KB, the same as at 10).

- New replicated table alarm_condition_comment, PK (scripted_alarm_id, seq),
  seq = the entry's ordinal in the trail. Ordinal (not content) keying makes
  the two pair nodes' copies of one logical entry (each stamped by its own
  clock, since both nodes apply every command) collapse to one row under LWW.
- LocalDbAlarmConditionStateStore.SaveAsync appends only entries past the
  stored MAX(seq) (INSERT OR IGNORE, append-only) and upserts a fixed-size
  condition row with comments_json = "[]", in one transaction. Remove drops
  the trail with the row. Load merges the table with any inline trail.
- comments_json column kept (always "[]") so the condition table's shape and
  replication digest are unchanged and old-shape rows still load exactly.
- AlarmConditionStateSchema.MigrateInlineComments: idempotent per-row move of
  inline trails, run by LocalDbSetup.OnReady after RegisterReplicated so the
  moved entries replicate. Comment table joins ReplicatedTables (now five).

Tests: constant transition payload at 10 vs 10k comments (+ instrument
positive control), old-shape back-compat load, migration idempotency /
partial re-run / lazy move, OnReady boot migration captured in the oplog,
and two loopback pair tests (A->B and B->A trail convergence; the shared
entry converges to one copy, with a distinct-alarm positive control).

Refs #748
Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 16:36:33 -04:00
Joseph Doherty 44ea6595e5 feat(arrays): advertise AbCip/AbLegacy/TwinCAT array writes in DriverArrayWriteSupport
The Wave-2 lanes gave AbCip, AbLegacy and TwinCAT real whole-array write
encoders (review 07 U10/U9), but the capability table still listed all three
as scalar-only, so every array node materialised read-only and a client write
was BadNotWritable before it reached the driver.

Rows added, each listing only what WriteAsync encodes end to end:
- AbCip: SInt/USInt/Int/UInt/DInt/UDInt/Real/Dt/LInt/ULInt/LReal, declared
  length, not a safetyTag. BOOL/STRING/Structure arrays stay read-only.
- AbLegacy: Int/AnalogInt on N/B/A, Long on L, Float on F (file-width match),
  plain <file><n>:<word> address, <= 256 elements; not /bit, .sub, indirect,
  S:/I:/O:, nor native PLC-5.
- TwinCAT: every codec element type, declared length; not a BOOL bit-in-word
  path.
Elements convert in all three encoders, so the rows never key on an exact
node DataType (ScadaBridge int[] -> Int16 stays writable); they leave out
only node types whose elements never convert (DateTime/String over numeric,
Boolean on AbLegacy, STRING under DateTime on TwinCAT: culture-dependent).

Tests: driver-level parity for each driver (every element type x node type
plus shape cases, against the real write path), the AbLegacy parity tests
un-skipped, Core.Abstractions row tests, a composer test (three drivers
writable, refused shapes read-only), and the existing AbCip negative cases
moved to a BOOL array. Docs: Uns.md table, per-driver docs, data-plane rules.

Refs #737 #738 #739

Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 16:17:48 -04:00
Joseph Doherty d2cc071517 feat(scripting): expose the run token to scripts as ctx.CancellationToken
ScriptContext.CancellationToken is backed by an AsyncLocal that
ScriptEvaluator.RunAsync scopes around the compiled delegate, so it is
exactly the token ScriptRunGuard cancels on timeout. A polling script
can now end its own abandoned run and free the slot. AsyncLocal rather
than a field because the scripted-alarm engine reuses one context per
alarm: an orphan keeps seeing its own cancelled token. The scoping
entry point is internal (IVT to Core.Scripting) so a script cannot
replace its own token.

Editor parity: ScriptAnalysisService dot-completion now walks the
receiver's base chain (instance members only, stopping before
System.Object). Before this, a member declared only on ScriptContext
compiled and published but never completed. DiagnoseTests pins the
member as allow-listed; CompletionTests pins the completion.

Refs #731 #734
Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 14:31:03 -04:00
Joseph Doherty 4615495f01 feat(scripting): bind the Scripting section node-wide and clamp its knobs
Program.cs binds ScriptingOptions from Scripting:* once, clamps every
numeric knob through ScriptingOptions.Sanitized (logging a Warning per
adjustment) and registers the one instance. It now reaches all three
script seams: RoslynVirtualTagEvaluator, the scripted-alarm engine
built in DriverHostActor (new trailing Props parameter, wired in
WithOtOpcUaRuntimeActors) and every Calculation driver (node options
as the base, per-driver config keys override one knob on a copy).

ScriptRunGuard sanitizes again in its ctor and computes the stuck
threshold with a saturating multiply, so an absurd
StuckOrphanQuarantineFactor can no longer throw OverflowException on
a refused run.

Refs #731 #734
Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 14:24:53 -04:00
Joseph Doherty 7bcb6331ed Merge lane C: control plane, host, security (#733)
Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 14:09:08 -04:00
Joseph Doherty a85d1c069a fix(scripting): quarantine a refusal behind a stuck orphan (fix round 2)
Ruling refinement on I1. Only timeouts are strikes, so a never-returning
while(true), which times out once and is then only refused, was never
quarantined, and that defeated E1. A refusal now quarantines immediately
when the orphan it was refused behind has been executing for longer than
ScriptingOptions.StuckOrphanQuarantineFactor x the script's timeout
(default 10, so 20 s at the 2 s default; 0 or less disables it). It uses
the same quarantine path: one Error, otopcua.script.quarantined with
reason=stuck, Bad, lifted on source change or redeploy. A young orphan
(a short overrun) is still a plain refusal.

The guard stamps the run's start (Stopwatch timestamp) when the pool
picks it up and clears it when the run really finishes.

Tests: the 30-trigger while(true) regression (paced) now ends
quarantined with reason "stuck", still with exactly one orphan and a
healthy sibling. Positive control: a 2x overrun is refused, not
quarantined, and evaluates again once the orphan completes. A factor of
0 disables the rule. Docs (ScriptEditor "Runtime safety", VirtualTags)
and the seam remarks are updated.

Refs #734

Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 13:59:10 -04:00
Joseph Doherty ef2eebe6b6 fix(scripting): review minors: configured limit in summaries, dispose gate/CTS, validator, docs (fix round 1)
- ScriptLogRateLimiter.LimitPerWindow. The ScriptLogTopicSink and
  VirtualTagActor summaries now print the configured limit, not
  DefaultLimitPerWindow. Tests assert "limit N/window" for a non-default
  N.
- CalculationDriver.Dispose cancels, then takes _evalGate (bounded 2 s)
  and disposes the gate and _evalCts. The pump and timer ticks enter the
  gate through TryEnterEvalGateAsync, which stops cleanly once the
  driver is disposed. Test: Dispose with a 6 s runaway run in flight
  returns promptly, disposes both, and a late trigger is harmless.
- ScriptedAlarmEngine captures _shutdownToken once and disposes
  _shutdownCts in Dispose when every drained task has completed. A task
  stuck past the grace leaves the CTS to the GC. Lifecycle tests assert
  the disposal.
- DraftValidator: CalculationNoEffectiveTrigger is skipped for a
  change-triggered tag whose scriptId does not resolve. That case is
  already CalculationScriptMissing or CalculationScriptNotFound, and
  there are no reads to count. Both existing tests now assert the
  absence (red before the change).
- docs/VirtualTags.md: System.Threading is denied as a whole namespace,
  with an allow-list; the type list is now accurate (Unsafe,
  RuntimeHelpers, and the ForbiddenMembers on System.Type); the resource
  limits section covers the run guard and the allocation budget.
  docs/ScriptEditor.md: accurate deny-list and a new "Runtime safety"
  section covering timeout, refusal, quarantine after 3 timeouts,
  allocation budget, lifting, and rate-limit summaries that appear only
  on the source's next entry after a burst then quiet.

Refs #734

Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 13:41:15 -04:00
Joseph Doherty 4d17b346c6 Merge branch 'master' into remed/lane-C 2026-10-22 13:31:03 -04:00
Joseph Doherty 6aa5edaf30 fix(scripting): only timeouts are quarantine strikes; alarm Bad annotation once per bucket change (fix round 1 I1, I3)
- I1 ScriptRunGuard: a run refused because the previous one is still
  executing is a Bad publish plus otopcua.script.run_refused, and no longer
  a quarantine strike. Only a timeout (and an allocation breach) counts;
  a refusal neither adds to nor resets the timeout streak; a completed run
  (value or throw) resets it. On a 1 Hz trigger with a 2 s budget a single
  4 s overrun is one timeout plus two refusals and must not quarantine.
  Tests updated across the guard, the alarm engine, the Calculation
  evaluator and the VT evaluator, plus
  Refusals_neither_count_toward_nor_reset_the_timeout_streak.
- I3 ScriptedAlarmEngine: the quality bucket is tracked only after the
  predicate completes (or in the inputs-not-ready path), so the tracked
  bucket is the last one annotated. A refused or quarantined predicate
  used to re-track the input bucket Good before the attempt and then flip
  it Bad, emitting QualityChanged on every push. The run-guard tests now
  count emissions (exactly 1 across N pushes, refused and quarantined
  cases), with a healthy sibling's dispatch count as the positive control
  that every push was processed. Both were red before the fix.

Refs #734

Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 13:26:12 -04:00
Joseph Doherty 909b061d3a Merge branch 'master' into remed/lane-E 2026-10-22 10:48:41 -04:00
Joseph Doherty 4d3fc9c816 fix(runtime): restart-safe alarm host, mux last-value replay, VT log rate limit, Calculation remediation (E4)
- R02:H1 ScriptedAlarmHostActor gains a factory Props
  (Func<(upstream, engine)>) so every incarnation builds and owns its own
  engine; a supervision restart no longer re-creates the actor around a
  disposed engine. The legacy instance-taking Props stays source-compatible
  and is now restart-safe in the weaker sense (PreRestart marks the stop as
  a restart, PostStop then detaches instead of disposing). DriverHostActor
  still spawns through the legacy overload (not owned by this lane — F3).
- R02:H3 OnAlarmCommand runs every engine op under a 10 s token linked to
  the actor's lifetime (DefaultCommandTimeout / Props commandTimeout) and
  logs a Warning on timeout instead of suspending the mailbox on a stalled
  store.
- R02:L3 DependencyMuxTagUpstreamSource.RetainOnly(loaded dep-ref union),
  called after every successful load.
- R02:P6 VirtualTagActor's per-failure ScriptLogEntry is rate-limited per
  tag (ScriptLogRateLimiter, 20/min + one summary); the Bad publish and the
  metric are untouched.
- R07:H6 DependencyMuxActor retains the last (value, status, ts) per
  ever-registered ref (through unregisters) and replays them in one batch to
  a RegisterInterest(ReplayRetained: true) — VirtualTagActor, the Calculation
  mux adapter and the alarm host opt in; the historization recorder does not
  (a replay there would record a stale sample per deploy). CalculationDriver
  records inputs regardless of _running so a replay landing before
  InitializeAsync still satisfies the change-gate.
- R07:P4 CalculationEvaluator negative-caches compile failures by source
  hash (one Roslyn attempt + one Warning per source until redeploy); the
  driver's script-log entry fires on the Good→Bad transition or a change of
  reason, matching the once-per-transition Bad publish.
- R07:H8 array results dedup structurally.
- R07:L2 ShutdownAsync drops the compiled scripts (collectible ALCs).
- R07:U5 DraftValidator rule CalculationNoEffectiveTrigger: a tag with no
  timer whose change trigger is off or whose script has no literal
  ctx.GetTag read can never evaluate.

Refs #734

Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 10:27:49 -04:00
Joseph Doherty 6cbf763719 fix(alarms): engine cache prune + bounded dispose, source Warning, cached dead-letter depth, script-log rate limit (E3)
- R05:1/12 ScriptedAlarmEngine: _valueCache is cleared in LoadAsync (it is
  re-seeded for the referenced paths) so retired tags no longer accrete across
  deploys; OnUpstreamChange no longer writes the cache after Dispose.
- R05:3 ScriptedAlarmEngine.Dispose drains in-flight work for at most 5 s
  (DisposeDrainTimeout), then cancels it through a shutdown token every
  DrainDirtyAsync/ShelvingCheckAsync now runs on, logs a Warning and gives
  one second of grace — a stalled IAlarmStateStore.SaveAsync can no longer
  hang host shutdown. A save that completes inside the bound is still drained
  (positive control test).
- R05:6 ScriptedAlarmSource logs a throwing OnAlarmEvent subscriber at
  Warning (naming the alarm) instead of swallowing it.
- R05:7 LocalDbStoreAndForwardSink keeps a cached dead-letter counter in step
  with every dead-letter/revive/purge path and resyncs it with the queued
  counter on the same cadence; GetStatus no longer runs a COUNT(*) scan.
- R05:9 ScriptLogRateLimiter (20 per source per minute, fixed window, one
  summary entry naming the dropped count when the window rolls) behind
  ScriptLogTopicSink (key = script|vtag|alarm) and ScriptLogCompanionSink
  (key = script name).

Refs #734

Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 10:17:30 -04:00
Joseph Doherty 2e9160317f feat(scripting): deny System.Threading, Type reflection entry points and RuntimeHelpers in the sandbox (E2)
R05:10 — the namespace deny-list covered System.Threading.Tasks but not
System.Threading, so SemaphoreSlim / Monitor / SpinWait / Mutex /
EventWaitHandle could park a script's pool thread past its budget (or touch a
system-global kernel object); System.Type.GetType(string) / GetMethod* /
InvokeMember and System.Runtime.CompilerServices.RuntimeHelpers resolved and
reach a forbidden type without naming it syntactically.

ForbiddenTypeAnalyzer now denies the whole System.Threading namespace with a
type-granular allow-list (CancellationToken, Interlocked, Volatile — the
sync-free primitives), adds a member deny-list for System.Type's reflection
entry points (GetType/GetMethod(s)/InvokeMember/GetProperty(ies)/GetField(s)/
GetMember(s)/GetConstructor(s)/GetEvent(s)/GetDefaultMembers; instance
object.GetType() stays allowed) and puts RuntimeHelpers on the type deny-list.
Thread/ThreadPool/Timer leave the type list — the prefix rule reaches them.

R05:11 — the allocation budget landed with E1's ScriptRunGuard
(Scripting:MaxAllocBytesPerRun, default 64 MB): sampled around each run,
breach quarantines. It is observation after the fact; a pre-emptive OOM guard
still needs an out-of-process runner (follow-up issue in the lane report).

The Monaco editor shares the analyzer, so DiagnoseTests pins every new form
as an OTSCRIPT_FORBIDDEN marker and every allow-listed form as clean.

Refs #734

Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 06:45:03 -04:00
Joseph Doherty eb9269c337 feat(scripting): ScriptRunGuard quarantines runaway scripts across every seam (E1)
A timed-out script is abandoned, not stopped, and every seam re-ran it on the
next trigger, leaking one pool thread per trigger (R03:1.1, R05:2, R07:L1).
A shared ScriptRunGuard, keyed per script instance, now:

- refuses a run while that script's previous orphan is still executing
  (Bad "previous evaluation still running" + otopcua.script.run_refused);
- quarantines after 3 consecutive non-completions or an allocation-budget
  breach (sampled with GC.GetAllocatedBytesForCurrentThread; default 64 MB):
  no further dispatch, Bad quality, one Error, otopcua.script.quarantined,
  cleared on source change / redeploy / reset;
- starts the budget clock when the pool picks the run up, so starvation is
  no longer charged to the script (R07:P5), and cancels the evaluator token
  on timeout.

Wired into RoslynVirtualTagEvaluator, CalculationEvaluator (new async
EvaluateAsync; the driver's pump and timer ticks now await the run behind a
SemaphoreSlim gate) and ScriptedAlarmEngine (a non-completing predicate
holds state and annotates the condition BadInternalError). ScriptingOptions
(Scripting:MaxAllocBytesPerRun, Scripting:QuarantineAfterConsecutiveTimeouts)
carries the knobs; the Calculation driver config accepts the same two keys.

Tests: guard unit suite (incl. the 30-trigger runaway leaving one orphan with
a healthy sibling as positive control), per-seam refusal/quarantine/reset
tests. The S12 race test now uses one vtag id per loop: twelve concurrent
evaluations of ONE id is a shape production never produces and the guard
refuses by design.

Refs #734

Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 06:39:43 -04:00
Joseph Doherty 73e74e270d fix(controlplane): validate and compose a deployment from ONE snapshot read
StartDeployment read the whole config DB twice with no transaction — DraftSnapshotFactory for
the validator, then ConfigComposer for the artifact, plus clusters/nodes and Script.SourceCode
again in between. An AdminUI save landing between the two shipped an artifact the DraftValidator
never saw.

- DraftSnapshot carries ServerClusters, ClusterNodes and ApiKeys; DraftSnapshotFactory reads
  every table inside one SNAPSHOT transaction on a relational provider (no-op on InMemory or
  under an ambient transaction).
- ConfigComposer.Compose(DraftSnapshot) is pure over the snapshot (ordinal-sorted keys);
  SnapshotAndFlattenAsync(db) delegates to it, byte-identical.
- AdminOperationsActor validates topology, estimates compile cost and composes the artifact
  from the one snapshot it holds.
- Migration EnableSnapshotIsolation: ALTER DATABASE CURRENT SET ALLOW_SNAPSHOT_ISOLATION ON
  (suppressTransaction; no model change).

Tests: the SQL Server snapshot read ignores a row committed mid-read (positive control: a later
read sees it); an edit injected between the Equipment and Tags reads is absent from the shipped
artifact and rejected by the next deploy; Compose(snapshot) == SnapshotAndFlattenAsync(db).

Refs #733

Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 06:36:12 -04:00
Joseph Doherty eb23053700 fix(runtime): dispose drivers on final stop, bound PostStop shutdown (A1)
DriverInstanceActor.PostStop now bounds IDriver.ShutdownAsync by a
shutdownTimeout (default 5 s, logged on overrun) and then disposes the
driver (IAsyncDisposable preferred, else IDisposable) — but only on a
real stop. PreRestart flags the restart so the PostStop it triggers
skips the dispose, because a supervision restart re-uses the same
driver object (Props closure) and the host re-initialises it.

DiscoveryDriverBrowser disposes its throwaway capture/probe driver
after the bounded shutdown, including after a failed or timed-out one.

Review refs: 02/L1, 02/P4, 06/G-1 (host context), 07/L2.

Refs #731

Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 06:23:39 -04:00
Joseph Doherty c8e3e279b4 Retain and replay ServiceLevel across publisher binding and failures
v2-ci / unit-tests (pull_request) Waiting to run
2026-10-13 13:26:27 -04:00
Joseph Doherty 109163621c fix(historian): preserve stable alarm delivery identities
v2-ci / build (pull_request) Successful in 5m52s
v2-ci / unit-tests (pull_request) Failing after 1h23m55s
2026-10-13 07:10:58 -04:00
Joseph Doherty 3c3513ea3e fix(historian): #694 page raw reads at a strict gateway's learned ceiling; opt-in EventReadHorizon
Item 1: IHistorianDataSource.MaxRawValuesPerRead (Gateway: learned ceiling - 16 = L; 0 elsewhere, guard-tested).
RawPageLimit plans every raw page on top of #692's lookahead: with L > 0 the read (page P + boundary ties k
+ lookahead) is <= L, P = min(N, L - k - 1) (uncapped: L - k - 1); no room => straight to the tie-cluster
over-fetch (never an uncapped or underflowing request). One re-plan + re-read when the read taught a new
limit or was refused for crowding pre-window rows (HistoryReadLimitExceededException.PreWindowRows): the
page shrinks by them. Over-fetch bound min(MaxTieClusterOverfetch, L - 1); slice size never 0.
Review fixes: F1 a window starting at/before the epoch counts AVEVA's synthetic row as pre-window (a
ceiling-truncated read was reported complete); F2 the page path disables ReverseRawReader's own re-run
(cold end-only ceiling 20/N=3000 cost 3,023 reads, now <= 40); F3 crowded look-back re-read around, not
refused; F4 the #687 crowding re-read room at least doubles (up to 8 re-reads), so a cluster of hundreds
just before a resume point no longer fails on a default gateway; a crowding refusal that leaves no room
is BadTooManyOperations, never a tick-by-tick cursor.

Item 2: ServerHistorian:EventReadHorizon (default 00:00:00 = exact, unchanged). When set, every fresh
HistoryReadEvents window's earlier bound is raised to now - horizon (both directions, all backends).
Lossy by design; documented.

Item 3 (native TimeAverage/Total re-check) stays blocked on historiangw#27.
2026-10-13 01:39:32 -04:00
Joseph Doherty 574c3aa1b0 fix(historian): #691.4 review — Null reports SupportsReverseEvents = true, like raw
The null source has no history in either direction; reporting false
made the server walk an empty reverse events read back through three
empty forward reads. It now answers the reverse overload with the empty
result itself (guard test unchanged: it still declares both members).
2026-10-12 17:37:32 -04:00
Joseph Doherty 410c8d8976 fix(historian): #686 serve reverse/end-only raw reads on forward-only backends (Gateway)
#669 turned an end-only capped HistoryRead-Raw into a reverse read (start > end)
for every backend; the gateway rejects start > end (InvalidArgument), so on the
default Gateway backend every end-only capped and every explicit reverse raw read
failed the node and was counted as a backend health failure.

- IHistorianDataSource.SupportsReverseRaw (default false; SqlServer/QuestDB/Null
  true, Gateway false; guard test pins that every production backend declares it).
- OtOpcUaNodeManager sends a reverse window only to a backend that declares it;
  otherwise ReverseRawReader walks (lo, hi] back in bounded forward segment reads
  (1 min, x4 while sparse; each <= MaxTieClusterOverfetch + 1 rows; a full read is
  re-read narrower, never answered truncated; filtered on sample timestamps, which
  also drops AVEVA's fabricated query-start rows; stops at the Unix epoch). A
  one-timestamp tie cluster over the bound => BadTooManyOperations.
- ServeRawPagedAsync maps HistoryReadLimitExceededException to BadTooManyOperations.
- GatewayHistorianDataSource.ReadRawAsync refuses start > end unsent, without
  recording a health failure.
2026-10-12 14:42:27 -04:00
Joseph Doherty 6ea3ba44da feat(historian): advertise only the backend's aggregates under AggregateFunctions (#682)
IHistorianDataSource.SupportedAggregates (default: every HistoryAggregateType)
is the one authority for both ServerCapabilities/AggregateFunctions (i=2997) and
HistoryServerCapabilities/AggregateFunctions (i=11201) and for the up-front
BadAggregateNotSupported in HistoryRead-Processed. Gateway declares
AggregateModeMapper.Served (4), SqlServer/QuestDB AggregateFamilies.Served (17),
Null none. Only the folder references are trimmed; the SDK's calculator
factories stay so live AggregateFilter subscriptions keep working.

Co-Authored-By: Claude <noreply@anthropic.com>
2026-10-12 08:52:47 -04:00
Joseph Doherty 93d6c1da67 fix(historian): #675 review follow-ups — SupportsReverseEvents capability, no catch-as-unsupported
- IHistorianDataSource.SupportsReverseEvents (default false; SqlServer/QuestDB true; Gateway/Null declare false).
  The node manager checks it BEFORE calling the source: an end-only capped events read becomes reverse only on
  a supporting backend (Gateway, the default, keeps the pre-#675 forward [MinValue, EndTime) paging), and an
  explicit reverse read on a non-supporting backend is BadHistoryOperationUnsupported without a call.
- Removed the broad catch (NotSupportedException) in the events path: a NotSupportedException from inside a
  backend is now logged as a backend fault (Error), not an 'unsupported' Warning.
- Null declares the reverse overload itself; guard test: every production IHistorianDataSource declares both
  the reverse overload and the capability (positive-control scan of all four backends).
- Conformance: capability reported + oversized reverse tie cluster (forward over-fetch reversed == reverse
  read order; capped reverse read stopping inside the cluster flags truncation) on SqlServer + QuestDB.
- docs/Historian.md: end-only capped events read is backward on SqlServer/QuestDB, forward on Gateway.
2026-10-11 11:36:12 -04:00
Joseph Doherty 8d154c6a81 feat(historian): #675 reverse HistoryRead-Events paging + end-only backward reads
- IHistorianDataSource.ReadEventsAsync(..., bool reverse, ...) overload: reverse = Part 11 window (lo, hi]
  newest-first by (time, id) DESC. Default forwards forward reads and throws NotSupportedException for
  reverse (Null); Gateway refuses explicitly (ascending-only stream). Server answers
  BadHistoryOperationUnsupported instead of silently returning the earliest N ascending.
- SqlServer / QuestDB serve reverse natively (ORDER BY time DESC, id DESC + TOP/LIMIT).
- Node manager: reverse event reads page with continuation points; direction stored in the cursor
  (HistoryContinuationState.Reverse) so a resume continues in the cursor's direction; reverse tie clusters
  over-fetched forward and sliced from the newest end (SliceTieCluster(reverse:)); end-only reads with a
  cap page backward from EndTime (mirrors raw).
- Resumed event pages over-fetch by the boundary skip, so every non-final page is a full cap page.
- Tests: node-manager reverse paging (ties, clusters, coarse resolution, overfetch bound, resume
  direction both ways, end-only, unsupported backend); shared EventReadConformance on SqlServer +
  QuestDB; over-the-wire reverse paging; offline Gateway pin of the UTC DateTime.MaxValue bound through
  the published client's proto conversion (in-process gRPC capture) and FILETIME.
- docs/Historian.md event paging section.
2026-10-11 11:19:54 -04:00
Joseph Doherty 11b703708c feat(historian): QuestDB backend skeleton — options, connection, schema bootstrap with TTL
Claude-Session: https://claude.ai/code/session_01Ef2ozxS7QPanGT8uoATbTe
2026-10-11 01:52:57 -04:00
Joseph Doherty 8c6d6aff6a refactor(historian): extract Driver.Historian.Sql.Common from the SQL Server backend (no behaviour change) (1a)
Moves the database-independent logic of the SQL Server historian backend into a new
shared library for the upcoming QuestDB backend (#666): status codes and bits, value/column
mapping, StoredSample + quality clamp, the AtTime and aggregate evaluators, BucketPlan,
LiveWritePreparer and HistoryUpdatePlanner (both with a resolutionTicks parameter, default 1),
tag-name rules, the bootstrap failure cache, health counters, alarm event ids, the event
continuation and severity clamp. The SQL Server backend now calls it; its T-SQL, TVP
builders, connection, bootstrapper and catalog stay.

Tests: StatusCodeOracleTests, ValueWriterPrepareTests, the column half of SampleRowMapperTests
and ProcessedReadTests.Every_enum_member_is_either_served_or_refused move to the new
Sql.Common.Tests under the same names. SqlServer integration tests get mechanical renames
only. Test-name set across the three projects is identical to the baseline (160 tests).

Claude-Session: https://claude.ai/code/session_01Ef2ozxS7QPanGT8uoATbTe
2026-10-11 01:26:49 -04:00
Joseph Doherty 17962d4233 feat(historian): SQL Server backend skeleton - schema DDL, locked bootstrapper, options, connection/mapper/tag-catalog helpers, status consts, env-gated fixture
Claude-Session: https://claude.ai/code/session_01Ef2ozxS7QPanGT8uoATbTe
2026-10-09 12:52:15 -04:00
Joseph Doherty ff7c6c531a feat(historian): typed HistorianValue contracts, IHistorianDataUpdater, IHistorianBackendProvider (no behaviour change)
Claude-Session: https://claude.ai/code/session_01Ef2ozxS7QPanGT8uoATbTe
2026-10-09 12:44:19 -04:00