Commit Graph

283 Commits

Author SHA1 Message Date
Joseph Doherty 7bcb6331ed Merge lane C: control plane, host, security (#733)
Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 14:09:08 -04:00
Joseph Doherty a85d1c069a fix(scripting): quarantine a refusal behind a stuck orphan (fix round 2)
Ruling refinement on I1. Only timeouts are strikes, so a never-returning
while(true), which times out once and is then only refused, was never
quarantined, and that defeated E1. A refusal now quarantines immediately
when the orphan it was refused behind has been executing for longer than
ScriptingOptions.StuckOrphanQuarantineFactor x the script's timeout
(default 10, so 20 s at the 2 s default; 0 or less disables it). It uses
the same quarantine path: one Error, otopcua.script.quarantined with
reason=stuck, Bad, lifted on source change or redeploy. A young orphan
(a short overrun) is still a plain refusal.

The guard stamps the run's start (Stopwatch timestamp) when the pool
picks it up and clears it when the run really finishes.

Tests: the 30-trigger while(true) regression (paced) now ends
quarantined with reason "stuck", still with exactly one orphan and a
healthy sibling. Positive control: a 2x overrun is refused, not
quarantined, and evaluates again once the orphan completes. A factor of
0 disables the rule. Docs (ScriptEditor "Runtime safety", VirtualTags)
and the seam remarks are updated.

Refs #734

Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 13:59:10 -04:00
Joseph Doherty ef2eebe6b6 fix(scripting): review minors: configured limit in summaries, dispose gate/CTS, validator, docs (fix round 1)
- ScriptLogRateLimiter.LimitPerWindow. The ScriptLogTopicSink and
  VirtualTagActor summaries now print the configured limit, not
  DefaultLimitPerWindow. Tests assert "limit N/window" for a non-default
  N.
- CalculationDriver.Dispose cancels, then takes _evalGate (bounded 2 s)
  and disposes the gate and _evalCts. The pump and timer ticks enter the
  gate through TryEnterEvalGateAsync, which stops cleanly once the
  driver is disposed. Test: Dispose with a 6 s runaway run in flight
  returns promptly, disposes both, and a late trigger is harmless.
- ScriptedAlarmEngine captures _shutdownToken once and disposes
  _shutdownCts in Dispose when every drained task has completed. A task
  stuck past the grace leaves the CTS to the GC. Lifecycle tests assert
  the disposal.
- DraftValidator: CalculationNoEffectiveTrigger is skipped for a
  change-triggered tag whose scriptId does not resolve. That case is
  already CalculationScriptMissing or CalculationScriptNotFound, and
  there are no reads to count. Both existing tests now assert the
  absence (red before the change).
- docs/VirtualTags.md: System.Threading is denied as a whole namespace,
  with an allow-list; the type list is now accurate (Unsafe,
  RuntimeHelpers, and the ForbiddenMembers on System.Type); the resource
  limits section covers the run guard and the allocation budget.
  docs/ScriptEditor.md: accurate deny-list and a new "Runtime safety"
  section covering timeout, refusal, quarantine after 3 timeouts,
  allocation budget, lifting, and rate-limit summaries that appear only
  on the source's next entry after a burst then quiet.

Refs #734

Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 13:41:15 -04:00
Joseph Doherty 4d17b346c6 Merge branch 'master' into remed/lane-C 2026-10-22 13:31:03 -04:00
Joseph Doherty 6aa5edaf30 fix(scripting): only timeouts are quarantine strikes; alarm Bad annotation once per bucket change (fix round 1 I1, I3)
- I1 ScriptRunGuard: a run refused because the previous one is still
  executing is a Bad publish plus otopcua.script.run_refused, and no longer
  a quarantine strike. Only a timeout (and an allocation breach) counts;
  a refusal neither adds to nor resets the timeout streak; a completed run
  (value or throw) resets it. On a 1 Hz trigger with a 2 s budget a single
  4 s overrun is one timeout plus two refusals and must not quarantine.
  Tests updated across the guard, the alarm engine, the Calculation
  evaluator and the VT evaluator, plus
  Refusals_neither_count_toward_nor_reset_the_timeout_streak.
- I3 ScriptedAlarmEngine: the quality bucket is tracked only after the
  predicate completes (or in the inputs-not-ready path), so the tracked
  bucket is the last one annotated. A refused or quarantined predicate
  used to re-track the input bucket Good before the attempt and then flip
  it Bad, emitting QualityChanged on every push. The run-guard tests now
  count emissions (exactly 1 across N pushes, refused and quarantined
  cases), with a healthy sibling's dispatch count as the positive control
  that every push was processed. Both were red before the fix.

Refs #734

Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 13:26:12 -04:00
Joseph Doherty 909b061d3a Merge branch 'master' into remed/lane-E 2026-10-22 10:48:41 -04:00
Joseph Doherty 4d3fc9c816 fix(runtime): restart-safe alarm host, mux last-value replay, VT log rate limit, Calculation remediation (E4)
- R02:H1 ScriptedAlarmHostActor gains a factory Props
  (Func<(upstream, engine)>) so every incarnation builds and owns its own
  engine; a supervision restart no longer re-creates the actor around a
  disposed engine. The legacy instance-taking Props stays source-compatible
  and is now restart-safe in the weaker sense (PreRestart marks the stop as
  a restart, PostStop then detaches instead of disposing). DriverHostActor
  still spawns through the legacy overload (not owned by this lane — F3).
- R02:H3 OnAlarmCommand runs every engine op under a 10 s token linked to
  the actor's lifetime (DefaultCommandTimeout / Props commandTimeout) and
  logs a Warning on timeout instead of suspending the mailbox on a stalled
  store.
- R02:L3 DependencyMuxTagUpstreamSource.RetainOnly(loaded dep-ref union),
  called after every successful load.
- R02:P6 VirtualTagActor's per-failure ScriptLogEntry is rate-limited per
  tag (ScriptLogRateLimiter, 20/min + one summary); the Bad publish and the
  metric are untouched.
- R07:H6 DependencyMuxActor retains the last (value, status, ts) per
  ever-registered ref (through unregisters) and replays them in one batch to
  a RegisterInterest(ReplayRetained: true) — VirtualTagActor, the Calculation
  mux adapter and the alarm host opt in; the historization recorder does not
  (a replay there would record a stale sample per deploy). CalculationDriver
  records inputs regardless of _running so a replay landing before
  InitializeAsync still satisfies the change-gate.
- R07:P4 CalculationEvaluator negative-caches compile failures by source
  hash (one Roslyn attempt + one Warning per source until redeploy); the
  driver's script-log entry fires on the Good→Bad transition or a change of
  reason, matching the once-per-transition Bad publish.
- R07:H8 array results dedup structurally.
- R07:L2 ShutdownAsync drops the compiled scripts (collectible ALCs).
- R07:U5 DraftValidator rule CalculationNoEffectiveTrigger: a tag with no
  timer whose change trigger is off or whose script has no literal
  ctx.GetTag read can never evaluate.

Refs #734

Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 10:27:49 -04:00
Joseph Doherty 6cbf763719 fix(alarms): engine cache prune + bounded dispose, source Warning, cached dead-letter depth, script-log rate limit (E3)
- R05:1/12 ScriptedAlarmEngine: _valueCache is cleared in LoadAsync (it is
  re-seeded for the referenced paths) so retired tags no longer accrete across
  deploys; OnUpstreamChange no longer writes the cache after Dispose.
- R05:3 ScriptedAlarmEngine.Dispose drains in-flight work for at most 5 s
  (DisposeDrainTimeout), then cancels it through a shutdown token every
  DrainDirtyAsync/ShelvingCheckAsync now runs on, logs a Warning and gives
  one second of grace — a stalled IAlarmStateStore.SaveAsync can no longer
  hang host shutdown. A save that completes inside the bound is still drained
  (positive control test).
- R05:6 ScriptedAlarmSource logs a throwing OnAlarmEvent subscriber at
  Warning (naming the alarm) instead of swallowing it.
- R05:7 LocalDbStoreAndForwardSink keeps a cached dead-letter counter in step
  with every dead-letter/revive/purge path and resyncs it with the queued
  counter on the same cadence; GetStatus no longer runs a COUNT(*) scan.
- R05:9 ScriptLogRateLimiter (20 per source per minute, fixed window, one
  summary entry naming the dropped count when the window rolls) behind
  ScriptLogTopicSink (key = script|vtag|alarm) and ScriptLogCompanionSink
  (key = script name).

Refs #734

Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 10:17:30 -04:00
Joseph Doherty 2e9160317f feat(scripting): deny System.Threading, Type reflection entry points and RuntimeHelpers in the sandbox (E2)
R05:10 — the namespace deny-list covered System.Threading.Tasks but not
System.Threading, so SemaphoreSlim / Monitor / SpinWait / Mutex /
EventWaitHandle could park a script's pool thread past its budget (or touch a
system-global kernel object); System.Type.GetType(string) / GetMethod* /
InvokeMember and System.Runtime.CompilerServices.RuntimeHelpers resolved and
reach a forbidden type without naming it syntactically.

ForbiddenTypeAnalyzer now denies the whole System.Threading namespace with a
type-granular allow-list (CancellationToken, Interlocked, Volatile — the
sync-free primitives), adds a member deny-list for System.Type's reflection
entry points (GetType/GetMethod(s)/InvokeMember/GetProperty(ies)/GetField(s)/
GetMember(s)/GetConstructor(s)/GetEvent(s)/GetDefaultMembers; instance
object.GetType() stays allowed) and puts RuntimeHelpers on the type deny-list.
Thread/ThreadPool/Timer leave the type list — the prefix rule reaches them.

R05:11 — the allocation budget landed with E1's ScriptRunGuard
(Scripting:MaxAllocBytesPerRun, default 64 MB): sampled around each run,
breach quarantines. It is observation after the fact; a pre-emptive OOM guard
still needs an out-of-process runner (follow-up issue in the lane report).

The Monaco editor shares the analyzer, so DiagnoseTests pins every new form
as an OTSCRIPT_FORBIDDEN marker and every allow-listed form as clean.

Refs #734

Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 06:45:03 -04:00
Joseph Doherty eb9269c337 feat(scripting): ScriptRunGuard quarantines runaway scripts across every seam (E1)
A timed-out script is abandoned, not stopped, and every seam re-ran it on the
next trigger, leaking one pool thread per trigger (R03:1.1, R05:2, R07:L1).
A shared ScriptRunGuard, keyed per script instance, now:

- refuses a run while that script's previous orphan is still executing
  (Bad "previous evaluation still running" + otopcua.script.run_refused);
- quarantines after 3 consecutive non-completions or an allocation-budget
  breach (sampled with GC.GetAllocatedBytesForCurrentThread; default 64 MB):
  no further dispatch, Bad quality, one Error, otopcua.script.quarantined,
  cleared on source change / redeploy / reset;
- starts the budget clock when the pool picks the run up, so starvation is
  no longer charged to the script (R07:P5), and cancels the evaluator token
  on timeout.

Wired into RoslynVirtualTagEvaluator, CalculationEvaluator (new async
EvaluateAsync; the driver's pump and timer ticks now await the run behind a
SemaphoreSlim gate) and ScriptedAlarmEngine (a non-completing predicate
holds state and annotates the condition BadInternalError). ScriptingOptions
(Scripting:MaxAllocBytesPerRun, Scripting:QuarantineAfterConsecutiveTimeouts)
carries the knobs; the Calculation driver config accepts the same two keys.

Tests: guard unit suite (incl. the 30-trigger runaway leaving one orphan with
a healthy sibling as positive control), per-seam refusal/quarantine/reset
tests. The S12 race test now uses one vtag id per loop: twelve concurrent
evaluations of ONE id is a shape production never produces and the guard
refuses by design.

Refs #734

Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 06:39:43 -04:00
Joseph Doherty 73e74e270d fix(controlplane): validate and compose a deployment from ONE snapshot read
StartDeployment read the whole config DB twice with no transaction — DraftSnapshotFactory for
the validator, then ConfigComposer for the artifact, plus clusters/nodes and Script.SourceCode
again in between. An AdminUI save landing between the two shipped an artifact the DraftValidator
never saw.

- DraftSnapshot carries ServerClusters, ClusterNodes and ApiKeys; DraftSnapshotFactory reads
  every table inside one SNAPSHOT transaction on a relational provider (no-op on InMemory or
  under an ambient transaction).
- ConfigComposer.Compose(DraftSnapshot) is pure over the snapshot (ordinal-sorted keys);
  SnapshotAndFlattenAsync(db) delegates to it, byte-identical.
- AdminOperationsActor validates topology, estimates compile cost and composes the artifact
  from the one snapshot it holds.
- Migration EnableSnapshotIsolation: ALTER DATABASE CURRENT SET ALLOW_SNAPSHOT_ISOLATION ON
  (suppressTransaction; no model change).

Tests: the SQL Server snapshot read ignores a row committed mid-read (positive control: a later
read sees it); an edit injected between the Equipment and Tags reads is absent from the shipped
artifact and rejected by the next deploy; Compose(snapshot) == SnapshotAndFlattenAsync(db).

Refs #733

Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 06:36:12 -04:00
Joseph Doherty eb23053700 fix(runtime): dispose drivers on final stop, bound PostStop shutdown (A1)
DriverInstanceActor.PostStop now bounds IDriver.ShutdownAsync by a
shutdownTimeout (default 5 s, logged on overrun) and then disposes the
driver (IAsyncDisposable preferred, else IDisposable) — but only on a
real stop. PreRestart flags the restart so the PostStop it triggers
skips the dispose, because a supervision restart re-uses the same
driver object (Props closure) and the host re-initialises it.

DiscoveryDriverBrowser disposes its throwaway capture/probe driver
after the bounded shutdown, including after a failed or timed-out one.

Review refs: 02/L1, 02/P4, 06/G-1 (host context), 07/L2.

Refs #731

Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 06:23:39 -04:00
Joseph Doherty c8e3e279b4 Retain and replay ServiceLevel across publisher binding and failures
v2-ci / unit-tests (pull_request) Waiting to run
2026-10-13 13:26:27 -04:00
Joseph Doherty 109163621c fix(historian): preserve stable alarm delivery identities
v2-ci / build (pull_request) Successful in 5m52s
v2-ci / unit-tests (pull_request) Failing after 1h23m55s
2026-10-13 07:10:58 -04:00
Joseph Doherty 3c3513ea3e fix(historian): #694 page raw reads at a strict gateway's learned ceiling; opt-in EventReadHorizon
Item 1: IHistorianDataSource.MaxRawValuesPerRead (Gateway: learned ceiling - 16 = L; 0 elsewhere, guard-tested).
RawPageLimit plans every raw page on top of #692's lookahead: with L > 0 the read (page P + boundary ties k
+ lookahead) is <= L, P = min(N, L - k - 1) (uncapped: L - k - 1); no room => straight to the tie-cluster
over-fetch (never an uncapped or underflowing request). One re-plan + re-read when the read taught a new
limit or was refused for crowding pre-window rows (HistoryReadLimitExceededException.PreWindowRows): the
page shrinks by them. Over-fetch bound min(MaxTieClusterOverfetch, L - 1); slice size never 0.
Review fixes: F1 a window starting at/before the epoch counts AVEVA's synthetic row as pre-window (a
ceiling-truncated read was reported complete); F2 the page path disables ReverseRawReader's own re-run
(cold end-only ceiling 20/N=3000 cost 3,023 reads, now <= 40); F3 crowded look-back re-read around, not
refused; F4 the #687 crowding re-read room at least doubles (up to 8 re-reads), so a cluster of hundreds
just before a resume point no longer fails on a default gateway; a crowding refusal that leaves no room
is BadTooManyOperations, never a tick-by-tick cursor.

Item 2: ServerHistorian:EventReadHorizon (default 00:00:00 = exact, unchanged). When set, every fresh
HistoryReadEvents window's earlier bound is raised to now - horizon (both directions, all backends).
Lossy by design; documented.

Item 3 (native TimeAverage/Total re-check) stays blocked on historiangw#27.
2026-10-13 01:39:32 -04:00
Joseph Doherty 574c3aa1b0 fix(historian): #691.4 review — Null reports SupportsReverseEvents = true, like raw
The null source has no history in either direction; reporting false
made the server walk an empty reverse events read back through three
empty forward reads. It now answers the reverse overload with the empty
result itself (guard test unchanged: it still declares both members).
2026-10-12 17:37:32 -04:00
Joseph Doherty 410c8d8976 fix(historian): #686 serve reverse/end-only raw reads on forward-only backends (Gateway)
#669 turned an end-only capped HistoryRead-Raw into a reverse read (start > end)
for every backend; the gateway rejects start > end (InvalidArgument), so on the
default Gateway backend every end-only capped and every explicit reverse raw read
failed the node and was counted as a backend health failure.

- IHistorianDataSource.SupportsReverseRaw (default false; SqlServer/QuestDB/Null
  true, Gateway false; guard test pins that every production backend declares it).
- OtOpcUaNodeManager sends a reverse window only to a backend that declares it;
  otherwise ReverseRawReader walks (lo, hi] back in bounded forward segment reads
  (1 min, x4 while sparse; each <= MaxTieClusterOverfetch + 1 rows; a full read is
  re-read narrower, never answered truncated; filtered on sample timestamps, which
  also drops AVEVA's fabricated query-start rows; stops at the Unix epoch). A
  one-timestamp tie cluster over the bound => BadTooManyOperations.
- ServeRawPagedAsync maps HistoryReadLimitExceededException to BadTooManyOperations.
- GatewayHistorianDataSource.ReadRawAsync refuses start > end unsent, without
  recording a health failure.
2026-10-12 14:42:27 -04:00
Joseph Doherty 6ea3ba44da feat(historian): advertise only the backend's aggregates under AggregateFunctions (#682)
IHistorianDataSource.SupportedAggregates (default: every HistoryAggregateType)
is the one authority for both ServerCapabilities/AggregateFunctions (i=2997) and
HistoryServerCapabilities/AggregateFunctions (i=11201) and for the up-front
BadAggregateNotSupported in HistoryRead-Processed. Gateway declares
AggregateModeMapper.Served (4), SqlServer/QuestDB AggregateFamilies.Served (17),
Null none. Only the folder references are trimmed; the SDK's calculator
factories stay so live AggregateFilter subscriptions keep working.

Co-Authored-By: Claude <noreply@anthropic.com>
2026-10-12 08:52:47 -04:00
Joseph Doherty 93d6c1da67 fix(historian): #675 review follow-ups — SupportsReverseEvents capability, no catch-as-unsupported
- IHistorianDataSource.SupportsReverseEvents (default false; SqlServer/QuestDB true; Gateway/Null declare false).
  The node manager checks it BEFORE calling the source: an end-only capped events read becomes reverse only on
  a supporting backend (Gateway, the default, keeps the pre-#675 forward [MinValue, EndTime) paging), and an
  explicit reverse read on a non-supporting backend is BadHistoryOperationUnsupported without a call.
- Removed the broad catch (NotSupportedException) in the events path: a NotSupportedException from inside a
  backend is now logged as a backend fault (Error), not an 'unsupported' Warning.
- Null declares the reverse overload itself; guard test: every production IHistorianDataSource declares both
  the reverse overload and the capability (positive-control scan of all four backends).
- Conformance: capability reported + oversized reverse tie cluster (forward over-fetch reversed == reverse
  read order; capped reverse read stopping inside the cluster flags truncation) on SqlServer + QuestDB.
- docs/Historian.md: end-only capped events read is backward on SqlServer/QuestDB, forward on Gateway.
2026-10-11 11:36:12 -04:00
Joseph Doherty 8d154c6a81 feat(historian): #675 reverse HistoryRead-Events paging + end-only backward reads
- IHistorianDataSource.ReadEventsAsync(..., bool reverse, ...) overload: reverse = Part 11 window (lo, hi]
  newest-first by (time, id) DESC. Default forwards forward reads and throws NotSupportedException for
  reverse (Null); Gateway refuses explicitly (ascending-only stream). Server answers
  BadHistoryOperationUnsupported instead of silently returning the earliest N ascending.
- SqlServer / QuestDB serve reverse natively (ORDER BY time DESC, id DESC + TOP/LIMIT).
- Node manager: reverse event reads page with continuation points; direction stored in the cursor
  (HistoryContinuationState.Reverse) so a resume continues in the cursor's direction; reverse tie clusters
  over-fetched forward and sliced from the newest end (SliceTieCluster(reverse:)); end-only reads with a
  cap page backward from EndTime (mirrors raw).
- Resumed event pages over-fetch by the boundary skip, so every non-final page is a full cap page.
- Tests: node-manager reverse paging (ties, clusters, coarse resolution, overfetch bound, resume
  direction both ways, end-only, unsupported backend); shared EventReadConformance on SqlServer +
  QuestDB; over-the-wire reverse paging; offline Gateway pin of the UTC DateTime.MaxValue bound through
  the published client's proto conversion (in-process gRPC capture) and FILETIME.
- docs/Historian.md event paging section.
2026-10-11 11:19:54 -04:00
Joseph Doherty 11b703708c feat(historian): QuestDB backend skeleton — options, connection, schema bootstrap with TTL
Claude-Session: https://claude.ai/code/session_01Ef2ozxS7QPanGT8uoATbTe
2026-10-11 01:52:57 -04:00
Joseph Doherty 8c6d6aff6a refactor(historian): extract Driver.Historian.Sql.Common from the SQL Server backend (no behaviour change) (1a)
Moves the database-independent logic of the SQL Server historian backend into a new
shared library for the upcoming QuestDB backend (#666): status codes and bits, value/column
mapping, StoredSample + quality clamp, the AtTime and aggregate evaluators, BucketPlan,
LiveWritePreparer and HistoryUpdatePlanner (both with a resolutionTicks parameter, default 1),
tag-name rules, the bootstrap failure cache, health counters, alarm event ids, the event
continuation and severity clamp. The SQL Server backend now calls it; its T-SQL, TVP
builders, connection, bootstrapper and catalog stay.

Tests: StatusCodeOracleTests, ValueWriterPrepareTests, the column half of SampleRowMapperTests
and ProcessedReadTests.Every_enum_member_is_either_served_or_refused move to the new
Sql.Common.Tests under the same names. SqlServer integration tests get mechanical renames
only. Test-name set across the three projects is identical to the baseline (160 tests).

Claude-Session: https://claude.ai/code/session_01Ef2ozxS7QPanGT8uoATbTe
2026-10-11 01:26:49 -04:00
Joseph Doherty 17962d4233 feat(historian): SQL Server backend skeleton - schema DDL, locked bootstrapper, options, connection/mapper/tag-catalog helpers, status consts, env-gated fixture
Claude-Session: https://claude.ai/code/session_01Ef2ozxS7QPanGT8uoATbTe
2026-10-09 12:52:15 -04:00
Joseph Doherty ff7c6c531a feat(historian): typed HistorianValue contracts, IHistorianDataUpdater, IHistorianBackendProvider (no behaviour change)
Claude-Session: https://claude.ai/code/session_01Ef2ozxS7QPanGT8uoATbTe
2026-10-09 12:44:19 -04:00
Joseph Doherty f7d497e532 refactor(#655): delete the dead driver-side history capability
HistoryRead is served server-side by IHistorianDataSource; nothing ever dispatched
to a driver's IHistoryProvider. Removes IHistoryProvider, the OpcUaClient history
passthrough, DriverCapability.HistoryRead (+ resilience tier rows and AdminUI row),
HistoryReadStatusException and its node-manager catches, and the tests that only
covered them. Shared result types and HistoryAggregateNotSupportedException stay
(moved to Historian/HistoryReadContracts.cs). A stored ResilienceConfig still
carrying a HistoryRead key parses (diagnostic only; new test).

Claude-Session: https://claude.ai/code/session_01Ef2ozxS7QPanGT8uoATbTe
2026-10-08 23:34:02 -04:00
Joseph Doherty 46b4ab6947 Merge b647-ci: Gitea-compatible artifact upload + deterministic CI-flaky tests (#606)
Claude-Session: https://claude.ai/code/session_01Ef2ozxS7QPanGT8uoATbTe
2026-10-08 20:45:50 -04:00
Joseph Doherty beccfc6404 test(#606): native-ack routing waits for the driver child to be Connected; widen a starved poll
DriverHostActorNativeAlarmAckRoutingTests: Applied does not mean the child DriverInstanceActor is
Connected, and an ack reaching a still-Connecting child is dropped by design — the single
positive-control ack was itself dropped (4/20 under DOTNET_PROCESSOR_COUNT=1 x8, still failing
with the presence budget dilated to 45 s, so not a budget problem). SpawnHostAndApply now waits
for SubscribeAlarmsAsync, which only the Connected state calls. ScriptedAlarmEngine
OnEvent_emission_happens_outside_evalGate: 2 s blocking poll -> 15 s (5/20 under starvation).

Claude-Session: https://claude.ai/code/session_01Ef2ozxS7QPanGT8uoATbTe
2026-10-08 20:37:15 -04:00
Joseph Doherty 55ae230fa4 ci(#606): Gitea-compatible trx upload + deterministic fixes for the CI-load test flakes
- v2-ci.yml: upload-artifact@v4 -> @v3 + continue-on-error (act_runner reports as GHES and
  v4 refuses GHES, failing every run); runner-capacity diagnostic step; AKKA_TEST_TIMEFACTOR=3
  on the test step.
- Runtime/ControlPlane harnesses: akka.test.timefactor from AKKA_TEST_TIMEFACTOR
  (TestTimeFactor), single-expect-default 15s; ControlPlane gains PresenceBudget.
- Leaked spinner threads: the infinite-loop timeout tests (Calculation, Host.IntegrationTests
  RoslynVirtualTagEvaluator, 3x Core.Scripting) pinned a core for the rest of the test process
  (abandoned, never stopped). Now a bounded 5 s wall-clock spin; outer compile-inclusive bound 30 s.
- Real test races fixed: ScriptedAlarmEngine push-during-drain (Inactive is also the start
  state + persist-before-update => wait for the Cleared save); DriverReconnectE2e (both nodes
  host the driver; Created[] is last-writer-wins and ForceReconnect is a no-op while
  Connecting); ControlPlaneStarterOrder (lost-race heal waited for a 60 s refresh tick).
- Fixed sleeps -> poll-for-condition: ReconnectSupervisor LastError, Modbus deadband steps
  (wait for observed reads), Modbus dispose hygiene precondition, FOCAS probe-loop log.
- Short presence budgets -> PresenceBudget in the Runtime/ControlPlane tests seen failing on
  Gitea; Calculation Settle 10 -> 30 s. No assertion removed, nothing skipped.
- infra/README.md §10 CI; stale TreatWarningsAsErrors comment in Directory.Build.props.

Claude-Session: https://claude.ai/code/session_01Ef2ozxS7QPanGT8uoATbTe
2026-10-08 19:44:31 -04:00
Joseph Doherty b8c2136732 fix(opcuaclient): #649 review minors - CodeBits match, Bad-only exception ctor, BadNoData + Raw-resume tests, latent-impact docs
Claude-Session: https://claude.ai/code/session_01Ef2ozxS7QPanGT8uoATbTe
2026-10-08 19:29:47 -04:00
Joseph Doherty 8801631739 Merge b640-601: honest deploy acks — Applied only after the address-space outcome, degraded flag + reason (#601)
v2-ci / build (push) Successful in 4m42s
v2-ci / unit-tests (push) Failing after 20m19s
Claude-Session: https://claude.ai/code/session_01Ef2ozxS7QPanGT8uoATbTe

# Conflicts:
#	src/Server/ZB.MOM.WW.OtOpcUa.Runtime/OpcUa/OpcUaPublishActor.cs
2026-10-08 18:00:23 -04:00
Joseph Doherty 521e1eb89b feat(deploy): additive degraded flag on ApplyAck + publish-actor apply outcome reply (#601)
ApplyAck gains trailing Degraded/DegradedReason (defaulted, so an old node's
DPS JSON or gRPC ack decodes exactly as before). mesh_transport.proto adds
ApplyAckMessage.degraded = 6 / optional degraded_reason = 7. OpcUaPublishActor
replies RebuildCompleted(Degraded, Reason) on every handler path when the
RebuildAddressSpace carries ReportOutcome.

Claude-Session: https://claude.ai/code/session_01Ef2ozxS7QPanGT8uoATbTe
2026-10-08 17:19:57 -04:00
Joseph Doherty ddd7678269 fix(opcua): carry native alarm condition state across a full rebuild (#641)
A full RebuildAddressSpace recreated every native Part 9 condition inactive +
acknowledged, and when the alarm-tag set was unchanged (e.g. one
UnsTagReference added to an alarm-bearing raw tag) nothing re-told the new node
its state: the host does not re-subscribe for an unchanged set, and Galaxy /
OpcUaClient have no snapshot to replay anyway. Active alarms read normal until
their next device transition.

OtOpcUaNodeManager now captures each NATIVE condition's live state (from the
node, so a client-applied ack is kept) before the rebuild clears, restores it
when the same id is re-materialised native in the same apply, and reports ONE
condition event per restored off-normal condition at the new
IOpcUaAddressSpaceSink.CompleteAddressSpaceApply (after notifier wiring).
Unclaimed captures are discarded there, so a later re-add is fresh. Quality is
carried silently for normal conditions too (it is only re-annotated on a
connectivity edge). Scripted conditions are untouched (#487 re-asserts them).
OpcUaPublishActor calls CompleteAddressSpaceApply at the end of every apply.

Claude-Session: https://claude.ai/code/session_01Ef2ozxS7QPanGT8uoATbTe
2026-10-08 17:02:12 -04:00
Joseph Doherty dcc827ca8c feat(mtconnect): one agent client per driver — a second /raw Device is a deploy error (#628)
Claude-Session: https://claude.ai/code/session_01Ef2ozxS7QPanGT8uoATbTe
2026-10-08 14:29:50 -04:00
Joseph Doherty 4a10ea1e1c fix(runtime): a removed driver's late writes no longer resurrect its resilience/telemetry state (#609)
DriverHostActor.StopChild released the tracker (Invalidate) and evicted the hub synchronously, but
Context.Stop is asynchronous: the stopping child's in-flight SubscribeBulk completed through its
capability invoker AFTER the release, and its RecordCallStart/RecordSuccess/RecordCallComplete
re-created the tracker entry, which DriverResilienceStatusPublisherService then re-emitted every tick
forever. Confirmed with a temporary stack capture in the tracker (removed): every post-Release write
came from DriverInstanceActor.HandleSubscribeAsync -> CapabilityInvoker.ExecuteAsync; the test failed
whenever it landed before the second publisher tick.

- DriverResilienceStatusTracker.Retire (Release) drops the id's snapshots and ignores every later
  Record* for it; one guarded write path re-checks after the upsert so a write racing Retire cannot
  survive it. Reinstate (called by DriverCapabilityInvokerFactory.Create) lifts it, so the same id
  re-added in a later deployment records again.
- TelemetryLocalHub.Evict now also holds the id (late Health/Resilience dropped, neither cached nor
  fanned out); ITelemetryLocalHub.Readmit (default no-op), called at the top of SpawnChild, lifts it.
- The removal test is now deterministic: an invoker decorator holds the child's subscribe inside the
  real invoker's bracket across the removal and signals when its late writes are done (positive
  control) — red 5/5 without the fix, green 20/20 with it. New unit tests for the tracker guard
  (incl. a Retire race), the factory Release/Create pair, the hub hold/readmit, and a re-add-same-id
  host test.

Claude-Session: https://claude.ai/code/session_01Ef2ozxS7QPanGT8uoATbTe
2026-10-08 09:07:11 -04:00
Joseph Doherty 52708bff52 fix(address-space): numeric array writes convert (checked) to the node element type; array-to-scalar refusal only for Modbus/S7 (#610 review R6)
MAJOR-1: R4's exact element-type match refused ScadaBridge's CLR-list
writes (its element types are only Boolean/Int32/Float/Double/String/
DateTime) to every Int16/UInt16/Byte/UInt32/Int64/UInt64 array node and a
Double list to a Float array. EvaluateEquipmentWriteStructure now converts
a numeric array of another numeric element type element-wise, CHECKED, into
the node's element type and hands the converted array back through
OnEquipmentTagWrite's ref value, so the node stores and the driver receives
the node's own type. Any out-of-range, fractional-to-integer, NaN/Inf-to-
integer or Float-overflowing element refuses the whole write BadTypeMismatch
(nothing dispatched) - so the UInt32 -> Int32 bit reinterpretation stays
closed. Non-numeric mismatches stay refused. The wrong 'no client relies on
a looser rule' comment is corrected.

MINOR-1: the array-value-to-scalar-node refusal broke writes back to
browse-committed array tags (committed scalar, isArray never set) on
pass-through drivers. It now applies only when the owning driver's scalar
encoder cannot take an array (DriverArrayWriteSupport.
RefusesArrayValueOnScalarTag: Modbus, S7), carried RawTagPlan.
ScalarRefusesArrayValue / UnsReferenceVariable.BackingScalarRefusesArrayValue
-> EnsureVariable(refuseArrayOnScalar) -> a per-node set in the node
manager. OpcUaClient/Galaxy/others keep pre-#610 dispatch. Not on the
classifier whitelist (driver-type edit rebuilds); a mirror whose flag
changes is recreated, not updated in place.

Claude-Session: https://claude.ai/code/session_01Ef2ozxS7QPanGT8uoATbTe
2026-10-08 08:31:46 -04:00
Joseph Doherty a702875d41 Merge review R4 fixes (#610): array writes require an exact element-type match (BadTypeMismatch, nothing sent); Galaxy encoder refuses uint[]/ulong[]; Modbus array writability bounded by maxRegistersPerWrite/coil cap; arrayLength >= 1 required; Galaxy arrays without a declared length stay read-only; arrays to scalar nodes refused
Claude-Session: https://claude.ai/code/session_01Ef2ozxS7QPanGT8uoATbTe
2026-10-08 08:08:03 -04:00
Joseph Doherty dabd8eb955 fix(address-space): exact element-type check for array writes + array-capability shape edges (#610 review R4)
MAJOR-1: the SDK skips TypeInfo.IsInstanceOfDataType on any node with an
OnWriteValue handler (every writable equipment node), and the node manager's
fail-fast treated numeric<->numeric as compatible, so a UInt32[] to a Galaxy
Int32 array matched MxValueEncoder's int[] arm by CLR covariance and was
written bit-reinterpreted with a Good result. EvaluateEquipmentWriteStructure
now matches an ARRAY write's element type exactly (Byte[]/ByteString the only
equivalence; Variant[]/object[] refused) -> BadTypeMismatch, nothing
dispatched. MxValueEncoder's int[]/long[] arms are exact-type guarded too, and
the Galaxy parity test's false "the SDK refuses" premise is corrected (the
UInt32/UInt64 exclusion is now the encoder's own answer). Scalar writes keep
numeric widening by decision: consumers write CLR-typed values (Double to a
Float node) and every driver's scalar encoder range-checks.

MINOR: Modbus row honours maxRegistersPerWrite (smallest of DriverConfig /
the tag's DeviceConfig, default 123) and FC15's 1968-coil ceiling, and
requires region + numeric address; Modbus/S7/Galaxy require isArray + integer
arrayLength >= 1 (Galaxy: an unbounded node would let a short write resize the
attribute); an array value to a scalar node is refused inline. Docs: Uns.md
array-write rules + upgrade note (ReadWrite array tags become client-writable
on first boot), Modbus.md, Galaxy.md.

Claude-Session: https://claude.ai/code/session_01Ef2ozxS7QPanGT8uoATbTe
2026-10-08 08:07:36 -04:00
Joseph Doherty b99aea46b2 Merge review R1 fixes: removed driver's resilience tracker released; SDK timed-unshelve no-op (engine owns expiry); commented re-ack stored as AddComment; native-ack Primary verdict published from start; drain gate SharesRowsWithPeer; malformed cache bytes don't adopt the revision; HostAbortedException passthrough; ClusterRoleInfo seed filtered
Claude-Session: https://claude.ai/code/session_01Ef2ozxS7QPanGT8uoATbTe
2026-10-08 07:47:57 -04:00
Joseph Doherty e3dcb180c3 fix(runtime): Primary verdict published from host start, cache boot never adopts malformed bytes, strict drain-gate predicate, HostAbortedException passthrough, seed only Up members
Review R1:
- M1 (#609): DriverHostActor.StopChild releases the instance's resilience state before evicting the hub.
- m2 (#597): DriverHostActor publishes ShouldServiceAsPrimary() to IDriverPrimaryRoleSource at start and on
  cluster membership changes (deduped), so the native-ack gate denies in a pair's boot window.
- m4 (#601): ApplyCachedArtifact returns false on empty/malformed bytes; the cached revision is not adopted,
  Direct mode goes Stale (no RUNNING FROM CACHE), FetchAndCache lands Steady-with-no-revision and a
  re-dispatch of the same revision really applies.
- m3 (#603): LocalDbReplicationPolicy splits into CapturesChanges (PeerAddress||ApiKey, registration) and
  SharesRowsWithPeer (ApiKey && (PeerAddress||SyncListenPort>0), alarm drain gate).
- m5 (#618): Program.cs catch-all filters HostAbortedException (design-time tooling).
- m6 (Cluster-002): ClusterRoleInfo seeds only Up/WeaklyUp members.

Claude-Session: https://claude.ai/code/session_01Ef2ozxS7QPanGT8uoATbTe
2026-10-08 07:47:34 -04:00
Joseph Doherty 83e9ffd27a fix(alarms): SDK timed-unshelve timer is a no-op; commented re-ack comment is durable (#597)
Review R1 M2: since #597 the SDK timer fires at real expiry, and OnTimedUnshelve routed an operator-style
Unshelve by 'system' from both pair nodes, pre-empting the engine's AutoUnshelve. The engine owns expiry,
so the callback now routes nothing. Integration test: an engine stand-in that acts on a routed Unshelve and
expires late, plus a wrapped OnTimedUnshelve (positive control: the SDK timer fired).
Review R1 m1: Part9StateMachine.ApplyAcknowledge records a commented re-ack as AddComment (persisted and
replicated) instead of a NoOp; an uncommented re-ack is still a no-op.

Claude-Session: https://claude.ai/code/session_01Ef2ozxS7QPanGT8uoATbTe
2026-10-08 07:47:28 -04:00
Joseph Doherty 18f8eb5d2f feat(address-space): server-side array writes gated per driver capability (#610)
Array nodes were forced read-only for every driver. A new single authority,
Core.Abstractions DriverArrayWriteSupport, says which (driver, element type)
pairs encode a whole-array write: Modbus (Coils / numeric HoldingRegisters),
S7 (Byte/Int16/UInt16/Int32/UInt32/Float32), OpcUaClient (pass-through) and
Galaxy (Boolean/Int32/Int64/Float/Double/String/DateTime). Everything else
stays read-only, so the server never advertises a write the driver refuses.

- Composer derives RawTagPlan.ArrayWritable + UnsReferenceVariable
  .BackingArrayWritable at compose time (both seams, byte-parity); the
  applier's one writability rule is Writable && (!IsArray || ArrayWritable)
  for materialise and surgical paths, raw and UNS alike.
- ArrayWritable is on the classifier's surgical raw-tag whitelist and
  mirrored on the UNS record (UnsReferenceBackingParityTests updated).
- Node manager refuses a wrong-length / non-array value on an array node
  (BadTypeMismatch) and any IndexRange write (BadWriteNotSupported) inline,
  before dispatch; WriteOperate/ACL gates and the revert are unchanged.
- Per-driver parity tests pin the table against the real Modbus write path,
  S7 EncodeArrayBlock and Galaxy MxValueEncoder (UInt32/UInt64 excluded:
  array covariance reinterprets their bits).
- Over-the-wire proof in OpcUaServer.IntegrationTests (ArrayWriteWireTests).
- ValueRank > 1 documented as out of scope by decision (docs/Uns.md).

Claude-Session: https://claude.ai/code/session_01Ef2ozxS7QPanGT8uoATbTe
2026-10-08 07:39:05 -04:00
Joseph Doherty e9e35baa70 Merge #600.1/#603: provision changed-to-historized tags; retire dormant VirtualTagEngine; cache ScriptSandbox references; one LocalDb replication predicate for registration and drain gate
Claude-Session: https://claude.ai/code/session_01Ef2ozxS7QPanGT8uoATbTe
2026-10-08 07:01:41 -04:00
Joseph Doherty 3fb6549209 fix(review): resolve deferred code-review findings; S7.Cli verdict, browser CP release, MemberLeft/Exited/Downed, Sql Detach hoist (#625)
Claude-Session: https://claude.ai/code/session_01Ef2ozxS7QPanGT8uoATbTe
2026-10-08 06:59:53 -04:00
Joseph Doherty 1b37b2f2b0 perf(scripting): cache ScriptSandbox BCL + pinned metadata references (#603)
Claude-Session: https://claude.ai/code/session_01Ef2ozxS7QPanGT8uoATbTe
2026-10-08 06:52:35 -04:00
Joseph Doherty a8703c6afe refactor(virtualtags): retire dormant VirtualTagEngine/Source/Definition + TimerTriggerScheduler (#603)
TimerTriggerScheduler was welded to the engine and has no production consumer.

Claude-Session: https://claude.ai/code/session_01Ef2ozxS7QPanGT8uoATbTe
2026-10-08 06:51:22 -04:00
Joseph Doherty 129b84fd69 fix(cluster): record RoleLeaderChanged leader from the event, not the lagging state snapshot (#590)
ClusterRoleInfo resolved the leader address through Cluster.State.Members, which is
published asynchronously; a miss stored null permanently, so RoleLeader(role) stayed null
(flaky test timed out under load: 6/10 failing with CPU contention, 0/20 after). Only the
address is ever read, so store it directly. Adds a deterministic unit test (fails without
the fix) and InternalsVisibleTo for the Cluster.Tests project.

Claude-Session: https://claude.ai/code/session_01Ef2ozxS7QPanGT8uoATbTe
2026-10-08 00:25:57 -04:00
Joseph Doherty 157d01c2c2 Merge branch 'fix/r3-02-stale-good-family' — R3-02: one quality contract at four seams — comms loss marks raw+UNS value variables BadNoCommunication (WriteStatuses sink member, recovery audit), Calculation + VT hold on Bad / worst-of sub-code / (value,status) dedup, MQTT initial push + ConnectionLost stale, Sparkplug birth replay (archreview 03/S15, 05/STAB-19, 05/STAB-18, 02/S14) 2026-08-20 11:49:06 -04:00
Joseph Doherty af39e61638 fix(cluster): T7's post-overlay guard was structurally unreachable — compare live Cluster:Roles, not the PostConfigured options.Roles (review finding, R3-04)
options.Roles is unconditionally overwritten by AddOtOpcUaCluster's PostConfigure to equal
EffectiveRoles.Gossiped (T2's own structural-identity fix), so comparing options.Roles against
Gossiped could never observe a divergence — an overlay setting Cluster:Roles was silently
ignored, not fail-start. The validator now injects IConfiguration and re-reads the raw
Cluster:Roles section LIVE (reflecting any overlay applied after EffectiveRoles was first
resolved), normalized through EffectiveRoles.Resolve for reuse of its own validation.

AddOtOpcUaCluster now TryAddSingleton(configuration) so a bare ServiceCollection test harness
(not just WebApplicationBuilder) can supply IConfiguration to the validator.

Both guard tests rewritten to go through the real AddOtOpcUaCluster + PostConfigure pipeline
over a live ConfigurationManager, simulating an overlay as a second provider added AFTER
EffectiveRoles was resolved off the same object — exactly Program.cs's own sequencing. Confirmed
red beforehand: the "changes" test failed to throw against the pre-fix validator.
2026-08-20 11:01:33 -04:00
Joseph Doherty 3648d3c129 fix(cluster): ClusterRoleInfo._roleLeaders stays Ordinal — Akka tracks role-leadership per exact gossiped role string (review finding, R3-04) 2026-08-20 10:52:51 -04:00
Joseph Doherty e2497c908d test(cluster): guard that role reads live only in the authority (archreview theme 4) 2026-08-20 10:21:36 -04:00