Commit Graph

4191 Commits

Author SHA1 Message Date
Joseph Doherty bfd5b7b015 fix(runtime): single-flight reconnect attempts, bounded pre-reconnect teardown (A2)
The Reconnecting retry timer started a new ShutdownAsync + InitializeAsync
pair on every tick whether or not the previous attempt had finished; the
generation token only discarded the older RESULT, so with an Initialize
bound longer than the cadence the teardown ran underneath an attempt still
inside the driver (the dispose-before-overwrite race). An _initInFlight
gate, cleared by the piped InitializeSucceeded/Failed, skips a tick while
an attempt is running; the pre-reconnect teardown is bounded by the
shutdown timeout; PostStop cancels an in-flight attempt before shutting
the driver down.

Review ref: 02/H2.

Refs #731

Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 06:26:45 -04:00
Joseph Doherty eb23053700 fix(runtime): dispose drivers on final stop, bound PostStop shutdown (A1)
DriverInstanceActor.PostStop now bounds IDriver.ShutdownAsync by a
shutdownTimeout (default 5 s, logged on overrun) and then disposes the
driver (IAsyncDisposable preferred, else IDisposable) — but only on a
real stop. PreRestart flags the restart so the PostStop it triggers
skips the dispose, because a supervision restart re-uses the same
driver object (Props closure) and the host re-initialises it.

DiscoveryDriverBrowser disposes its throwaway capture/probe driver
after the bounded shutdown, including after a failed or timed-out one.

Review refs: 02/L1, 02/P4, 06/G-1 (host context), 07/L2.

Refs #731

Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 06:23:39 -04:00
Joseph Doherty ad2b5c4044 docs: add 2026-10-21 system review + remediation plan
Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 06:04:58 -04:00
Joseph Doherty d51d35bd64 Merge branch 'docs/xml-doc-cleanup'
v2-ci / unit-tests (push) Failing after 3m14s
2026-10-15 06:25:09 -04:00
Joseph Doherty 8c6067b62e docs: move subsystem gotchas out of CLAUDE.md into path-scoped .claude/rules
Keeps the root CLAUDE.md lean by lazy-loading per-area detail (drivers, topology,
data-plane auth, alarms, historian) only when matching files are touched.
2026-10-15 06:14:58 -04:00
Joseph Doherty 826312befe docs: fill missing XML docs and strip tracking IDs from src comments
Clears CommentChecker findings in src/ (3,280 -> 209, the remainder being false
positives like PLC-5/IEEE-754/M0.0) by adding missing summaries/params/returns,
normalising inheritdoc usage, and removing Gitea/review tracking IDs while keeping
the rationale they annotated. Comment-only: comment-stripped sources are identical
to the parent commit.
2026-10-15 06:14:58 -04:00
dohertj2 ebcbd81fca Merge pull request 'Ensure driver test cleanup preserves primary failures' (#730) from test/task-assertion-return-discard into master
v2-ci / unit-tests (push) Successful in 1h1m21s
2026-10-13 21:06:31 -04:00
Joseph Doherty 4f537a5652 Merge remote-tracking branch 'origin/master' into test/task-assertion-return-discard
v2-ci / unit-tests (pull_request) Successful in 1h1m9s
2026-10-13 19:03:52 -04:00
dohertj2 c89912a93b Merge pull request 'Fix DB-less ServiceLevel freshness and authorization remarks' (#724) from fix/dbless-servicelevel-freshness into master
v2-ci / unit-tests (push) Waiting to run
2026-10-13 19:03:42 -04:00
Joseph Doherty c4b4046832 test: verify missing-snapshot ServiceLevel timer behavior
v2-ci / unit-tests (pull_request) Successful in 1h0m42s
2026-10-13 17:19:50 -04:00
Joseph Doherty eed7cc7d75 test: guarantee driver fixture cleanup without masking failures 2026-10-13 16:36:38 -04:00
Joseph Doherty 5310a7e07a Discard asserted probe-task return values without joining loops 2026-10-13 16:12:31 -04:00
Joseph Doherty e1feb9fbb6 Merge remote-tracking branch 'origin/master' into fix/dbless-servicelevel-freshness
v2-ci / unit-tests (pull_request) Waiting to run
2026-10-13 15:58:35 -04:00
dohertj2 1bf8c614a9 Merge pull request 'Retain ServiceLevel requests across binding and publisher failures' (#720) from fix/servicelevel-binding-replay into master
v2-ci / unit-tests (push) Waiting to run
2026-10-13 15:58:19 -04:00
Joseph Doherty bc80b53e0e Merge ServiceLevel replay fixes for freshness integration 2026-10-13 15:11:20 -04:00
Joseph Doherty f478b844a1 docs: clarify anonymous session ACL enforcement 2026-10-13 14:29:00 -04:00
Joseph Doherty 1e71a25e3d fix: age DB-less ServiceLevel snapshots periodically 2026-10-13 14:29:00 -04:00
Joseph Doherty 8828e999c4 Merge remote-tracking branch 'origin/master' into fix/servicelevel-binding-replay
v2-ci / unit-tests (pull_request) Successful in 1h1m55s
2026-10-13 13:56:26 -04:00
dohertj2 4f7a07fbd3 Merge pull request 'Count individual skipped TRX results' (#714) from fix/trx-skip-reporting into master
v2-ci / unit-tests (push) Has been cancelled
2026-10-13 13:56:04 -04:00
Joseph Doherty da2e441bb9 Merge branch 'fix/trx-skip-reporting' of https://gitea.dohertylan.com/dohertj2/lmxopcua into fix/servicelevel-binding-replay
v2-ci / unit-tests (pull_request) Has been cancelled
2026-10-13 13:54:54 -04:00
Joseph Doherty c8e3e279b4 Retain and replay ServiceLevel across publisher binding and failures
v2-ci / unit-tests (pull_request) Waiting to run
2026-10-13 13:26:27 -04:00
Joseph Doherty 468f13cb56 Merge remote-tracking branch 'origin/master' into fix/trx-skip-reporting
v2-ci / unit-tests (pull_request) Waiting to run
2026-10-13 11:55:13 -04:00
Joseph Doherty f71fe13826 docs: record successful bounded CI baseline 2026-10-13 11:55:13 -04:00
dohertj2 6f0674296b Merge pull request 'Clarify S7 counter evidence and fixture limits' (#715) from docs/s7-counter-evidence into master
v2-ci / unit-tests (push) Has been cancelled
2026-10-13 11:54:57 -04:00
dohertj2 19bc13bf6a Merge pull request 'Bound CI concurrency and stabilize test fixtures' (#711) from fix/bootstrap-and-boundary-test-fixtures into master
v2-ci / unit-tests (push) Waiting to run
2026-10-13 11:54:15 -04:00
Joseph Doherty cd50db136e docs: clarify S7 counter evidence and fixture limits 2026-10-13 11:34:59 -04:00
Joseph Doherty eecf301200 ci: count individual skipped TRX results 2026-10-13 11:05:49 -04:00
Joseph Doherty 76bdca90e9 ci: bound build and test concurrency
v2-ci / unit-tests (pull_request) Successful in 1h1m1s
2026-10-13 10:52:12 -04:00
Joseph Doherty 8ff9f38bbd Make bootstrap cancellation test deterministic and deduplicate boundary cases 2026-10-13 09:56:37 -04:00
dohertj2 495606b9fb Merge pull request 'Clarify Galaxy native acknowledgement outcome documentation' (#708) from docs/galaxy-ack-rationale into master
v2-ci / build (push) Successful in 7m24s
v2-ci / unit-tests (push) Failing after 31m46s
2026-10-13 09:10:19 -04:00
Joseph Doherty f794dc50b6 docs(galaxy): clarify native alarm acknowledgement outcomes
v2-ci / build (pull_request) Successful in 7m16s
v2-ci / unit-tests (pull_request) Failing after 3h13m50s
2026-10-13 09:07:13 -04:00
dohertj2 7b23875070 Merge pull request 'fix(historian): preserve alarm delivery identity across retries' (#706) from fix/alarm-delivery-identity into master
v2-ci / build (push) Successful in 11m27s
v2-ci / unit-tests (push) Failing after 1h18m15s
2026-10-13 07:11:09 -04:00
Joseph Doherty 109163621c fix(historian): preserve stable alarm delivery identities
v2-ci / build (pull_request) Successful in 5m52s
v2-ci / unit-tests (pull_request) Failing after 1h23m55s
2026-10-13 07:10:58 -04:00
dohertj2 0ccfbbee5c Merge pull request 'Accept supported definitive loss reasons in restart wire test' (#704) from fix/restart-loss-reason into master
v2-ci / build (push) Successful in 12m15s
v2-ci / unit-tests (push) Failing after 35m1s
2026-10-13 06:30:22 -04:00
Joseph Doherty dccab57d39 test: accept definitive session loss reasons during restart
v2-ci / build (pull_request) Successful in 12m16s
v2-ci / unit-tests (pull_request) Failing after 3h13m34s
2026-10-13 06:30:08 -04:00
dohertj2 a935e21bf8 Merge pull request 'Cover concurrent gateway tag cache lookups' (#703) from test/historian-cache-concurrency into master
v2-ci / build (push) Successful in 6m57s
v2-ci / unit-tests (push) Failing after 3h11m16s
2026-10-13 05:22:42 -04:00
Joseph Doherty 3ec6006f57 test(historian): cover concurrent gateway tag cache lookups
v2-ci / build (pull_request) Successful in 6m52s
v2-ci / unit-tests (pull_request) Failing after 3h12m20s
2026-10-13 05:21:46 -04:00
dohertj2 618d98e822 Merge pull request 'Fix historian missing-tag latency, paging limits and response contracts' (#699) from fix/historian-653-followups into master
v2-ci / build (push) Successful in 6m15s
v2-ci / unit-tests (push) Failing after 3h11m48s
2026-10-13 04:45:00 -04:00
Joseph Doherty 5897764024 fix(historian): fail fast for missing tags and reduce gateway read cost
v2-ci / build (pull_request) Successful in 5m6s
v2-ci / unit-tests (pull_request) Failing after 29m54s
2026-10-13 04:31:52 -04:00
Joseph Doherty 97668d26d3 fix(historian): respect gateway ceilings and history value contracts
v2-ci / build (pull_request) Successful in 5m10s
v2-ci / unit-tests (pull_request) Failing after 28m34s
2026-10-13 03:51:04 -04:00
Joseph Doherty 66a2b0c3e9 Merge fix/694-gateway-ceiling-events: strict-ceiling raw paging, epoch boundary-row truncation, crowding re-read, EventReadHorizon (#694.1/.2)
v2-ci / build (push) Successful in 5m30s
v2-ci / unit-tests (push) Failing after 3h14m14s
2026-10-13 01:53:30 -04:00
Joseph Doherty de8bc7a12c fix(historian): #694 R2 cap the crowding re-read room at MaxTieClusterOverfetch
The #687/F4 crowding re-read doubled its room up to 8 re-reads with no cap, so a pathological single-ms
cluster on a streaming gateway was streamed several times over for one page. The room is now capped at
ServerHistorian:MaxTieClusterOverfetch (plumbed into GatewayHistorianDataSource as maxPreWindowRows,
default 65536); past it, or once a read has already seen more, the read is BadTooManyOperations with
PreWindowRows set.
2026-10-13 01:53:14 -04:00
Joseph Doherty 3c3513ea3e fix(historian): #694 page raw reads at a strict gateway's learned ceiling; opt-in EventReadHorizon
Item 1: IHistorianDataSource.MaxRawValuesPerRead (Gateway: learned ceiling - 16 = L; 0 elsewhere, guard-tested).
RawPageLimit plans every raw page on top of #692's lookahead: with L > 0 the read (page P + boundary ties k
+ lookahead) is <= L, P = min(N, L - k - 1) (uncapped: L - k - 1); no room => straight to the tie-cluster
over-fetch (never an uncapped or underflowing request). One re-plan + re-read when the read taught a new
limit or was refused for crowding pre-window rows (HistoryReadLimitExceededException.PreWindowRows): the
page shrinks by them. Over-fetch bound min(MaxTieClusterOverfetch, L - 1); slice size never 0.
Review fixes: F1 a window starting at/before the epoch counts AVEVA's synthetic row as pre-window (a
ceiling-truncated read was reported complete); F2 the page path disables ReverseRawReader's own re-run
(cold end-only ceiling 20/N=3000 cost 3,023 reads, now <= 40); F3 crowded look-back re-read around, not
refused; F4 the #687 crowding re-read room at least doubles (up to 8 re-reads), so a cluster of hundreds
just before a resume point no longer fails on a default gateway; a crowding refusal that leaves no room
is BadTooManyOperations, never a tick-by-tick cursor.

Item 2: ServerHistorian:EventReadHorizon (default 00:00:00 = exact, unchanged). When set, every fresh
HistoryReadEvents window's earlier bound is raised to now - horizon (both directions, all backends).
Lossy by design; documented.

Item 3 (native TimeAverage/Total re-check) stays blocked on historiangw#27.
2026-10-13 01:39:32 -04:00
Joseph Doherty a5a0496176 Merge fix/693-gateway-attime: Gateway AtTime from raw reads + shared evaluator, node type hint (#693) 2026-10-13 01:18:09 -04:00
Joseph Doherty 2609c5462e fix(historian): #693 re-review — AtTime read cap 64 + 16 per time (bursty tags need ~6)
A time inside a gap of a bursty tag costs a 64-row jump read landing in the next
burst plus a bisection back across the gap (~6 reads), so the 64 + 4/time cap
refused a 1 h/day 1 Hz tag asked at noon on 30/100/300 consecutive days
(186/606/1,806 reads) that AVEVA's own AtTime served. The cap is a runaway guard
(HistoryReadDeadline bounds wall clock): now 64 + 16 per distinct time.
Regression tests for x30/x100/x300 (each <= 7 reads a time, values checked).
Docs: DateTime-hinted nodes are stepped but keep the gateway's Double (the
Gateway never historizes DateTime nodes; the SQL backends return DateTime).
2026-10-13 01:17:51 -04:00
Joseph Doherty 68a1f31397 fix(historian): #693 review — bisecting walk-back, small jump reads, read cap, DateTime/NaN hints
H1: the walk-back reset its width after a full segment and discarded a full
whole-remainder read, so a dense tag that stopped reporting weeks ago gave up
after 65 reads (BadTooManyOperations). It now gallops (x4, whole-remainder read
after two empties) and then BISECTS once any stored row is known below — a full
segment's (incl. a full whole-remainder read's) last row, or an earlier window
read's last row: (mid, hi] empty => hi = mid; full => known = its last row;
complete with rows => done; one density-aimed segment after a full read until an
aimed segment comes back empty; below 2 ms the whole stretch from the known row
(never a 100 ns step). Stopped 1 h..365 d with 5 y history: 6-18 reads at
1/min, 21-25 at 1 Hz, 25-31 at 10 Hz.
M1: a jump read asks for 64 rows (1,000 times over 10 y: ~66k rows, was ~1M).
L1: per-request read cap 64 + 4 x distinct times with a clear message; a strict
ceiling below 4 rows is refused naming the ceiling.
L2: DateTime hints are stepped. L3: under an integer hint a NaN/inf or
out-of-Int64-range value is unusable, never saturated Good.
L5: conformance Int64 AtTime asserts the type. L4: docs name the backend
differences. Nit: the window read end is clamped to now + 1 day.

Reviewer harnesses (Scratch693Fuzz/Cost, not committed): 0 differential diffs
over 2,400 cases.
2026-10-13 01:09:54 -04:00
Joseph Doherty aa923cea83 Merge fix/692-n1-paging: every HistoryRead-Raw page is full; N=1 no longer alternates empty pages (#692) 2026-10-13 01:02:31 -04:00
Joseph Doherty e04e1bf127 fix(historian): #692 review — a drained tie cluster never resumes onto the exclusive EndTime
F1 (pre-existing): SliceTieCluster continued a forward read while next <= endUtc, but the
forward window is end-exclusive; an oversized cluster one step below EndTime resumed AT
EndTime, the resume became a start == end point read and returned the EndTime samples as
data (microsecond: EndTime 8.1 us truncated to 8.0). Continue only while next is strictly
before EndTime truncated to the backend resolution.

Tests: A_tie_cluster_just_below_the_end_time_never_resumes_onto_the_end_time (native +
microsecond; both fail on the old rule) and
A_cursor_inside_a_tie_cluster_over_fetches_without_a_normal_read_first (pins the
WithinTieCluster shortcut by per-page read count; fails with the shortcut disabled).
Docs: the oversized-cluster exception to 'every page full / no lookahead => last' (F2),
and the N + skip + 1 strict-ceiling edge note referencing #694 (F4).
2026-10-13 01:02:12 -04:00
Joseph Doherty 0d249f3ee9 fix(historian): #693 AtTime type hint — Boolean nodes stepped, integer nodes rounded to their type
Review follow-up: on the Gateway every value is a Double (AVEVA stores a Boolean
as a 0/1 Int2), so the raw-evaluated AtTime interpolated a Boolean node to 0.5.

- IHistorianDataSource gains a default-interface overload
  ReadAtTimeAsync(fullReference, timestamps, TypeCode? valueType, ct) that
  forwards to the existing one (SqlServer/QuestDB/Null/fakes unchanged).
- OtOpcUaNodeManager.HistoryReadAtTime passes the node's scalar DataType
  (AtTimeValueType: built-in type -> TypeCode; arrays/other types => null);
  Raw and UNS NodeIds alike.
- GatewayAtTimeReader converts each raw row to the hinted kind BEFORE the shared
  AtTimeEvaluator runs (Boolean = value != 0 => stepped; integers => Int64 =>
  interpolated, rounded half away from zero, returned as the node's integer
  type; floats Double), so the evaluator's own type branch decides.
- AtTimeEvaluator boxed every interpolated Int64 as a Double (`cond ? (long)x :
  v` widens to double); its "keeping its type" test passed because Shouldly
  ShouldBe compares numbers across types. Fixed (affects SqlServer/QuestDB too)
  and pinned with ShouldBeOfType.
2026-10-13 00:52:03 -04:00
Joseph Doherty 3d7eea6280 docs(historian): #692 raw paging fills every page; lookahead row; marked placeholders 2026-10-13 00:42:13 -04:00