Commit Graph

4197 Commits

Author SHA1 Message Date
Joseph Doherty 953bd27533 fix(runtime,adminui): per-tag write ordering, pessimistic init bound, unique generations, PostStop budget (fix round 1)
- Writes: a second write to a tag already inside the driver waits in the
  queue behind it (_tagsInFlight) while other tags still run under the
  cap of 4; queued writes are stamped and failed BadTimeout at dequeue
  once older than the 8 s host Ask budget (Props queuedWriteBudget)
  instead of being written late.
- Initialise attempts are bounded pessimistically (initializeTimeout,
  default 30 s = 2x Tier C) on top of the invoker's token-only timeout:
  an overrun is logged as abandoned, reported InitializeFailed so the
  retry loop continues, and its late result is dropped by generation.
  Generations come from a process-wide Interlocked counter so a prior
  incarnation's late result cannot collide (F6).
- PostStop shares one shutdownTimeout budget across shutdown + dispose.
- TelemetryDialSupervisor cancels the establishment deadline on
  StreamStopped; FleetDiagnosticsClient builds the ActorSelection inside
  the try (a malformed node degrades to "unavailable"); the diagnostics
  page labels the column "Last health publish".
- Tests: same-tag serialisation with a distinct-tag positive control;
  stale queued write BadTimeout; token-ignoring hung init -> later
  attempt connects; stopwatch assertion demoted to a ceiling.

Refs #731

Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 10:36:30 -04:00
Joseph Doherty 59b72fe385 fix(runtime): diagnostics Name stays the string instance id; order-tolerant write-cap test (A6 follow-through)
DriverInstanceDiagnostics.Name carries the string driver-instance id
(what /hosts, telemetry and the artifact key on) and DriverInstanceId
the DB row Guid; the wire record has no slot for the authored display
name. The write-cap test asserts entry order per slot pair instead of
completion order of two concurrent writes.

Refs #731

Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 10:15:01 -04:00
Joseph Doherty d1cc9e2cfc feat(runtime,adminui): real GetDiagnostics from child health + /fleet/diagnostics consumer (A6)
DriverHostActor wraps the health publisher it hands its children in a
ChildHealthLedger (last snapshot per driver id, forwarded to the real
publisher), so GetDiagnostics now answers with each child's row id,
authored name, last-published state, device counts (host statuses when
the driver probes hosts, else one device) and last-change time — instead
of Guid.Empty / "Spawned" / 0 / now. Entries are forgotten on StopChild
and operator Restart.

IFleetDiagnosticsClient gains its first consumer: the new
/fleet/diagnostics page (node selector + ?node= query, revision,
running-from-cache notice, per-driver table). FleetDiagnosticsClient
logs the degrade-to-empty path at Warning.

Review ref: 02/U1.

Refs #731

Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 06:44:38 -04:00
Joseph Doherty c6d4b14c93 feat(telemetry): wire TelemetryDial:CallTimeoutSeconds as the dial-establishment deadline (A5)
CallTimeoutSeconds was bound, defaulted and documented but consumed
nowhere. TelemetryDialSupervisor now arms a per-generation deadline on
every dial: a dial that has delivered no event by the deadline is
cancelled and treated as a transport failure (reconnect with a failure
streak). A live node replays its snapshots on subscribe, so this bites
only a dial that never connects (a black-holed host, where the socket
connect has no timeout). An established stream is deliberately not
bounded — the wire has no heartbeat, keepalive pings own its liveness.

Review ref: 02/U2.

Refs #731

Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 06:38:32 -04:00
Joseph Doherty 79d6b0a4b2 fix(runtime): lifecycle housekeeping — reprovision stamp, Stale drops, off-thread DB probe, tombstone sweep (A4)
- DriverHostActor.StopChild removes the driver's _lastReprovisionUtc stamp
  (and its deferred-reprovision timer): the map grew per removed id and a
  re-added driver inherited a backoff it never earned (02/L2).
- Stale now registers explicit drops for AttributeValueBatchPublished,
  AttributeAlarmPublished, ArtifactForApply and ApplyWedged instead of
  letting them hit Unhandled (02/H5).
- DbHealthProbeActor runs its SELECT on the thread pool and pipes the
  outcome back (single-flight), with a 2 s command timeout on relational
  providers, so a hung SQL Server no longer pins the actor thread and the
  cached GetStatus answer stays reachable (02/P7).
- TelemetryLocalHub eviction tombstones are timestamped and swept past a
  10 min TTL on the next Evict; a TimeProvider ctor is the test seam (02/L4).
- Tests: ValuePublishes.Single helper replaces the per-value
  AttributeValuePublished in CalculationDependencyFlowTests (02/U4, test
  half; the production receive is owned by Lane E).
- R05/8 audit: PollGroupEngine.Unsubscribe has no production caller; every
  driver already uses UnsubscribeAsync (no change).

Refs #731

Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 06:36:12 -04:00
Joseph Doherty 6e2febbf3d fix(runtime): run driver writes off the mailbox under an in-flight cap (A3)
WriteAttribute was a ReceiveAsync, suspending the child's mailbox for the
whole device round-trip (up to 5 s): writes serialised, the second of two
operator writes to a slow device missed the host's 8 s Ask and was
reverted although the device took it, and every data change queued behind
the write. Writes are now a synchronous Receive that launches the device
call on the thread pool (sender captured, reply from the task, slot freed
by a self-sent WriteCompleted) under a per-driver cap (default 4; excess
writes queue in order). Queued writes fail fast on every exit from
Connected. Subscribe/unsubscribe stay ReceiveAsync.

Review ref: 02/P2.

Refs #731

Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 06:30:57 -04:00
Joseph Doherty bfd5b7b015 fix(runtime): single-flight reconnect attempts, bounded pre-reconnect teardown (A2)
The Reconnecting retry timer started a new ShutdownAsync + InitializeAsync
pair on every tick whether or not the previous attempt had finished; the
generation token only discarded the older RESULT, so with an Initialize
bound longer than the cadence the teardown ran underneath an attempt still
inside the driver (the dispose-before-overwrite race). An _initInFlight
gate, cleared by the piped InitializeSucceeded/Failed, skips a tick while
an attempt is running; the pre-reconnect teardown is bounded by the
shutdown timeout; PostStop cancels an in-flight attempt before shutting
the driver down.

Review ref: 02/H2.

Refs #731

Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 06:26:45 -04:00
Joseph Doherty eb23053700 fix(runtime): dispose drivers on final stop, bound PostStop shutdown (A1)
DriverInstanceActor.PostStop now bounds IDriver.ShutdownAsync by a
shutdownTimeout (default 5 s, logged on overrun) and then disposes the
driver (IAsyncDisposable preferred, else IDisposable) — but only on a
real stop. PreRestart flags the restart so the PostStop it triggers
skips the dispose, because a supervision restart re-uses the same
driver object (Props closure) and the host re-initialises it.

DiscoveryDriverBrowser disposes its throwaway capture/probe driver
after the bounded shutdown, including after a failed or timed-out one.

Review refs: 02/L1, 02/P4, 06/G-1 (host context), 07/L2.

Refs #731

Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 06:23:39 -04:00
Joseph Doherty ad2b5c4044 docs: add 2026-10-21 system review + remediation plan
Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 06:04:58 -04:00
Joseph Doherty d51d35bd64 Merge branch 'docs/xml-doc-cleanup'
v2-ci / unit-tests (push) Failing after 3m14s
2026-10-15 06:25:09 -04:00
Joseph Doherty 8c6067b62e docs: move subsystem gotchas out of CLAUDE.md into path-scoped .claude/rules
Keeps the root CLAUDE.md lean by lazy-loading per-area detail (drivers, topology,
data-plane auth, alarms, historian) only when matching files are touched.
2026-10-15 06:14:58 -04:00
Joseph Doherty 826312befe docs: fill missing XML docs and strip tracking IDs from src comments
Clears CommentChecker findings in src/ (3,280 -> 209, the remainder being false
positives like PLC-5/IEEE-754/M0.0) by adding missing summaries/params/returns,
normalising inheritdoc usage, and removing Gitea/review tracking IDs while keeping
the rationale they annotated. Comment-only: comment-stripped sources are identical
to the parent commit.
2026-10-15 06:14:58 -04:00
dohertj2 ebcbd81fca Merge pull request 'Ensure driver test cleanup preserves primary failures' (#730) from test/task-assertion-return-discard into master
v2-ci / unit-tests (push) Successful in 1h1m21s
2026-10-13 21:06:31 -04:00
Joseph Doherty 4f537a5652 Merge remote-tracking branch 'origin/master' into test/task-assertion-return-discard
v2-ci / unit-tests (pull_request) Successful in 1h1m9s
2026-10-13 19:03:52 -04:00
dohertj2 c89912a93b Merge pull request 'Fix DB-less ServiceLevel freshness and authorization remarks' (#724) from fix/dbless-servicelevel-freshness into master
v2-ci / unit-tests (push) Waiting to run
2026-10-13 19:03:42 -04:00
Joseph Doherty c4b4046832 test: verify missing-snapshot ServiceLevel timer behavior
v2-ci / unit-tests (pull_request) Successful in 1h0m42s
2026-10-13 17:19:50 -04:00
Joseph Doherty eed7cc7d75 test: guarantee driver fixture cleanup without masking failures 2026-10-13 16:36:38 -04:00
Joseph Doherty 5310a7e07a Discard asserted probe-task return values without joining loops 2026-10-13 16:12:31 -04:00
Joseph Doherty e1feb9fbb6 Merge remote-tracking branch 'origin/master' into fix/dbless-servicelevel-freshness
v2-ci / unit-tests (pull_request) Waiting to run
2026-10-13 15:58:35 -04:00
dohertj2 1bf8c614a9 Merge pull request 'Retain ServiceLevel requests across binding and publisher failures' (#720) from fix/servicelevel-binding-replay into master
v2-ci / unit-tests (push) Waiting to run
2026-10-13 15:58:19 -04:00
Joseph Doherty bc80b53e0e Merge ServiceLevel replay fixes for freshness integration 2026-10-13 15:11:20 -04:00
Joseph Doherty f478b844a1 docs: clarify anonymous session ACL enforcement 2026-10-13 14:29:00 -04:00
Joseph Doherty 1e71a25e3d fix: age DB-less ServiceLevel snapshots periodically 2026-10-13 14:29:00 -04:00
Joseph Doherty 8828e999c4 Merge remote-tracking branch 'origin/master' into fix/servicelevel-binding-replay
v2-ci / unit-tests (pull_request) Successful in 1h1m55s
2026-10-13 13:56:26 -04:00
dohertj2 4f7a07fbd3 Merge pull request 'Count individual skipped TRX results' (#714) from fix/trx-skip-reporting into master
v2-ci / unit-tests (push) Has been cancelled
2026-10-13 13:56:04 -04:00
Joseph Doherty da2e441bb9 Merge branch 'fix/trx-skip-reporting' of https://gitea.dohertylan.com/dohertj2/lmxopcua into fix/servicelevel-binding-replay
v2-ci / unit-tests (pull_request) Has been cancelled
2026-10-13 13:54:54 -04:00
Joseph Doherty c8e3e279b4 Retain and replay ServiceLevel across publisher binding and failures
v2-ci / unit-tests (pull_request) Waiting to run
2026-10-13 13:26:27 -04:00
Joseph Doherty 468f13cb56 Merge remote-tracking branch 'origin/master' into fix/trx-skip-reporting
v2-ci / unit-tests (pull_request) Waiting to run
2026-10-13 11:55:13 -04:00
Joseph Doherty f71fe13826 docs: record successful bounded CI baseline 2026-10-13 11:55:13 -04:00
dohertj2 6f0674296b Merge pull request 'Clarify S7 counter evidence and fixture limits' (#715) from docs/s7-counter-evidence into master
v2-ci / unit-tests (push) Has been cancelled
2026-10-13 11:54:57 -04:00
dohertj2 19bc13bf6a Merge pull request 'Bound CI concurrency and stabilize test fixtures' (#711) from fix/bootstrap-and-boundary-test-fixtures into master
v2-ci / unit-tests (push) Waiting to run
2026-10-13 11:54:15 -04:00
Joseph Doherty cd50db136e docs: clarify S7 counter evidence and fixture limits 2026-10-13 11:34:59 -04:00
Joseph Doherty eecf301200 ci: count individual skipped TRX results 2026-10-13 11:05:49 -04:00
Joseph Doherty 76bdca90e9 ci: bound build and test concurrency
v2-ci / unit-tests (pull_request) Successful in 1h1m1s
2026-10-13 10:52:12 -04:00
Joseph Doherty 8ff9f38bbd Make bootstrap cancellation test deterministic and deduplicate boundary cases 2026-10-13 09:56:37 -04:00
dohertj2 495606b9fb Merge pull request 'Clarify Galaxy native acknowledgement outcome documentation' (#708) from docs/galaxy-ack-rationale into master
v2-ci / build (push) Successful in 7m24s
v2-ci / unit-tests (push) Failing after 31m46s
2026-10-13 09:10:19 -04:00
Joseph Doherty f794dc50b6 docs(galaxy): clarify native alarm acknowledgement outcomes
v2-ci / build (pull_request) Successful in 7m16s
v2-ci / unit-tests (pull_request) Failing after 3h13m50s
2026-10-13 09:07:13 -04:00
dohertj2 7b23875070 Merge pull request 'fix(historian): preserve alarm delivery identity across retries' (#706) from fix/alarm-delivery-identity into master
v2-ci / build (push) Successful in 11m27s
v2-ci / unit-tests (push) Failing after 1h18m15s
2026-10-13 07:11:09 -04:00
Joseph Doherty 109163621c fix(historian): preserve stable alarm delivery identities
v2-ci / build (pull_request) Successful in 5m52s
v2-ci / unit-tests (pull_request) Failing after 1h23m55s
2026-10-13 07:10:58 -04:00
dohertj2 0ccfbbee5c Merge pull request 'Accept supported definitive loss reasons in restart wire test' (#704) from fix/restart-loss-reason into master
v2-ci / build (push) Successful in 12m15s
v2-ci / unit-tests (push) Failing after 35m1s
2026-10-13 06:30:22 -04:00
Joseph Doherty dccab57d39 test: accept definitive session loss reasons during restart
v2-ci / build (pull_request) Successful in 12m16s
v2-ci / unit-tests (pull_request) Failing after 3h13m34s
2026-10-13 06:30:08 -04:00
dohertj2 a935e21bf8 Merge pull request 'Cover concurrent gateway tag cache lookups' (#703) from test/historian-cache-concurrency into master
v2-ci / build (push) Successful in 6m57s
v2-ci / unit-tests (push) Failing after 3h11m16s
2026-10-13 05:22:42 -04:00
Joseph Doherty 3ec6006f57 test(historian): cover concurrent gateway tag cache lookups
v2-ci / build (pull_request) Successful in 6m52s
v2-ci / unit-tests (pull_request) Failing after 3h12m20s
2026-10-13 05:21:46 -04:00
dohertj2 618d98e822 Merge pull request 'Fix historian missing-tag latency, paging limits and response contracts' (#699) from fix/historian-653-followups into master
v2-ci / build (push) Successful in 6m15s
v2-ci / unit-tests (push) Failing after 3h11m48s
2026-10-13 04:45:00 -04:00
Joseph Doherty 5897764024 fix(historian): fail fast for missing tags and reduce gateway read cost
v2-ci / build (pull_request) Successful in 5m6s
v2-ci / unit-tests (pull_request) Failing after 29m54s
2026-10-13 04:31:52 -04:00
Joseph Doherty 97668d26d3 fix(historian): respect gateway ceilings and history value contracts
v2-ci / build (pull_request) Successful in 5m10s
v2-ci / unit-tests (pull_request) Failing after 28m34s
2026-10-13 03:51:04 -04:00
Joseph Doherty 66a2b0c3e9 Merge fix/694-gateway-ceiling-events: strict-ceiling raw paging, epoch boundary-row truncation, crowding re-read, EventReadHorizon (#694.1/.2)
v2-ci / build (push) Successful in 5m30s
v2-ci / unit-tests (push) Failing after 3h14m14s
2026-10-13 01:53:30 -04:00
Joseph Doherty de8bc7a12c fix(historian): #694 R2 cap the crowding re-read room at MaxTieClusterOverfetch
The #687/F4 crowding re-read doubled its room up to 8 re-reads with no cap, so a pathological single-ms
cluster on a streaming gateway was streamed several times over for one page. The room is now capped at
ServerHistorian:MaxTieClusterOverfetch (plumbed into GatewayHistorianDataSource as maxPreWindowRows,
default 65536); past it, or once a read has already seen more, the read is BadTooManyOperations with
PreWindowRows set.
2026-10-13 01:53:14 -04:00
Joseph Doherty 3c3513ea3e fix(historian): #694 page raw reads at a strict gateway's learned ceiling; opt-in EventReadHorizon
Item 1: IHistorianDataSource.MaxRawValuesPerRead (Gateway: learned ceiling - 16 = L; 0 elsewhere, guard-tested).
RawPageLimit plans every raw page on top of #692's lookahead: with L > 0 the read (page P + boundary ties k
+ lookahead) is <= L, P = min(N, L - k - 1) (uncapped: L - k - 1); no room => straight to the tie-cluster
over-fetch (never an uncapped or underflowing request). One re-plan + re-read when the read taught a new
limit or was refused for crowding pre-window rows (HistoryReadLimitExceededException.PreWindowRows): the
page shrinks by them. Over-fetch bound min(MaxTieClusterOverfetch, L - 1); slice size never 0.
Review fixes: F1 a window starting at/before the epoch counts AVEVA's synthetic row as pre-window (a
ceiling-truncated read was reported complete); F2 the page path disables ReverseRawReader's own re-run
(cold end-only ceiling 20/N=3000 cost 3,023 reads, now <= 40); F3 crowded look-back re-read around, not
refused; F4 the #687 crowding re-read room at least doubles (up to 8 re-reads), so a cluster of hundreds
just before a resume point no longer fails on a default gateway; a crowding refusal that leaves no room
is BadTooManyOperations, never a tick-by-tick cursor.

Item 2: ServerHistorian:EventReadHorizon (default 00:00:00 = exact, unchanged). When set, every fresh
HistoryReadEvents window's earlier bound is raised to now - horizon (both directions, all backends).
Lossy by design; documented.

Item 3 (native TimeAverage/Total re-check) stays blocked on historiangw#27.
2026-10-13 01:39:32 -04:00
Joseph Doherty a5a0496176 Merge fix/693-gateway-attime: Gateway AtTime from raw reads + shared evaluator, node type hint (#693) 2026-10-13 01:18:09 -04:00