Commit Graph

4228 Commits

Author SHA1 Message Date
Joseph Doherty 34e455f412 test(ablegacy): dead-device gate tolerates a coalesced single-unit pass
Live run against ab_server (slc500/micrologix/plc5 x libplctag/native): with
coalescing on, the blackholed device's 20 tags are one block, i.e. one comms
fault per poll, so its backoff window opens on the second poll rather than the
first. The gate now asserts the actual contract: at most two timeouts across
three polls, the live device Good on every poll, and the third poll fast.

Refs #738
Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 14:48:35 -04:00
Joseph Doherty 61dd498eab test(ablegacy): pin the array-write capability row against the encoder
DriverArrayWriteSupport (Core.Abstractions, not this lane's) still lists
AbLegacy as scalar-only, so every AbLegacy array node is materialised
read-only and the new array encoder is unreachable from OPC UA. Adds the
Modbus/S7-style parity test, skipped until follow-up F1 adds the AbLegacy row
(Int/AnalogInt/Long/Float with a declared length), plus an active test pinning
exactly which element types the encoder accepts.

Refs #738
Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 14:46:04 -04:00
Joseph Doherty ffa5b2a4fa fix(ablegacy-cli): Ctrl-C prints Cancelled instead of a stack trace
Review 07 U7 (AbLegacy part): probe/read/write had no OperationCanceledException
arm, so cancelling a slow read printed an unhandled-exception stack trace. The
driver work now runs through AbLegacyCommandBase.RunCancellableAsync, which
reports an operator cancellation as "Cancelled." and still propagates anything
else. (The AbCip subscribe lock and SnapshotFormatter array rendering are
other lanes' files.)

Refs #738
Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 14:44:51 -04:00
Joseph Doherty 66927a3faa fix(ablegacy): wire the authored probeTimeoutSeconds into Test Connect
Dead config field (user decision: wire them all): the AdminUI form authors
probeTimeoutSeconds ("Admin UI probe timeout", default 5) but AbLegacyDriverProbe
never read it, so Test Connect always ran on the caller's budget. It now
tightens that budget (never raises it), the same rule as the Galaxy probe; a
non-positive value keeps the caller's budget.

Refs #738
Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 14:43:23 -04:00
Joseph Doherty 582e82899b feat(ablegacy): implement IBatchSubscribable
Review 02 P1 / Wave 2 rule (d): AbLegacy runs PollGroupEngine but raised only
the per-tag OnDataChange, so every changed tag paid its own trip through the
actor chain. The engine's per-cycle batch is now raised as OnDataChangeBatch
(one per poll cycle), alongside the per-tag event for legacy consumers.

Refs #738
Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 14:42:23 -04:00
Joseph Doherty 530f4bdcd4 perf(ablegacy): coalesce adjacent N/B/F/L scalar reads into one PCCC request
Review 07 P7 (rest): adjacent words on one data file were never coalesced
(one round trip per tag) although the array path proves the transport can read
a span. Within a device pass, scalar tags on one N/B/F/L file within 8
elements of each other (block <= 200 data bytes, under the 236-byte typed-read
limit) now ride one multi-element read and are decoded from their offsets with
exactly the CLR types the single-tag path returns (int / float / bool).

Only shapes the single path decodes identically are coalesced: indirect,
sub-element, array, file-number-less (S:/I:/O:) and width-mismatched tags keep
the single path. A block the device refuses (an authored element past the end
of the file, a native PLC-5 non-N file) falls back to per-tag reads for its
members and its shape is remembered; a comms fault fails the members together.

New driver-config key coalesceReads (default true) with an AdminUI switch
(Razor change: AbLegacyDriverForm, for the Wave 4 live check). Adds an
env-gated ab_server parity case (coalesced == per-tag values).

Refs #738
Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 14:41:20 -04:00
Joseph Doherty e5527d0201 fix(ablegacy): isolate a dead device — parallel devices, per-device backoff, single-flight connect
Review 07 P7 (High): reads walked every reference of every device serially
with no backoff, so a dead device cost Timeout per tag per poll (50 tags x 2 s
= 100 s) and starved every healthy device long enough to trip the comms-loss
watchdog against a reachable one.

- ReadAsync buckets references by device and reads up to 8 devices
  concurrently; each device's tags are still read in order.
- Each device has a ConnectionBackoff (1 s doubling to 30 s). Two consecutive
  comms faults mark it down: the rest of that pass and every later pass fail
  fast with BadCommunicationError and touch no wire until the window expires or
  any answer (data path or probe) resets it. One timing-out tag does not
  blacken its answering siblings.
- Cold connects are single-flight per device (ConnectGate), replacing the
  per-key creation locks; a cached handle never touches the gate.
- A status/exception proving the device ANSWERED with a deterministic refusal
  (not found, not allowed, out of bounds, type/size mismatch) is per-tag Bad,
  keeps its handle, warns once per tag and advances LastSuccessfulRead instead
  of evicting and degrading. Writes classify the same way.
- Health is written once per read: Degraded if any device faulted, Healthy if
  anything read Good, LastSuccessfulRead advanced when any device answered.

Adds an env-gated ab_server case with a blackholed sibling device.

Refs #738
Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 14:33:54 -04:00
Joseph Doherty 517837e798 feat(ablegacy): encode array writes element by element
Review 07 U10 (user decision: implement array encode now). Both runtimes
encoded a scalar for an array tag: an int[] threw InvalidCastException into
BadTypeMismatch + Degraded, a scalar wrote element 0 natively, and libplctag's
SetInt32(0, ..) re-sent the stale last-read buffer for elements 1..n-1.

IAbLegacyTagRuntime gains EncodeArray; both runtimes write every element at
the element width (Int/AnalogInt 2 bytes, Long/Float 4). The driver converts
the written value before any runtime exists: each element is converted, not
exact-matched (ScadaBridge writes int[] to Int16 nodes), a length other than
the tag's element count is BadOutOfRange, an overflowing element is
BadOutOfRange, a non-array or non-numeric payload is BadTypeMismatch, and a Bit
array (bit-packed words; an encode would clobber neighbour bits) is
BadNotSupported. None of these touch health or the wire. Adds an env-gated
ab_server round trip for an N-file array write.

Refs #738
Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 14:25:12 -04:00
Joseph Doherty 8b50b43048 fix(ablegacy): deterministic NotSupported refusals are BadNotSupported with no health write
Review 07 H16: an array of String/Timer/Counter/Control passed the mapper, so
every poll created a handle (one init read), the decode threw
NotSupportedException, the generic catch evicted the handle and latched
Degraded. FromTagConfig now rejects that combination (the skip carries the
reason, Inspect/deploy gate and the AdminUI model state it), and a runtime
NotSupportedException on read or write is BadNotSupported: handle kept, no
health write, one Warning per tag and operation. Write refusals (native PLC-5
and status-bit writes, libplctag translated addresses) now warn once per tag too.

Refs #738
Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 14:20:17 -04:00
Joseph Doherty 7377daa9b0 fix(ablegacy): prune per-key gates when an indirect index moves
Review 07 L7: each index value an indirect tag (N7:[N7:0]) swept through left a
per-runtime and a creation SemaphoreSlim in the device's gate tables until
shutdown. A moved index now drops the stale resolution's gates along with its
handle, and sweeps any other stale resolution of the same tag a concurrent
reader cached, so the tables stay bounded by the authored set.

Refs #738
Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 14:17:16 -04:00
Joseph Doherty 92171a01ec fix(ablegacy): build-and-swap device/tag tables; teardown safe vs Dispose and abandoned init
Review 07 H13: _devices and _tagsByRawPath were plain dictionaries cleared and
refilled in place while the retained poll loop, the actor's GetHostStatuses and
server reads enumerated them. Both are now built locally and published by
reference under a lifecycle lock; teardown swaps in empty tables.

Lane A contract: ShutdownAsync and Dispose each retire a disjoint set of device
states, so Dispose running concurrently with a still-joining ShutdownAsync is
safe. A lifecycle epoch makes an InitializeAsync that the runtime abandoned
(overlapping the next ShutdownAsync) refuse to publish its tables, start probe
loops or write health. A runtime cached on an already-retired device state by an
in-flight read is disposed instead of leaked, and late reads/writes no longer
overwrite shutdown's health. Per-key semaphores are no longer disposed under
callers that may still hold them.

Refs #738
Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 14:04:51 -04:00
Joseph Doherty 8a4cf8b823 Merge lane F: Galaxy driver teardown, probing, knobs (#735)
Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 13:50:53 -04:00
Joseph Doherty 0d367590db test(galaxy): isolate process-wide telemetry tests; bounded stream-cancel wait
GalaxyTelemetryTests (ActivityListener) and GatewayGalaxyDataWriterFailClosedTests
(MeterListener, filtered only by meter and instrument name) observe every Galaxy span or
measurement in the process. They now run in a collection with parallelization disabled
(F-FU6), because a parallel driver test, or its background probe seed, polluted their counts.
The stream-cancel assertion in GalaxyDriverShutdownTests goes back to a bounded wait: the
instantaneous IsCompleted check introduced in round 1 was racy.

Refs #735

Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 13:47:34 -04:00
Joseph Doherty 3e60a925dc fix(galaxy): background probe seed; generation-checked registrations
Review fix round 2 for lane F:
- N1: the init platform seed no longer runs inline under the init token. Galaxy's Tier-A
  Initialize budget is 5 s, and a cold gateway DiscoverHierarchy can spend all of it, so the
  OperationCanceledException rethrow failed init, opened the Initialize breaker and cost a full
  rebuild per retry. InitializeAsync now only resolves the platform source (no network) and
  queues a background seed tied to a per-runtime CTS that the teardown cancels. A result that
  lands after its runtime was torn down is discarded.
- N2: the seed worker serves the runtime that is live when each pass STARTS, so a request made
  under a new runtime while a stale pass is in flight always gets a pass of its own.
- N3: SubscribeAsync and the subscribe-once read capture the runtime generation before
  SubscribeBulk and register only if it is unchanged, atomically with the teardown's registry
  Clear (_registryGate). On a mismatch they release the handles and fail the call. Re-subscribe
  Rebind uses the same gate, and the production rebuild clears the registry unconditionally.
- PerPlatformProbeWatcher re-checks its stop after the SubscribeBulk await, atomically with
  the record and publish. A pass that straddles StopAsync releases the handles it acquired
  instead of writing the probe entry back into a cleared registry.

Refs #735

Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 13:47:34 -04:00
Joseph Doherty f327dd6cdb Merge lane A: fix hung-write test fixture (#731)
Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 13:39:58 -04:00
Joseph Doherty 9e180b72a3 test(runtime): de-flake the hung-write release test; name the real write bound
The fixture wedged EVERY write on one shared token-ignoring release gate, so
the later same-tag write was also hung, and the 300 ms pessimistic bound
abandoned it while the test was still polling before ReleaseWrites(). Its
reason was "write did not complete within" (the bound), not "queued for" (the
queue budget), and it had already entered the driver, so the tag had been
released. The release path is correct. Only the first write now hangs, and
later writes complete at once.

The abandon reason formatted the bound with TotalSeconds:F0, which printed a
300 ms bound as "0s". It now uses milliseconds, and the test asserts it.

Refs #731

Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 13:39:33 -04:00
Joseph Doherty 8a0acc4649 Merge master into remed/lane-F
Refs #735

Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 13:29:15 -04:00
Joseph Doherty 77ca10aa37 test(galaxy): de-flake writer-lane and supervisor tests; align EffectiveTimeout doc
The writer concurrency fake parks on its release signal (the 300 ms delay could release the
gate early on a starved pool), the supervisor test's cross-thread hold flag is Volatile, and
the EffectiveTimeout doc now states the 1 s floor the code and tests implement.

Refs #735

Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 13:29:11 -04:00
Joseph Doherty 3456756cd1 fix(galaxy): seed the probe set at init, refuse pumps mid-teardown, bound the close
Review fix round 1 for lane F:
- The ScanState probe watcher is seeded at InitializeAsync from a platform-only
  DiscoverHierarchy query (IGalaxyPlatformSource / GatewayGalaxyPlatformSource over the owned
  repository client) and re-seeded on every deploy-watcher rediscovery. The runtime never
  calls DiscoverAsync on a live driver, so per-platform host status was unreachable in
  production. A failed fetch logs a Warning and never fails init; re-seeds coalesce so a
  deploy during an in-flight pass still gets a pass that starts after it.
- EnsureEventPumpStarted refuses while the runtime is stopping, BEFORE reusing the live pump
  (that pump belongs to the dying subscriber), and the production rebuild disposes any stale
  pump before wiring the new subscriber.
- TeardownRuntimeAsync closes the MX session under the caller's token linked with a 5 s
  SessionCloseDeadline, so a blackholed gateway cannot hold DriverInstanceActor.PostStop for 30 s.
- A runtime generation counter, bumped by every teardown, is re-checked before each Rebind
  in ResubscribeAllAsync, so a replay-gap re-snapshot that straddles a teardown cannot
  resurrect handles into the cleared registry.

Refs #735

Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 13:29:11 -04:00
Joseph Doherty 05a0e51fc1 fix(galaxy): GalaxyMxSession.CloseAsync(ct) bounds the gateway CloseSession
The SDK's DisposeAsync closes with no token, i.e. under the 30 s DefaultCallTimeout, so a
blackholed gateway parked the driver's teardown for the whole of it. CloseAsync(ct) runs the
close under the caller's token, drops the channel on a cancelled/failed close (the gateway
reaps the session on its idle lease) and leaves the session re-openable. A close followed by
the final dispose closes the gateway session once.

Refs #735

Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 13:29:11 -04:00
Joseph Doherty d8c39933ed Merge lane A: runtime driver lifecycle (#731)
Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 10:44:22 -04:00
Joseph Doherty 7ae74ef147 fix(runtime): pessimistic write bound, incarnation-stamped write completions (fix round 2)
- RunWriteAsync wraps the invoker call in WaitAsync(writeTimeout) (Props,
  default 10 s = 2x the 5 s write cap): a token-ignoring hung write now
  answers BadTimeout, releases its slot and tag, and its orphan outcome
  is observed and discarded - later writes to that tag reach the device.
- WriteCompleted carries the launching incarnation (from the process-wide
  generation source); a late completion from a previous incarnation after
  a supervision restart is ignored instead of freeing the new
  incarnation's slot/tag.
- Test: a hung token-ignoring tag-a write is abandoned and a later tag-a
  write reaches the device while the abandoned one is still inside it.

Refs #731

Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 10:43:45 -04:00
Joseph Doherty d422eb747b Merge lane H1: HistoryRead raw cap (#736)
Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 10:41:54 -04:00
Joseph Doherty 3e62683517 chore(historian): review cleanups for the raw cap
- GatewayHistorianDataSource.ReadRawWindowAsync keeps the cap + 1 probe budget
  (budgetCap) separate from the client's cap: a strict-ceiling or crowding
  refusal now reports Limit = the server cap that applied and advises setting
  NumValuesPerNode at or below it (it said "lower NumValuesPerNode" with
  Limit = cap + 1 to a client that sent 0).
- Fuzz: the oracle-source over-cap assertion was vacuous (cap 0 never counts);
  it now checks no reverse window reached the forward-only shape. Class docs
  name the post-floor cap set (3 ... 1000) and distinguish NumValuesPerNode
  1 / 2 from the smallest server cap.
- HistoryPaging.IsFullPage doc says it judges the REQUEST cap (page + ties +
  lookahead), which is finite under any limit; AdvertiseMaxReturnDataValues
  swallows any exception (returns false for the caller to log), as its doc
  promised.

Refs #736

Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 10:41:37 -04:00
Joseph Doherty a01f5e6705 build: pin patched System.Security.Cryptography.* / SignalR to clear fresh-restore NuGet audit
Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 10:38:07 -04:00
Joseph Doherty 953bd27533 fix(runtime,adminui): per-tag write ordering, pessimistic init bound, unique generations, PostStop budget (fix round 1)
- Writes: a second write to a tag already inside the driver waits in the
  queue behind it (_tagsInFlight) while other tags still run under the
  cap of 4; queued writes are stamped and failed BadTimeout at dequeue
  once older than the 8 s host Ask budget (Props queuedWriteBudget)
  instead of being written late.
- Initialise attempts are bounded pessimistically (initializeTimeout,
  default 30 s = 2x Tier C) on top of the invoker's token-only timeout:
  an overrun is logged as abandoned, reported InitializeFailed so the
  retry loop continues, and its late result is dropped by generation.
  Generations come from a process-wide Interlocked counter so a prior
  incarnation's late result cannot collide (F6).
- PostStop shares one shutdownTimeout budget across shutdown + dispose.
- TelemetryDialSupervisor cancels the establishment deadline on
  StreamStopped; FleetDiagnosticsClient builds the ActorSelection inside
  the try (a malformed node degrades to "unavailable"); the diagnostics
  page labels the column "Last health publish".
- Tests: same-tag serialisation with a distinct-tag positive control;
  stale queued write BadTimeout; token-ignoring hung init -> later
  attempt connects; stopwatch assertion demoted to a ceiling.

Refs #731

Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 10:36:30 -04:00
Joseph Doherty 7e8941fbdc test(historian): paged-vs-unpaged raw-cap fuzz, MaxReturnDataValues advertisement, cap floor of 3
- NodeManagerHistoryReadRawCapFuzzTests: 10k seeds (env OTOPCUA_PAGING_FUZZ_ITERATIONS)
  through a real SDK node manager against wire-faithful fakes of SqlServer / QuestDB /
  Gateway that enforce the backend half of the cap; random series with ties, random
  windows (empty, on/past a tie, between samples, forward/reverse/end-only, bounds),
  N in {0,1,2,small,huge}, cap in {off,3..1000}, tie budget {3,8,65536}. Pages must
  concatenate to the cap-off one-shot answer (checked against the series), terminate
  within values + 2 (+ one empty step per oversized boundary cluster), never exceed
  the cap, never ask the backend for 0 / over-cap; refusals only past the budget.
- HistoryPaging.AdvertiseMaxReturnDataValues writes HistoryServerCapabilities/
  MaxReturnDataValues (i=11273), read back through the SDK master node manager.
- MinMaxRawValuesPerRead raised to 3: at 2 a resumed page has no room (value + its
  boundary tie + lookahead) and every value cost an empty page + CP (seed 15).
- RawPageLimit rows pin the default-cap arithmetic (9 999 first page, 9 998 resumed).

Fuzz findings recorded as follow-ups: a bounded read whose data fits one page
returns cap + 1 (end bound fitted against N, not the cap; node manager, F2);
the Gateway walk-back's cost on a sparse series under a tiny tie budget is
pre-existing (seed 320: 82 reads uncapped, 93 capped).

Refs #736

Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 10:15:21 -04:00
Joseph Doherty 59b72fe385 fix(runtime): diagnostics Name stays the string instance id; order-tolerant write-cap test (A6 follow-through)
DriverInstanceDiagnostics.Name carries the string driver-instance id
(what /hosts, telemetry and the artifact key on) and DriverInstanceId
the DB row Guid; the wire record has no slot for the authored display
name. The write-cap test asserts entry order per slot pair instead of
completion order of two concurrent writes.

Refs #731

Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 10:15:01 -04:00
Joseph Doherty d1cc9e2cfc feat(runtime,adminui): real GetDiagnostics from child health + /fleet/diagnostics consumer (A6)
DriverHostActor wraps the health publisher it hands its children in a
ChildHealthLedger (last snapshot per driver id, forwarded to the real
publisher), so GetDiagnostics now answers with each child's row id,
authored name, last-published state, device counts (host statuses when
the driver probes hosts, else one device) and last-change time — instead
of Guid.Empty / "Spawned" / 0 / now. Entries are forgotten on StopChild
and operator Restart.

IFleetDiagnosticsClient gains its first consumer: the new
/fleet/diagnostics page (node selector + ?node= query, revision,
running-from-cache notice, per-driver table). FleetDiagnosticsClient
logs the degrade-to-empty path at Warning.

Review ref: 02/U1.

Refs #731

Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 06:44:38 -04:00
Joseph Doherty 6dedf5158f feat(galaxy): wire DiscoverPageSize and ProbeTimeoutSeconds; concurrent first-touch write lanes
Review 2026-10-21 G-5, G-6, G-8 ("wire them all").

- G-5: GatewayGalaxyHierarchySource now pages the DiscoverHierarchy RPC
  itself (DiscoverHierarchyRawAsync) at Repository.DiscoverPageSize. The
  mxaccessgw client's typed DiscoverHierarchyAsync overloads hard-code a
  5000-row page and DiscoverHierarchyOptions carries no page size, so the
  authored knob reached nothing; the raw request takes page_size on the
  wire, so it is honoured here (non-positive → 5000). The loop keeps the
  client's repeated-page-token guard.
- G-6: GalaxyDriverProbe reads ProbeTimeoutSeconds from the config and
  runs under min(caller budget, configured cap) — the knob can shorten a
  probe, never extend it past the host's budget. The TCP preflight now
  shares that deadline instead of hanging on the caller's token alone.
- G-8: GatewayGalaxyDataWriter runs a batch as one lane per distinct tag,
  concurrently and bounded at 8 (MaxConcurrentWriteLanes); same-tag
  writes keep their authored order inside one lane and results land at
  their request's index. Acks stay serial by design.
- GalaxyDriverForm: help text now says what the two knobs actually do.

Refs #735

Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 06:41:47 -04:00
Joseph Doherty c6d4b14c93 feat(telemetry): wire TelemetryDial:CallTimeoutSeconds as the dial-establishment deadline (A5)
CallTimeoutSeconds was bound, defaulted and documented but consumed
nowhere. TelemetryDialSupervisor now arms a per-generation deadline on
every dial: a dial that has delivered no event by the deadline is
cancelled and treated as a transport failure (reconnect with a failure
streak). A live node replays its snapshots on subscribe, so this bites
only a dial that never connects (a black-holed host, where the socket
connect has no timeout). An established stream is deliberately not
bounded — the wire has no heartbeat, keepalive pings own its liveness.

Review ref: 02/U2.

Refs #731

Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 06:38:32 -04:00
Joseph Doherty 79d6b0a4b2 fix(runtime): lifecycle housekeeping — reprovision stamp, Stale drops, off-thread DB probe, tombstone sweep (A4)
- DriverHostActor.StopChild removes the driver's _lastReprovisionUtc stamp
  (and its deferred-reprovision timer): the map grew per removed id and a
  re-added driver inherited a backoff it never earned (02/L2).
- Stale now registers explicit drops for AttributeValueBatchPublished,
  AttributeAlarmPublished, ArtifactForApply and ApplyWedged instead of
  letting them hit Unhandled (02/H5).
- DbHealthProbeActor runs its SELECT on the thread pool and pipes the
  outcome back (single-flight), with a 2 s command timeout on relational
  providers, so a hung SQL Server no longer pins the actor thread and the
  cached GetStatus answer stays reachable (02/P7).
- TelemetryLocalHub eviction tombstones are timestamped and swept past a
  10 min TTL on the next Evict; a TimeProvider ctor is the test seam (02/L4).
- Tests: ValuePublishes.Single helper replaces the per-value
  AttributeValuePublished in CalculationDependencyFlowTests (02/U4, test
  half; the production receive is owned by Lane E).
- R05/8 audit: PollGroupEngine.Unsubscribe has no production caller; every
  driver already uses UnsubscribeAsync (no change).

Refs #731

Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 06:36:12 -04:00
Joseph Doherty 1b3169273d fix(galaxy): wire ScanState probing, single-sweep discovery, liveness stamp, dispatch-loop fault
Review 2026-10-21 G-4, G-7, G-11, G-10.

- G-4: PerPlatformProbeWatcher publishes its probe handles into the
  driver's SubscriptionRegistry under the reserved ProbeSubscriptionId
  (-1), so the EventPump's fan-out finally reaches the watcher — every
  ScanState event used to be dropped as "stale after unsubscribe" and
  per-platform host status never updated. ResubscribeAllAsync delegates
  the reserved entry to the watcher's new ResubscribeAsync, so the probe
  survives a reconnect; StopAsync removes the entry. Probe traffic is
  routed by the reserved handle and never reaches the public OnDataChange.
  The watcher is also built on the seam path (subscriber only), which is
  what makes the wiring testable without a gateway.
- G-7: GalaxyDiscoverer.DiscoverAsync returns the hierarchy it fetched and
  the driver selects its $WinPlatform/$AppEngine probe targets from that
  same sweep instead of asking the gateway for the whole galaxy twice.
- G-11: OnPumpDataChange stamps LastSuccessfulRead from inbound stream
  data, throttled (1 s default), so the comms-loss watchdog has honest
  liveness evidence under a Degraded supervisor.
- G-10: an EventPump dispatch-loop exit that is not a cancellation now
  reports a stream fault so the supervisor recreates the pump, instead of
  leaving a deaf pump reporting Healthy.

Refs #735

Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 06:35:48 -04:00
Joseph Doherty 6e2febbf3d fix(runtime): run driver writes off the mailbox under an in-flight cap (A3)
WriteAttribute was a ReceiveAsync, suspending the child's mailbox for the
whole device round-trip (up to 5 s): writes serialised, the second of two
operator writes to a slow device missed the host's 8 s Ask and was
reverted although the device took it, and every data change queued behind
the write. Writes are now a synchronous Receive that launches the device
call on the thread pool (sender captured, reply from the task, slot freed
by a self-sent WriteCompleted) under a per-driver cap (default 4; excess
writes queue in order). Queued writes fail fast on every exit from
Connected. Subscribe/unsubscribe stay ReceiveAsync.

Review ref: 02/P2.

Refs #731

Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 06:30:57 -04:00
Joseph Doherty 2d25c916b6 fix(galaxy): make ShutdownAsync a real runtime teardown so re-init rebuilds
GalaxyDriver.ShutdownAsync was a no-op: the MX session, both owned gRPC
channels, the EventPump, the alarm feed, the DeployWatcher, the probe
watcher and the reconnect supervisor were released only by DisposeAsync,
which the host never calls — one leaked gateway session per respawn,
removal and /raw browse, and a watchdog Shutdown→Initialize that neither
tore down nor rebuilt anything (review 2026-10-21 G-1, G-2, G-9, G-3).

- ShutdownAsync and DisposeAsync share one ordered TeardownRuntimeAsync
  (supervisor first, then deploy/probe watchers, pump, alarm feed,
  session, channels); owned seams are nulled so the next InitializeAsync
  rebuilds; injected seams survive a shutdown and drop only on the final
  dispose. Lifecycle calls are serialised behind a gate.
- A failed production connect releases its partial runtime (no orphaned
  client per watchdog retry) and lands Faulted before and after.
- StartDeployWatcherAsync stops the previous watcher before starting a
  replacement, and now starts on a production instance's FIRST
  InitializeAsync (the old guard returned whenever discovery had not run
  here, which on a runtime instance is always).
- ReconnectSupervisor releases each cycle's CancellationTokenSource when
  its loop exits and gains an awaitable StopAsync; PerPlatformProbeWatcher
  gains an awaitable StopAsync.
- EventPump treats any exception under cancellation as a clean stop, so a
  teardown never wakes the supervisor.
- Internal seams for tests: an IGalaxyDeployWatchSource ctor parameter and
  a session-open override forwarded to the owned GalaxyMxSession.

Refs #735

Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 06:30:14 -04:00
Joseph Doherty 6669fae7e5 feat(historian): ServerHistorian:MaxRawValuesPerRead caps raw pages and backend reads
Review R08:1.1: an uncapped HistoryRead-Raw (NumValuesPerNode = 0) read a
tag's whole window into memory on every backend (SQL TOP(int.MaxValue),
QuestDB LIMIT, Gateway max_values = int.MaxValue + a second snapshot copy).

New ServerHistorian:MaxRawValuesPerRead (default 10 000; <= 0 unbounded with
a Validate() warning; 1 raised to 2). Each backend data source takes it,
reports it as IHistorianDataSource.MaxRawValuesPerRead (the Gateway: min
with a strict gateway's learned ceiling), so the node manager's existing
RawPageLimit paging serves N = 0 and N > cap as pages of at most the cap
with a continuation point. Independently, a backend raw read asking for 0
or more than the cap reads cap + 1 rows and is REFUSED
(HistoryReadLimitExceededException => BadTooManyOperations) when the window
holds more - never answered truncated, since the server's paging reads a
short answer as the last page. Bound through the Host registry, the
SqlServer/QuestDb providers and GatewayHistorian.CreateDataSource.

Tests: options (default/binding/warnings), provider binding, Gateway unit +
through a real SDK node manager (uncapped -> 999/998/998/605 + CPs), and a
conformance case for the SQL backends.

Refs #736

Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 06:29:10 -04:00
Joseph Doherty bfd5b7b015 fix(runtime): single-flight reconnect attempts, bounded pre-reconnect teardown (A2)
The Reconnecting retry timer started a new ShutdownAsync + InitializeAsync
pair on every tick whether or not the previous attempt had finished; the
generation token only discarded the older RESULT, so with an Initialize
bound longer than the cadence the teardown ran underneath an attempt still
inside the driver (the dispose-before-overwrite race). An _initInFlight
gate, cleared by the piped InitializeSucceeded/Failed, skips a tick while
an attempt is running; the pre-reconnect teardown is bounded by the
shutdown timeout; PostStop cancels an in-flight attempt before shutting
the driver down.

Review ref: 02/H2.

Refs #731

Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 06:26:45 -04:00
Joseph Doherty eb23053700 fix(runtime): dispose drivers on final stop, bound PostStop shutdown (A1)
DriverInstanceActor.PostStop now bounds IDriver.ShutdownAsync by a
shutdownTimeout (default 5 s, logged on overrun) and then disposes the
driver (IAsyncDisposable preferred, else IDisposable) — but only on a
real stop. PreRestart flags the restart so the PostStop it triggers
skips the dispose, because a supervision restart re-uses the same
driver object (Props closure) and the host re-initialises it.

DiscoveryDriverBrowser disposes its throwaway capture/probe driver
after the bounded shutdown, including after a failed or timed-out one.

Review refs: 02/L1, 02/P4, 06/G-1 (host context), 07/L2.

Refs #731

Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 06:23:39 -04:00
Joseph Doherty 7513ce5144 docs(review): record pre-remediation baseline test run
Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 06:11:55 -04:00
Joseph Doherty ad2b5c4044 docs: add 2026-10-21 system review + remediation plan
Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 06:04:58 -04:00
Joseph Doherty d51d35bd64 Merge branch 'docs/xml-doc-cleanup'
v2-ci / unit-tests (push) Failing after 3m14s
2026-10-15 06:25:09 -04:00
Joseph Doherty 8c6067b62e docs: move subsystem gotchas out of CLAUDE.md into path-scoped .claude/rules
Keeps the root CLAUDE.md lean by lazy-loading per-area detail (drivers, topology,
data-plane auth, alarms, historian) only when matching files are touched.
2026-10-15 06:14:58 -04:00
Joseph Doherty 826312befe docs: fill missing XML docs and strip tracking IDs from src comments
Clears CommentChecker findings in src/ (3,280 -> 209, the remainder being false
positives like PLC-5/IEEE-754/M0.0) by adding missing summaries/params/returns,
normalising inheritdoc usage, and removing Gitea/review tracking IDs while keeping
the rationale they annotated. Comment-only: comment-stripped sources are identical
to the parent commit.
2026-10-15 06:14:58 -04:00
dohertj2 ebcbd81fca Merge pull request 'Ensure driver test cleanup preserves primary failures' (#730) from test/task-assertion-return-discard into master
v2-ci / unit-tests (push) Successful in 1h1m21s
2026-10-13 21:06:31 -04:00
Joseph Doherty 4f537a5652 Merge remote-tracking branch 'origin/master' into test/task-assertion-return-discard
v2-ci / unit-tests (pull_request) Successful in 1h1m9s
2026-10-13 19:03:52 -04:00
dohertj2 c89912a93b Merge pull request 'Fix DB-less ServiceLevel freshness and authorization remarks' (#724) from fix/dbless-servicelevel-freshness into master
v2-ci / unit-tests (push) Waiting to run
2026-10-13 19:03:42 -04:00
Joseph Doherty c4b4046832 test: verify missing-snapshot ServiceLevel timer behavior
v2-ci / unit-tests (pull_request) Successful in 1h0m42s
2026-10-13 17:19:50 -04:00
Joseph Doherty eed7cc7d75 test: guarantee driver fixture cleanup without masking failures 2026-10-13 16:36:38 -04:00
Joseph Doherty 5310a7e07a Discard asserted probe-task return values without joining loops 2026-10-13 16:12:31 -04:00
Joseph Doherty e1feb9fbb6 Merge remote-tracking branch 'origin/master' into fix/dbless-servicelevel-freshness
v2-ci / unit-tests (pull_request) Waiting to run
2026-10-13 15:58:35 -04:00