Commit Graph

4228 Commits

Author SHA1 Message Date
Joseph Doherty f179829ca6 fix(twincat): review round 1 - PLC-not-in-RUN degrades, per-device health, retry backoff
- IMPORTANT: a successful probe records the PLC's ADS state; a PLC not in Run
  keeps the driver Degraded with 'PLC in {state}' while LastSuccessfulRead
  still advances (the device answered). RecordDeviceAnswered never restores
  Healthy while any device is out of Run.
- Health is per device, rolled up worst-of: a comms fault is remembered on the
  device it was seen on and only that device answering clears it (device A
  answering no longer clears device B's fault).
- Refused (non-comms) dead registrations back off 30 s doubling to 10 min and
  are re-armed by a symbol-version change or retried by the reconnect replay;
  comms-class failures keep the every-tick retry. A retry whose failure class
  changes republishes the tag's status, once.
- Bit-RMW gates are kept across generations and teardowns (never cleared).
- ShutdownAsync honours its CancellationToken: it stops waiting, finishes the
  bookkeeping and throws OperationCanceledException.
- Remarks on why a superseded InitializeAsync throws OperationCanceledException.
- REAL / LREAL values (scalar or array element) outside the target range are
  OverflowException -> BadOutOfRange instead of being written as Infinity.
- String / array write encoding factored into AdsTwinCATClient.PlanStringWrite
  / PlanArrayWrite; the fake runs them so a driver-level test covers the real
  encode path against a fake resolved symbol.
- Timing-fragile tests use condition waits / measured probe-tick windows; the
  dispose-during-shutdown test asserts ordering with a 1 s bound.

Refs #739
Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 14:47:24 -04:00
Joseph Doherty 8de5947e2d test(twincat): XAR-gated wire cases for H11/H12/H18/U9
Native-subscribe isolation of an unknown symbol, LastSuccessfulRead advance
from pushes alone, a WORD bit write flipping exactly one bit (bit 16
refused), a whole-array REAL write round-trip (length mismatch refused) and a
STRING(80) write round-trip (overflow refused). Every writing case restores
the seed it touched. Skipped without the TwinCAT XAR fixture.

Refs #739
Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 14:30:05 -04:00
Joseph Doherty 6201bafea4 test(twincat): pass the test token to the abandoned-init wait
Refs #739
Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 14:28:53 -04:00
Joseph Doherty 9beeed7a76 feat(twincat): implement IBatchSubscribable (R02:P1)
The poll engine already builds a per-cycle batch; TwinCAT now forwards it as
OnDataChangeBatch, so DriverInstanceActor gets one message per poll cycle
instead of one per changed tag. Because the actor subscribes ONLY the batch
event of a batch-capable driver, every native-path data change (pushes, the
BadNodeIdUnknown / refusal / unavailable snapshots) is raised through a single
RaiseDataChange that fires both the per-tag event and a fresh one-item batch.

Refs #739
Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 14:25:20 -04:00
Joseph Doherty cbacaee4ee fix(twincat): bounded, race-safe lifecycle; drop counter; post-unsubscribe purge (L8)
- Device and tag tables are build-and-swap: InitializeAsync builds a fresh
  generation and publishes it by one swap under a lifecycle lock, then
  releases the replaced generation; teardown swaps in empty tables. A
  retained poll loop never sees a table mid-Clear (it used to read
  _tagsByRawPath while a re-init only ever added to it), and each device
  set is released by exactly one caller.
- A lifecycle generation makes an InitializeAsync overtaken by a shutdown
  (or a newer init) release what it built instead of installing it - the
  Lane A abandoned-init-overlaps-shutdown case.
- ShutdownAsync joins all probe loops together within 2 s (it awaited each
  unbounded - a probe wedged in the blocking ADS Connect hung the actor's
  teardown) in parallel with the poll-engine dispose, the whole bounded at
  4.5 s; DisposeAsync may run concurrently and returns at once. A connect
  that completes on a released device disposes its client.
- Releasing a generation disposes its native registrations, so a direct
  re-init no longer leaves live notifications bound to dead DeviceStates.
- Connect gates and RMW semaphores are no longer disposed under in-flight
  users (no kernel handle is ever allocated); a fault on a released device
  is not recorded as driver health.
- The notification channel counts DropWrite overflows
  (ITwinCATClient.DroppedNotificationCount); the probe tick warns with the
  delta. A queued value whose registration was disposed is purged in the
  client dispatch loop and dropped driver-side for any client.

Refs #739
Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 14:24:10 -04:00
Joseph Doherty f17ba8e258 feat(twincat): array and STRING writes through the codecs (U9)
Array tags were advertised writable but every write threw InvalidCastException
in ConvertForWrite (reported BadTypeMismatch); STRING writes went through the
Beckhoff marshaller with no length check against STRING(n) and the host's
ANSI code page, which the read side deliberately avoids.

- Array write: TwinCATArrayCodec.ToWriteElements converts exactly ArrayLength
  elements to the declared type (CLR lists convert - ScadaBridge writes int[]
  to Int16 nodes; a length mismatch or out-of-range element is BadOutOfRange, a
  scalar value BadTypeMismatch) before any wire traffic; the ADS client plans
  the layout from the resolved PLC array (TwinCATTypeGuard.CheckArray, as the
  read does), encodes with TwinCATArrayCodec.Encode - the exact inverse of
  Decode - and writes the raw bytes (new ITwinCATClient.WriteArrayAsync).
- STRING / WSTRING write: encoded at the PLC's declared width by
  TwinCATStringCodec.EncodeString/EncodeWString (Windows-1252 or UTF-8 per
  target, NUL-padded, strict: an over-long value or unrepresentable character
  is OverflowException -> BadOutOfRange) and written as raw bytes.
- TwinCAT CLI read/write gain --array-length (+ --separator) for the hardware
  gate.

Refs #739
Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 14:15:29 -04:00
Joseph Doherty 40c7493829 fix(twincat): BOOL-within-word writes RMW the parent at its own width (H18)
The bit write read and wrote the parent word as a 32-bit UDInt with no parent
check: a bit beyond a 16-bit parent was written into neighbouring bytes, and a
signed parent with its sign bit set read back >= 2^31, so the checked
conversion turned any bit write into BadOutOfRange.

The write now goes through ITwinCATClient.WriteBitAsync: the ADS client
resolves the parent symbol, checks the bit against it (TwinCATBitAccess.
CheckParent, as the notification path already did), reads exactly
parent.ByteSize raw bytes, flips one bit (TwinCATBitAccess.ApplyBitWrite) and
writes the bytes back by index group/offset or variable handle. A bit beyond
the parent is BadOutOfRange, a non-integer parent BadTypeMismatch, a short
parent read BadCommunicationError (never written back), all without a health
write. The per-parent RMW lock stays in the driver.

Refs #739
Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 14:10:12 -04:00
Joseph Doherty 4083b4aada fix(twincat): warn once per bad tag; a device refusal is not a comms fault (P10)
CompleteRead logged a Warning for every non-Good status on every poll and
latched Degraded for all of them. Now a bad tag warns once per bad episode
(re-armed when it reads Good, and on re-initialise), a comms-class status
(BadCommunicationError / BadTimeout) degrades with LastSuccessfulRead
preserved, and any other status - the device answered and refused this tag -
advances LastSuccessfulRead with no state change. Read-path exception
handlers now go through MarkCommsFault, so they never downgrade Faulted.

Refs #739
Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 14:07:03 -04:00
Joseph Doherty ecdd6dbecb fix(twincat): native/probe evidence advances LastSuccessfulRead; probe failures degrade (H12)
In native-notification mode LastSuccessfulRead never moved past the init
stamp, so one Degraded write on a connection older than 90 s latched the
comms-loss watchdog into a full reconnect (every notification on every device
deleted and re-registered). Conversely an unreachable AMS router was swallowed
by the probe and never left Healthy.

- a native push and a successful probe now stamp LastSuccessfulRead (throttled
  to one health record per 250 ms) and restore Healthy unless a device is still
  probe-unreachable;
- 3 consecutive failed probe ticks write Degraded with the stamp preserved;
- a NotSupportedException write is BadNotSupported with no health write (#611);
- a reconnect replay the PLC refuses republishes the refusal's status without
  degrading (it used to report BadCommunicationError + Degraded on every
  reconnect for a correctly-isolated type-mismatch tag).

Refs #739
Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 14:03:28 -04:00
Joseph Doherty 17f4bf6d33 fix(twincat): isolate native registrations per tag and per device (H11)
Native subscribe was all-or-nothing: any registration failure other than a
type refusal disposed every registration and rethrew, so one typo'd symbol or
one unreachable PLC in a multi-device driver served zero data, churned
register/delete every reconcile tick, and left health Healthy.

Now each registration fails on its own: a refusal (the device answered: symbol
not found, PLC not ready, type mismatch) publishes its status with no health
write; any other per-tag failure publishes BadCommunicationError and degrades;
a connect failure marks the device unreachable for the rest of the call and
stores its tags as dead intents. Every failed tag is kept as a dead intent that
the probe-tick retry and the reconnect replay re-register. The ADS client now
classifies a failed symbol resolve / AddDeviceNotification by status instead of
throwing InvalidOperationException for all of them. Cancellation still tears
the partial subscription down.

Refs #739
Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 14:00:38 -04:00
Joseph Doherty de21a54949 fix(twincat): publish BadNodeIdUnknown for unresolvable native-subscribe references (U12)
The native subscribe path silently skipped a reference it could not resolve
(unknown RawPath, or a tag routed to an unconfigured device), leaving the node
on whatever status the server seeded. It now publishes one BadNodeIdUnknown
snapshot, matching the poll path.

Refs #739
Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 13:57:01 -04:00
Joseph Doherty 8a4cf8b823 Merge lane F: Galaxy driver teardown, probing, knobs (#735)
Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 13:50:53 -04:00
Joseph Doherty 0d367590db test(galaxy): isolate process-wide telemetry tests; bounded stream-cancel wait
GalaxyTelemetryTests (ActivityListener) and GatewayGalaxyDataWriterFailClosedTests
(MeterListener, filtered only by meter and instrument name) observe every Galaxy span or
measurement in the process. They now run in a collection with parallelization disabled
(F-FU6), because a parallel driver test, or its background probe seed, polluted their counts.
The stream-cancel assertion in GalaxyDriverShutdownTests goes back to a bounded wait: the
instantaneous IsCompleted check introduced in round 1 was racy.

Refs #735

Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 13:47:34 -04:00
Joseph Doherty 3e60a925dc fix(galaxy): background probe seed; generation-checked registrations
Review fix round 2 for lane F:
- N1: the init platform seed no longer runs inline under the init token. Galaxy's Tier-A
  Initialize budget is 5 s, and a cold gateway DiscoverHierarchy can spend all of it, so the
  OperationCanceledException rethrow failed init, opened the Initialize breaker and cost a full
  rebuild per retry. InitializeAsync now only resolves the platform source (no network) and
  queues a background seed tied to a per-runtime CTS that the teardown cancels. A result that
  lands after its runtime was torn down is discarded.
- N2: the seed worker serves the runtime that is live when each pass STARTS, so a request made
  under a new runtime while a stale pass is in flight always gets a pass of its own.
- N3: SubscribeAsync and the subscribe-once read capture the runtime generation before
  SubscribeBulk and register only if it is unchanged, atomically with the teardown's registry
  Clear (_registryGate). On a mismatch they release the handles and fail the call. Re-subscribe
  Rebind uses the same gate, and the production rebuild clears the registry unconditionally.
- PerPlatformProbeWatcher re-checks its stop after the SubscribeBulk await, atomically with
  the record and publish. A pass that straddles StopAsync releases the handles it acquired
  instead of writing the probe entry back into a cleared registry.

Refs #735

Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 13:47:34 -04:00
Joseph Doherty f327dd6cdb Merge lane A: fix hung-write test fixture (#731)
Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 13:39:58 -04:00
Joseph Doherty 9e180b72a3 test(runtime): de-flake the hung-write release test; name the real write bound
The fixture wedged EVERY write on one shared token-ignoring release gate, so
the later same-tag write was also hung, and the 300 ms pessimistic bound
abandoned it while the test was still polling before ReleaseWrites(). Its
reason was "write did not complete within" (the bound), not "queued for" (the
queue budget), and it had already entered the driver, so the tag had been
released. The release path is correct. Only the first write now hangs, and
later writes complete at once.

The abandon reason formatted the bound with TotalSeconds:F0, which printed a
300 ms bound as "0s". It now uses milliseconds, and the test asserts it.

Refs #731

Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 13:39:33 -04:00
Joseph Doherty 8a0acc4649 Merge master into remed/lane-F
Refs #735

Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 13:29:15 -04:00
Joseph Doherty 77ca10aa37 test(galaxy): de-flake writer-lane and supervisor tests; align EffectiveTimeout doc
The writer concurrency fake parks on its release signal (the 300 ms delay could release the
gate early on a starved pool), the supervisor test's cross-thread hold flag is Volatile, and
the EffectiveTimeout doc now states the 1 s floor the code and tests implement.

Refs #735

Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 13:29:11 -04:00
Joseph Doherty 3456756cd1 fix(galaxy): seed the probe set at init, refuse pumps mid-teardown, bound the close
Review fix round 1 for lane F:
- The ScanState probe watcher is seeded at InitializeAsync from a platform-only
  DiscoverHierarchy query (IGalaxyPlatformSource / GatewayGalaxyPlatformSource over the owned
  repository client) and re-seeded on every deploy-watcher rediscovery. The runtime never
  calls DiscoverAsync on a live driver, so per-platform host status was unreachable in
  production. A failed fetch logs a Warning and never fails init; re-seeds coalesce so a
  deploy during an in-flight pass still gets a pass that starts after it.
- EnsureEventPumpStarted refuses while the runtime is stopping, BEFORE reusing the live pump
  (that pump belongs to the dying subscriber), and the production rebuild disposes any stale
  pump before wiring the new subscriber.
- TeardownRuntimeAsync closes the MX session under the caller's token linked with a 5 s
  SessionCloseDeadline, so a blackholed gateway cannot hold DriverInstanceActor.PostStop for 30 s.
- A runtime generation counter, bumped by every teardown, is re-checked before each Rebind
  in ResubscribeAllAsync, so a replay-gap re-snapshot that straddles a teardown cannot
  resurrect handles into the cleared registry.

Refs #735

Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 13:29:11 -04:00
Joseph Doherty 05a0e51fc1 fix(galaxy): GalaxyMxSession.CloseAsync(ct) bounds the gateway CloseSession
The SDK's DisposeAsync closes with no token, i.e. under the 30 s DefaultCallTimeout, so a
blackholed gateway parked the driver's teardown for the whole of it. CloseAsync(ct) runs the
close under the caller's token, drops the channel on a cancelled/failed close (the gateway
reaps the session on its idle lease) and leaves the session re-openable. A close followed by
the final dispose closes the gateway session once.

Refs #735

Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 13:29:11 -04:00
Joseph Doherty d8c39933ed Merge lane A: runtime driver lifecycle (#731)
Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 10:44:22 -04:00
Joseph Doherty 7ae74ef147 fix(runtime): pessimistic write bound, incarnation-stamped write completions (fix round 2)
- RunWriteAsync wraps the invoker call in WaitAsync(writeTimeout) (Props,
  default 10 s = 2x the 5 s write cap): a token-ignoring hung write now
  answers BadTimeout, releases its slot and tag, and its orphan outcome
  is observed and discarded - later writes to that tag reach the device.
- WriteCompleted carries the launching incarnation (from the process-wide
  generation source); a late completion from a previous incarnation after
  a supervision restart is ignored instead of freeing the new
  incarnation's slot/tag.
- Test: a hung token-ignoring tag-a write is abandoned and a later tag-a
  write reaches the device while the abandoned one is still inside it.

Refs #731

Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 10:43:45 -04:00
Joseph Doherty d422eb747b Merge lane H1: HistoryRead raw cap (#736)
Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 10:41:54 -04:00
Joseph Doherty 3e62683517 chore(historian): review cleanups for the raw cap
- GatewayHistorianDataSource.ReadRawWindowAsync keeps the cap + 1 probe budget
  (budgetCap) separate from the client's cap: a strict-ceiling or crowding
  refusal now reports Limit = the server cap that applied and advises setting
  NumValuesPerNode at or below it (it said "lower NumValuesPerNode" with
  Limit = cap + 1 to a client that sent 0).
- Fuzz: the oracle-source over-cap assertion was vacuous (cap 0 never counts);
  it now checks no reverse window reached the forward-only shape. Class docs
  name the post-floor cap set (3 ... 1000) and distinguish NumValuesPerNode
  1 / 2 from the smallest server cap.
- HistoryPaging.IsFullPage doc says it judges the REQUEST cap (page + ties +
  lookahead), which is finite under any limit; AdvertiseMaxReturnDataValues
  swallows any exception (returns false for the caller to log), as its doc
  promised.

Refs #736

Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 10:41:37 -04:00
Joseph Doherty a01f5e6705 build: pin patched System.Security.Cryptography.* / SignalR to clear fresh-restore NuGet audit
Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 10:38:07 -04:00
Joseph Doherty 953bd27533 fix(runtime,adminui): per-tag write ordering, pessimistic init bound, unique generations, PostStop budget (fix round 1)
- Writes: a second write to a tag already inside the driver waits in the
  queue behind it (_tagsInFlight) while other tags still run under the
  cap of 4; queued writes are stamped and failed BadTimeout at dequeue
  once older than the 8 s host Ask budget (Props queuedWriteBudget)
  instead of being written late.
- Initialise attempts are bounded pessimistically (initializeTimeout,
  default 30 s = 2x Tier C) on top of the invoker's token-only timeout:
  an overrun is logged as abandoned, reported InitializeFailed so the
  retry loop continues, and its late result is dropped by generation.
  Generations come from a process-wide Interlocked counter so a prior
  incarnation's late result cannot collide (F6).
- PostStop shares one shutdownTimeout budget across shutdown + dispose.
- TelemetryDialSupervisor cancels the establishment deadline on
  StreamStopped; FleetDiagnosticsClient builds the ActorSelection inside
  the try (a malformed node degrades to "unavailable"); the diagnostics
  page labels the column "Last health publish".
- Tests: same-tag serialisation with a distinct-tag positive control;
  stale queued write BadTimeout; token-ignoring hung init -> later
  attempt connects; stopwatch assertion demoted to a ceiling.

Refs #731

Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 10:36:30 -04:00
Joseph Doherty 7e8941fbdc test(historian): paged-vs-unpaged raw-cap fuzz, MaxReturnDataValues advertisement, cap floor of 3
- NodeManagerHistoryReadRawCapFuzzTests: 10k seeds (env OTOPCUA_PAGING_FUZZ_ITERATIONS)
  through a real SDK node manager against wire-faithful fakes of SqlServer / QuestDB /
  Gateway that enforce the backend half of the cap; random series with ties, random
  windows (empty, on/past a tie, between samples, forward/reverse/end-only, bounds),
  N in {0,1,2,small,huge}, cap in {off,3..1000}, tie budget {3,8,65536}. Pages must
  concatenate to the cap-off one-shot answer (checked against the series), terminate
  within values + 2 (+ one empty step per oversized boundary cluster), never exceed
  the cap, never ask the backend for 0 / over-cap; refusals only past the budget.
- HistoryPaging.AdvertiseMaxReturnDataValues writes HistoryServerCapabilities/
  MaxReturnDataValues (i=11273), read back through the SDK master node manager.
- MinMaxRawValuesPerRead raised to 3: at 2 a resumed page has no room (value + its
  boundary tie + lookahead) and every value cost an empty page + CP (seed 15).
- RawPageLimit rows pin the default-cap arithmetic (9 999 first page, 9 998 resumed).

Fuzz findings recorded as follow-ups: a bounded read whose data fits one page
returns cap + 1 (end bound fitted against N, not the cap; node manager, F2);
the Gateway walk-back's cost on a sparse series under a tiny tie budget is
pre-existing (seed 320: 82 reads uncapped, 93 capped).

Refs #736

Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 10:15:21 -04:00
Joseph Doherty 59b72fe385 fix(runtime): diagnostics Name stays the string instance id; order-tolerant write-cap test (A6 follow-through)
DriverInstanceDiagnostics.Name carries the string driver-instance id
(what /hosts, telemetry and the artifact key on) and DriverInstanceId
the DB row Guid; the wire record has no slot for the authored display
name. The write-cap test asserts entry order per slot pair instead of
completion order of two concurrent writes.

Refs #731

Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 10:15:01 -04:00
Joseph Doherty d1cc9e2cfc feat(runtime,adminui): real GetDiagnostics from child health + /fleet/diagnostics consumer (A6)
DriverHostActor wraps the health publisher it hands its children in a
ChildHealthLedger (last snapshot per driver id, forwarded to the real
publisher), so GetDiagnostics now answers with each child's row id,
authored name, last-published state, device counts (host statuses when
the driver probes hosts, else one device) and last-change time — instead
of Guid.Empty / "Spawned" / 0 / now. Entries are forgotten on StopChild
and operator Restart.

IFleetDiagnosticsClient gains its first consumer: the new
/fleet/diagnostics page (node selector + ?node= query, revision,
running-from-cache notice, per-driver table). FleetDiagnosticsClient
logs the degrade-to-empty path at Warning.

Review ref: 02/U1.

Refs #731

Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 06:44:38 -04:00
Joseph Doherty 6dedf5158f feat(galaxy): wire DiscoverPageSize and ProbeTimeoutSeconds; concurrent first-touch write lanes
Review 2026-10-21 G-5, G-6, G-8 ("wire them all").

- G-5: GatewayGalaxyHierarchySource now pages the DiscoverHierarchy RPC
  itself (DiscoverHierarchyRawAsync) at Repository.DiscoverPageSize. The
  mxaccessgw client's typed DiscoverHierarchyAsync overloads hard-code a
  5000-row page and DiscoverHierarchyOptions carries no page size, so the
  authored knob reached nothing; the raw request takes page_size on the
  wire, so it is honoured here (non-positive → 5000). The loop keeps the
  client's repeated-page-token guard.
- G-6: GalaxyDriverProbe reads ProbeTimeoutSeconds from the config and
  runs under min(caller budget, configured cap) — the knob can shorten a
  probe, never extend it past the host's budget. The TCP preflight now
  shares that deadline instead of hanging on the caller's token alone.
- G-8: GatewayGalaxyDataWriter runs a batch as one lane per distinct tag,
  concurrently and bounded at 8 (MaxConcurrentWriteLanes); same-tag
  writes keep their authored order inside one lane and results land at
  their request's index. Acks stay serial by design.
- GalaxyDriverForm: help text now says what the two knobs actually do.

Refs #735

Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 06:41:47 -04:00
Joseph Doherty c6d4b14c93 feat(telemetry): wire TelemetryDial:CallTimeoutSeconds as the dial-establishment deadline (A5)
CallTimeoutSeconds was bound, defaulted and documented but consumed
nowhere. TelemetryDialSupervisor now arms a per-generation deadline on
every dial: a dial that has delivered no event by the deadline is
cancelled and treated as a transport failure (reconnect with a failure
streak). A live node replays its snapshots on subscribe, so this bites
only a dial that never connects (a black-holed host, where the socket
connect has no timeout). An established stream is deliberately not
bounded — the wire has no heartbeat, keepalive pings own its liveness.

Review ref: 02/U2.

Refs #731

Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 06:38:32 -04:00
Joseph Doherty 79d6b0a4b2 fix(runtime): lifecycle housekeeping — reprovision stamp, Stale drops, off-thread DB probe, tombstone sweep (A4)
- DriverHostActor.StopChild removes the driver's _lastReprovisionUtc stamp
  (and its deferred-reprovision timer): the map grew per removed id and a
  re-added driver inherited a backoff it never earned (02/L2).
- Stale now registers explicit drops for AttributeValueBatchPublished,
  AttributeAlarmPublished, ArtifactForApply and ApplyWedged instead of
  letting them hit Unhandled (02/H5).
- DbHealthProbeActor runs its SELECT on the thread pool and pipes the
  outcome back (single-flight), with a 2 s command timeout on relational
  providers, so a hung SQL Server no longer pins the actor thread and the
  cached GetStatus answer stays reachable (02/P7).
- TelemetryLocalHub eviction tombstones are timestamped and swept past a
  10 min TTL on the next Evict; a TimeProvider ctor is the test seam (02/L4).
- Tests: ValuePublishes.Single helper replaces the per-value
  AttributeValuePublished in CalculationDependencyFlowTests (02/U4, test
  half; the production receive is owned by Lane E).
- R05/8 audit: PollGroupEngine.Unsubscribe has no production caller; every
  driver already uses UnsubscribeAsync (no change).

Refs #731

Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 06:36:12 -04:00
Joseph Doherty 1b3169273d fix(galaxy): wire ScanState probing, single-sweep discovery, liveness stamp, dispatch-loop fault
Review 2026-10-21 G-4, G-7, G-11, G-10.

- G-4: PerPlatformProbeWatcher publishes its probe handles into the
  driver's SubscriptionRegistry under the reserved ProbeSubscriptionId
  (-1), so the EventPump's fan-out finally reaches the watcher — every
  ScanState event used to be dropped as "stale after unsubscribe" and
  per-platform host status never updated. ResubscribeAllAsync delegates
  the reserved entry to the watcher's new ResubscribeAsync, so the probe
  survives a reconnect; StopAsync removes the entry. Probe traffic is
  routed by the reserved handle and never reaches the public OnDataChange.
  The watcher is also built on the seam path (subscriber only), which is
  what makes the wiring testable without a gateway.
- G-7: GalaxyDiscoverer.DiscoverAsync returns the hierarchy it fetched and
  the driver selects its $WinPlatform/$AppEngine probe targets from that
  same sweep instead of asking the gateway for the whole galaxy twice.
- G-11: OnPumpDataChange stamps LastSuccessfulRead from inbound stream
  data, throttled (1 s default), so the comms-loss watchdog has honest
  liveness evidence under a Degraded supervisor.
- G-10: an EventPump dispatch-loop exit that is not a cancellation now
  reports a stream fault so the supervisor recreates the pump, instead of
  leaving a deaf pump reporting Healthy.

Refs #735

Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 06:35:48 -04:00
Joseph Doherty 6e2febbf3d fix(runtime): run driver writes off the mailbox under an in-flight cap (A3)
WriteAttribute was a ReceiveAsync, suspending the child's mailbox for the
whole device round-trip (up to 5 s): writes serialised, the second of two
operator writes to a slow device missed the host's 8 s Ask and was
reverted although the device took it, and every data change queued behind
the write. Writes are now a synchronous Receive that launches the device
call on the thread pool (sender captured, reply from the task, slot freed
by a self-sent WriteCompleted) under a per-driver cap (default 4; excess
writes queue in order). Queued writes fail fast on every exit from
Connected. Subscribe/unsubscribe stay ReceiveAsync.

Review ref: 02/P2.

Refs #731

Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 06:30:57 -04:00
Joseph Doherty 2d25c916b6 fix(galaxy): make ShutdownAsync a real runtime teardown so re-init rebuilds
GalaxyDriver.ShutdownAsync was a no-op: the MX session, both owned gRPC
channels, the EventPump, the alarm feed, the DeployWatcher, the probe
watcher and the reconnect supervisor were released only by DisposeAsync,
which the host never calls — one leaked gateway session per respawn,
removal and /raw browse, and a watchdog Shutdown→Initialize that neither
tore down nor rebuilt anything (review 2026-10-21 G-1, G-2, G-9, G-3).

- ShutdownAsync and DisposeAsync share one ordered TeardownRuntimeAsync
  (supervisor first, then deploy/probe watchers, pump, alarm feed,
  session, channels); owned seams are nulled so the next InitializeAsync
  rebuilds; injected seams survive a shutdown and drop only on the final
  dispose. Lifecycle calls are serialised behind a gate.
- A failed production connect releases its partial runtime (no orphaned
  client per watchdog retry) and lands Faulted before and after.
- StartDeployWatcherAsync stops the previous watcher before starting a
  replacement, and now starts on a production instance's FIRST
  InitializeAsync (the old guard returned whenever discovery had not run
  here, which on a runtime instance is always).
- ReconnectSupervisor releases each cycle's CancellationTokenSource when
  its loop exits and gains an awaitable StopAsync; PerPlatformProbeWatcher
  gains an awaitable StopAsync.
- EventPump treats any exception under cancellation as a clean stop, so a
  teardown never wakes the supervisor.
- Internal seams for tests: an IGalaxyDeployWatchSource ctor parameter and
  a session-open override forwarded to the owned GalaxyMxSession.

Refs #735

Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 06:30:14 -04:00
Joseph Doherty 6669fae7e5 feat(historian): ServerHistorian:MaxRawValuesPerRead caps raw pages and backend reads
Review R08:1.1: an uncapped HistoryRead-Raw (NumValuesPerNode = 0) read a
tag's whole window into memory on every backend (SQL TOP(int.MaxValue),
QuestDB LIMIT, Gateway max_values = int.MaxValue + a second snapshot copy).

New ServerHistorian:MaxRawValuesPerRead (default 10 000; <= 0 unbounded with
a Validate() warning; 1 raised to 2). Each backend data source takes it,
reports it as IHistorianDataSource.MaxRawValuesPerRead (the Gateway: min
with a strict gateway's learned ceiling), so the node manager's existing
RawPageLimit paging serves N = 0 and N > cap as pages of at most the cap
with a continuation point. Independently, a backend raw read asking for 0
or more than the cap reads cap + 1 rows and is REFUSED
(HistoryReadLimitExceededException => BadTooManyOperations) when the window
holds more - never answered truncated, since the server's paging reads a
short answer as the last page. Bound through the Host registry, the
SqlServer/QuestDb providers and GatewayHistorian.CreateDataSource.

Tests: options (default/binding/warnings), provider binding, Gateway unit +
through a real SDK node manager (uncapped -> 999/998/998/605 + CPs), and a
conformance case for the SQL backends.

Refs #736

Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 06:29:10 -04:00
Joseph Doherty bfd5b7b015 fix(runtime): single-flight reconnect attempts, bounded pre-reconnect teardown (A2)
The Reconnecting retry timer started a new ShutdownAsync + InitializeAsync
pair on every tick whether or not the previous attempt had finished; the
generation token only discarded the older RESULT, so with an Initialize
bound longer than the cadence the teardown ran underneath an attempt still
inside the driver (the dispose-before-overwrite race). An _initInFlight
gate, cleared by the piped InitializeSucceeded/Failed, skips a tick while
an attempt is running; the pre-reconnect teardown is bounded by the
shutdown timeout; PostStop cancels an in-flight attempt before shutting
the driver down.

Review ref: 02/H2.

Refs #731

Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 06:26:45 -04:00
Joseph Doherty eb23053700 fix(runtime): dispose drivers on final stop, bound PostStop shutdown (A1)
DriverInstanceActor.PostStop now bounds IDriver.ShutdownAsync by a
shutdownTimeout (default 5 s, logged on overrun) and then disposes the
driver (IAsyncDisposable preferred, else IDisposable) — but only on a
real stop. PreRestart flags the restart so the PostStop it triggers
skips the dispose, because a supervision restart re-uses the same
driver object (Props closure) and the host re-initialises it.

DiscoveryDriverBrowser disposes its throwaway capture/probe driver
after the bounded shutdown, including after a failed or timed-out one.

Review refs: 02/L1, 02/P4, 06/G-1 (host context), 07/L2.

Refs #731

Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 06:23:39 -04:00
Joseph Doherty 7513ce5144 docs(review): record pre-remediation baseline test run
Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 06:11:55 -04:00
Joseph Doherty ad2b5c4044 docs: add 2026-10-21 system review + remediation plan
Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 06:04:58 -04:00
Joseph Doherty d51d35bd64 Merge branch 'docs/xml-doc-cleanup'
v2-ci / unit-tests (push) Failing after 3m14s
2026-10-15 06:25:09 -04:00
Joseph Doherty 8c6067b62e docs: move subsystem gotchas out of CLAUDE.md into path-scoped .claude/rules
Keeps the root CLAUDE.md lean by lazy-loading per-area detail (drivers, topology,
data-plane auth, alarms, historian) only when matching files are touched.
2026-10-15 06:14:58 -04:00
Joseph Doherty 826312befe docs: fill missing XML docs and strip tracking IDs from src comments
Clears CommentChecker findings in src/ (3,280 -> 209, the remainder being false
positives like PLC-5/IEEE-754/M0.0) by adding missing summaries/params/returns,
normalising inheritdoc usage, and removing Gitea/review tracking IDs while keeping
the rationale they annotated. Comment-only: comment-stripped sources are identical
to the parent commit.
2026-10-15 06:14:58 -04:00
dohertj2 ebcbd81fca Merge pull request 'Ensure driver test cleanup preserves primary failures' (#730) from test/task-assertion-return-discard into master
v2-ci / unit-tests (push) Successful in 1h1m21s
2026-10-13 21:06:31 -04:00
Joseph Doherty 4f537a5652 Merge remote-tracking branch 'origin/master' into test/task-assertion-return-discard
v2-ci / unit-tests (pull_request) Successful in 1h1m9s
2026-10-13 19:03:52 -04:00
dohertj2 c89912a93b Merge pull request 'Fix DB-less ServiceLevel freshness and authorization remarks' (#724) from fix/dbless-servicelevel-freshness into master
v2-ci / unit-tests (push) Waiting to run
2026-10-13 19:03:42 -04:00
Joseph Doherty c4b4046832 test: verify missing-snapshot ServiceLevel timer behavior
v2-ci / unit-tests (pull_request) Successful in 1h0m42s
2026-10-13 17:19:50 -04:00
Joseph Doherty eed7cc7d75 test: guarantee driver fixture cleanup without masking failures 2026-10-13 16:36:38 -04:00
Joseph Doherty 5310a7e07a Discard asserted probe-task return values without joining loops 2026-10-13 16:12:31 -04:00
Joseph Doherty e1feb9fbb6 Merge remote-tracking branch 'origin/master' into fix/dbless-servicelevel-freshness
v2-ci / unit-tests (pull_request) Waiting to run
2026-10-13 15:58:35 -04:00