Commit Graph

4227 Commits

Author SHA1 Message Date
Joseph Doherty fc94b87634 perf(abcip): cache the declaration-only UDT layout per member list
P11: AbCipUdtReadPlanner called AbCipUdtMemberLayout.TryBuild(parent.Members)
for every member reference on every poll - O(members^2) per parent per tick,
a dictionary allocation each time. GetOrBuild caches the layout (including a
"not groupable" verdict) in a ConditionalWeakTable keyed by the member list's
identity, which is immutable for the life of the tag table; a re-init's new
lists replace the entries and the old ones are collected with them.

Refs #737
Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 14:31:32 -04:00
Joseph Doherty 7cc8415310 feat(abcip): array writes encode count elements at the element width
U10 (user decision: implement array encode now): config, the AdminUI and
discovery all advertised array tags Operate-writable, but both runtimes encoded
a scalar - an int[] threw InvalidCastException (BadTypeMismatch AND Degraded),
a scalar wrote element 0 natively, and libplctag re-sent its whole last-read
buffer with element 0 patched (stale elements 1..n-1).

- AbCipArrayCodec encodes count elements little-endian at the element width,
  CONVERTING each element (ScadaBridge writes int[] to INT arrays; overflow is
  BadOutOfRange, unconvertible BadTypeMismatch, both before anything is sent).
- The driver checks the shape before any runtime exists: not a list ->
  BadTypeMismatch, length != elementCount -> BadOutOfRange, BOOL/STRING/
  structure element types -> BadNotSupported; and, before sending, a controller
  buffer that is not width x count -> BadTypeMismatch. Refusals warn once and
  write no health.
- Native: ONE Write Tag of count elements with the controller's type code (the
  client fragments a large one); libplctag: the codec's bytes copied into the
  elem_count-sized buffer, refused TooSmall if it does not fit.
- IAbCipTagRuntime.EncodeArray (default: NotSupportedException).

Real-controller proof is a hardware userGate (see the lane report recipe).

Refs #737
Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 14:30:11 -04:00
Joseph Doherty 82a485e42e fix(abcip): a BOOL-array element reads its own bit, not "low byte != 0"
U11: a scalar BOOL authored over Bits[n] requested element n and decoded
"any of bits 0-7 of the reply set", so Bits[n] read Good with the subscript
ignored. A Logix BOOL array is stored as DWORDs (0xD3) and its element segment
addresses them, so the native runtime now requests DWORD n / 32 and decodes bit
n % 32 when the controller answers 0xD3 (a plain BOOL reply still decodes its
byte). libplctag exposes no type code and so cannot say which DWORD it read: a
4-byte reply for such a tag is refused BadNotSupported (warned once) instead of
publishing a wrong value as Good. Real-controller proof is a hardware gate.

Refs #737
Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 14:27:34 -04:00
Joseph Doherty 895cc28886 fix(abcip): write-side health neither latches nor overwrites
H19: Format/InvalidCast/Overflow on a client value never touched the wire but
latched Degraded; a good write set Healthy and so undid a Degraded the poll
loop had just recorded for other tags (the batch read path is worst-wins for
exactly this reason).

- The three conversion failures map to BadTypeMismatch/BadOutOfRange with a
  once-per-tag Warning and no health write (AbLegacy's rule).
- A good write (scalar or bit RMW) advances LastSuccessfulRead only - the
  device answered - and leaves the state alone.
- A write comms fault (thrown, or a non-refusal status word, which used to
  write no health at all) is Degraded with LastSuccessfulRead preserved and
  never downgrades Faulted.

Refs #737
Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 14:24:06 -04:00
Joseph Doherty cb16ebc75e fix(abcip): bit-in-word writes take the parent width from its read
H17: the .n read-modify-write hard-coded a DINT parent. On an INT or SINT host
(both legal in Logix, and reads of Flags.3 there already worked) the DINT
decode of a 2-byte buffer threw OutOfBounds, the generic catch evicted the BIT
tag's (non-existent) runtime and reported BadCommunicationError + Degraded.

The parent width now comes from the parent read's size (1 SINT / 2 INT /
4 DINT / 8 LINT; unknown keeps the DINT assumption). Narrow hosts use unsigned
arithmetic so a set top bit encodes without overflow; a bit past the host's
width is BadOutOfRange and a non-integer width BadTypeMismatch, both refused
before anything is written, warned once, no health write. A transport failure
in the RMW now evicts the parent runtime it actually used. Native end-to-end:
an INT host is written as ONE INT (type 0xC3, 2 bytes).

Refs #737
Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 14:21:43 -04:00
Joseph Doherty 4e9bd9d966 perf(abcip): native read window >= 64; cold waiters hold no window slot
P8: on the native transport the bounded-parallel read window is also the cap
on services per Multiple Service Packet (the device's shared scheduler packs
whatever is queued, within the connection's request/reply byte budget), so the
default 8 cost a 1000-tag controller ~125 round trips per poll instead of ~10.
The effective window (AbCipDriver.ReadWindow) is now max(64, option) when the
native tag factory serves reads; libplctag and injected factories keep the
option exactly. Option doc + AdminUI help text updated (Razor: Wave 4 live
check).

P7 addendum: a work item now takes its window slot only after its runtime
exists. A cold create queues on its device's single-flight connect gate, and
holding a slot there let one dead device's backoff-expiry attempt occupy the
whole window for a full Timeout, stalling every other device's reads.

Refs #737
Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 14:19:38 -04:00
Joseph Doherty 48b0173ae9 fix(abcip): a reader racing a re-init neither leaks a runtime nor degrades
Found by the H13 stress test (200 re-inits under a concurrent reader): a read
that resolved the device just before the table swap could cold-create a
runtime into the retired state after its teardown had disposed it (leaked
native handle / lease), and could hit a per-tag SemaphoreSlim the teardown had
disposed, which the generic catch reported as a comms fault (Degraded).

- DeviceState.Retire marks the state retired before disposing handles; each
  handle is removed-then-disposed exactly once. A creator re-checks the flag
  after its TryAdd (fenced, Dekker-paired with Retire) and disposes the orphan.
- A failure on a retired device is the teardown's doing: BadCommunicationError
  for that one call, no evict, no health signal.
- Per-tag/RMW gates are no longer disposed (async-only waits hold no handle),
  matching the ConnectGate rationale.

Refs #737
Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 14:17:16 -04:00
Joseph Doherty 22d4680e7b fix(abcip): a refused probe tag proves the device reachable (no blackout)
H10: the probe loop recorded EVERY exception - including the controller
answering NotFound for a typo'd or download-deleted Probe.ProbeTagPath - as a
Forward-Open failure. At a 5 s tick against a 30 s backoff cap ShouldAttempt
was never true again, so every cold or evicted data tag on a reachable device
read BadCommunicationError indefinitely and the host showed Stopped.

The probe now decides the backoff verdict and the host state separately:
- a device refusal (thrown or as a status word) is backoff success + host
  Running + one Warning per device naming the probe tag; an initialised probe
  runtime is kept so its lease is not re-opened every tick;
- a malformed probe tag name (AbCipTagConfigurationException) never reached
  the device: no backoff signal, no host transition, one Warning;
- a genuine transport failure is unchanged (backoff failure, host Stopped).

Refs #737
Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 14:17:03 -04:00
Joseph Doherty f087f01d2b fix(abcip): classify every controller refusal as per-tag Bad, keep the runtime
H1/L4: one classification (AbCipRefusals.IsDeviceRefusal) of the statuses
that prove the controller answered - NotFound, OutOfBounds, NotAllowed,
TooLarge, TooSmall, TypeMismatch, Unsupported - shared by the data path, the
Test-Connect probe (IsReachableStatus widened) and the native runtime.

- A refused read/write (thrown or as a status word) is per-tag Bad with a
  once-per-tag Warning, keeps the runtime (no evict, so no lease/Forward-Open
  churn for a sole lease holder), writes no Degraded and, since the device
  answered, advances LastSuccessfulRead without touching the state.
- A refused cold create records the backoff as reachable, so siblings are no
  longer throttled for up to 30 s by one Arr[999].
- The native runtime completes InitializeAsync on any refusal (it used to on
  TypeMismatch/Unsupported only) and reports it on every later read.
- A refused READ of NotAllowed reads BadNotReadable, not BadNotWritable.
- Test Connect names the refusal ("probe tag refused: OutOfBounds").

0x1C/0x20/0xFF-0x2104 still fold into BadReply inside Driver.Ab.Transport
(Lane K follow-up); they remain comms faults until mapped there.

Refs #737
Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 14:07:50 -04:00
Joseph Doherty 20cb2a52d6 fix(abcip): build-and-swap device tables, volatile health, async-only dispose
H13: _devices/_devicesByRoutingKey are published by reference swap under a
lifecycle lock and never mutated after publish, so the retained poll engine,
discovery and the actor's GetHostStatuses pull never read a table mid-Clear.
Every retire (init's own release, shutdown, dispose) takes the table it
replaces, so ShutdownAsync racing DisposeAsync tears each state down once, and
a generation check makes an InitializeAsync overtaken by a shutdown publish
nothing (it throws OperationCanceledException) instead of resurrecting probe
loops behind the shutdown. Probe join is one bounded 5 s wait independent of
the caller token.

H21: _health is volatile and read-modify-write updates go through a CAS.

F10: the sync Dispose() (sync-over-async) is removed; DisposeAsync is the
complete, idempotent teardown the runtime calls after ShutdownAsync.

Refs #737
Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 14:02:00 -04:00
Joseph Doherty 8a4cf8b823 Merge lane F: Galaxy driver teardown, probing, knobs (#735)
Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 13:50:53 -04:00
Joseph Doherty 0d367590db test(galaxy): isolate process-wide telemetry tests; bounded stream-cancel wait
GalaxyTelemetryTests (ActivityListener) and GatewayGalaxyDataWriterFailClosedTests
(MeterListener, filtered only by meter and instrument name) observe every Galaxy span or
measurement in the process. They now run in a collection with parallelization disabled
(F-FU6), because a parallel driver test, or its background probe seed, polluted their counts.
The stream-cancel assertion in GalaxyDriverShutdownTests goes back to a bounded wait: the
instantaneous IsCompleted check introduced in round 1 was racy.

Refs #735

Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 13:47:34 -04:00
Joseph Doherty 3e60a925dc fix(galaxy): background probe seed; generation-checked registrations
Review fix round 2 for lane F:
- N1: the init platform seed no longer runs inline under the init token. Galaxy's Tier-A
  Initialize budget is 5 s, and a cold gateway DiscoverHierarchy can spend all of it, so the
  OperationCanceledException rethrow failed init, opened the Initialize breaker and cost a full
  rebuild per retry. InitializeAsync now only resolves the platform source (no network) and
  queues a background seed tied to a per-runtime CTS that the teardown cancels. A result that
  lands after its runtime was torn down is discarded.
- N2: the seed worker serves the runtime that is live when each pass STARTS, so a request made
  under a new runtime while a stale pass is in flight always gets a pass of its own.
- N3: SubscribeAsync and the subscribe-once read capture the runtime generation before
  SubscribeBulk and register only if it is unchanged, atomically with the teardown's registry
  Clear (_registryGate). On a mismatch they release the handles and fail the call. Re-subscribe
  Rebind uses the same gate, and the production rebuild clears the registry unconditionally.
- PerPlatformProbeWatcher re-checks its stop after the SubscribeBulk await, atomically with
  the record and publish. A pass that straddles StopAsync releases the handles it acquired
  instead of writing the probe entry back into a cleared registry.

Refs #735

Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 13:47:34 -04:00
Joseph Doherty f327dd6cdb Merge lane A: fix hung-write test fixture (#731)
Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 13:39:58 -04:00
Joseph Doherty 9e180b72a3 test(runtime): de-flake the hung-write release test; name the real write bound
The fixture wedged EVERY write on one shared token-ignoring release gate, so
the later same-tag write was also hung, and the 300 ms pessimistic bound
abandoned it while the test was still polling before ReleaseWrites(). Its
reason was "write did not complete within" (the bound), not "queued for" (the
queue budget), and it had already entered the driver, so the tag had been
released. The release path is correct. Only the first write now hangs, and
later writes complete at once.

The abandon reason formatted the bound with TotalSeconds:F0, which printed a
300 ms bound as "0s". It now uses milliseconds, and the test asserts it.

Refs #731

Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 13:39:33 -04:00
Joseph Doherty 8a0acc4649 Merge master into remed/lane-F
Refs #735

Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 13:29:15 -04:00
Joseph Doherty 77ca10aa37 test(galaxy): de-flake writer-lane and supervisor tests; align EffectiveTimeout doc
The writer concurrency fake parks on its release signal (the 300 ms delay could release the
gate early on a starved pool), the supervisor test's cross-thread hold flag is Volatile, and
the EffectiveTimeout doc now states the 1 s floor the code and tests implement.

Refs #735

Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 13:29:11 -04:00
Joseph Doherty 3456756cd1 fix(galaxy): seed the probe set at init, refuse pumps mid-teardown, bound the close
Review fix round 1 for lane F:
- The ScanState probe watcher is seeded at InitializeAsync from a platform-only
  DiscoverHierarchy query (IGalaxyPlatformSource / GatewayGalaxyPlatformSource over the owned
  repository client) and re-seeded on every deploy-watcher rediscovery. The runtime never
  calls DiscoverAsync on a live driver, so per-platform host status was unreachable in
  production. A failed fetch logs a Warning and never fails init; re-seeds coalesce so a
  deploy during an in-flight pass still gets a pass that starts after it.
- EnsureEventPumpStarted refuses while the runtime is stopping, BEFORE reusing the live pump
  (that pump belongs to the dying subscriber), and the production rebuild disposes any stale
  pump before wiring the new subscriber.
- TeardownRuntimeAsync closes the MX session under the caller's token linked with a 5 s
  SessionCloseDeadline, so a blackholed gateway cannot hold DriverInstanceActor.PostStop for 30 s.
- A runtime generation counter, bumped by every teardown, is re-checked before each Rebind
  in ResubscribeAllAsync, so a replay-gap re-snapshot that straddles a teardown cannot
  resurrect handles into the cleared registry.

Refs #735

Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 13:29:11 -04:00
Joseph Doherty 05a0e51fc1 fix(galaxy): GalaxyMxSession.CloseAsync(ct) bounds the gateway CloseSession
The SDK's DisposeAsync closes with no token, i.e. under the 30 s DefaultCallTimeout, so a
blackholed gateway parked the driver's teardown for the whole of it. CloseAsync(ct) runs the
close under the caller's token, drops the channel on a cancelled/failed close (the gateway
reaps the session on its idle lease) and leaves the session re-openable. A close followed by
the final dispose closes the gateway session once.

Refs #735

Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 13:29:11 -04:00
Joseph Doherty d8c39933ed Merge lane A: runtime driver lifecycle (#731)
Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 10:44:22 -04:00
Joseph Doherty 7ae74ef147 fix(runtime): pessimistic write bound, incarnation-stamped write completions (fix round 2)
- RunWriteAsync wraps the invoker call in WaitAsync(writeTimeout) (Props,
  default 10 s = 2x the 5 s write cap): a token-ignoring hung write now
  answers BadTimeout, releases its slot and tag, and its orphan outcome
  is observed and discarded - later writes to that tag reach the device.
- WriteCompleted carries the launching incarnation (from the process-wide
  generation source); a late completion from a previous incarnation after
  a supervision restart is ignored instead of freeing the new
  incarnation's slot/tag.
- Test: a hung token-ignoring tag-a write is abandoned and a later tag-a
  write reaches the device while the abandoned one is still inside it.

Refs #731

Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 10:43:45 -04:00
Joseph Doherty d422eb747b Merge lane H1: HistoryRead raw cap (#736)
Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 10:41:54 -04:00
Joseph Doherty 3e62683517 chore(historian): review cleanups for the raw cap
- GatewayHistorianDataSource.ReadRawWindowAsync keeps the cap + 1 probe budget
  (budgetCap) separate from the client's cap: a strict-ceiling or crowding
  refusal now reports Limit = the server cap that applied and advises setting
  NumValuesPerNode at or below it (it said "lower NumValuesPerNode" with
  Limit = cap + 1 to a client that sent 0).
- Fuzz: the oracle-source over-cap assertion was vacuous (cap 0 never counts);
  it now checks no reverse window reached the forward-only shape. Class docs
  name the post-floor cap set (3 ... 1000) and distinguish NumValuesPerNode
  1 / 2 from the smallest server cap.
- HistoryPaging.IsFullPage doc says it judges the REQUEST cap (page + ties +
  lookahead), which is finite under any limit; AdvertiseMaxReturnDataValues
  swallows any exception (returns false for the caller to log), as its doc
  promised.

Refs #736

Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 10:41:37 -04:00
Joseph Doherty a01f5e6705 build: pin patched System.Security.Cryptography.* / SignalR to clear fresh-restore NuGet audit
Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 10:38:07 -04:00
Joseph Doherty 953bd27533 fix(runtime,adminui): per-tag write ordering, pessimistic init bound, unique generations, PostStop budget (fix round 1)
- Writes: a second write to a tag already inside the driver waits in the
  queue behind it (_tagsInFlight) while other tags still run under the
  cap of 4; queued writes are stamped and failed BadTimeout at dequeue
  once older than the 8 s host Ask budget (Props queuedWriteBudget)
  instead of being written late.
- Initialise attempts are bounded pessimistically (initializeTimeout,
  default 30 s = 2x Tier C) on top of the invoker's token-only timeout:
  an overrun is logged as abandoned, reported InitializeFailed so the
  retry loop continues, and its late result is dropped by generation.
  Generations come from a process-wide Interlocked counter so a prior
  incarnation's late result cannot collide (F6).
- PostStop shares one shutdownTimeout budget across shutdown + dispose.
- TelemetryDialSupervisor cancels the establishment deadline on
  StreamStopped; FleetDiagnosticsClient builds the ActorSelection inside
  the try (a malformed node degrades to "unavailable"); the diagnostics
  page labels the column "Last health publish".
- Tests: same-tag serialisation with a distinct-tag positive control;
  stale queued write BadTimeout; token-ignoring hung init -> later
  attempt connects; stopwatch assertion demoted to a ceiling.

Refs #731

Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 10:36:30 -04:00
Joseph Doherty 7e8941fbdc test(historian): paged-vs-unpaged raw-cap fuzz, MaxReturnDataValues advertisement, cap floor of 3
- NodeManagerHistoryReadRawCapFuzzTests: 10k seeds (env OTOPCUA_PAGING_FUZZ_ITERATIONS)
  through a real SDK node manager against wire-faithful fakes of SqlServer / QuestDB /
  Gateway that enforce the backend half of the cap; random series with ties, random
  windows (empty, on/past a tie, between samples, forward/reverse/end-only, bounds),
  N in {0,1,2,small,huge}, cap in {off,3..1000}, tie budget {3,8,65536}. Pages must
  concatenate to the cap-off one-shot answer (checked against the series), terminate
  within values + 2 (+ one empty step per oversized boundary cluster), never exceed
  the cap, never ask the backend for 0 / over-cap; refusals only past the budget.
- HistoryPaging.AdvertiseMaxReturnDataValues writes HistoryServerCapabilities/
  MaxReturnDataValues (i=11273), read back through the SDK master node manager.
- MinMaxRawValuesPerRead raised to 3: at 2 a resumed page has no room (value + its
  boundary tie + lookahead) and every value cost an empty page + CP (seed 15).
- RawPageLimit rows pin the default-cap arithmetic (9 999 first page, 9 998 resumed).

Fuzz findings recorded as follow-ups: a bounded read whose data fits one page
returns cap + 1 (end bound fitted against N, not the cap; node manager, F2);
the Gateway walk-back's cost on a sparse series under a tiny tie budget is
pre-existing (seed 320: 82 reads uncapped, 93 capped).

Refs #736

Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 10:15:21 -04:00
Joseph Doherty 59b72fe385 fix(runtime): diagnostics Name stays the string instance id; order-tolerant write-cap test (A6 follow-through)
DriverInstanceDiagnostics.Name carries the string driver-instance id
(what /hosts, telemetry and the artifact key on) and DriverInstanceId
the DB row Guid; the wire record has no slot for the authored display
name. The write-cap test asserts entry order per slot pair instead of
completion order of two concurrent writes.

Refs #731

Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 10:15:01 -04:00
Joseph Doherty d1cc9e2cfc feat(runtime,adminui): real GetDiagnostics from child health + /fleet/diagnostics consumer (A6)
DriverHostActor wraps the health publisher it hands its children in a
ChildHealthLedger (last snapshot per driver id, forwarded to the real
publisher), so GetDiagnostics now answers with each child's row id,
authored name, last-published state, device counts (host statuses when
the driver probes hosts, else one device) and last-change time — instead
of Guid.Empty / "Spawned" / 0 / now. Entries are forgotten on StopChild
and operator Restart.

IFleetDiagnosticsClient gains its first consumer: the new
/fleet/diagnostics page (node selector + ?node= query, revision,
running-from-cache notice, per-driver table). FleetDiagnosticsClient
logs the degrade-to-empty path at Warning.

Review ref: 02/U1.

Refs #731

Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 06:44:38 -04:00
Joseph Doherty 6dedf5158f feat(galaxy): wire DiscoverPageSize and ProbeTimeoutSeconds; concurrent first-touch write lanes
Review 2026-10-21 G-5, G-6, G-8 ("wire them all").

- G-5: GatewayGalaxyHierarchySource now pages the DiscoverHierarchy RPC
  itself (DiscoverHierarchyRawAsync) at Repository.DiscoverPageSize. The
  mxaccessgw client's typed DiscoverHierarchyAsync overloads hard-code a
  5000-row page and DiscoverHierarchyOptions carries no page size, so the
  authored knob reached nothing; the raw request takes page_size on the
  wire, so it is honoured here (non-positive → 5000). The loop keeps the
  client's repeated-page-token guard.
- G-6: GalaxyDriverProbe reads ProbeTimeoutSeconds from the config and
  runs under min(caller budget, configured cap) — the knob can shorten a
  probe, never extend it past the host's budget. The TCP preflight now
  shares that deadline instead of hanging on the caller's token alone.
- G-8: GatewayGalaxyDataWriter runs a batch as one lane per distinct tag,
  concurrently and bounded at 8 (MaxConcurrentWriteLanes); same-tag
  writes keep their authored order inside one lane and results land at
  their request's index. Acks stay serial by design.
- GalaxyDriverForm: help text now says what the two knobs actually do.

Refs #735

Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 06:41:47 -04:00
Joseph Doherty c6d4b14c93 feat(telemetry): wire TelemetryDial:CallTimeoutSeconds as the dial-establishment deadline (A5)
CallTimeoutSeconds was bound, defaulted and documented but consumed
nowhere. TelemetryDialSupervisor now arms a per-generation deadline on
every dial: a dial that has delivered no event by the deadline is
cancelled and treated as a transport failure (reconnect with a failure
streak). A live node replays its snapshots on subscribe, so this bites
only a dial that never connects (a black-holed host, where the socket
connect has no timeout). An established stream is deliberately not
bounded — the wire has no heartbeat, keepalive pings own its liveness.

Review ref: 02/U2.

Refs #731

Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 06:38:32 -04:00
Joseph Doherty 79d6b0a4b2 fix(runtime): lifecycle housekeeping — reprovision stamp, Stale drops, off-thread DB probe, tombstone sweep (A4)
- DriverHostActor.StopChild removes the driver's _lastReprovisionUtc stamp
  (and its deferred-reprovision timer): the map grew per removed id and a
  re-added driver inherited a backoff it never earned (02/L2).
- Stale now registers explicit drops for AttributeValueBatchPublished,
  AttributeAlarmPublished, ArtifactForApply and ApplyWedged instead of
  letting them hit Unhandled (02/H5).
- DbHealthProbeActor runs its SELECT on the thread pool and pipes the
  outcome back (single-flight), with a 2 s command timeout on relational
  providers, so a hung SQL Server no longer pins the actor thread and the
  cached GetStatus answer stays reachable (02/P7).
- TelemetryLocalHub eviction tombstones are timestamped and swept past a
  10 min TTL on the next Evict; a TimeProvider ctor is the test seam (02/L4).
- Tests: ValuePublishes.Single helper replaces the per-value
  AttributeValuePublished in CalculationDependencyFlowTests (02/U4, test
  half; the production receive is owned by Lane E).
- R05/8 audit: PollGroupEngine.Unsubscribe has no production caller; every
  driver already uses UnsubscribeAsync (no change).

Refs #731

Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 06:36:12 -04:00
Joseph Doherty 1b3169273d fix(galaxy): wire ScanState probing, single-sweep discovery, liveness stamp, dispatch-loop fault
Review 2026-10-21 G-4, G-7, G-11, G-10.

- G-4: PerPlatformProbeWatcher publishes its probe handles into the
  driver's SubscriptionRegistry under the reserved ProbeSubscriptionId
  (-1), so the EventPump's fan-out finally reaches the watcher — every
  ScanState event used to be dropped as "stale after unsubscribe" and
  per-platform host status never updated. ResubscribeAllAsync delegates
  the reserved entry to the watcher's new ResubscribeAsync, so the probe
  survives a reconnect; StopAsync removes the entry. Probe traffic is
  routed by the reserved handle and never reaches the public OnDataChange.
  The watcher is also built on the seam path (subscriber only), which is
  what makes the wiring testable without a gateway.
- G-7: GalaxyDiscoverer.DiscoverAsync returns the hierarchy it fetched and
  the driver selects its $WinPlatform/$AppEngine probe targets from that
  same sweep instead of asking the gateway for the whole galaxy twice.
- G-11: OnPumpDataChange stamps LastSuccessfulRead from inbound stream
  data, throttled (1 s default), so the comms-loss watchdog has honest
  liveness evidence under a Degraded supervisor.
- G-10: an EventPump dispatch-loop exit that is not a cancellation now
  reports a stream fault so the supervisor recreates the pump, instead of
  leaving a deaf pump reporting Healthy.

Refs #735

Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 06:35:48 -04:00
Joseph Doherty 6e2febbf3d fix(runtime): run driver writes off the mailbox under an in-flight cap (A3)
WriteAttribute was a ReceiveAsync, suspending the child's mailbox for the
whole device round-trip (up to 5 s): writes serialised, the second of two
operator writes to a slow device missed the host's 8 s Ask and was
reverted although the device took it, and every data change queued behind
the write. Writes are now a synchronous Receive that launches the device
call on the thread pool (sender captured, reply from the task, slot freed
by a self-sent WriteCompleted) under a per-driver cap (default 4; excess
writes queue in order). Queued writes fail fast on every exit from
Connected. Subscribe/unsubscribe stay ReceiveAsync.

Review ref: 02/P2.

Refs #731

Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 06:30:57 -04:00
Joseph Doherty 2d25c916b6 fix(galaxy): make ShutdownAsync a real runtime teardown so re-init rebuilds
GalaxyDriver.ShutdownAsync was a no-op: the MX session, both owned gRPC
channels, the EventPump, the alarm feed, the DeployWatcher, the probe
watcher and the reconnect supervisor were released only by DisposeAsync,
which the host never calls — one leaked gateway session per respawn,
removal and /raw browse, and a watchdog Shutdown→Initialize that neither
tore down nor rebuilt anything (review 2026-10-21 G-1, G-2, G-9, G-3).

- ShutdownAsync and DisposeAsync share one ordered TeardownRuntimeAsync
  (supervisor first, then deploy/probe watchers, pump, alarm feed,
  session, channels); owned seams are nulled so the next InitializeAsync
  rebuilds; injected seams survive a shutdown and drop only on the final
  dispose. Lifecycle calls are serialised behind a gate.
- A failed production connect releases its partial runtime (no orphaned
  client per watchdog retry) and lands Faulted before and after.
- StartDeployWatcherAsync stops the previous watcher before starting a
  replacement, and now starts on a production instance's FIRST
  InitializeAsync (the old guard returned whenever discovery had not run
  here, which on a runtime instance is always).
- ReconnectSupervisor releases each cycle's CancellationTokenSource when
  its loop exits and gains an awaitable StopAsync; PerPlatformProbeWatcher
  gains an awaitable StopAsync.
- EventPump treats any exception under cancellation as a clean stop, so a
  teardown never wakes the supervisor.
- Internal seams for tests: an IGalaxyDeployWatchSource ctor parameter and
  a session-open override forwarded to the owned GalaxyMxSession.

Refs #735

Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 06:30:14 -04:00
Joseph Doherty 6669fae7e5 feat(historian): ServerHistorian:MaxRawValuesPerRead caps raw pages and backend reads
Review R08:1.1: an uncapped HistoryRead-Raw (NumValuesPerNode = 0) read a
tag's whole window into memory on every backend (SQL TOP(int.MaxValue),
QuestDB LIMIT, Gateway max_values = int.MaxValue + a second snapshot copy).

New ServerHistorian:MaxRawValuesPerRead (default 10 000; <= 0 unbounded with
a Validate() warning; 1 raised to 2). Each backend data source takes it,
reports it as IHistorianDataSource.MaxRawValuesPerRead (the Gateway: min
with a strict gateway's learned ceiling), so the node manager's existing
RawPageLimit paging serves N = 0 and N > cap as pages of at most the cap
with a continuation point. Independently, a backend raw read asking for 0
or more than the cap reads cap + 1 rows and is REFUSED
(HistoryReadLimitExceededException => BadTooManyOperations) when the window
holds more - never answered truncated, since the server's paging reads a
short answer as the last page. Bound through the Host registry, the
SqlServer/QuestDb providers and GatewayHistorian.CreateDataSource.

Tests: options (default/binding/warnings), provider binding, Gateway unit +
through a real SDK node manager (uncapped -> 999/998/998/605 + CPs), and a
conformance case for the SQL backends.

Refs #736

Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 06:29:10 -04:00
Joseph Doherty bfd5b7b015 fix(runtime): single-flight reconnect attempts, bounded pre-reconnect teardown (A2)
The Reconnecting retry timer started a new ShutdownAsync + InitializeAsync
pair on every tick whether or not the previous attempt had finished; the
generation token only discarded the older RESULT, so with an Initialize
bound longer than the cadence the teardown ran underneath an attempt still
inside the driver (the dispose-before-overwrite race). An _initInFlight
gate, cleared by the piped InitializeSucceeded/Failed, skips a tick while
an attempt is running; the pre-reconnect teardown is bounded by the
shutdown timeout; PostStop cancels an in-flight attempt before shutting
the driver down.

Review ref: 02/H2.

Refs #731

Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 06:26:45 -04:00
Joseph Doherty eb23053700 fix(runtime): dispose drivers on final stop, bound PostStop shutdown (A1)
DriverInstanceActor.PostStop now bounds IDriver.ShutdownAsync by a
shutdownTimeout (default 5 s, logged on overrun) and then disposes the
driver (IAsyncDisposable preferred, else IDisposable) — but only on a
real stop. PreRestart flags the restart so the PostStop it triggers
skips the dispose, because a supervision restart re-uses the same
driver object (Props closure) and the host re-initialises it.

DiscoveryDriverBrowser disposes its throwaway capture/probe driver
after the bounded shutdown, including after a failed or timed-out one.

Review refs: 02/L1, 02/P4, 06/G-1 (host context), 07/L2.

Refs #731

Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 06:23:39 -04:00
Joseph Doherty 7513ce5144 docs(review): record pre-remediation baseline test run
Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 06:11:55 -04:00
Joseph Doherty ad2b5c4044 docs: add 2026-10-21 system review + remediation plan
Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 06:04:58 -04:00
Joseph Doherty d51d35bd64 Merge branch 'docs/xml-doc-cleanup'
v2-ci / unit-tests (push) Failing after 3m14s
2026-10-15 06:25:09 -04:00
Joseph Doherty 8c6067b62e docs: move subsystem gotchas out of CLAUDE.md into path-scoped .claude/rules
Keeps the root CLAUDE.md lean by lazy-loading per-area detail (drivers, topology,
data-plane auth, alarms, historian) only when matching files are touched.
2026-10-15 06:14:58 -04:00
Joseph Doherty 826312befe docs: fill missing XML docs and strip tracking IDs from src comments
Clears CommentChecker findings in src/ (3,280 -> 209, the remainder being false
positives like PLC-5/IEEE-754/M0.0) by adding missing summaries/params/returns,
normalising inheritdoc usage, and removing Gitea/review tracking IDs while keeping
the rationale they annotated. Comment-only: comment-stripped sources are identical
to the parent commit.
2026-10-15 06:14:58 -04:00
dohertj2 ebcbd81fca Merge pull request 'Ensure driver test cleanup preserves primary failures' (#730) from test/task-assertion-return-discard into master
v2-ci / unit-tests (push) Successful in 1h1m21s
2026-10-13 21:06:31 -04:00
Joseph Doherty 4f537a5652 Merge remote-tracking branch 'origin/master' into test/task-assertion-return-discard
v2-ci / unit-tests (pull_request) Successful in 1h1m9s
2026-10-13 19:03:52 -04:00
dohertj2 c89912a93b Merge pull request 'Fix DB-less ServiceLevel freshness and authorization remarks' (#724) from fix/dbless-servicelevel-freshness into master
v2-ci / unit-tests (push) Waiting to run
2026-10-13 19:03:42 -04:00
Joseph Doherty c4b4046832 test: verify missing-snapshot ServiceLevel timer behavior
v2-ci / unit-tests (pull_request) Successful in 1h0m42s
2026-10-13 17:19:50 -04:00
Joseph Doherty eed7cc7d75 test: guarantee driver fixture cleanup without masking failures 2026-10-13 16:36:38 -04:00
Joseph Doherty 5310a7e07a Discard asserted probe-task return values without joining loops 2026-10-13 16:12:31 -04:00
Joseph Doherty e1feb9fbb6 Merge remote-tracking branch 'origin/master' into fix/dbless-servicelevel-freshness
v2-ci / unit-tests (pull_request) Waiting to run
2026-10-13 15:58:35 -04:00
dohertj2 1bf8c614a9 Merge pull request 'Retain ServiceLevel requests across binding and publisher failures' (#720) from fix/servicelevel-binding-replay into master
v2-ci / unit-tests (push) Waiting to run
2026-10-13 15:58:19 -04:00