The gateway's StreamEvents with after_worker_sequence = 0 is a fresh stream
and never replays, and the event pump started only on the first
Read/Subscribe, so platform ScanState probes advised before then lost their
initial value (and a driver with no authored tags never got any host status).
PerPlatformProbeWatcher now awaits an ensure-event-stream hook before every
probe SubscribeBulk (seed, discover sync, reconnect ResubscribeStaleAsync).
The driver's hook starts the pump (refused while a teardown is in progress)
and waits, bounded, on the new EventPump.StreamRequested signal - completed
once the stream loop's first MoveNext has issued the StreamEvents RPC, or
when the loop ends - so the advise never precedes the stream request.
Claude-Session: https://claude.ai/code/session_01GT67AD99zQQqpLCVm3p7xk
BLOCKER: the reconnect replay chose work from the registry entries' sessions,
but ONE registry entry holds every probe handle and any probe publish on the
new session (a seed in the reopen→replay gap, or the straddle retry) re-stamps
it as current — the replay then skipped the probe watcher and the old
platforms stayed bound to handle 0 (host status frozen); a publish after the
snapshot made it re-subscribe everything (double advise, leaked handle).
ReplayAsync now excludes the probe entry from its snapshot and calls the new
PerPlatformProbeWatcher.ResubscribeStaleAsync on every reconnect, which
re-subscribes exactly the platforms whose ProbeHandle.Session predates the
current session. The replay-gap path keeps ResubscribeAsync (all platforms).
Also: ClearRegistry (teardown) starts a new registry session, so a late
PurgeParked from a teardown-orphaned call cannot reach the next runtime's
parked values; the read path's unused pump local is gone (call kept).
Tests go through the driver's reopen/replay: the probe old-session-number test
is rewritten that way, plus the seed-in-the-gap scenario (only P and Q are
re-subscribed, R keeps its new-session handle, all three resolve). Both checked
red with the probe replay removed / re-subscribing all platforms.
Refs #859, #861.
Claude-Session: https://claude.ai/code/session_01GT67AD99zQQqpLCVm3p7xk
B-1: the subscribe-once read's cleanup unsubscribes what the registry holds at
removal (the replay may have rebound it to new-session handles), not its
original bindings; nothing when the entry still holds dead old-session
handles. Found while testing it: reads were attached to the pump INSTANCE, so
a read across a reconnect's pump restart never saw the replayed value — reads
now listen on a driver-level ReadDataChange raised for whichever pump is live.
I-A: PerPlatformProbeWatcher records the registry session with each probe
handle. Publishing goes through the new fenced TryUpsert and binds only
current-session handles; removal/stop unsubscribe and purge only those; a
seed reply that straddled a reconnect records nothing and is re-issued (max 3).
I-B: PurgeParked takes the session captured BEFORE the unsubscribe RPC and is
a no-op under the park gate if the session changed.
I-C: ReopenAsync starts a new registry session before AND after the gateway
session swap; ReplayAsync does so only when no reopen ran (injected subscriber
/ tests) and replays only subscriptions still holding older-session handles,
so a subscribe issued after the swap is live and not double-advised.
P-1: ReplayAsync restarts the pump before any early return (no entries used to
leave the dead pump in place). P-2: ReplayOnSessionLost=false logs a Warning;
misleading comment about gateway-side replay fixed.
M-a: TryRebind returns BindOutcome; UnknownId releases the handles just
advised for a subscription removed mid-replay. M-b: the replay/re-snapshot
pass reads the session once and stops if it changes. M-c: parked entries drop
their MxEvent when they leave the parked set. M-d (retry a straddled read)
not done.
Tests (TestWait) for each, each checked red against a mutation of its fix.
Refs #859, #861.
Claude-Session: https://claude.ai/code/session_01GT67AD99zQQqpLCVm3p7xk
I-2: unclaimed parked events are trimmed by window sequence. Each window gets a
monotonically increasing sequence at Begin, each parked event is stamped with
the newest one, and on every End events stamped before the oldest open
window's sequence are dropped (amortised O(1) via a park-order queue). Under
concurrent reads the open count rarely hit 0, so trailing events used to pile
up to the bound and push genuine early values into overflow.
I-1 / #861: SubscriptionRegistry.StartNewSession() — called FIRST in
ReplayAsync on every reconnect — clears held events, invalidates the reverse
map (subscriptions kept) and bumps a session number. New-session events whose
number an old binding still held now park until the replay's per-subscription
Rebind promotes them, instead of resolving to the wrong tag. TryRegister /
TryRebind refuse a stale session atomically under the park gate: a
SubscribeAsync whose reply straddled a reconnect is re-issued on the new
session (max 3 attempts), a subscribe-once read returns BadCommunicationError,
ResubscribeAllAsync stops if the session changes again. Old-session handle
numbers are never unsubscribed (Remove reports whether the entry belongs to
the current session; the read's late-reply path checks too), and an old
session's pump can no longer resolve through the map (fast path checks the
session).
I-3 (interim): a completed UnsubscribeBulk purges events parked for its
handles (driver + probe watcher). Full fix needs a worker sequence on the
SubscribeBulk reply (mxaccessgw, optional) — documented.
M-1: a resolved handle with a stranded parked list promotes it and resolves
the live event (never parks it). M-2: ResolveOrPark returns the ready events
to replay first, one gate acquisition per event while anything is held.
M-3: SubscribeAsync/read dispose the window before the orphan-release RPC.
M-5: ResolveOrPark remarks document the single-dispatching-pump invariant.
Tests (TestWait): overlapping windows don't accumulate, purge, stranded list,
session fence + map invalidation, stale-session registration refused,
end-to-end replay with swapped handle numbers and event-before-reply,
subscribe and read straddling a reconnect, read deadline closes its window.
Each was checked red against a mutation of the fix it pins.
Refs #859, #861.
Claude-Session: https://claude.ai/code/session_01GT67AD99zQQqpLCVm3p7xk
mxaccessgw gives no reply-before-event ordering: the worker completes the STA
command via a RunContinuationsAsynchronously TCS, so the reply is framed on a
thread-pool continuation while the STA already pumps the OnDataChange Advise
queued (drained/framed by a separate loop), and the reply (unary Invoke) and
the event (StreamEvents) are separate HTTP/2 streams. The driver then also
registers handles concurrently with the pump's dispatch loop. MXAccess's
initial value could therefore be dropped as "stale after unsubscribe",
leaving the node at its pre-subscribe value until the next change.
SubscriptionRegistry gains an early-event window: every SubscribeBulk call
site (SubscribeAsync, ResubscribeAllAsync, PerPlatformProbeWatcher) opens
BeginPendingSubscribe() before the RPC and closes it after registering.
Inside the window unknown-handle events are parked (bounded, 100k); the
Register/Rebind that binds a handle promotes them, and the EventPump replays
them on its single dispatch thread before any later event for that handle
(woken by a sentinel when no further event arrives). Closing the last window
discards unclaimed events, so the stale drop is unchanged outside it.
New counters galaxy.events.early_{parked,replayed,overflow}.
Claude-Session: https://claude.ai/code/session_01GT67AD99zQQqpLCVm3p7xk
PerPlatformProbeWatcher records a platform's handle and publishes the probe
binding into the driver registry as one step under _syncLock, but
WatchedPlatforms read the handle map lock-free - so it could report a
platform one step ahead of the registry Upsert. A ScanState event raised in
that window was dispatched by the EventPump, missed ResolveSubscribers and
was dropped as "stale after unsubscribe", leaving the host status empty.
That is the #856 flake: A_fetch_completing_after_init_populates_host_statuses
waited for WatchedPlatforms, emitted ScanState, and timed out at 10 s on the
final Running wait. Reproduced 11/700 (28 parallel runners x 25 + 14 CPU
burners), every failure at that step with the binding present by the time of
diagnosis and hostStatuses=[] - i.e. the event was dropped. 0/1400 after.
WatchedPlatforms now reads under _syncLock (test seam only; the dispatch path
already checks the map after the registry, the safe order). New
PerPlatformProbeWatcherVisibilityTests spins a reader on another thread and
asserts the registry entry exists the moment the platform is visible
(old code: torn in 1691/2000 iterations). WaitUntilAsync gains a describe
overload; the #856 test's waits now report elapsed time and the observed
seed/binding/pump/host-status chain on timeout.
Claude-Session: https://claude.ai/code/session_01GT67AD99zQQqpLCVm3p7xk
Review fix round 2 for lane F:
- N1: the init platform seed no longer runs inline under the init token. Galaxy's Tier-A
Initialize budget is 5 s, and a cold gateway DiscoverHierarchy can spend all of it, so the
OperationCanceledException rethrow failed init, opened the Initialize breaker and cost a full
rebuild per retry. InitializeAsync now only resolves the platform source (no network) and
queues a background seed tied to a per-runtime CTS that the teardown cancels. A result that
lands after its runtime was torn down is discarded.
- N2: the seed worker serves the runtime that is live when each pass STARTS, so a request made
under a new runtime while a stale pass is in flight always gets a pass of its own.
- N3: SubscribeAsync and the subscribe-once read capture the runtime generation before
SubscribeBulk and register only if it is unchanged, atomically with the teardown's registry
Clear (_registryGate). On a mismatch they release the handles and fail the call. Re-subscribe
Rebind uses the same gate, and the production rebuild clears the registry unconditionally.
- PerPlatformProbeWatcher re-checks its stop after the SubscribeBulk await, atomically with
the record and publish. A pass that straddles StopAsync releases the handles it acquired
instead of writing the probe entry back into a cleared registry.
Refs #735
Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
Review 2026-10-21 G-4, G-7, G-11, G-10.
- G-4: PerPlatformProbeWatcher publishes its probe handles into the
driver's SubscriptionRegistry under the reserved ProbeSubscriptionId
(-1), so the EventPump's fan-out finally reaches the watcher — every
ScanState event used to be dropped as "stale after unsubscribe" and
per-platform host status never updated. ResubscribeAllAsync delegates
the reserved entry to the watcher's new ResubscribeAsync, so the probe
survives a reconnect; StopAsync removes the entry. Probe traffic is
routed by the reserved handle and never reaches the public OnDataChange.
The watcher is also built on the seam path (subscriber only), which is
what makes the wiring testable without a gateway.
- G-7: GalaxyDiscoverer.DiscoverAsync returns the hierarchy it fetched and
the driver selects its $WinPlatform/$AppEngine probe targets from that
same sweep instead of asking the gateway for the whole galaxy twice.
- G-11: OnPumpDataChange stamps LastSuccessfulRead from inbound stream
data, throttled (1 s default), so the comms-loss watchdog has honest
liveness evidence under a Degraded supervisor.
- G-10: an EventPump dispatch-loop exit that is not a cancellation now
reports a stream fault so the supervisor recreates the pump, instead of
leaving a deaf pump reporting Healthy.
Refs #735
Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
GalaxyDriver.ShutdownAsync was a no-op: the MX session, both owned gRPC
channels, the EventPump, the alarm feed, the DeployWatcher, the probe
watcher and the reconnect supervisor were released only by DisposeAsync,
which the host never calls — one leaked gateway session per respawn,
removal and /raw browse, and a watchdog Shutdown→Initialize that neither
tore down nor rebuilt anything (review 2026-10-21 G-1, G-2, G-9, G-3).
- ShutdownAsync and DisposeAsync share one ordered TeardownRuntimeAsync
(supervisor first, then deploy/probe watchers, pump, alarm feed,
session, channels); owned seams are nulled so the next InitializeAsync
rebuilds; injected seams survive a shutdown and drop only on the final
dispose. Lifecycle calls are serialised behind a gate.
- A failed production connect releases its partial runtime (no orphaned
client per watchdog retry) and lands Faulted before and after.
- StartDeployWatcherAsync stops the previous watcher before starting a
replacement, and now starts on a production instance's FIRST
InitializeAsync (the old guard returned whenever discovery had not run
here, which on a runtime instance is always).
- ReconnectSupervisor releases each cycle's CancellationTokenSource when
its loop exits and gains an awaitable StopAsync; PerPlatformProbeWatcher
gains an awaitable StopAsync.
- EventPump treats any exception under cancellation as a clean stop, so a
teardown never wakes the supervisor.
- Internal seams for tests: an IGalaxyDeployWatchSource ctor parameter and
a session-open override forwarded to the owned GalaxyMxSession.
Refs #735
Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
Adds <summary>/<param>/<returns>/<inheritdoc> where missing and removes
project bookkeeping IDs (task/tracking refs) from shipped code comments,
so the docs read cleanly and CommentChecker is quiet except for known
false positives (PLC/protocol terms, event/IEqualityComparer inheritdoc).
Doc/comment-only; no logic changed; solution builds clean.
Adds <summary>, <param>, <typeparam>, and <inheritdoc/> tags to public
members surfaced by commentchecker — resolves 5,847 of 5,869 issues
(99.6%) across three /fixdocs passes.
Group all 69 projects into category subfolders under src/ and tests/ so the
Rider Solution Explorer mirrors the module structure. Folders: Core, Server,
Drivers (with a nested Driver CLIs subfolder), Client, Tooling.
- Move every project folder on disk with git mv (history preserved as renames).
- Recompute relative paths in 57 .csproj files: cross-category ProjectReferences,
the lib/ HintPath+None refs in Driver.Historian.Wonderware, and the external
mxaccessgw refs in Driver.Galaxy and its test project.
- Rebuild ZB.MOM.WW.OtOpcUa.slnx with nested solution folders.
- Re-prefix project paths in functional scripts (e2e, compliance, smoke SQL,
integration, install).
Build green (0 errors); unit tests pass. Docs left for a separate pass.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>