fix(galaxy): #859/#861 third review — read rebind release, probe session fence, reopen fence, pump restart
B-1: the subscribe-once read's cleanup unsubscribes what the registry holds at removal (the replay may have rebound it to new-session handles), not its original bindings; nothing when the entry still holds dead old-session handles. Found while testing it: reads were attached to the pump INSTANCE, so a read across a reconnect's pump restart never saw the replayed value — reads now listen on a driver-level ReadDataChange raised for whichever pump is live. I-A: PerPlatformProbeWatcher records the registry session with each probe handle. Publishing goes through the new fenced TryUpsert and binds only current-session handles; removal/stop unsubscribe and purge only those; a seed reply that straddled a reconnect records nothing and is re-issued (max 3). I-B: PurgeParked takes the session captured BEFORE the unsubscribe RPC and is a no-op under the park gate if the session changed. I-C: ReopenAsync starts a new registry session before AND after the gateway session swap; ReplayAsync does so only when no reopen ran (injected subscriber / tests) and replays only subscriptions still holding older-session handles, so a subscribe issued after the swap is live and not double-advised. P-1: ReplayAsync restarts the pump before any early return (no entries used to leave the dead pump in place). P-2: ReplayOnSessionLost=false logs a Warning; misleading comment about gateway-side replay fixed. M-a: TryRebind returns BindOutcome; UnknownId releases the handles just advised for a subscription removed mid-replay. M-b: the replay/re-snapshot pass reads the session once and stops if it changes. M-c: parked entries drop their MxEvent when they leave the parked set. M-d (retry a straddled read) not done. Tests (TestWait) for each, each checked red against a mutation of its fix. Refs #859, #861. Claude-Session: https://claude.ai/code/session_01GT67AD99zQQqpLCVm3p7xk
This commit is contained in:
@@ -172,7 +172,7 @@ Every (re)open of `StreamAlarms` replays the gateway's active-alarm snapshot. `G
|
||||
- **Replay gaps** (`galaxy.events.replay_gaps`): the gateway signals that events were evicted from its replay ring before we could read them. The driver responds by re-issuing `SubscribeBulk` for every tracked subscription — MXAccess delivers an initial value per item on subscribe, so re-subscribing *is* the re-snapshot the contract asks for. It does **not** restart the event pump (the stream carrying the sentinel is healthy) and it honours `Reconnect:ReplayOnSessionLost=false` by logging and standing down, on the grounds that an operator who delegated recovery to the gateway did not ask for a re-subscribe on a different trigger.
|
||||
⚠️ This cannot currently fire: the sentinel is only emitted on a stream opened with `after_worker_sequence > 0`, and the driver opens with `0`, which the gateway treats as "fresh stream, not a resume". Enabling resume is a bigger change than it looks — the sequence is only meaningful **within one gateway session**, and the reconnect path recreates the session, so a remembered sequence replayed against a fresh session would set the server's live watermark *above* every event it is about to send, silencing the stream rather than recovering it.
|
||||
- **Undecodable values** (`galaxy.events.undecodable`): a value kind this driver has no CLR representation for is published as **`BadTypeMismatch` with a null value**, and the first occurrence of each distinct kind is logged. It is *not* published as a null carrying the event's own quality: an MXAccess event can be Good-quality while the payload is one we cannot map, and a good null is indistinguishable from a genuine null reading — a client would consume a value it never received as a real one, with the failure visible only in a log. Non-zero here means the gateway is sending a wire type `MxValueDecoder` has no mapping for and those readings are being lost; the fix is a decoder mapping, not a quality change.
|
||||
- **Initial values can overtake the subscribe reply (#859)** (`galaxy.events.early_parked` / `early_replayed` / `early_overflow`): mxaccessgw gives no reply-before-event ordering — the worker frames the SubscribeBulk reply on a thread-pool continuation while the STA is already pumping the OnDataChange that Advise queued, and the reply (unary Invoke) and the event (StreamEvents) are separate HTTP/2 streams anyway. An event for an unregistered handle is therefore only "stale after unsubscribe" when no subscribe is in flight. Every SubscribeBulk call site (`SubscribeAsync`, the subscribe-once `ReadAsync` — which waits only for the initial value and would otherwise read `BadTimeout` — the reconnect/replay-gap `ResubscribeAllAsync` (one window per subscription, not across the whole replay), `PerPlatformProbeWatcher`) opens `SubscriptionRegistry.BeginPendingSubscribe()` before the RPC and closes it right after registering, never later; inside the window unknown-handle events are parked, promoted by the `Register`/`Rebind` that binds the handle — map writes and promotion are one critical section, so no later event can resolve in between — and replayed on the pump's dispatch thread before any later event for that handle. Held events (parked + promoted-but-unreplayed) are bounded at 100 000 together. Unclaimed parked events are trimmed by **window sequence**, not only when every window has closed (concurrent reads keep some window open almost permanently): each window gets a monotonically increasing sequence when it opens, each parked event is stamped with the newest sequence, and once every window that opened at or before that stamp has closed the event is discarded — outside any window the stale drop is unchanged. **Sessions (#861):** item handles are per gateway session and the next session reuses the numbers. On reconnect `ReplayAsync` first calls `SubscriptionRegistry.StartNewSession()`, which discards everything held, **invalidates the reverse map** (subscriptions are kept; new-session events park until the replay's per-subscription `Rebind` promotes them, instead of resolving to the tag that owned the number before), and bumps a session number. Registration and rebind are refused for a stale session, so a `SubscribeAsync` whose reply straddled the reconnect is **re-issued on the new session** (up to 3 attempts), a subscribe-once read returns `BadCommunicationError`, and dead old-session handle numbers are never sent to `UnsubscribeBulk`; an old session's pump can no longer resolve through the map. **Interim (I-3):** a completed `UnsubscribeBulk` purges anything parked for its handles, so a trailing event of a removed item is not promoted onto a reused handle number within one session. It cannot catch an event still in flight after the unsubscribe, and it also drops a genuine early value for a number reused that fast (a timeout rather than another tag's value). The complete fix needs a worker sequence on the SubscribeBulk reply so the driver can tell pre-subscribe events from post-subscribe ones — an optional cross-repo change in mxaccessgw.
|
||||
- **Initial values can overtake the subscribe reply (#859)** (`galaxy.events.early_parked` / `early_replayed` / `early_overflow`): mxaccessgw gives no reply-before-event ordering — the worker frames the SubscribeBulk reply on a thread-pool continuation while the STA is already pumping the OnDataChange that Advise queued, and the reply (unary Invoke) and the event (StreamEvents) are separate HTTP/2 streams anyway. An event for an unregistered handle is therefore only "stale after unsubscribe" when no subscribe is in flight. Every SubscribeBulk call site (`SubscribeAsync`, the subscribe-once `ReadAsync` — which waits only for the initial value and would otherwise read `BadTimeout` — the reconnect/replay-gap `ResubscribeAllAsync` (one window per subscription, not across the whole replay), `PerPlatformProbeWatcher`) opens `SubscriptionRegistry.BeginPendingSubscribe()` before the RPC and closes it right after registering, never later; inside the window unknown-handle events are parked, promoted by the `Register`/`Rebind` that binds the handle — map writes and promotion are one critical section, so no later event can resolve in between — and replayed on the pump's dispatch thread before any later event for that handle. Held events (parked + promoted-but-unreplayed) are bounded at 100 000 together. Unclaimed parked events are trimmed by **window sequence**, not only when every window has closed (concurrent reads keep some window open almost permanently): each window gets a monotonically increasing sequence when it opens, each parked event is stamped with the newest sequence, and once every window that opened at or before that stamp has closed the event is discarded — outside any window the stale drop is unchanged. **Sessions (#861):** item handles are per gateway session and the next session reuses the numbers. On reconnect `ReopenAsync` calls `SubscriptionRegistry.StartNewSession()` both before and after the gateway session swap (`ReplayAsync` does it only when no reopen ran), and the replay re-subscribes only subscriptions still holding the previous session's handles (one registered after the swap is already live). The pump is always restarted, even with nothing to replay. `StartNewSession()` discards everything held, **invalidates the reverse map** (subscriptions are kept; new-session events park until the replay's per-subscription `Rebind` promotes them, instead of resolving to the tag that owned the number before), and bumps a session number. Registration and rebind are refused for a stale session, so a `SubscribeAsync` whose reply straddled the reconnect is **re-issued on the new session** (up to 3 attempts), a subscribe-once read returns `BadCommunicationError`, and dead old-session handle numbers are never sent to `UnsubscribeBulk` (a read rebound by the replay releases the handles the registry holds, not its original ones); an old session's pump can no longer resolve through the map. Reads listen on the driver, not on a pump instance, so a read survives the reconnect's pump restart. `PerPlatformProbeWatcher` records the session with each probe handle: only current-session handles are published (fenced `TryUpsert`), unsubscribed or purged, and a seed reply that straddled a reconnect is re-issued. Purges are fenced by the session captured before the unsubscribe RPC. **Interim (I-3):** a completed `UnsubscribeBulk` purges anything parked for its handles, so a trailing event of a removed item is not promoted onto a reused handle number within one session. It cannot catch an event still in flight after the unsubscribe, and it also drops a genuine early value for a number reused that fast (a timeout rather than another tag's value). The complete fix needs a worker sequence on the SubscribeBulk reply so the driver can tell pre-subscribe events from post-subscribe ones — an optional cross-repo change in mxaccessgw.
|
||||
- **Duplicate `attributeRef` (#605 / 06-S-13)**: two authored RawPaths dialling the same Galaxy attribute are both dialled, but the reverse map is first-wins, so value fan-out and alarm `ConditionId` surface under the first RawPath only. The loser is logged at Warning and recorded in the skipped-tag ledger with the winner path in the reason.
|
||||
- **Subscribe-once reads are bounded (#605 / 06-S-3)**: `ReadAsync` on the production path (SubscribeBulk, first event, Unsubscribe) carries an internal 30 s deadline in addition to the caller token; slots that never publish come back `BadTimeout`, so `CancellationToken.None` can no longer hang on a silent tag.
|
||||
- **Supervisory advise is retried after a thrown failure (#605 / 06-S-12)**: the writer marks an item handle advised before the call; a thrown advise (not just a non-OK status) now unmarks it, fails the write closed and re-advises on the next write.
|
||||
|
||||
Reference in New Issue
Block a user