feat(events): opt-in stream-attached acknowledgement on StreamEvents (#227)
A fresh StreamEvents is never replayed, so an initial value published before
the gateway registers the stream's subscriber is lost (lmxopcua#865 —
slow-changing ScanState). Clients had nothing deterministic to wait on.
Contract (additive, proto3):
- StreamEventsRequest.emit_stream_attached = 3 (bool, default false)
- MxEvent.stream_attached = 15 (optional StreamAttached)
- message StreamAttached { uint64 live_after_sequence = 1; }
- OpenSessionReply capability "stream-events-attached-ack"
(GatewayCapabilities.StreamEventsAttachedAck)
When requested, EventStreamService yields the sentinel as the FIRST message,
after AttachEventSubscriber/AttachEventSubscriberWithReplay has returned, so
receipt implies registration; refused attaches never see it. It precedes the
ReplayGap sentinel and replay on a resume. Shape mirrors ReplayGap: family
UNSPECIFIED, no body, worker_sequence 0.
Opt-in rather than always-on: ScadaBridge's MxGateway adapter stores every
streamed event's worker_sequence as its resume cursor, so an unrequested
sentinel would reset it to 0. Default wire behaviour is unchanged.
.NET client: StreamEventsAsync/StreamEventItemsAsync(after, emitStreamAttached),
MxEventStreamItem.IsStreamAttached, MxGatewaySession.SupportsStreamAttachedAck,
and OpenEventStreamAsync -> MxGatewayEventStream (awaits the sentinel, with
attach timeout; refuses older gateways).
Regenerated Contracts/Generated, the client protoset, Go and Python bindings
(also picks up comment-only drift already on main), synced the Rust vendored
proto and set emit_stream_attached in the Rust StreamEventsRequest literals.
No package versions bumped.
Tests: ordering (sentinel first, then events), race proof (event fanned right
after the sentinel is delivered — mutation-checked: fails when the sentinel is
yielded before registration), resume-with-gap order, refused attach emits no
sentinel, default emits none, contract field numbers, capability, and .NET
client wire tests (await, no-capability, older gateway, timeout).
This commit is contained in:
+1
-1
@@ -335,7 +335,7 @@ The pipeline has four stages:
|
||||
1. The session is resolved via `sessionManager.TryGetSession`. A miss raises `SessionManagerException(SessionNotFound)` so the handler reports `StatusCode.NotFound`.
|
||||
2. The attach is owner-scoped: the caller's API key id must match the key that opened the session, or the call fails with `PermissionDenied` before anything is attached or delivered.
|
||||
3. The call registers a subscriber on the distributor. A fresh stream (`after_worker_sequence == 0`) uses `session.AttachEventSubscriber(...)`. A resume uses `session.AttachEventSubscriberWithReplay(...)`, which snapshots the replay ring for events newer than the cursor and registers the live subscriber under one lock, so the replay-to-live handoff has no gap and no duplicate. In single-subscriber mode (the default) a second attach fails with `EventSubscriberAlreadyActive`; with `Sessions:AllowMultipleEventSubscribers` enabled, an attach beyond `MaxEventSubscribersPerSession` fails with `EventSubscriberLimitReached`. Both map to `ResourceExhausted`. The lease is released in the `finally` block, so the subscriber slot is freed even when the client cancels mid-stream.
|
||||
4. The call yields events from the lease's bounded channel (capacity `Events:QueueCapacity`). On a resume it first yields a `ReplayGap` sentinel when the cursor predates the oldest retained event, then the replayed batch, then live events filtered to sequences above the last replayed one. Per-session ordering is preserved end-to-end, because the pump is the single writer to each subscriber channel and writes in `WorkerSequence` order.
|
||||
4. The call yields events from the lease's bounded channel (capacity `Events:QueueCapacity`). When the request set `emit_stream_attached` it first yields the StreamAttached sentinel — after step 3, so the client's receipt of it proves the subscriber is registered (see [Sessions](./Sessions.md#attach-acknowledgement)). On a resume it then yields a `ReplayGap` sentinel when the cursor predates the oldest retained event, then the replayed batch, then live events filtered to sequences above the last replayed one. Per-session ordering is preserved end-to-end, because the pump is the single writer to each subscriber channel and writes in `WorkerSequence` order.
|
||||
|
||||
The replay ring is server-side state that outlives a single RPC: with the default `DetachGraceSeconds` and `ReplayRetentionSeconds`, a client that drops can reconnect with its last `worker_sequence` and receive what it missed, as long as the events are still retained.
|
||||
|
||||
|
||||
@@ -282,6 +282,7 @@ The handoff is sealed by a watermark. `RegisterWithReplay` returns `LiveResumeSe
|
||||
|
||||
Emit order on a resumed stream:
|
||||
|
||||
0. **StreamAttached sentinel (only when requested).** See [Attach acknowledgement](#attach-acknowledgement) below; when asked for, it precedes everything else on any stream, fresh or resumed.
|
||||
1. **ReplayGap sentinel (only when events were evicted).** If the requested `after_worker_sequence` predates the oldest event still retained — i.e. events in the open interval were dropped by capacity or age eviction and are unrecoverable — the gateway first yields a single sentinel `MxEvent` with `replay_gap` populated (`requested_after_sequence` = the requested watermark, `oldest_available_sequence` = the resume anchor described below). The sentinel carries the session id; its `family` is `UNSPECIFIED`, its `body` oneof is unset, and no per-item fields are populated. It is an explicit, documented control signal — *not* a synthesized MXAccess event — telling the client to discard local state and re-snapshot. A client that wants to resume without another gap should set `after_worker_sequence = oldest_available_sequence - 1` on its next request.
|
||||
2. **Retained replay batch.** The still-retained events newer than the requested watermark, in ascending `worker_sequence` order.
|
||||
3. **Live events**, resuming strictly after `LiveResumeSequence`.
|
||||
@@ -292,6 +293,16 @@ When `after_worker_sequence` is inside the retained window (nothing was evicted)
|
||||
|
||||
The ReplayGap sentinel is emitted **only** on the `StreamEvents` server stream and only to the resuming subscriber — it is never fanned to other subscribers and never appears in `DrainEventsReply` (the diagnostic drain path is untouched). Replay retention itself is bounded by `MxGateway:Events:ReplayBufferCapacity` (count), `ReplayRetentionSeconds` (age) and `ReplayBufferMaxBytes` (summed serialized size, so a burst of large-array events cannot pin memory the count cap would allow); whichever binds first evicts oldest-first. Evicted slots are cleared at once, so an evicted event is released immediately rather than when its slot is next reused. See [Configuration](GatewayConfiguration.md).
|
||||
|
||||
#### Attach acknowledgement
|
||||
|
||||
A fresh stream is not replayed, so an event the session publishes before the gateway registers the stream's subscriber never reaches that stream. A client that advises an item right after issuing `StreamEvents` can lose the item's initial value that way, and for a slow-changing attribute (`ScanState`, a setpoint) the next value may be hours away (lmxopcua#865). The client cannot see when the HTTP/2 request has been processed, so it has nothing to wait on.
|
||||
|
||||
`StreamEventsRequest.emit_stream_attached` (#227, default `false`) gives it something. When set, `EventStreamService.StreamEventsAsync` yields one sentinel `MxEvent` with `stream_attached` populated as the **first** message of the stream, and yields it only after `AttachEventSubscriber` / `AttachEventSubscriberWithReplay` has returned — that is, after the subscriber channel is registered on the distributor. Every event the pump fans from that point lands in the channel, and the client cannot receive the sentinel before the server has executed that line, so "sentinel received" implies "registered": an advise issued after it cannot lose its initial value to the attach race. An attach that is refused (owner mismatch, `EventSubscriberAlreadyActive`, limit reached, session not ready) throws before the sentinel, so a refused stream never sees it.
|
||||
|
||||
The sentinel carries the session id and `StreamAttached.live_after_sequence` — the watermark live events are filtered at (`0` on a fresh stream, `LiveResumeSequence` on a resume). Like the ReplayGap sentinel its `family` is `UNSPECIFIED`, its `body` oneof is unset, its `worker_sequence` is `0` (so clients must not store it as a resume cursor), and it is a documented control signal rather than a synthesized MXAccess event. It is sent only to the requesting subscriber and never appears in `DrainEventsReply`.
|
||||
|
||||
It is opt-in because existing consumers treat every streamed `MxEvent` as data: ScadaBridge's MxGateway adapter stores every event's `worker_sequence` as its reconnect cursor, so an unrequested sentinel would reset that cursor to `0`. A gateway that predates the field ignores it (proto3 unknown field) and never sends the sentinel; gateways that honour it list `stream-events-attached-ack` (`GatewayCapabilities.StreamEventsAttachedAck`) in `OpenSessionReply.capabilities`, so a client can check before it waits.
|
||||
|
||||
### Close
|
||||
|
||||
`GatewaySession.CloseAsync` is serialized by a per-session `SemaphoreSlim` (`_closeLock`) so only one close runs at a time, but every read/write of `_state` still passes through `_syncRoot` (via `TryBeginClose` and `MarkClosed`). The close path therefore obeys the same lock discipline as `TransitionTo` / `MarkFaulted`: it transitions to `Closing`, asks the worker client to shut down within `ShutdownTimeout`, and on success transitions to `Closed`. `DisposeAsync` waits on `_closeLock` once before disposing the semaphore so an in-flight close's `Release()` cannot race against the dispose. `WorkerClient.ShutdownAsync` is bounded by `ShutdownTimeout` end to end — queuing the `WorkerShutdown` envelope included, since a worker that stopped reading the pipe can leave the bounded outbound channel full with no watchdog left to clear it once the client is `Closing` — so a close never holds `_closeLock` longer than the timeout before reaching the fallback. If `WorkerClient.ShutdownAsync` throws, the session falls back to `IWorkerClient.Kill` (forced close):
|
||||
|
||||
Reference in New Issue
Block a user