fix(dcl): MxGateway resume cursor ignores worker_sequence-0 sentinels (#128)

RealMxGatewayClient.RunEventLoopAsync stored every streamed MxEvent's
WorkerSequence as its StreamEvents resume cursor before filtering by family.
The gateway's control sentinels (ReplayGap on a resume that hit evicted
events; the opt-in StreamAttached ack) carry worker_sequence 0, so a gap
reset the cursor to 0 and the next resume became a fresh stream: no replay,
no gap signal, silent loss.

Cursor rule: advance-only (max), from real worker events (sequence > 0) only.
A ReplayGap anchors the cursor at oldest_available_sequence - 1 (the gateway's
documented gap-free resume point, valid for an empty ring too) and is logged
as a warning naming the lost interval. Sentinels never reach the value
callback.

The loop body is split into internal RunEventLoopCoreAsync(IAsyncEnumerable)
so tests drive a scripted stream; 4 of the 7 new tests fail on the old loop.

Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi
This commit is contained in:
Joseph Doherty
2026-11-12 07:35:34 -05:00
parent 08ef94bb21
commit 94ae186050
3 changed files with 239 additions and 4 deletions
@@ -101,7 +101,9 @@ public interface IMxGatewayClient : IAsyncDisposable
/// <summary>
/// Long-running event consumer. Invokes <paramref name="onUpdate"/> for each advised-tag
/// data change. Resumes from the last delivered worker sequence on reconnect. Completes
/// data change. Re-invoking it on the same client resumes after the last real worker
/// sequence seen (gateway control sentinels never move that cursor; a replay gap is logged).
/// The adapter's reconnect builds a new client and session, which streams fresh. Completes
/// (or throws) when the stream ends — the adapter treats that as a disconnect.
/// </summary>
/// <param name="onUpdate">Callback invoked per advised-tag value change.</param>
@@ -393,11 +393,53 @@ public sealed class RealMxGatewayClient : IMxGatewayClient
}
/// <inheritdoc />
public async Task RunEventLoopAsync(Action<MxValueUpdate> onUpdate, CancellationToken ct = default)
public Task RunEventLoopAsync(Action<MxValueUpdate> onUpdate, CancellationToken ct = default)
=> RunEventLoopCoreAsync(_session!.StreamEventsAsync(_lastSeq, ct), onUpdate, ct);
/// <summary>The worker sequence the next <see cref="RunEventLoopAsync"/> resumes after.</summary>
internal ulong ResumeCursor => _lastSeq;
private void AdvanceResumeCursor(ulong sequence)
{
await foreach (var ev in _session!.StreamEventsAsync(_lastSeq, ct).ConfigureAwait(false))
if (sequence > _lastSeq)
_lastSeq = sequence;
}
/// <summary>Consumes one event stream; split out so tests can drive a scripted stream.</summary>
/// <remarks>
/// Resume-cursor rule (ScadaBridge#128): the cursor only ever advances, and only from
/// real worker events. The gateway's control sentinels — the ReplayGap sentinel on a
/// resume that hit evicted events, and the opt-in StreamAttached ack this adapter does not
/// request — carry <c>worker_sequence</c> 0; storing that would turn the next resume into
/// a fresh stream (no replay, no gap signal) and silently lose everything in between.
/// A ReplayGap anchors the cursor at <c>oldest_available_sequence - 1</c>, the gateway's
/// documented gap-free resume point (it also holds when the ring is empty, where the
/// gateway reports <c>highest observed + 1</c>), so a stream that drops before any
/// replayed or live event arrives does not hit — and re-report — the same gap again.
/// </remarks>
internal async Task RunEventLoopCoreAsync(
IAsyncEnumerable<MxEvent> events, Action<MxValueUpdate> onUpdate, CancellationToken ct)
{
await foreach (var ev in events.WithCancellation(ct).ConfigureAwait(false))
{
_lastSeq = ev.WorkerSequence;
if (ev.ReplayGap is { } gap)
{
// Events in (requested, oldest) were evicted from the gateway's replay ring and
// are unrecoverable: tags whose only change fell in that window keep a stale
// value until they next change. Surface it — never hide the loss.
_logger.LogWarning(
"MxGateway event stream resumed after worker sequence {RequestedAfterSequence} but the gateway only retains events from {OldestAvailableSequence}; the events in between were lost and subscribed values may be stale until they next change",
gap.RequestedAfterSequence, gap.OldestAvailableSequence);
if (gap.OldestAvailableSequence > 0)
AdvanceResumeCursor(gap.OldestAvailableSequence - 1);
continue;
}
// A 0 sequence is a control sentinel (e.g. StreamAttached), never a worker event.
if (ev.WorkerSequence == 0)
continue;
AdvanceResumeCursor(ev.WorkerSequence);
if (ev.Family != MxEventFamily.OnDataChange)
continue;
if (!_handleToTag.TryGetValue(ev.ItemHandle, out var tag))