dohertj2/lmxopcua
1
0
Fork 0
Code Issues 17 Pull Requests Actions Packages Projects Releases Wiki Activity

fix(mqtt): make connect idempotent, bound the reconnect callback, stop State lying

Browse Source 
Closes the connect-vs-connect defect: MQTTnet throws (and raises no DisconnectedAsync)
on a connect-while-connected, so a caller's ConnectAsync and the reconnect supervisor
corrupted each other in both directions. Both paths now check first; when the supervisor
finds the session already restored it stands down WITHOUT firing Reconnected.

Reconnected becomes Func<CancellationToken, Task>, fed from the lifetime token and capped
at ConnectTimeoutSeconds, so a hung re-subscribe can no longer park the supervisor.
Connected is published only after the re-subscribe succeeds; a supervisor that dies now
reports Faulted instead of Reconnecting forever.

Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
This commit is contained in:
Joseph Doherty
2026-07-24 15:32:41 -04:00
parent 768fd87774
commit a6370a26f8
2 changed files with 545 additions and 56 deletions
Download Patch File Download Diff File Expand all files Collapse all files
src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Mqtt/MqttConnection.cs
+268 -46
View File
@@ -22,8 +22,10 @@ public enum MqttConnectionState
Reconnecting = 2,
/// <summary>
/// Unrecoverable <i>configuration</i>: retrying cannot help, so the supervisor stops. Only
/// failures raised while assembling the client options (before any I/O) land here.
/// Unrecoverable without a driver restart: retrying cannot help, so the supervisor has
/// stopped. Two causes — unusable configuration (a failure raised while assembling the client
/// options, before any I/O), or the supervisor itself dying on an unexpected exception. A
/// broker that is merely unreachable never lands here.
/// </summary>
Faulted = 3,
@@ -69,7 +71,64 @@ public enum MqttConnectionState
/// persistent-session reconnects, where re-subscribing is merely redundant. Re-subscribing
/// an already-subscribed topic is harmless; missing one is silent death. If the callback
/// fails, the freshly-established session is torn down and retried rather than left
/// connected-but-deaf.
/// connected-but-deaf. It is invoked under a cancellation token and a
/// <see cref="MqttDriverOptions.ConnectTimeoutSeconds"/> ceiling: a subscriber that hangs on
/// a broker which accepts SUBSCRIBE and never SUBACKs — the frozen-peer shape again — must
/// not be able to park the supervisor or outlive <see cref="DisposeAsync"/>.
/// </para>
/// <para>
/// <b>Connect is idempotent.</b> MQTTnet throws (and raises no <c>DisconnectedAsync</c>) if
/// asked to connect a client that is already connected, so both connect paths check first and
/// return a no-op. The two paths genuinely race: the supervisor is told to expect the driver
/// host to re-run <c>InitializeAsync</c>, so a caller can restore the session while the
/// supervisor sits in backoff, and the supervisor can restore it a moment before a caller
/// asks. Without the check, the first case leaves the supervisor failing forever against a
/// perfectly healthy connection (inflating <c>attempt</c>, so the <i>next</i> genuine outage
/// waits <see cref="MqttDriverOptions.ReconnectMaxBackoffSeconds"/> instead of min) and the
/// second throws an undocumented <see cref="InvalidOperationException"/> out of
/// <see cref="ConnectAsync"/> on a working session. When the supervisor finds the session
/// already restored it stops retrying <b>without</b> firing <see cref="Reconnected"/> — the
/// caller that reconnected owns its own subscribe, per the <see cref="ConnectAsync"/>
/// contract.
/// </para>
/// <para>
/// <b>State transitions.</b> <see cref="MqttConnectionState.Disposed"/> is terminal; nothing
/// moves off it.
/// <list type="table">
/// <item>
/// <term><see cref="MqttConnectionState.Connected"/></term>
/// <description>
/// set by <see cref="ConnectAsync"/> on success (or on finding the session
/// already up), and by the supervisor <i>only after</i>
/// <see cref="Reconnected"/> has completed — so on the reconnect path
/// <c>Connected</c> means "connected <i>and</i> re-subscribed".
/// </description>
/// </item>
/// <item>
/// <term><see cref="MqttConnectionState.Reconnecting"/></term>
/// <description>
/// set by the <c>DisconnectedAsync</c> callback when a live session drops, and
/// held by the supervisor for the whole retry + re-subscribe sequence.
/// </description>
/// </item>
/// <item>
/// <term><see cref="MqttConnectionState.Faulted"/></term>
/// <description>
/// set when the client options cannot be assembled, or when the supervisor exits
/// on an unexpected exception. Both mean no further recovery without a restart.
/// </description>
/// </item>
/// <item>
/// <term><see cref="MqttConnectionState.Disposed"/></term>
/// <description>set first thing in <see cref="DisposeAsync"/>.</description>
/// </item>
/// </list>
/// <see cref="State"/> and <see cref="IsConnected"/> can legitimately disagree in <b>both</b>
/// directions and are not interchangeable: <c>Reconnecting</c> with
/// <c>IsConnected == true</c> is the window where the socket is up but the re-subscribe has
/// not finished, and <c>Connected</c> with <c>IsConnected == false</c> is the instant between
/// the socket dropping and MQTTnet raising its callback. <see cref="State"/> is the health
/// surface; <see cref="IsConnected"/> is the transport fact.
/// </para>
/// <para>
/// <b>Credentials never leak.</b> Nothing here logs or formats
@@ -99,7 +158,15 @@ public sealed class MqttConnection : IAsyncDisposable
/// </summary>
private readonly SemaphoreSlim _lifecycleGate = new(1, 1);
/// <summary>Cancelled by <see cref="DisposeAsync"/>; aborts the supervisor and any in-flight connect.</summary>
/// <summary>
/// Cancelled by <see cref="DisposeAsync"/>; aborts the supervisor, any in-flight connect and
/// any in-flight <see cref="Reconnected"/> callback. Never disposed — deliberately, like the
/// two semaphores: a connect racing dispose reads <see cref="CancellationTokenSource.Token"/>
/// after its own disposed check, and disposing the source would turn that benign loser into an
/// <see cref="ObjectDisposedException"/> naming the wrong type. It holds no timer and no
/// surviving registrations (every linked source here is <c>using</c>-scoped), so leaving it
/// undisposed costs nothing.
/// </summary>
private readonly CancellationTokenSource _lifetimeCts = new();
private readonly ILogger? _logger;
@@ -135,12 +202,20 @@ public sealed class MqttConnection : IAsyncDisposable
/// <summary>
/// Raised after every successful <i>re</i>connect (never after the initial
/// <see cref="ConnectAsync"/>, which the caller already sequences its own subscribe behind).
/// <see cref="ConnectAsync"/>, which the caller already sequences its own subscribe behind,
/// and never when the supervisor merely finds the session already restored by such a caller).
/// Subscribers must re-establish their MQTT subscriptions here — see the type remarks for
/// why skipping it is silent death. Every subscriber is invoked even if an earlier one
/// throws; a throwing subscriber causes the session to be torn down and retried.
/// </summary>
public event Func<Task>? Reconnected;
/// <remarks>
/// The supplied token is cancelled by <see cref="DisposeAsync"/>, and the whole fan-out is
/// additionally capped at <see cref="MqttDriverOptions.ConnectTimeoutSeconds"/> — a subscriber
/// that ignores its token cannot park the supervisor or make it outlive teardown. Subscribers
/// should flow the token into their own network calls rather than relying on that ceiling,
/// which abandons rather than stops the offending work.
/// </remarks>
public event Func<CancellationToken, Task>? Reconnected;
/// <summary>Whether the underlying client currently holds an established MQTT session.</summary>
public bool IsConnected => _client?.IsConnected ?? false;
@@ -285,11 +360,20 @@ public sealed class MqttConnection : IAsyncDisposable
/// supervisor takes over keeping the session alive.
/// </summary>
/// <remarks>
/// May be retried on the same instance after a failed attempt — the underlying
/// <see cref="IMqttClient"/> is created once and reused across attempts. Safe to call
/// concurrently with <see cref="DisposeAsync"/>: the two are serialised, and a connect that
/// loses the race disposes whatever it built and throws
/// <see cref="ObjectDisposedException"/>.
/// <para>
/// May be retried on the same instance after a failed attempt — the underlying
/// <see cref="IMqttClient"/> is created once and reused across attempts. Safe to call
/// concurrently with <see cref="DisposeAsync"/>: the two are serialised, and a connect
/// that loses the race disposes whatever it built and throws
/// <see cref="ObjectDisposedException"/>.
/// </para>
/// <para>
/// <b>Idempotent.</b> Calling it on a session that is already established — including one
/// the reconnect supervisor restored a moment earlier — is a no-op that returns normally,
/// not the <see cref="InvalidOperationException"/> MQTTnet would raise for a
/// connect-while-connected. The caller still owns re-establishing its own subscriptions
/// after this returns; <see cref="Reconnected"/> is not fired for this path.
/// </para>
/// </remarks>
/// <param name="cancellationToken">Caller cancellation; linked with the connect deadline.</param>
/// <exception cref="TimeoutException">The connect deadline elapsed.</exception>
@@ -297,21 +381,22 @@ public sealed class MqttConnection : IAsyncDisposable
/// <exception cref="ObjectDisposedException">
/// The connection was disposed, either before the attempt started or while it was in flight.
/// </exception>
public Task ConnectAsync(CancellationToken cancellationToken)
public async Task ConnectAsync(CancellationToken cancellationToken)
{
ObjectDisposedException.ThrowIf(Disposed, this);
return ConnectCoreAsync(startSupervisor: true, cancellationToken);
await ConnectCoreAsync(supervisorAttempt: false, cancellationToken).ConfigureAwait(false);
}
/// <summary>
/// Invokes every <see cref="Reconnected"/> subscriber. Walks the invocation list explicitly:
/// awaiting a multicast <see cref="Func{Task}"/> directly yields only the <i>last</i>
/// subscriber's task and abandons the earlier ones, so a single throwing subscriber would
/// silently skip every subscriber behind it — and a skipped re-subscribe is a topic that goes
/// dark. Failures are collected and rethrown after every subscriber has had its turn.
/// awaiting a multicast delegate directly yields only the <i>last</i> subscriber's task and
/// abandons the earlier ones, so a single throwing subscriber would silently skip every
/// subscriber behind it — and a skipped re-subscribe is a topic that goes dark. Failures are
/// collected and rethrown after every subscriber has had its turn.
/// </summary>
internal async Task FireReconnectedAsync()
/// <param name="cancellationToken">Passed to every subscriber; cancelled by <see cref="DisposeAsync"/>.</param>
internal async Task FireReconnectedAsync(CancellationToken cancellationToken)
{
var subscribers = Reconnected;
if (subscribers is null)
@@ -320,11 +405,11 @@ public sealed class MqttConnection : IAsyncDisposable
}
List<Exception>? failures = null;
foreach (var subscriber in subscribers.GetInvocationList().Cast<Func<Task>>())
foreach (var subscriber in subscribers.GetInvocationList().Cast<Func<CancellationToken, Task>>())
{
try
{
await subscriber().ConfigureAwait(false);
await subscriber(cancellationToken).ConfigureAwait(false);
}
catch (Exception ex)
{
@@ -340,7 +425,23 @@ public sealed class MqttConnection : IAsyncDisposable
}
}
private async Task ConnectCoreAsync(bool startSupervisor, CancellationToken cancellationToken)
/// <summary>
/// The single gated connect both entry points funnel through.
/// </summary>
/// <param name="supervisorAttempt">
/// <c>true</c> when the reconnect supervisor is calling. It suppresses publishing
/// <see cref="MqttConnectionState.Connected"/> on success, because on the reconnect path
/// "connected" is only true once <see cref="Reconnected"/> has re-subscribed — publishing it
/// here would advertise a healthy connection during the window in which it has no
/// subscriptions at all. It also suppresses starting the supervisor (it is the supervisor).
/// </param>
/// <param name="cancellationToken">Caller cancellation; linked with the connect deadline.</param>
/// <returns>
/// <c>true</c> if this call established the session; <c>false</c> if it found one already
/// established and did nothing. The supervisor uses <c>false</c> to stop retrying <i>without</i>
/// firing <see cref="Reconnected"/> — whoever established that session owns its subscribe.
/// </returns>
private async Task<bool> ConnectCoreAsync(bool supervisorAttempt, CancellationToken cancellationToken)
{
// The options are rebuilt per attempt so a rotated CA file is picked up on the next connect
// rather than being pinned for the life of the process. This is also the only step that can
@@ -397,6 +498,23 @@ public sealed class MqttConnection : IAsyncDisposable
_client = client;
}
// Idempotence, and the whole point of the gate: MQTTnet throws
// "not allowed to connect with a server after the connection is established" for a
// connect-while-connected AND raises no DisconnectedAsync for it, so neither the retry
// loop nor Classify would ever recover. Both entry points race for real — see the type
// remarks — so whichever arrives second must find the session and leave it alone.
if (client.IsConnected)
{
SetState(MqttConnectionState.Connected);
StartSupervisorIfCallerConnect(supervisorAttempt);
_logger?.LogDebug(
"MQTT driver '{DriverId}': connect to {Host}:{Port} skipped — the session is already established.",
_driverId,
_options.Host,
_options.Port);
return false;
}
await client.ConnectAsync(clientOptions, linked.Token).ConfigureAwait(false);
if (AfterConnectHookForTests is { } hook)
@@ -417,12 +535,15 @@ public sealed class MqttConnection : IAsyncDisposable
+ $"{_options.Host}:{_options.Port} was in flight; the established session was torn down.");
}
SetState(MqttConnectionState.Connected);
if (startSupervisor)
// On the supervisor path the session is up but has no subscriptions yet, so State stays
// Reconnecting until FireReconnectedAsync succeeds — see the state table in the remarks.
if (!supervisorAttempt)
{
_reconnectWorker ??= Task.Run(() => ReconnectSupervisorAsync(_lifetimeCts.Token));
SetState(MqttConnectionState.Connected);
}
StartSupervisorIfCallerConnect(supervisorAttempt);
return true;
}
catch (ObjectDisposedException)
{
@@ -438,6 +559,18 @@ public sealed class MqttConnection : IAsyncDisposable
}
}
/// <summary>
/// Starts the supervisor on the first <i>successful</i> caller connect. Always called while
/// holding the lifecycle gate, so the <c>??=</c> needs no further synchronisation.
/// </summary>
private void StartSupervisorIfCallerConnect(bool supervisorAttempt)
{
if (!supervisorAttempt)
{
_reconnectWorker ??= Task.Run(() => ReconnectSupervisorAsync(_lifetimeCts.Token));
}
}
/// <summary>
/// Maps a connect failure onto this type's documented contract. MQTTnet wraps a cancelled
/// connect in <c>MqttConnectingFailedException</c> rather than letting the
@@ -537,6 +670,14 @@ public sealed class MqttConnection : IAsyncDisposable
for (var attempt = 0; !cancellationToken.IsCancellationRequested && !Disposed; attempt++)
{
// A caller's ConnectAsync may have restored the session while we slept. Bail
// before burning the backoff, not just before burning an attempt.
if (IsConnected)
{
AdoptSessionRestoredByCaller();
break;
}
await Task
.Delay(
NextBackoff(
@@ -546,11 +687,13 @@ public sealed class MqttConnection : IAsyncDisposable
cancellationToken)
.ConfigureAwait(false);
bool established;
try
{
await ConnectCoreAsync(startSupervisor: false, cancellationToken).ConfigureAwait(false);
established = await ConnectCoreAsync(supervisorAttempt: true, cancellationToken)
.ConfigureAwait(false);
}
catch (ObjectDisposedException)
catch (ObjectDisposedException) when (Disposed)
{
return;
}
@@ -575,25 +718,24 @@ public sealed class MqttConnection : IAsyncDisposable
continue;
}
try
if (!established)
{
// ALWAYS, on every reconnect — including persistent sessions, where this is
// merely redundant. Subscriptions do not survive a clean session, and a
// reconnect that skips this leaves a healthy-looking, permanently deaf client.
await FireReconnectedAsync().ConfigureAwait(false);
// Someone else restored the session between the backoff and the connect.
// Firing Reconnected here would re-subscribe on top of a caller that is about
// to do exactly that itself, and — worse — leaving the loop running would
// keep failing against a healthy connection forever.
AdoptSessionRestoredByCaller();
break;
}
catch (Exception ex)
if (!await TryReSubscribeAsync(cancellationToken).ConfigureAwait(false))
{
_logger?.LogError(
ex,
"MQTT driver '{DriverId}': re-subscribe after reconnect failed; tearing the session down "
+ "and retrying rather than serving a connection that receives nothing.",
_driverId);
SetState(MqttConnectionState.Reconnecting);
await ForceDisconnectAsync().ConfigureAwait(false);
continue;
}
// Only now is "Connected" true in the sense this type sells it: connected AND
// subscribed.
SetState(MqttConnectionState.Connected);
_logger?.LogInformation(
"MQTT driver '{DriverId}': reconnected to {Host}:{Port} after {Attempts} attempt(s); "
+ "subscriptions re-established.",
@@ -612,7 +754,9 @@ public sealed class MqttConnection : IAsyncDisposable
catch (Exception ex)
{
// Reaching here means the connection is permanently dark, which is exactly the failure
// this supervisor exists to prevent — it must never be swallowed quietly.
// this supervisor exists to prevent — it must never be swallowed quietly, and State must
// not keep reporting Reconnecting, which every consumer reads as "recovering".
SetState(MqttConnectionState.Faulted);
_logger?.LogError(
ex,
"MQTT driver '{DriverId}': reconnect supervisor stopped unexpectedly; the broker session will not "
@@ -621,6 +765,71 @@ public sealed class MqttConnection : IAsyncDisposable
}
}
/// <summary>
/// The session came back without this supervisor's help — a caller's <see cref="ConnectAsync"/>
/// won the race while we were in backoff. Publish the truth and stand down; deliberately does
/// <b>not</b> fire <see cref="Reconnected"/>, because that caller owns its own re-subscribe.
/// </summary>
private void AdoptSessionRestoredByCaller()
{
SetState(MqttConnectionState.Connected);
_logger?.LogDebug(
"MQTT driver '{DriverId}': reconnect stood down — the session to {Host}:{Port} was restored elsewhere.",
_driverId,
_options.Host,
_options.Port);
}
/// <summary>
/// Fires <see cref="Reconnected"/> — ALWAYS, on every reconnect, including persistent
/// sessions where it is merely redundant, because subscriptions do not survive a clean
/// session and a reconnect that skips this leaves a healthy-looking, permanently deaf client.
/// </summary>
/// <returns>
/// <c>false</c> if the re-subscribe failed or overran its deadline, having torn the session
/// down so the next attempt starts from a clean CONNECT — serving a connection that receives
/// nothing is the one outcome this type must never produce.
/// </returns>
private async Task<bool> TryReSubscribeAsync(CancellationToken cancellationToken)
{
// Bounded like every other wait here. A subscriber that flows the token cancels promptly; one
// that ignores it is abandoned at the ceiling rather than being allowed to park the
// supervisor past DisposeAsync.
var fanOut = FireReconnectedAsync(cancellationToken);
// The ceiling abandons the task rather than stopping it, so its eventual failure must still
// be observed or it resurfaces as an unobserved TaskException on the finalizer thread.
_ = fanOut.ContinueWith(
static abandoned => _ = abandoned.Exception,
CancellationToken.None,
TaskContinuationOptions.OnlyOnFaulted | TaskContinuationOptions.ExecuteSynchronously,
TaskScheduler.Default);
try
{
await fanOut
.WaitAsync(TimeSpan.FromSeconds(_options.ConnectTimeoutSeconds), cancellationToken)
.ConfigureAwait(false);
return true;
}
catch (OperationCanceledException) when (cancellationToken.IsCancellationRequested)
{
throw; // Teardown — let the supervisor's own cancellation handling end the loop.
}
catch (Exception ex)
{
_logger?.LogError(
ex,
"MQTT driver '{DriverId}': re-subscribe after reconnect failed or overran {TimeoutSeconds}s; tearing "
+ "the session down and retrying rather than serving a connection that receives nothing.",
_driverId,
_options.ConnectTimeoutSeconds);
SetState(MqttConnectionState.Reconnecting);
await ForceDisconnectAsync().ConfigureAwait(false);
return false;
}
}
/// <summary>
/// Drops a session that is connected but unusable, so the next supervisor pass starts from a
/// clean CONNECT. Best-effort and bounded — failure here just means the next attempt sees a
@@ -666,15 +875,28 @@ public sealed class MqttConnection : IAsyncDisposable
}
}
/// <summary><see cref="MqttConnectionState.Disposed"/> is terminal — nothing may move off it.</summary>
/// <summary>
/// <see cref="MqttConnectionState.Disposed"/> is terminal — nothing may move off it. A
/// check-then-write would let a thread preempted between the two steps resurrect a disposed
/// connection's state, so the transition is a compare-and-swap: the terminal check and the
/// write are the same atomic operation.
/// </summary>
private void SetState(MqttConnectionState state)
{
if (Volatile.Read(ref _state) == (int)MqttConnectionState.Disposed)
var desired = (int)state;
while (true)
{
return;
}
var current = Volatile.Read(ref _state);
if (current == (int)MqttConnectionState.Disposed || current == desired)
{
return;
}
Volatile.Write(ref _state, (int)state);
if (Interlocked.CompareExchange(ref _state, desired, current) == current)
{
return;
}
}
}
/// <summary>