dohertj2/ScadaBridge
1
0
Fork 0
Code Issues 2 Pull Requests Actions Packages Projects Releases Wiki Activity

fix(dcl): generation-fence ConnectResult + roll back failed OPC UA subscribe batches (arch-review 2026-08-19 P2-3 + P2-4)

Browse Source 
P2-3. ConnectResult was the last piped completion with no adapter generation on
it, so a connect launched against generation N and answered after the adapter had
been swapped (failover, or the row-97 in-place reconfigure) was applied to its
REPLACEMENT. A stale success drove ReSubscribeAll + BecomeConnected against an
adapter that had never connected, and the replacement's own result then landed in
the Connected state where nothing handles it. With a bad new endpoint the batch
subscribes failed at connection level with a log line only, no AdapterDisconnected
ever fired, and the idempotent-reconfigure guard refused a re-send of the same
configuration — only a restart recovered the connection. A stale FAILURE was as
wrong the other way: it counted toward failover and re-armed the reconnect timer
underneath the attempt genuinely in flight.

HandleAttemptConnect now stamps the generation and both handlers open with the
same guard the subscribe leg got in c4d3c8b8, dropping a mismatched result whole.
The adapter instance is deliberately NOT carried: unlike a subscribe completion a
connect result holds no adapter-scoped handles to release, and SwapAdapter has
already disposed the outgoing adapter. Dropping cannot wedge the machine — every
generation bump (CountFailureAndMaybeFailover, BecomeReconnecting's failover
branch, HandleReconfigure through it) arms an AttemptConnect for the new
generation, and the state stays Connecting/Reconnecting until that answers.

P2-4. CreateSubscriptionsCoreAsync registers each monitored item as it places it
(_callbacks, _itemShard, shard budget, the SDK subscription's item list) and only
then issues ApplyChanges, so a throw there propagated with every registration
still in place and nothing holding a handle to release them. The actor reads the
throw as a whole-chunk fault and re-issues the SAME batch on its backoff timer, so
each probe grew the bookkeeping again. Placement + apply moved to
PlaceAndApplyAsync behind a try/catch that unwinds via the existing
ReleaseFailedItem (per-item try, so one failure neither stops the rest nor
replaces the original fault); no ApplyChanges is re-issued, and items a partial
apply did reach the server with are reclaimed when the shard and session are torn
down on reconnect.

That reconnect is the other half: IsConnectionLevelFailure knew nothing about
ServiceResultException, which is how the SDK reports a dead session — so a dead
session classified as a per-tag resolution failure for the whole chunk. It now
also matches a narrow set of session/channel status codes (BadSessionIdInvalid,
BadSessionClosed, BadSessionNotActivated, BadSecureChannelClosed,
BadSecureChannelIdInvalid, BadServerNotConnected, BadConnectionClosed,
BadCommunicationError, BadNotConnected). Per-node codes such as BadNodeIdUnknown
stay per-tag — that is exactly the device-still-booting case the resolution timer
exists for.

Tests: DataConnectionStaleConnectResultTests (stale success neither connects nor
re-subscribes against the new adapter; stale failure does not disturb the current
attempt; the current generation's result is still processed after a stale drop) —
all 3 fail with the guards removed. DataConnectionOpcUaFaultClassificationTests
(bad-session drives reconnect — fails unfixed; bad-node-id stays per-tag).
RealOpcUaClientBatchRollbackTests (no residual registrations after an ApplyChanges
throw; repeated failed batches do not grow the bookkeeping) — both fail with the
rollback removed. Full DCL suite 342/342 green.
This commit is contained in:
Joseph Doherty
2026-08-19 14:40:56 -04:00
parent e2d149089c
commit 26ab6f8f09
5 changed files with 632 additions and 46 deletions
Download Patch File Download Diff File Expand all files Collapse all files
src/ZB.MOM.WW.ScadaBridge.DataConnectionLayer/Actors/DataConnectionActor.cs
+85 -6
View File
@@ -778,16 +778,60 @@ public class DataConnectionActor : UntypedActor, IWithStash, IWithTimers
{
_log.Debug("[{0}] Attempting connection...", _connectionName);
var self = Self;
// Stamp the generation this attempt belongs to, exactly as every other piped
// completion does. A failover or an in-place reconfigure landing while ConnectAsync
// is still in flight swaps the adapter and bumps the counter, and the OUTGOING
// adapter's answer must never be applied to its replacement.
var generation = _adapterGeneration;
_adapter.ConnectAsync(_connectionDetails).ContinueWith(t =>
{
if (t.IsCompletedSuccessfully)
return new ConnectResult(true, null);
return new ConnectResult(false, t.Exception?.GetBaseException().Message);
return new ConnectResult(true, null, generation);
return new ConnectResult(false, t.Exception?.GetBaseException().Message, generation);
}).PipeTo(self);
}
/// <summary>
/// True when this connect result was produced by an adapter that has since been swapped
/// out, in which case it is dropped WHOLE (arch-review 2026-08-19 P2-3).
///
/// <para>
/// A success from the dead adapter used to drive <see cref="ReSubscribeAll"/> and
/// <see cref="BecomeConnected"/> against the NEW, never-connected adapter — and the new
/// adapter's own result then arrived in the Connected state, where nothing handles it. If the
/// new endpoint was bad the batch subscribes failed at connection level with a log line only,
/// no <c>AdapterDisconnected</c> ever fired, and the idempotent-reconfigure guard rejected
/// re-sending the same configuration: only a restart recovered the connection. A failure from
/// the dead adapter is just as wrong — it would count toward failover and re-arm the reconnect
/// timer underneath the attempt that is genuinely in flight.
/// </para>
///
/// <para>
/// Dropping cannot wedge the state machine: every site that bumps the generation
/// (<see cref="CountFailureAndMaybeFailover"/>, the failover branch of
/// <see cref="BecomeReconnecting"/>, and <see cref="HandleReconfigure"/> through it) arms an
/// <see cref="AttemptConnect"/> for the new generation, and the state stays Connecting /
/// Reconnecting until that attempt's own result arrives.
/// </para>
/// </summary>
private bool IsStaleConnectResult(ConnectResult result)
{
if (result.AdapterGeneration == _adapterGeneration)
return false;
_log.Debug(
"[{0}] Dropping {1} connect result from adapter generation {2} (current {3}) — "
+ "the adapter it ran against has been replaced",
_connectionName, result.Success ? "successful" : "failed",
result.AdapterGeneration, _adapterGeneration);
return true;
}
private void HandleConnectResult(ConnectResult result)
{
if (IsStaleConnectResult(result))
return;
if (result.Success)
{
_log.Info("[{0}] Connection established", _connectionName);
@@ -809,6 +853,9 @@ public class DataConnectionActor : UntypedActor, IWithStash, IWithTimers
private void HandleReconnectResult(ConnectResult result)
{
if (IsStaleConnectResult(result))
return;
if (result.Success)
{
_log.Info("[{0}] Reconnected successfully on {1} endpoint", _connectionName, _activeEndpoint);
@@ -1538,11 +1585,38 @@ public class DataConnectionActor : UntypedActor, IWithStash, IWithTimers
{
var baseEx = ex is AggregateException agg ? agg.GetBaseException() : ex;
return baseEx is InvalidOperationException
or System.Net.Sockets.SocketException
or TimeoutException
or System.IO.IOException;
or System.Net.Sockets.SocketException
or TimeoutException
or System.IO.IOException
|| (baseEx is Opc.Ua.ServiceResultException sre && IsConnectionLevelStatus(sre.StatusCode));
}
/// <summary>
/// OPC UA status codes that mean the SESSION or the SECURE CHANNEL is gone, not that one
/// node is unusable (arch-review 2026-08-19 P2-4). The SDK reports a dead session by
/// throwing <see cref="Opc.Ua.ServiceResultException"/> out of the batch subscribe, which
/// classified as a per-tag fault: every tag in the chunk was marked unresolved and each
/// backoff probe re-issued the whole batch against the same dead session, growing the
/// adapter's monitored-item bookkeeping on every round instead of reconnecting once.
///
/// <para>
/// Deliberately narrow. A per-node code — <c>BadNodeIdUnknown</c>, <c>BadNodeIdInvalid</c>,
/// <c>BadUserAccessDenied</c> — must stay a tag-resolution failure, because that is exactly
/// the "device still booting / tag path wrong" case the retry timer exists for; treating it
/// as connection-level would put the connection into a reconnect loop over one bad tag path.
/// </para>
/// </summary>
private static bool IsConnectionLevelStatus(uint statusCode) => statusCode is
Opc.Ua.StatusCodes.BadSessionIdInvalid
or Opc.Ua.StatusCodes.BadSessionClosed
or Opc.Ua.StatusCodes.BadSessionNotActivated
or Opc.Ua.StatusCodes.BadSecureChannelClosed
or Opc.Ua.StatusCodes.BadSecureChannelIdInvalid
or Opc.Ua.StatusCodes.BadServerNotConnected
or Opc.Ua.StatusCodes.BadConnectionClosed
or Opc.Ua.StatusCodes.BadCommunicationError
or Opc.Ua.StatusCodes.BadNotConnected;
private void HandleUnsubscribe(UnsubscribeTagsRequest request)
{
_log.Debug("[{0}] Unsubscribing all tags for instance {1}",
@@ -2881,7 +2955,12 @@ public class DataConnectionActor : UntypedActor, IWithStash, IWithTimers
// ── Internal messages ──
internal record AttemptConnect;
internal record ConnectResult(bool Success, string? Error);
/// <summary>
/// Outcome of one <see cref="AttemptConnect"/>. Carries the adapter generation it ran
/// against so a result produced by an adapter that has since been swapped out is dropped
/// rather than applied to its replacement (see <see cref="IsStaleConnectResult"/>).
/// </summary>
internal record ConnectResult(bool Success, string? Error, int AdapterGeneration);
internal record AdapterDisconnected;
internal record TagValueReceived(string TagPath, TagValue Value, int AdapterGeneration);
internal record RetryTagResolution;