dohertj2/ScadaBridge
1
0
Fork 0
Code Issues 2 Pull Requests 1 Actions Packages Projects Releases Wiki Activity

fix(dcl): derive tag-resolution health counts from per-tag authoritative state

Browse Source 
Closes arch-review remediation residual #1 (DCL unsubscribe-during-reconnect
count staleness).

DataConnectionActor tracked TotalSubscribedTags/ResolvedTags as two int fields
incremented and decremented at five independent sites. ReSubscribeAll clears the
very maps those decrements key off (_subscriptionIds, _unresolvedTags) while
deliberately preserving _subscriptionsByInstance, so an unsubscribe landing
inside a reconnect window matched NEITHER decrement branch: the total leaked +1
per subscribe/reconnect/unsubscribe churn cycle, permanently and cumulatively.
The 37f13e2e discard gate stopped the orphan-handle half of that race; it could
not stop the counters drifting, because they were state of their own.

Both counts are now DERIVED at report time from the authoritative per-tag
collections, which makes the drift unrepresentable rather than merely guarded:

  total    = _instancesByTag.Count   (the per-tag counted set the residual
                                      called for — distinct tags with at least
                                      one subscribing instance)
  resolved = _subscriptionIds.Count  (tags for which the adapter holds a handle)

Two semantic corrections fall out of the derivation:

- A tag whose subscribe failed at CONNECTION level now counts toward the total.
  It was excluded before, yet the reconnect re-subscribe re-issued it from
  _subscriptionsByInstance and booked it as resolved — resolved above total, and
  a total driven negative by the eventual unsubscribe.
- _tagSubscriberCount is deleted. It duplicated _instancesByTag exactly, so
  HandleUnsubscribe's last-subscriber test is now "did UnindexTag drop the key?"
  — still O(1), with no parallel count that can disagree about when a handle is
  released. The subscribe-success promotion split (fresh vs. unresolved→resolved)
  also goes: it existed only to pick which scalar to bump; set sizes get
  DataConnectionLayer-020's double-count cases right for free.

Behavior is otherwise unchanged — same logging, same handle release, same
unresolved-tag probing, same in-flight-unsubscribe discard semantics (the long
comment block there is updated for the mechanics that changed).

Tests: five TagResolutionCounts_* cases in DataConnectionActorBatchTests
covering the churn repro (3 cycles), a shared tag losing one instance mid
reconnect, connection-level failure then recovery, plain subscribe/unsubscribe
cycles, and a completed reconnect re-subscribe. Verified failing against the
pre-fix actor (churn: total 1 not 0; connection-level: total 0 not 1) and
passing after. Full DCL suite 319/319; solution builds with 0 warnings.

Docs: Component-DataConnectionLayer.md health-reporting section describes the
derived counts; residuals register item 1 marked RESOLVED.
This commit is contained in:
Joseph Doherty
2026-08-15 02:05:35 -04:00
parent 986e6e7ad5
commit 491df111ea
5 changed files with 389 additions and 125 deletions
Download Patch File Download Diff File Expand all files Collapse all files
src/ZB.MOM.WW.ScadaBridge.DataConnectionLayer/Actors/DataConnectionActor.cs
+134 -118
View File
@@ -55,21 +55,25 @@ public class DataConnectionActor : UntypedActor, IWithStash, IWithTimers
private readonly Dictionary<string, string> _subscriptionIds = new();
/// <summary>
/// Reverse index of how many instances subscribe to each
/// tag path. Lets <see cref="HandleUnsubscribe"/> decide whether any other instance
/// still needs a tag in O(1) instead of scanning every instance's tag set.
/// </summary>
private readonly Dictionary<string, int> _tagSubscriberCount = new();
/// <summary>
/// Reverse index of which instances subscribe to each tag path: tagPath → set of
/// instanceUniqueName. Mirrors <see cref="_subscriptionsByInstance"/> inverted so the
/// <see cref="HandleTagValueReceived"/> hot path fans a value out to exactly the
/// interested instances in O(subscribers) instead of scanning every instance's tag set
/// Per-tag COUNTED SET — the reverse index of which instances subscribe to each tag
/// path: tagPath → set of instanceUniqueName. Mirrors <see cref="_subscriptionsByInstance"/>
/// inverted so the <see cref="HandleTagValueReceived"/> hot path fans a value out to exactly
/// the interested instances in O(subscribers) instead of scanning every instance's tag set
/// on every tag update. Maintained via <see cref="IndexTag"/>/<see cref="UnindexTag"/>
/// at every <see cref="_subscriptionsByInstance"/> tag-set mutation; a tag key is dropped
/// when its instance set empties. Derived purely from <see cref="_subscriptionsByInstance"/>,
/// so it is preserved across reconnect exactly as that map is (see <see cref="ReSubscribeAll"/>).
///
/// It is also THE authority for three other questions, so that no independently-mutated
/// counter can drift away from it:
/// <list type="bullet">
/// <item>"does any instance still need this tag?" — <see cref="HandleUnsubscribe"/>'s
/// last-subscriber test is simply "did <see cref="UnindexTag"/> drop the key?" (O(1),
/// replacing a separate <c>_tagSubscriberCount</c> map that had to be kept in step);</item>
/// <item>"was this tag unsubscribed while a batch subscribe was in flight?" — the discard
/// gates in <see cref="HandleBatchSubscribeCompleted"/>;</item>
/// <item>the reported <see cref="TotalSubscribedTagCount"/>.</item>
/// </list>
/// </summary>
private readonly Dictionary<string, HashSet<string>> _instancesByTag = new();
@@ -130,10 +134,34 @@ public class DataConnectionActor : UntypedActor, IWithStash, IWithTimers
private readonly HashSet<string> _alarmSubscribesInFlight = new();
/// <summary>
/// Tracks total subscribed and resolved tags for health reporting.
/// Total subscribed tags for health reporting: the number of DISTINCT tag paths that at
/// least one instance currently subscribes to.
///
/// DERIVED, never accumulated. This used to be an <c>int</c> field incremented and
/// decremented at five independent sites (initial subscribe success, initial subscribe
/// resolution failure, both unsubscribe branches) while the reconnect path cleared the
/// maps those decrements keyed off — so an unsubscribe that landed inside a reconnect
/// window matched NEITHER decrement branch and leaked +1 per churn cycle, permanently.
/// Reading the counted set at report time makes that class of drift unrepresentable:
/// the number can only ever be what the authoritative per-tag state says it is.
///
/// A tag counts from the moment it is registered against an instance, whatever its
/// resolution outcome — resolved, awaiting a resolution retry, or failed at connection
/// level (that last case used to be excluded, which let <see cref="ResolvedTagCount"/>
/// climb ABOVE the total once the reconnect re-subscribe resolved it).
/// </summary>
private int _totalSubscribed;
private int _resolvedTags;
private int TotalSubscribedTagCount => _instancesByTag.Count;
/// <summary>
/// Resolved tags for health reporting — DERIVED, for the same reason as
/// <see cref="TotalSubscribedTagCount"/>. A tag is resolved exactly when the adapter has
/// handed back a subscription handle for it, so <see cref="_subscriptionIds"/> IS the
/// count; the retired <c>_resolvedTags</c> field was incremented and decremented in
/// lock-step with every mutation of that dictionary anyway. Never exceeds
/// <see cref="TotalSubscribedTagCount"/>, because a handle is only ever stored for a tag
/// that is present in <see cref="_instancesByTag"/> (enforced at every store site).
/// </summary>
private int ResolvedTagCount => _subscriptionIds.Count;
private int _tagsGoodQuality;
private int _tagsBadQuality;
@@ -363,7 +391,7 @@ public class DataConnectionActor : UntypedActor, IWithStash, IWithTimers
_log.Info("[{0}] Entering Connected state", _connectionName);
_lastConnectedAt = DateTimeOffset.UtcNow;
_healthCollector.UpdateConnectionHealth(_connectionName, ConnectionHealth.Connected);
_healthCollector.UpdateTagResolution(_connectionName, _totalSubscribed, _resolvedTags);
_healthCollector.UpdateTagResolution(_connectionName, TotalSubscribedTagCount, ResolvedTagCount);
var endpointLabel = _backupConfig == null ? "Connected" : $"Connected to {_activeEndpoint.ToString().ToLower()}";
_healthCollector.UpdateConnectionEndpoint(_connectionName, endpointLabel);
Become(Connected);
@@ -1104,13 +1132,13 @@ public class DataConnectionActor : UntypedActor, IWithStash, IWithTimers
{
// The instance was unsubscribed while the
// subscribe I/O was in flight. Re-creating the per-instance entry and
// applying counter/handle mutations here would permanently leak state
// applying the handle mutations here would permanently leak state
// — _subscriptionsByInstance[instanceName] resurrected with no
// subscriber to receive callbacks, _tagSubscriberCount inflated forever
// (no future HandleUnsubscribe will drop it), and _totalSubscribed /
// _resolvedTags drifting above the real instance count across the
// adapter lifetime (also re-issued by ReSubscribeAll on every
// reconnect). Instead: drop all state mutations for this stale
// subscriber to receive callbacks, _instancesByTag re-indexed against
// an instance no future HandleUnsubscribe will ever visit, and the
// health counts derived from both reporting tags nobody subscribes to
// for the rest of the adapter lifetime (also re-issued by ReSubscribeAll
// on every reconnect). Instead: drop all state mutations for this stale
// message and release the adapter-level monitored items we just
// created so the device doesn't keep streaming change notifications
// for a tag nobody is subscribed to.
@@ -1167,15 +1195,14 @@ public class DataConnectionActor : UntypedActor, IWithStash, IWithTimers
if (!result.AlreadySubscribed)
_subscribesInFlight.Remove(result.TagPath);
// Only a tag newly added to THIS instance's set
// increments the reference count, so the count stays an accurate "number
// of distinct instances subscribed to this tag".
if (instanceTags.Add(result.TagPath))
{
_tagSubscriberCount[result.TagPath] =
_tagSubscriberCount.GetValueOrDefault(result.TagPath) + 1;
IndexTag(result.TagPath, instanceName);
}
// Register the tag against this instance in both directions. The per-tag
// counted set _instancesByTag is what makes the tag "subscribed" for every
// consumer — the fan-out, the last-subscriber test in HandleUnsubscribe, the
// in-flight-unsubscribe discard gates, and TotalSubscribedTagCount. Both
// adds are idempotent, so this runs unconditionally: there is no separate
// counter left that a repeated add could inflate.
instanceTags.Add(result.TagPath);
IndexTag(result.TagPath, instanceName);
// Re-check against current state: another subscribe may have resolved the
// same tag while this request's I/O was in flight.
@@ -1201,28 +1228,18 @@ public class DataConnectionActor : UntypedActor, IWithStash, IWithTimers
if (result.Success)
{
// Storing the handle IS the resolution: ResolvedTagCount is
// _subscriptionIds.Count, and the tag is already in the per-tag counted
// set, so it is already inside TotalSubscribedTagCount. The former
// fresh-subscribe vs. unresolved→resolved-promotion split existed only to
// decide which of the two scalar counters to bump (a promotion had to move
// _resolvedTags WITHOUT re-bumping _totalSubscribed, or a tag that a second
// instance resolved after a first instance failed it counted twice —
// DataConnectionLayer-020). Derived counts get both cases right for free;
// all that is left of the promotion is dropping the retry bookkeeping.
_subscriptionIds[result.TagPath] = result.SubscriptionId!;
// Distinguish fresh subscribe from
// unresolved → resolved promotion. If an earlier instance's
// subscribe for this tag had failed at the resolution layer
// (the tag was already added to _unresolvedTags AND already
// counted in _totalSubscribed), this success transitions it
// from unresolved to resolved — increment _resolvedTags ONLY.
// Incrementing _totalSubscribed again here would over-count by
// one until HandleTagResolutionSucceeded reconciled. Mirrors
// HandleTagResolutionSucceeded's promotion shape so both paths
// resolve a previously-failed tag identically.
if (_unresolvedTags.Remove(result.TagPath))
{
_resolutionInFlight.Remove(result.TagPath);
_resolvedTags++;
}
else
{
_totalSubscribed++;
_resolvedTags++;
}
}
else if (result.ConnectionLevelFailure)
{
@@ -1234,18 +1251,13 @@ public class DataConnectionActor : UntypedActor, IWithStash, IWithTimers
}
else
{
// Genuine tag resolution failure — mark unresolved so the
// periodic retry timer picks it up. Only increment _totalSubscribed
// when the tag is genuinely
// newly-tracked. A second instance failing to resolve a tag the
// first instance already added to _unresolvedTags is the same
// logical tag, counted once — bumping _totalSubscribed again
// would over-report TotalSubscribedTags forever.
var newlyUnresolved = _unresolvedTags.Add(result.TagPath);
if (newlyUnresolved)
{
_totalSubscribed++;
}
// Genuine tag resolution failure — mark unresolved so the periodic retry
// timer picks it up. The tag still counts toward TotalSubscribedTagCount
// (an instance subscribes to it, so it is in the per-tag counted set); it
// simply does not count as resolved. Two instances failing the SAME tag is
// still one logical tag in both counts, because both counts are set sizes
// rather than accumulated increments (DataConnectionLayer-020).
_unresolvedTags.Add(result.TagPath);
_log.Debug("[{0}] Tag resolution failed for {1}: {2}",
_connectionName, result.TagPath, result.Error);
@@ -1331,38 +1343,27 @@ public class DataConnectionActor : UntypedActor, IWithStash, IWithTimers
// Cleanup on Instance Actor stop
foreach (var tagPath in tags)
{
// Drop this instance from the fan-out reverse index (mirrors the
// _subscriptionsByInstance removal at the end of this method).
// Drop this instance from the per-tag counted set. UnindexTag removes the tag
// key entirely when its last subscriber leaves, so "the key is gone" IS the
// last-subscriber test — O(1), and with no parallel reference count that could
// ever disagree with it. TotalSubscribedTagCount drops by exactly the number of
// keys this loop retires, whatever state the tag was in (resolved, unresolved,
// or mid-reconnect with both maps cleared — the case the retired scalar
// counters silently failed to decrement at all).
UnindexTag(tagPath, request.InstanceUniqueName);
// Drop this instance's reference; the tag is only
// released at the adapter when no other instance still subscribes to it.
// The reference count makes this O(1) instead of an O(instances) scan.
var remaining = _tagSubscriberCount.GetValueOrDefault(tagPath) - 1;
if (remaining > 0)
{
_tagSubscriberCount[tagPath] = remaining;
if (_instancesByTag.ContainsKey(tagPath))
continue;
}
_tagSubscriberCount.Remove(tagPath);
// Last subscriber gone. A tag with a subscription id is a resolved tag;
// an unresolved tag never has a subscription id, so reaching this branch
// via TryGetValue means the tag was resolved — decrement _resolvedTags
// unconditionally (the previous `!_unresolvedTags.Contains` re-check after
// an unconditional Remove was always-true dead logic).
if (_subscriptionIds.TryGetValue(tagPath, out var subId))
// Last subscriber gone. A tag with a subscription id is a resolved tag and its
// adapter handle must be released; an unresolved tag never has one.
if (_subscriptionIds.Remove(tagPath, out var subId))
{
idsToRelease.Add(subId);
_subscriptionIds.Remove(tagPath);
_resolutionInFlight.Remove(tagPath);
_totalSubscribed--;
_resolvedTags--;
// Drop the tag's tracked quality so it is no
// longer counted by PushBadQualityForAllTags (which sets _tagsBadQuality
// from _lastTagQuality.Count). Leaving it here drifts the quality
// counters above _totalSubscribed across disconnect cycles.
// counters above the reported subscribed total across disconnect cycles.
if (_lastTagQuality.Remove(tagPath, out var droppedQuality))
{
switch (droppedQuality)
@@ -1373,16 +1374,13 @@ public class DataConnectionActor : UntypedActor, IWithStash, IWithTimers
}
}
}
else if (_unresolvedTags.Remove(tagPath))
{
// Last subscriber gone for a tag that had never resolved: stop
// retrying it and drop it from the subscribed total. The previous
// implementation never reached this case (its guard required a
// subscription id), so an unresolved tag leaked into the retry timer
// and TotalSubscribedTags forever after its instance unsubscribed.
_resolutionInFlight.Remove(tagPath);
_totalSubscribed--;
}
// Stop probing a tag nobody subscribes to any more. Unconditional and
// idempotent: a tag is in at most one of these sets (and in NEITHER during a
// reconnect window, where ReSubscribeAll has just cleared both), so there is
// no branch here to get wrong.
_unresolvedTags.Remove(tagPath);
_resolutionInFlight.Remove(tagPath);
}
_subscriptionsByInstance.Remove(request.InstanceUniqueName);
@@ -1395,7 +1393,7 @@ public class DataConnectionActor : UntypedActor, IWithStash, IWithTimers
// unsubscribed tags' buckets were decremented above. SYNCHRONOUS flush: the
// coalescing timer must never delay a count that just dropped.
FlushQualityCounters();
_healthCollector.UpdateTagResolution(_connectionName, _totalSubscribed, _resolvedTags);
_healthCollector.UpdateTagResolution(_connectionName, TotalSubscribedTagCount, ResolvedTagCount);
}
// ── Write Support ──
@@ -1905,9 +1903,11 @@ public class DataConnectionActor : UntypedActor, IWithStash, IWithTimers
/// <summary>
/// Applies one chunk of batch-subscribe results on the actor thread — the shared tail
/// of the reconnect re-subscribe and the tag-resolution probe. Both paths subscribe
/// tags that are ALREADY counted in <see cref="_totalSubscribed"/> (they come from
/// <see cref="_subscriptionsByInstance"/>), so neither touches that counter.
/// of the reconnect re-subscribe and the tag-resolution probe. Both paths re-subscribe
/// tags that some instance already subscribes to (they come from
/// <see cref="_subscriptionsByInstance"/>), so they only ever move a tag between
/// unresolved and resolved — <see cref="TotalSubscribedTagCount"/> is unaffected by
/// construction, because neither path adds to or removes from the per-tag counted set.
/// </summary>
private void HandleBatchSubscribeCompleted(BatchSubscribeCompleted msg)
{
@@ -1933,18 +1933,27 @@ public class DataConnectionActor : UntypedActor, IWithStash, IWithTimers
// The tag lost its last subscriber while this batch was in flight — an
// instance unsubscribe (disable/undeploy/redeploy) raced the resolution
// probe or the reconnect re-subscribe. _instancesByTag is the authority:
// it is the inverse of _subscriptionsByInstance and, like it, is preserved
// across reconnect, so an empty entry means nobody wants this tag any more.
// Applying the row anyway would store a handle no future unsubscribe can
// ever release (HandleUnsubscribe already ran for this tag) AND push
// _resolvedTags above the _totalSubscribed that same unsubscribe just
// decremented — permanently corrupting the health counters and driving
// _totalSubscribed negative on the next redeploy round trip. So discard
// every state mutation for this row and release the adapter handle the
// batch just created. This restores the gate the pre-merge per-tag handler
// had (it applied only when the tag was still in _unresolvedTags) and
// additionally releases the handle that path leaked.
// probe or the reconnect re-subscribe. _instancesByTag, the per-tag counted
// set, is the authority: it is the inverse of _subscriptionsByInstance and,
// like it, is preserved across reconnect, so a missing key means nobody
// wants this tag any more. Applying the row anyway would store a handle no
// future unsubscribe can ever release (HandleUnsubscribe already ran for
// this tag) AND — because ResolvedTagCount is _subscriptionIds.Count while
// TotalSubscribedTagCount is _instancesByTag.Count — report a resolved tag
// that is not in the subscribed total at all, i.e. resolved > total. So
// discard every state mutation for this row and release the adapter handle
// the batch just created. This restores the gate the pre-merge per-tag
// handler had (it applied only when the tag was still in _unresolvedTags)
// and additionally releases the handle that path leaked.
//
// Deriving the counts is what makes this gate SUFFICIENT rather than merely
// necessary. It used to be one of several places that had to agree on scalar
// counters mutated elsewhere, and the counters could still drift when the
// unsubscribe landed inside a reconnect window (ReSubscribeAll clears the
// maps HandleUnsubscribe's decrement branches keyed off, so neither branch
// fired and the total leaked +1 per churn cycle). Now the gate has exactly
// one job — do not store an orphan handle — and both counts follow from the
// collections it protects.
if (!_instancesByTag.ContainsKey(row.TagPath))
{
_log.Debug(
@@ -1965,7 +1974,6 @@ public class DataConnectionActor : UntypedActor, IWithStash, IWithTimers
else
{
_subscriptionIds[row.TagPath] = row.SubscriptionId;
_resolvedTags++;
anyResolved = true;
if (wasUnresolved)
_log.Info("[{0}] Tag resolved: {1}", _connectionName, row.TagPath);
@@ -1984,17 +1992,16 @@ public class DataConnectionActor : UntypedActor, IWithStash, IWithTimers
_connectionName, row.TagPath, row.Error);
// Same in-flight-unsubscribe race as the success branch above: the tag was
// already dropped from _unresolvedTags AND from _totalSubscribed by
// already dropped from _unresolvedTags and from the per-tag counted set by
// HandleUnsubscribe, so re-adding it here would probe a tag nobody
// subscribes to forever, at a retry count TotalSubscribedTags no longer
// accounts for.
// subscribes to, forever.
if (_instancesByTag.ContainsKey(row.TagPath))
_unresolvedTags.Add(row.TagPath);
}
}
_ = UnsubscribeIdsAsync(_adapter, idsToRelease);
_healthCollector.UpdateTagResolution(_connectionName, _totalSubscribed, _resolvedTags);
_healthCollector.UpdateTagResolution(_connectionName, TotalSubscribedTagCount, ResolvedTagCount);
// Backoff bookkeeping belongs to the probe round only: a reconnect re-subscribe
// chunk is not a "retry round" and must not double the interval.
@@ -2100,11 +2107,19 @@ public class DataConnectionActor : UntypedActor, IWithStash, IWithTimers
// the in-flight tracking; the stale completion simply has nothing to
// remove (idempotent HashSet.Remove on a missing key).
_subscribesInFlight.Clear();
_resolvedTags = 0;
// NOTE: clearing _subscriptionIds above IS the resolved-count reset —
// ResolvedTagCount is derived from it. The retired _resolvedTags scalar had to be
// zeroed here by hand; its sibling _totalSubscribed deliberately was NOT, because
// the tags stay subscribed across a reconnect. That asymmetry is what left an
// unsubscribe arriving inside this window matching neither of _totalSubscribed's
// decrement branches (both keyed off maps this method just cleared), leaking a
// phantom tag into the reported total for the lifetime of the actor. Both counts
// now follow from the collections themselves: _instancesByTag is untouched here, so
// the total correctly rides through the reconnect and still drops on unsubscribe.
// Reset the quality tracking too. Otherwise tags
// resolved for the first time after reconnect (never in _lastTagQuality) only
// increment their bucket and the totals drift above _totalSubscribed. They are
// increment their bucket and the totals drift above the subscribed total. They are
// repopulated from fresh TagValueReceived messages once subscriptions activate.
_lastTagQuality.Clear();
_tagsGoodQuality = 0;
@@ -2196,7 +2211,8 @@ public class DataConnectionActor : UntypedActor, IWithStash, IWithTimers
? "Primary (no backup)"
: _activeEndpoint.ToString();
Sender.Tell(new DataConnectionHealthReport(
_connectionName, status, _totalSubscribed, _resolvedTags, endpointLabel, DateTimeOffset.UtcNow));
_connectionName, status, TotalSubscribedTagCount, ResolvedTagCount,
endpointLabel, DateTimeOffset.UtcNow));
}
// ── Internal message handlers for piped async results ──