fix(dcl): derive tag-resolution health counts from per-tag authoritative state
Closes arch-review remediation residual #1 (DCL unsubscribe-during-reconnect
count staleness).
DataConnectionActor tracked TotalSubscribedTags/ResolvedTags as two int fields
incremented and decremented at five independent sites. ReSubscribeAll clears the
very maps those decrements key off (_subscriptionIds, _unresolvedTags) while
deliberately preserving _subscriptionsByInstance, so an unsubscribe landing
inside a reconnect window matched NEITHER decrement branch: the total leaked +1
per subscribe/reconnect/unsubscribe churn cycle, permanently and cumulatively.
The 37f13e2e discard gate stopped the orphan-handle half of that race; it could
not stop the counters drifting, because they were state of their own.
Both counts are now DERIVED at report time from the authoritative per-tag
collections, which makes the drift unrepresentable rather than merely guarded:
total = _instancesByTag.Count (the per-tag counted set the residual
called for — distinct tags with at least
one subscribing instance)
resolved = _subscriptionIds.Count (tags for which the adapter holds a handle)
Two semantic corrections fall out of the derivation:
- A tag whose subscribe failed at CONNECTION level now counts toward the total.
It was excluded before, yet the reconnect re-subscribe re-issued it from
_subscriptionsByInstance and booked it as resolved — resolved above total, and
a total driven negative by the eventual unsubscribe.
- _tagSubscriberCount is deleted. It duplicated _instancesByTag exactly, so
HandleUnsubscribe's last-subscriber test is now "did UnindexTag drop the key?"
— still O(1), with no parallel count that can disagree about when a handle is
released. The subscribe-success promotion split (fresh vs. unresolved→resolved)
also goes: it existed only to pick which scalar to bump; set sizes get
DataConnectionLayer-020's double-count cases right for free.
Behavior is otherwise unchanged — same logging, same handle release, same
unresolved-tag probing, same in-flight-unsubscribe discard semantics (the long
comment block there is updated for the mechanics that changed).
Tests: five TagResolutionCounts_* cases in DataConnectionActorBatchTests
covering the churn repro (3 cycles), a shared tag losing one instance mid
reconnect, connection-level failure then recovery, plain subscribe/unsubscribe
cycles, and a completed reconnect re-subscribe. Verified failing against the
pre-fix actor (churn: total 1 not 0; connection-level: total 0 not 1) and
passing after. Full DCL suite 319/319; solution builds with 0 warnings.
Docs: Component-DataConnectionLayer.md health-reporting section describes the
derived counts; residuals register item 1 marked RESOLVED.
This commit is contained in:
@@ -92,10 +92,15 @@ each instructed to try to refute the fixes. Confirmed findings landed as targete
|
||||
|
||||
Deliberately not fixed in this program — each has a stated reason, not an oversight:
|
||||
|
||||
1. **DCL unsubscribe-during-reconnect count staleness.** The `37f13e2e` fix discards orphaned
|
||||
1. ~~**DCL unsubscribe-during-reconnect count staleness.** The `37f13e2e` fix discards orphaned
|
||||
in-flight results but a per-connection counter can still drift under rapid
|
||||
subscribe/unsubscribe churn during a reconnect; needs a per-tag counted set. Low severity,
|
||||
cosmetic (a health-report number), deferred.
|
||||
cosmetic (a health-report number), deferred.~~ **RESOLVED 2026-08-15** — `DataConnectionActor`'s
|
||||
`_totalSubscribed`/`_resolvedTags` scalars are deleted and both health counts are now DERIVED at
|
||||
report time from the authoritative per-tag state (`_instancesByTag.Count`, the per-tag counted
|
||||
set the residual called for, and `_subscriptionIds.Count`), so no accumulated counter exists to
|
||||
drift; this also closes the connection-level-failure case that let resolved climb above total.
|
||||
Regression tests: `TagResolutionCounts_*` in `DataConnectionActorBatchTests`.
|
||||
2. **Per-table `needs_snapshot` in LocalDb.** Baselining one table currently re-streams every
|
||||
registered table in both directions. Narrowing it needs an on-disk schema change LocalDb 0.2.1
|
||||
deliberately avoided (wire/schema compatibility). Documented as a follow-up in the library's
|
||||
|
||||
@@ -414,7 +414,7 @@ Note: Pre-deployment validation at central does **not** verify that tag paths re
|
||||
The DCL reports the following metrics to the Health Monitoring component via the existing periodic heartbeat:
|
||||
|
||||
- **Connection status**: `connected`, `disconnected`, or `reconnecting` per data connection.
|
||||
- **Tag resolution counts**: Per connection, the number of total subscribed tags vs. successfully resolved tags. This gives operators visibility into misconfigured templates without needing to open the debug view for individual instances.
|
||||
- **Tag resolution counts**: Per connection, the number of total subscribed tags vs. successfully resolved tags. This gives operators visibility into misconfigured templates without needing to open the debug view for individual instances. Both numbers are **derived at report time from the connection actor's authoritative per-tag state**, never accumulated in counters: the total is the number of distinct tag paths at least one instance currently subscribes to (the per-tag counted set that also drives value fan-out and the last-subscriber release decision), and the resolved count is the number of tags for which the adapter currently holds a subscription handle. A tag counts toward the total from the moment an instance registers it, whatever its resolution outcome — resolved, awaiting a resolution retry, or failed at connection level — so resolved can never exceed total. Deriving rather than accumulating is deliberate: increment/decrement counters drifted whenever an unsubscribe landed inside a reconnect window (the reconnect clears the per-tag maps the decrements keyed off), leaking a phantom tag into the reported total on every churn cycle.
|
||||
- **Tag quality counters** are pushed on a genuine quality **transition** only, coalesced onto a `QualityFlushInterval` (1s) single-shot timer. Counter arithmetic still runs per message; only the collector push is deferred, and a value whose quality is unchanged moves no counter at all. Health reports poll at 30s, so the coalescing loses nothing. Three paths flush **synchronously** because their correctness depends on it: the bad-quality push on disconnect, unsubscribe, and the reconnect counter reset.
|
||||
|
||||
## Dependencies
|
||||
|
||||
@@ -55,21 +55,25 @@ public class DataConnectionActor : UntypedActor, IWithStash, IWithTimers
|
||||
private readonly Dictionary<string, string> _subscriptionIds = new();
|
||||
|
||||
/// <summary>
|
||||
/// Reverse index of how many instances subscribe to each
|
||||
/// tag path. Lets <see cref="HandleUnsubscribe"/> decide whether any other instance
|
||||
/// still needs a tag in O(1) instead of scanning every instance's tag set.
|
||||
/// </summary>
|
||||
private readonly Dictionary<string, int> _tagSubscriberCount = new();
|
||||
|
||||
/// <summary>
|
||||
/// Reverse index of which instances subscribe to each tag path: tagPath → set of
|
||||
/// instanceUniqueName. Mirrors <see cref="_subscriptionsByInstance"/> inverted so the
|
||||
/// <see cref="HandleTagValueReceived"/> hot path fans a value out to exactly the
|
||||
/// interested instances in O(subscribers) instead of scanning every instance's tag set
|
||||
/// Per-tag COUNTED SET — the reverse index of which instances subscribe to each tag
|
||||
/// path: tagPath → set of instanceUniqueName. Mirrors <see cref="_subscriptionsByInstance"/>
|
||||
/// inverted so the <see cref="HandleTagValueReceived"/> hot path fans a value out to exactly
|
||||
/// the interested instances in O(subscribers) instead of scanning every instance's tag set
|
||||
/// on every tag update. Maintained via <see cref="IndexTag"/>/<see cref="UnindexTag"/>
|
||||
/// at every <see cref="_subscriptionsByInstance"/> tag-set mutation; a tag key is dropped
|
||||
/// when its instance set empties. Derived purely from <see cref="_subscriptionsByInstance"/>,
|
||||
/// so it is preserved across reconnect exactly as that map is (see <see cref="ReSubscribeAll"/>).
|
||||
///
|
||||
/// It is also THE authority for three other questions, so that no independently-mutated
|
||||
/// counter can drift away from it:
|
||||
/// <list type="bullet">
|
||||
/// <item>"does any instance still need this tag?" — <see cref="HandleUnsubscribe"/>'s
|
||||
/// last-subscriber test is simply "did <see cref="UnindexTag"/> drop the key?" (O(1),
|
||||
/// replacing a separate <c>_tagSubscriberCount</c> map that had to be kept in step);</item>
|
||||
/// <item>"was this tag unsubscribed while a batch subscribe was in flight?" — the discard
|
||||
/// gates in <see cref="HandleBatchSubscribeCompleted"/>;</item>
|
||||
/// <item>the reported <see cref="TotalSubscribedTagCount"/>.</item>
|
||||
/// </list>
|
||||
/// </summary>
|
||||
private readonly Dictionary<string, HashSet<string>> _instancesByTag = new();
|
||||
|
||||
@@ -130,10 +134,34 @@ public class DataConnectionActor : UntypedActor, IWithStash, IWithTimers
|
||||
private readonly HashSet<string> _alarmSubscribesInFlight = new();
|
||||
|
||||
/// <summary>
|
||||
/// Tracks total subscribed and resolved tags for health reporting.
|
||||
/// Total subscribed tags for health reporting: the number of DISTINCT tag paths that at
|
||||
/// least one instance currently subscribes to.
|
||||
///
|
||||
/// DERIVED, never accumulated. This used to be an <c>int</c> field incremented and
|
||||
/// decremented at five independent sites (initial subscribe success, initial subscribe
|
||||
/// resolution failure, both unsubscribe branches) while the reconnect path cleared the
|
||||
/// maps those decrements keyed off — so an unsubscribe that landed inside a reconnect
|
||||
/// window matched NEITHER decrement branch and leaked +1 per churn cycle, permanently.
|
||||
/// Reading the counted set at report time makes that class of drift unrepresentable:
|
||||
/// the number can only ever be what the authoritative per-tag state says it is.
|
||||
///
|
||||
/// A tag counts from the moment it is registered against an instance, whatever its
|
||||
/// resolution outcome — resolved, awaiting a resolution retry, or failed at connection
|
||||
/// level (that last case used to be excluded, which let <see cref="ResolvedTagCount"/>
|
||||
/// climb ABOVE the total once the reconnect re-subscribe resolved it).
|
||||
/// </summary>
|
||||
private int _totalSubscribed;
|
||||
private int _resolvedTags;
|
||||
private int TotalSubscribedTagCount => _instancesByTag.Count;
|
||||
|
||||
/// <summary>
|
||||
/// Resolved tags for health reporting — DERIVED, for the same reason as
|
||||
/// <see cref="TotalSubscribedTagCount"/>. A tag is resolved exactly when the adapter has
|
||||
/// handed back a subscription handle for it, so <see cref="_subscriptionIds"/> IS the
|
||||
/// count; the retired <c>_resolvedTags</c> field was incremented and decremented in
|
||||
/// lock-step with every mutation of that dictionary anyway. Never exceeds
|
||||
/// <see cref="TotalSubscribedTagCount"/>, because a handle is only ever stored for a tag
|
||||
/// that is present in <see cref="_instancesByTag"/> (enforced at every store site).
|
||||
/// </summary>
|
||||
private int ResolvedTagCount => _subscriptionIds.Count;
|
||||
|
||||
private int _tagsGoodQuality;
|
||||
private int _tagsBadQuality;
|
||||
@@ -363,7 +391,7 @@ public class DataConnectionActor : UntypedActor, IWithStash, IWithTimers
|
||||
_log.Info("[{0}] Entering Connected state", _connectionName);
|
||||
_lastConnectedAt = DateTimeOffset.UtcNow;
|
||||
_healthCollector.UpdateConnectionHealth(_connectionName, ConnectionHealth.Connected);
|
||||
_healthCollector.UpdateTagResolution(_connectionName, _totalSubscribed, _resolvedTags);
|
||||
_healthCollector.UpdateTagResolution(_connectionName, TotalSubscribedTagCount, ResolvedTagCount);
|
||||
var endpointLabel = _backupConfig == null ? "Connected" : $"Connected to {_activeEndpoint.ToString().ToLower()}";
|
||||
_healthCollector.UpdateConnectionEndpoint(_connectionName, endpointLabel);
|
||||
Become(Connected);
|
||||
@@ -1104,13 +1132,13 @@ public class DataConnectionActor : UntypedActor, IWithStash, IWithTimers
|
||||
{
|
||||
// The instance was unsubscribed while the
|
||||
// subscribe I/O was in flight. Re-creating the per-instance entry and
|
||||
// applying counter/handle mutations here would permanently leak state
|
||||
// applying the handle mutations here would permanently leak state
|
||||
// — _subscriptionsByInstance[instanceName] resurrected with no
|
||||
// subscriber to receive callbacks, _tagSubscriberCount inflated forever
|
||||
// (no future HandleUnsubscribe will drop it), and _totalSubscribed /
|
||||
// _resolvedTags drifting above the real instance count across the
|
||||
// adapter lifetime (also re-issued by ReSubscribeAll on every
|
||||
// reconnect). Instead: drop all state mutations for this stale
|
||||
// subscriber to receive callbacks, _instancesByTag re-indexed against
|
||||
// an instance no future HandleUnsubscribe will ever visit, and the
|
||||
// health counts derived from both reporting tags nobody subscribes to
|
||||
// for the rest of the adapter lifetime (also re-issued by ReSubscribeAll
|
||||
// on every reconnect). Instead: drop all state mutations for this stale
|
||||
// message and release the adapter-level monitored items we just
|
||||
// created so the device doesn't keep streaming change notifications
|
||||
// for a tag nobody is subscribed to.
|
||||
@@ -1167,15 +1195,14 @@ public class DataConnectionActor : UntypedActor, IWithStash, IWithTimers
|
||||
if (!result.AlreadySubscribed)
|
||||
_subscribesInFlight.Remove(result.TagPath);
|
||||
|
||||
// Only a tag newly added to THIS instance's set
|
||||
// increments the reference count, so the count stays an accurate "number
|
||||
// of distinct instances subscribed to this tag".
|
||||
if (instanceTags.Add(result.TagPath))
|
||||
{
|
||||
_tagSubscriberCount[result.TagPath] =
|
||||
_tagSubscriberCount.GetValueOrDefault(result.TagPath) + 1;
|
||||
// Register the tag against this instance in both directions. The per-tag
|
||||
// counted set _instancesByTag is what makes the tag "subscribed" for every
|
||||
// consumer — the fan-out, the last-subscriber test in HandleUnsubscribe, the
|
||||
// in-flight-unsubscribe discard gates, and TotalSubscribedTagCount. Both
|
||||
// adds are idempotent, so this runs unconditionally: there is no separate
|
||||
// counter left that a repeated add could inflate.
|
||||
instanceTags.Add(result.TagPath);
|
||||
IndexTag(result.TagPath, instanceName);
|
||||
}
|
||||
|
||||
// Re-check against current state: another subscribe may have resolved the
|
||||
// same tag while this request's I/O was in flight.
|
||||
@@ -1201,28 +1228,18 @@ public class DataConnectionActor : UntypedActor, IWithStash, IWithTimers
|
||||
|
||||
if (result.Success)
|
||||
{
|
||||
// Storing the handle IS the resolution: ResolvedTagCount is
|
||||
// _subscriptionIds.Count, and the tag is already in the per-tag counted
|
||||
// set, so it is already inside TotalSubscribedTagCount. The former
|
||||
// fresh-subscribe vs. unresolved→resolved-promotion split existed only to
|
||||
// decide which of the two scalar counters to bump (a promotion had to move
|
||||
// _resolvedTags WITHOUT re-bumping _totalSubscribed, or a tag that a second
|
||||
// instance resolved after a first instance failed it counted twice —
|
||||
// DataConnectionLayer-020). Derived counts get both cases right for free;
|
||||
// all that is left of the promotion is dropping the retry bookkeeping.
|
||||
_subscriptionIds[result.TagPath] = result.SubscriptionId!;
|
||||
|
||||
// Distinguish fresh subscribe from
|
||||
// unresolved → resolved promotion. If an earlier instance's
|
||||
// subscribe for this tag had failed at the resolution layer
|
||||
// (the tag was already added to _unresolvedTags AND already
|
||||
// counted in _totalSubscribed), this success transitions it
|
||||
// from unresolved to resolved — increment _resolvedTags ONLY.
|
||||
// Incrementing _totalSubscribed again here would over-count by
|
||||
// one until HandleTagResolutionSucceeded reconciled. Mirrors
|
||||
// HandleTagResolutionSucceeded's promotion shape so both paths
|
||||
// resolve a previously-failed tag identically.
|
||||
if (_unresolvedTags.Remove(result.TagPath))
|
||||
{
|
||||
_resolutionInFlight.Remove(result.TagPath);
|
||||
_resolvedTags++;
|
||||
}
|
||||
else
|
||||
{
|
||||
_totalSubscribed++;
|
||||
_resolvedTags++;
|
||||
}
|
||||
}
|
||||
else if (result.ConnectionLevelFailure)
|
||||
{
|
||||
@@ -1234,18 +1251,13 @@ public class DataConnectionActor : UntypedActor, IWithStash, IWithTimers
|
||||
}
|
||||
else
|
||||
{
|
||||
// Genuine tag resolution failure — mark unresolved so the
|
||||
// periodic retry timer picks it up. Only increment _totalSubscribed
|
||||
// when the tag is genuinely
|
||||
// newly-tracked. A second instance failing to resolve a tag the
|
||||
// first instance already added to _unresolvedTags is the same
|
||||
// logical tag, counted once — bumping _totalSubscribed again
|
||||
// would over-report TotalSubscribedTags forever.
|
||||
var newlyUnresolved = _unresolvedTags.Add(result.TagPath);
|
||||
if (newlyUnresolved)
|
||||
{
|
||||
_totalSubscribed++;
|
||||
}
|
||||
// Genuine tag resolution failure — mark unresolved so the periodic retry
|
||||
// timer picks it up. The tag still counts toward TotalSubscribedTagCount
|
||||
// (an instance subscribes to it, so it is in the per-tag counted set); it
|
||||
// simply does not count as resolved. Two instances failing the SAME tag is
|
||||
// still one logical tag in both counts, because both counts are set sizes
|
||||
// rather than accumulated increments (DataConnectionLayer-020).
|
||||
_unresolvedTags.Add(result.TagPath);
|
||||
_log.Debug("[{0}] Tag resolution failed for {1}: {2}",
|
||||
_connectionName, result.TagPath, result.Error);
|
||||
|
||||
@@ -1331,38 +1343,27 @@ public class DataConnectionActor : UntypedActor, IWithStash, IWithTimers
|
||||
// Cleanup on Instance Actor stop
|
||||
foreach (var tagPath in tags)
|
||||
{
|
||||
// Drop this instance from the fan-out reverse index (mirrors the
|
||||
// _subscriptionsByInstance removal at the end of this method).
|
||||
// Drop this instance from the per-tag counted set. UnindexTag removes the tag
|
||||
// key entirely when its last subscriber leaves, so "the key is gone" IS the
|
||||
// last-subscriber test — O(1), and with no parallel reference count that could
|
||||
// ever disagree with it. TotalSubscribedTagCount drops by exactly the number of
|
||||
// keys this loop retires, whatever state the tag was in (resolved, unresolved,
|
||||
// or mid-reconnect with both maps cleared — the case the retired scalar
|
||||
// counters silently failed to decrement at all).
|
||||
UnindexTag(tagPath, request.InstanceUniqueName);
|
||||
|
||||
// Drop this instance's reference; the tag is only
|
||||
// released at the adapter when no other instance still subscribes to it.
|
||||
// The reference count makes this O(1) instead of an O(instances) scan.
|
||||
var remaining = _tagSubscriberCount.GetValueOrDefault(tagPath) - 1;
|
||||
if (remaining > 0)
|
||||
{
|
||||
_tagSubscriberCount[tagPath] = remaining;
|
||||
if (_instancesByTag.ContainsKey(tagPath))
|
||||
continue;
|
||||
}
|
||||
_tagSubscriberCount.Remove(tagPath);
|
||||
|
||||
// Last subscriber gone. A tag with a subscription id is a resolved tag;
|
||||
// an unresolved tag never has a subscription id, so reaching this branch
|
||||
// via TryGetValue means the tag was resolved — decrement _resolvedTags
|
||||
// unconditionally (the previous `!_unresolvedTags.Contains` re-check after
|
||||
// an unconditional Remove was always-true dead logic).
|
||||
if (_subscriptionIds.TryGetValue(tagPath, out var subId))
|
||||
// Last subscriber gone. A tag with a subscription id is a resolved tag and its
|
||||
// adapter handle must be released; an unresolved tag never has one.
|
||||
if (_subscriptionIds.Remove(tagPath, out var subId))
|
||||
{
|
||||
idsToRelease.Add(subId);
|
||||
_subscriptionIds.Remove(tagPath);
|
||||
_resolutionInFlight.Remove(tagPath);
|
||||
_totalSubscribed--;
|
||||
_resolvedTags--;
|
||||
|
||||
// Drop the tag's tracked quality so it is no
|
||||
// longer counted by PushBadQualityForAllTags (which sets _tagsBadQuality
|
||||
// from _lastTagQuality.Count). Leaving it here drifts the quality
|
||||
// counters above _totalSubscribed across disconnect cycles.
|
||||
// counters above the reported subscribed total across disconnect cycles.
|
||||
if (_lastTagQuality.Remove(tagPath, out var droppedQuality))
|
||||
{
|
||||
switch (droppedQuality)
|
||||
@@ -1373,16 +1374,13 @@ public class DataConnectionActor : UntypedActor, IWithStash, IWithTimers
|
||||
}
|
||||
}
|
||||
}
|
||||
else if (_unresolvedTags.Remove(tagPath))
|
||||
{
|
||||
// Last subscriber gone for a tag that had never resolved: stop
|
||||
// retrying it and drop it from the subscribed total. The previous
|
||||
// implementation never reached this case (its guard required a
|
||||
// subscription id), so an unresolved tag leaked into the retry timer
|
||||
// and TotalSubscribedTags forever after its instance unsubscribed.
|
||||
|
||||
// Stop probing a tag nobody subscribes to any more. Unconditional and
|
||||
// idempotent: a tag is in at most one of these sets (and in NEITHER during a
|
||||
// reconnect window, where ReSubscribeAll has just cleared both), so there is
|
||||
// no branch here to get wrong.
|
||||
_unresolvedTags.Remove(tagPath);
|
||||
_resolutionInFlight.Remove(tagPath);
|
||||
_totalSubscribed--;
|
||||
}
|
||||
}
|
||||
|
||||
_subscriptionsByInstance.Remove(request.InstanceUniqueName);
|
||||
@@ -1395,7 +1393,7 @@ public class DataConnectionActor : UntypedActor, IWithStash, IWithTimers
|
||||
// unsubscribed tags' buckets were decremented above. SYNCHRONOUS flush: the
|
||||
// coalescing timer must never delay a count that just dropped.
|
||||
FlushQualityCounters();
|
||||
_healthCollector.UpdateTagResolution(_connectionName, _totalSubscribed, _resolvedTags);
|
||||
_healthCollector.UpdateTagResolution(_connectionName, TotalSubscribedTagCount, ResolvedTagCount);
|
||||
}
|
||||
|
||||
// ── Write Support ──
|
||||
@@ -1905,9 +1903,11 @@ public class DataConnectionActor : UntypedActor, IWithStash, IWithTimers
|
||||
|
||||
/// <summary>
|
||||
/// Applies one chunk of batch-subscribe results on the actor thread — the shared tail
|
||||
/// of the reconnect re-subscribe and the tag-resolution probe. Both paths subscribe
|
||||
/// tags that are ALREADY counted in <see cref="_totalSubscribed"/> (they come from
|
||||
/// <see cref="_subscriptionsByInstance"/>), so neither touches that counter.
|
||||
/// of the reconnect re-subscribe and the tag-resolution probe. Both paths re-subscribe
|
||||
/// tags that some instance already subscribes to (they come from
|
||||
/// <see cref="_subscriptionsByInstance"/>), so they only ever move a tag between
|
||||
/// unresolved and resolved — <see cref="TotalSubscribedTagCount"/> is unaffected by
|
||||
/// construction, because neither path adds to or removes from the per-tag counted set.
|
||||
/// </summary>
|
||||
private void HandleBatchSubscribeCompleted(BatchSubscribeCompleted msg)
|
||||
{
|
||||
@@ -1933,18 +1933,27 @@ public class DataConnectionActor : UntypedActor, IWithStash, IWithTimers
|
||||
|
||||
// The tag lost its last subscriber while this batch was in flight — an
|
||||
// instance unsubscribe (disable/undeploy/redeploy) raced the resolution
|
||||
// probe or the reconnect re-subscribe. _instancesByTag is the authority:
|
||||
// it is the inverse of _subscriptionsByInstance and, like it, is preserved
|
||||
// across reconnect, so an empty entry means nobody wants this tag any more.
|
||||
// Applying the row anyway would store a handle no future unsubscribe can
|
||||
// ever release (HandleUnsubscribe already ran for this tag) AND push
|
||||
// _resolvedTags above the _totalSubscribed that same unsubscribe just
|
||||
// decremented — permanently corrupting the health counters and driving
|
||||
// _totalSubscribed negative on the next redeploy round trip. So discard
|
||||
// every state mutation for this row and release the adapter handle the
|
||||
// batch just created. This restores the gate the pre-merge per-tag handler
|
||||
// had (it applied only when the tag was still in _unresolvedTags) and
|
||||
// additionally releases the handle that path leaked.
|
||||
// probe or the reconnect re-subscribe. _instancesByTag, the per-tag counted
|
||||
// set, is the authority: it is the inverse of _subscriptionsByInstance and,
|
||||
// like it, is preserved across reconnect, so a missing key means nobody
|
||||
// wants this tag any more. Applying the row anyway would store a handle no
|
||||
// future unsubscribe can ever release (HandleUnsubscribe already ran for
|
||||
// this tag) AND — because ResolvedTagCount is _subscriptionIds.Count while
|
||||
// TotalSubscribedTagCount is _instancesByTag.Count — report a resolved tag
|
||||
// that is not in the subscribed total at all, i.e. resolved > total. So
|
||||
// discard every state mutation for this row and release the adapter handle
|
||||
// the batch just created. This restores the gate the pre-merge per-tag
|
||||
// handler had (it applied only when the tag was still in _unresolvedTags)
|
||||
// and additionally releases the handle that path leaked.
|
||||
//
|
||||
// Deriving the counts is what makes this gate SUFFICIENT rather than merely
|
||||
// necessary. It used to be one of several places that had to agree on scalar
|
||||
// counters mutated elsewhere, and the counters could still drift when the
|
||||
// unsubscribe landed inside a reconnect window (ReSubscribeAll clears the
|
||||
// maps HandleUnsubscribe's decrement branches keyed off, so neither branch
|
||||
// fired and the total leaked +1 per churn cycle). Now the gate has exactly
|
||||
// one job — do not store an orphan handle — and both counts follow from the
|
||||
// collections it protects.
|
||||
if (!_instancesByTag.ContainsKey(row.TagPath))
|
||||
{
|
||||
_log.Debug(
|
||||
@@ -1965,7 +1974,6 @@ public class DataConnectionActor : UntypedActor, IWithStash, IWithTimers
|
||||
else
|
||||
{
|
||||
_subscriptionIds[row.TagPath] = row.SubscriptionId;
|
||||
_resolvedTags++;
|
||||
anyResolved = true;
|
||||
if (wasUnresolved)
|
||||
_log.Info("[{0}] Tag resolved: {1}", _connectionName, row.TagPath);
|
||||
@@ -1984,17 +1992,16 @@ public class DataConnectionActor : UntypedActor, IWithStash, IWithTimers
|
||||
_connectionName, row.TagPath, row.Error);
|
||||
|
||||
// Same in-flight-unsubscribe race as the success branch above: the tag was
|
||||
// already dropped from _unresolvedTags AND from _totalSubscribed by
|
||||
// already dropped from _unresolvedTags and from the per-tag counted set by
|
||||
// HandleUnsubscribe, so re-adding it here would probe a tag nobody
|
||||
// subscribes to forever, at a retry count TotalSubscribedTags no longer
|
||||
// accounts for.
|
||||
// subscribes to, forever.
|
||||
if (_instancesByTag.ContainsKey(row.TagPath))
|
||||
_unresolvedTags.Add(row.TagPath);
|
||||
}
|
||||
}
|
||||
|
||||
_ = UnsubscribeIdsAsync(_adapter, idsToRelease);
|
||||
_healthCollector.UpdateTagResolution(_connectionName, _totalSubscribed, _resolvedTags);
|
||||
_healthCollector.UpdateTagResolution(_connectionName, TotalSubscribedTagCount, ResolvedTagCount);
|
||||
|
||||
// Backoff bookkeeping belongs to the probe round only: a reconnect re-subscribe
|
||||
// chunk is not a "retry round" and must not double the interval.
|
||||
@@ -2100,11 +2107,19 @@ public class DataConnectionActor : UntypedActor, IWithStash, IWithTimers
|
||||
// the in-flight tracking; the stale completion simply has nothing to
|
||||
// remove (idempotent HashSet.Remove on a missing key).
|
||||
_subscribesInFlight.Clear();
|
||||
_resolvedTags = 0;
|
||||
// NOTE: clearing _subscriptionIds above IS the resolved-count reset —
|
||||
// ResolvedTagCount is derived from it. The retired _resolvedTags scalar had to be
|
||||
// zeroed here by hand; its sibling _totalSubscribed deliberately was NOT, because
|
||||
// the tags stay subscribed across a reconnect. That asymmetry is what left an
|
||||
// unsubscribe arriving inside this window matching neither of _totalSubscribed's
|
||||
// decrement branches (both keyed off maps this method just cleared), leaking a
|
||||
// phantom tag into the reported total for the lifetime of the actor. Both counts
|
||||
// now follow from the collections themselves: _instancesByTag is untouched here, so
|
||||
// the total correctly rides through the reconnect and still drops on unsubscribe.
|
||||
|
||||
// Reset the quality tracking too. Otherwise tags
|
||||
// resolved for the first time after reconnect (never in _lastTagQuality) only
|
||||
// increment their bucket and the totals drift above _totalSubscribed. They are
|
||||
// increment their bucket and the totals drift above the subscribed total. They are
|
||||
// repopulated from fresh TagValueReceived messages once subscriptions activate.
|
||||
_lastTagQuality.Clear();
|
||||
_tagsGoodQuality = 0;
|
||||
@@ -2196,7 +2211,8 @@ public class DataConnectionActor : UntypedActor, IWithStash, IWithTimers
|
||||
? "Primary (no backup)"
|
||||
: _activeEndpoint.ToString();
|
||||
Sender.Tell(new DataConnectionHealthReport(
|
||||
_connectionName, status, _totalSubscribed, _resolvedTags, endpointLabel, DateTimeOffset.UtcNow));
|
||||
_connectionName, status, TotalSubscribedTagCount, ResolvedTagCount,
|
||||
endpointLabel, DateTimeOffset.UtcNow));
|
||||
}
|
||||
|
||||
// ── Internal message handlers for piped async results ──
|
||||
|
||||
+238
@@ -417,6 +417,244 @@ public class DataConnectionActorBatchTests : TestKit
|
||||
}, TimeSpan.FromSeconds(5));
|
||||
}
|
||||
|
||||
// ── Tag-resolution health counts: derived from the per-tag counted set ──
|
||||
//
|
||||
// Arch-review remediation residual #1. TotalSubscribedTags / ResolvedTags used to be
|
||||
// two int fields incremented and decremented at five independent sites. The reconnect
|
||||
// path cleared the very maps those decrements keyed off (_subscriptionIds and
|
||||
// _unresolvedTags), so an unsubscribe landing inside a reconnect window matched NEITHER
|
||||
// decrement branch and leaked a phantom tag into the total — permanently, and cumulative
|
||||
// under churn. They are now derived at report time from _instancesByTag (the per-tag
|
||||
// counted set) and _subscriptionIds, which makes that class of drift unrepresentable.
|
||||
|
||||
[Fact]
|
||||
public void TagResolutionCounts_UnsubscribeInsideAReconnectWindow_DoesNotLeakASubscribedTag()
|
||||
{
|
||||
// THE residual repro. Each cycle: subscribe → drop the connection → hold the
|
||||
// reconnect re-subscribe in flight → unsubscribe inside that window → let the batch
|
||||
// land. Pre-fix the total read 1 after the first cycle's unsubscribe (nothing
|
||||
// decremented it) and then 2, 3 … at the head of each following cycle — a +1 leak
|
||||
// per churn round that never recovers. Post-fix every cycle books exactly one tag
|
||||
// in and one tag out.
|
||||
var options = Options();
|
||||
options.SubscribeBatchSize = 10;
|
||||
|
||||
var adapter = new FakeBatchDataConnection();
|
||||
var actor = CreateActor(adapter, options, "counts-reconnect-churn");
|
||||
|
||||
for (var cycle = 1; cycle <= 3; cycle++)
|
||||
{
|
||||
actor.Tell(new SubscribeTagsRequest(
|
||||
$"s{cycle}", "inst1", "counts-reconnect-churn", ["tag1"], DateTimeOffset.UtcNow));
|
||||
FishForMessage<SubscribeTagsResponse>(m => m.Success, TimeSpan.FromSeconds(10));
|
||||
|
||||
var afterSubscribe = Health(actor);
|
||||
Assert.Equal(1, afterSubscribe.TotalSubscribedTags);
|
||||
Assert.Equal(1, afterSubscribe.ResolvedTags);
|
||||
|
||||
// Park the reconnect re-subscribe: ReSubscribeAll has already cleared
|
||||
// _subscriptionIds and _unresolvedTags, and the batch has not come back yet.
|
||||
var gate = new TaskCompletionSource(TaskCreationOptions.RunContinuationsAsynchronously);
|
||||
adapter.SubscribeGate = gate.Task;
|
||||
|
||||
var beforeReconnect = adapter.SubscribeBatches.Count;
|
||||
adapter.RaiseDisconnected();
|
||||
AwaitCondition(() => adapter.SubscribeBatches.Count > beforeReconnect, TimeSpan.FromSeconds(10));
|
||||
|
||||
// Unsubscribe INSIDE the reconnect window (same-sender ordering makes the
|
||||
// health report a barrier proving it was applied first).
|
||||
actor.Tell(new UnsubscribeTagsRequest(
|
||||
$"u{cycle}", "inst1", "counts-reconnect-churn", DateTimeOffset.UtcNow));
|
||||
|
||||
var afterUnsubscribe = Health(actor);
|
||||
Assert.Equal(0, afterUnsubscribe.TotalSubscribedTags);
|
||||
Assert.Equal(0, afterUnsubscribe.ResolvedTags);
|
||||
|
||||
// Release the raced batch. Its rows are discarded and its handle released —
|
||||
// one release per cycle, which is also the barrier that the discard ran.
|
||||
gate.SetResult();
|
||||
adapter.SubscribeGate = null;
|
||||
AwaitCondition(() => adapter.UnsubscribeBatches.Count >= cycle, TimeSpan.FromSeconds(10));
|
||||
|
||||
var afterDiscard = Health(actor);
|
||||
Assert.Equal(0, afterDiscard.TotalSubscribedTags);
|
||||
Assert.Equal(0, afterDiscard.ResolvedTags);
|
||||
}
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void TagResolutionCounts_SharedTagUnsubscribedInsideAReconnectWindow_StillCountsForTheOtherInstance()
|
||||
{
|
||||
// The counted-set semantics, not a bare reference count: one instance leaving during
|
||||
// a reconnect window must leave the tag counted (the other instance still subscribes)
|
||||
// and must not release its handle.
|
||||
var options = Options();
|
||||
options.SubscribeBatchSize = 10;
|
||||
|
||||
var adapter = new FakeBatchDataConnection();
|
||||
var actor = CreateActor(adapter, options, "counts-shared-reconnect");
|
||||
|
||||
actor.Tell(new SubscribeTagsRequest("s1", "instA", "counts-shared-reconnect", ["tag1"], DateTimeOffset.UtcNow));
|
||||
FishForMessage<SubscribeTagsResponse>(m => m.Success, TimeSpan.FromSeconds(10));
|
||||
actor.Tell(new SubscribeTagsRequest("s2", "instB", "counts-shared-reconnect", ["tag1"], DateTimeOffset.UtcNow));
|
||||
FishForMessage<SubscribeTagsResponse>(m => m.Success, TimeSpan.FromSeconds(10));
|
||||
|
||||
var shared = Health(actor);
|
||||
Assert.Equal(1, shared.TotalSubscribedTags);
|
||||
Assert.Equal(1, shared.ResolvedTags);
|
||||
|
||||
var gate = new TaskCompletionSource(TaskCreationOptions.RunContinuationsAsynchronously);
|
||||
adapter.SubscribeGate = gate.Task;
|
||||
|
||||
var beforeReconnect = adapter.SubscribeBatches.Count;
|
||||
adapter.RaiseDisconnected();
|
||||
AwaitCondition(() => adapter.SubscribeBatches.Count > beforeReconnect, TimeSpan.FromSeconds(10));
|
||||
|
||||
actor.Tell(new UnsubscribeTagsRequest("u1", "instA", "counts-shared-reconnect", DateTimeOffset.UtcNow));
|
||||
var afterOneLeft = Health(actor);
|
||||
Assert.Equal(1, afterOneLeft.TotalSubscribedTags);
|
||||
Assert.Equal(0, afterOneLeft.ResolvedTags); // ReSubscribeAll cleared the handles
|
||||
|
||||
// instB still wants the tag, so the raced batch row is APPLIED (not discarded) and
|
||||
// nothing is released back to the adapter.
|
||||
gate.SetResult();
|
||||
adapter.SubscribeGate = null;
|
||||
AwaitAssert(() =>
|
||||
{
|
||||
var report = Health(actor);
|
||||
Assert.Equal(1, report.TotalSubscribedTags);
|
||||
Assert.Equal(1, report.ResolvedTags);
|
||||
}, TimeSpan.FromSeconds(10));
|
||||
Assert.Empty(adapter.UnsubscribeBatches);
|
||||
|
||||
// The last subscriber leaving drops the tag from both counts and releases it.
|
||||
actor.Tell(new UnsubscribeTagsRequest("u2", "instB", "counts-shared-reconnect", DateTimeOffset.UtcNow));
|
||||
var afterBothLeft = Health(actor);
|
||||
Assert.Equal(0, afterBothLeft.TotalSubscribedTags);
|
||||
Assert.Equal(0, afterBothLeft.ResolvedTags);
|
||||
AwaitCondition(() => adapter.UnsubscribeBatches.Any(b => b.Count == 1), TimeSpan.FromSeconds(10));
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void TagResolutionCounts_ConnectionLevelFailureThenRecovery_KeepResolvedWithinTheTotal()
|
||||
{
|
||||
// A tag whose FIRST subscribe fails at connection level is subscribed by an instance
|
||||
// (it is re-issued from _subscriptionsByInstance on reconnect) but is neither
|
||||
// resolved nor unresolved. The retired scalar counted it in neither, so once the
|
||||
// reconnect re-subscribe resolved it the report read ResolvedTags=1 against
|
||||
// TotalSubscribedTags=0 — resolved above total, and a total driven negative by the
|
||||
// eventual unsubscribe.
|
||||
var options = Options();
|
||||
options.SubscribeBatchSize = 10;
|
||||
|
||||
var calls = 0;
|
||||
var adapter = new FakeBatchDataConnection
|
||||
{
|
||||
// Fail the initial subscribe at connection level; let the reconnect through.
|
||||
BatchSubscribeThrows = () => Interlocked.Increment(ref calls) == 1
|
||||
? new InvalidOperationException("client is not connected")
|
||||
: null
|
||||
};
|
||||
var actor = CreateActor(adapter, options, "counts-connection-level");
|
||||
|
||||
actor.Tell(new SubscribeTagsRequest(
|
||||
"s1", "inst1", "counts-connection-level", ["tag1"], DateTimeOffset.UtcNow));
|
||||
FishForMessage<SubscribeTagsResponse>(m => !m.Success, TimeSpan.FromSeconds(10));
|
||||
|
||||
// The tag is subscribed by the instance from the moment it is registered, whatever
|
||||
// the adapter said — so it counts toward the total and not toward resolved.
|
||||
var afterFault = Health(actor);
|
||||
Assert.Equal(1, afterFault.TotalSubscribedTags);
|
||||
Assert.Equal(0, afterFault.ResolvedTags);
|
||||
|
||||
// The reconnect re-subscribe resolves it: 1/1, never 1/0-then-resolved-above-total.
|
||||
AwaitAssert(() =>
|
||||
{
|
||||
var report = Health(actor);
|
||||
Assert.Equal(1, report.TotalSubscribedTags);
|
||||
Assert.Equal(1, report.ResolvedTags);
|
||||
}, TimeSpan.FromSeconds(15));
|
||||
|
||||
actor.Tell(new UnsubscribeTagsRequest("u1", "inst1", "counts-connection-level", DateTimeOffset.UtcNow));
|
||||
var afterUnsubscribe = Health(actor);
|
||||
Assert.Equal(0, afterUnsubscribe.TotalSubscribedTags);
|
||||
Assert.Equal(0, afterUnsubscribe.ResolvedTags);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void TagResolutionCounts_SurvivePlainSubscribeUnsubscribeCycles()
|
||||
{
|
||||
// The undramatic path, pinned so the derived counts cannot regress on it: repeated
|
||||
// deploy/undeploy round trips with a mix of resolved and unresolved tags always read
|
||||
// back the same numbers, with no accumulation across cycles.
|
||||
var options = Options();
|
||||
options.SubscribeBatchSize = 10;
|
||||
// Long enough that a resolution probe cannot resolve tag4 mid-assertion.
|
||||
options.TagResolutionRetryInterval = TimeSpan.FromSeconds(30);
|
||||
options.TagResolutionRetryMaxInterval = TimeSpan.FromSeconds(60);
|
||||
|
||||
var adapter = new FakeBatchDataConnection();
|
||||
adapter.FailingTags.Add("tag4");
|
||||
var actor = CreateActor(adapter, options, "counts-plain-cycles");
|
||||
|
||||
for (var cycle = 1; cycle <= 3; cycle++)
|
||||
{
|
||||
actor.Tell(new SubscribeTagsRequest(
|
||||
$"s{cycle}", "inst1", "counts-plain-cycles", Tags(4), DateTimeOffset.UtcNow));
|
||||
FishForMessage<SubscribeTagsResponse>(m => m.Success, TimeSpan.FromSeconds(10));
|
||||
|
||||
var afterSubscribe = Health(actor);
|
||||
Assert.Equal(4, afterSubscribe.TotalSubscribedTags); // 3 resolved + 1 unresolved
|
||||
Assert.Equal(3, afterSubscribe.ResolvedTags);
|
||||
|
||||
actor.Tell(new UnsubscribeTagsRequest(
|
||||
$"u{cycle}", "inst1", "counts-plain-cycles", DateTimeOffset.UtcNow));
|
||||
|
||||
var afterUnsubscribe = Health(actor);
|
||||
Assert.Equal(0, afterUnsubscribe.TotalSubscribedTags);
|
||||
Assert.Equal(0, afterUnsubscribe.ResolvedTags);
|
||||
}
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void TagResolutionCounts_AreCarriedThroughACompletedReconnectReSubscribe()
|
||||
{
|
||||
// A completed reconnect re-subscribe restores the resolved count and leaves the
|
||||
// total exactly where it was — the tags never stopped being subscribed.
|
||||
var options = Options();
|
||||
options.SubscribeBatchSize = 10;
|
||||
|
||||
var adapter = new FakeBatchDataConnection();
|
||||
adapter.FailingTags.Add("tag5");
|
||||
var actor = CreateActor(adapter, options, "counts-reconnect-restore");
|
||||
|
||||
actor.Tell(new SubscribeTagsRequest(
|
||||
"s1", "inst1", "counts-reconnect-restore", Tags(5), DateTimeOffset.UtcNow));
|
||||
FishForMessage<SubscribeTagsResponse>(m => m.Success, TimeSpan.FromSeconds(10));
|
||||
|
||||
var beforeReconnect = Health(actor);
|
||||
Assert.Equal(5, beforeReconnect.TotalSubscribedTags);
|
||||
Assert.Equal(4, beforeReconnect.ResolvedTags);
|
||||
|
||||
adapter.RaiseDisconnected();
|
||||
|
||||
AwaitAssert(() =>
|
||||
{
|
||||
var report = Health(actor);
|
||||
Assert.Equal(5, report.TotalSubscribedTags);
|
||||
Assert.Equal(4, report.ResolvedTags);
|
||||
}, TimeSpan.FromSeconds(15));
|
||||
|
||||
// The still-unresolved tag resolving on a later probe moves ONLY the resolved count.
|
||||
adapter.FailingTags.Clear();
|
||||
AwaitAssert(() =>
|
||||
{
|
||||
var report = Health(actor);
|
||||
Assert.Equal(5, report.TotalSubscribedTags);
|
||||
Assert.Equal(5, report.ResolvedTags);
|
||||
}, TimeSpan.FromSeconds(15));
|
||||
}
|
||||
|
||||
/// <summary>Point-in-time counters straight off the actor, fished past any pending pushes.</summary>
|
||||
private DataConnectionHealthReport Health(IActorRef actor)
|
||||
{
|
||||
|
||||
+9
-4
@@ -31,8 +31,13 @@ public sealed class FakeBatchDataConnection
|
||||
|
||||
/// <summary>Tags reported as failed rows (per-tag resolution failure).</summary>
|
||||
public readonly HashSet<string> FailingTags = new(StringComparer.Ordinal);
|
||||
/// <summary>When set, every batch subscribe throws this — a batch-level fault.</summary>
|
||||
public Func<Exception>? BatchSubscribeThrows;
|
||||
/// <summary>
|
||||
/// When set, a batch subscribe throws whatever this returns — a batch-level fault.
|
||||
/// Returning <c>null</c> lets that call through, so a test can make the fault TRANSIENT
|
||||
/// (e.g. fail the initial subscribe at connection level, then let the reconnect
|
||||
/// re-subscribe succeed).
|
||||
/// </summary>
|
||||
public Func<Exception?>? BatchSubscribeThrows;
|
||||
/// <summary>When true, bulk reads never return until the caller's token cancels.</summary>
|
||||
public bool HangReads;
|
||||
/// <summary>
|
||||
@@ -81,8 +86,8 @@ public sealed class FakeBatchDataConnection
|
||||
SubscribeBatchTimes.Enqueue(DateTimeOffset.UtcNow);
|
||||
ValueCallback = callback;
|
||||
|
||||
if (BatchSubscribeThrows is { } factory)
|
||||
throw factory();
|
||||
if (BatchSubscribeThrows?.Invoke() is { } fault)
|
||||
throw fault;
|
||||
|
||||
if (SubscribeGate is { } gate)
|
||||
await gate;
|
||||
|
||||
Reference in New Issue
Block a user