fix(dcl): derive tag-resolution health counts from per-tag authoritative state

Closes arch-review remediation residual #1 (DCL unsubscribe-during-reconnect
count staleness).

DataConnectionActor tracked TotalSubscribedTags/ResolvedTags as two int fields
incremented and decremented at five independent sites. ReSubscribeAll clears the
very maps those decrements key off (_subscriptionIds, _unresolvedTags) while
deliberately preserving _subscriptionsByInstance, so an unsubscribe landing
inside a reconnect window matched NEITHER decrement branch: the total leaked +1
per subscribe/reconnect/unsubscribe churn cycle, permanently and cumulatively.
The 37f13e2e discard gate stopped the orphan-handle half of that race; it could
not stop the counters drifting, because they were state of their own.

Both counts are now DERIVED at report time from the authoritative per-tag
collections, which makes the drift unrepresentable rather than merely guarded:

  total    = _instancesByTag.Count   (the per-tag counted set the residual
                                      called for — distinct tags with at least
                                      one subscribing instance)
  resolved = _subscriptionIds.Count  (tags for which the adapter holds a handle)

Two semantic corrections fall out of the derivation:

- A tag whose subscribe failed at CONNECTION level now counts toward the total.
  It was excluded before, yet the reconnect re-subscribe re-issued it from
  _subscriptionsByInstance and booked it as resolved — resolved above total, and
  a total driven negative by the eventual unsubscribe.
- _tagSubscriberCount is deleted. It duplicated _instancesByTag exactly, so
  HandleUnsubscribe's last-subscriber test is now "did UnindexTag drop the key?"
  — still O(1), with no parallel count that can disagree about when a handle is
  released. The subscribe-success promotion split (fresh vs. unresolved→resolved)
  also goes: it existed only to pick which scalar to bump; set sizes get
  DataConnectionLayer-020's double-count cases right for free.

Behavior is otherwise unchanged — same logging, same handle release, same
unresolved-tag probing, same in-flight-unsubscribe discard semantics (the long
comment block there is updated for the mechanics that changed).

Tests: five TagResolutionCounts_* cases in DataConnectionActorBatchTests
covering the churn repro (3 cycles), a shared tag losing one instance mid
reconnect, connection-level failure then recovery, plain subscribe/unsubscribe
cycles, and a completed reconnect re-subscribe. Verified failing against the
pre-fix actor (churn: total 1 not 0; connection-level: total 0 not 1) and
passing after. Full DCL suite 319/319; solution builds with 0 warnings.

Docs: Component-DataConnectionLayer.md health-reporting section describes the
derived counts; residuals register item 1 marked RESOLVED.
This commit is contained in:
Joseph Doherty
2026-08-15 02:05:35 -04:00
parent 986e6e7ad5
commit 491df111ea
5 changed files with 389 additions and 125 deletions
@@ -92,10 +92,15 @@ each instructed to try to refute the fixes. Confirmed findings landed as targete
Deliberately not fixed in this program — each has a stated reason, not an oversight:
1. **DCL unsubscribe-during-reconnect count staleness.** The `37f13e2e` fix discards orphaned
1. ~~**DCL unsubscribe-during-reconnect count staleness.** The `37f13e2e` fix discards orphaned
in-flight results but a per-connection counter can still drift under rapid
subscribe/unsubscribe churn during a reconnect; needs a per-tag counted set. Low severity,
cosmetic (a health-report number), deferred.
cosmetic (a health-report number), deferred.~~ **RESOLVED 2026-08-15**`DataConnectionActor`'s
`_totalSubscribed`/`_resolvedTags` scalars are deleted and both health counts are now DERIVED at
report time from the authoritative per-tag state (`_instancesByTag.Count`, the per-tag counted
set the residual called for, and `_subscriptionIds.Count`), so no accumulated counter exists to
drift; this also closes the connection-level-failure case that let resolved climb above total.
Regression tests: `TagResolutionCounts_*` in `DataConnectionActorBatchTests`.
2. **Per-table `needs_snapshot` in LocalDb.** Baselining one table currently re-streams every
registered table in both directions. Narrowing it needs an on-disk schema change LocalDb 0.2.1
deliberately avoided (wire/schema compatibility). Documented as a follow-up in the library's
@@ -414,7 +414,7 @@ Note: Pre-deployment validation at central does **not** verify that tag paths re
The DCL reports the following metrics to the Health Monitoring component via the existing periodic heartbeat:
- **Connection status**: `connected`, `disconnected`, or `reconnecting` per data connection.
- **Tag resolution counts**: Per connection, the number of total subscribed tags vs. successfully resolved tags. This gives operators visibility into misconfigured templates without needing to open the debug view for individual instances.
- **Tag resolution counts**: Per connection, the number of total subscribed tags vs. successfully resolved tags. This gives operators visibility into misconfigured templates without needing to open the debug view for individual instances. Both numbers are **derived at report time from the connection actor's authoritative per-tag state**, never accumulated in counters: the total is the number of distinct tag paths at least one instance currently subscribes to (the per-tag counted set that also drives value fan-out and the last-subscriber release decision), and the resolved count is the number of tags for which the adapter currently holds a subscription handle. A tag counts toward the total from the moment an instance registers it, whatever its resolution outcome — resolved, awaiting a resolution retry, or failed at connection level — so resolved can never exceed total. Deriving rather than accumulating is deliberate: increment/decrement counters drifted whenever an unsubscribe landed inside a reconnect window (the reconnect clears the per-tag maps the decrements keyed off), leaking a phantom tag into the reported total on every churn cycle.
- **Tag quality counters** are pushed on a genuine quality **transition** only, coalesced onto a `QualityFlushInterval` (1s) single-shot timer. Counter arithmetic still runs per message; only the collector push is deferred, and a value whose quality is unchanged moves no counter at all. Health reports poll at 30s, so the coalescing loses nothing. Three paths flush **synchronously** because their correctness depends on it: the bad-quality push on disconnect, unsubscribe, and the reconnect counter reset.
## Dependencies