fix(dashboard): clamped CAS retry loop in snapshot hub connection counter + direct counter tests

Decrement was decrement-first with a single non-retried repair CAS. From zero,
two unmatched decrements (SignalR calls OnDisconnectedAsync for a connection
whose OnConnectedAsync faulted) capture -1 and -2; a real Increment then makes
the count -1, and the first decrementer's stale CompareExchange(0, -1) matches
and resets to zero — erasing a live connection, so the idle gate freezes an open
dashboard. The same lost race also made Decrement report 0 when it had not
written 0.

Clamping now happens inside the compare-and-swap: read, clamp, publish, retry on
loss. A lost race re-reads the fresh value instead of repairing a stale one.

The counter moves to its own file per the one-public-type-per-file convention and
gains direct tests: the zero floor under concurrent unmatched decrements, matched
pairs settling at zero, and an interleaved connect/disconnect stress round. The
stress test asserts the observable invariants only — the specific interleaving
cannot be forced through the public API (verified: the previous implementation
passes it), which its remarks now state rather than implying a reproducer. A hub
wiring test is skipped for the EventsHub reason: driving Hub.OnConnectedAsync
needs caller-clients and connection-context fakes, and the overrides are two
lines of delegation to the tested type.

Also documents that the API-key refresh's pre-gate time check races benignly.
This commit is contained in:
Joseph Doherty
2026-08-15 12:32:58 -04:00
parent 44ca7c8623
commit f920b4cbf5
4 changed files with 209 additions and 40 deletions
@@ -294,6 +294,9 @@ public sealed class DashboardSnapshotService : IDashboardSnapshotService
// Inside the refresh window: reuse the cached summaries rather than
// re-reading the API key table on this tick. Only a *successful* refresh
// moves the timestamp, so a failed read is retried on the next tick.
// This check is deliberately outside the refresh gate, so it races
// benignly: if two callers both read a stale timestamp, the zero-timeout
// gate below admits one and the other returns without touching the store.
return;
}
@@ -39,43 +39,3 @@ public sealed class DashboardSnapshotHub(
await base.OnDisconnectedAsync(exception).ConfigureAwait(false);
}
}
/// <summary>
/// Process-wide count of live <see cref="DashboardSnapshotHub"/> connections.
/// Registered as a singleton and read by <see cref="DashboardSnapshotPublisher"/>
/// to idle-gate the snapshot tick: with no dashboard connected there is nothing
/// to broadcast to, so no snapshot is built.
/// </summary>
public sealed class DashboardSnapshotHubConnectionCounter
{
private int _count;
/// <summary>Gets the number of live snapshot hub connections.</summary>
public int Count => Volatile.Read(ref _count);
/// <summary>Records a new snapshot hub connection.</summary>
/// <returns>The connection count after the increment.</returns>
public int Increment()
{
return Interlocked.Increment(ref _count);
}
/// <summary>
/// Records a snapshot hub disconnection. The count is clamped at zero: SignalR
/// can invoke <c>OnDisconnectedAsync</c> for a connection whose
/// <c>OnConnectedAsync</c> faulted, and a negative count would idle-gate the
/// publisher while viewers are still attached.
/// </summary>
/// <returns>The connection count after the decrement.</returns>
public int Decrement()
{
int updated = Interlocked.Decrement(ref _count);
if (updated >= 0)
{
return updated;
}
Interlocked.CompareExchange(ref _count, 0, updated);
return 0;
}
}
@@ -0,0 +1,52 @@
namespace ZB.MOM.WW.MxGateway.Server.Dashboard.Hubs;
/// <summary>
/// Process-wide count of live <see cref="DashboardSnapshotHub"/> connections.
/// Registered as a singleton and read by <see cref="DashboardSnapshotPublisher"/>
/// to idle-gate the snapshot tick: with no dashboard connected there is nothing
/// to broadcast to, so no snapshot is built.
/// </summary>
public sealed class DashboardSnapshotHubConnectionCounter
{
private int _count;
/// <summary>Gets the number of live snapshot hub connections.</summary>
public int Count => Volatile.Read(ref _count);
/// <summary>Records a new snapshot hub connection.</summary>
/// <returns>The connection count after the increment.</returns>
public int Increment()
{
return Interlocked.Increment(ref _count);
}
/// <summary>
/// Records a snapshot hub disconnection, clamped at zero: SignalR can invoke
/// <c>OnDisconnectedAsync</c> for a connection whose <c>OnConnectedAsync</c>
/// faulted, and a negative count would idle-gate the publisher while viewers
/// are still attached.
/// </summary>
/// <remarks>
/// The clamp is applied inside the compare-and-swap rather than as a repair
/// afterwards. Decrementing first and then correcting a negative result races:
/// two unmatched decrements from zero would both plan a repair, a real
/// connection could increment in between, and the stale repair would then
/// overwrite that live connection's increment — freezing a real viewer's
/// dashboard behind the idle gate. Reading, clamping, and publishing as one
/// atomic step means a lost race simply retries against the fresh value.
/// </remarks>
/// <returns>The connection count after the decrement.</returns>
public int Decrement()
{
int current;
int next;
do
{
current = Volatile.Read(ref _count);
next = current > 0 ? current - 1 : 0;
}
while (Interlocked.CompareExchange(ref _count, next, current) != current);
return next;
}
}