feat(transport): counter for import-session cap rejections (register row 66)
Row 53 shipped the scadabridge.central.transport.import_sessions.open gauge
and a Warning log on a cap hit, but a log line is not something an operator
can alert on. Adds the companion counter
scadabridge.central.transport.import_sessions.rejected
on the existing ZB.MOM.WW.ScadaBridge meter, so no ObservedMeters allowlist
change is needed (an unlisted meter exports nothing, silently).
The emit point is deliberately narrow. BundleSessionStore.Open tests the cap
TWICE — once on entry and again after EvictExpired() — and only the second
test is a rejection: a store full of merely expired sessions reclaims a slot
and admits the caller. Counting the first would make the series fire
routinely on any node whose sweep had not yet run, turning a page-worthy
signal into background noise. Re-Opening an already-open session id skips
the cap block entirely and is likewise not counted.
The two instruments are diagnostic as a pair: rejections rising while the
gauge oscillates below the cap is genuine concurrency (raise the cap);
rejections rising while the gauge sits pinned at the cap is a session leak
(register row 43's signature), where raising the cap only defers the problem.
Tests assert a delta over a scoped action rather than an absolute value, so
they do not inherit the register row 32 static-state flake that makes
QueueDepthGaugeTests fail only under the full concurrent suite. Negative
controls cover under-cap, expired-reclaim, and duplicate-session-id.
Tests: 11 passed in BundleSessionObservabilityTests (6 existing + 5 new).
This commit is contained in:
@@ -93,6 +93,23 @@ public static class ScadaBridgeTelemetry
|
||||
Meter.CreateCounter<long>("scadabridge.site.stream.events_dropped", unit: "1",
|
||||
description: "Events evicted from a site gRPC stream's bounded send channel, tagged by stream kind.");
|
||||
|
||||
/// <summary>
|
||||
/// Register row 66: incremented each time <c>BundleSessionStore.Open</c> refuses a
|
||||
/// bundle-import session because <c>TransportOptions.MaxConcurrentImportSessions</c>
|
||||
/// (default 8) is genuinely exhausted — i.e. the store was still at the cap AFTER
|
||||
/// expired sessions were reclaimed. Reclaim-then-succeed and re-opening an already
|
||||
/// open session id are deliberately NOT counted: neither turns an operator away.
|
||||
///
|
||||
/// Row 53 shipped the companion open-sessions gauge and a Warning log, but a log line
|
||||
/// is not alertable. The pair is what distinguishes the two failure modes: rejections
|
||||
/// climbing while the gauge oscillates below the cap is genuine concurrency (raise the
|
||||
/// cap), whereas rejections climbing while the gauge sits pinned at the cap is a
|
||||
/// session leak like register row 43's (find the leak — raising the cap only defers it).
|
||||
/// </summary>
|
||||
private static readonly Counter<long> _transportImportSessionsRejected =
|
||||
Meter.CreateCounter<long>("scadabridge.central.transport.import_sessions.rejected", unit: "1",
|
||||
description: "Bundle-import sessions rejected because the concurrent-import-session cap was exhausted.");
|
||||
|
||||
// ---------------- Histograms ----------------
|
||||
|
||||
/// <summary>
|
||||
@@ -186,6 +203,14 @@ public static class ScadaBridgeTelemetry
|
||||
/// <summary>Records one resync whose ack window expired.</summary>
|
||||
public static void RecordSfResyncAckMissing() => _sfResyncAckMissing.Add(1);
|
||||
|
||||
/// <summary>
|
||||
/// Records one bundle-import session rejected on a genuine concurrent-session cap hit
|
||||
/// (register row 66). Call ONLY from the post-eviction recheck in
|
||||
/// <c>BundleSessionStore.Open</c> — counting the pre-eviction check would inflate the
|
||||
/// series with reclaim-then-succeed cases that never turned an operator away.
|
||||
/// </summary>
|
||||
public static void RecordImportSessionRejected() => _transportImportSessionsRejected.Add(1);
|
||||
|
||||
/// <summary>Records that a site connection opened (increments the up-count gauge).</summary>
|
||||
public static void SiteConnectionOpened() => Interlocked.Increment(ref _siteConnectionsUp);
|
||||
|
||||
|
||||
@@ -3,6 +3,7 @@ using Microsoft.Extensions.Logging;
|
||||
using Microsoft.Extensions.Logging.Abstractions;
|
||||
using Microsoft.Extensions.Options;
|
||||
using ZB.MOM.WW.ScadaBridge.Commons.Interfaces.Transport;
|
||||
using ZB.MOM.WW.ScadaBridge.Commons.Observability;
|
||||
using ZB.MOM.WW.ScadaBridge.Commons.Types.Transport;
|
||||
|
||||
namespace ZB.MOM.WW.ScadaBridge.Transport.Import;
|
||||
@@ -115,6 +116,15 @@ public sealed class BundleSessionStore : IBundleSessionStore
|
||||
EvictExpired();
|
||||
if (_sessions.Count >= cap)
|
||||
{
|
||||
// Register row 66: the alertable half of row 53's observability. The
|
||||
// Warning below carries the diagnosis but a log line cannot be alerted
|
||||
// on; this counter can. Deliberately incremented HERE, after
|
||||
// EvictExpired() has already failed to free a slot — the outer check
|
||||
// at the top of this block is not a rejection (a store full of stale
|
||||
// entries reclaims and succeeds), so counting there would inflate the
|
||||
// series with imports that were never actually turned away.
|
||||
ScadaBridgeTelemetry.RecordImportSessionRejected();
|
||||
|
||||
// Register row 53: this rejection used to be silent apart from the
|
||||
// exception the caller saw. A cap hit is either genuine concurrency
|
||||
// (many operators importing at once) or a session leak like register
|
||||
|
||||
Reference in New Issue
Block a user