feat(overview): show MXAccess session counts on the gateway card

The gateway publishes its live MXAccess session counts, and the dashboard could not reach them. Four things stood between the data and the card, and only the last one is cosmetic: The active tier was probed only for instances declared HasActiveRole. A standalone gateway has no active/standby role, so its active tier was never requested at all. The tempting fix is to declare mxgw HasActiveRole, and it is wrong: that flag also paints an Active/Standby badge, so it would buy the data by asserting a role that does not exist. The probe is now unconditional for any reachable instance while ObserveActive stays role-gated, separating "what does this tier report" from "am I the active half". The active probe's parsed body was then discarded - only ready.Report was kept - so InstanceSnapshot gains ActiveReport. The two tiers stay separate rather than merged, because a check name is only unique within a tier. The card's one data-rendering path, ClusterDataLine, is located by the Akka leader/activeNode keys and would ignore session data even once present. The new row follows that helper's own precedent and locates the check by the data it carries - the total/ready/faulted triple - not by name, requiring all three so a future check publishing a lone "total" cannot be mistaken for it. Rendered on the card FACE, not inside the collapsed check list: a count behind a disclosure triangle is one nobody reads. Zero is shown rather than hidden, since a gateway holding no sessions is the interesting case and suppressing it would make the row appear only when the news is good. Cost, stated plainly: every reachable instance now makes one extra HTTP request per poll. The unreachable-instance skip is untouched, so the worst-case sweep duration - which is set by timeouts, not by healthy round-trips - is unchanged. Four cadence tests were counting raw requests as a proxy for "instances polled"; they now count ready-tier probes, which is what they always meant and is robust to tier changes. The Wonder registry comment claiming the active tier ran an empty check set was true when written and stopped being true when the gateway added the check; it now warns against the HasActiveRole shortcut instead. 183 tests pass (177 + 6), including a positive control on the absence assertion so "row not rendered" cannot pass vacuously.
2026-08-12 04:47:06 -04:00
parent 81e0005628
commit da276ff15a
6 changed files with 234 additions and 12 deletions
@@ -42,6 +42,16 @@
            <span class="v @ChecksClass">@ChecksText</span>
        </div>

+        @if (SessionText is { Length: > 0 } sessionText)
+        {
+            @* Only the gateway publishes these counts, so the row is absent everywhere else
+               rather than showing an empty or zero value on apps the concept does not apply to. *@
+            <div class="kv" data-testid="mxaccess-sessions">
+                <span class="k">MXAccess</span>
+                <span class="v @SessionClass">@sessionText</span>
+            </div>
+        }
+
        @if (Instance.Error is { Length: > 0 } error)
        {
            <div class="kv">
@@ -260,6 +270,66 @@
        }
    }

+    /// <summary>
+    /// The MXAccess session check's entry from the <b>active</b> tier, or null when this instance
+    /// publishes none.
+    /// </summary>
+    /// <remarks>
+    ///     <para>
+    ///         Located by the data it carries rather than by check name, for the same reason
+    ///         <see cref="ClusterDataLine"/> is: a name is a rename away from silently dropping the
+    ///         line, whereas the <c>total</c>/<c>ready</c>/<c>faulted</c> triple is the check's
+    ///         actual contract. All three are required so a future check that happens to publish a
+    ///         lone <c>total</c> cannot be mistaken for this one.
+    ///     </para>
+    ///     <para>
+    ///         It comes from <see cref="InstanceSnapshot.ActiveReport"/>, never the ready tier —
+    ///         the gateway registers this check on <c>active</c> because a gateway with no sessions
+    ///         is still ready to serve.
+    ///     </para>
+    /// </remarks>
+    private ZbHealthEntry? SessionEntry =>
+        Instance.ActiveReport?.Entries.Values.FirstOrDefault(e =>
+            e.DataInt("total") is not null
+            && e.DataInt("ready") is not null
+            && e.DataInt("faulted") is not null);
+
+    /// <summary>The session counts as a short face-level line, or null when there are none to show.</summary>
+    /// <remarks>
+    /// Rendered on the card face rather than inside the collapsed check list: "how many MXAccess
+    /// connections are up" is a number an operator scans for, and a value behind a disclosure
+    /// triangle is one nobody sees. Zero is shown, not hidden — a gateway holding no sessions at
+    /// all is precisely the interesting case, and suppressing it would make the row appear only
+    /// when the news is good.
+    /// </remarks>
+    private string? SessionText
+    {
+        get
+        {
+            var entry = SessionEntry;
+            if (entry is null)
+                return null;
+
+            int total = entry.DataInt("total") ?? 0;
+            if (total == 0)
+                return "0";
+
+            var parts = new List<string> { total.ToString(System.Globalization.CultureInfo.InvariantCulture) };
+            parts.Add($"{entry.DataInt("ready") ?? 0} ready");
+
+            foreach (var (key, label) in new[] { ("faulted", "faulted"), ("starting", "starting") })
+            {
+                if (entry.DataInt(key) is > 0 and var n)
+                    parts.Add($"{n} {label}");
+            }
+
+            return string.Join(" · ", parts);
+        }
+    }
+
+    /// <summary>Highlights the session row only when a session is actually broken.</summary>
+    private string? SessionClass => SessionEntry?.DataInt("faulted") is > 0 ? "bad" : null;
+
    private static string DotClass(string? status) => status switch
    {
        "Healthy" => "ok",
@@ -158,8 +158,16 @@ public sealed class OverviewPollerService : BackgroundService
        // timeout to learn what we already know, and ObserveActive maps a missing probe to Unknown
        // anyway. A 503 from ready is NOT a skip — that instance is talking, and whether it is the
        // active half is exactly what an operator wants to know about a sick node.
+        //
+        // Deliberately NOT gated on HasActiveRole. The active tier answers two different questions
+        // and they have to be separated: (1) am I the active half of a pair, and (2) what does this
+        // tier report about me. A standalone gateway has no answer to (1) but can have plenty to say
+        // about (2) — MxAccessGateway publishes its live MXAccess session counts there. Gating the
+        // probe on the role made that data unreachable, and the tempting fix (declare the gateway
+        // HasActiveRole) is wrong: it would paint an Active/Standby badge asserting a role that does
+        // not exist. So probe unconditionally and let ObserveActive stay role-gated below.
        HealthProbeResult? active = null;
-        if (target.Instance.HasActiveRole && ready.Reachable)
+        if (ready.Reachable)
            active = await probe.ProbeStatusAsync(target.ActiveUrl, timeout, cancellationToken).ConfigureAwait(false);

        var observed = StatusDerivation.Observe(ready);
@@ -178,6 +186,7 @@ public sealed class OverviewPollerService : BackgroundService
            Latency = ready.Latency,
            Error = ready.Error,
            Report = ready.Report,
+            ActiveReport = active?.Report,
        };
    }

@@ -67,6 +67,17 @@ public sealed record InstanceSnapshot
    /// <summary>The last parsed ready-tier body, which drives the per-check detail list.</summary>
    public ZbHealthReport? Report { get; init; }

+    /// <summary>
+    /// The last parsed <b>active</b>-tier body, when that tier answered with one.
+    /// </summary>
+    /// <remarks>
+    /// Kept separate from <see cref="Report"/> rather than merged into it: the two tiers answer
+    /// different questions, and a check name is only unique within a tier. It is null whenever the
+    /// active tier was not probed, did not answer, or answered without a canonical body — every one
+    /// of which is normal, so nothing driven by it may be required for the card to render.
+    /// </remarks>
+    public ZbHealthReport? ActiveReport { get; init; }
+
    /// <summary>This instance's effective staleness window.</summary>
    public TimeSpan StaleAfter { get; init; }

@@ -118,9 +118,17 @@
        "ManagementLabel": "Dashboard",
        "Instances": [
          {
-            // HasActiveRole false: mxgw is a standalone gateway with no cluster. Its
-            // /health/active does answer 200, but only because the active tier runs an EMPTY
-            // check set — an Active badge here would assert a role that does not exist.
+            // HasActiveRole false: mxgw is a standalone gateway with no cluster, so an Active
+            // badge here would assert a role that does not exist.
+            //
+            // This flag no longer decides whether the active tier is PROBED — since 2026-08-12 it
+            // always is, for any reachable instance. That tier is where mxgw publishes its live
+            // MXAccess session counts (the "MXAccess" row on the card), and gating the probe on the
+            // role made those counts unreachable. Do NOT flip this to true to get them back: the
+            // probe and the badge are deliberately separate, and true would paint a false badge.
+            //
+            // (The previous note here said the active tier ran an EMPTY check set. That was true
+            // when written and stopped being true when the gateway added its session check.)
            "Name": "wonder",
            "BaseUrl": "http://127.0.0.1:5130",
            "ManagementUrl": "http://10.220.157.247:5130/",
@@ -21,6 +21,7 @@ public class InstanceCardTests : TestContext
        TimeSpan? staleAfter = null,
        string? error = null,
        ZbHealthReport? report = null,
+        ZbHealthReport? activeReport = null,
        string? managementUrl = null) =>
        new()
        {
@@ -38,6 +39,7 @@ public class InstanceCardTests : TestContext
            Latency = TimeSpan.FromMilliseconds(18),
            Error = error,
            Report = report,
+            ActiveReport = activeReport,
            StaleAfter = staleAfter ?? TimeSpan.FromSeconds(45),
        };

@@ -246,4 +248,78 @@ public class InstanceCardTests : TestContext

    private static ZbHealthReport Report(params (string Name, string Status, string? Leader)[] entries) =>
        System.Text.Json.JsonSerializer.Deserialize<ZbHealthReport>(HealthBody.Ready("Healthy", entries))!;
+
+    // ---- MXAccess session counts (active tier) -------------------------------------------------
+    // The gateway publishes live MXAccess session counts on its ACTIVE tier. These pin that the
+    // number reaches the card face, since a count behind a disclosure triangle is one nobody reads.
+
+    private static ZbHealthReport SessionReport(int total, int ready, int faulted = 0, int starting = 0) =>
+        System.Text.Json.JsonSerializer.Deserialize<ZbHealthReport>(
+            HealthBody.WithData(
+                "Healthy",
+                ("mxaccess-sessions", "Healthy", new Dictionary<string, object>(StringComparer.Ordinal)
+                {
+                    ["total"] = total,
+                    ["ready"] = ready,
+                    ["faulted"] = faulted,
+                    ["starting"] = starting,
+                    ["closing"] = 0,
+                })))!;
+
+    [Fact]
+    public void Card_ShowsMxAccessSessionCount_FromTheActiveTier()
+    {
+        var card = Render(Card(activeReport: SessionReport(total: 2, ready: 2)));
+
+        var row = card.Find("[data-testid=mxaccess-sessions]");
+        Assert.Contains("MXAccess", row.TextContent, StringComparison.Ordinal);
+        Assert.Contains("2 · 2 ready", row.TextContent, StringComparison.Ordinal);
+    }
+
+    [Fact]
+    public void Card_ShowsZeroSessions_RatherThanHidingTheRow()
+    {
+        // A gateway holding no MXAccess sessions is the interesting case. Suppressing zero would
+        // make the row appear only when the news is good, which is when nobody needs it.
+        var card = Render(Card(activeReport: SessionReport(total: 0, ready: 0)));
+
+        Assert.Equal("0", card.Find("[data-testid=mxaccess-sessions] .v").TextContent.Trim());
+    }
+
+    [Fact]
+    public void Card_CallsOutFaultedSessions()
+    {
+        var card = Render(Card(activeReport: SessionReport(total: 3, ready: 2, faulted: 1)));
+
+        var value = card.Find("[data-testid=mxaccess-sessions] .v");
+        Assert.Contains("1 faulted", value.TextContent, StringComparison.Ordinal);
+        Assert.Contains("bad", value.ClassList, StringComparer.Ordinal);
+    }
+
+    [Fact]
+    public void Card_OmitsTheMxAccessRow_ForAppsThatPublishNoSessionCounts()
+    {
+        // Positive control first: the SAME assertion must find the row when the data IS present,
+        // or "not found" below would prove nothing about the omission.
+        Assert.NotEmpty(Render(Card(activeReport: SessionReport(1, 1))).FindAll("[data-testid=mxaccess-sessions]"));
+
+        Assert.Empty(Render(Card()).FindAll("[data-testid=mxaccess-sessions]"));
+    }
+
+    [Fact]
+    public void Card_IgnoresActiveTierChecksThatAreNotSessionCounts()
+    {
+        // The row is located by the total/ready/faulted triple, not by check name. An active-node
+        // check publishes its own data on the same tier and must not be mistaken for a session count.
+        var activeNode = System.Text.Json.JsonSerializer.Deserialize<ZbHealthReport>(
+            HealthBody.WithData(
+                "Healthy",
+                ("active-node", "Healthy", new Dictionary<string, object>(StringComparer.Ordinal)
+                {
+                    ["activeNode"] = "site-a-a",
+                    ["total"] = 3,
+                })))!;
+
+        Assert.Empty(Render(Card(activeReport: activeNode)).FindAll("[data-testid=mxaccess-sessions]"));
+    }
 }
@@ -75,7 +75,10 @@ public class OverviewPollerServiceTests

        Assert.Equal(2, snapshot.AllInstances.Count());
        Assert.All(snapshot.AllInstances, i => Assert.Equal(InstanceStatus.Up, i.Status));
-        Assert.Equal(2, handler.Requests.Count);
+        // Counted on the READY tier alone: one ready probe == one polled instance, which is what
+        // this test means. A reachable instance is also probed on the active tier for its data, so
+        // a raw request count would pin the number of tiers rather than the polling behaviour.
+        Assert.Equal(2, ReadyProbes(handler));
    }

    [Fact]
@@ -92,12 +95,18 @@ public class OverviewPollerServiceTests
    }

    [Fact]
-    public async Task Sweep_ActiveTierProbedOnlyForInstancesThatHaveTheRole()
+    public async Task Sweep_ProbesTheActiveTierEvenWithoutTheRole_ButStillAwardsNoBadge()
    {
+        // The active tier answers two separable questions: "am I the active half of a pair" and
+        // "what does this tier report about me". A standalone gateway has no answer to the first
+        // and plenty to say about the second (its live MXAccess session counts), so the PROBE is
+        // unconditional while the BADGE stays role-gated. Declaring such a gateway hasActiveRole
+        // would fetch the data at the cost of asserting a role that does not exist.
        var handler = new RoutingHandler()
            .Json("http://a:8080/health/ready", HttpStatusCode.OK, HealthBody.Ready("Healthy"))
            .Status("http://a:8080/health/active", HttpStatusCode.OK)
-            .Json("http://gw:5120/health/ready", HttpStatusCode.OK, HealthBody.Ready("Healthy"));
+            .Json("http://gw:5120/health/ready", HttpStatusCode.OK, HealthBody.Ready("Healthy"))
+            .Status("http://gw:5120/health/active", HttpStatusCode.OK);
        var (poller, _) = Build(
            Registry(App("App", Node("a", "http://a:8080", "pair", hasActiveRole: true), Node("gw", "http://gw:5120"))),
            handler,
@@ -105,12 +114,39 @@ public class OverviewPollerServiceTests

        var snapshot = await poller.SweepAsync();

-        Assert.DoesNotContain("http://gw:5120/health/active", handler.Requests, StringComparer.Ordinal);
+        Assert.Contains("http://gw:5120/health/active", handler.Requests, StringComparer.Ordinal);
        var byName = snapshot.AllInstances.ToDictionary(i => i.Instance, StringComparer.Ordinal);
        Assert.Equal(ActiveState.Active, byName["a"].Active);
        Assert.Equal(ActiveState.NotApplicable, byName["gw"].Active);
    }

+    [Fact]
+    public async Task Sweep_RetainsTheActiveTierBody_SoItsDataCanBeRendered()
+    {
+        var body = HealthBody.WithData(
+            "Healthy",
+            ("mxaccess-sessions", "Healthy", new Dictionary<string, object>(StringComparer.Ordinal)
+            {
+                ["total"] = 2,
+                ["ready"] = 2,
+                ["faulted"] = 0,
+            }));
+        var handler = new RoutingHandler()
+            .Json("http://gw:5120/health/ready", HttpStatusCode.OK, HealthBody.Ready("Healthy"))
+            .Json("http://gw:5120/health/active", HttpStatusCode.OK, body);
+        var (poller, _) = Build(
+            Registry(App("App", Node("gw", "http://gw:5120"))),
+            handler,
+            new FakeTimeProvider(Start));
+
+        var instance = Assert.Single((await poller.SweepAsync()).AllInstances);
+
+        Assert.NotNull(instance.ActiveReport);
+        Assert.Equal(2, instance.ActiveReport!.Entries["mxaccess-sessions"].DataInt("total"));
+        // The tiers stay separate: a check name is only unique within a tier.
+        Assert.DoesNotContain("mxaccess-sessions", instance.Report!.Entries.Keys, StringComparer.Ordinal);
+    }
+
    [Fact]
    public async Task Sweep_StandbyAnswers503OnActive_AndIsStillUp()
    {
@@ -220,7 +256,9 @@ public class OverviewPollerServiceTests
        time.Advance(TimeSpan.FromSeconds(1));
        await poller.SweepAsync();

-        Assert.Single(handler.Requests);
+        // One ready probe == one poll. The second sweep must not poll at all; counting ready
+        // probes says exactly that without also asserting how many tiers a poll touches.
+        Assert.Equal(1, ReadyProbes(handler));
    }

    [Fact]
@@ -245,8 +283,8 @@ public class OverviewPollerServiceTests
            time.Advance(TimeSpan.FromSeconds(5));
        }

-        Assert.Equal(4, handler.Requests.Count(r => r.Contains("fast", StringComparison.Ordinal)));
-        Assert.Equal(1, handler.Requests.Count(r => r.Contains("slow", StringComparison.Ordinal)));
+        Assert.Equal(4, ReadyProbes(handler, "fast"));
+        Assert.Equal(1, ReadyProbes(handler, "slow"));
    }

    [Fact]
@@ -262,7 +300,7 @@ public class OverviewPollerServiceTests
        time.Advance(TimeSpan.FromMilliseconds(9_900));
        await poller.SweepAsync();

-        Assert.Equal(2, handler.Requests.Count);
+        Assert.Equal(2, ReadyProbes(handler));
    }

    [Fact]
@@ -430,4 +468,14 @@ public class OverviewPollerServiceTests
            return await inner.HandleAsync(request).ConfigureAwait(false);
        }
    }
+
+    /// <summary>
+    /// Ready-tier probes only - one per polled instance. Tests about polling cadence must not
+    /// count total requests, which also includes the active-tier probe every reachable instance
+    /// now receives.
+    /// </summary>
+    private static int ReadyProbes(RoutingHandler handler, string? hostContains = null) =>
+        handler.Requests.Count(r =>
+            r.EndsWith("/health/ready", StringComparison.Ordinal)
+            && (hostContains is null || r.Contains(hostContains, StringComparison.Ordinal)));
 }