feat(overview): poller, leader aggregation and the dashboard UI
Tasks 3.4-3.6 of the overview-dashboard plan. Polling (3.4/3.5): - OverviewPollerService: one timer at the fastest configured cadence, each target polled when its own interval is due, fan-out per sweep, no retries (the next tick is the retry, so damping stays the only place that decides whether a failure is worth showing). - OverviewSnapshotStore: atomic swap of a complete immutable graph plus a change event. Pages read it and never probe — the cache-first requirement. - LeaderResolver: per-group leader from akka-cluster data, with the split-brain flag scoped to members that are actually answering, so an ordinary failover is not misread as a disagreement. UI (3.6): ThemeShell + StatusPill/TechCard composition against the mockup; instance cards carry the status stripe (dashed for Unreachable), the check detail list and the raw ready/active signal line. Two defects found by verification rather than by review: - The active tier answers with a status code and an EMPTY body, but the client demanded parseable JSON on every probe — so a healthy pair rendered as "role unknown" instead of Active/Standby. Split into ProbeAsync (ready tier, body is the payload) and ProbeStatusAsync (active tier, code is the answer). Caught by a live smoke run against two fake health endpoints; the poller tests now serve an empty body so they hold the fix. - SweepAsync published even when cancellation was already requested, leaving a half-probed registry as the store's final state on shutdown. It now checks the token itself rather than relying on the transport to throw. Also: app.UseAntiforgery() is required despite the anonymous-by-design pipeline — AddRazorComponents stamps antiforgery metadata unconditionally and the endpoint middleware hard-fails without it (every page was 500). Chosen over DisableAntiforgery() so a future form is protected by default. 147 tests, 0 warnings. Live-verified end to end against fake endpoints: Up/Unreachable/Active/Standby, leader chip, cluster-data line, KPI counts.
This commit is contained in:
Joseph Doherty
@@ -0,0 +1,22 @@
|
||||
@* Host page. Head order matters: Bootstrap first (the Theme kit's shell and
|
||||
TechButton are built on Bootstrap utility classes), then <ThemeHead/> which
|
||||
brings the design tokens and IBM Plex, then site.css so this app's own
|
||||
dashboard anatomy can override either. *@
|
||||
<!doctype html>
|
||||
<html lang="en">
|
||||
<head>
|
||||
<meta charset="utf-8" />
|
||||
<meta name="viewport" content="width=device-width, initial-scale=1" />
|
||||
<base href="/" />
|
||||
<link rel="stylesheet" href="/lib/bootstrap/css/bootstrap.min.css" />
|
||||
<ThemeHead />
|
||||
<link rel="stylesheet" href="/css/site.css" />
|
||||
<HeadOutlet @rendermode="InteractiveServer" />
|
||||
</head>
|
||||
<body>
|
||||
<Routes @rendermode="InteractiveServer" />
|
||||
<script src="/lib/bootstrap/js/bootstrap.bundle.min.js"></script>
|
||||
<ThemeScripts />
|
||||
<script src="/_framework/blazor.web.js"></script>
|
||||
</body>
|
||||
</html>
|
||||
@@ -0,0 +1,43 @@
|
||||
@inherits LayoutComponentBase
|
||||
@inject OverviewSnapshotStore Store
|
||||
|
||||
@* Thin layout: the side-rail chassis belongs to the shared kit's <ThemeShell>.
|
||||
No AuthorizeView anywhere — this dashboard is anonymous by requirement, so
|
||||
there is no user to show and nothing to sign out of. The rail's application
|
||||
links are fragment anchors into the single page rather than routes: the whole
|
||||
fleet is one view, and splitting it per application would defeat the point. *@
|
||||
<ThemeShell Product="SCADA Overview" Accent="#6d28d9">
|
||||
<Nav>
|
||||
<NavRailItem Href="/" Text="Overview" Match="NavLinkMatch.All" />
|
||||
@if (Applications.Count > 0)
|
||||
{
|
||||
<NavRailSection Title="Applications" Key="overview-apps">
|
||||
@foreach (var application in Applications)
|
||||
{
|
||||
<a class="rail-link" href="@($"#{Slug(application)}")">@application</a>
|
||||
}
|
||||
</NavRailSection>
|
||||
}
|
||||
</Nav>
|
||||
<RailFooter>
|
||||
<div class="rail-note">read-only · anonymous<br />ZB.MOM.WW.Overview @Version</div>
|
||||
</RailFooter>
|
||||
<ChildContent>@Body</ChildContent>
|
||||
</ThemeShell>
|
||||
|
||||
@code {
|
||||
private static readonly string Version =
|
||||
typeof(MainLayout).Assembly.GetName().Version?.ToString(3) ?? "0.0.0";
|
||||
|
||||
private IReadOnlyList<string> Applications =>
|
||||
Store.Current.Applications.Select(a => a.Name).ToList();
|
||||
|
||||
/// <summary>
|
||||
/// Builds the fragment id for an application section. Must stay identical to the id the
|
||||
/// Overview page emits, or every rail link becomes a dead anchor.
|
||||
/// </summary>
|
||||
/// <param name="name">The application's display name.</param>
|
||||
/// <returns>A URL-fragment-safe slug.</returns>
|
||||
internal static string Slug(string name) =>
|
||||
new(name.Where(c => char.IsLetterOrDigit(c) || c is '-' or '_').Select(char.ToLowerInvariant).ToArray());
|
||||
}
|
||||
@@ -0,0 +1,245 @@
|
||||
@page "/"
|
||||
@implements IDisposable
|
||||
@inject OverviewSnapshotStore Store
|
||||
@inject TimeProvider Time
|
||||
|
||||
@* The whole dashboard, rendered from the snapshot store. This page never polls
|
||||
an instance: it subscribes to the store and re-renders when the poller swaps
|
||||
a snapshot in, plus a slow local tick so ages and staleness advance on their
|
||||
own between sweeps. *@
|
||||
|
||||
@* Read once per render and threaded through everything below, so every card, KPI
|
||||
and group header on one paint agrees on what "now" is. Reading the clock at
|
||||
render — rather than caching it when a snapshot arrives — is what lets the view
|
||||
age on its own: a paused or stuck poller publishes nothing, and the cards still
|
||||
cross into Stale. *@
|
||||
@{
|
||||
var now = Time.GetUtcNow();
|
||||
}
|
||||
|
||||
<PageTitle>SCADA Overview</PageTitle>
|
||||
|
||||
<div class="page-head rise">
|
||||
<h1>Fleet Overview</h1>
|
||||
<span class="page-meta">@Meta</span>
|
||||
<span class="spacer"></span>
|
||||
<span class="conn-pill" data-state="@(_paused ? "disconnected" : "connected")">
|
||||
<span class="dot"></span>@(_paused ? "Paused" : "Polling")
|
||||
</span>
|
||||
<button class="rail-btn" type="button" data-testid="pause-toggle" @onclick="TogglePause">
|
||||
@(_paused ? "Resume auto-refresh" : "Pause auto-refresh")
|
||||
</button>
|
||||
</div>
|
||||
|
||||
@if (_snapshot.Applications.Count == 0)
|
||||
{
|
||||
<p class="page-meta" data-testid="empty-registry">
|
||||
No instances have been polled yet.
|
||||
</p>
|
||||
}
|
||||
else
|
||||
{
|
||||
<div class="agg-grid rise">
|
||||
<div class="agg-card">
|
||||
<div class="agg-label">Instances</div>
|
||||
<div class="agg-value">@_instances.Count<span class="agg-sub">registered</span></div>
|
||||
</div>
|
||||
<div class="agg-card good">
|
||||
<div class="agg-label">Up</div>
|
||||
<div class="agg-value" data-testid="kpi-up">@Count(now, InstanceStatus.Up)</div>
|
||||
</div>
|
||||
<div class="agg-card @(Count(now, InstanceStatus.Degraded) > 0 ? "caution" : null)">
|
||||
<div class="agg-label">Degraded</div>
|
||||
<div class="agg-value" data-testid="kpi-degraded">@Count(now, InstanceStatus.Degraded)</div>
|
||||
</div>
|
||||
<div class="agg-card @(Count(now, InstanceStatus.Down) > 0 ? "alert" : null)">
|
||||
<div class="agg-label">Down</div>
|
||||
<div class="agg-value" data-testid="kpi-down">@Count(now, InstanceStatus.Down)</div>
|
||||
</div>
|
||||
<div class="agg-card @(Count(now, InstanceStatus.Unreachable) > 0 ? "alert" : null)">
|
||||
<div class="agg-label">Unreachable</div>
|
||||
<div class="agg-value" data-testid="kpi-unreachable">@Count(now, InstanceStatus.Unreachable)</div>
|
||||
</div>
|
||||
<div class="agg-card quiet">
|
||||
<div class="agg-label">Stale</div>
|
||||
<div class="agg-value" data-testid="kpi-stale">@StaleCount(now)</div>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
@foreach (var application in _snapshot.Applications)
|
||||
{
|
||||
<section class="app-section rise" id="@MainLayout.Slug(application.Name)" data-testid="application">
|
||||
<div class="app-head">
|
||||
<h2>@application.Name</h2>
|
||||
<StatusPill State="RollupState(application)">@RollupLabel(application)</StatusPill>
|
||||
<span class="spacer"></span>
|
||||
<span class="app-meta">@AppMeta(application)</span>
|
||||
</div>
|
||||
|
||||
@foreach (var group in Layout(application))
|
||||
{
|
||||
<div class="group">
|
||||
@if (group.Key is { } groupName)
|
||||
{
|
||||
var state = application.GroupFor(groupName);
|
||||
<div class="group-head">
|
||||
<span class="group-name">@groupName</span>
|
||||
@if (state?.LeaderInstance is { } leaderInstance)
|
||||
{
|
||||
<StatusPill State="StatusState.Info">Leader · @leaderInstance</StatusPill>
|
||||
}
|
||||
else if (state?.LeaderAddress is { } leaderAddress)
|
||||
{
|
||||
@* Unresolved address: shown raw rather than hidden — a leader on a
|
||||
host that is not in the registry is itself the finding. *@
|
||||
<StatusPill State="StatusState.Info">Leader · @leaderAddress</StatusPill>
|
||||
}
|
||||
@if (state?.Disagreement == true)
|
||||
{
|
||||
<StatusPill State="StatusState.Bad">
|
||||
Split brain · @string.Join(" vs ", state.ReportedLeaders)
|
||||
</StatusPill>
|
||||
}
|
||||
<span class="group-meta">@GroupMeta(group, now)</span>
|
||||
</div>
|
||||
}
|
||||
<div class="card-grid">
|
||||
@foreach (var instance in group)
|
||||
{
|
||||
<InstanceCard Instance="instance"
|
||||
Now="now"
|
||||
IsLeader="IsLeader(application, instance)" />
|
||||
}
|
||||
</div>
|
||||
</div>
|
||||
}
|
||||
</section>
|
||||
}
|
||||
}
|
||||
|
||||
@code {
|
||||
/// <summary>
|
||||
/// How often the page re-renders on its own. Independent of the poll interval: its job is to
|
||||
/// age the "last poll" figures and let a card cross into Stale even when no sweep lands —
|
||||
/// which is exactly the case a stuck poller produces.
|
||||
/// </summary>
|
||||
private static readonly TimeSpan UiTick = TimeSpan.FromSeconds(5);
|
||||
|
||||
private OverviewSnapshot _snapshot = OverviewSnapshot.Empty;
|
||||
private IReadOnlyList<InstanceSnapshot> _instances = [];
|
||||
private bool _paused;
|
||||
private ITimer? _timer;
|
||||
|
||||
protected override void OnInitialized()
|
||||
{
|
||||
Adopt(Store.Current);
|
||||
Store.Changed += OnSnapshotChanged;
|
||||
_timer = Time.CreateTimer(_ => InvokeAsync(StateHasChanged), null, UiTick, UiTick);
|
||||
}
|
||||
|
||||
private void OnSnapshotChanged(OverviewSnapshot snapshot)
|
||||
{
|
||||
// Pause stops ADOPTING new snapshots; the poller keeps sweeping. That is what makes the
|
||||
// pause useful — the frozen view then visibly ages into Stale, so a paused dashboard can
|
||||
// never be mistaken for a live one.
|
||||
if (_paused)
|
||||
return;
|
||||
|
||||
InvokeAsync(() =>
|
||||
{
|
||||
Adopt(snapshot);
|
||||
StateHasChanged();
|
||||
});
|
||||
}
|
||||
|
||||
private void Adopt(OverviewSnapshot snapshot)
|
||||
{
|
||||
_snapshot = snapshot;
|
||||
_instances = snapshot.AllInstances.ToList();
|
||||
}
|
||||
|
||||
private void TogglePause()
|
||||
{
|
||||
_paused = !_paused;
|
||||
|
||||
if (!_paused)
|
||||
Adopt(Store.Current);
|
||||
}
|
||||
|
||||
private int Count(DateTimeOffset now, InstanceStatus status) =>
|
||||
_instances.Count(i => i.Status == status && !i.IsStale(now));
|
||||
|
||||
private int StaleCount(DateTimeOffset now) => _instances.Count(i => i.IsStale(now));
|
||||
|
||||
private string Meta
|
||||
{
|
||||
get
|
||||
{
|
||||
var generated = _snapshot.GeneratedUtc == default
|
||||
? "never"
|
||||
: _snapshot.GeneratedUtc.ToLocalTime().ToString("HH:mm:ss", System.Globalization.CultureInfo.InvariantCulture);
|
||||
|
||||
return $"last sweep {generated} · {_instances.Count} instances";
|
||||
}
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Groups an application's instances for display: named cluster groups first in registry
|
||||
/// order, then the ungrouped ones.
|
||||
/// </summary>
|
||||
/// <param name="application">The application to lay out.</param>
|
||||
/// <returns>The instance groups.</returns>
|
||||
private static IEnumerable<IGrouping<string?, InstanceSnapshot>> Layout(ApplicationSnapshot application) =>
|
||||
application.Instances
|
||||
.GroupBy(i => i.Group, StringComparer.Ordinal)
|
||||
.OrderBy(g => g.Key is null)
|
||||
.ThenBy(g => g.Key, StringComparer.Ordinal);
|
||||
|
||||
private static string AppMeta(ApplicationSnapshot application)
|
||||
{
|
||||
var clusters = application.Groups.Count;
|
||||
var instances = application.Instances.Count;
|
||||
var clusterText = clusters == 0 ? "" : $" · {clusters} cluster{(clusters == 1 ? "" : "s")}";
|
||||
|
||||
return $"{instances} instance{(instances == 1 ? "" : "s")}{clusterText}";
|
||||
}
|
||||
|
||||
private static string GroupMeta(IGrouping<string?, InstanceSnapshot> group, DateTimeOffset now)
|
||||
{
|
||||
var members = group.Count();
|
||||
var missing = group.Count(i => i.Status is InstanceStatus.Unreachable || i.IsStale(now));
|
||||
|
||||
return $"{members} member{(members == 1 ? "" : "s")} · {missing} unreachable";
|
||||
}
|
||||
|
||||
private static bool IsLeader(ApplicationSnapshot application, InstanceSnapshot instance) =>
|
||||
instance.Group is { } group &&
|
||||
application.GroupFor(group)?.LeaderInstance is { } leader &&
|
||||
string.Equals(leader, instance.Instance, StringComparison.Ordinal);
|
||||
|
||||
private static StatusState RollupState(ApplicationSnapshot application) => application.Worst switch
|
||||
{
|
||||
InstanceStatus.Up => StatusState.Ok,
|
||||
InstanceStatus.Degraded => StatusState.Warn,
|
||||
InstanceStatus.Down or InstanceStatus.Unreachable => StatusState.Bad,
|
||||
_ => StatusState.Idle,
|
||||
};
|
||||
|
||||
private static string RollupLabel(ApplicationSnapshot application) => application.Worst switch
|
||||
{
|
||||
InstanceStatus.Up => "Up",
|
||||
InstanceStatus.Degraded => "Degraded",
|
||||
InstanceStatus.Down => "Down",
|
||||
InstanceStatus.Unreachable => "Unreachable",
|
||||
_ => "Pending",
|
||||
};
|
||||
|
||||
/// <inheritdoc/>
|
||||
public void Dispose()
|
||||
{
|
||||
// Both are required: an un-detached handler would call StateHasChanged on a disposed
|
||||
// renderer once per sweep for the life of the process.
|
||||
Store.Changed -= OnSnapshotChanged;
|
||||
_timer?.Dispose();
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,18 @@
|
||||
@* Plain RouteView, not AuthorizeRouteView. The dashboard is anonymous and
|
||||
read-only by requirement (design §2): there is no authentication pipeline to
|
||||
authorize against, and wiring one in would be the opposite of the decision. *@
|
||||
<Router AppAssembly="@typeof(Routes).Assembly">
|
||||
<Found Context="routeData">
|
||||
<RouteView RouteData="@routeData" DefaultLayout="@typeof(MainLayout)" />
|
||||
<FocusOnNavigate RouteData="@routeData" Selector="h1" />
|
||||
</Found>
|
||||
<NotFound>
|
||||
<LayoutView Layout="@typeof(MainLayout)">
|
||||
<PageTitle>Overview — Not Found</PageTitle>
|
||||
<section class="app-section">
|
||||
<h1 class="page-head">Not Found</h1>
|
||||
<p class="mb-0">The requested page does not exist.</p>
|
||||
</section>
|
||||
</LayoutView>
|
||||
</NotFound>
|
||||
</Router>
|
||||
@@ -0,0 +1,277 @@
|
||||
@* One instance = one card. Every value shown here comes from the snapshot the
|
||||
poller published; the component never probes anything itself. *@
|
||||
|
||||
<article class="inst @StateClass" data-testid="instance-card" data-instance="@Instance.Instance">
|
||||
<div class="inst-head">
|
||||
<div>
|
||||
<span class="inst-name">@Instance.Instance</span>
|
||||
<span class="inst-host">@HostAndPort</span>
|
||||
</div>
|
||||
<div class="inst-chips">
|
||||
@if (IsLeader)
|
||||
{
|
||||
<StatusPill State="StatusState.Info">Leader</StatusPill>
|
||||
}
|
||||
@if (Instance.Active is ActiveState.Active)
|
||||
{
|
||||
<StatusPill State="StatusState.Info">Active</StatusPill>
|
||||
}
|
||||
else if (Instance.Active is ActiveState.Standby)
|
||||
{
|
||||
<StatusPill State="StatusState.Idle">Standby</StatusPill>
|
||||
}
|
||||
else if (Instance.Active is ActiveState.Unknown)
|
||||
{
|
||||
<StatusPill State="StatusState.Idle">Role ?</StatusPill>
|
||||
}
|
||||
<StatusPill State="StatusPillState">@StatusLabel</StatusPill>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<div class="inst-body">
|
||||
<div class="kv">
|
||||
<span class="k">Latency</span>
|
||||
<span class="v">@LatencyText</span>
|
||||
</div>
|
||||
<div class="kv">
|
||||
<span class="k">@(IsStale ? "Stale since" : "Last poll")</span>
|
||||
<span class="v @(IsStale ? "idle" : null)">@LastPollText</span>
|
||||
</div>
|
||||
<div class="kv">
|
||||
<span class="k">Checks</span>
|
||||
<span class="v @ChecksClass">@ChecksText</span>
|
||||
</div>
|
||||
|
||||
@if (Instance.Error is { Length: > 0 } error)
|
||||
{
|
||||
<div class="kv">
|
||||
<span class="k">Error</span>
|
||||
<span class="v bad" title="@error">@Truncate(error, 48)</span>
|
||||
</div>
|
||||
}
|
||||
|
||||
@if (Entries.Count > 0)
|
||||
{
|
||||
<details class="checks">
|
||||
<summary>@Entries.Count check@(Entries.Count == 1 ? "" : "s")</summary>
|
||||
@foreach (var entry in Entries)
|
||||
{
|
||||
<div class="check-row">
|
||||
<span class="check-dot @DotClass(entry.Value.Status)"></span>
|
||||
<span class="check-name">@entry.Key</span>
|
||||
<span class="check-desc">@entry.Value.Description</span>
|
||||
</div>
|
||||
}
|
||||
@if (ClusterDataLine is { Length: > 0 } clusterData)
|
||||
{
|
||||
@* The cluster view, rendered verbatim rather than interpreted: when a pair
|
||||
disagrees about the leader this line is the raw evidence an operator needs
|
||||
to see, not a summary of it. *@
|
||||
<div class="check-data">@((MarkupString)clusterData)</div>
|
||||
}
|
||||
</details>
|
||||
}
|
||||
</div>
|
||||
|
||||
<div class="panel-foot">
|
||||
@if (Instance.ManagementUrl is { Length: > 0 } managementUrl)
|
||||
{
|
||||
<a href="@managementUrl" target="_blank" rel="noopener">Open @Instance.ManagementLabel ↗</a>
|
||||
}
|
||||
else
|
||||
{
|
||||
<span class="foot-note">no management link</span>
|
||||
}
|
||||
<span class="foot-note" data-testid="raw-signals">@RawSignals</span>
|
||||
</div>
|
||||
</article>
|
||||
|
||||
@code {
|
||||
/// <summary>The instance to render.</summary>
|
||||
[Parameter, EditorRequired] public InstanceSnapshot Instance { get; set; } = default!;
|
||||
|
||||
/// <summary>The render-time clock, passed down so every card on a page agrees on "now".</summary>
|
||||
[Parameter, EditorRequired] public DateTimeOffset Now { get; set; }
|
||||
|
||||
/// <summary>Whether this instance is its group's leader.</summary>
|
||||
[Parameter] public bool IsLeader { get; set; }
|
||||
|
||||
private bool IsStale => Instance.IsStale(Now);
|
||||
|
||||
private IReadOnlyList<KeyValuePair<string, ZbHealthEntry>> Entries =>
|
||||
Instance.Report?.Entries.OrderBy(e => e.Key, StringComparer.Ordinal).ToList() ?? [];
|
||||
|
||||
/// <summary>
|
||||
/// Stale wins over the stored status in the class, so the card visibly greys out. What it
|
||||
/// shows is still the last real reading — the point is to mark it as no longer current, not
|
||||
/// to replace it with a guess.
|
||||
/// </summary>
|
||||
private string StateClass => Instance.Status switch
|
||||
{
|
||||
null => "pending",
|
||||
_ when IsStale => $"{Instance.Status.ToString()!.ToLowerInvariant()} stale",
|
||||
var status => status.ToString()!.ToLowerInvariant(),
|
||||
};
|
||||
|
||||
private string StatusLabel => Instance.Status switch
|
||||
{
|
||||
null => "Pending",
|
||||
_ when IsStale => "Stale",
|
||||
InstanceStatus.Up => "Up",
|
||||
InstanceStatus.Degraded => "Degraded",
|
||||
InstanceStatus.Down => "Down",
|
||||
_ => "Unreachable",
|
||||
};
|
||||
|
||||
private StatusState StatusPillState => Instance.Status switch
|
||||
{
|
||||
null => StatusState.Idle,
|
||||
_ when IsStale => StatusState.Idle,
|
||||
InstanceStatus.Up => StatusState.Ok,
|
||||
InstanceStatus.Degraded => StatusState.Warn,
|
||||
_ => StatusState.Bad,
|
||||
};
|
||||
|
||||
private string HostAndPort
|
||||
{
|
||||
get
|
||||
{
|
||||
if (!Uri.TryCreate(Instance.BaseUrl, UriKind.Absolute, out var uri))
|
||||
return Instance.BaseUrl;
|
||||
|
||||
return uri.IsDefaultPort ? uri.Host : $"{uri.Host}:{uri.Port}";
|
||||
}
|
||||
}
|
||||
|
||||
private string LatencyText =>
|
||||
Instance.Status is null ? "—" : $"{Instance.Latency.TotalMilliseconds:0} ms";
|
||||
|
||||
private string LastPollText
|
||||
{
|
||||
get
|
||||
{
|
||||
// Stale cards count from the last successful contact, not the last poll attempt:
|
||||
// "stale since 4 s ago" while an instance has actually been gone for ten minutes
|
||||
// would be actively misleading.
|
||||
var reference = IsStale ? Instance.LastReachableUtc ?? Instance.LastPolledUtc : Instance.LastPolledUtc;
|
||||
return reference is { } at ? Relative(Now - at) : "never";
|
||||
}
|
||||
}
|
||||
|
||||
private string ChecksText
|
||||
{
|
||||
get
|
||||
{
|
||||
if (Entries.Count == 0)
|
||||
return Instance.Status is null ? "—" : "no detail";
|
||||
|
||||
var healthy = Entries.Count(e => string.Equals(e.Value.Status, "Healthy", StringComparison.Ordinal));
|
||||
return $"{healthy} / {Entries.Count} healthy";
|
||||
}
|
||||
}
|
||||
|
||||
private string? ChecksClass
|
||||
{
|
||||
get
|
||||
{
|
||||
if (Entries.Count == 0)
|
||||
return null;
|
||||
|
||||
if (Entries.Any(e => string.Equals(e.Value.Status, "Unhealthy", StringComparison.Ordinal)))
|
||||
return "bad";
|
||||
|
||||
return Entries.Any(e => string.Equals(e.Value.Status, "Degraded", StringComparison.Ordinal))
|
||||
? "warn"
|
||||
: "ok";
|
||||
}
|
||||
}
|
||||
|
||||
private string RawSignals
|
||||
{
|
||||
get
|
||||
{
|
||||
var ready = Instance.Status switch
|
||||
{
|
||||
null => "—",
|
||||
InstanceStatus.Unreachable => "fail",
|
||||
InstanceStatus.Down => "503",
|
||||
_ => "200",
|
||||
};
|
||||
|
||||
var active = Instance.Active switch
|
||||
{
|
||||
ActiveState.Active => "200",
|
||||
ActiveState.Standby => "503",
|
||||
ActiveState.Unknown => "—",
|
||||
_ => null,
|
||||
};
|
||||
|
||||
return active is null ? $"ready {ready}" : $"ready {ready} · active {active}";
|
||||
}
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Renders the akka-cluster check's data as the mockup's monospace evidence line.
|
||||
/// </summary>
|
||||
private string? ClusterDataLine
|
||||
{
|
||||
get
|
||||
{
|
||||
if (Instance.Report?.Entries.GetValueOrDefault(LeaderResolver.ClusterCheckName) is not { Data: not null } entry)
|
||||
return null;
|
||||
|
||||
var parts = new List<string>();
|
||||
|
||||
void Add(string label, string? value)
|
||||
{
|
||||
if (!string.IsNullOrEmpty(value))
|
||||
parts.Add($"<b>{Escape(label)}</b> {Escape(value)}");
|
||||
}
|
||||
|
||||
Add("leader", entry.DataString(LeaderResolver.LeaderDataKey));
|
||||
Add("members", entry.DataInt("memberCount")?.ToString());
|
||||
Add("unreachable", entry.DataInt("unreachableCount")?.ToString());
|
||||
Add("self", entry.DataString("selfAddress"));
|
||||
|
||||
return parts.Count == 0 ? null : string.Join(" · ", parts);
|
||||
}
|
||||
}
|
||||
|
||||
private static string DotClass(string? status) => status switch
|
||||
{
|
||||
"Healthy" => "ok",
|
||||
"Degraded" => "warn",
|
||||
"Unhealthy" => "bad",
|
||||
_ => "idle",
|
||||
};
|
||||
|
||||
/// <summary>Formats an age the way an operator reads a dashboard: coarse and instant.</summary>
|
||||
/// <param name="age">The elapsed time.</param>
|
||||
/// <returns>A short relative string.</returns>
|
||||
internal static string Relative(TimeSpan age)
|
||||
{
|
||||
if (age < TimeSpan.Zero)
|
||||
age = TimeSpan.Zero;
|
||||
|
||||
if (age.TotalSeconds < 60)
|
||||
return $"{age.TotalSeconds:0} s ago";
|
||||
|
||||
if (age.TotalMinutes < 60)
|
||||
return $"{age.TotalMinutes:0} min ago";
|
||||
|
||||
return age.TotalHours < 24 ? $"{age.TotalHours:0} h ago" : $"{age.TotalDays:0} d ago";
|
||||
}
|
||||
|
||||
private static string Truncate(string value, int max) =>
|
||||
value.Length <= max ? value : string.Concat(value.AsSpan(0, max - 1), "…");
|
||||
|
||||
/// <summary>
|
||||
/// HTML-escapes a value before it goes into the one MarkupString on this card. Health data is
|
||||
/// remote input from another application — trusted operationally, but never trusted to be
|
||||
/// markup-safe.
|
||||
/// </summary>
|
||||
/// <param name="value">The raw value.</param>
|
||||
/// <returns>The escaped value.</returns>
|
||||
private static string Escape(string value) =>
|
||||
System.Net.WebUtility.HtmlEncode(value);
|
||||
}
|
||||
@@ -0,0 +1,10 @@
|
||||
@using Microsoft.AspNetCore.Components.Routing
|
||||
@using Microsoft.AspNetCore.Components.Web
|
||||
@using ZB.MOM.WW.Overview.Components
|
||||
@using ZB.MOM.WW.Overview.Components.Layout
|
||||
@using ZB.MOM.WW.Overview.Components.Pages
|
||||
@using ZB.MOM.WW.Overview.Components.Widgets
|
||||
@using ZB.MOM.WW.Overview.Polling
|
||||
@using ZB.MOM.WW.Overview.Registry
|
||||
@using ZB.MOM.WW.Theme
|
||||
@using static Microsoft.AspNetCore.Components.Web.RenderMode
|
||||
@@ -0,0 +1,132 @@
|
||||
namespace ZB.MOM.WW.Overview.Polling;
|
||||
|
||||
/// <summary>
|
||||
/// Derives per-group leader state from what the members of a cluster group report in their
|
||||
/// <c>akka-cluster</c> health data.
|
||||
/// </summary>
|
||||
/// <remarks>
|
||||
/// Pure and static: given the same instance snapshots it always produces the same answer, which is
|
||||
/// what makes the split-brain rule — the one output an operator would act on at 3am — exhaustively
|
||||
/// testable without a cluster.
|
||||
/// </remarks>
|
||||
public static class LeaderResolver
|
||||
{
|
||||
/// <summary>The health check whose data carries the cluster view.</summary>
|
||||
public const string ClusterCheckName = "akka-cluster";
|
||||
|
||||
/// <summary>The data key holding the Akka leader address.</summary>
|
||||
public const string LeaderDataKey = "leader";
|
||||
|
||||
/// <summary>Resolves leader state for every cluster group present in <paramref name="instances"/>.</summary>
|
||||
/// <param name="instances">One application's instances.</param>
|
||||
/// <returns>Group state in first-appearance order; empty when the application has no groups.</returns>
|
||||
public static IReadOnlyList<GroupSnapshot> Resolve(IEnumerable<InstanceSnapshot> instances)
|
||||
{
|
||||
ArgumentNullException.ThrowIfNull(instances);
|
||||
|
||||
var all = instances as IReadOnlyList<InstanceSnapshot> ?? instances.ToList();
|
||||
var groups = new List<GroupSnapshot>();
|
||||
|
||||
foreach (var name in all
|
||||
.Select(i => i.Group)
|
||||
.Where(g => !string.IsNullOrWhiteSpace(g))
|
||||
.Distinct(StringComparer.Ordinal))
|
||||
{
|
||||
var members = all.Where(i => string.Equals(i.Group, name, StringComparison.Ordinal)).ToList();
|
||||
|
||||
// A group of instances that have no active/standby concept has no leader to report;
|
||||
// rendering a leader chip for one would invent a distinction that does not exist.
|
||||
if (!members.Exists(m => m.HasActiveRole))
|
||||
continue;
|
||||
|
||||
groups.Add(ResolveGroup(name!, members, all));
|
||||
}
|
||||
|
||||
return groups;
|
||||
}
|
||||
|
||||
private static GroupSnapshot ResolveGroup(
|
||||
string name,
|
||||
IReadOnlyList<InstanceSnapshot> members,
|
||||
IReadOnlyList<InstanceSnapshot> allInstances)
|
||||
{
|
||||
// Only members that are actually talking get a vote. A Down or Unreachable node's last
|
||||
// known leader is stale by definition, and counting it would manufacture a disagreement
|
||||
// every time a node goes away mid-failover — the exact moment the flag must stay honest.
|
||||
var votes = members
|
||||
.Where(m => m.Status is InstanceStatus.Up or InstanceStatus.Degraded)
|
||||
.Select(m => m.Report?.Entries.GetValueOrDefault(ClusterCheckName)?.DataString(LeaderDataKey))
|
||||
.Where(address => !string.IsNullOrWhiteSpace(address))
|
||||
.Select(address => address!)
|
||||
.ToList();
|
||||
|
||||
if (votes.Count == 0)
|
||||
return new GroupSnapshot(name, null, null, Disagreement: false, []);
|
||||
|
||||
var distinct = votes.Distinct(StringComparer.Ordinal).OrderBy(a => a, StringComparer.Ordinal).ToList();
|
||||
|
||||
// The most-reported address wins the chip, ties broken ordinally so the display is stable
|
||||
// rather than flickering between two equally-supported claims on consecutive sweeps.
|
||||
var winner = votes
|
||||
.GroupBy(a => a, StringComparer.Ordinal)
|
||||
.OrderByDescending(g => g.Count())
|
||||
.ThenBy(g => g.Key, StringComparer.Ordinal)
|
||||
.First()
|
||||
.Key;
|
||||
|
||||
return new GroupSnapshot(
|
||||
name,
|
||||
ResolveToInstanceName(winner, allInstances),
|
||||
winner,
|
||||
Disagreement: distinct.Count > 1,
|
||||
distinct);
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Maps an Akka address to a registry instance name by host.
|
||||
/// </summary>
|
||||
/// <param name="akkaAddress">An address of the form <c>akka.tcp://system@host:port</c>.</param>
|
||||
/// <param name="instances">Candidate instances.</param>
|
||||
/// <returns>The matching instance's name, or null when nothing matches.</returns>
|
||||
/// <remarks>
|
||||
/// Host only, deliberately: the Akka remoting port and the HTTP port are different by design, so
|
||||
/// comparing ports would never match. An unmatched address is not an error — the card falls back
|
||||
/// to showing the raw address, which is still the useful thing to see.
|
||||
/// </remarks>
|
||||
internal static string? ResolveToInstanceName(string akkaAddress, IReadOnlyList<InstanceSnapshot> instances)
|
||||
{
|
||||
var host = HostOf(akkaAddress);
|
||||
if (host is null)
|
||||
return null;
|
||||
|
||||
foreach (var instance in instances)
|
||||
{
|
||||
if (Uri.TryCreate(instance.BaseUrl, UriKind.Absolute, out var uri) &&
|
||||
string.Equals(uri.Host, host, StringComparison.OrdinalIgnoreCase))
|
||||
{
|
||||
return instance.Instance;
|
||||
}
|
||||
}
|
||||
|
||||
return null;
|
||||
}
|
||||
|
||||
/// <summary>Extracts the host from an Akka address.</summary>
|
||||
/// <param name="akkaAddress">The address to parse.</param>
|
||||
/// <returns>The host, or null when the address is not in the expected form.</returns>
|
||||
internal static string? HostOf(string? akkaAddress)
|
||||
{
|
||||
if (string.IsNullOrWhiteSpace(akkaAddress))
|
||||
return null;
|
||||
|
||||
var at = akkaAddress.LastIndexOf('@');
|
||||
if (at < 0 || at == akkaAddress.Length - 1)
|
||||
return null;
|
||||
|
||||
var hostAndPort = akkaAddress[(at + 1)..];
|
||||
var colon = hostAndPort.LastIndexOf(':');
|
||||
var host = colon > 0 ? hostAndPort[..colon] : hostAndPort;
|
||||
|
||||
return host.Length == 0 ? null : host;
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,239 @@
|
||||
using Microsoft.Extensions.Options;
|
||||
using ZB.MOM.WW.Overview.Registry;
|
||||
|
||||
namespace ZB.MOM.WW.Overview.Polling;
|
||||
|
||||
/// <summary>
|
||||
/// The single writer: sweeps every registered instance on a timer and publishes each result set to
|
||||
/// <see cref="OverviewSnapshotStore"/>.
|
||||
/// </summary>
|
||||
/// <remarks>
|
||||
/// No retries. A failed probe is simply reported as failed and the next tick is the retry, which
|
||||
/// keeps the sweep bounded in time and keeps the two-strike damping in
|
||||
/// <see cref="StatusDerivation.Damp"/> the only place that decides whether a failure is worth
|
||||
/// showing. Retrying inside a sweep would silently double the effective strike count.
|
||||
/// </remarks>
|
||||
public sealed class OverviewPollerService : BackgroundService
|
||||
{
|
||||
/// <summary>Name of the pooled <see cref="HttpClient"/> used for probes.</summary>
|
||||
public const string HttpClientName = "overview-health";
|
||||
|
||||
/// <summary>
|
||||
/// Slack allowed when deciding whether an instance is due. A timer tick can arrive a hair early
|
||||
/// relative to the recorded sweep time, and without the tolerance such an instance would be
|
||||
/// skipped and end up polling at half its configured rate.
|
||||
/// </summary>
|
||||
private static readonly TimeSpan DueTolerance = TimeSpan.FromMilliseconds(250);
|
||||
|
||||
private readonly IHttpClientFactory _httpClientFactory;
|
||||
private readonly OverviewSnapshotStore _store;
|
||||
private readonly TimeProvider _timeProvider;
|
||||
private readonly ILogger<OverviewPollerService> _logger;
|
||||
private readonly List<PollTarget> _targets;
|
||||
|
||||
/// <summary>Initializes a new <see cref="OverviewPollerService"/>.</summary>
|
||||
/// <param name="options">The validated registry.</param>
|
||||
/// <param name="httpClientFactory">Factory for the pooled probe client.</param>
|
||||
/// <param name="store">The snapshot store this service writes to.</param>
|
||||
/// <param name="timeProvider">Clock and timer source.</param>
|
||||
/// <param name="logger">Logger.</param>
|
||||
public OverviewPollerService(
|
||||
IOptions<OverviewOptions> options,
|
||||
IHttpClientFactory httpClientFactory,
|
||||
OverviewSnapshotStore store,
|
||||
TimeProvider timeProvider,
|
||||
ILogger<OverviewPollerService> logger)
|
||||
{
|
||||
ArgumentNullException.ThrowIfNull(options);
|
||||
|
||||
_httpClientFactory = httpClientFactory ?? throw new ArgumentNullException(nameof(httpClientFactory));
|
||||
_store = store ?? throw new ArgumentNullException(nameof(store));
|
||||
_timeProvider = timeProvider ?? throw new ArgumentNullException(nameof(timeProvider));
|
||||
_logger = logger ?? throw new ArgumentNullException(nameof(logger));
|
||||
_targets = BuildTargets(options.Value);
|
||||
}
|
||||
|
||||
/// <inheritdoc/>
|
||||
protected override async Task ExecuteAsync(CancellationToken stoppingToken)
|
||||
{
|
||||
if (_targets.Count == 0)
|
||||
{
|
||||
_logger.LogWarning("Overview registry contains no instances; the poller has nothing to do.");
|
||||
return;
|
||||
}
|
||||
|
||||
// Tick at the fastest configured cadence and let each target decide whether it is due.
|
||||
// A single timer honours per-instance overrides without spawning a timer per instance.
|
||||
var tick = _targets.Min(t => t.Timings.PollInterval);
|
||||
|
||||
// Publish the registry before probing anything, so the page paints its full layout at once
|
||||
// instead of appearing empty for the duration of the first sweep.
|
||||
_store.Publish(BuildSnapshot(_timeProvider.GetUtcNow()));
|
||||
|
||||
using var timer = new PeriodicTimer(tick, _timeProvider);
|
||||
|
||||
try
|
||||
{
|
||||
do
|
||||
{
|
||||
await SweepAsync(stoppingToken).ConfigureAwait(false);
|
||||
}
|
||||
while (await timer.WaitForNextTickAsync(stoppingToken).ConfigureAwait(false));
|
||||
}
|
||||
catch (OperationCanceledException) when (stoppingToken.IsCancellationRequested)
|
||||
{
|
||||
// Ordinary shutdown.
|
||||
}
|
||||
}
|
||||
|
||||
/// <summary>Runs one sweep: probes every due instance and publishes the resulting snapshot.</summary>
|
||||
/// <param name="cancellationToken">Cancelled on host shutdown.</param>
|
||||
/// <returns>The snapshot that was published.</returns>
|
||||
internal async Task<OverviewSnapshot> SweepAsync(CancellationToken cancellationToken = default)
|
||||
{
|
||||
// Checked here rather than left to the transport: whether an in-flight request observes the
|
||||
// token is a property of the handler stack, but "a sweep that started after shutdown must
|
||||
// not publish" has to hold unconditionally, or the store keeps a half-probed registry as
|
||||
// its final state.
|
||||
cancellationToken.ThrowIfCancellationRequested();
|
||||
|
||||
var now = _timeProvider.GetUtcNow();
|
||||
var due = _targets.Where(t => IsDue(t, now)).ToList();
|
||||
|
||||
if (due.Count > 0)
|
||||
{
|
||||
// One client per sweep, not per probe: IHttpClientFactory pools the handler either way,
|
||||
// and a single instance keeps connection reuse across the fan-out.
|
||||
var client = _httpClientFactory.CreateClient(HttpClientName);
|
||||
var probe = new ZbHealthReportClient(client);
|
||||
|
||||
var results = await Task
|
||||
.WhenAll(due.Select(target => ProbeAsync(probe, target, now, cancellationToken)))
|
||||
.ConfigureAwait(false);
|
||||
|
||||
for (var i = 0; i < due.Count; i++)
|
||||
due[i].Last = results[i];
|
||||
}
|
||||
|
||||
// Re-checked after the fan-out: a shutdown that lands mid-sweep leaves the probed targets
|
||||
// in an arbitrary mix of fresh and stale results, and publishing that mix would freeze a
|
||||
// misleading picture into the store for as long as the process takes to exit.
|
||||
cancellationToken.ThrowIfCancellationRequested();
|
||||
|
||||
var snapshot = BuildSnapshot(now);
|
||||
_store.Publish(snapshot);
|
||||
|
||||
return snapshot;
|
||||
}
|
||||
|
||||
private bool IsDue(PollTarget target, DateTimeOffset now) =>
|
||||
target.Last?.LastPolledUtc is not { } polled || now - polled + DueTolerance >= target.Timings.PollInterval;
|
||||
|
||||
private static async Task<InstanceSnapshot> ProbeAsync(
|
||||
ZbHealthReportClient probe,
|
||||
PollTarget target,
|
||||
DateTimeOffset now,
|
||||
CancellationToken cancellationToken)
|
||||
{
|
||||
var timeout = target.Timings.Timeout;
|
||||
var ready = await probe.ProbeAsync(target.ReadyUrl, timeout, cancellationToken).ConfigureAwait(false);
|
||||
|
||||
// Skip the active probe when nothing answered the ready probe: it would spend a second full
|
||||
// timeout to learn what we already know, and ObserveActive maps a missing probe to Unknown
|
||||
// anyway. A 503 from ready is NOT a skip — that instance is talking, and whether it is the
|
||||
// active half is exactly what an operator wants to know about a sick node.
|
||||
HealthProbeResult? active = null;
|
||||
if (target.Instance.HasActiveRole && ready.Reachable)
|
||||
active = await probe.ProbeStatusAsync(target.ActiveUrl, timeout, cancellationToken).ConfigureAwait(false);
|
||||
|
||||
var observed = StatusDerivation.Observe(ready);
|
||||
var (status, failures) = StatusDerivation.Damp(
|
||||
target.Last?.Status,
|
||||
target.Last?.ConsecutiveFailures ?? 0,
|
||||
observed);
|
||||
|
||||
return target.Template with
|
||||
{
|
||||
Status = status,
|
||||
Active = StatusDerivation.ObserveActive(target.Instance.HasActiveRole, active),
|
||||
ConsecutiveFailures = failures,
|
||||
LastPolledUtc = now,
|
||||
LastReachableUtc = ready.Reachable ? now : target.Last?.LastReachableUtc,
|
||||
Latency = ready.Latency,
|
||||
Error = ready.Error,
|
||||
Report = ready.Report,
|
||||
};
|
||||
}
|
||||
|
||||
private OverviewSnapshot BuildSnapshot(DateTimeOffset now)
|
||||
{
|
||||
var applications = _targets
|
||||
.GroupBy(t => t.Application, StringComparer.Ordinal)
|
||||
.Select(group =>
|
||||
{
|
||||
var instances = group.Select(t => t.Last ?? t.Template).ToList();
|
||||
return new ApplicationSnapshot(group.Key, instances, LeaderResolver.Resolve(instances));
|
||||
})
|
||||
.ToList();
|
||||
|
||||
return new OverviewSnapshot(now, applications);
|
||||
}
|
||||
|
||||
private static List<PollTarget> BuildTargets(OverviewOptions options)
|
||||
{
|
||||
var targets = new List<PollTarget>();
|
||||
|
||||
foreach (var application in options.Applications)
|
||||
{
|
||||
foreach (var instance in application.Instances)
|
||||
{
|
||||
var baseUrl = instance.BaseUrl.TrimEnd('/');
|
||||
|
||||
targets.Add(new PollTarget
|
||||
{
|
||||
Application = application.Name,
|
||||
Instance = instance,
|
||||
Timings = EffectiveTimings.Resolve(options, application, instance),
|
||||
ReadyUrl = $"{baseUrl}/health/ready",
|
||||
ActiveUrl = $"{baseUrl}/health/active",
|
||||
Template = new InstanceSnapshot
|
||||
{
|
||||
Application = application.Name,
|
||||
Instance = instance.Name,
|
||||
Group = instance.Group,
|
||||
BaseUrl = baseUrl,
|
||||
ManagementUrl = instance.ManagementUrl,
|
||||
ManagementLabel = application.ManagementLabel,
|
||||
HasActiveRole = instance.HasActiveRole,
|
||||
StaleAfter = EffectiveTimings.Resolve(options, application, instance).StaleAfter,
|
||||
},
|
||||
});
|
||||
}
|
||||
}
|
||||
|
||||
return targets;
|
||||
}
|
||||
|
||||
/// <summary>One instance's fixed probe configuration plus its most recent result.</summary>
|
||||
private sealed class PollTarget
|
||||
{
|
||||
public required string Application { get; init; }
|
||||
|
||||
public required InstanceEntry Instance { get; init; }
|
||||
|
||||
public required EffectiveTimings Timings { get; init; }
|
||||
|
||||
public required string ReadyUrl { get; init; }
|
||||
|
||||
public required string ActiveUrl { get; init; }
|
||||
|
||||
/// <summary>The registry-derived fields, reused as the base of every result.</summary>
|
||||
public required InstanceSnapshot Template { get; init; }
|
||||
|
||||
/// <summary>
|
||||
/// The last published result, or null before the first probe. Written only by the sweep
|
||||
/// loop after its <c>Task.WhenAll</c> has completed, so it is never touched concurrently.
|
||||
/// </summary>
|
||||
public InstanceSnapshot? Last { get; set; }
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,172 @@
|
||||
namespace ZB.MOM.WW.Overview.Polling;
|
||||
|
||||
/// <summary>
|
||||
/// Everything the dashboard knows about one instance, as of the last sweep that included it.
|
||||
/// </summary>
|
||||
/// <remarks>
|
||||
/// Immutable by construction: the poller builds a whole new object graph each sweep and swaps it
|
||||
/// into <see cref="OverviewSnapshotStore"/> atomically, so a render can never observe a half-updated
|
||||
/// registry.
|
||||
/// </remarks>
|
||||
public sealed record InstanceSnapshot
|
||||
{
|
||||
/// <summary>Owning application's display name.</summary>
|
||||
public required string Application { get; init; }
|
||||
|
||||
/// <summary>This instance's display name, unique within its application.</summary>
|
||||
public required string Instance { get; init; }
|
||||
|
||||
/// <summary>Cluster group, or null for a standalone instance.</summary>
|
||||
public string? Group { get; init; }
|
||||
|
||||
/// <summary>The probed base URL, shown on the card for diagnosis.</summary>
|
||||
public required string BaseUrl { get; init; }
|
||||
|
||||
/// <summary>Management UI link, when the registry supplies one.</summary>
|
||||
public string? ManagementUrl { get; init; }
|
||||
|
||||
/// <summary>Label for <see cref="ManagementUrl"/>, from the application entry.</summary>
|
||||
public string ManagementLabel { get; init; } = "UI";
|
||||
|
||||
/// <summary>Whether this instance is half of an active/standby pair.</summary>
|
||||
public bool HasActiveRole { get; init; }
|
||||
|
||||
/// <summary>
|
||||
/// The damped, displayable status — null until the first sweep has actually probed this
|
||||
/// instance.
|
||||
/// </summary>
|
||||
/// <remarks>
|
||||
/// Null is a real, distinct state and not a stand-in for "Up": the registry is published to the
|
||||
/// store before the first probe returns so the page renders its full layout immediately, and
|
||||
/// showing an unmeasured instance as healthy for that half-second would be a fabricated reading.
|
||||
/// </remarks>
|
||||
public InstanceStatus? Status { get; init; }
|
||||
|
||||
/// <summary>Active/standby state from the active tier.</summary>
|
||||
public ActiveState Active { get; init; } = ActiveState.NotApplicable;
|
||||
|
||||
/// <summary>Consecutive failing observations, carried across sweeps to drive flap damping.</summary>
|
||||
public int ConsecutiveFailures { get; init; }
|
||||
|
||||
/// <summary>
|
||||
/// When the sweep that produced this snapshot started; null before the first probe. Staleness
|
||||
/// is measured from here, so a stuck or paused poller ages every card even though no probe
|
||||
/// failed.
|
||||
/// </summary>
|
||||
public DateTimeOffset? LastPolledUtc { get; init; }
|
||||
|
||||
/// <summary>When this instance last answered a probe at all, regardless of what it said.</summary>
|
||||
public DateTimeOffset? LastReachableUtc { get; init; }
|
||||
|
||||
/// <summary>Wall-clock duration of the last ready probe.</summary>
|
||||
public TimeSpan Latency { get; init; }
|
||||
|
||||
/// <summary>Why the last probe failed, when it did.</summary>
|
||||
public string? Error { get; init; }
|
||||
|
||||
/// <summary>The last parsed ready-tier body, which drives the per-check detail list.</summary>
|
||||
public ZbHealthReport? Report { get; init; }
|
||||
|
||||
/// <summary>This instance's effective staleness window.</summary>
|
||||
public TimeSpan StaleAfter { get; init; }
|
||||
|
||||
/// <summary>Whether this snapshot is too old to be trusted as current.</summary>
|
||||
/// <param name="now">The current time.</param>
|
||||
/// <returns>True when the snapshot has aged past its staleness window.</returns>
|
||||
/// <remarks>
|
||||
/// Evaluated at render rather than stored, because staleness is a function of the clock and not
|
||||
/// of anything the poller observed — a snapshot that was fresh when written goes stale on its
|
||||
/// own while nothing at all happens.
|
||||
/// </remarks>
|
||||
public bool IsStale(DateTimeOffset now) =>
|
||||
Status is not null && LastPolledUtc is { } polled && now - polled > StaleAfter;
|
||||
}
|
||||
|
||||
/// <summary>Leader state for one cluster group, derived from what its members report.</summary>
|
||||
/// <param name="Name">The group name from the registry.</param>
|
||||
/// <param name="LeaderInstance">
|
||||
/// Registry name of the instance the reported leader address resolves to, or null when no member
|
||||
/// reported a leader or the address matches no registered instance.
|
||||
/// </param>
|
||||
/// <param name="LeaderAddress">The reported Akka leader address, shown raw when unresolved.</param>
|
||||
/// <param name="Disagreement">
|
||||
/// True when reachable members reported different leaders — the split-brain tell.
|
||||
/// </param>
|
||||
/// <param name="ReportedLeaders">Every distinct address reported, for the disagreement detail.</param>
|
||||
public sealed record GroupSnapshot(
|
||||
string Name,
|
||||
string? LeaderInstance,
|
||||
string? LeaderAddress,
|
||||
bool Disagreement,
|
||||
IReadOnlyList<string> ReportedLeaders);
|
||||
|
||||
/// <summary>One application section of the dashboard.</summary>
|
||||
/// <param name="Name">The application's display name.</param>
|
||||
/// <param name="Instances">Its instances, in registry order.</param>
|
||||
/// <param name="Groups">Leader state per cluster group, in registry order.</param>
|
||||
public sealed record ApplicationSnapshot(
|
||||
string Name,
|
||||
IReadOnlyList<InstanceSnapshot> Instances,
|
||||
IReadOnlyList<GroupSnapshot> Groups)
|
||||
{
|
||||
/// <summary>Finds the group state for <paramref name="group"/>.</summary>
|
||||
/// <param name="group">The group name, or null for standalone instances.</param>
|
||||
/// <returns>The group snapshot, or null when there is none.</returns>
|
||||
public GroupSnapshot? GroupFor(string? group) =>
|
||||
group is null ? null : Groups.FirstOrDefault(g => string.Equals(g.Name, group, StringComparison.Ordinal));
|
||||
|
||||
/// <summary>
|
||||
/// The most severe status across this application's probed instances, for the rollup pill;
|
||||
/// null while nothing has been probed yet.
|
||||
/// </summary>
|
||||
public InstanceStatus? Worst
|
||||
{
|
||||
get
|
||||
{
|
||||
InstanceStatus? worst = null;
|
||||
foreach (var instance in Instances)
|
||||
{
|
||||
if (instance.Status is not { } status)
|
||||
continue;
|
||||
|
||||
if (worst is null || StatusSeverity.Rank(status) > StatusSeverity.Rank(worst.Value))
|
||||
worst = status;
|
||||
}
|
||||
|
||||
return worst;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// <summary>A complete, atomically-published view of the whole registry.</summary>
|
||||
/// <param name="GeneratedUtc">When the sweep that produced this snapshot started.</param>
|
||||
/// <param name="Applications">The applications, in registry order.</param>
|
||||
public sealed record OverviewSnapshot(DateTimeOffset GeneratedUtc, IReadOnlyList<ApplicationSnapshot> Applications)
|
||||
{
|
||||
/// <summary>The pre-registry snapshot served before the poller has published anything.</summary>
|
||||
public static OverviewSnapshot Empty { get; } = new(default, []);
|
||||
|
||||
/// <summary>Every instance across every application, flattened.</summary>
|
||||
public IEnumerable<InstanceSnapshot> AllInstances => Applications.SelectMany(a => a.Instances);
|
||||
}
|
||||
|
||||
/// <summary>Severity ordering for rollups.</summary>
|
||||
public static class StatusSeverity
|
||||
{
|
||||
/// <summary>Ranks a status; higher is worse.</summary>
|
||||
/// <param name="status">The status to rank.</param>
|
||||
/// <returns>The severity rank.</returns>
|
||||
/// <remarks>
|
||||
/// <see cref="InstanceStatus.Unreachable"/> outranks <see cref="InstanceStatus.Down"/> because
|
||||
/// it carries strictly less information: a Down instance told us what is wrong, an Unreachable
|
||||
/// one did not answer at all and could be anything up to and including gone.
|
||||
/// </remarks>
|
||||
public static int Rank(InstanceStatus status) => status switch
|
||||
{
|
||||
InstanceStatus.Up => 0,
|
||||
InstanceStatus.Degraded => 1,
|
||||
InstanceStatus.Down => 2,
|
||||
InstanceStatus.Unreachable => 3,
|
||||
_ => 3,
|
||||
};
|
||||
}
|
||||
@@ -0,0 +1,38 @@
|
||||
namespace ZB.MOM.WW.Overview.Polling;
|
||||
|
||||
/// <summary>
|
||||
/// Holds the current <see cref="OverviewSnapshot"/> and notifies subscribers when it is replaced.
|
||||
/// </summary>
|
||||
/// <remarks>
|
||||
/// This is the cache-first requirement made concrete: the poller is the only writer, and every
|
||||
/// render reads whatever is already here. A page load therefore paints the last known state
|
||||
/// immediately and never waits on — or triggers — a probe of its own.
|
||||
/// </remarks>
|
||||
public sealed class OverviewSnapshotStore
|
||||
{
|
||||
private OverviewSnapshot _current = OverviewSnapshot.Empty;
|
||||
|
||||
/// <summary>Raised after a new snapshot has been published.</summary>
|
||||
/// <remarks>
|
||||
/// Handlers run on the poller's thread, so a Blazor subscriber must marshal onto its own
|
||||
/// synchronization context (<c>InvokeAsync(StateHasChanged)</c>) and must unsubscribe when
|
||||
/// disposed.
|
||||
/// </remarks>
|
||||
public event Action<OverviewSnapshot>? Changed;
|
||||
|
||||
/// <summary>The current snapshot; never null.</summary>
|
||||
public OverviewSnapshot Current => Volatile.Read(ref _current);
|
||||
|
||||
/// <summary>Replaces the current snapshot and notifies subscribers.</summary>
|
||||
/// <param name="snapshot">The new snapshot.</param>
|
||||
public void Publish(OverviewSnapshot snapshot)
|
||||
{
|
||||
ArgumentNullException.ThrowIfNull(snapshot);
|
||||
|
||||
// A single reference swap of an already-complete graph. Readers hold their own reference
|
||||
// for the duration of a render, so a sweep landing mid-render cannot tear the view.
|
||||
Volatile.Write(ref _current, snapshot);
|
||||
|
||||
Changed?.Invoke(snapshot);
|
||||
}
|
||||
}
|
||||
+33
@@ -0,0 +1,33 @@
|
||||
using Microsoft.Extensions.DependencyInjection.Extensions;
|
||||
|
||||
namespace ZB.MOM.WW.Overview.Polling;
|
||||
|
||||
/// <summary>Registers the polling pipeline.</summary>
|
||||
public static class PollingServiceCollectionExtensions
|
||||
{
|
||||
/// <summary>Adds the snapshot store, the probe client and the poller.</summary>
|
||||
/// <param name="services">The service collection.</param>
|
||||
/// <returns>The same collection, for chaining.</returns>
|
||||
public static IServiceCollection AddOverviewPolling(this IServiceCollection services)
|
||||
{
|
||||
ArgumentNullException.ThrowIfNull(services);
|
||||
|
||||
services.TryAddSingleton(TimeProvider.System);
|
||||
services.TryAddSingleton<OverviewSnapshotStore>();
|
||||
|
||||
services.AddHttpClient(OverviewPollerService.HttpClientName, client =>
|
||||
{
|
||||
// The per-request CancellationTokenSource in ZbHealthReportClient is the only timeout
|
||||
// that should ever fire. Leaving HttpClient's own 100s default in place would let it
|
||||
// pre-empt a per-instance override and report "timed out after 3s" for a 100s wait.
|
||||
client.Timeout = Timeout.InfiniteTimeSpan;
|
||||
|
||||
// Health bodies are small and must not be served from anywhere but the instance itself.
|
||||
client.DefaultRequestHeaders.CacheControl = new() { NoCache = true, NoStore = true };
|
||||
});
|
||||
|
||||
services.AddHostedService<OverviewPollerService>();
|
||||
|
||||
return services;
|
||||
}
|
||||
}
|
||||
@@ -44,15 +44,43 @@ public sealed class ZbHealthReportClient
|
||||
public ZbHealthReportClient(HttpClient httpClient) =>
|
||||
_httpClient = httpClient ?? throw new ArgumentNullException(nameof(httpClient));
|
||||
|
||||
/// <summary>Probes <paramref name="url"/> and parses the response.</summary>
|
||||
/// <summary>Probes <paramref name="url"/> and parses the response body.</summary>
|
||||
/// <param name="url">Absolute URL of a health endpoint.</param>
|
||||
/// <param name="timeout">Per-request timeout.</param>
|
||||
/// <param name="cancellationToken">Token cancelled when the host is shutting down.</param>
|
||||
/// <returns>The probe result; this never throws for an unreachable endpoint.</returns>
|
||||
public async Task<HealthProbeResult> ProbeAsync(
|
||||
public Task<HealthProbeResult> ProbeAsync(
|
||||
string url,
|
||||
TimeSpan timeout,
|
||||
CancellationToken cancellationToken = default)
|
||||
CancellationToken cancellationToken = default) =>
|
||||
SendAsync(url, timeout, bodyRequired: true, cancellationToken);
|
||||
|
||||
/// <summary>
|
||||
/// Probes <paramref name="url"/> for its status code alone, parsing the body only if one
|
||||
/// happens to be there.
|
||||
/// </summary>
|
||||
/// <param name="url">Absolute URL of a health endpoint.</param>
|
||||
/// <param name="timeout">Per-request timeout.</param>
|
||||
/// <param name="cancellationToken">Token cancelled when the host is shutting down.</param>
|
||||
/// <returns>The probe result; this never throws for an unreachable endpoint.</returns>
|
||||
/// <remarks>
|
||||
/// For the active tier the status code IS the answer — 200 means active, 503 means standby —
|
||||
/// and the body carries nothing the dashboard uses. Demanding a parseable body here would
|
||||
/// report every node behind anything that answers the tier with an empty response (a bare
|
||||
/// <c>IActiveNodeGate</c> endpoint, a load balancer, a proxy) as "role unknown" while it is
|
||||
/// plainly telling us the role.
|
||||
/// </remarks>
|
||||
public Task<HealthProbeResult> ProbeStatusAsync(
|
||||
string url,
|
||||
TimeSpan timeout,
|
||||
CancellationToken cancellationToken = default) =>
|
||||
SendAsync(url, timeout, bodyRequired: false, cancellationToken);
|
||||
|
||||
private async Task<HealthProbeResult> SendAsync(
|
||||
string url,
|
||||
TimeSpan timeout,
|
||||
bool bodyRequired,
|
||||
CancellationToken cancellationToken)
|
||||
{
|
||||
var startedAt = Stopwatch.GetTimestamp();
|
||||
|
||||
@@ -81,10 +109,12 @@ public sealed class ZbHealthReportClient
|
||||
// Answered, but not with this contract — a reverse proxy error page, an HTML login
|
||||
// redirect, a wrong port. That is a registry/plumbing problem, not a sick app, so
|
||||
// it lands as Unreachable rather than Down.
|
||||
return Failed(startedAt, statusCode, $"response body is not canonical health JSON: {ex.Message}");
|
||||
return bodyRequired
|
||||
? Failed(startedAt, statusCode, $"response body is not canonical health JSON: {ex.Message}")
|
||||
: new HealthProbeResult(true, statusCode, null, Stopwatch.GetElapsedTime(startedAt), null);
|
||||
}
|
||||
|
||||
if (report is null)
|
||||
if (report is null && bodyRequired)
|
||||
return Failed(startedAt, statusCode, "response body deserialized to null");
|
||||
|
||||
return new HealthProbeResult(true, statusCode, report, Stopwatch.GetElapsedTime(startedAt), null);
|
||||
|
||||
@@ -1,4 +1,6 @@
|
||||
using ZB.MOM.WW.Configuration;
|
||||
using ZB.MOM.WW.Overview.Components;
|
||||
using ZB.MOM.WW.Overview.Polling;
|
||||
using ZB.MOM.WW.Overview.Registry;
|
||||
|
||||
var builder = WebApplication.CreateBuilder(args);
|
||||
@@ -15,9 +17,29 @@ ConfigPreflight.For(builder.Configuration)
|
||||
builder.Services.AddValidatedOptions<OverviewOptions, OverviewOptionsValidator>(
|
||||
builder.Configuration, OverviewOptions.SectionName);
|
||||
|
||||
builder.Services.AddOverviewPolling();
|
||||
|
||||
builder.Services.AddRazorComponents()
|
||||
.AddInteractiveServerComponents();
|
||||
|
||||
var app = builder.Build();
|
||||
|
||||
app.MapGet("/", () => "ZB.MOM.WW.Overview");
|
||||
app.UseStaticFiles();
|
||||
|
||||
// No UseAuthentication/UseAuthorization and no cascading auth state: the dashboard is anonymous and
|
||||
// read-only by requirement (design §2). It renders health data that every one of these apps already
|
||||
// serves unauthenticated, and it has no endpoint that mutates anything.
|
||||
//
|
||||
// UseAntiforgery IS required despite that. AddRazorComponents stamps antiforgery metadata onto every
|
||||
// component endpoint unconditionally, and the endpoint middleware hard-fails a request whose endpoint
|
||||
// carries that metadata with no middleware to honour it — so without this line every page returns
|
||||
// 500. The alternative, MapRazorComponents(...).DisableAntiforgery(), would also boot, but it turns
|
||||
// the first form anyone ever adds into a silent CSRF hole. This app has no forms, so the middleware
|
||||
// is inert; it is here so that stays true by default rather than by vigilance.
|
||||
app.UseAntiforgery();
|
||||
|
||||
app.MapRazorComponents<App>()
|
||||
.AddInteractiveServerRenderMode();
|
||||
|
||||
await app.RunAsync();
|
||||
|
||||
|
||||
@@ -0,0 +1,112 @@
|
||||
/* ═══════════════════════════════════════════════════════════════════════════
|
||||
ZB.MOM.WW.Overview — app-local styles.
|
||||
|
||||
Everything here is what the shared ZB.MOM.WW.Theme kit does NOT already
|
||||
provide. The kit owns the design tokens, the shell, and the generic
|
||||
vocabulary this file builds on: .chip-*, .panel, .panel-foot, .kv, .agg-*,
|
||||
.card-grid, .conn-pill, .rail-btn, .mono, .rise. None of those are
|
||||
redefined here — this file adds only the dashboard-specific anatomy from
|
||||
docs/mockups/overview-dashboard-mockup.html (page-head, app/group headers,
|
||||
the instance card, the check-detail list).
|
||||
═══════════════════════════════════════════════════════════════════════════ */
|
||||
|
||||
/* ── Page header ────────────────────────────────────────────────────────── */
|
||||
.page-head { display: flex; align-items: baseline; gap: 1rem; flex-wrap: wrap; margin: 0 0 1rem; }
|
||||
.page-head h1 { font-size: 1.25rem; font-weight: 600; letter-spacing: 0.01em; margin: 0; text-wrap: balance; }
|
||||
.page-head .spacer { flex: 1; }
|
||||
.page-meta { font-family: var(--mono); font-size: 0.78rem; color: var(--ink-soft); }
|
||||
|
||||
/* KPI variants the kit does not carry (it ships .alert / .caution only). */
|
||||
.agg-card.good .agg-value { color: var(--ok); }
|
||||
.agg-card.quiet .agg-value { color: var(--idle); }
|
||||
.kv .v.idle { color: var(--idle); }
|
||||
|
||||
/* ── Application sections ───────────────────────────────────────────────── */
|
||||
.app-section { margin-bottom: 1.5rem; }
|
||||
.app-head {
|
||||
display: flex; align-items: baseline; gap: 0.6rem; flex-wrap: wrap;
|
||||
padding-bottom: 0.4rem; border-bottom: 1px solid var(--rule-strong); margin-bottom: 0.75rem;
|
||||
}
|
||||
.app-head h2 { font-size: 1.02rem; font-weight: 600; margin: 0; }
|
||||
.app-head .spacer { flex: 1; }
|
||||
.app-meta { font-family: var(--mono); font-size: 0.76rem; color: var(--ink-faint); }
|
||||
|
||||
.group { margin-bottom: 0.9rem; }
|
||||
.group-head { display: flex; align-items: center; gap: 0.55rem; flex-wrap: wrap; margin-bottom: 0.5rem; }
|
||||
.group-name {
|
||||
font-size: 0.74rem; font-weight: 600; text-transform: uppercase;
|
||||
letter-spacing: 0.07em; color: var(--ink-soft);
|
||||
}
|
||||
.group-meta { font-family: var(--mono); font-size: 0.74rem; color: var(--ink-faint); }
|
||||
|
||||
/* ── Instance card ──────────────────────────────────────────────────────── */
|
||||
.inst {
|
||||
background: var(--card); border: 1px solid var(--rule); border-radius: 8px;
|
||||
overflow: hidden; display: flex; flex-direction: column;
|
||||
}
|
||||
|
||||
/* Status is carried by the top stripe. Unreachable additionally goes DASHED:
|
||||
colour alone cannot distinguish it from Down, and the two send an operator
|
||||
to completely different places (network vs. the application itself). */
|
||||
.inst.up { border-top: 3px solid var(--ok); }
|
||||
.inst.degraded { border-top: 3px solid var(--warn); }
|
||||
.inst.down { border-top: 3px solid var(--bad); }
|
||||
.inst.unreachable { border-top: 3px solid var(--bad); border-style: dashed; border-top-style: solid; }
|
||||
.inst.pending { border-top: 3px solid var(--rule-strong); }
|
||||
.inst.stale { border-top: 3px solid var(--idle); }
|
||||
.inst.stale .inst-body,
|
||||
.inst.stale .inst-head .inst-host { opacity: 0.55; }
|
||||
|
||||
.inst-head {
|
||||
display: flex; align-items: flex-start; justify-content: space-between;
|
||||
gap: 0.6rem; padding: 0.6rem 0.9rem 0.45rem;
|
||||
}
|
||||
.inst-name { font-weight: 600; font-size: 0.92rem; }
|
||||
.inst-host {
|
||||
display: block; font-family: var(--mono); font-size: 0.72rem;
|
||||
color: var(--ink-faint); margin-top: 0.15rem;
|
||||
}
|
||||
.inst-chips { display: flex; gap: 0.3rem; flex-wrap: wrap; justify-content: flex-end; }
|
||||
|
||||
.inst .panel-foot {
|
||||
margin-top: auto; display: flex; align-items: center;
|
||||
justify-content: space-between; gap: 0.6rem;
|
||||
}
|
||||
.foot-note { font-family: var(--mono); font-size: 0.72rem; color: var(--ink-faint); }
|
||||
|
||||
/* ── Check detail (native <details>, no JS) ─────────────────────────────── */
|
||||
details.checks { border-top: 1px solid var(--rule); }
|
||||
details.checks summary {
|
||||
list-style: none; cursor: pointer; padding: 0.45rem 0.9rem; font-size: 0.72rem;
|
||||
font-weight: 600; text-transform: uppercase; letter-spacing: 0.06em;
|
||||
color: var(--ink-faint); user-select: none;
|
||||
}
|
||||
details.checks summary::-webkit-details-marker { display: none; }
|
||||
details.checks summary::before { content: '\25B6'; font-size: 0.55rem; margin-right: 0.4rem; }
|
||||
details.checks[open] summary::before { content: '\25BC'; }
|
||||
details.checks summary:hover { color: var(--ink); }
|
||||
|
||||
.check-row { display: flex; align-items: baseline; gap: 0.5rem; padding: 0.28rem 0.9rem; font-size: 0.8rem; }
|
||||
.check-row:nth-child(even) { background: var(--zebra-bg); }
|
||||
.check-name { font-family: var(--mono); font-size: 0.78rem; min-width: 9.5rem; }
|
||||
.check-desc { color: var(--ink-soft); font-size: 0.78rem; }
|
||||
.check-dot { width: 8px; height: 8px; border-radius: 2px; flex: 0 0 8px; align-self: center; }
|
||||
.check-dot.ok { background: var(--ok); }
|
||||
.check-dot.warn { background: var(--warn); }
|
||||
.check-dot.bad { background: var(--bad); }
|
||||
.check-dot.idle { background: var(--idle); }
|
||||
|
||||
.check-data {
|
||||
margin: 0.35rem 0.9rem 0.6rem; padding: 0.4rem 0.6rem; background: var(--zebra-bg);
|
||||
border: 1px solid var(--rule); border-radius: 4px;
|
||||
font-family: var(--mono); font-size: 0.72rem; color: var(--ink-soft); line-height: 1.6;
|
||||
/* Scrolls inside its own box: an Akka address plus member counts is wider than a
|
||||
310px card, and letting it widen the card would break the whole grid. */
|
||||
overflow-x: auto; white-space: nowrap;
|
||||
}
|
||||
.check-data b { color: var(--accent-deep); font-weight: 500; }
|
||||
|
||||
.rail-note {
|
||||
padding: 0 0.6rem 0.6rem; font-family: var(--mono); font-size: 0.72rem;
|
||||
color: var(--ink-faint); line-height: 1.5;
|
||||
}
|
||||
+6
File diff suppressed because one or more lines are too long
+7
File diff suppressed because one or more lines are too long
@@ -0,0 +1,249 @@
|
||||
using Bunit;
|
||||
using ZB.MOM.WW.Overview.Components.Widgets;
|
||||
using ZB.MOM.WW.Overview.Polling;
|
||||
|
||||
namespace ZB.MOM.WW.Overview.Tests;
|
||||
|
||||
/// <summary>
|
||||
/// The instance card is where a reading becomes a decision, so the tests pin what an operator can
|
||||
/// actually distinguish at a glance: which of the four states is shown, whether it is still
|
||||
/// current, and whether the raw evidence behind it is visible.
|
||||
/// </summary>
|
||||
public class InstanceCardTests : TestContext
|
||||
{
|
||||
private static readonly DateTimeOffset Now = new(2026, 7, 24, 12, 0, 0, TimeSpan.Zero);
|
||||
|
||||
private static InstanceSnapshot Card(
|
||||
InstanceStatus? status = InstanceStatus.Up,
|
||||
ActiveState active = ActiveState.NotApplicable,
|
||||
DateTimeOffset? lastPolled = null,
|
||||
DateTimeOffset? lastReachable = null,
|
||||
TimeSpan? staleAfter = null,
|
||||
string? error = null,
|
||||
ZbHealthReport? report = null,
|
||||
string? managementUrl = null) =>
|
||||
new()
|
||||
{
|
||||
Application = "ScadaBridge",
|
||||
Instance = "site-a-a",
|
||||
Group = "site-a",
|
||||
BaseUrl = "http://site-a-a:8084",
|
||||
ManagementUrl = managementUrl,
|
||||
ManagementLabel = "AdminUI",
|
||||
HasActiveRole = active != ActiveState.NotApplicable,
|
||||
Status = status,
|
||||
Active = active,
|
||||
LastPolledUtc = lastPolled ?? Now,
|
||||
LastReachableUtc = lastReachable ?? Now,
|
||||
Latency = TimeSpan.FromMilliseconds(18),
|
||||
Error = error,
|
||||
Report = report,
|
||||
StaleAfter = staleAfter ?? TimeSpan.FromSeconds(45),
|
||||
};
|
||||
|
||||
private IRenderedComponent<InstanceCard> Render(InstanceSnapshot instance, bool isLeader = false) =>
|
||||
RenderComponent<InstanceCard>(p => p
|
||||
.Add(c => c.Instance, instance)
|
||||
.Add(c => c.Now, Now)
|
||||
.Add(c => c.IsLeader, isLeader));
|
||||
|
||||
[Theory]
|
||||
[InlineData(InstanceStatus.Up, "up", "Up")]
|
||||
[InlineData(InstanceStatus.Degraded, "degraded", "Degraded")]
|
||||
[InlineData(InstanceStatus.Down, "down", "Down")]
|
||||
[InlineData(InstanceStatus.Unreachable, "unreachable", "Unreachable")]
|
||||
public void Card_CarriesItsStatusAsBothClassAndLabel(InstanceStatus status, string cssClass, string label)
|
||||
{
|
||||
var card = Render(Card(status));
|
||||
|
||||
Assert.Contains(cssClass, card.Find("article").ClassList, StringComparer.Ordinal);
|
||||
Assert.Contains(label, card.Markup, StringComparison.Ordinal);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void UnreachableAndDown_AreVisuallyDistinct_NotJustDifferentWords()
|
||||
{
|
||||
// The CSS gives Unreachable a dashed border precisely because both are red. If the class
|
||||
// did not differ, the two would be indistinguishable on a wall display.
|
||||
var down = Render(Card(InstanceStatus.Down)).Find("article").ClassList;
|
||||
var unreachable = Render(Card(InstanceStatus.Unreachable)).Find("article").ClassList;
|
||||
|
||||
Assert.NotEqual(down, unreachable);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void NeverPolled_RendersPending_NotHealthy()
|
||||
{
|
||||
var card = Render(Card(status: null, lastPolled: null));
|
||||
|
||||
Assert.Contains("pending", card.Find("article").ClassList, StringComparer.Ordinal);
|
||||
Assert.Contains("Pending", card.Markup, StringComparison.Ordinal);
|
||||
Assert.DoesNotContain(">Up<", card.Markup, StringComparison.Ordinal);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void StaleCard_IsMarkedStaleButKeepsItsLastKnownStatusClass()
|
||||
{
|
||||
// Both facts matter: the reading was Up, and it is no longer current. Dropping either one
|
||||
// loses information the operator needs.
|
||||
var card = Render(Card(lastPolled: Now.AddMinutes(-5)));
|
||||
|
||||
var classes = card.Find("article").ClassList;
|
||||
Assert.Contains("stale", classes, StringComparer.Ordinal);
|
||||
Assert.Contains("up", classes, StringComparer.Ordinal);
|
||||
Assert.Contains("Stale", card.Markup, StringComparison.Ordinal);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void StaleCard_CountsFromLastContact_NotFromTheLastPollAttempt()
|
||||
{
|
||||
// The poller kept trying for ten minutes and kept failing. "Stale since 5 s ago" would
|
||||
// suggest the instance was fine moments ago; it has actually been gone for ten minutes.
|
||||
var card = Render(Card(
|
||||
status: InstanceStatus.Unreachable,
|
||||
lastPolled: Now.AddSeconds(-5),
|
||||
lastReachable: Now.AddMinutes(-10),
|
||||
staleAfter: TimeSpan.FromSeconds(1)));
|
||||
|
||||
Assert.Contains("Stale since", card.Markup, StringComparison.Ordinal);
|
||||
Assert.Contains("10 min ago", card.Markup, StringComparison.Ordinal);
|
||||
}
|
||||
|
||||
[Theory]
|
||||
[InlineData(ActiveState.Active, "Active")]
|
||||
[InlineData(ActiveState.Standby, "Standby")]
|
||||
[InlineData(ActiveState.Unknown, "Role ?")]
|
||||
public void ActiveState_IsBadged(ActiveState state, string expected)
|
||||
{
|
||||
Assert.Contains(expected, Render(Card(active: state)).Markup, StringComparison.Ordinal);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void NoActiveRole_ShowsNoActivityBadgeAtAll()
|
||||
{
|
||||
// A single-instance gateway has no active/standby concept; a badge would invent one.
|
||||
var markup = Render(Card(active: ActiveState.NotApplicable)).Markup;
|
||||
|
||||
Assert.DoesNotContain("Standby", markup, StringComparison.Ordinal);
|
||||
Assert.DoesNotContain("Role ?", markup, StringComparison.Ordinal);
|
||||
Assert.DoesNotContain("· active", markup, StringComparison.Ordinal);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void LeaderBadge_IsShownOnlyWhenTheCardIsTheLeader()
|
||||
{
|
||||
Assert.Contains("Leader", Render(Card(), isLeader: true).Markup, StringComparison.Ordinal);
|
||||
Assert.DoesNotContain("Leader", Render(Card(), isLeader: false).Markup, StringComparison.Ordinal);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void Checks_AreListedWithAHealthyCount()
|
||||
{
|
||||
var card = Render(Card(report: Report(("localdb", "Healthy", null), ("akka-cluster", "Unhealthy", null))));
|
||||
|
||||
Assert.Contains("1 / 2 healthy", card.Markup, StringComparison.Ordinal);
|
||||
Assert.Equal(2, card.FindAll(".check-row").Count);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void ClusterData_IsRenderedAsRawEvidence()
|
||||
{
|
||||
var card = Render(Card(report: Report(("akka-cluster", "Healthy", "akka.tcp://scadabridge@site-a-a:8082"))));
|
||||
|
||||
var data = card.Find(".check-data");
|
||||
Assert.Contains("akka.tcp://scadabridge@site-a-a:8082", data.TextContent, StringComparison.Ordinal);
|
||||
Assert.Contains("leader", data.TextContent, StringComparison.Ordinal);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void ClusterData_FromAnInstanceWithoutIt_IsOmittedEntirely()
|
||||
{
|
||||
// Every pre-0.2.0 node, and every check that publishes no data. The card must simply not
|
||||
// show the line rather than showing an empty one.
|
||||
var card = Render(Card(report: Report(("localdb", "Healthy", null))));
|
||||
|
||||
Assert.Empty(card.FindAll(".check-data"));
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void ClusterData_IsHtmlEscaped()
|
||||
{
|
||||
// The value is remote input from another application. Operationally trusted, but a health
|
||||
// check that ever emits markup must not be able to inject it into this page.
|
||||
var card = Render(Card(report: Report(("akka-cluster", "Healthy", "<img src=x onerror=alert(1)>"))));
|
||||
|
||||
var data = card.Find(".check-data");
|
||||
Assert.Empty(data.QuerySelectorAll("img"));
|
||||
Assert.Contains("<img src=x onerror=alert(1)>", data.TextContent, StringComparison.Ordinal);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void ManagementLink_OpensInANewTabWithNoopener()
|
||||
{
|
||||
var link = Render(Card(managementUrl: "http://site-a-a:9000/")).Find(".panel-foot a");
|
||||
|
||||
Assert.Equal("http://site-a-a:9000/", link.GetAttribute("href"));
|
||||
Assert.Equal("_blank", link.GetAttribute("target"));
|
||||
Assert.Equal("noopener", link.GetAttribute("rel"));
|
||||
Assert.Contains("AdminUI", link.TextContent, StringComparison.Ordinal);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void NoManagementUrl_RendersNoDeadLink()
|
||||
{
|
||||
var card = Render(Card(managementUrl: null));
|
||||
|
||||
Assert.Empty(card.FindAll(".panel-foot a"));
|
||||
Assert.Contains("no management link", card.Markup, StringComparison.Ordinal);
|
||||
}
|
||||
|
||||
[Theory]
|
||||
[InlineData(InstanceStatus.Up, ActiveState.Active, "ready 200 · active 200")]
|
||||
[InlineData(InstanceStatus.Up, ActiveState.Standby, "ready 200 · active 503")]
|
||||
[InlineData(InstanceStatus.Down, ActiveState.NotApplicable, "ready 503")]
|
||||
[InlineData(InstanceStatus.Unreachable, ActiveState.NotApplicable, "ready fail")]
|
||||
public void RawSignalLine_ShowsWhatWasActuallyReceived(
|
||||
InstanceStatus status, ActiveState active, string expected)
|
||||
{
|
||||
// The footer is the escape hatch from every derived label above it: when the dashboard and
|
||||
// a curl disagree, this line is what reconciles them.
|
||||
Assert.Contains(
|
||||
expected,
|
||||
Render(Card(status, active)).Find("[data-testid=raw-signals]").TextContent,
|
||||
StringComparison.Ordinal);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void ProbeError_IsSurfacedOnTheCard()
|
||||
{
|
||||
var card = Render(Card(InstanceStatus.Unreachable, error: "Connection refused"));
|
||||
|
||||
Assert.Contains("Connection refused", card.Markup, StringComparison.Ordinal);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void Host_IsShownWithoutTheScheme()
|
||||
{
|
||||
Assert.Contains("site-a-a:8084", Render(Card()).Find(".inst-host").TextContent, StringComparison.Ordinal);
|
||||
}
|
||||
|
||||
[Theory]
|
||||
[InlineData(4, "4 s ago")]
|
||||
[InlineData(90, "2 min ago")]
|
||||
[InlineData(7200, "2 h ago")]
|
||||
[InlineData(172800, "2 d ago")]
|
||||
public void Relative_ScalesTheUnit(int seconds, string expected)
|
||||
{
|
||||
Assert.Equal(expected, InstanceCard.Relative(TimeSpan.FromSeconds(seconds)));
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void Relative_NegativeAge_ReadsAsNow_NotAsTheFuture()
|
||||
{
|
||||
// Clock skew between this host and a probed one is routine; "-3 s ago" is nonsense.
|
||||
Assert.Equal("0 s ago", InstanceCard.Relative(TimeSpan.FromSeconds(-3)));
|
||||
}
|
||||
|
||||
private static ZbHealthReport Report(params (string Name, string Status, string? Leader)[] entries) =>
|
||||
System.Text.Json.JsonSerializer.Deserialize<ZbHealthReport>(HealthBody.Ready("Healthy", entries))!;
|
||||
}
|
||||
@@ -0,0 +1,185 @@
|
||||
using ZB.MOM.WW.Overview.Polling;
|
||||
|
||||
namespace ZB.MOM.WW.Overview.Tests;
|
||||
|
||||
/// <summary>
|
||||
/// Leader aggregation and the split-brain flag. The disagreement rule is the one output an operator
|
||||
/// would be woken by, so both directions are pinned: it must fire when two members genuinely claim
|
||||
/// different leaders, and it must NOT fire for the ordinary cases that superficially resemble that
|
||||
/// — a node that stopped answering, a node running a pre-0.2.0 Health package.
|
||||
/// </summary>
|
||||
public class LeaderResolverTests
|
||||
{
|
||||
private const string LeaderA = "akka.tcp://scadabridge@site-a-a:8082";
|
||||
private const string LeaderB = "akka.tcp://scadabridge@site-a-b:8082";
|
||||
|
||||
[Fact]
|
||||
public void Resolve_AgreeingPair_NamesTheLeaderInstance()
|
||||
{
|
||||
var groups = LeaderResolver.Resolve(
|
||||
[
|
||||
Snapshots.Instance("site-a-a", "site-a", "http://site-a-a:8084", leader: LeaderA),
|
||||
Snapshots.Instance("site-a-b", "site-a", "http://site-a-b:8084", leader: LeaderA),
|
||||
]);
|
||||
|
||||
var group = Assert.Single(groups);
|
||||
Assert.Equal("site-a", group.Name);
|
||||
Assert.Equal("site-a-a", group.LeaderInstance);
|
||||
Assert.Equal(LeaderA, group.LeaderAddress);
|
||||
Assert.False(group.Disagreement);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void Resolve_MembersClaimDifferentLeaders_FlagsDisagreement()
|
||||
{
|
||||
var groups = LeaderResolver.Resolve(
|
||||
[
|
||||
Snapshots.Instance("site-a-a", "site-a", "http://site-a-a:8084", leader: LeaderA),
|
||||
Snapshots.Instance("site-a-b", "site-a", "http://site-a-b:8084", leader: LeaderB),
|
||||
]);
|
||||
|
||||
var group = Assert.Single(groups);
|
||||
Assert.True(group.Disagreement);
|
||||
Assert.Equal([LeaderA, LeaderB], group.ReportedLeaders);
|
||||
}
|
||||
|
||||
[Theory]
|
||||
[InlineData(InstanceStatus.Down)]
|
||||
[InlineData(InstanceStatus.Unreachable)]
|
||||
public void Resolve_FailedMemberDoesNotVote_SoAFailoverIsNotMisreadAsSplitBrain(InstanceStatus failed)
|
||||
{
|
||||
// Mid-failover the departed node's last body still names the old leader. Counting it would
|
||||
// raise a split-brain warning during the most normal event in a redundant pair's life.
|
||||
var groups = LeaderResolver.Resolve(
|
||||
[
|
||||
Snapshots.Instance("site-a-a", "site-a", "http://site-a-a:8084", status: failed, leader: LeaderA),
|
||||
Snapshots.Instance("site-a-b", "site-a", "http://site-a-b:8084", leader: LeaderB),
|
||||
]);
|
||||
|
||||
var group = Assert.Single(groups);
|
||||
Assert.False(group.Disagreement);
|
||||
Assert.Equal("site-a-b", group.LeaderInstance);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void Resolve_DegradedMemberStillVotes()
|
||||
{
|
||||
// Degraded means answering with a full cluster view — its opinion on the leader is current.
|
||||
var groups = LeaderResolver.Resolve(
|
||||
[
|
||||
Snapshots.Instance("site-a-a", "site-a", "http://site-a-a:8084", status: InstanceStatus.Degraded, leader: LeaderA),
|
||||
Snapshots.Instance("site-a-b", "site-a", "http://site-a-b:8084", leader: LeaderB),
|
||||
]);
|
||||
|
||||
Assert.True(Assert.Single(groups).Disagreement);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void Resolve_NoMemberPublishesLeaderData_YieldsNoLeaderAndNoWarning()
|
||||
{
|
||||
// The whole-fleet-on-Health-0.1.0 case: no data key anywhere. It must degrade to "leader
|
||||
// unknown", never to a false split-brain warning.
|
||||
var groups = LeaderResolver.Resolve(
|
||||
[
|
||||
Snapshots.Instance("site-a-a", "site-a", "http://site-a-a:8084"),
|
||||
Snapshots.Instance("site-a-b", "site-a", "http://site-a-b:8084"),
|
||||
]);
|
||||
|
||||
var group = Assert.Single(groups);
|
||||
Assert.Null(group.LeaderAddress);
|
||||
Assert.Null(group.LeaderInstance);
|
||||
Assert.False(group.Disagreement);
|
||||
Assert.Empty(group.ReportedLeaders);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void Resolve_OneMemberOnOldHealthPackage_UsesTheOneThatDoesReport()
|
||||
{
|
||||
// A partly-bumped fleet is the expected state during rollout, and one silent member is not
|
||||
// a disagreement.
|
||||
var groups = LeaderResolver.Resolve(
|
||||
[
|
||||
Snapshots.Instance("site-a-a", "site-a", "http://site-a-a:8084", leader: LeaderA),
|
||||
Snapshots.Instance("site-a-b", "site-a", "http://site-a-b:8084"),
|
||||
]);
|
||||
|
||||
var group = Assert.Single(groups);
|
||||
Assert.Equal("site-a-a", group.LeaderInstance);
|
||||
Assert.False(group.Disagreement);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void Resolve_UnresolvableAddress_KeepsTheRawAddress()
|
||||
{
|
||||
// A leader on a host that is not in the registry (a node nobody added) still tells the
|
||||
// operator something useful — showing nothing would hide the discovery.
|
||||
var groups = LeaderResolver.Resolve(
|
||||
[
|
||||
Snapshots.Instance("site-a-a", "site-a", "http://site-a-a:8084", leader: "akka.tcp://scadabridge@ghost:8082"),
|
||||
]);
|
||||
|
||||
var group = Assert.Single(groups);
|
||||
Assert.Null(group.LeaderInstance);
|
||||
Assert.Equal("akka.tcp://scadabridge@ghost:8082", group.LeaderAddress);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void Resolve_MatchesOnHostOnly_BecauseAkkaAndHttpPortsDiffer()
|
||||
{
|
||||
// The registry knows the HTTP port (8084); the leader address carries the Akka remoting
|
||||
// port (8082). Matching on authority instead of host would never resolve anything.
|
||||
var name = LeaderResolver.ResolveToInstanceName(
|
||||
LeaderA,
|
||||
[Snapshots.Instance("site-a-a", "site-a", "http://site-a-a:8084")]);
|
||||
|
||||
Assert.Equal("site-a-a", name);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void Resolve_GroupWithoutActiveRole_IsNotAGroupAtAll()
|
||||
{
|
||||
// Instances grouped only for layout have no leader; a chip there would invent a concept.
|
||||
var groups = LeaderResolver.Resolve(
|
||||
[
|
||||
Snapshots.Instance("gw-1", "gateways", "http://gw-1:5120", hasActiveRole: false),
|
||||
Snapshots.Instance("gw-2", "gateways", "http://gw-2:5120", hasActiveRole: false),
|
||||
]);
|
||||
|
||||
Assert.Empty(groups);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void Resolve_UngroupedInstances_ProduceNoGroups()
|
||||
{
|
||||
Assert.Empty(LeaderResolver.Resolve([Snapshots.Instance("solo")]));
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void Resolve_SeparateGroups_AreScopedIndependently()
|
||||
{
|
||||
// Two site pairs, each with its own leader: per-Cluster election is the family's actual
|
||||
// topology, so a leader from one group must never be attributed to another.
|
||||
var groups = LeaderResolver.Resolve(
|
||||
[
|
||||
Snapshots.Instance("site-a-a", "site-a", "http://site-a-a:8084", leader: LeaderA),
|
||||
Snapshots.Instance("site-b-a", "site-b", "http://site-b-a:8084", leader: "akka.tcp://scadabridge@site-b-a:8082"),
|
||||
]);
|
||||
|
||||
Assert.Equal(2, groups.Count);
|
||||
Assert.Equal("site-a-a", groups[0].LeaderInstance);
|
||||
Assert.Equal("site-b-a", groups[1].LeaderInstance);
|
||||
Assert.All(groups, g => Assert.False(g.Disagreement));
|
||||
}
|
||||
|
||||
[Theory]
|
||||
[InlineData("akka.tcp://scadabridge@site-a-a:8082", "site-a-a")]
|
||||
[InlineData("akka.tcp://otopcua@10.100.0.35:4053", "10.100.0.35")]
|
||||
[InlineData("akka.tcp://sys@host", "host")]
|
||||
[InlineData("not-an-address", null)]
|
||||
[InlineData("", null)]
|
||||
[InlineData(null, null)]
|
||||
public void HostOf_ParsesAkkaAddresses(string? address, string? expected)
|
||||
{
|
||||
Assert.Equal(expected, LeaderResolver.HostOf(address));
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,251 @@
|
||||
using Bunit;
|
||||
using Microsoft.Extensions.DependencyInjection;
|
||||
using ZB.MOM.WW.Overview.Polling;
|
||||
|
||||
namespace ZB.MOM.WW.Overview.Tests;
|
||||
|
||||
/// <summary>
|
||||
/// The page as a whole: the KPI strip an operator scans first, the group headers that carry the
|
||||
/// split-brain warning, and the pause control — which has to freeze the view without pretending
|
||||
/// the view is still live.
|
||||
/// </summary>
|
||||
public class OverviewPageTests : TestContext
|
||||
{
|
||||
private static readonly DateTimeOffset Now = new(2026, 7, 24, 12, 0, 0, TimeSpan.Zero);
|
||||
|
||||
private readonly OverviewSnapshotStore _store = new();
|
||||
private readonly FakeTimeProvider _time = new(Now);
|
||||
|
||||
public OverviewPageTests()
|
||||
{
|
||||
Services.AddSingleton(_store);
|
||||
Services.AddSingleton<TimeProvider>(_time);
|
||||
}
|
||||
|
||||
private static InstanceSnapshot Node(
|
||||
string name,
|
||||
InstanceStatus? status,
|
||||
string? group = null,
|
||||
string? leader = null,
|
||||
DateTimeOffset? lastPolled = null) =>
|
||||
new()
|
||||
{
|
||||
Application = "ScadaBridge",
|
||||
Instance = name,
|
||||
Group = group,
|
||||
BaseUrl = $"http://{name}:8084",
|
||||
HasActiveRole = group is not null,
|
||||
Status = status,
|
||||
LastPolledUtc = lastPolled ?? Now,
|
||||
LastReachableUtc = lastPolled ?? Now,
|
||||
StaleAfter = TimeSpan.FromSeconds(45),
|
||||
Report = leader is null
|
||||
? null
|
||||
: System.Text.Json.JsonSerializer.Deserialize<ZbHealthReport>(
|
||||
HealthBody.Ready("Healthy", ("akka-cluster", "Healthy", leader))),
|
||||
};
|
||||
|
||||
private static OverviewSnapshot Snapshot(params InstanceSnapshot[] instances)
|
||||
{
|
||||
var applications = instances
|
||||
.GroupBy(i => i.Application, StringComparer.Ordinal)
|
||||
.Select(g => new ApplicationSnapshot(g.Key, g.ToList(), LeaderResolver.Resolve(g.ToList())))
|
||||
.ToList();
|
||||
|
||||
return new OverviewSnapshot(Now, applications);
|
||||
}
|
||||
|
||||
private IRenderedComponent<Components.Pages.Overview> RenderPage() =>
|
||||
RenderComponent<Components.Pages.Overview>();
|
||||
|
||||
[Fact]
|
||||
public void EmptyStore_RendersAnExplicitEmptyState()
|
||||
{
|
||||
// Before the first sweep. It must say so rather than render a KPI strip full of zeroes,
|
||||
// which would read as "the whole fleet is registered and nothing is up".
|
||||
var page = RenderPage();
|
||||
|
||||
Assert.NotNull(page.Find("[data-testid=empty-registry]"));
|
||||
Assert.Empty(page.FindAll(".agg-grid"));
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void KpiStrip_CountsEachStatus()
|
||||
{
|
||||
_store.Publish(Snapshot(
|
||||
Node("a", InstanceStatus.Up),
|
||||
Node("b", InstanceStatus.Up),
|
||||
Node("c", InstanceStatus.Degraded),
|
||||
Node("d", InstanceStatus.Down),
|
||||
Node("e", InstanceStatus.Unreachable)));
|
||||
|
||||
var page = RenderPage();
|
||||
|
||||
Assert.Equal("2", page.Find("[data-testid=kpi-up]").TextContent);
|
||||
Assert.Equal("1", page.Find("[data-testid=kpi-degraded]").TextContent);
|
||||
Assert.Equal("1", page.Find("[data-testid=kpi-down]").TextContent);
|
||||
Assert.Equal("1", page.Find("[data-testid=kpi-unreachable]").TextContent);
|
||||
Assert.Equal("0", page.Find("[data-testid=kpi-stale]").TextContent);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void StaleInstances_CountAsStaleAndNotAlsoAsUp()
|
||||
{
|
||||
// Double-counting would make the KPI totals exceed the instance count and would let a
|
||||
// frozen dashboard keep reporting a healthy fleet.
|
||||
_store.Publish(Snapshot(
|
||||
Node("a", InstanceStatus.Up),
|
||||
Node("b", InstanceStatus.Up, lastPolled: Now.AddMinutes(-5))));
|
||||
|
||||
var page = RenderPage();
|
||||
|
||||
Assert.Equal("1", page.Find("[data-testid=kpi-up]").TextContent);
|
||||
Assert.Equal("1", page.Find("[data-testid=kpi-stale]").TextContent);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void GroupHeader_NamesTheLeader()
|
||||
{
|
||||
const string leader = "akka.tcp://scadabridge@a:8082";
|
||||
_store.Publish(Snapshot(
|
||||
Node("a", InstanceStatus.Up, "site-a", leader),
|
||||
Node("b", InstanceStatus.Up, "site-a", leader)));
|
||||
|
||||
Assert.Contains("Leader · a", RenderPage().Markup, StringComparison.Ordinal);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void GroupHeader_WarnsOnDisagreement()
|
||||
{
|
||||
// The one output on this page that means "go look right now".
|
||||
_store.Publish(Snapshot(
|
||||
Node("a", InstanceStatus.Up, "site-a", "akka.tcp://scadabridge@a:8082"),
|
||||
Node("b", InstanceStatus.Up, "site-a", "akka.tcp://scadabridge@b:8082")));
|
||||
|
||||
Assert.Contains("Split brain", RenderPage().Markup, StringComparison.Ordinal);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void AgreeingGroup_ShowsNoWarning()
|
||||
{
|
||||
const string leader = "akka.tcp://scadabridge@a:8082";
|
||||
_store.Publish(Snapshot(
|
||||
Node("a", InstanceStatus.Up, "site-a", leader),
|
||||
Node("b", InstanceStatus.Up, "site-a", leader)));
|
||||
|
||||
Assert.DoesNotContain("Split brain", RenderPage().Markup, StringComparison.Ordinal);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void PublishedSnapshot_ReRendersThePage()
|
||||
{
|
||||
_store.Publish(Snapshot(Node("a", InstanceStatus.Up)));
|
||||
var page = RenderPage();
|
||||
Assert.Equal("1", page.Find("[data-testid=kpi-up]").TextContent);
|
||||
|
||||
_store.Publish(Snapshot(Node("a", InstanceStatus.Down)));
|
||||
|
||||
page.WaitForAssertion(() => Assert.Equal("0", page.Find("[data-testid=kpi-up]").TextContent));
|
||||
Assert.Equal("1", page.Find("[data-testid=kpi-down]").TextContent);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void Paused_StopsAdoptingNewSnapshots()
|
||||
{
|
||||
_store.Publish(Snapshot(Node("a", InstanceStatus.Up)));
|
||||
var page = RenderPage();
|
||||
|
||||
page.Find("[data-testid=pause-toggle]").Click();
|
||||
_store.Publish(Snapshot(Node("a", InstanceStatus.Down)));
|
||||
|
||||
Assert.Equal("1", page.Find("[data-testid=kpi-up]").TextContent);
|
||||
Assert.Contains("Resume auto-refresh", page.Markup, StringComparison.Ordinal);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void Resumed_AdoptsWhateverTheStoreHoldsNow()
|
||||
{
|
||||
// Resume must not replay the snapshots missed while paused — it must jump to current.
|
||||
_store.Publish(Snapshot(Node("a", InstanceStatus.Up)));
|
||||
var page = RenderPage();
|
||||
page.Find("[data-testid=pause-toggle]").Click();
|
||||
_store.Publish(Snapshot(Node("a", InstanceStatus.Down)));
|
||||
|
||||
page.Find("[data-testid=pause-toggle]").Click();
|
||||
|
||||
Assert.Equal("1", page.Find("[data-testid=kpi-down]").TextContent);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void PausedView_AgesIntoStaleOnItsOwn()
|
||||
{
|
||||
// This is why pause freezes adoption rather than stopping the clock: a paused dashboard
|
||||
// greys out by itself, so it cannot be mistaken for a live one.
|
||||
_store.Publish(Snapshot(Node("a", InstanceStatus.Up)));
|
||||
var page = RenderPage();
|
||||
page.Find("[data-testid=pause-toggle]").Click();
|
||||
|
||||
_time.Advance(TimeSpan.FromMinutes(5));
|
||||
page.Render();
|
||||
|
||||
Assert.Equal("1", page.Find("[data-testid=kpi-stale]").TextContent);
|
||||
Assert.Equal("0", page.Find("[data-testid=kpi-up]").TextContent);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void Disposal_UnsubscribesFromTheStore()
|
||||
{
|
||||
// A leaked handler would call StateHasChanged on a disposed renderer every sweep, for the
|
||||
// life of the process — one leak per page load.
|
||||
_store.Publish(Snapshot(Node("a", InstanceStatus.Up)));
|
||||
var page = RenderPage();
|
||||
|
||||
page.Instance.Dispose();
|
||||
|
||||
var exception = Record.Exception(() => _store.Publish(Snapshot(Node("a", InstanceStatus.Down))));
|
||||
Assert.Null(exception);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void ApplicationSection_CarriesAnAnchorMatchingTheRailLink()
|
||||
{
|
||||
// The rail's application links are fragment anchors; a mismatch makes every one dead.
|
||||
_store.Publish(Snapshot(Node("a", InstanceStatus.Up)));
|
||||
|
||||
var section = RenderPage().Find("[data-testid=application]");
|
||||
|
||||
Assert.Equal(
|
||||
ZB.MOM.WW.Overview.Components.Layout.MainLayout.Slug("ScadaBridge"),
|
||||
section.GetAttribute("id"));
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void OneCardPerRegisteredInstance()
|
||||
{
|
||||
_store.Publish(Snapshot(
|
||||
Node("a", InstanceStatus.Up, "site-a"),
|
||||
Node("b", InstanceStatus.Up, "site-a"),
|
||||
Node("gw", InstanceStatus.Up)));
|
||||
|
||||
Assert.Equal(3, RenderPage().FindAll("[data-testid=instance-card]").Count);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void UngroupedInstances_RenderAfterTheClusterGroups()
|
||||
{
|
||||
_store.Publish(Snapshot(
|
||||
Node("gw", InstanceStatus.Up),
|
||||
Node("a", InstanceStatus.Up, "site-a")));
|
||||
|
||||
// Enumerated, not indexed: bunit 1.40 was built against AngleSharp 1.2 and its
|
||||
// RefreshableElementCollection indexer calls an IHtmlCollection member whose signature
|
||||
// changed by 1.5.2 — the version this project pins for GHSA reasons. LINQ goes through
|
||||
// IEnumerable and is unaffected.
|
||||
var cards = RenderPage()
|
||||
.FindAll("[data-testid=instance-card]")
|
||||
.Select(c => c.GetAttribute("data-instance"))
|
||||
.ToList();
|
||||
|
||||
Assert.Equal(["a", "gw"], cards);
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,430 @@
|
||||
using System.Net;
|
||||
using Microsoft.Extensions.Logging.Abstractions;
|
||||
using Microsoft.Extensions.Options;
|
||||
using ZB.MOM.WW.Overview.Polling;
|
||||
using ZB.MOM.WW.Overview.Registry;
|
||||
|
||||
namespace ZB.MOM.WW.Overview.Tests;
|
||||
|
||||
/// <summary>
|
||||
/// Sweep behaviour, driven directly rather than through the timer: the rules worth pinning are what
|
||||
/// a sweep probes, what it does with the answers, and when it declines to probe at all.
|
||||
/// </summary>
|
||||
public class OverviewPollerServiceTests
|
||||
{
|
||||
private static readonly DateTimeOffset Start = new(2026, 7, 24, 12, 0, 0, TimeSpan.Zero);
|
||||
|
||||
private static OverviewOptions Registry(params ApplicationEntry[] applications) =>
|
||||
new()
|
||||
{
|
||||
PollIntervalSeconds = 10,
|
||||
TimeoutSeconds = 3,
|
||||
StaleAfterSeconds = 45,
|
||||
Applications = applications,
|
||||
};
|
||||
|
||||
private static ApplicationEntry App(string name, params InstanceEntry[] instances) =>
|
||||
new() { Name = name, ManagementLabel = "AdminUI", Instances = instances };
|
||||
|
||||
private static InstanceEntry Node(
|
||||
string name,
|
||||
string baseUrl,
|
||||
string? group = null,
|
||||
bool hasActiveRole = false,
|
||||
int? pollIntervalSeconds = null) =>
|
||||
new()
|
||||
{
|
||||
Name = name,
|
||||
BaseUrl = baseUrl,
|
||||
Group = group,
|
||||
HasActiveRole = hasActiveRole,
|
||||
PollIntervalSeconds = pollIntervalSeconds,
|
||||
};
|
||||
|
||||
private static (OverviewPollerService Poller, OverviewSnapshotStore Store) Build(
|
||||
OverviewOptions options,
|
||||
HttpMessageHandler handler,
|
||||
TimeProvider timeProvider)
|
||||
{
|
||||
var store = new OverviewSnapshotStore();
|
||||
var poller = new OverviewPollerService(
|
||||
Options.Create(options),
|
||||
new StubHttpClientFactory(handler),
|
||||
store,
|
||||
timeProvider,
|
||||
NullLogger<OverviewPollerService>.Instance);
|
||||
|
||||
return (poller, store);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task Sweep_ProbesReadyForEveryInstance()
|
||||
{
|
||||
var handler = new RoutingHandler()
|
||||
.Json("http://a:8080/health/ready", HttpStatusCode.OK, HealthBody.Ready("Healthy"))
|
||||
.Json("http://b:8080/health/ready", HttpStatusCode.OK, HealthBody.Ready("Healthy"));
|
||||
var (poller, _) = Build(
|
||||
Registry(App("App", Node("a", "http://a:8080"), Node("b", "http://b:8080"))),
|
||||
handler,
|
||||
new FakeTimeProvider(Start));
|
||||
|
||||
var snapshot = await poller.SweepAsync();
|
||||
|
||||
Assert.Equal(2, snapshot.AllInstances.Count());
|
||||
Assert.All(snapshot.AllInstances, i => Assert.Equal(InstanceStatus.Up, i.Status));
|
||||
Assert.Equal(2, handler.Requests.Count);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task Sweep_TrailingSlashInBaseUrl_DoesNotDoubleTheSeparator()
|
||||
{
|
||||
// "//health/ready" would 404 on every instance — a registry typo class that must not matter.
|
||||
var handler = new RoutingHandler()
|
||||
.Json("http://a:8080/health/ready", HttpStatusCode.OK, HealthBody.Ready("Healthy"));
|
||||
var (poller, _) = Build(Registry(App("App", Node("a", "http://a:8080/"))), handler, new FakeTimeProvider(Start));
|
||||
|
||||
var snapshot = await poller.SweepAsync();
|
||||
|
||||
Assert.Equal(InstanceStatus.Up, Assert.Single(snapshot.AllInstances).Status);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task Sweep_ActiveTierProbedOnlyForInstancesThatHaveTheRole()
|
||||
{
|
||||
var handler = new RoutingHandler()
|
||||
.Json("http://a:8080/health/ready", HttpStatusCode.OK, HealthBody.Ready("Healthy"))
|
||||
.Status("http://a:8080/health/active", HttpStatusCode.OK)
|
||||
.Json("http://gw:5120/health/ready", HttpStatusCode.OK, HealthBody.Ready("Healthy"));
|
||||
var (poller, _) = Build(
|
||||
Registry(App("App", Node("a", "http://a:8080", "pair", hasActiveRole: true), Node("gw", "http://gw:5120"))),
|
||||
handler,
|
||||
new FakeTimeProvider(Start));
|
||||
|
||||
var snapshot = await poller.SweepAsync();
|
||||
|
||||
Assert.DoesNotContain("http://gw:5120/health/active", handler.Requests, StringComparer.Ordinal);
|
||||
var byName = snapshot.AllInstances.ToDictionary(i => i.Instance, StringComparer.Ordinal);
|
||||
Assert.Equal(ActiveState.Active, byName["a"].Active);
|
||||
Assert.Equal(ActiveState.NotApplicable, byName["gw"].Active);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task Sweep_StandbyAnswers503OnActive_AndIsStillUp()
|
||||
{
|
||||
// 503 on the ACTIVE tier is the standby's correct answer and says nothing about health —
|
||||
// conflating the two tiers would paint every healthy standby in the fleet as Down.
|
||||
var handler = new RoutingHandler()
|
||||
.Json("http://b:8080/health/ready", HttpStatusCode.OK, HealthBody.Ready("Healthy"))
|
||||
.Status("http://b:8080/health/active", HttpStatusCode.ServiceUnavailable);
|
||||
var (poller, _) = Build(
|
||||
Registry(App("App", Node("b", "http://b:8080", "pair", hasActiveRole: true))),
|
||||
handler,
|
||||
new FakeTimeProvider(Start));
|
||||
|
||||
var instance = Assert.Single((await poller.SweepAsync()).AllInstances);
|
||||
|
||||
Assert.Equal(InstanceStatus.Up, instance.Status);
|
||||
Assert.Equal(ActiveState.Standby, instance.Active);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task Sweep_UnreachableInstance_SkipsTheActiveProbe()
|
||||
{
|
||||
// Spending a second full timeout to confirm what the first one established would double
|
||||
// the worst-case sweep duration for every dead node in the registry.
|
||||
var handler = new RoutingHandler().Fails("http://a:8080/health/ready", "Connection refused");
|
||||
var (poller, _) = Build(
|
||||
Registry(App("App", Node("a", "http://a:8080", "pair", hasActiveRole: true))),
|
||||
handler,
|
||||
new FakeTimeProvider(Start));
|
||||
|
||||
await poller.SweepAsync();
|
||||
|
||||
Assert.DoesNotContain("http://a:8080/health/active", handler.Requests, StringComparer.Ordinal);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task Sweep_SickInstanceStillGetsItsActiveProbe()
|
||||
{
|
||||
// A 503 on ready means the app is talking. Whether the sick node is the ACTIVE half is
|
||||
// precisely what decides whether this is an outage or a standby nobody is using.
|
||||
var handler = new RoutingHandler()
|
||||
.Json("http://a:8080/health/ready", HttpStatusCode.ServiceUnavailable, HealthBody.Ready("Unhealthy"))
|
||||
.Status("http://a:8080/health/active", HttpStatusCode.OK);
|
||||
var (poller, _) = Build(
|
||||
Registry(App("App", Node("a", "http://a:8080", "pair", hasActiveRole: true))),
|
||||
handler,
|
||||
new FakeTimeProvider(Start));
|
||||
|
||||
var instance = Assert.Single((await poller.SweepAsync()).AllInstances);
|
||||
|
||||
Assert.Equal(InstanceStatus.Down, instance.Status);
|
||||
Assert.Equal(ActiveState.Active, instance.Active);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task Sweep_FirstFailureAfterAGoodPoll_IsDampedThenAdopted()
|
||||
{
|
||||
var handler = new RoutingHandler()
|
||||
.Json("http://a:8080/health/ready", HttpStatusCode.OK, HealthBody.Ready("Healthy"));
|
||||
var time = new FakeTimeProvider(Start);
|
||||
var (poller, _) = Build(Registry(App("App", Node("a", "http://a:8080"))), handler, time);
|
||||
|
||||
Assert.Equal(InstanceStatus.Up, Assert.Single((await poller.SweepAsync()).AllInstances).Status);
|
||||
|
||||
handler.Fails("http://a:8080/health/ready", "Connection refused");
|
||||
|
||||
time.Advance(TimeSpan.FromSeconds(10));
|
||||
var damped = Assert.Single((await poller.SweepAsync()).AllInstances);
|
||||
Assert.Equal(InstanceStatus.Up, damped.Status);
|
||||
Assert.Equal(1, damped.ConsecutiveFailures);
|
||||
|
||||
time.Advance(TimeSpan.FromSeconds(10));
|
||||
var adopted = Assert.Single((await poller.SweepAsync()).AllInstances);
|
||||
Assert.Equal(InstanceStatus.Unreachable, adopted.Status);
|
||||
Assert.Equal(2, adopted.ConsecutiveFailures);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task Sweep_LastReachableSurvivesAFailedPoll()
|
||||
{
|
||||
// "last seen" must keep pointing at the last real contact, not reset to null the moment
|
||||
// the instance goes away — that timestamp is how an operator judges how long it has been out.
|
||||
var handler = new RoutingHandler()
|
||||
.Json("http://a:8080/health/ready", HttpStatusCode.OK, HealthBody.Ready("Healthy"));
|
||||
var time = new FakeTimeProvider(Start);
|
||||
var (poller, _) = Build(Registry(App("App", Node("a", "http://a:8080"))), handler, time);
|
||||
|
||||
await poller.SweepAsync();
|
||||
handler.Fails("http://a:8080/health/ready", "Connection refused");
|
||||
time.Advance(TimeSpan.FromSeconds(10));
|
||||
|
||||
var instance = Assert.Single((await poller.SweepAsync()).AllInstances);
|
||||
|
||||
Assert.Equal(Start, instance.LastReachableUtc);
|
||||
Assert.Equal(Start.AddSeconds(10), instance.LastPolledUtc);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task Sweep_BeforeAnInstanceIsDue_LeavesItAlone()
|
||||
{
|
||||
var handler = new RoutingHandler()
|
||||
.Json("http://a:8080/health/ready", HttpStatusCode.OK, HealthBody.Ready("Healthy"));
|
||||
var time = new FakeTimeProvider(Start);
|
||||
var (poller, _) = Build(Registry(App("App", Node("a", "http://a:8080"))), handler, time);
|
||||
|
||||
await poller.SweepAsync();
|
||||
time.Advance(TimeSpan.FromSeconds(1));
|
||||
await poller.SweepAsync();
|
||||
|
||||
Assert.Single(handler.Requests);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task Sweep_PerInstanceInterval_IsHonoured()
|
||||
{
|
||||
// The override is only real if a slow instance actually skips ticks while a fast one runs.
|
||||
var handler = new RoutingHandler()
|
||||
.Json("http://fast:8080/health/ready", HttpStatusCode.OK, HealthBody.Ready("Healthy"))
|
||||
.Json("http://slow:8080/health/ready", HttpStatusCode.OK, HealthBody.Ready("Healthy"));
|
||||
var time = new FakeTimeProvider(Start);
|
||||
var (poller, _) = Build(
|
||||
Registry(App(
|
||||
"App",
|
||||
Node("fast", "http://fast:8080", pollIntervalSeconds: 5),
|
||||
Node("slow", "http://slow:8080", pollIntervalSeconds: 30))),
|
||||
handler,
|
||||
time);
|
||||
|
||||
for (var i = 0; i < 4; i++)
|
||||
{
|
||||
await poller.SweepAsync();
|
||||
time.Advance(TimeSpan.FromSeconds(5));
|
||||
}
|
||||
|
||||
Assert.Equal(4, handler.Requests.Count(r => r.Contains("fast", StringComparison.Ordinal)));
|
||||
Assert.Equal(1, handler.Requests.Count(r => r.Contains("slow", StringComparison.Ordinal)));
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task Sweep_DueCheckToleratesAnEarlyTick()
|
||||
{
|
||||
// A tick arriving a few ms early must not push the instance to half its configured rate.
|
||||
var handler = new RoutingHandler()
|
||||
.Json("http://a:8080/health/ready", HttpStatusCode.OK, HealthBody.Ready("Healthy"));
|
||||
var time = new FakeTimeProvider(Start);
|
||||
var (poller, _) = Build(Registry(App("App", Node("a", "http://a:8080"))), handler, time);
|
||||
|
||||
await poller.SweepAsync();
|
||||
time.Advance(TimeSpan.FromMilliseconds(9_900));
|
||||
await poller.SweepAsync();
|
||||
|
||||
Assert.Equal(2, handler.Requests.Count);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task Sweep_PublishesToTheStore()
|
||||
{
|
||||
var handler = new RoutingHandler()
|
||||
.Json("http://a:8080/health/ready", HttpStatusCode.OK, HealthBody.Ready("Healthy"));
|
||||
var (poller, store) = Build(Registry(App("App", Node("a", "http://a:8080"))), handler, new FakeTimeProvider(Start));
|
||||
OverviewSnapshot? notified = null;
|
||||
store.Changed += s => notified = s;
|
||||
|
||||
var returned = await poller.SweepAsync();
|
||||
|
||||
Assert.Same(returned, store.Current);
|
||||
Assert.Same(returned, notified);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task Sweep_CarriesRegistryMetadataOntoEveryCard()
|
||||
{
|
||||
var handler = new RoutingHandler()
|
||||
.Json("http://a:8080/health/ready", HttpStatusCode.OK, HealthBody.Ready("Healthy"))
|
||||
.Status("http://a:8080/health/active", HttpStatusCode.OK);
|
||||
var application = App("ScadaBridge", Node("central-a", "http://a:8080", "central", hasActiveRole: true));
|
||||
application.Instances[0].ManagementUrl = "http://a:9000/";
|
||||
var (poller, _) = Build(Registry(application), handler, new FakeTimeProvider(Start));
|
||||
|
||||
var instance = Assert.Single((await poller.SweepAsync()).AllInstances);
|
||||
|
||||
Assert.Equal("ScadaBridge", instance.Application);
|
||||
Assert.Equal("central", instance.Group);
|
||||
Assert.Equal("AdminUI", instance.ManagementLabel);
|
||||
Assert.Equal("http://a:9000/", instance.ManagementUrl);
|
||||
Assert.Equal(TimeSpan.FromSeconds(45), instance.StaleAfter);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task Sweep_ResolvesGroupLeadersIntoTheSnapshot()
|
||||
{
|
||||
const string leader = "akka.tcp://scadabridge@a:8082";
|
||||
var handler = new RoutingHandler()
|
||||
.Json("http://a:8084/health/ready", HttpStatusCode.OK, HealthBody.Ready("Healthy", ("akka-cluster", "Healthy", leader)))
|
||||
.Status("http://a:8084/health/active", HttpStatusCode.OK)
|
||||
.Json("http://b:8084/health/ready", HttpStatusCode.OK, HealthBody.Ready("Healthy", ("akka-cluster", "Healthy", leader)))
|
||||
.Status("http://b:8084/health/active", HttpStatusCode.ServiceUnavailable);
|
||||
var (poller, _) = Build(
|
||||
Registry(App(
|
||||
"ScadaBridge",
|
||||
Node("site-a-a", "http://a:8084", "site-a", hasActiveRole: true),
|
||||
Node("site-a-b", "http://b:8084", "site-a", hasActiveRole: true))),
|
||||
handler,
|
||||
new FakeTimeProvider(Start));
|
||||
|
||||
var application = Assert.Single((await poller.SweepAsync()).Applications);
|
||||
var group = Assert.Single(application.Groups);
|
||||
|
||||
Assert.Equal("site-a", group.Name);
|
||||
Assert.Equal("site-a-a", group.LeaderInstance);
|
||||
Assert.False(group.Disagreement);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task Sweep_WorstStatusRollsUpAcrossTheApplication()
|
||||
{
|
||||
var handler = new RoutingHandler()
|
||||
.Json("http://a:8080/health/ready", HttpStatusCode.OK, HealthBody.Ready("Healthy"))
|
||||
.Json("http://b:8080/health/ready", HttpStatusCode.ServiceUnavailable, HealthBody.Ready("Unhealthy"));
|
||||
var (poller, _) = Build(
|
||||
Registry(App("App", Node("a", "http://a:8080"), Node("b", "http://b:8080"))),
|
||||
handler,
|
||||
new FakeTimeProvider(Start));
|
||||
|
||||
var application = Assert.Single((await poller.SweepAsync()).Applications);
|
||||
|
||||
Assert.Equal(InstanceStatus.Down, application.Worst);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task Sweep_OneSlowInstanceDoesNotDelayTheOthers()
|
||||
{
|
||||
// The fan-out must be genuinely parallel: a serial sweep over a fleet with several dead
|
||||
// nodes would take timeout × dead-node-count and starve the whole dashboard.
|
||||
var handler = new RoutingHandler()
|
||||
.Json("http://a:8080/health/ready", HttpStatusCode.OK, HealthBody.Ready("Healthy"))
|
||||
.Json("http://b:8080/health/ready", HttpStatusCode.OK, HealthBody.Ready("Healthy"));
|
||||
var options = Registry(App("App", Node("a", "http://a:8080"), Node("b", "http://b:8080")));
|
||||
var gate = new SemaphoreSlim(0);
|
||||
var (poller, _) = Build(options, new GatedHandler(handler, gate), new FakeTimeProvider(Start));
|
||||
|
||||
var sweep = poller.SweepAsync();
|
||||
Assert.False(sweep.IsCompleted);
|
||||
|
||||
// Both probes must already be in flight; releasing one at a time proves neither waited.
|
||||
gate.Release(2);
|
||||
var snapshot = await sweep.WaitAsync(TimeSpan.FromSeconds(10));
|
||||
|
||||
Assert.All(snapshot.AllInstances, i => Assert.Equal(InstanceStatus.Up, i.Status));
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task Sweep_UnregisteredInstances_RenderAsPendingNotAsHealthy()
|
||||
{
|
||||
// Between host start and the first probe the registry is already published so the page has
|
||||
// its layout. Those cards must read "pending", never an unmeasured "Up".
|
||||
var handler = new RoutingHandler();
|
||||
var (poller, store) = Build(Registry(App("App", Node("a", "http://a:8080"))), handler, new FakeTimeProvider(Start));
|
||||
|
||||
Assert.Empty(store.Current.Applications);
|
||||
|
||||
handler.Fails("http://a:8080/health/ready", "Connection refused");
|
||||
var instance = Assert.Single((await poller.SweepAsync()).AllInstances);
|
||||
|
||||
Assert.Equal(InstanceStatus.Unreachable, instance.Status);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void PendingSnapshot_IsNeverStale()
|
||||
{
|
||||
// Staleness on a card that has never been polled would be meaningless and alarming.
|
||||
var pending = Snapshots.Instance("a", status: null);
|
||||
|
||||
Assert.False(pending.IsStale(Start.AddYears(1)));
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void Snapshot_GoesStaleOnceItsWindowElapses()
|
||||
{
|
||||
var polled = Snapshots.Instance("a") with { LastPolledUtc = Start, StaleAfter = TimeSpan.FromSeconds(45) };
|
||||
|
||||
Assert.False(polled.IsStale(Start.AddSeconds(45)));
|
||||
Assert.True(polled.IsStale(Start.AddSeconds(46)));
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task Sweep_HostShutdown_AbandonsTheSweepInsteadOfRecordingFailures()
|
||||
{
|
||||
// Writing "Unreachable" across the registry on the way out would leave a lie in the store.
|
||||
var handler = new RoutingHandler()
|
||||
.Json("http://a:8080/health/ready", HttpStatusCode.OK, HealthBody.Ready("Healthy"));
|
||||
var (poller, store) = Build(Registry(App("App", Node("a", "http://a:8080"))), handler, new FakeTimeProvider(Start));
|
||||
using var shutdown = new CancellationTokenSource();
|
||||
await shutdown.CancelAsync();
|
||||
|
||||
await Assert.ThrowsAnyAsync<OperationCanceledException>(() => poller.SweepAsync(shutdown.Token));
|
||||
|
||||
Assert.Empty(store.Current.Applications);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task EmptyRegistry_StartsAndStopsWithoutProbing()
|
||||
{
|
||||
var (poller, _) = Build(Registry(), new RoutingHandler(), new FakeTimeProvider(Start));
|
||||
|
||||
await poller.StartAsync(CancellationToken.None);
|
||||
await poller.StopAsync(CancellationToken.None);
|
||||
}
|
||||
|
||||
/// <summary>Holds every request until the test releases it, proving the fan-out is concurrent.</summary>
|
||||
private sealed class GatedHandler(RoutingHandler inner, SemaphoreSlim gate) : HttpMessageHandler
|
||||
{
|
||||
protected override async Task<HttpResponseMessage> SendAsync(
|
||||
HttpRequestMessage request, CancellationToken cancellationToken)
|
||||
{
|
||||
await gate.WaitAsync(cancellationToken).ConfigureAwait(false);
|
||||
return await inner.HandleAsync(request).ConfigureAwait(false);
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,97 @@
|
||||
using ZB.MOM.WW.Overview.Polling;
|
||||
|
||||
namespace ZB.MOM.WW.Overview.Tests;
|
||||
|
||||
/// <summary>
|
||||
/// The store is the cache-first seam: a page reads it and never probes. These tests pin that it
|
||||
/// always has something to serve and that subscribers hear about replacements.
|
||||
/// </summary>
|
||||
public class OverviewSnapshotStoreTests
|
||||
{
|
||||
private static OverviewSnapshot SnapshotAt(int second) =>
|
||||
new(new DateTimeOffset(2026, 7, 24, 12, 0, second, TimeSpan.Zero), []);
|
||||
|
||||
[Fact]
|
||||
public void Current_BeforeAnyPublish_IsEmptyNotNull()
|
||||
{
|
||||
// A page can render before the poller's first sweep finishes; a null here would be a
|
||||
// NullReferenceException on the landing page of a monitoring dashboard.
|
||||
var store = new OverviewSnapshotStore();
|
||||
|
||||
Assert.NotNull(store.Current);
|
||||
Assert.Empty(store.Current.Applications);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void Publish_ReplacesCurrent()
|
||||
{
|
||||
var store = new OverviewSnapshotStore();
|
||||
var snapshot = SnapshotAt(1);
|
||||
|
||||
store.Publish(snapshot);
|
||||
|
||||
Assert.Same(snapshot, store.Current);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void Publish_NotifiesSubscribersWithTheNewSnapshot()
|
||||
{
|
||||
var store = new OverviewSnapshotStore();
|
||||
OverviewSnapshot? received = null;
|
||||
store.Changed += s => received = s;
|
||||
|
||||
var snapshot = SnapshotAt(2);
|
||||
store.Publish(snapshot);
|
||||
|
||||
Assert.Same(snapshot, received);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void Publish_AfterUnsubscribe_DoesNotNotify()
|
||||
{
|
||||
// Blazor circuits come and go; a handler that outlives its component would call
|
||||
// StateHasChanged on a disposed renderer on every sweep.
|
||||
var store = new OverviewSnapshotStore();
|
||||
var calls = 0;
|
||||
void Handler(OverviewSnapshot _) => calls++;
|
||||
|
||||
store.Changed += Handler;
|
||||
store.Publish(SnapshotAt(1));
|
||||
store.Changed -= Handler;
|
||||
store.Publish(SnapshotAt(2));
|
||||
|
||||
Assert.Equal(1, calls);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void Publish_Null_Throws()
|
||||
{
|
||||
Assert.Throws<ArgumentNullException>(() => new OverviewSnapshotStore().Publish(null!));
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task Current_ReadDuringPublish_ObservesOneWholeSnapshot()
|
||||
{
|
||||
// The tearing guarantee: a reader always sees a complete graph, never a half-swapped one.
|
||||
var store = new OverviewSnapshotStore();
|
||||
var a = SnapshotAt(1);
|
||||
var b = SnapshotAt(2);
|
||||
store.Publish(a);
|
||||
|
||||
using var stop = new CancellationTokenSource();
|
||||
var reader = Task.Run(() =>
|
||||
{
|
||||
while (!stop.IsCancellationRequested)
|
||||
{
|
||||
var seen = store.Current;
|
||||
Assert.True(ReferenceEquals(seen, a) || ReferenceEquals(seen, b));
|
||||
}
|
||||
});
|
||||
|
||||
for (var i = 0; i < 1_000; i++)
|
||||
store.Publish(i % 2 == 0 ? b : a);
|
||||
|
||||
await stop.CancelAsync();
|
||||
await reader;
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,142 @@
|
||||
using System.Net;
|
||||
using System.Text;
|
||||
using System.Text.Json;
|
||||
using ZB.MOM.WW.Overview.Polling;
|
||||
|
||||
namespace ZB.MOM.WW.Overview.Tests;
|
||||
|
||||
/// <summary>A clock the test drives by hand.</summary>
|
||||
/// <remarks>
|
||||
/// Only <see cref="GetUtcNow"/> is overridden: the poller's sweep — the part with rules worth
|
||||
/// pinning — is exercised directly, so no test here needs a fake timer.
|
||||
/// </remarks>
|
||||
internal sealed class FakeTimeProvider(DateTimeOffset start) : TimeProvider
|
||||
{
|
||||
private DateTimeOffset _now = start;
|
||||
|
||||
public override DateTimeOffset GetUtcNow() => _now;
|
||||
|
||||
public void Advance(TimeSpan by) => _now += by;
|
||||
}
|
||||
|
||||
/// <summary>Serves canned responses per absolute URL and counts what was requested.</summary>
|
||||
internal sealed class RoutingHandler : HttpMessageHandler
|
||||
{
|
||||
private readonly Dictionary<string, Func<HttpResponseMessage>> _routes = new(StringComparer.OrdinalIgnoreCase);
|
||||
|
||||
public List<string> Requests { get; } = [];
|
||||
|
||||
public RoutingHandler Json(string url, HttpStatusCode status, string body)
|
||||
{
|
||||
_routes[url] = () => new HttpResponseMessage(status)
|
||||
{
|
||||
Content = new StringContent(body, Encoding.UTF8, "application/json"),
|
||||
};
|
||||
return this;
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// A status-code-only response with an EMPTY body — what the active tier actually returns when
|
||||
/// it is a bare gate endpoint rather than a full health report.
|
||||
/// </summary>
|
||||
public RoutingHandler Status(string url, HttpStatusCode status)
|
||||
{
|
||||
_routes[url] = () => new HttpResponseMessage(status)
|
||||
{
|
||||
Content = new StringContent(string.Empty, Encoding.UTF8, "application/json"),
|
||||
};
|
||||
return this;
|
||||
}
|
||||
|
||||
public RoutingHandler Fails(string url, string message)
|
||||
{
|
||||
_routes[url] = () => throw new HttpRequestException(message);
|
||||
return this;
|
||||
}
|
||||
|
||||
/// <summary>Lets a wrapping handler delegate here without re-entering the protected member.</summary>
|
||||
public Task<HttpResponseMessage> HandleAsync(HttpRequestMessage request) =>
|
||||
SendAsync(request, CancellationToken.None);
|
||||
|
||||
protected override Task<HttpResponseMessage> SendAsync(
|
||||
HttpRequestMessage request, CancellationToken cancellationToken)
|
||||
{
|
||||
var url = request.RequestUri!.ToString();
|
||||
|
||||
lock (Requests)
|
||||
Requests.Add(url);
|
||||
|
||||
if (!_routes.TryGetValue(url, out var factory))
|
||||
return Task.FromException<HttpResponseMessage>(new HttpRequestException($"no route for {url}"));
|
||||
|
||||
try
|
||||
{
|
||||
return Task.FromResult(factory());
|
||||
}
|
||||
catch (HttpRequestException ex)
|
||||
{
|
||||
return Task.FromException<HttpResponseMessage>(ex);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// <summary>Hands every named client the same test handler.</summary>
|
||||
internal sealed class StubHttpClientFactory(HttpMessageHandler handler) : IHttpClientFactory
|
||||
{
|
||||
public HttpClient CreateClient(string name) =>
|
||||
new(handler, disposeHandler: false) { Timeout = Timeout.InfiniteTimeSpan };
|
||||
}
|
||||
|
||||
/// <summary>Builds canonical health bodies so tests state intent, not JSON.</summary>
|
||||
internal static class HealthBody
|
||||
{
|
||||
public static string Ready(string status, params (string Name, string Status, string? Leader)[] entries)
|
||||
{
|
||||
var payload = new
|
||||
{
|
||||
status,
|
||||
totalDurationMs = 1.5,
|
||||
entries = entries.ToDictionary(
|
||||
e => e.Name,
|
||||
e => (object)new
|
||||
{
|
||||
status = e.Status,
|
||||
description = $"{e.Name} says {e.Status}",
|
||||
durationMs = 0.5,
|
||||
data = e.Leader is null ? null : new Dictionary<string, object> { ["leader"] = e.Leader },
|
||||
},
|
||||
StringComparer.Ordinal),
|
||||
};
|
||||
|
||||
return JsonSerializer.Serialize(payload);
|
||||
}
|
||||
}
|
||||
|
||||
/// <summary>Builds <see cref="InstanceSnapshot"/> values for the pure-logic tests.</summary>
|
||||
internal static class Snapshots
|
||||
{
|
||||
public static InstanceSnapshot Instance(
|
||||
string name,
|
||||
string? group = null,
|
||||
string baseUrl = "http://host:8080",
|
||||
bool hasActiveRole = true,
|
||||
InstanceStatus? status = InstanceStatus.Up,
|
||||
string? leader = null) =>
|
||||
new()
|
||||
{
|
||||
Application = "App",
|
||||
Instance = name,
|
||||
Group = group,
|
||||
BaseUrl = baseUrl,
|
||||
HasActiveRole = hasActiveRole,
|
||||
Status = status,
|
||||
Report = leader is null ? null : ReportWithLeader(leader),
|
||||
StaleAfter = TimeSpan.FromSeconds(45),
|
||||
};
|
||||
|
||||
private static ZbHealthReport ReportWithLeader(string leader)
|
||||
{
|
||||
var body = HealthBody.Ready("Healthy", ("akka-cluster", "Healthy", leader));
|
||||
return JsonSerializer.Deserialize<ZbHealthReport>(body)!;
|
||||
}
|
||||
}
|
||||
@@ -169,6 +169,59 @@ public class ZbHealthReportClientTests
|
||||
() => client.ProbeAsync("http://node/health/ready", ProbeTimeout, shutdown.Token));
|
||||
}
|
||||
|
||||
[Theory]
|
||||
[InlineData(HttpStatusCode.OK, 200)]
|
||||
[InlineData(HttpStatusCode.ServiceUnavailable, 503)]
|
||||
public async Task ProbeStatus_EmptyBody_IsStillAnAnswer(HttpStatusCode status, int expected)
|
||||
{
|
||||
// The active tier answers with a status code; the body is not part of that contract. An
|
||||
// empty 200/503 is the endpoint telling us the role plainly, and insisting on parseable
|
||||
// JSON here reports every such node as "role unknown".
|
||||
var client = ClientReturning(status, string.Empty);
|
||||
|
||||
var result = await client.ProbeStatusAsync("http://node/health/active", ProbeTimeout);
|
||||
|
||||
Assert.True(result.Reachable);
|
||||
Assert.Equal(expected, result.StatusCode);
|
||||
Assert.Null(result.Report);
|
||||
Assert.Null(result.Error);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task ProbeStatus_StillParsesABodyWhenOneIsThere()
|
||||
{
|
||||
var client = ClientReturning(HttpStatusCode.OK, ReadyHealthyWithData);
|
||||
|
||||
var result = await client.ProbeStatusAsync("http://node/health/active", ProbeTimeout);
|
||||
|
||||
Assert.Equal("Healthy", result.Report!.Status);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task ProbeStatus_TransportFailure_IsStillUnreachable()
|
||||
{
|
||||
// Relaxing the body requirement must not relax what "reachable" means.
|
||||
var client = new ZbHealthReportClient(
|
||||
new HttpClient(new ThrowingHandler(new HttpRequestException("Connection refused"))));
|
||||
|
||||
var result = await client.ProbeStatusAsync("http://node/health/active", ProbeTimeout);
|
||||
|
||||
Assert.False(result.Reachable);
|
||||
Assert.Null(result.StatusCode);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task Probe_ReadyTierStillDemandsAParseableBody()
|
||||
{
|
||||
// The ready tier's body IS the payload — the per-check list and the leader data come from
|
||||
// it. An empty 200 there means something other than a family app is on that port.
|
||||
var client = ClientReturning(HttpStatusCode.OK, string.Empty);
|
||||
|
||||
var result = await client.ProbeAsync("http://node/health/ready", ProbeTimeout);
|
||||
|
||||
Assert.False(result.Reachable);
|
||||
}
|
||||
|
||||
private sealed class StubHandler((HttpStatusCode Status, string Body, string ContentType) response)
|
||||
: HttpMessageHandler
|
||||
{
|
||||
|
||||
@@ -104,7 +104,7 @@
|
||||
{
|
||||
"id": "14",
|
||||
"subject": "3.4 Overview: poller + snapshot store",
|
||||
"status": "pending",
|
||||
"status": "completed",
|
||||
"blockedBy": [
|
||||
"12",
|
||||
"13"
|
||||
@@ -113,7 +113,7 @@
|
||||
{
|
||||
"id": "15",
|
||||
"subject": "3.5 Overview: leader aggregation",
|
||||
"status": "pending",
|
||||
"status": "completed",
|
||||
"blockedBy": [
|
||||
"13"
|
||||
]
|
||||
@@ -121,7 +121,7 @@
|
||||
{
|
||||
"id": "16",
|
||||
"subject": "3.6 Overview: UI (ThemeShell, Overview page, InstanceCard)",
|
||||
"status": "pending",
|
||||
"status": "completed",
|
||||
"blockedBy": [
|
||||
"14",
|
||||
"15"
|
||||
@@ -170,14 +170,19 @@
|
||||
"Phase 1 DoD rig curls (site :8084 ready/active split) \u2014 live docker rig, folded into Phase 4."
|
||||
],
|
||||
"branches": {
|
||||
"scadaproj": "feat/health-0.2.0-cluster-data (80668a0) -> feat/overview-dashboard (d0110cf), stacked",
|
||||
"scadaproj": "feat/health-0.2.0-cluster-data (80668a0) -> feat/overview-dashboard, stacked",
|
||||
"ScadaBridge": "feat/site-node-health (commit 47850c0f)",
|
||||
"OtOpcUa": "feat/health-0.2.0-bump (commit 2e515c34, made in an isolated git worktree \u2014 feat/mesh-phase6 working tree untouched)",
|
||||
"HistorianGateway": "chore/health-0.2.0-bump (commit 51f0c2b)",
|
||||
"MxAccessGateway": "chore/health-0.2.0-bump (commit 2d54ace)"
|
||||
},
|
||||
"front_loaded": [
|
||||
"Validator / health-client / status-derivation tests from task 3.8 were written with tasks 3.2-3.3 so each batch is verified; 3.8 covers the remainder (poller, leader, store, bUnit, boot)."
|
||||
"Validator / health-client / status-derivation tests from task 3.8 were written with tasks 3.2-3.3 so each batch is verified; 3.8 covers the remainder (poller, leader, store, bUnit, boot).",
|
||||
"Poller / snapshot-store / leader-resolver / bUnit InstanceCard + Overview page tests were written with tasks 3.4-3.6; task 3.8 now covers only the WebApplicationFactory boot tests."
|
||||
],
|
||||
"deviations": [
|
||||
"Task 3.6 said 'no antiforgery'. AddRazorComponents stamps antiforgery metadata on every component endpoint unconditionally and the endpoint middleware hard-fails without a matching middleware, so every page returned 500. Resolved with app.UseAntiforgery() (inert here, no forms) rather than MapRazorComponents(...).DisableAntiforgery(), which would silently expose the first form anyone adds. No authentication was added.",
|
||||
"Bootstrap IS vendored (copied from HistorianGateway, v5.3.3). Not optional: ThemeShell and TechButton are built on Bootstrap utility classes, so the shared kit does not render without it."
|
||||
]
|
||||
}
|
||||
}
|
||||
|
||||
Reference in New Issue
Block a user