feat(overview): poller, leader aggregation and the dashboard UI
Tasks 3.4-3.6 of the overview-dashboard plan. Polling (3.4/3.5): - OverviewPollerService: one timer at the fastest configured cadence, each target polled when its own interval is due, fan-out per sweep, no retries (the next tick is the retry, so damping stays the only place that decides whether a failure is worth showing). - OverviewSnapshotStore: atomic swap of a complete immutable graph plus a change event. Pages read it and never probe — the cache-first requirement. - LeaderResolver: per-group leader from akka-cluster data, with the split-brain flag scoped to members that are actually answering, so an ordinary failover is not misread as a disagreement. UI (3.6): ThemeShell + StatusPill/TechCard composition against the mockup; instance cards carry the status stripe (dashed for Unreachable), the check detail list and the raw ready/active signal line. Two defects found by verification rather than by review: - The active tier answers with a status code and an EMPTY body, but the client demanded parseable JSON on every probe — so a healthy pair rendered as "role unknown" instead of Active/Standby. Split into ProbeAsync (ready tier, body is the payload) and ProbeStatusAsync (active tier, code is the answer). Caught by a live smoke run against two fake health endpoints; the poller tests now serve an empty body so they hold the fix. - SweepAsync published even when cancellation was already requested, leaving a half-probed registry as the store's final state on shutdown. It now checks the token itself rather than relying on the transport to throw. Also: app.UseAntiforgery() is required despite the anonymous-by-design pipeline — AddRazorComponents stamps antiforgery metadata unconditionally and the endpoint middleware hard-fails without it (every page was 500). Chosen over DisableAntiforgery() so a future form is protected by default. 147 tests, 0 warnings. Live-verified end to end against fake endpoints: Up/Unreachable/Active/Standby, leader chip, cluster-data line, KPI counts.
This commit is contained in:
Joseph Doherty
@@ -0,0 +1,22 @@
|
||||
@* Host page. Head order matters: Bootstrap first (the Theme kit's shell and
|
||||
TechButton are built on Bootstrap utility classes), then <ThemeHead/> which
|
||||
brings the design tokens and IBM Plex, then site.css so this app's own
|
||||
dashboard anatomy can override either. *@
|
||||
<!doctype html>
|
||||
<html lang="en">
|
||||
<head>
|
||||
<meta charset="utf-8" />
|
||||
<meta name="viewport" content="width=device-width, initial-scale=1" />
|
||||
<base href="/" />
|
||||
<link rel="stylesheet" href="/lib/bootstrap/css/bootstrap.min.css" />
|
||||
<ThemeHead />
|
||||
<link rel="stylesheet" href="/css/site.css" />
|
||||
<HeadOutlet @rendermode="InteractiveServer" />
|
||||
</head>
|
||||
<body>
|
||||
<Routes @rendermode="InteractiveServer" />
|
||||
<script src="/lib/bootstrap/js/bootstrap.bundle.min.js"></script>
|
||||
<ThemeScripts />
|
||||
<script src="/_framework/blazor.web.js"></script>
|
||||
</body>
|
||||
</html>
|
||||
@@ -0,0 +1,43 @@
|
||||
@inherits LayoutComponentBase
|
||||
@inject OverviewSnapshotStore Store
|
||||
|
||||
@* Thin layout: the side-rail chassis belongs to the shared kit's <ThemeShell>.
|
||||
No AuthorizeView anywhere — this dashboard is anonymous by requirement, so
|
||||
there is no user to show and nothing to sign out of. The rail's application
|
||||
links are fragment anchors into the single page rather than routes: the whole
|
||||
fleet is one view, and splitting it per application would defeat the point. *@
|
||||
<ThemeShell Product="SCADA Overview" Accent="#6d28d9">
|
||||
<Nav>
|
||||
<NavRailItem Href="/" Text="Overview" Match="NavLinkMatch.All" />
|
||||
@if (Applications.Count > 0)
|
||||
{
|
||||
<NavRailSection Title="Applications" Key="overview-apps">
|
||||
@foreach (var application in Applications)
|
||||
{
|
||||
<a class="rail-link" href="@($"#{Slug(application)}")">@application</a>
|
||||
}
|
||||
</NavRailSection>
|
||||
}
|
||||
</Nav>
|
||||
<RailFooter>
|
||||
<div class="rail-note">read-only · anonymous<br />ZB.MOM.WW.Overview @Version</div>
|
||||
</RailFooter>
|
||||
<ChildContent>@Body</ChildContent>
|
||||
</ThemeShell>
|
||||
|
||||
@code {
|
||||
private static readonly string Version =
|
||||
typeof(MainLayout).Assembly.GetName().Version?.ToString(3) ?? "0.0.0";
|
||||
|
||||
private IReadOnlyList<string> Applications =>
|
||||
Store.Current.Applications.Select(a => a.Name).ToList();
|
||||
|
||||
/// <summary>
|
||||
/// Builds the fragment id for an application section. Must stay identical to the id the
|
||||
/// Overview page emits, or every rail link becomes a dead anchor.
|
||||
/// </summary>
|
||||
/// <param name="name">The application's display name.</param>
|
||||
/// <returns>A URL-fragment-safe slug.</returns>
|
||||
internal static string Slug(string name) =>
|
||||
new(name.Where(c => char.IsLetterOrDigit(c) || c is '-' or '_').Select(char.ToLowerInvariant).ToArray());
|
||||
}
|
||||
@@ -0,0 +1,245 @@
|
||||
@page "/"
|
||||
@implements IDisposable
|
||||
@inject OverviewSnapshotStore Store
|
||||
@inject TimeProvider Time
|
||||
|
||||
@* The whole dashboard, rendered from the snapshot store. This page never polls
|
||||
an instance: it subscribes to the store and re-renders when the poller swaps
|
||||
a snapshot in, plus a slow local tick so ages and staleness advance on their
|
||||
own between sweeps. *@
|
||||
|
||||
@* Read once per render and threaded through everything below, so every card, KPI
|
||||
and group header on one paint agrees on what "now" is. Reading the clock at
|
||||
render — rather than caching it when a snapshot arrives — is what lets the view
|
||||
age on its own: a paused or stuck poller publishes nothing, and the cards still
|
||||
cross into Stale. *@
|
||||
@{
|
||||
var now = Time.GetUtcNow();
|
||||
}
|
||||
|
||||
<PageTitle>SCADA Overview</PageTitle>
|
||||
|
||||
<div class="page-head rise">
|
||||
<h1>Fleet Overview</h1>
|
||||
<span class="page-meta">@Meta</span>
|
||||
<span class="spacer"></span>
|
||||
<span class="conn-pill" data-state="@(_paused ? "disconnected" : "connected")">
|
||||
<span class="dot"></span>@(_paused ? "Paused" : "Polling")
|
||||
</span>
|
||||
<button class="rail-btn" type="button" data-testid="pause-toggle" @onclick="TogglePause">
|
||||
@(_paused ? "Resume auto-refresh" : "Pause auto-refresh")
|
||||
</button>
|
||||
</div>
|
||||
|
||||
@if (_snapshot.Applications.Count == 0)
|
||||
{
|
||||
<p class="page-meta" data-testid="empty-registry">
|
||||
No instances have been polled yet.
|
||||
</p>
|
||||
}
|
||||
else
|
||||
{
|
||||
<div class="agg-grid rise">
|
||||
<div class="agg-card">
|
||||
<div class="agg-label">Instances</div>
|
||||
<div class="agg-value">@_instances.Count<span class="agg-sub">registered</span></div>
|
||||
</div>
|
||||
<div class="agg-card good">
|
||||
<div class="agg-label">Up</div>
|
||||
<div class="agg-value" data-testid="kpi-up">@Count(now, InstanceStatus.Up)</div>
|
||||
</div>
|
||||
<div class="agg-card @(Count(now, InstanceStatus.Degraded) > 0 ? "caution" : null)">
|
||||
<div class="agg-label">Degraded</div>
|
||||
<div class="agg-value" data-testid="kpi-degraded">@Count(now, InstanceStatus.Degraded)</div>
|
||||
</div>
|
||||
<div class="agg-card @(Count(now, InstanceStatus.Down) > 0 ? "alert" : null)">
|
||||
<div class="agg-label">Down</div>
|
||||
<div class="agg-value" data-testid="kpi-down">@Count(now, InstanceStatus.Down)</div>
|
||||
</div>
|
||||
<div class="agg-card @(Count(now, InstanceStatus.Unreachable) > 0 ? "alert" : null)">
|
||||
<div class="agg-label">Unreachable</div>
|
||||
<div class="agg-value" data-testid="kpi-unreachable">@Count(now, InstanceStatus.Unreachable)</div>
|
||||
</div>
|
||||
<div class="agg-card quiet">
|
||||
<div class="agg-label">Stale</div>
|
||||
<div class="agg-value" data-testid="kpi-stale">@StaleCount(now)</div>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
@foreach (var application in _snapshot.Applications)
|
||||
{
|
||||
<section class="app-section rise" id="@MainLayout.Slug(application.Name)" data-testid="application">
|
||||
<div class="app-head">
|
||||
<h2>@application.Name</h2>
|
||||
<StatusPill State="RollupState(application)">@RollupLabel(application)</StatusPill>
|
||||
<span class="spacer"></span>
|
||||
<span class="app-meta">@AppMeta(application)</span>
|
||||
</div>
|
||||
|
||||
@foreach (var group in Layout(application))
|
||||
{
|
||||
<div class="group">
|
||||
@if (group.Key is { } groupName)
|
||||
{
|
||||
var state = application.GroupFor(groupName);
|
||||
<div class="group-head">
|
||||
<span class="group-name">@groupName</span>
|
||||
@if (state?.LeaderInstance is { } leaderInstance)
|
||||
{
|
||||
<StatusPill State="StatusState.Info">Leader · @leaderInstance</StatusPill>
|
||||
}
|
||||
else if (state?.LeaderAddress is { } leaderAddress)
|
||||
{
|
||||
@* Unresolved address: shown raw rather than hidden — a leader on a
|
||||
host that is not in the registry is itself the finding. *@
|
||||
<StatusPill State="StatusState.Info">Leader · @leaderAddress</StatusPill>
|
||||
}
|
||||
@if (state?.Disagreement == true)
|
||||
{
|
||||
<StatusPill State="StatusState.Bad">
|
||||
Split brain · @string.Join(" vs ", state.ReportedLeaders)
|
||||
</StatusPill>
|
||||
}
|
||||
<span class="group-meta">@GroupMeta(group, now)</span>
|
||||
</div>
|
||||
}
|
||||
<div class="card-grid">
|
||||
@foreach (var instance in group)
|
||||
{
|
||||
<InstanceCard Instance="instance"
|
||||
Now="now"
|
||||
IsLeader="IsLeader(application, instance)" />
|
||||
}
|
||||
</div>
|
||||
</div>
|
||||
}
|
||||
</section>
|
||||
}
|
||||
}
|
||||
|
||||
@code {
|
||||
/// <summary>
|
||||
/// How often the page re-renders on its own. Independent of the poll interval: its job is to
|
||||
/// age the "last poll" figures and let a card cross into Stale even when no sweep lands —
|
||||
/// which is exactly the case a stuck poller produces.
|
||||
/// </summary>
|
||||
private static readonly TimeSpan UiTick = TimeSpan.FromSeconds(5);
|
||||
|
||||
private OverviewSnapshot _snapshot = OverviewSnapshot.Empty;
|
||||
private IReadOnlyList<InstanceSnapshot> _instances = [];
|
||||
private bool _paused;
|
||||
private ITimer? _timer;
|
||||
|
||||
protected override void OnInitialized()
|
||||
{
|
||||
Adopt(Store.Current);
|
||||
Store.Changed += OnSnapshotChanged;
|
||||
_timer = Time.CreateTimer(_ => InvokeAsync(StateHasChanged), null, UiTick, UiTick);
|
||||
}
|
||||
|
||||
private void OnSnapshotChanged(OverviewSnapshot snapshot)
|
||||
{
|
||||
// Pause stops ADOPTING new snapshots; the poller keeps sweeping. That is what makes the
|
||||
// pause useful — the frozen view then visibly ages into Stale, so a paused dashboard can
|
||||
// never be mistaken for a live one.
|
||||
if (_paused)
|
||||
return;
|
||||
|
||||
InvokeAsync(() =>
|
||||
{
|
||||
Adopt(snapshot);
|
||||
StateHasChanged();
|
||||
});
|
||||
}
|
||||
|
||||
private void Adopt(OverviewSnapshot snapshot)
|
||||
{
|
||||
_snapshot = snapshot;
|
||||
_instances = snapshot.AllInstances.ToList();
|
||||
}
|
||||
|
||||
private void TogglePause()
|
||||
{
|
||||
_paused = !_paused;
|
||||
|
||||
if (!_paused)
|
||||
Adopt(Store.Current);
|
||||
}
|
||||
|
||||
private int Count(DateTimeOffset now, InstanceStatus status) =>
|
||||
_instances.Count(i => i.Status == status && !i.IsStale(now));
|
||||
|
||||
private int StaleCount(DateTimeOffset now) => _instances.Count(i => i.IsStale(now));
|
||||
|
||||
private string Meta
|
||||
{
|
||||
get
|
||||
{
|
||||
var generated = _snapshot.GeneratedUtc == default
|
||||
? "never"
|
||||
: _snapshot.GeneratedUtc.ToLocalTime().ToString("HH:mm:ss", System.Globalization.CultureInfo.InvariantCulture);
|
||||
|
||||
return $"last sweep {generated} · {_instances.Count} instances";
|
||||
}
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Groups an application's instances for display: named cluster groups first in registry
|
||||
/// order, then the ungrouped ones.
|
||||
/// </summary>
|
||||
/// <param name="application">The application to lay out.</param>
|
||||
/// <returns>The instance groups.</returns>
|
||||
private static IEnumerable<IGrouping<string?, InstanceSnapshot>> Layout(ApplicationSnapshot application) =>
|
||||
application.Instances
|
||||
.GroupBy(i => i.Group, StringComparer.Ordinal)
|
||||
.OrderBy(g => g.Key is null)
|
||||
.ThenBy(g => g.Key, StringComparer.Ordinal);
|
||||
|
||||
private static string AppMeta(ApplicationSnapshot application)
|
||||
{
|
||||
var clusters = application.Groups.Count;
|
||||
var instances = application.Instances.Count;
|
||||
var clusterText = clusters == 0 ? "" : $" · {clusters} cluster{(clusters == 1 ? "" : "s")}";
|
||||
|
||||
return $"{instances} instance{(instances == 1 ? "" : "s")}{clusterText}";
|
||||
}
|
||||
|
||||
private static string GroupMeta(IGrouping<string?, InstanceSnapshot> group, DateTimeOffset now)
|
||||
{
|
||||
var members = group.Count();
|
||||
var missing = group.Count(i => i.Status is InstanceStatus.Unreachable || i.IsStale(now));
|
||||
|
||||
return $"{members} member{(members == 1 ? "" : "s")} · {missing} unreachable";
|
||||
}
|
||||
|
||||
private static bool IsLeader(ApplicationSnapshot application, InstanceSnapshot instance) =>
|
||||
instance.Group is { } group &&
|
||||
application.GroupFor(group)?.LeaderInstance is { } leader &&
|
||||
string.Equals(leader, instance.Instance, StringComparison.Ordinal);
|
||||
|
||||
private static StatusState RollupState(ApplicationSnapshot application) => application.Worst switch
|
||||
{
|
||||
InstanceStatus.Up => StatusState.Ok,
|
||||
InstanceStatus.Degraded => StatusState.Warn,
|
||||
InstanceStatus.Down or InstanceStatus.Unreachable => StatusState.Bad,
|
||||
_ => StatusState.Idle,
|
||||
};
|
||||
|
||||
private static string RollupLabel(ApplicationSnapshot application) => application.Worst switch
|
||||
{
|
||||
InstanceStatus.Up => "Up",
|
||||
InstanceStatus.Degraded => "Degraded",
|
||||
InstanceStatus.Down => "Down",
|
||||
InstanceStatus.Unreachable => "Unreachable",
|
||||
_ => "Pending",
|
||||
};
|
||||
|
||||
/// <inheritdoc/>
|
||||
public void Dispose()
|
||||
{
|
||||
// Both are required: an un-detached handler would call StateHasChanged on a disposed
|
||||
// renderer once per sweep for the life of the process.
|
||||
Store.Changed -= OnSnapshotChanged;
|
||||
_timer?.Dispose();
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,18 @@
|
||||
@* Plain RouteView, not AuthorizeRouteView. The dashboard is anonymous and
|
||||
read-only by requirement (design §2): there is no authentication pipeline to
|
||||
authorize against, and wiring one in would be the opposite of the decision. *@
|
||||
<Router AppAssembly="@typeof(Routes).Assembly">
|
||||
<Found Context="routeData">
|
||||
<RouteView RouteData="@routeData" DefaultLayout="@typeof(MainLayout)" />
|
||||
<FocusOnNavigate RouteData="@routeData" Selector="h1" />
|
||||
</Found>
|
||||
<NotFound>
|
||||
<LayoutView Layout="@typeof(MainLayout)">
|
||||
<PageTitle>Overview — Not Found</PageTitle>
|
||||
<section class="app-section">
|
||||
<h1 class="page-head">Not Found</h1>
|
||||
<p class="mb-0">The requested page does not exist.</p>
|
||||
</section>
|
||||
</LayoutView>
|
||||
</NotFound>
|
||||
</Router>
|
||||
@@ -0,0 +1,277 @@
|
||||
@* One instance = one card. Every value shown here comes from the snapshot the
|
||||
poller published; the component never probes anything itself. *@
|
||||
|
||||
<article class="inst @StateClass" data-testid="instance-card" data-instance="@Instance.Instance">
|
||||
<div class="inst-head">
|
||||
<div>
|
||||
<span class="inst-name">@Instance.Instance</span>
|
||||
<span class="inst-host">@HostAndPort</span>
|
||||
</div>
|
||||
<div class="inst-chips">
|
||||
@if (IsLeader)
|
||||
{
|
||||
<StatusPill State="StatusState.Info">Leader</StatusPill>
|
||||
}
|
||||
@if (Instance.Active is ActiveState.Active)
|
||||
{
|
||||
<StatusPill State="StatusState.Info">Active</StatusPill>
|
||||
}
|
||||
else if (Instance.Active is ActiveState.Standby)
|
||||
{
|
||||
<StatusPill State="StatusState.Idle">Standby</StatusPill>
|
||||
}
|
||||
else if (Instance.Active is ActiveState.Unknown)
|
||||
{
|
||||
<StatusPill State="StatusState.Idle">Role ?</StatusPill>
|
||||
}
|
||||
<StatusPill State="StatusPillState">@StatusLabel</StatusPill>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<div class="inst-body">
|
||||
<div class="kv">
|
||||
<span class="k">Latency</span>
|
||||
<span class="v">@LatencyText</span>
|
||||
</div>
|
||||
<div class="kv">
|
||||
<span class="k">@(IsStale ? "Stale since" : "Last poll")</span>
|
||||
<span class="v @(IsStale ? "idle" : null)">@LastPollText</span>
|
||||
</div>
|
||||
<div class="kv">
|
||||
<span class="k">Checks</span>
|
||||
<span class="v @ChecksClass">@ChecksText</span>
|
||||
</div>
|
||||
|
||||
@if (Instance.Error is { Length: > 0 } error)
|
||||
{
|
||||
<div class="kv">
|
||||
<span class="k">Error</span>
|
||||
<span class="v bad" title="@error">@Truncate(error, 48)</span>
|
||||
</div>
|
||||
}
|
||||
|
||||
@if (Entries.Count > 0)
|
||||
{
|
||||
<details class="checks">
|
||||
<summary>@Entries.Count check@(Entries.Count == 1 ? "" : "s")</summary>
|
||||
@foreach (var entry in Entries)
|
||||
{
|
||||
<div class="check-row">
|
||||
<span class="check-dot @DotClass(entry.Value.Status)"></span>
|
||||
<span class="check-name">@entry.Key</span>
|
||||
<span class="check-desc">@entry.Value.Description</span>
|
||||
</div>
|
||||
}
|
||||
@if (ClusterDataLine is { Length: > 0 } clusterData)
|
||||
{
|
||||
@* The cluster view, rendered verbatim rather than interpreted: when a pair
|
||||
disagrees about the leader this line is the raw evidence an operator needs
|
||||
to see, not a summary of it. *@
|
||||
<div class="check-data">@((MarkupString)clusterData)</div>
|
||||
}
|
||||
</details>
|
||||
}
|
||||
</div>
|
||||
|
||||
<div class="panel-foot">
|
||||
@if (Instance.ManagementUrl is { Length: > 0 } managementUrl)
|
||||
{
|
||||
<a href="@managementUrl" target="_blank" rel="noopener">Open @Instance.ManagementLabel ↗</a>
|
||||
}
|
||||
else
|
||||
{
|
||||
<span class="foot-note">no management link</span>
|
||||
}
|
||||
<span class="foot-note" data-testid="raw-signals">@RawSignals</span>
|
||||
</div>
|
||||
</article>
|
||||
|
||||
@code {
|
||||
/// <summary>The instance to render.</summary>
|
||||
[Parameter, EditorRequired] public InstanceSnapshot Instance { get; set; } = default!;
|
||||
|
||||
/// <summary>The render-time clock, passed down so every card on a page agrees on "now".</summary>
|
||||
[Parameter, EditorRequired] public DateTimeOffset Now { get; set; }
|
||||
|
||||
/// <summary>Whether this instance is its group's leader.</summary>
|
||||
[Parameter] public bool IsLeader { get; set; }
|
||||
|
||||
private bool IsStale => Instance.IsStale(Now);
|
||||
|
||||
private IReadOnlyList<KeyValuePair<string, ZbHealthEntry>> Entries =>
|
||||
Instance.Report?.Entries.OrderBy(e => e.Key, StringComparer.Ordinal).ToList() ?? [];
|
||||
|
||||
/// <summary>
|
||||
/// Stale wins over the stored status in the class, so the card visibly greys out. What it
|
||||
/// shows is still the last real reading — the point is to mark it as no longer current, not
|
||||
/// to replace it with a guess.
|
||||
/// </summary>
|
||||
private string StateClass => Instance.Status switch
|
||||
{
|
||||
null => "pending",
|
||||
_ when IsStale => $"{Instance.Status.ToString()!.ToLowerInvariant()} stale",
|
||||
var status => status.ToString()!.ToLowerInvariant(),
|
||||
};
|
||||
|
||||
private string StatusLabel => Instance.Status switch
|
||||
{
|
||||
null => "Pending",
|
||||
_ when IsStale => "Stale",
|
||||
InstanceStatus.Up => "Up",
|
||||
InstanceStatus.Degraded => "Degraded",
|
||||
InstanceStatus.Down => "Down",
|
||||
_ => "Unreachable",
|
||||
};
|
||||
|
||||
private StatusState StatusPillState => Instance.Status switch
|
||||
{
|
||||
null => StatusState.Idle,
|
||||
_ when IsStale => StatusState.Idle,
|
||||
InstanceStatus.Up => StatusState.Ok,
|
||||
InstanceStatus.Degraded => StatusState.Warn,
|
||||
_ => StatusState.Bad,
|
||||
};
|
||||
|
||||
private string HostAndPort
|
||||
{
|
||||
get
|
||||
{
|
||||
if (!Uri.TryCreate(Instance.BaseUrl, UriKind.Absolute, out var uri))
|
||||
return Instance.BaseUrl;
|
||||
|
||||
return uri.IsDefaultPort ? uri.Host : $"{uri.Host}:{uri.Port}";
|
||||
}
|
||||
}
|
||||
|
||||
private string LatencyText =>
|
||||
Instance.Status is null ? "—" : $"{Instance.Latency.TotalMilliseconds:0} ms";
|
||||
|
||||
private string LastPollText
|
||||
{
|
||||
get
|
||||
{
|
||||
// Stale cards count from the last successful contact, not the last poll attempt:
|
||||
// "stale since 4 s ago" while an instance has actually been gone for ten minutes
|
||||
// would be actively misleading.
|
||||
var reference = IsStale ? Instance.LastReachableUtc ?? Instance.LastPolledUtc : Instance.LastPolledUtc;
|
||||
return reference is { } at ? Relative(Now - at) : "never";
|
||||
}
|
||||
}
|
||||
|
||||
private string ChecksText
|
||||
{
|
||||
get
|
||||
{
|
||||
if (Entries.Count == 0)
|
||||
return Instance.Status is null ? "—" : "no detail";
|
||||
|
||||
var healthy = Entries.Count(e => string.Equals(e.Value.Status, "Healthy", StringComparison.Ordinal));
|
||||
return $"{healthy} / {Entries.Count} healthy";
|
||||
}
|
||||
}
|
||||
|
||||
private string? ChecksClass
|
||||
{
|
||||
get
|
||||
{
|
||||
if (Entries.Count == 0)
|
||||
return null;
|
||||
|
||||
if (Entries.Any(e => string.Equals(e.Value.Status, "Unhealthy", StringComparison.Ordinal)))
|
||||
return "bad";
|
||||
|
||||
return Entries.Any(e => string.Equals(e.Value.Status, "Degraded", StringComparison.Ordinal))
|
||||
? "warn"
|
||||
: "ok";
|
||||
}
|
||||
}
|
||||
|
||||
private string RawSignals
|
||||
{
|
||||
get
|
||||
{
|
||||
var ready = Instance.Status switch
|
||||
{
|
||||
null => "—",
|
||||
InstanceStatus.Unreachable => "fail",
|
||||
InstanceStatus.Down => "503",
|
||||
_ => "200",
|
||||
};
|
||||
|
||||
var active = Instance.Active switch
|
||||
{
|
||||
ActiveState.Active => "200",
|
||||
ActiveState.Standby => "503",
|
||||
ActiveState.Unknown => "—",
|
||||
_ => null,
|
||||
};
|
||||
|
||||
return active is null ? $"ready {ready}" : $"ready {ready} · active {active}";
|
||||
}
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Renders the akka-cluster check's data as the mockup's monospace evidence line.
|
||||
/// </summary>
|
||||
private string? ClusterDataLine
|
||||
{
|
||||
get
|
||||
{
|
||||
if (Instance.Report?.Entries.GetValueOrDefault(LeaderResolver.ClusterCheckName) is not { Data: not null } entry)
|
||||
return null;
|
||||
|
||||
var parts = new List<string>();
|
||||
|
||||
void Add(string label, string? value)
|
||||
{
|
||||
if (!string.IsNullOrEmpty(value))
|
||||
parts.Add($"<b>{Escape(label)}</b> {Escape(value)}");
|
||||
}
|
||||
|
||||
Add("leader", entry.DataString(LeaderResolver.LeaderDataKey));
|
||||
Add("members", entry.DataInt("memberCount")?.ToString());
|
||||
Add("unreachable", entry.DataInt("unreachableCount")?.ToString());
|
||||
Add("self", entry.DataString("selfAddress"));
|
||||
|
||||
return parts.Count == 0 ? null : string.Join(" · ", parts);
|
||||
}
|
||||
}
|
||||
|
||||
private static string DotClass(string? status) => status switch
|
||||
{
|
||||
"Healthy" => "ok",
|
||||
"Degraded" => "warn",
|
||||
"Unhealthy" => "bad",
|
||||
_ => "idle",
|
||||
};
|
||||
|
||||
/// <summary>Formats an age the way an operator reads a dashboard: coarse and instant.</summary>
|
||||
/// <param name="age">The elapsed time.</param>
|
||||
/// <returns>A short relative string.</returns>
|
||||
internal static string Relative(TimeSpan age)
|
||||
{
|
||||
if (age < TimeSpan.Zero)
|
||||
age = TimeSpan.Zero;
|
||||
|
||||
if (age.TotalSeconds < 60)
|
||||
return $"{age.TotalSeconds:0} s ago";
|
||||
|
||||
if (age.TotalMinutes < 60)
|
||||
return $"{age.TotalMinutes:0} min ago";
|
||||
|
||||
return age.TotalHours < 24 ? $"{age.TotalHours:0} h ago" : $"{age.TotalDays:0} d ago";
|
||||
}
|
||||
|
||||
private static string Truncate(string value, int max) =>
|
||||
value.Length <= max ? value : string.Concat(value.AsSpan(0, max - 1), "…");
|
||||
|
||||
/// <summary>
|
||||
/// HTML-escapes a value before it goes into the one MarkupString on this card. Health data is
|
||||
/// remote input from another application — trusted operationally, but never trusted to be
|
||||
/// markup-safe.
|
||||
/// </summary>
|
||||
/// <param name="value">The raw value.</param>
|
||||
/// <returns>The escaped value.</returns>
|
||||
private static string Escape(string value) =>
|
||||
System.Net.WebUtility.HtmlEncode(value);
|
||||
}
|
||||
@@ -0,0 +1,10 @@
|
||||
@using Microsoft.AspNetCore.Components.Routing
|
||||
@using Microsoft.AspNetCore.Components.Web
|
||||
@using ZB.MOM.WW.Overview.Components
|
||||
@using ZB.MOM.WW.Overview.Components.Layout
|
||||
@using ZB.MOM.WW.Overview.Components.Pages
|
||||
@using ZB.MOM.WW.Overview.Components.Widgets
|
||||
@using ZB.MOM.WW.Overview.Polling
|
||||
@using ZB.MOM.WW.Overview.Registry
|
||||
@using ZB.MOM.WW.Theme
|
||||
@using static Microsoft.AspNetCore.Components.Web.RenderMode
|
||||
@@ -0,0 +1,132 @@
|
||||
namespace ZB.MOM.WW.Overview.Polling;
|
||||
|
||||
/// <summary>
|
||||
/// Derives per-group leader state from what the members of a cluster group report in their
|
||||
/// <c>akka-cluster</c> health data.
|
||||
/// </summary>
|
||||
/// <remarks>
|
||||
/// Pure and static: given the same instance snapshots it always produces the same answer, which is
|
||||
/// what makes the split-brain rule — the one output an operator would act on at 3am — exhaustively
|
||||
/// testable without a cluster.
|
||||
/// </remarks>
|
||||
public static class LeaderResolver
|
||||
{
|
||||
/// <summary>The health check whose data carries the cluster view.</summary>
|
||||
public const string ClusterCheckName = "akka-cluster";
|
||||
|
||||
/// <summary>The data key holding the Akka leader address.</summary>
|
||||
public const string LeaderDataKey = "leader";
|
||||
|
||||
/// <summary>Resolves leader state for every cluster group present in <paramref name="instances"/>.</summary>
|
||||
/// <param name="instances">One application's instances.</param>
|
||||
/// <returns>Group state in first-appearance order; empty when the application has no groups.</returns>
|
||||
public static IReadOnlyList<GroupSnapshot> Resolve(IEnumerable<InstanceSnapshot> instances)
|
||||
{
|
||||
ArgumentNullException.ThrowIfNull(instances);
|
||||
|
||||
var all = instances as IReadOnlyList<InstanceSnapshot> ?? instances.ToList();
|
||||
var groups = new List<GroupSnapshot>();
|
||||
|
||||
foreach (var name in all
|
||||
.Select(i => i.Group)
|
||||
.Where(g => !string.IsNullOrWhiteSpace(g))
|
||||
.Distinct(StringComparer.Ordinal))
|
||||
{
|
||||
var members = all.Where(i => string.Equals(i.Group, name, StringComparison.Ordinal)).ToList();
|
||||
|
||||
// A group of instances that have no active/standby concept has no leader to report;
|
||||
// rendering a leader chip for one would invent a distinction that does not exist.
|
||||
if (!members.Exists(m => m.HasActiveRole))
|
||||
continue;
|
||||
|
||||
groups.Add(ResolveGroup(name!, members, all));
|
||||
}
|
||||
|
||||
return groups;
|
||||
}
|
||||
|
||||
private static GroupSnapshot ResolveGroup(
|
||||
string name,
|
||||
IReadOnlyList<InstanceSnapshot> members,
|
||||
IReadOnlyList<InstanceSnapshot> allInstances)
|
||||
{
|
||||
// Only members that are actually talking get a vote. A Down or Unreachable node's last
|
||||
// known leader is stale by definition, and counting it would manufacture a disagreement
|
||||
// every time a node goes away mid-failover — the exact moment the flag must stay honest.
|
||||
var votes = members
|
||||
.Where(m => m.Status is InstanceStatus.Up or InstanceStatus.Degraded)
|
||||
.Select(m => m.Report?.Entries.GetValueOrDefault(ClusterCheckName)?.DataString(LeaderDataKey))
|
||||
.Where(address => !string.IsNullOrWhiteSpace(address))
|
||||
.Select(address => address!)
|
||||
.ToList();
|
||||
|
||||
if (votes.Count == 0)
|
||||
return new GroupSnapshot(name, null, null, Disagreement: false, []);
|
||||
|
||||
var distinct = votes.Distinct(StringComparer.Ordinal).OrderBy(a => a, StringComparer.Ordinal).ToList();
|
||||
|
||||
// The most-reported address wins the chip, ties broken ordinally so the display is stable
|
||||
// rather than flickering between two equally-supported claims on consecutive sweeps.
|
||||
var winner = votes
|
||||
.GroupBy(a => a, StringComparer.Ordinal)
|
||||
.OrderByDescending(g => g.Count())
|
||||
.ThenBy(g => g.Key, StringComparer.Ordinal)
|
||||
.First()
|
||||
.Key;
|
||||
|
||||
return new GroupSnapshot(
|
||||
name,
|
||||
ResolveToInstanceName(winner, allInstances),
|
||||
winner,
|
||||
Disagreement: distinct.Count > 1,
|
||||
distinct);
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Maps an Akka address to a registry instance name by host.
|
||||
/// </summary>
|
||||
/// <param name="akkaAddress">An address of the form <c>akka.tcp://system@host:port</c>.</param>
|
||||
/// <param name="instances">Candidate instances.</param>
|
||||
/// <returns>The matching instance's name, or null when nothing matches.</returns>
|
||||
/// <remarks>
|
||||
/// Host only, deliberately: the Akka remoting port and the HTTP port are different by design, so
|
||||
/// comparing ports would never match. An unmatched address is not an error — the card falls back
|
||||
/// to showing the raw address, which is still the useful thing to see.
|
||||
/// </remarks>
|
||||
internal static string? ResolveToInstanceName(string akkaAddress, IReadOnlyList<InstanceSnapshot> instances)
|
||||
{
|
||||
var host = HostOf(akkaAddress);
|
||||
if (host is null)
|
||||
return null;
|
||||
|
||||
foreach (var instance in instances)
|
||||
{
|
||||
if (Uri.TryCreate(instance.BaseUrl, UriKind.Absolute, out var uri) &&
|
||||
string.Equals(uri.Host, host, StringComparison.OrdinalIgnoreCase))
|
||||
{
|
||||
return instance.Instance;
|
||||
}
|
||||
}
|
||||
|
||||
return null;
|
||||
}
|
||||
|
||||
/// <summary>Extracts the host from an Akka address.</summary>
|
||||
/// <param name="akkaAddress">The address to parse.</param>
|
||||
/// <returns>The host, or null when the address is not in the expected form.</returns>
|
||||
internal static string? HostOf(string? akkaAddress)
|
||||
{
|
||||
if (string.IsNullOrWhiteSpace(akkaAddress))
|
||||
return null;
|
||||
|
||||
var at = akkaAddress.LastIndexOf('@');
|
||||
if (at < 0 || at == akkaAddress.Length - 1)
|
||||
return null;
|
||||
|
||||
var hostAndPort = akkaAddress[(at + 1)..];
|
||||
var colon = hostAndPort.LastIndexOf(':');
|
||||
var host = colon > 0 ? hostAndPort[..colon] : hostAndPort;
|
||||
|
||||
return host.Length == 0 ? null : host;
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,239 @@
|
||||
using Microsoft.Extensions.Options;
|
||||
using ZB.MOM.WW.Overview.Registry;
|
||||
|
||||
namespace ZB.MOM.WW.Overview.Polling;
|
||||
|
||||
/// <summary>
|
||||
/// The single writer: sweeps every registered instance on a timer and publishes each result set to
|
||||
/// <see cref="OverviewSnapshotStore"/>.
|
||||
/// </summary>
|
||||
/// <remarks>
|
||||
/// No retries. A failed probe is simply reported as failed and the next tick is the retry, which
|
||||
/// keeps the sweep bounded in time and keeps the two-strike damping in
|
||||
/// <see cref="StatusDerivation.Damp"/> the only place that decides whether a failure is worth
|
||||
/// showing. Retrying inside a sweep would silently double the effective strike count.
|
||||
/// </remarks>
|
||||
public sealed class OverviewPollerService : BackgroundService
|
||||
{
|
||||
/// <summary>Name of the pooled <see cref="HttpClient"/> used for probes.</summary>
|
||||
public const string HttpClientName = "overview-health";
|
||||
|
||||
/// <summary>
|
||||
/// Slack allowed when deciding whether an instance is due. A timer tick can arrive a hair early
|
||||
/// relative to the recorded sweep time, and without the tolerance such an instance would be
|
||||
/// skipped and end up polling at half its configured rate.
|
||||
/// </summary>
|
||||
private static readonly TimeSpan DueTolerance = TimeSpan.FromMilliseconds(250);
|
||||
|
||||
private readonly IHttpClientFactory _httpClientFactory;
|
||||
private readonly OverviewSnapshotStore _store;
|
||||
private readonly TimeProvider _timeProvider;
|
||||
private readonly ILogger<OverviewPollerService> _logger;
|
||||
private readonly List<PollTarget> _targets;
|
||||
|
||||
/// <summary>Initializes a new <see cref="OverviewPollerService"/>.</summary>
|
||||
/// <param name="options">The validated registry.</param>
|
||||
/// <param name="httpClientFactory">Factory for the pooled probe client.</param>
|
||||
/// <param name="store">The snapshot store this service writes to.</param>
|
||||
/// <param name="timeProvider">Clock and timer source.</param>
|
||||
/// <param name="logger">Logger.</param>
|
||||
public OverviewPollerService(
|
||||
IOptions<OverviewOptions> options,
|
||||
IHttpClientFactory httpClientFactory,
|
||||
OverviewSnapshotStore store,
|
||||
TimeProvider timeProvider,
|
||||
ILogger<OverviewPollerService> logger)
|
||||
{
|
||||
ArgumentNullException.ThrowIfNull(options);
|
||||
|
||||
_httpClientFactory = httpClientFactory ?? throw new ArgumentNullException(nameof(httpClientFactory));
|
||||
_store = store ?? throw new ArgumentNullException(nameof(store));
|
||||
_timeProvider = timeProvider ?? throw new ArgumentNullException(nameof(timeProvider));
|
||||
_logger = logger ?? throw new ArgumentNullException(nameof(logger));
|
||||
_targets = BuildTargets(options.Value);
|
||||
}
|
||||
|
||||
/// <inheritdoc/>
|
||||
protected override async Task ExecuteAsync(CancellationToken stoppingToken)
|
||||
{
|
||||
if (_targets.Count == 0)
|
||||
{
|
||||
_logger.LogWarning("Overview registry contains no instances; the poller has nothing to do.");
|
||||
return;
|
||||
}
|
||||
|
||||
// Tick at the fastest configured cadence and let each target decide whether it is due.
|
||||
// A single timer honours per-instance overrides without spawning a timer per instance.
|
||||
var tick = _targets.Min(t => t.Timings.PollInterval);
|
||||
|
||||
// Publish the registry before probing anything, so the page paints its full layout at once
|
||||
// instead of appearing empty for the duration of the first sweep.
|
||||
_store.Publish(BuildSnapshot(_timeProvider.GetUtcNow()));
|
||||
|
||||
using var timer = new PeriodicTimer(tick, _timeProvider);
|
||||
|
||||
try
|
||||
{
|
||||
do
|
||||
{
|
||||
await SweepAsync(stoppingToken).ConfigureAwait(false);
|
||||
}
|
||||
while (await timer.WaitForNextTickAsync(stoppingToken).ConfigureAwait(false));
|
||||
}
|
||||
catch (OperationCanceledException) when (stoppingToken.IsCancellationRequested)
|
||||
{
|
||||
// Ordinary shutdown.
|
||||
}
|
||||
}
|
||||
|
||||
/// <summary>Runs one sweep: probes every due instance and publishes the resulting snapshot.</summary>
|
||||
/// <param name="cancellationToken">Cancelled on host shutdown.</param>
|
||||
/// <returns>The snapshot that was published.</returns>
|
||||
internal async Task<OverviewSnapshot> SweepAsync(CancellationToken cancellationToken = default)
|
||||
{
|
||||
// Checked here rather than left to the transport: whether an in-flight request observes the
|
||||
// token is a property of the handler stack, but "a sweep that started after shutdown must
|
||||
// not publish" has to hold unconditionally, or the store keeps a half-probed registry as
|
||||
// its final state.
|
||||
cancellationToken.ThrowIfCancellationRequested();
|
||||
|
||||
var now = _timeProvider.GetUtcNow();
|
||||
var due = _targets.Where(t => IsDue(t, now)).ToList();
|
||||
|
||||
if (due.Count > 0)
|
||||
{
|
||||
// One client per sweep, not per probe: IHttpClientFactory pools the handler either way,
|
||||
// and a single instance keeps connection reuse across the fan-out.
|
||||
var client = _httpClientFactory.CreateClient(HttpClientName);
|
||||
var probe = new ZbHealthReportClient(client);
|
||||
|
||||
var results = await Task
|
||||
.WhenAll(due.Select(target => ProbeAsync(probe, target, now, cancellationToken)))
|
||||
.ConfigureAwait(false);
|
||||
|
||||
for (var i = 0; i < due.Count; i++)
|
||||
due[i].Last = results[i];
|
||||
}
|
||||
|
||||
// Re-checked after the fan-out: a shutdown that lands mid-sweep leaves the probed targets
|
||||
// in an arbitrary mix of fresh and stale results, and publishing that mix would freeze a
|
||||
// misleading picture into the store for as long as the process takes to exit.
|
||||
cancellationToken.ThrowIfCancellationRequested();
|
||||
|
||||
var snapshot = BuildSnapshot(now);
|
||||
_store.Publish(snapshot);
|
||||
|
||||
return snapshot;
|
||||
}
|
||||
|
||||
private bool IsDue(PollTarget target, DateTimeOffset now) =>
|
||||
target.Last?.LastPolledUtc is not { } polled || now - polled + DueTolerance >= target.Timings.PollInterval;
|
||||
|
||||
private static async Task<InstanceSnapshot> ProbeAsync(
|
||||
ZbHealthReportClient probe,
|
||||
PollTarget target,
|
||||
DateTimeOffset now,
|
||||
CancellationToken cancellationToken)
|
||||
{
|
||||
var timeout = target.Timings.Timeout;
|
||||
var ready = await probe.ProbeAsync(target.ReadyUrl, timeout, cancellationToken).ConfigureAwait(false);
|
||||
|
||||
// Skip the active probe when nothing answered the ready probe: it would spend a second full
|
||||
// timeout to learn what we already know, and ObserveActive maps a missing probe to Unknown
|
||||
// anyway. A 503 from ready is NOT a skip — that instance is talking, and whether it is the
|
||||
// active half is exactly what an operator wants to know about a sick node.
|
||||
HealthProbeResult? active = null;
|
||||
if (target.Instance.HasActiveRole && ready.Reachable)
|
||||
active = await probe.ProbeStatusAsync(target.ActiveUrl, timeout, cancellationToken).ConfigureAwait(false);
|
||||
|
||||
var observed = StatusDerivation.Observe(ready);
|
||||
var (status, failures) = StatusDerivation.Damp(
|
||||
target.Last?.Status,
|
||||
target.Last?.ConsecutiveFailures ?? 0,
|
||||
observed);
|
||||
|
||||
return target.Template with
|
||||
{
|
||||
Status = status,
|
||||
Active = StatusDerivation.ObserveActive(target.Instance.HasActiveRole, active),
|
||||
ConsecutiveFailures = failures,
|
||||
LastPolledUtc = now,
|
||||
LastReachableUtc = ready.Reachable ? now : target.Last?.LastReachableUtc,
|
||||
Latency = ready.Latency,
|
||||
Error = ready.Error,
|
||||
Report = ready.Report,
|
||||
};
|
||||
}
|
||||
|
||||
private OverviewSnapshot BuildSnapshot(DateTimeOffset now)
|
||||
{
|
||||
var applications = _targets
|
||||
.GroupBy(t => t.Application, StringComparer.Ordinal)
|
||||
.Select(group =>
|
||||
{
|
||||
var instances = group.Select(t => t.Last ?? t.Template).ToList();
|
||||
return new ApplicationSnapshot(group.Key, instances, LeaderResolver.Resolve(instances));
|
||||
})
|
||||
.ToList();
|
||||
|
||||
return new OverviewSnapshot(now, applications);
|
||||
}
|
||||
|
||||
private static List<PollTarget> BuildTargets(OverviewOptions options)
|
||||
{
|
||||
var targets = new List<PollTarget>();
|
||||
|
||||
foreach (var application in options.Applications)
|
||||
{
|
||||
foreach (var instance in application.Instances)
|
||||
{
|
||||
var baseUrl = instance.BaseUrl.TrimEnd('/');
|
||||
|
||||
targets.Add(new PollTarget
|
||||
{
|
||||
Application = application.Name,
|
||||
Instance = instance,
|
||||
Timings = EffectiveTimings.Resolve(options, application, instance),
|
||||
ReadyUrl = $"{baseUrl}/health/ready",
|
||||
ActiveUrl = $"{baseUrl}/health/active",
|
||||
Template = new InstanceSnapshot
|
||||
{
|
||||
Application = application.Name,
|
||||
Instance = instance.Name,
|
||||
Group = instance.Group,
|
||||
BaseUrl = baseUrl,
|
||||
ManagementUrl = instance.ManagementUrl,
|
||||
ManagementLabel = application.ManagementLabel,
|
||||
HasActiveRole = instance.HasActiveRole,
|
||||
StaleAfter = EffectiveTimings.Resolve(options, application, instance).StaleAfter,
|
||||
},
|
||||
});
|
||||
}
|
||||
}
|
||||
|
||||
return targets;
|
||||
}
|
||||
|
||||
/// <summary>One instance's fixed probe configuration plus its most recent result.</summary>
|
||||
private sealed class PollTarget
|
||||
{
|
||||
public required string Application { get; init; }
|
||||
|
||||
public required InstanceEntry Instance { get; init; }
|
||||
|
||||
public required EffectiveTimings Timings { get; init; }
|
||||
|
||||
public required string ReadyUrl { get; init; }
|
||||
|
||||
public required string ActiveUrl { get; init; }
|
||||
|
||||
/// <summary>The registry-derived fields, reused as the base of every result.</summary>
|
||||
public required InstanceSnapshot Template { get; init; }
|
||||
|
||||
/// <summary>
|
||||
/// The last published result, or null before the first probe. Written only by the sweep
|
||||
/// loop after its <c>Task.WhenAll</c> has completed, so it is never touched concurrently.
|
||||
/// </summary>
|
||||
public InstanceSnapshot? Last { get; set; }
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,172 @@
|
||||
namespace ZB.MOM.WW.Overview.Polling;
|
||||
|
||||
/// <summary>
|
||||
/// Everything the dashboard knows about one instance, as of the last sweep that included it.
|
||||
/// </summary>
|
||||
/// <remarks>
|
||||
/// Immutable by construction: the poller builds a whole new object graph each sweep and swaps it
|
||||
/// into <see cref="OverviewSnapshotStore"/> atomically, so a render can never observe a half-updated
|
||||
/// registry.
|
||||
/// </remarks>
|
||||
public sealed record InstanceSnapshot
|
||||
{
|
||||
/// <summary>Owning application's display name.</summary>
|
||||
public required string Application { get; init; }
|
||||
|
||||
/// <summary>This instance's display name, unique within its application.</summary>
|
||||
public required string Instance { get; init; }
|
||||
|
||||
/// <summary>Cluster group, or null for a standalone instance.</summary>
|
||||
public string? Group { get; init; }
|
||||
|
||||
/// <summary>The probed base URL, shown on the card for diagnosis.</summary>
|
||||
public required string BaseUrl { get; init; }
|
||||
|
||||
/// <summary>Management UI link, when the registry supplies one.</summary>
|
||||
public string? ManagementUrl { get; init; }
|
||||
|
||||
/// <summary>Label for <see cref="ManagementUrl"/>, from the application entry.</summary>
|
||||
public string ManagementLabel { get; init; } = "UI";
|
||||
|
||||
/// <summary>Whether this instance is half of an active/standby pair.</summary>
|
||||
public bool HasActiveRole { get; init; }
|
||||
|
||||
/// <summary>
|
||||
/// The damped, displayable status — null until the first sweep has actually probed this
|
||||
/// instance.
|
||||
/// </summary>
|
||||
/// <remarks>
|
||||
/// Null is a real, distinct state and not a stand-in for "Up": the registry is published to the
|
||||
/// store before the first probe returns so the page renders its full layout immediately, and
|
||||
/// showing an unmeasured instance as healthy for that half-second would be a fabricated reading.
|
||||
/// </remarks>
|
||||
public InstanceStatus? Status { get; init; }
|
||||
|
||||
/// <summary>Active/standby state from the active tier.</summary>
|
||||
public ActiveState Active { get; init; } = ActiveState.NotApplicable;
|
||||
|
||||
/// <summary>Consecutive failing observations, carried across sweeps to drive flap damping.</summary>
|
||||
public int ConsecutiveFailures { get; init; }
|
||||
|
||||
/// <summary>
|
||||
/// When the sweep that produced this snapshot started; null before the first probe. Staleness
|
||||
/// is measured from here, so a stuck or paused poller ages every card even though no probe
|
||||
/// failed.
|
||||
/// </summary>
|
||||
public DateTimeOffset? LastPolledUtc { get; init; }
|
||||
|
||||
/// <summary>When this instance last answered a probe at all, regardless of what it said.</summary>
|
||||
public DateTimeOffset? LastReachableUtc { get; init; }
|
||||
|
||||
/// <summary>Wall-clock duration of the last ready probe.</summary>
|
||||
public TimeSpan Latency { get; init; }
|
||||
|
||||
/// <summary>Why the last probe failed, when it did.</summary>
|
||||
public string? Error { get; init; }
|
||||
|
||||
/// <summary>The last parsed ready-tier body, which drives the per-check detail list.</summary>
|
||||
public ZbHealthReport? Report { get; init; }
|
||||
|
||||
/// <summary>This instance's effective staleness window.</summary>
|
||||
public TimeSpan StaleAfter { get; init; }
|
||||
|
||||
/// <summary>Whether this snapshot is too old to be trusted as current.</summary>
|
||||
/// <param name="now">The current time.</param>
|
||||
/// <returns>True when the snapshot has aged past its staleness window.</returns>
|
||||
/// <remarks>
|
||||
/// Evaluated at render rather than stored, because staleness is a function of the clock and not
|
||||
/// of anything the poller observed — a snapshot that was fresh when written goes stale on its
|
||||
/// own while nothing at all happens.
|
||||
/// </remarks>
|
||||
public bool IsStale(DateTimeOffset now) =>
|
||||
Status is not null && LastPolledUtc is { } polled && now - polled > StaleAfter;
|
||||
}
|
||||
|
||||
/// <summary>Leader state for one cluster group, derived from what its members report.</summary>
|
||||
/// <param name="Name">The group name from the registry.</param>
|
||||
/// <param name="LeaderInstance">
|
||||
/// Registry name of the instance the reported leader address resolves to, or null when no member
|
||||
/// reported a leader or the address matches no registered instance.
|
||||
/// </param>
|
||||
/// <param name="LeaderAddress">The reported Akka leader address, shown raw when unresolved.</param>
|
||||
/// <param name="Disagreement">
|
||||
/// True when reachable members reported different leaders — the split-brain tell.
|
||||
/// </param>
|
||||
/// <param name="ReportedLeaders">Every distinct address reported, for the disagreement detail.</param>
|
||||
public sealed record GroupSnapshot(
|
||||
string Name,
|
||||
string? LeaderInstance,
|
||||
string? LeaderAddress,
|
||||
bool Disagreement,
|
||||
IReadOnlyList<string> ReportedLeaders);
|
||||
|
||||
/// <summary>One application section of the dashboard.</summary>
|
||||
/// <param name="Name">The application's display name.</param>
|
||||
/// <param name="Instances">Its instances, in registry order.</param>
|
||||
/// <param name="Groups">Leader state per cluster group, in registry order.</param>
|
||||
public sealed record ApplicationSnapshot(
|
||||
string Name,
|
||||
IReadOnlyList<InstanceSnapshot> Instances,
|
||||
IReadOnlyList<GroupSnapshot> Groups)
|
||||
{
|
||||
/// <summary>Finds the group state for <paramref name="group"/>.</summary>
|
||||
/// <param name="group">The group name, or null for standalone instances.</param>
|
||||
/// <returns>The group snapshot, or null when there is none.</returns>
|
||||
public GroupSnapshot? GroupFor(string? group) =>
|
||||
group is null ? null : Groups.FirstOrDefault(g => string.Equals(g.Name, group, StringComparison.Ordinal));
|
||||
|
||||
/// <summary>
|
||||
/// The most severe status across this application's probed instances, for the rollup pill;
|
||||
/// null while nothing has been probed yet.
|
||||
/// </summary>
|
||||
public InstanceStatus? Worst
|
||||
{
|
||||
get
|
||||
{
|
||||
InstanceStatus? worst = null;
|
||||
foreach (var instance in Instances)
|
||||
{
|
||||
if (instance.Status is not { } status)
|
||||
continue;
|
||||
|
||||
if (worst is null || StatusSeverity.Rank(status) > StatusSeverity.Rank(worst.Value))
|
||||
worst = status;
|
||||
}
|
||||
|
||||
return worst;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// <summary>A complete, atomically-published view of the whole registry.</summary>
|
||||
/// <param name="GeneratedUtc">When the sweep that produced this snapshot started.</param>
|
||||
/// <param name="Applications">The applications, in registry order.</param>
|
||||
public sealed record OverviewSnapshot(DateTimeOffset GeneratedUtc, IReadOnlyList<ApplicationSnapshot> Applications)
|
||||
{
|
||||
/// <summary>The pre-registry snapshot served before the poller has published anything.</summary>
|
||||
public static OverviewSnapshot Empty { get; } = new(default, []);
|
||||
|
||||
/// <summary>Every instance across every application, flattened.</summary>
|
||||
public IEnumerable<InstanceSnapshot> AllInstances => Applications.SelectMany(a => a.Instances);
|
||||
}
|
||||
|
||||
/// <summary>Severity ordering for rollups.</summary>
|
||||
public static class StatusSeverity
|
||||
{
|
||||
/// <summary>Ranks a status; higher is worse.</summary>
|
||||
/// <param name="status">The status to rank.</param>
|
||||
/// <returns>The severity rank.</returns>
|
||||
/// <remarks>
|
||||
/// <see cref="InstanceStatus.Unreachable"/> outranks <see cref="InstanceStatus.Down"/> because
|
||||
/// it carries strictly less information: a Down instance told us what is wrong, an Unreachable
|
||||
/// one did not answer at all and could be anything up to and including gone.
|
||||
/// </remarks>
|
||||
public static int Rank(InstanceStatus status) => status switch
|
||||
{
|
||||
InstanceStatus.Up => 0,
|
||||
InstanceStatus.Degraded => 1,
|
||||
InstanceStatus.Down => 2,
|
||||
InstanceStatus.Unreachable => 3,
|
||||
_ => 3,
|
||||
};
|
||||
}
|
||||
@@ -0,0 +1,38 @@
|
||||
namespace ZB.MOM.WW.Overview.Polling;
|
||||
|
||||
/// <summary>
|
||||
/// Holds the current <see cref="OverviewSnapshot"/> and notifies subscribers when it is replaced.
|
||||
/// </summary>
|
||||
/// <remarks>
|
||||
/// This is the cache-first requirement made concrete: the poller is the only writer, and every
|
||||
/// render reads whatever is already here. A page load therefore paints the last known state
|
||||
/// immediately and never waits on — or triggers — a probe of its own.
|
||||
/// </remarks>
|
||||
public sealed class OverviewSnapshotStore
|
||||
{
|
||||
private OverviewSnapshot _current = OverviewSnapshot.Empty;
|
||||
|
||||
/// <summary>Raised after a new snapshot has been published.</summary>
|
||||
/// <remarks>
|
||||
/// Handlers run on the poller's thread, so a Blazor subscriber must marshal onto its own
|
||||
/// synchronization context (<c>InvokeAsync(StateHasChanged)</c>) and must unsubscribe when
|
||||
/// disposed.
|
||||
/// </remarks>
|
||||
public event Action<OverviewSnapshot>? Changed;
|
||||
|
||||
/// <summary>The current snapshot; never null.</summary>
|
||||
public OverviewSnapshot Current => Volatile.Read(ref _current);
|
||||
|
||||
/// <summary>Replaces the current snapshot and notifies subscribers.</summary>
|
||||
/// <param name="snapshot">The new snapshot.</param>
|
||||
public void Publish(OverviewSnapshot snapshot)
|
||||
{
|
||||
ArgumentNullException.ThrowIfNull(snapshot);
|
||||
|
||||
// A single reference swap of an already-complete graph. Readers hold their own reference
|
||||
// for the duration of a render, so a sweep landing mid-render cannot tear the view.
|
||||
Volatile.Write(ref _current, snapshot);
|
||||
|
||||
Changed?.Invoke(snapshot);
|
||||
}
|
||||
}
|
||||
+33
@@ -0,0 +1,33 @@
|
||||
using Microsoft.Extensions.DependencyInjection.Extensions;
|
||||
|
||||
namespace ZB.MOM.WW.Overview.Polling;
|
||||
|
||||
/// <summary>Registers the polling pipeline.</summary>
|
||||
public static class PollingServiceCollectionExtensions
|
||||
{
|
||||
/// <summary>Adds the snapshot store, the probe client and the poller.</summary>
|
||||
/// <param name="services">The service collection.</param>
|
||||
/// <returns>The same collection, for chaining.</returns>
|
||||
public static IServiceCollection AddOverviewPolling(this IServiceCollection services)
|
||||
{
|
||||
ArgumentNullException.ThrowIfNull(services);
|
||||
|
||||
services.TryAddSingleton(TimeProvider.System);
|
||||
services.TryAddSingleton<OverviewSnapshotStore>();
|
||||
|
||||
services.AddHttpClient(OverviewPollerService.HttpClientName, client =>
|
||||
{
|
||||
// The per-request CancellationTokenSource in ZbHealthReportClient is the only timeout
|
||||
// that should ever fire. Leaving HttpClient's own 100s default in place would let it
|
||||
// pre-empt a per-instance override and report "timed out after 3s" for a 100s wait.
|
||||
client.Timeout = Timeout.InfiniteTimeSpan;
|
||||
|
||||
// Health bodies are small and must not be served from anywhere but the instance itself.
|
||||
client.DefaultRequestHeaders.CacheControl = new() { NoCache = true, NoStore = true };
|
||||
});
|
||||
|
||||
services.AddHostedService<OverviewPollerService>();
|
||||
|
||||
return services;
|
||||
}
|
||||
}
|
||||
@@ -44,15 +44,43 @@ public sealed class ZbHealthReportClient
|
||||
public ZbHealthReportClient(HttpClient httpClient) =>
|
||||
_httpClient = httpClient ?? throw new ArgumentNullException(nameof(httpClient));
|
||||
|
||||
/// <summary>Probes <paramref name="url"/> and parses the response.</summary>
|
||||
/// <summary>Probes <paramref name="url"/> and parses the response body.</summary>
|
||||
/// <param name="url">Absolute URL of a health endpoint.</param>
|
||||
/// <param name="timeout">Per-request timeout.</param>
|
||||
/// <param name="cancellationToken">Token cancelled when the host is shutting down.</param>
|
||||
/// <returns>The probe result; this never throws for an unreachable endpoint.</returns>
|
||||
public async Task<HealthProbeResult> ProbeAsync(
|
||||
public Task<HealthProbeResult> ProbeAsync(
|
||||
string url,
|
||||
TimeSpan timeout,
|
||||
CancellationToken cancellationToken = default)
|
||||
CancellationToken cancellationToken = default) =>
|
||||
SendAsync(url, timeout, bodyRequired: true, cancellationToken);
|
||||
|
||||
/// <summary>
|
||||
/// Probes <paramref name="url"/> for its status code alone, parsing the body only if one
|
||||
/// happens to be there.
|
||||
/// </summary>
|
||||
/// <param name="url">Absolute URL of a health endpoint.</param>
|
||||
/// <param name="timeout">Per-request timeout.</param>
|
||||
/// <param name="cancellationToken">Token cancelled when the host is shutting down.</param>
|
||||
/// <returns>The probe result; this never throws for an unreachable endpoint.</returns>
|
||||
/// <remarks>
|
||||
/// For the active tier the status code IS the answer — 200 means active, 503 means standby —
|
||||
/// and the body carries nothing the dashboard uses. Demanding a parseable body here would
|
||||
/// report every node behind anything that answers the tier with an empty response (a bare
|
||||
/// <c>IActiveNodeGate</c> endpoint, a load balancer, a proxy) as "role unknown" while it is
|
||||
/// plainly telling us the role.
|
||||
/// </remarks>
|
||||
public Task<HealthProbeResult> ProbeStatusAsync(
|
||||
string url,
|
||||
TimeSpan timeout,
|
||||
CancellationToken cancellationToken = default) =>
|
||||
SendAsync(url, timeout, bodyRequired: false, cancellationToken);
|
||||
|
||||
private async Task<HealthProbeResult> SendAsync(
|
||||
string url,
|
||||
TimeSpan timeout,
|
||||
bool bodyRequired,
|
||||
CancellationToken cancellationToken)
|
||||
{
|
||||
var startedAt = Stopwatch.GetTimestamp();
|
||||
|
||||
@@ -81,10 +109,12 @@ public sealed class ZbHealthReportClient
|
||||
// Answered, but not with this contract — a reverse proxy error page, an HTML login
|
||||
// redirect, a wrong port. That is a registry/plumbing problem, not a sick app, so
|
||||
// it lands as Unreachable rather than Down.
|
||||
return Failed(startedAt, statusCode, $"response body is not canonical health JSON: {ex.Message}");
|
||||
return bodyRequired
|
||||
? Failed(startedAt, statusCode, $"response body is not canonical health JSON: {ex.Message}")
|
||||
: new HealthProbeResult(true, statusCode, null, Stopwatch.GetElapsedTime(startedAt), null);
|
||||
}
|
||||
|
||||
if (report is null)
|
||||
if (report is null && bodyRequired)
|
||||
return Failed(startedAt, statusCode, "response body deserialized to null");
|
||||
|
||||
return new HealthProbeResult(true, statusCode, report, Stopwatch.GetElapsedTime(startedAt), null);
|
||||
|
||||
@@ -1,4 +1,6 @@
|
||||
using ZB.MOM.WW.Configuration;
|
||||
using ZB.MOM.WW.Overview.Components;
|
||||
using ZB.MOM.WW.Overview.Polling;
|
||||
using ZB.MOM.WW.Overview.Registry;
|
||||
|
||||
var builder = WebApplication.CreateBuilder(args);
|
||||
@@ -15,9 +17,29 @@ ConfigPreflight.For(builder.Configuration)
|
||||
builder.Services.AddValidatedOptions<OverviewOptions, OverviewOptionsValidator>(
|
||||
builder.Configuration, OverviewOptions.SectionName);
|
||||
|
||||
builder.Services.AddOverviewPolling();
|
||||
|
||||
builder.Services.AddRazorComponents()
|
||||
.AddInteractiveServerComponents();
|
||||
|
||||
var app = builder.Build();
|
||||
|
||||
app.MapGet("/", () => "ZB.MOM.WW.Overview");
|
||||
app.UseStaticFiles();
|
||||
|
||||
// No UseAuthentication/UseAuthorization and no cascading auth state: the dashboard is anonymous and
|
||||
// read-only by requirement (design §2). It renders health data that every one of these apps already
|
||||
// serves unauthenticated, and it has no endpoint that mutates anything.
|
||||
//
|
||||
// UseAntiforgery IS required despite that. AddRazorComponents stamps antiforgery metadata onto every
|
||||
// component endpoint unconditionally, and the endpoint middleware hard-fails a request whose endpoint
|
||||
// carries that metadata with no middleware to honour it — so without this line every page returns
|
||||
// 500. The alternative, MapRazorComponents(...).DisableAntiforgery(), would also boot, but it turns
|
||||
// the first form anyone ever adds into a silent CSRF hole. This app has no forms, so the middleware
|
||||
// is inert; it is here so that stays true by default rather than by vigilance.
|
||||
app.UseAntiforgery();
|
||||
|
||||
app.MapRazorComponents<App>()
|
||||
.AddInteractiveServerRenderMode();
|
||||
|
||||
await app.RunAsync();
|
||||
|
||||
|
||||
@@ -0,0 +1,112 @@
|
||||
/* ═══════════════════════════════════════════════════════════════════════════
|
||||
ZB.MOM.WW.Overview — app-local styles.
|
||||
|
||||
Everything here is what the shared ZB.MOM.WW.Theme kit does NOT already
|
||||
provide. The kit owns the design tokens, the shell, and the generic
|
||||
vocabulary this file builds on: .chip-*, .panel, .panel-foot, .kv, .agg-*,
|
||||
.card-grid, .conn-pill, .rail-btn, .mono, .rise. None of those are
|
||||
redefined here — this file adds only the dashboard-specific anatomy from
|
||||
docs/mockups/overview-dashboard-mockup.html (page-head, app/group headers,
|
||||
the instance card, the check-detail list).
|
||||
═══════════════════════════════════════════════════════════════════════════ */
|
||||
|
||||
/* ── Page header ────────────────────────────────────────────────────────── */
|
||||
.page-head { display: flex; align-items: baseline; gap: 1rem; flex-wrap: wrap; margin: 0 0 1rem; }
|
||||
.page-head h1 { font-size: 1.25rem; font-weight: 600; letter-spacing: 0.01em; margin: 0; text-wrap: balance; }
|
||||
.page-head .spacer { flex: 1; }
|
||||
.page-meta { font-family: var(--mono); font-size: 0.78rem; color: var(--ink-soft); }
|
||||
|
||||
/* KPI variants the kit does not carry (it ships .alert / .caution only). */
|
||||
.agg-card.good .agg-value { color: var(--ok); }
|
||||
.agg-card.quiet .agg-value { color: var(--idle); }
|
||||
.kv .v.idle { color: var(--idle); }
|
||||
|
||||
/* ── Application sections ───────────────────────────────────────────────── */
|
||||
.app-section { margin-bottom: 1.5rem; }
|
||||
.app-head {
|
||||
display: flex; align-items: baseline; gap: 0.6rem; flex-wrap: wrap;
|
||||
padding-bottom: 0.4rem; border-bottom: 1px solid var(--rule-strong); margin-bottom: 0.75rem;
|
||||
}
|
||||
.app-head h2 { font-size: 1.02rem; font-weight: 600; margin: 0; }
|
||||
.app-head .spacer { flex: 1; }
|
||||
.app-meta { font-family: var(--mono); font-size: 0.76rem; color: var(--ink-faint); }
|
||||
|
||||
.group { margin-bottom: 0.9rem; }
|
||||
.group-head { display: flex; align-items: center; gap: 0.55rem; flex-wrap: wrap; margin-bottom: 0.5rem; }
|
||||
.group-name {
|
||||
font-size: 0.74rem; font-weight: 600; text-transform: uppercase;
|
||||
letter-spacing: 0.07em; color: var(--ink-soft);
|
||||
}
|
||||
.group-meta { font-family: var(--mono); font-size: 0.74rem; color: var(--ink-faint); }
|
||||
|
||||
/* ── Instance card ──────────────────────────────────────────────────────── */
|
||||
.inst {
|
||||
background: var(--card); border: 1px solid var(--rule); border-radius: 8px;
|
||||
overflow: hidden; display: flex; flex-direction: column;
|
||||
}
|
||||
|
||||
/* Status is carried by the top stripe. Unreachable additionally goes DASHED:
|
||||
colour alone cannot distinguish it from Down, and the two send an operator
|
||||
to completely different places (network vs. the application itself). */
|
||||
.inst.up { border-top: 3px solid var(--ok); }
|
||||
.inst.degraded { border-top: 3px solid var(--warn); }
|
||||
.inst.down { border-top: 3px solid var(--bad); }
|
||||
.inst.unreachable { border-top: 3px solid var(--bad); border-style: dashed; border-top-style: solid; }
|
||||
.inst.pending { border-top: 3px solid var(--rule-strong); }
|
||||
.inst.stale { border-top: 3px solid var(--idle); }
|
||||
.inst.stale .inst-body,
|
||||
.inst.stale .inst-head .inst-host { opacity: 0.55; }
|
||||
|
||||
.inst-head {
|
||||
display: flex; align-items: flex-start; justify-content: space-between;
|
||||
gap: 0.6rem; padding: 0.6rem 0.9rem 0.45rem;
|
||||
}
|
||||
.inst-name { font-weight: 600; font-size: 0.92rem; }
|
||||
.inst-host {
|
||||
display: block; font-family: var(--mono); font-size: 0.72rem;
|
||||
color: var(--ink-faint); margin-top: 0.15rem;
|
||||
}
|
||||
.inst-chips { display: flex; gap: 0.3rem; flex-wrap: wrap; justify-content: flex-end; }
|
||||
|
||||
.inst .panel-foot {
|
||||
margin-top: auto; display: flex; align-items: center;
|
||||
justify-content: space-between; gap: 0.6rem;
|
||||
}
|
||||
.foot-note { font-family: var(--mono); font-size: 0.72rem; color: var(--ink-faint); }
|
||||
|
||||
/* ── Check detail (native <details>, no JS) ─────────────────────────────── */
|
||||
details.checks { border-top: 1px solid var(--rule); }
|
||||
details.checks summary {
|
||||
list-style: none; cursor: pointer; padding: 0.45rem 0.9rem; font-size: 0.72rem;
|
||||
font-weight: 600; text-transform: uppercase; letter-spacing: 0.06em;
|
||||
color: var(--ink-faint); user-select: none;
|
||||
}
|
||||
details.checks summary::-webkit-details-marker { display: none; }
|
||||
details.checks summary::before { content: '\25B6'; font-size: 0.55rem; margin-right: 0.4rem; }
|
||||
details.checks[open] summary::before { content: '\25BC'; }
|
||||
details.checks summary:hover { color: var(--ink); }
|
||||
|
||||
.check-row { display: flex; align-items: baseline; gap: 0.5rem; padding: 0.28rem 0.9rem; font-size: 0.8rem; }
|
||||
.check-row:nth-child(even) { background: var(--zebra-bg); }
|
||||
.check-name { font-family: var(--mono); font-size: 0.78rem; min-width: 9.5rem; }
|
||||
.check-desc { color: var(--ink-soft); font-size: 0.78rem; }
|
||||
.check-dot { width: 8px; height: 8px; border-radius: 2px; flex: 0 0 8px; align-self: center; }
|
||||
.check-dot.ok { background: var(--ok); }
|
||||
.check-dot.warn { background: var(--warn); }
|
||||
.check-dot.bad { background: var(--bad); }
|
||||
.check-dot.idle { background: var(--idle); }
|
||||
|
||||
.check-data {
|
||||
margin: 0.35rem 0.9rem 0.6rem; padding: 0.4rem 0.6rem; background: var(--zebra-bg);
|
||||
border: 1px solid var(--rule); border-radius: 4px;
|
||||
font-family: var(--mono); font-size: 0.72rem; color: var(--ink-soft); line-height: 1.6;
|
||||
/* Scrolls inside its own box: an Akka address plus member counts is wider than a
|
||||
310px card, and letting it widen the card would break the whole grid. */
|
||||
overflow-x: auto; white-space: nowrap;
|
||||
}
|
||||
.check-data b { color: var(--accent-deep); font-weight: 500; }
|
||||
|
||||
.rail-note {
|
||||
padding: 0 0.6rem 0.6rem; font-family: var(--mono); font-size: 0.72rem;
|
||||
color: var(--ink-faint); line-height: 1.5;
|
||||
}
|
||||
+6
File diff suppressed because one or more lines are too long
+7
File diff suppressed because one or more lines are too long
Reference in New Issue
Block a user