92997b40dc
Closes Phase 3 of the overview-dashboard plan (tasks 3.7-3.9).
Self-observability
- OverviewMetrics: an observable gauge read from the published snapshot at scrape
time, so the metric and the page cannot disagree about what the dashboard
believes, plus a sweep-duration histogram. Instrument names follow the family
METRIC-CONVENTIONS spec (overview.instance.status / overview.poll.duration,
unit s) rather than the plan's literal zb_ names; Prometheus renders them as
overview_instance_status and overview_poll_duration_seconds. Labels are
application/node/group - "node", not "instance", because Prometheus stamps its
own instance label on every scraped series and a colliding metric label is
silently renamed exported_instance. Instances never probed emit nothing: the
enum's zero ordinal is Up, so a placeholder would publish the single most
reassuring answer about a node nobody has heard from.
- Two readiness checks make the dashboard's own /health/ready mean something:
registry-loaded (reports the counts it bound) and last-poll-cycle-completed,
which catches the failure where Kestrel is fine, the page still renders, and
every card on it is quietly frozen. The heartbeat is stamped by the sweep, not
derived from the snapshot's timestamp - the poller publishes the registry
before it probes anything, so a stuck poller would otherwise look healthy.
No cycle yet is Degraded inside the grace window and Unhealthy past it, so a
restart does not fail readiness for its first seconds and a poller that never
started cannot hide behind "still starting".
Configuration
- appsettings.json ships tuning defaults with an EMPTY Applications list, so a
deployment that supplies no registry fails at ConfigPreflight naming the key
instead of booting into a dashboard of nothing.
- Three registries, because reachability genuinely differs: Development is the
host-reachable subset (4 instances), Docker is the full 16-instance fleet by
container DNS name. The rigs publish almost no per-node HTTP port to the host
- OtOpcUa publishes none (only Traefik :9200, load-balanced across the central
pair, so it cannot identify a node) and ScadaBridge keeps site :8084
container-internal. Ports were established by probing the running rigs, not
read off the compose files, because the two disagree: OtOpcUa site nodes serve
health on :8080, not the :9000 their compose anchor implies.
- Real Serilog sinks, not the family's empty "Serilog": {}. AddZbSerilog drives
sinks entirely from configuration and adds none of its own, so the empty
section produces an app that logs NOWHERE - verified against the running
HistorianGateway container, whose log stream is completely empty for exactly
this reason. The HttpClient level override matters too: 16 instances on a 10s
cadence emit several Information lines a second of pure probe noise.
Container
- HistorianGateway-pattern runtime-only image; the compose stack joins the three
rig networks as external, so bringing the dashboard up or down cannot disturb
a rig. The Dockerfile clears ASPNETCORE_HTTP_PORTS as well as ASPNETCORE_URLS:
on .NET 8+ the base image declares its binding via the former, and clearing
only the latter leaves Kestrel logging "Overriding address(es)" every boot.
No healthcheck - aspnet:10.0 has neither curl nor wget, and adding a
--healthcheck entry point to production code to satisfy Compose is not a
trade worth making.
Tests
- BootTests boot the real Program.cs: every endpoint anonymous, the page
rendering from cache while nothing it watches is reachable, the Meters
allowlist proven by an end-to-end /metrics scrape, and both refuse-to-boot
paths (missing registry, stale window inside the poll interval).
The registry is supplied via environment variables, not
ConfigureAppConfiguration, because ConfigPreflight runs before Build() - when
the factory's callbacks are applied - and would not see them.
170 tests, 0 warnings. Verified live against the running rigs: 16 cards, 16
gauge series, page and gauge agreeing exactly.
172 lines
5.6 KiB
C#
172 lines
5.6 KiB
C#
using System.Diagnostics.Metrics;
|
|
using System.Net;
|
|
using System.Text;
|
|
using System.Text.Json;
|
|
using ZB.MOM.WW.Overview.Polling;
|
|
|
|
namespace ZB.MOM.WW.Overview.Tests;
|
|
|
|
/// <summary>A meter factory with no listener attached, for tests that only need the poller to run.</summary>
|
|
internal sealed class DummyMeterFactory : IMeterFactory
|
|
{
|
|
private readonly List<Meter> _meters = [];
|
|
|
|
public Meter Create(MeterOptions options)
|
|
{
|
|
ArgumentNullException.ThrowIfNull(options);
|
|
|
|
// Stamp ourselves as the scope, exactly as the real factory does. Meter names are global,
|
|
// so a MeterListener that matched on name alone would pick up meters created by tests
|
|
// running in parallel; the scope is what makes one test's instruments identifiable.
|
|
options.Scope = this;
|
|
|
|
var meter = new Meter(options);
|
|
_meters.Add(meter);
|
|
return meter;
|
|
}
|
|
|
|
public void Dispose()
|
|
{
|
|
foreach (var meter in _meters)
|
|
meter.Dispose();
|
|
|
|
_meters.Clear();
|
|
}
|
|
}
|
|
|
|
/// <summary>A clock the test drives by hand.</summary>
|
|
/// <remarks>
|
|
/// Only <see cref="GetUtcNow"/> is overridden: the poller's sweep — the part with rules worth
|
|
/// pinning — is exercised directly, so no test here needs a fake timer.
|
|
/// </remarks>
|
|
internal sealed class FakeTimeProvider(DateTimeOffset start) : TimeProvider
|
|
{
|
|
private DateTimeOffset _now = start;
|
|
|
|
public override DateTimeOffset GetUtcNow() => _now;
|
|
|
|
public void Advance(TimeSpan by) => _now += by;
|
|
}
|
|
|
|
/// <summary>Serves canned responses per absolute URL and counts what was requested.</summary>
|
|
internal sealed class RoutingHandler : HttpMessageHandler
|
|
{
|
|
private readonly Dictionary<string, Func<HttpResponseMessage>> _routes = new(StringComparer.OrdinalIgnoreCase);
|
|
|
|
public List<string> Requests { get; } = [];
|
|
|
|
public RoutingHandler Json(string url, HttpStatusCode status, string body)
|
|
{
|
|
_routes[url] = () => new HttpResponseMessage(status)
|
|
{
|
|
Content = new StringContent(body, Encoding.UTF8, "application/json"),
|
|
};
|
|
return this;
|
|
}
|
|
|
|
/// <summary>
|
|
/// A status-code-only response with an EMPTY body — what the active tier actually returns when
|
|
/// it is a bare gate endpoint rather than a full health report.
|
|
/// </summary>
|
|
public RoutingHandler Status(string url, HttpStatusCode status)
|
|
{
|
|
_routes[url] = () => new HttpResponseMessage(status)
|
|
{
|
|
Content = new StringContent(string.Empty, Encoding.UTF8, "application/json"),
|
|
};
|
|
return this;
|
|
}
|
|
|
|
public RoutingHandler Fails(string url, string message)
|
|
{
|
|
_routes[url] = () => throw new HttpRequestException(message);
|
|
return this;
|
|
}
|
|
|
|
/// <summary>Lets a wrapping handler delegate here without re-entering the protected member.</summary>
|
|
public Task<HttpResponseMessage> HandleAsync(HttpRequestMessage request) =>
|
|
SendAsync(request, CancellationToken.None);
|
|
|
|
protected override Task<HttpResponseMessage> SendAsync(
|
|
HttpRequestMessage request, CancellationToken cancellationToken)
|
|
{
|
|
var url = request.RequestUri!.ToString();
|
|
|
|
lock (Requests)
|
|
Requests.Add(url);
|
|
|
|
if (!_routes.TryGetValue(url, out var factory))
|
|
return Task.FromException<HttpResponseMessage>(new HttpRequestException($"no route for {url}"));
|
|
|
|
try
|
|
{
|
|
return Task.FromResult(factory());
|
|
}
|
|
catch (HttpRequestException ex)
|
|
{
|
|
return Task.FromException<HttpResponseMessage>(ex);
|
|
}
|
|
}
|
|
}
|
|
|
|
/// <summary>Hands every named client the same test handler.</summary>
|
|
internal sealed class StubHttpClientFactory(HttpMessageHandler handler) : IHttpClientFactory
|
|
{
|
|
public HttpClient CreateClient(string name) =>
|
|
new(handler, disposeHandler: false) { Timeout = Timeout.InfiniteTimeSpan };
|
|
}
|
|
|
|
/// <summary>Builds canonical health bodies so tests state intent, not JSON.</summary>
|
|
internal static class HealthBody
|
|
{
|
|
public static string Ready(string status, params (string Name, string Status, string? Leader)[] entries)
|
|
{
|
|
var payload = new
|
|
{
|
|
status,
|
|
totalDurationMs = 1.5,
|
|
entries = entries.ToDictionary(
|
|
e => e.Name,
|
|
e => (object)new
|
|
{
|
|
status = e.Status,
|
|
description = $"{e.Name} says {e.Status}",
|
|
durationMs = 0.5,
|
|
data = e.Leader is null ? null : new Dictionary<string, object> { ["leader"] = e.Leader },
|
|
},
|
|
StringComparer.Ordinal),
|
|
};
|
|
|
|
return JsonSerializer.Serialize(payload);
|
|
}
|
|
}
|
|
|
|
/// <summary>Builds <see cref="InstanceSnapshot"/> values for the pure-logic tests.</summary>
|
|
internal static class Snapshots
|
|
{
|
|
public static InstanceSnapshot Instance(
|
|
string name,
|
|
string? group = null,
|
|
string baseUrl = "http://host:8080",
|
|
bool hasActiveRole = true,
|
|
InstanceStatus? status = InstanceStatus.Up,
|
|
string? leader = null) =>
|
|
new()
|
|
{
|
|
Application = "App",
|
|
Instance = name,
|
|
Group = group,
|
|
BaseUrl = baseUrl,
|
|
HasActiveRole = hasActiveRole,
|
|
Status = status,
|
|
Report = leader is null ? null : ReportWithLeader(leader),
|
|
StaleAfter = TimeSpan.FromSeconds(45),
|
|
};
|
|
|
|
private static ZbHealthReport ReportWithLeader(string leader)
|
|
{
|
|
var body = HealthBody.Ready("Healthy", ("akka-cluster", "Healthy", leader));
|
|
return JsonSerializer.Deserialize<ZbHealthReport>(body)!;
|
|
}
|
|
}
|