feat(overview): self-observability, config and the container image
Closes Phase 3 of the overview-dashboard plan (tasks 3.7-3.9).
Self-observability
- OverviewMetrics: an observable gauge read from the published snapshot at scrape
time, so the metric and the page cannot disagree about what the dashboard
believes, plus a sweep-duration histogram. Instrument names follow the family
METRIC-CONVENTIONS spec (overview.instance.status / overview.poll.duration,
unit s) rather than the plan's literal zb_ names; Prometheus renders them as
overview_instance_status and overview_poll_duration_seconds. Labels are
application/node/group - "node", not "instance", because Prometheus stamps its
own instance label on every scraped series and a colliding metric label is
silently renamed exported_instance. Instances never probed emit nothing: the
enum's zero ordinal is Up, so a placeholder would publish the single most
reassuring answer about a node nobody has heard from.
- Two readiness checks make the dashboard's own /health/ready mean something:
registry-loaded (reports the counts it bound) and last-poll-cycle-completed,
which catches the failure where Kestrel is fine, the page still renders, and
every card on it is quietly frozen. The heartbeat is stamped by the sweep, not
derived from the snapshot's timestamp - the poller publishes the registry
before it probes anything, so a stuck poller would otherwise look healthy.
No cycle yet is Degraded inside the grace window and Unhealthy past it, so a
restart does not fail readiness for its first seconds and a poller that never
started cannot hide behind "still starting".
Configuration
- appsettings.json ships tuning defaults with an EMPTY Applications list, so a
deployment that supplies no registry fails at ConfigPreflight naming the key
instead of booting into a dashboard of nothing.
- Three registries, because reachability genuinely differs: Development is the
host-reachable subset (4 instances), Docker is the full 16-instance fleet by
container DNS name. The rigs publish almost no per-node HTTP port to the host
- OtOpcUa publishes none (only Traefik :9200, load-balanced across the central
pair, so it cannot identify a node) and ScadaBridge keeps site :8084
container-internal. Ports were established by probing the running rigs, not
read off the compose files, because the two disagree: OtOpcUa site nodes serve
health on :8080, not the :9000 their compose anchor implies.
- Real Serilog sinks, not the family's empty "Serilog": {}. AddZbSerilog drives
sinks entirely from configuration and adds none of its own, so the empty
section produces an app that logs NOWHERE - verified against the running
HistorianGateway container, whose log stream is completely empty for exactly
this reason. The HttpClient level override matters too: 16 instances on a 10s
cadence emit several Information lines a second of pure probe noise.
Container
- HistorianGateway-pattern runtime-only image; the compose stack joins the three
rig networks as external, so bringing the dashboard up or down cannot disturb
a rig. The Dockerfile clears ASPNETCORE_HTTP_PORTS as well as ASPNETCORE_URLS:
on .NET 8+ the base image declares its binding via the former, and clearing
only the latter leaves Kestrel logging "Overriding address(es)" every boot.
No healthcheck - aspnet:10.0 has neither curl nor wget, and adding a
--healthcheck entry point to production code to satisfy Compose is not a
trade worth making.
Tests
- BootTests boot the real Program.cs: every endpoint anonymous, the page
rendering from cache while nothing it watches is reachable, the Meters
allowlist proven by an end-to-end /metrics scrape, and both refuse-to-boot
paths (missing registry, stale window inside the poll interval).
The registry is supplied via environment variables, not
ConfigureAppConfiguration, because ConfigPreflight runs before Build() - when
the factory's callbacks are applied - and would not see them.
170 tests, 0 warnings. Verified live against the running rigs: 16 cards, 16
gauge series, page and gauge agreeing exactly.
This commit is contained in:
@@ -0,0 +1,193 @@
|
||||
using System.Net;
|
||||
using Microsoft.AspNetCore.Mvc.Testing;
|
||||
using Microsoft.Extensions.DependencyInjection;
|
||||
using Microsoft.Extensions.Hosting;
|
||||
using ZB.MOM.WW.Overview.Observability;
|
||||
using ZB.MOM.WW.Overview.Polling;
|
||||
|
||||
namespace ZB.MOM.WW.Overview.Tests;
|
||||
|
||||
/// <summary>
|
||||
/// Boots the REAL <c>Program.cs</c> and asks the questions no in-process test can: does every
|
||||
/// endpoint answer without a login, and does a bad registry stop the host rather than produce a
|
||||
/// dashboard full of nothing.
|
||||
/// </summary>
|
||||
/// <remarks>
|
||||
/// The registry is supplied through environment variables rather than
|
||||
/// <c>ConfigureAppConfiguration</c>. <c>ConfigPreflight</c> runs against
|
||||
/// <c>builder.Configuration</c> in the top-level statements, before <c>Build()</c> — which is when
|
||||
/// the factory's configuration callbacks are applied — so a callback-supplied registry would arrive
|
||||
/// too late for the very check these tests are about. Environment variables are read by
|
||||
/// <c>CreateBuilder</c> itself, so they are in place from the first line.
|
||||
///
|
||||
/// <para>
|
||||
/// Environment variables are process-global, so every test here lives in one class: xunit runs a
|
||||
/// class's tests sequentially, which is what keeps two hosts from fighting over the same keys.
|
||||
/// </para>
|
||||
/// </remarks>
|
||||
public sealed class BootTests : IDisposable
|
||||
{
|
||||
private readonly List<string> _keys = [];
|
||||
|
||||
private void Set(string key, string? value)
|
||||
{
|
||||
_keys.Add(key);
|
||||
Environment.SetEnvironmentVariable(key, value);
|
||||
}
|
||||
|
||||
/// <summary>Restores the environment so a later-running test class sees a clean process.</summary>
|
||||
public void Dispose()
|
||||
{
|
||||
foreach (var key in _keys)
|
||||
Environment.SetEnvironmentVariable(key, null);
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// A registry with one instance pointing at a port nothing listens on. The dashboard must boot
|
||||
/// and serve regardless of whether anything it watches is up — that is the entire premise of
|
||||
/// rendering from cache.
|
||||
/// </summary>
|
||||
private void SetValidRegistry()
|
||||
{
|
||||
Set("ASPNETCORE_ENVIRONMENT", "Testing");
|
||||
Set("Overview__PollIntervalSeconds", "10");
|
||||
Set("Overview__TimeoutSeconds", "1");
|
||||
Set("Overview__StaleAfterSeconds", "45");
|
||||
Set("Overview__Applications__0__Name", "TestApp");
|
||||
Set("Overview__Applications__0__ManagementLabel", "UI");
|
||||
Set("Overview__Applications__0__Instances__0__Name", "node-a");
|
||||
Set("Overview__Applications__0__Instances__0__BaseUrl", "http://127.0.0.1:1");
|
||||
Set("Overview__Applications__0__Instances__0__HasActiveRole", "false");
|
||||
}
|
||||
|
||||
private static WebApplicationFactory<Program> Factory() => new();
|
||||
|
||||
[Fact]
|
||||
public async Task Page_IsServedAnonymously()
|
||||
{
|
||||
SetValidRegistry();
|
||||
using var factory = Factory();
|
||||
|
||||
// AllowAutoRedirect off so an auth redirect would surface as a 302 here rather than being
|
||||
// silently followed to a login page that then returns 200.
|
||||
using var client = factory.CreateClient(new WebApplicationFactoryClientOptions
|
||||
{
|
||||
AllowAutoRedirect = false,
|
||||
});
|
||||
|
||||
using var response = await client.GetAsync("/");
|
||||
|
||||
Assert.Equal(HttpStatusCode.OK, response.StatusCode);
|
||||
Assert.Contains("SCADA Overview", await response.Content.ReadAsStringAsync(), StringComparison.Ordinal);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task Page_RendersEvenThoughNothingItWatchesIsReachable()
|
||||
{
|
||||
// Cache-first, stated as a test: the registry above points at a dead port, and the page
|
||||
// must still paint its full layout instead of waiting on a sweep.
|
||||
SetValidRegistry();
|
||||
using var factory = Factory();
|
||||
using var client = factory.CreateClient();
|
||||
|
||||
var html = await client.GetStringAsync("/");
|
||||
|
||||
Assert.Contains("data-testid=\"instance-card\"", html, StringComparison.Ordinal);
|
||||
Assert.Contains("node-a", html, StringComparison.Ordinal);
|
||||
}
|
||||
|
||||
[Theory]
|
||||
[InlineData("/health/ready")]
|
||||
[InlineData("/health/active")]
|
||||
[InlineData("/healthz")]
|
||||
[InlineData("/metrics")]
|
||||
public async Task HealthAndMetrics_AreServedAnonymously(string path)
|
||||
{
|
||||
SetValidRegistry();
|
||||
using var factory = Factory();
|
||||
using var client = factory.CreateClient(new WebApplicationFactoryClientOptions
|
||||
{
|
||||
AllowAutoRedirect = false,
|
||||
});
|
||||
|
||||
using var response = await client.GetAsync(path);
|
||||
|
||||
Assert.Equal(HttpStatusCode.OK, response.StatusCode);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task Ready_ReportsTheDashboardsOwnChecks()
|
||||
{
|
||||
SetValidRegistry();
|
||||
using var factory = Factory();
|
||||
using var client = factory.CreateClient();
|
||||
|
||||
var body = await client.GetStringAsync("/health/ready");
|
||||
|
||||
Assert.Contains(ObservabilityServiceCollectionExtensions.RegistryLoadedCheckName, body, StringComparison.Ordinal);
|
||||
Assert.Contains(ObservabilityServiceCollectionExtensions.PollCycleCheckName, body, StringComparison.Ordinal);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task Metrics_ExportsTheAppsOwnMeter()
|
||||
{
|
||||
// The Meters allowlist in AddZbTelemetry is silent when it is wrong: a missing name drops
|
||||
// the app's whole instrument surface from /metrics with no warning anywhere. Only an
|
||||
// end-to-end scrape catches that.
|
||||
SetValidRegistry();
|
||||
using var factory = Factory();
|
||||
using var client = factory.CreateClient();
|
||||
|
||||
// Force a sweep so the observable gauge has a probed instance to report. The instance is
|
||||
// unreachable, which is a perfectly good status to publish.
|
||||
var poller = factory.Services.GetServices<IHostedService>().OfType<OverviewPollerService>().Single();
|
||||
await poller.SweepAsync();
|
||||
|
||||
var body = await client.GetStringAsync("/metrics");
|
||||
|
||||
Assert.Contains("overview_instance_status", body, StringComparison.Ordinal);
|
||||
Assert.Contains("node=\"node-a\"", body, StringComparison.Ordinal);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task MissingRegistry_FailsToBootAndNamesTheKey()
|
||||
{
|
||||
// appsettings.json ships an EMPTY Applications list precisely so this is the outcome when a
|
||||
// deployment forgets its registry — an empty dashboard that boots would look like a healthy
|
||||
// fleet of nothing.
|
||||
Set("ASPNETCORE_ENVIRONMENT", "Testing");
|
||||
|
||||
using var factory = Factory();
|
||||
|
||||
var error = await Record.ExceptionAsync(() => factory.CreateClient().GetAsync("/"));
|
||||
|
||||
Assert.NotNull(error);
|
||||
Assert.Contains("Overview:Applications:0:Name", Flatten(error), StringComparison.Ordinal);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task StaleWindowInsideThePollInterval_FailsToBootWithTheValidatorMessage()
|
||||
{
|
||||
// Past preflight and into the validator: a staleness window shorter than the poll interval
|
||||
// would mark every instance Stale between two perfectly healthy polls.
|
||||
SetValidRegistry();
|
||||
Set("Overview__StaleAfterSeconds", "5");
|
||||
|
||||
using var factory = Factory();
|
||||
|
||||
var error = await Record.ExceptionAsync(() => factory.CreateClient().GetAsync("/"));
|
||||
|
||||
Assert.NotNull(error);
|
||||
Assert.Contains("StaleAfterSeconds", Flatten(error), StringComparison.Ordinal);
|
||||
}
|
||||
|
||||
private static string Flatten(Exception exception)
|
||||
{
|
||||
var text = new System.Text.StringBuilder();
|
||||
|
||||
for (var current = exception; current is not null; current = current.InnerException)
|
||||
text.AppendLine(current.Message);
|
||||
|
||||
return text.ToString();
|
||||
}
|
||||
}
|
||||
Reference in New Issue
Block a user