diff --git a/ZB.MOM.WW.Overview/.gitignore b/ZB.MOM.WW.Overview/.gitignore new file mode 100644 index 0000000..eace692 --- /dev/null +++ b/ZB.MOM.WW.Overview/.gitignore @@ -0,0 +1,5 @@ +# Host-side publish output copied into the runtime-only Docker image. +docker/publish/ + +# Serilog rolling file sink, written relative to the content root. +logs/ diff --git a/ZB.MOM.WW.Overview/docker/.dockerignore b/ZB.MOM.WW.Overview/docker/.dockerignore new file mode 100644 index 0000000..f9aff11 --- /dev/null +++ b/ZB.MOM.WW.Overview/docker/.dockerignore @@ -0,0 +1,2 @@ +**/*.md +**/.DS_Store diff --git a/ZB.MOM.WW.Overview/docker/Dockerfile b/ZB.MOM.WW.Overview/docker/Dockerfile new file mode 100644 index 0000000..be08112 --- /dev/null +++ b/ZB.MOM.WW.Overview/docker/Dockerfile @@ -0,0 +1,29 @@ +# Runtime-only image for the ZB.MOM.WW.Overview dashboard. +# +# The app is published FRAMEWORK-DEPENDENT on the host (which holds the Gitea NuGet feed +# credentials in ~/.nuget) into ./publish, then copied in here. Docker performs NO restore and +# NO build, so the authenticated Gitea feed is never needed at image-build time. Regenerate +# ./publish with: +# dotnet publish src/ZB.MOM.WW.Overview/ZB.MOM.WW.Overview.csproj \ +# -c Release -o docker/publish -p:UseAppHost=false +# +# The published IL is architecture-neutral, so the multi-arch aspnet:10.0 base runs as-is on +# both Apple Silicon and amd64 hosts. +FROM mcr.microsoft.com/dotnet/aspnet:10.0 AS runtime +WORKDIR /app + +# The base image declares its own binding — as ASPNETCORE_HTTP_PORTS=8080 on .NET 8+, NOT the +# ASPNETCORE_URLS everyone reaches for first (clearing only URLS leaves the warning in place, which +# is a good way to spend an afternoon). Left set, it collides with the Kestrel endpoint section in +# appsettings.Docker.json: Kestrel wins, but logs "Overriding address(es) 'http://*:8080'" on every +# boot — the family's URLs-override trap arriving by default rather than by mistake. Clearing both +# at the layer that introduced them leaves the Kestrel section as the single authority. +ENV ASPNETCORE_URLS= \ + ASPNETCORE_HTTP_PORTS= + +COPY publish/ ./ + +# Dashboard + /health/* + /metrics, all HTTP/1.1 and all anonymous. +EXPOSE 5320 + +ENTRYPOINT ["dotnet", "ZB.MOM.WW.Overview.dll"] diff --git a/ZB.MOM.WW.Overview/docker/README.md b/ZB.MOM.WW.Overview/docker/README.md new file mode 100644 index 0000000..5d6fbee --- /dev/null +++ b/ZB.MOM.WW.Overview/docker/README.md @@ -0,0 +1,81 @@ +# `docker/` — containerised overview dashboard + +Runtime-only image, the HistorianGateway pattern: publish on the host (which has the Gitea feed +credentials), copy the output into `mcr.microsoft.com/dotnet/aspnet:10.0`. Docker never restores. + +```bash +# from ZB.MOM.WW.Overview/ +dotnet publish src/ZB.MOM.WW.Overview/ZB.MOM.WW.Overview.csproj \ + -c Release -o docker/publish -p:UseAppHost=false + +cd docker && docker compose up -d --build +open http://localhost:5320 +``` + +`docker/publish/` is build output and is gitignored. + +## Why the container, and not just `dotnet run` + +The dev rigs publish only some node ports to the host: + +| Fleet | Health endpoint | Reachable from the Mac host? | +|---|---|---| +| ScadaBridge central-a/b | container `:5000` | yes — `localhost:9001` / `:9002` | +| ScadaBridge site-\* (6) | container `:8084` | **no** — not published | +| OtOpcUa central-1/2 | container `:9000` | **no** — only Traefik `:9200`, load-balanced across the pair | +| OtOpcUa site-\* (4) | container `:8080` | **no** — not published | +| HistorianGateway | container `:5220` | yes — `localhost:5220` | +| MxAccessGateway | windev `:5130` | yes, with the VPN up | + +So a host-run dashboard can only see part of the fleet. This stack joins the three rig networks +(`otopcua-dev_default`, `scadabridge-net`, `zb-historiangw_default`, all declared `external`) and +addresses every node by container DNS name, which is how it sees all of them. + +Ports above were established by probing the running rigs on 2026-07-24, **not** read off the +compose files — the two disagree. In particular OtOpcUa's site nodes serve health on `:8080`, not +the `:9000` their compose anchor implies, because their explicit Kestrel listeners for LocalDb sync +re-bind the primary HTTP port. + +Start the rigs before this stack. The networks are `external`, so bringing the dashboard up or +down never disturbs a running rig. + +## Registry shape + +Bound from the `Overview` section; `appsettings.Docker.json` is the worked example. + +```jsonc +"Overview": { + "PollIntervalSeconds": 10, // sweep cadence; per-app and per-instance overrides allowed + "TimeoutSeconds": 3, // per-request timeout + "StaleAfterSeconds": 45, // must exceed the poll interval, or every card reads Stale + "Applications": [ + { + "Name": "ScadaBridge", + "ManagementLabel": "CentralUI", // label on each instance's management link + "Instances": [ + { + "Name": "central-a", + "Group": "central", // one Akka cluster = one group; leader + split-brain + // detection are scoped to it. Omit for standalone apps. + "BaseUrl": "http://host:5000", // /health/ready and /health/active are probed under this + "ManagementUrl": "http://localhost:9001/", // followed by the OPERATOR'S browser, so it + // must be host-reachable, not container DNS + "HasActiveRole": true // false for single-instance gateways: no Active badge, + // and /health/active is never probed + } + ] + } + ] +} +``` + +`appsettings.json` ships the tuning defaults with an **empty** `Applications` list, so a deployment +that supplies no registry fails at startup naming the missing key rather than booting into an empty +dashboard. + +## Configuration traps + +- **Never set `ASPNETCORE_URLS` alongside `Kestrel__Endpoints__*`.** Declaring both puts Kestrel + into explicit-endpoints mode and one of them is silently discarded. +- No `env_file`, no secrets, no login. The dashboard reads anonymous health endpoints and holds no + credentials. That is a requirement of the design, not an omission. diff --git a/ZB.MOM.WW.Overview/docker/docker-compose.yml b/ZB.MOM.WW.Overview/docker/docker-compose.yml new file mode 100644 index 0000000..755307b --- /dev/null +++ b/ZB.MOM.WW.Overview/docker/docker-compose.yml @@ -0,0 +1,54 @@ +# Local Docker deployment of the SCADA family overview dashboard. +# +# Why this exists rather than just `dotnet run`: the dev rigs publish only some node ports to the +# host. OtOpcUa publishes no per-node HTTP port at all (only Traefik's load-balanced :9200, which +# round-robins the central pair and so cannot identify a node), and ScadaBridge's site nodes keep +# their health port container-internal. A dashboard on the host can therefore see a fraction of the +# fleet; one joined to the rig networks sees all of it, addressing each node by container DNS name. +# +# Registry: appsettings.Docker.json, baked into the image (ASPNETCORE_ENVIRONMENT=Docker below). +# +# No env_file and no secrets: the dashboard reads anonymous /health endpoints, holds no +# credentials, and serves no login. That is a requirement, not an oversight. +# +# Explicit project name so this stack is isolated from the other compose projects on the host. +name: zb-overview + +services: + overview: + build: + context: . + dockerfile: Dockerfile + image: zb-overview:local + container_name: zb-overview + restart: unless-stopped + ports: + - "5320:5320" # dashboard + /health/* + /metrics + environment: + ASPNETCORE_ENVIRONMENT: Docker + + # The endpoint lives in appsettings.Docker.json — ONE authority for the binding. Do NOT add + # ASPNETCORE_URLS here; the Dockerfile clears the base image's default for that reason. + + # No container healthcheck on purpose. aspnet:10.0 ships neither curl nor wget, and adding a + # `--healthcheck` mode to the app to work around that would put a second entry point into + # production code purely to satisfy Compose. The app's real health surface is HTTP and already + # honest — `curl localhost:5320/health/ready` from the host, or scrape /metrics. + + networks: + - otopcua + - scadabridge + - historiangw + +# All three are created by the rigs themselves; this stack joins them and never defines them, so +# bringing the dashboard up or down cannot disturb a running rig. Start the rigs first. +networks: + otopcua: + name: otopcua-dev_default + external: true + scadabridge: + name: scadabridge-net + external: true + historiangw: + name: zb-historiangw_default + external: true diff --git a/ZB.MOM.WW.Overview/src/ZB.MOM.WW.Overview/Observability/ObservabilityServiceCollectionExtensions.cs b/ZB.MOM.WW.Overview/src/ZB.MOM.WW.Overview/Observability/ObservabilityServiceCollectionExtensions.cs new file mode 100644 index 0000000..37fdb57 --- /dev/null +++ b/ZB.MOM.WW.Overview/src/ZB.MOM.WW.Overview/Observability/ObservabilityServiceCollectionExtensions.cs @@ -0,0 +1,37 @@ +using ZB.MOM.WW.Health; + +namespace ZB.MOM.WW.Overview.Observability; + +/// Registers the dashboard's own health checks. +public static class ObservabilityServiceCollectionExtensions +{ + /// Check name for the registry-loaded probe. + public const string RegistryLoadedCheckName = "registry-loaded"; + + /// Check name for the poll-cycle liveness probe. + public const string PollCycleCheckName = "last-poll-cycle-completed"; + + /// + /// Adds the two readiness checks that make the dashboard's own /health/ready mean + /// something: that it bound a registry, and that it is still polling it. + /// + /// The service collection. + /// The same collection, for chaining. + /// + /// Both are tagged only. The dashboard is a single stateless + /// process with no active/standby concept, so it registers nothing on the active tier — its own + /// /health/active is deliberately an empty, always-200 report. + /// + public static IServiceCollection AddOverviewHealthChecks(this IServiceCollection services) + { + ArgumentNullException.ThrowIfNull(services); + + services.AddHealthChecks() + .AddTypeActivatedCheck( + RegistryLoadedCheckName, failureStatus: null, tags: [ZbHealthTags.Ready]) + .AddTypeActivatedCheck( + PollCycleCheckName, failureStatus: null, tags: [ZbHealthTags.Ready]); + + return services; + } +} diff --git a/ZB.MOM.WW.Overview/src/ZB.MOM.WW.Overview/Observability/OverviewMetrics.cs b/ZB.MOM.WW.Overview/src/ZB.MOM.WW.Overview/Observability/OverviewMetrics.cs new file mode 100644 index 0000000..dfc67e3 --- /dev/null +++ b/ZB.MOM.WW.Overview/src/ZB.MOM.WW.Overview/Observability/OverviewMetrics.cs @@ -0,0 +1,90 @@ +using System.Diagnostics.Metrics; +using ZB.MOM.WW.Overview.Polling; + +namespace ZB.MOM.WW.Overview.Observability; + +/// +/// The dashboard's own Meter: what it currently believes about every registered instance, and how +/// long its sweeps take. +/// +/// +/// +/// The instance gauge is observable — it is read from the published snapshot at scrape +/// time rather than written on every poll. The snapshot is already the single authority for what +/// the dashboard believes, so deriving the gauge from it makes a disagreement between the page and +/// the metric impossible by construction. +/// +/// +/// The instrument names follow the family convention in +/// components/observability/spec/METRIC-CONVENTIONS.md §2 — lower-case, dot-separated +/// <app>.<subsystem>.<event>, durations in seconds — so the Prometheus +/// exporter renders them as overview_instance_status and +/// overview_poll_duration_seconds. +/// +/// +public sealed class OverviewMetrics : IDisposable +{ + /// The app's single Meter name, per the family one-meter-per-app convention. + public const string MeterName = "ZB.MOM.WW.Overview"; + + /// Per-instance status gauge; the value is the ordinal. + public const string InstanceStatusInstrument = "overview.instance.status"; + + /// Sweep-duration histogram, in seconds. + public const string PollDurationInstrument = "overview.poll.duration"; + + private readonly Meter _meter; + private readonly Histogram _pollDuration; + private readonly OverviewSnapshotStore _store; + + /// Initializes the meter and its instruments. + /// Factory that scopes the meter's lifetime to the container. + /// The snapshot the observable gauge reads at scrape time. + public OverviewMetrics(IMeterFactory meterFactory, OverviewSnapshotStore store) + { + ArgumentNullException.ThrowIfNull(meterFactory); + + _store = store ?? throw new ArgumentNullException(nameof(store)); + _meter = meterFactory.Create(MeterName); + + _meter.CreateObservableGauge( + InstanceStatusInstrument, + ObserveInstanceStatus, + unit: "1", + description: "Derived status of each registered instance (0=Up, 1=Degraded, 2=Down, 3=Unreachable)."); + + _pollDuration = _meter.CreateHistogram( + PollDurationInstrument, + unit: "s", + description: "Wall-clock duration of one poll sweep across every due instance."); + } + + /// Records the duration of one completed sweep. + /// Wall-clock time the sweep took. + public void RecordSweep(TimeSpan duration) => _pollDuration.Record(duration.TotalSeconds); + + /// + public void Dispose() => _meter.Dispose(); + + private IEnumerable> ObserveInstanceStatus() + { + foreach (var instance in _store.Current.AllInstances) + { + // Instances that have never been probed are skipped rather than reported as anything. + // Emitting a value for them would publish a status the dashboard does not yet hold — + // and the enum's zero ordinal is Up, so the fabrication would be the reassuring one. + if (instance.Status is not { } status) + continue; + + yield return new Measurement( + (int)status, + new KeyValuePair("application", instance.Application), + + // Deliberately NOT named "instance": Prometheus attaches its own `instance` label to + // every scraped series, and a colliding metric label is silently renamed to + // `exported_instance`, which no dashboard query would be written against. + new KeyValuePair("node", instance.Instance), + new KeyValuePair("group", instance.Group ?? string.Empty)); + } + } +} diff --git a/ZB.MOM.WW.Overview/src/ZB.MOM.WW.Overview/Observability/PollCycleHealthCheck.cs b/ZB.MOM.WW.Overview/src/ZB.MOM.WW.Overview/Observability/PollCycleHealthCheck.cs new file mode 100644 index 0000000..4aeb8c9 --- /dev/null +++ b/ZB.MOM.WW.Overview/src/ZB.MOM.WW.Overview/Observability/PollCycleHealthCheck.cs @@ -0,0 +1,110 @@ +using Microsoft.Extensions.Diagnostics.HealthChecks; +using Microsoft.Extensions.Options; +using ZB.MOM.WW.Overview.Registry; + +namespace ZB.MOM.WW.Overview.Observability; + +/// +/// Fails when the poller has stopped sweeping — the one failure mode that would otherwise leave the +/// dashboard looking perfectly healthy while every card it serves quietly went stale. +/// +/// +/// This is what makes the dashboard's own /health/ready honest. Without it the process +/// answers 200 for as long as Kestrel is up, which says nothing about whether the data behind the +/// page is still being refreshed. +/// +public sealed class PollCycleHealthCheck : IHealthCheck +{ + private readonly PollCycleHeartbeat _heartbeat; + private readonly TimeProvider _timeProvider; + private readonly TimeSpan _grace; + + /// Initializes a new . + /// The poller's completed-sweep heartbeat. + /// The registry, used to derive the tolerated gap. + /// Clock. + public PollCycleHealthCheck( + PollCycleHeartbeat heartbeat, + IOptions options, + TimeProvider timeProvider) + { + ArgumentNullException.ThrowIfNull(options); + + _heartbeat = heartbeat ?? throw new ArgumentNullException(nameof(heartbeat)); + _timeProvider = timeProvider ?? throw new ArgumentNullException(nameof(timeProvider)); + _grace = GraceFor(options.Value); + } + + /// + /// The longest gap between completed sweeps that is still considered healthy. + /// + /// The registry. + /// The tolerated gap. + /// + /// A sweep completes on every timer tick, and the timer ticks at the fastest configured + /// interval — so the expected gap is one interval plus however long the probes took, which is + /// bounded by the slowest configured timeout. Twice that is comfortably past "slow" and + /// unambiguously "stopped". + /// + internal static TimeSpan GraceFor(OverviewOptions options) + { + ArgumentNullException.ThrowIfNull(options); + + var timings = options.Applications + .SelectMany(app => app.Instances.Select(instance => + EffectiveTimings.Resolve(options, app, instance))) + .ToList(); + + // An empty registry cannot poll at all; RegistryLoadedHealthCheck is the check that reports + // that, so fall back to the global defaults rather than dividing by an empty set here. + var interval = timings.Count > 0 + ? timings.Min(t => t.PollInterval) + : TimeSpan.FromSeconds(options.PollIntervalSeconds); + var timeout = timings.Count > 0 + ? timings.Max(t => t.Timeout) + : TimeSpan.FromSeconds(options.TimeoutSeconds); + + return 2 * (interval + timeout); + } + + /// + public Task CheckHealthAsync( + HealthCheckContext context, + CancellationToken cancellationToken = default) + { + var now = _timeProvider.GetUtcNow(); + var last = _heartbeat.LastCompletedUtc; + + var data = new Dictionary(StringComparer.Ordinal) + { + ["graceSeconds"] = Math.Round(_grace.TotalSeconds, 3), + }; + + if (last is null) + { + // Nothing has swept yet. Within the grace window that is ordinary startup, so report + // Degraded (still a 200) rather than failing readiness for the first few seconds of + // every deployment. Past it, the poller never started and that IS a failure. + var waiting = now - _heartbeat.StartedUtc; + data["waitingSeconds"] = Math.Round(waiting.TotalSeconds, 3); + + return Task.FromResult(waiting <= _grace + ? HealthCheckResult.Degraded("No poll cycle has completed yet; the poller is still starting.", data: data) + : HealthCheckResult.Unhealthy( + $"No poll cycle has completed in {waiting.TotalSeconds:0.#}s since startup.", + exception: null, + data)); + } + + var age = now - last.Value; + data["lastCycleUtc"] = last.Value.ToString("O", System.Globalization.CultureInfo.InvariantCulture); + data["ageSeconds"] = Math.Round(age.TotalSeconds, 3); + + return Task.FromResult(age <= _grace + ? HealthCheckResult.Healthy($"Last poll cycle completed {age.TotalSeconds:0.#}s ago.", data) + : HealthCheckResult.Unhealthy( + $"No poll cycle has completed in {age.TotalSeconds:0.#}s; the displayed data is stale.", + exception: null, + data)); + } +} diff --git a/ZB.MOM.WW.Overview/src/ZB.MOM.WW.Overview/Observability/PollCycleHeartbeat.cs b/ZB.MOM.WW.Overview/src/ZB.MOM.WW.Overview/Observability/PollCycleHeartbeat.cs new file mode 100644 index 0000000..5f6de79 --- /dev/null +++ b/ZB.MOM.WW.Overview/src/ZB.MOM.WW.Overview/Observability/PollCycleHeartbeat.cs @@ -0,0 +1,46 @@ +namespace ZB.MOM.WW.Overview.Observability; + +/// +/// When the poller last finished a full sweep. Written by the poller, read by +/// . +/// +/// +/// Deliberately not derived from the published snapshot's GeneratedUtc: the poller publishes +/// the registry once before it probes anything, so a snapshot exists from the moment the host +/// starts. A stuck poller would therefore look like a working one. This records only sweeps that +/// actually completed. +/// +public sealed class PollCycleHeartbeat +{ + private long _lastCompletedTicks; + + /// Initializes the heartbeat and stamps the host start time. + /// Clock. + public PollCycleHeartbeat(TimeProvider timeProvider) + { + ArgumentNullException.ThrowIfNull(timeProvider); + StartedUtc = timeProvider.GetUtcNow(); + } + + /// When this heartbeat was constructed — effectively when the host started. + /// + /// Lets the health check distinguish "started a moment ago and has not swept yet" from "has + /// not swept in minutes", which are the same null reading but opposite conclusions. + /// + public DateTimeOffset StartedUtc { get; } + + /// When the last full sweep completed, or null if none has. + public DateTimeOffset? LastCompletedUtc + { + get + { + var ticks = Interlocked.Read(ref _lastCompletedTicks); + return ticks == 0 ? null : new DateTimeOffset(ticks, TimeSpan.Zero); + } + } + + /// Records that a sweep completed. + /// When it completed. + public void RecordCycleCompleted(DateTimeOffset at) => + Interlocked.Exchange(ref _lastCompletedTicks, at.UtcTicks); +} diff --git a/ZB.MOM.WW.Overview/src/ZB.MOM.WW.Overview/Observability/RegistryLoadedHealthCheck.cs b/ZB.MOM.WW.Overview/src/ZB.MOM.WW.Overview/Observability/RegistryLoadedHealthCheck.cs new file mode 100644 index 0000000..92c9ba7 --- /dev/null +++ b/ZB.MOM.WW.Overview/src/ZB.MOM.WW.Overview/Observability/RegistryLoadedHealthCheck.cs @@ -0,0 +1,52 @@ +using Microsoft.Extensions.Diagnostics.HealthChecks; +using Microsoft.Extensions.Options; +using ZB.MOM.WW.Overview.Registry; + +namespace ZB.MOM.WW.Overview.Observability; + +/// +/// Reports how much registry the dashboard actually bound, and fails if it bound none. +/// +/// +/// The pre-host ConfigPreflight and the options validator already refuse to start on an +/// empty registry, so the unhealthy branch is a backstop rather than an expected state. The check +/// earns its place on the reported counts: an operator looking at /health/ready after a +/// config change can see how many instances this process is watching without reading its +/// configuration files. +/// +public sealed class RegistryLoadedHealthCheck : IHealthCheck +{ + private readonly OverviewOptions _options; + + /// Initializes a new . + /// The bound registry. + public RegistryLoadedHealthCheck(IOptions options) + { + ArgumentNullException.ThrowIfNull(options); + _options = options.Value; + } + + /// + public Task CheckHealthAsync( + HealthCheckContext context, + CancellationToken cancellationToken = default) + { + var applications = _options.Applications.Count; + var instances = _options.Applications.Sum(a => a.Instances.Count); + + var data = new Dictionary(StringComparer.Ordinal) + { + ["applications"] = applications, + ["instances"] = instances, + }; + + return Task.FromResult(instances > 0 + ? HealthCheckResult.Healthy( + $"Watching {instances} instance(s) across {applications} application(s).", + data) + : HealthCheckResult.Unhealthy( + "The registry contains no instances; the dashboard has nothing to watch.", + exception: null, + data)); + } +} diff --git a/ZB.MOM.WW.Overview/src/ZB.MOM.WW.Overview/Polling/OverviewPollerService.cs b/ZB.MOM.WW.Overview/src/ZB.MOM.WW.Overview/Polling/OverviewPollerService.cs index 89284d4..7805f2a 100644 --- a/ZB.MOM.WW.Overview/src/ZB.MOM.WW.Overview/Polling/OverviewPollerService.cs +++ b/ZB.MOM.WW.Overview/src/ZB.MOM.WW.Overview/Polling/OverviewPollerService.cs @@ -1,4 +1,5 @@ using Microsoft.Extensions.Options; +using ZB.MOM.WW.Overview.Observability; using ZB.MOM.WW.Overview.Registry; namespace ZB.MOM.WW.Overview.Polling; @@ -29,6 +30,8 @@ public sealed class OverviewPollerService : BackgroundService private readonly OverviewSnapshotStore _store; private readonly TimeProvider _timeProvider; private readonly ILogger _logger; + private readonly OverviewMetrics _metrics; + private readonly PollCycleHeartbeat _heartbeat; private readonly List _targets; /// Initializes a new . @@ -36,12 +39,16 @@ public sealed class OverviewPollerService : BackgroundService /// Factory for the pooled probe client. /// The snapshot store this service writes to. /// Clock and timer source. + /// The app's own instruments. + /// Records that a sweep completed, for the self health check. /// Logger. public OverviewPollerService( IOptions options, IHttpClientFactory httpClientFactory, OverviewSnapshotStore store, TimeProvider timeProvider, + OverviewMetrics metrics, + PollCycleHeartbeat heartbeat, ILogger logger) { ArgumentNullException.ThrowIfNull(options); @@ -49,6 +56,8 @@ public sealed class OverviewPollerService : BackgroundService _httpClientFactory = httpClientFactory ?? throw new ArgumentNullException(nameof(httpClientFactory)); _store = store ?? throw new ArgumentNullException(nameof(store)); _timeProvider = timeProvider ?? throw new ArgumentNullException(nameof(timeProvider)); + _metrics = metrics ?? throw new ArgumentNullException(nameof(metrics)); + _heartbeat = heartbeat ?? throw new ArgumentNullException(nameof(heartbeat)); _logger = logger ?? throw new ArgumentNullException(nameof(logger)); _targets = BuildTargets(options.Value); } @@ -97,6 +106,7 @@ public sealed class OverviewPollerService : BackgroundService // its final state. cancellationToken.ThrowIfCancellationRequested(); + var startedAt = _timeProvider.GetTimestamp(); var now = _timeProvider.GetUtcNow(); var due = _targets.Where(t => IsDue(t, now)).ToList(); @@ -123,6 +133,12 @@ public sealed class OverviewPollerService : BackgroundService var snapshot = BuildSnapshot(now); _store.Publish(snapshot); + _metrics.RecordSweep(_timeProvider.GetElapsedTime(startedAt)); + + // Stamped after the publish, so the heartbeat can never claim a cycle the store did not + // receive — the health check's whole job is to notice exactly that gap. + _heartbeat.RecordCycleCompleted(_timeProvider.GetUtcNow()); + return snapshot; } diff --git a/ZB.MOM.WW.Overview/src/ZB.MOM.WW.Overview/Polling/PollingServiceCollectionExtensions.cs b/ZB.MOM.WW.Overview/src/ZB.MOM.WW.Overview/Polling/PollingServiceCollectionExtensions.cs index acab3d3..470b5cf 100644 --- a/ZB.MOM.WW.Overview/src/ZB.MOM.WW.Overview/Polling/PollingServiceCollectionExtensions.cs +++ b/ZB.MOM.WW.Overview/src/ZB.MOM.WW.Overview/Polling/PollingServiceCollectionExtensions.cs @@ -1,4 +1,5 @@ using Microsoft.Extensions.DependencyInjection.Extensions; +using ZB.MOM.WW.Overview.Observability; namespace ZB.MOM.WW.Overview.Polling; @@ -15,6 +16,12 @@ public static class PollingServiceCollectionExtensions services.TryAddSingleton(TimeProvider.System); services.TryAddSingleton(); + // IMeterFactory comes from AddMetrics, which the default web host already registers; the + // call is idempotent and makes the dependency explicit for any non-web host. + services.AddMetrics(); + services.TryAddSingleton(); + services.TryAddSingleton(); + services.AddHttpClient(OverviewPollerService.HttpClientName, client => { // The per-request CancellationTokenSource in ZbHealthReportClient is the only timeout diff --git a/ZB.MOM.WW.Overview/src/ZB.MOM.WW.Overview/Program.cs b/ZB.MOM.WW.Overview/src/ZB.MOM.WW.Overview/Program.cs index be31ed6..a35205d 100644 --- a/ZB.MOM.WW.Overview/src/ZB.MOM.WW.Overview/Program.cs +++ b/ZB.MOM.WW.Overview/src/ZB.MOM.WW.Overview/Program.cs @@ -1,7 +1,11 @@ using ZB.MOM.WW.Configuration; +using ZB.MOM.WW.Health; using ZB.MOM.WW.Overview.Components; +using ZB.MOM.WW.Overview.Observability; using ZB.MOM.WW.Overview.Polling; using ZB.MOM.WW.Overview.Registry; +using ZB.MOM.WW.Telemetry; +using ZB.MOM.WW.Telemetry.Serilog; var builder = WebApplication.CreateBuilder(args); @@ -17,7 +21,20 @@ ConfigPreflight.For(builder.Configuration) builder.Services.AddValidatedOptions( builder.Configuration, OverviewOptions.SectionName); +builder.AddZbSerilog(o => o.ServiceName = "overview"); + +builder.AddZbTelemetry(o => +{ + o.ServiceName = "overview"; + + // Meters is an ALLOWLIST, not a filter: a Meter absent from this array is never registered with + // the MeterProvider, and its instruments vanish from /metrics with no warning anywhere. The + // dashboard's own gauge is the only thing here that would go missing silently. + o.Meters = [OverviewMetrics.MeterName]; +}); + builder.Services.AddOverviewPolling(); +builder.Services.AddOverviewHealthChecks(); builder.Services.AddRazorComponents() .AddInteractiveServerComponents(); @@ -41,6 +58,12 @@ app.UseAntiforgery(); app.MapRazorComponents() .AddInteractiveServerRenderMode(); +// The dashboard's own three-tier health surface and Prometheus endpoint. Both are anonymous, like +// every /health/* endpoint this app polls — which also means one Overview instance can be pointed +// at another and would render it correctly. +app.MapZbHealth(); +app.MapZbMetrics(); + await app.RunAsync(); /// diff --git a/ZB.MOM.WW.Overview/src/ZB.MOM.WW.Overview/appsettings.Development.json b/ZB.MOM.WW.Overview/src/ZB.MOM.WW.Overview/appsettings.Development.json new file mode 100644 index 0000000..3c0de0f --- /dev/null +++ b/ZB.MOM.WW.Overview/src/ZB.MOM.WW.Overview/appsettings.Development.json @@ -0,0 +1,89 @@ +{ + "Logging": { + "LogLevel": { + "Default": "Information", + "Microsoft.AspNetCore": "Warning" + } + }, + + // Kestrel endpoint config, NOT ASPNETCORE_URLS. Setting both puts Kestrel into + // explicit-endpoints mode and silently discards one of them — the family's recurring + // URLs-override trap. + "Kestrel": { + "Endpoints": { + "Http": { + "Url": "http://localhost:5320" + } + } + }, + + // Development registry for `dotnet run` ON THE MAC HOST. This is deliberately a SUBSET of the + // fleet: the dev rigs publish only some node ports to the host, so these are the instances a + // host-run dashboard can genuinely reach. Verified against the running containers 2026-07-24. + // + // ScadaBridge central pair localhost:9001 / :9002 (container :5000 published) + // ScadaBridge site nodes NOT published — their health port (8084) is container-internal + // OtOpcUa (all six nodes) NOT published — only Traefik's load-balanced :9200, which + // round-robins the central pair and so cannot identify a node + // HistorianGateway localhost:5220 + // MxAccessGateway windev 10.100.0.48:5130 (needs the VPN up) + // + // Run the containerised dashboard (docker/, environment "Docker") to see the whole fleet: it + // joins the rig networks and addresses every node by container DNS name. + "Overview": { + "PollIntervalSeconds": 10, + "TimeoutSeconds": 3, + "StaleAfterSeconds": 45, + "Applications": [ + { + "Name": "ScadaBridge", + "ManagementLabel": "CentralUI", + "Instances": [ + { + "Name": "central-a", + "Group": "central", + "BaseUrl": "http://localhost:9001", + "ManagementUrl": "http://localhost:9001/", + "HasActiveRole": true + }, + { + "Name": "central-b", + "Group": "central", + "BaseUrl": "http://localhost:9002", + "ManagementUrl": "http://localhost:9002/", + "HasActiveRole": true + } + ] + }, + { + "Name": "HistorianGateway", + "ManagementLabel": "Dashboard", + "Instances": [ + { + "Name": "histgw", + "BaseUrl": "http://localhost:5220", + "ManagementUrl": "http://localhost:5220/", + "HasActiveRole": false, + + // Its readiness probe reaches the real historian over the VPN; when that is down the + // probe hangs rather than refusing, so it needs more room than the 3s default before + // it is honestly called unreachable. + "TimeoutSeconds": 10 + } + ] + }, + { + "Name": "MxAccessGateway", + "ManagementLabel": "Dashboard", + "Instances": [ + { + "Name": "windev", + "BaseUrl": "http://10.100.0.48:5130", + "ManagementUrl": "http://10.100.0.48:5130/", + "HasActiveRole": false + } + ] + } + ] + } +} diff --git a/ZB.MOM.WW.Overview/src/ZB.MOM.WW.Overview/appsettings.Docker.json b/ZB.MOM.WW.Overview/src/ZB.MOM.WW.Overview/appsettings.Docker.json new file mode 100644 index 0000000..e8f4751 --- /dev/null +++ b/ZB.MOM.WW.Overview/src/ZB.MOM.WW.Overview/appsettings.Docker.json @@ -0,0 +1,163 @@ +{ + "Logging": { + "LogLevel": { + "Default": "Information", + "Microsoft.AspNetCore": "Warning" + } + }, + + "Kestrel": { + "Endpoints": { + "Http": { + "Url": "http://+:5320" + } + } + }, + + // Full-fleet registry for the CONTAINERISED dashboard, which joins the rig networks + // (otopcua-dev_default, scadabridge-net, zb-historiangw_default) and therefore reaches every + // node by container DNS name — including the ones whose ports are never published to the host. + // + // Ports below were verified by probing the running rigs on 2026-07-24, not read off the compose + // files, because the two disagree: + // OtOpcUa central-1/2 :9000 (ASPNETCORE_URLS) + // OtOpcUa site-* :8080 (NOT 9000 — the site nodes' explicit Kestrel listeners for + // LocalDb sync re-bind the primary HTTP port) + // ScadaBridge central :5000 + // ScadaBridge site :8084 (MetricsPort; MapZbHealth lands here from the site-health work) + // + // ManagementUrl values stay host-relative (localhost:*) on purpose: they are followed by the + // OPERATOR'S BROWSER, not by this container, so container DNS names would be dead links. + "Overview": { + "PollIntervalSeconds": 10, + "TimeoutSeconds": 3, + "StaleAfterSeconds": 45, + "Applications": [ + { + "Name": "ScadaBridge", + "ManagementLabel": "CentralUI", + "Instances": [ + { + "Name": "central-a", + "Group": "central", + "BaseUrl": "http://scadabridge-central-a:5000", + "ManagementUrl": "http://localhost:9001/", + "HasActiveRole": true + }, + { + "Name": "central-b", + "Group": "central", + "BaseUrl": "http://scadabridge-central-b:5000", + "ManagementUrl": "http://localhost:9002/", + "HasActiveRole": true + }, + { + "Name": "site-a-a", + "Group": "site-a", + "BaseUrl": "http://scadabridge-site-a-a:8084", + "HasActiveRole": true + }, + { + "Name": "site-a-b", + "Group": "site-a", + "BaseUrl": "http://scadabridge-site-a-b:8084", + "HasActiveRole": true + }, + { + "Name": "site-b-a", + "Group": "site-b", + "BaseUrl": "http://scadabridge-site-b-a:8084", + "HasActiveRole": true + }, + { + "Name": "site-b-b", + "Group": "site-b", + "BaseUrl": "http://scadabridge-site-b-b:8084", + "HasActiveRole": true + }, + { + "Name": "site-c-a", + "Group": "site-c", + "BaseUrl": "http://scadabridge-site-c-a:8084", + "HasActiveRole": true + }, + { + "Name": "site-c-b", + "Group": "site-c", + "BaseUrl": "http://scadabridge-site-c-b:8084", + "HasActiveRole": true + } + ] + }, + { + "Name": "OtOpcUa", + "ManagementLabel": "AdminUI", + "Instances": [ + { + "Name": "central-1", + "Group": "MAIN", + "BaseUrl": "http://central-1:9000", + "ManagementUrl": "http://localhost:9200/", + "HasActiveRole": true + }, + { + "Name": "central-2", + "Group": "MAIN", + "BaseUrl": "http://central-2:9000", + "ManagementUrl": "http://localhost:9200/", + "HasActiveRole": true + }, + { + "Name": "site-a-1", + "Group": "SITE-A", + "BaseUrl": "http://site-a-1:8080", + "HasActiveRole": true + }, + { + "Name": "site-a-2", + "Group": "SITE-A", + "BaseUrl": "http://site-a-2:8080", + "HasActiveRole": true + }, + { + "Name": "site-b-1", + "Group": "SITE-B", + "BaseUrl": "http://site-b-1:8080", + "HasActiveRole": true + }, + { + "Name": "site-b-2", + "Group": "SITE-B", + "BaseUrl": "http://site-b-2:8080", + "HasActiveRole": true + } + ] + }, + { + "Name": "HistorianGateway", + "ManagementLabel": "Dashboard", + "Instances": [ + { + "Name": "histgw", + "BaseUrl": "http://zb-historiangw:5220", + "ManagementUrl": "http://localhost:5220/", + "HasActiveRole": false, + "TimeoutSeconds": 10 + } + ] + }, + { + "Name": "MxAccessGateway", + "ManagementLabel": "Dashboard", + "Instances": [ + { + "Name": "windev", + "BaseUrl": "http://10.100.0.48:5130", + "ManagementUrl": "http://10.100.0.48:5130/", + "HasActiveRole": false + } + ] + } + ] + } +} diff --git a/ZB.MOM.WW.Overview/src/ZB.MOM.WW.Overview/appsettings.json b/ZB.MOM.WW.Overview/src/ZB.MOM.WW.Overview/appsettings.json new file mode 100644 index 0000000..f66f47d --- /dev/null +++ b/ZB.MOM.WW.Overview/src/ZB.MOM.WW.Overview/appsettings.json @@ -0,0 +1,54 @@ +{ + "Logging": { + "LogLevel": { + "Default": "Information", + "Microsoft.AspNetCore": "Warning" + } + }, + "AllowedHosts": "*", + + // AddZbSerilog drives its sinks entirely from this section (ReadFrom.Configuration) and adds + // none of its own. An empty "Serilog": {} therefore produces an app that logs NOWHERE — verified + // against the running HistorianGateway container, whose log stream is completely empty for + // exactly this reason. Declaring the sinks here is what keeps this app diagnosable. + "Serilog": { + // Every probe emits four Information-level HttpClient lines. At 16 instances on a 10s cadence + // that is several lines a second of pure noise from an app whose entire job is polling — it + // would bury anything worth reading. Note this override, not Logging:LogLevel, is what governs + // once Serilog takes over. + "MinimumLevel": { + "Default": "Information", + "Override": { + "Microsoft.AspNetCore": "Warning", + "System.Net.Http.HttpClient": "Warning" + } + }, + "Using": [ + "Serilog.Sinks.Console", + "Serilog.Sinks.File" + ], + "WriteTo": [ + { "Name": "Console" }, + { + "Name": "File", + "Args": { + "path": "logs/overview-.log", + "rollingInterval": "Day" + } + } + ] + }, + + "Telemetry": {}, + + // Tuning defaults only. `Applications` is deliberately EMPTY here: the registry is a + // per-deployment fact, and shipping a placeholder fleet would let a misconfigured deployment + // boot and render instances nobody asked about. With no registry supplied, ConfigPreflight + // fails startup and names the missing key. See docker/README.md for the full shape. + "Overview": { + "PollIntervalSeconds": 10, + "TimeoutSeconds": 3, + "StaleAfterSeconds": 45, + "Applications": [] + } +} diff --git a/ZB.MOM.WW.Overview/tests/ZB.MOM.WW.Overview.Tests/BootTests.cs b/ZB.MOM.WW.Overview/tests/ZB.MOM.WW.Overview.Tests/BootTests.cs new file mode 100644 index 0000000..e5a1714 --- /dev/null +++ b/ZB.MOM.WW.Overview/tests/ZB.MOM.WW.Overview.Tests/BootTests.cs @@ -0,0 +1,193 @@ +using System.Net; +using Microsoft.AspNetCore.Mvc.Testing; +using Microsoft.Extensions.DependencyInjection; +using Microsoft.Extensions.Hosting; +using ZB.MOM.WW.Overview.Observability; +using ZB.MOM.WW.Overview.Polling; + +namespace ZB.MOM.WW.Overview.Tests; + +/// +/// Boots the REAL Program.cs and asks the questions no in-process test can: does every +/// endpoint answer without a login, and does a bad registry stop the host rather than produce a +/// dashboard full of nothing. +/// +/// +/// The registry is supplied through environment variables rather than +/// ConfigureAppConfiguration. ConfigPreflight runs against +/// builder.Configuration in the top-level statements, before Build() — which is when +/// the factory's configuration callbacks are applied — so a callback-supplied registry would arrive +/// too late for the very check these tests are about. Environment variables are read by +/// CreateBuilder itself, so they are in place from the first line. +/// +/// +/// Environment variables are process-global, so every test here lives in one class: xunit runs a +/// class's tests sequentially, which is what keeps two hosts from fighting over the same keys. +/// +/// +public sealed class BootTests : IDisposable +{ + private readonly List _keys = []; + + private void Set(string key, string? value) + { + _keys.Add(key); + Environment.SetEnvironmentVariable(key, value); + } + + /// Restores the environment so a later-running test class sees a clean process. + public void Dispose() + { + foreach (var key in _keys) + Environment.SetEnvironmentVariable(key, null); + } + + /// + /// A registry with one instance pointing at a port nothing listens on. The dashboard must boot + /// and serve regardless of whether anything it watches is up — that is the entire premise of + /// rendering from cache. + /// + private void SetValidRegistry() + { + Set("ASPNETCORE_ENVIRONMENT", "Testing"); + Set("Overview__PollIntervalSeconds", "10"); + Set("Overview__TimeoutSeconds", "1"); + Set("Overview__StaleAfterSeconds", "45"); + Set("Overview__Applications__0__Name", "TestApp"); + Set("Overview__Applications__0__ManagementLabel", "UI"); + Set("Overview__Applications__0__Instances__0__Name", "node-a"); + Set("Overview__Applications__0__Instances__0__BaseUrl", "http://127.0.0.1:1"); + Set("Overview__Applications__0__Instances__0__HasActiveRole", "false"); + } + + private static WebApplicationFactory Factory() => new(); + + [Fact] + public async Task Page_IsServedAnonymously() + { + SetValidRegistry(); + using var factory = Factory(); + + // AllowAutoRedirect off so an auth redirect would surface as a 302 here rather than being + // silently followed to a login page that then returns 200. + using var client = factory.CreateClient(new WebApplicationFactoryClientOptions + { + AllowAutoRedirect = false, + }); + + using var response = await client.GetAsync("/"); + + Assert.Equal(HttpStatusCode.OK, response.StatusCode); + Assert.Contains("SCADA Overview", await response.Content.ReadAsStringAsync(), StringComparison.Ordinal); + } + + [Fact] + public async Task Page_RendersEvenThoughNothingItWatchesIsReachable() + { + // Cache-first, stated as a test: the registry above points at a dead port, and the page + // must still paint its full layout instead of waiting on a sweep. + SetValidRegistry(); + using var factory = Factory(); + using var client = factory.CreateClient(); + + var html = await client.GetStringAsync("/"); + + Assert.Contains("data-testid=\"instance-card\"", html, StringComparison.Ordinal); + Assert.Contains("node-a", html, StringComparison.Ordinal); + } + + [Theory] + [InlineData("/health/ready")] + [InlineData("/health/active")] + [InlineData("/healthz")] + [InlineData("/metrics")] + public async Task HealthAndMetrics_AreServedAnonymously(string path) + { + SetValidRegistry(); + using var factory = Factory(); + using var client = factory.CreateClient(new WebApplicationFactoryClientOptions + { + AllowAutoRedirect = false, + }); + + using var response = await client.GetAsync(path); + + Assert.Equal(HttpStatusCode.OK, response.StatusCode); + } + + [Fact] + public async Task Ready_ReportsTheDashboardsOwnChecks() + { + SetValidRegistry(); + using var factory = Factory(); + using var client = factory.CreateClient(); + + var body = await client.GetStringAsync("/health/ready"); + + Assert.Contains(ObservabilityServiceCollectionExtensions.RegistryLoadedCheckName, body, StringComparison.Ordinal); + Assert.Contains(ObservabilityServiceCollectionExtensions.PollCycleCheckName, body, StringComparison.Ordinal); + } + + [Fact] + public async Task Metrics_ExportsTheAppsOwnMeter() + { + // The Meters allowlist in AddZbTelemetry is silent when it is wrong: a missing name drops + // the app's whole instrument surface from /metrics with no warning anywhere. Only an + // end-to-end scrape catches that. + SetValidRegistry(); + using var factory = Factory(); + using var client = factory.CreateClient(); + + // Force a sweep so the observable gauge has a probed instance to report. The instance is + // unreachable, which is a perfectly good status to publish. + var poller = factory.Services.GetServices().OfType().Single(); + await poller.SweepAsync(); + + var body = await client.GetStringAsync("/metrics"); + + Assert.Contains("overview_instance_status", body, StringComparison.Ordinal); + Assert.Contains("node=\"node-a\"", body, StringComparison.Ordinal); + } + + [Fact] + public async Task MissingRegistry_FailsToBootAndNamesTheKey() + { + // appsettings.json ships an EMPTY Applications list precisely so this is the outcome when a + // deployment forgets its registry — an empty dashboard that boots would look like a healthy + // fleet of nothing. + Set("ASPNETCORE_ENVIRONMENT", "Testing"); + + using var factory = Factory(); + + var error = await Record.ExceptionAsync(() => factory.CreateClient().GetAsync("/")); + + Assert.NotNull(error); + Assert.Contains("Overview:Applications:0:Name", Flatten(error), StringComparison.Ordinal); + } + + [Fact] + public async Task StaleWindowInsideThePollInterval_FailsToBootWithTheValidatorMessage() + { + // Past preflight and into the validator: a staleness window shorter than the poll interval + // would mark every instance Stale between two perfectly healthy polls. + SetValidRegistry(); + Set("Overview__StaleAfterSeconds", "5"); + + using var factory = Factory(); + + var error = await Record.ExceptionAsync(() => factory.CreateClient().GetAsync("/")); + + Assert.NotNull(error); + Assert.Contains("StaleAfterSeconds", Flatten(error), StringComparison.Ordinal); + } + + private static string Flatten(Exception exception) + { + var text = new System.Text.StringBuilder(); + + for (var current = exception; current is not null; current = current.InnerException) + text.AppendLine(current.Message); + + return text.ToString(); + } +} diff --git a/ZB.MOM.WW.Overview/tests/ZB.MOM.WW.Overview.Tests/ObservabilityTests.cs b/ZB.MOM.WW.Overview/tests/ZB.MOM.WW.Overview.Tests/ObservabilityTests.cs new file mode 100644 index 0000000..94919ef --- /dev/null +++ b/ZB.MOM.WW.Overview/tests/ZB.MOM.WW.Overview.Tests/ObservabilityTests.cs @@ -0,0 +1,309 @@ +using System.Diagnostics.Metrics; +using Microsoft.Extensions.Diagnostics.HealthChecks; +using Microsoft.Extensions.Options; +using ZB.MOM.WW.Overview.Observability; +using ZB.MOM.WW.Overview.Polling; +using ZB.MOM.WW.Overview.Registry; + +namespace ZB.MOM.WW.Overview.Tests; + +/// +/// The dashboard watching itself: the gauge it exports about the fleet, and the two checks that +/// decide whether its own /health/ready is telling the truth. +/// +public class ObservabilityTests +{ + private static readonly DateTimeOffset Start = new(2026, 7, 24, 12, 0, 0, TimeSpan.Zero); + + private static OverviewOptions Registry( + int pollIntervalSeconds = 10, + int timeoutSeconds = 3, + params ApplicationEntry[] applications) => + new() + { + PollIntervalSeconds = pollIntervalSeconds, + TimeoutSeconds = timeoutSeconds, + StaleAfterSeconds = 45, + Applications = applications.Length > 0 + ? applications + : [new ApplicationEntry + { + Name = "App", + Instances = [new InstanceEntry { Name = "a", BaseUrl = "http://a:8080" }], + }], + }; + + private static HealthCheckContext Context() => new(); + + // ---------------------------------------------------------------- registry-loaded + + [Fact] + public async Task RegistryLoaded_ReportsTheCountsItBound() + { + var options = Registry(applications: + [ + new ApplicationEntry + { + Name = "ScadaBridge", + Instances = + [ + new InstanceEntry { Name = "central-a", BaseUrl = "http://central-a:9000" }, + new InstanceEntry { Name = "central-b", BaseUrl = "http://central-b:9100" }, + ], + }, + new ApplicationEntry + { + Name = "OtOpcUa", + Instances = [new InstanceEntry { Name = "admin", BaseUrl = "http://admin:9200" }], + }, + ]); + + var result = await new RegistryLoadedHealthCheck(Options.Create(options)) + .CheckHealthAsync(Context()); + + Assert.Equal(HealthStatus.Healthy, result.Status); + Assert.Equal(2, result.Data["applications"]); + Assert.Equal(3, result.Data["instances"]); + } + + [Fact] + public async Task RegistryLoaded_EmptyRegistry_IsUnhealthy() + { + // Unreachable through the real startup path — preflight and the validator both refuse it — + // but the check must still be able to say so, or it is only ever decorative. + var options = new OverviewOptions { Applications = [] }; + + var result = await new RegistryLoadedHealthCheck(Options.Create(options)) + .CheckHealthAsync(Context()); + + Assert.Equal(HealthStatus.Unhealthy, result.Status); + Assert.Equal(0, result.Data["instances"]); + } + + // ---------------------------------------------------------------- grace window + + [Fact] + public void Grace_IsTwiceTheFastestIntervalPlusTheSlowestTimeout() + { + var options = Registry(pollIntervalSeconds: 10, timeoutSeconds: 3); + + Assert.Equal(TimeSpan.FromSeconds(26), PollCycleHealthCheck.GraceFor(options)); + } + + [Fact] + public void Grace_HonoursPerInstanceOverrides() + { + // The sweep loop ticks at the FASTEST configured interval, and the slowest timeout bounds + // how long any one sweep can take — so the window has to be derived from both extremes, + // not from the global defaults. + var options = Registry(pollIntervalSeconds: 30, timeoutSeconds: 3, applications: + [ + new ApplicationEntry + { + Name = "App", + Instances = + [ + new InstanceEntry { Name = "fast", BaseUrl = "http://fast:8080", PollIntervalSeconds = 5 }, + new InstanceEntry { Name = "slow", BaseUrl = "http://slow:8080", TimeoutSeconds = 20 }, + ], + }, + ]); + + Assert.Equal(TimeSpan.FromSeconds(50), PollCycleHealthCheck.GraceFor(options)); + } + + // ---------------------------------------------------------------- last-poll-cycle-completed + + [Fact] + public async Task PollCycle_NoCycleYetButStillStarting_IsDegradedNotUnhealthy() + { + // Every deployment passes through this state. Failing readiness here would make the + // container look broken for the first seconds of its life, every single restart. + var time = new FakeTimeProvider(Start); + var check = new PollCycleHealthCheck(new PollCycleHeartbeat(time), Options.Create(Registry()), time); + + time.Advance(TimeSpan.FromSeconds(5)); + var result = await check.CheckHealthAsync(Context()); + + Assert.Equal(HealthStatus.Degraded, result.Status); + } + + [Fact] + public async Task PollCycle_NeverSwept_EventuallyGoesUnhealthy() + { + // A poller that never started must not hide behind "still starting" forever. + var time = new FakeTimeProvider(Start); + var check = new PollCycleHealthCheck(new PollCycleHeartbeat(time), Options.Create(Registry()), time); + + time.Advance(TimeSpan.FromMinutes(5)); + var result = await check.CheckHealthAsync(Context()); + + Assert.Equal(HealthStatus.Unhealthy, result.Status); + Assert.Contains("since startup", result.Description, StringComparison.Ordinal); + } + + [Fact] + public async Task PollCycle_RecentSweep_IsHealthyAndPublishesItsAge() + { + var time = new FakeTimeProvider(Start); + var heartbeat = new PollCycleHeartbeat(time); + var check = new PollCycleHealthCheck(heartbeat, Options.Create(Registry()), time); + + heartbeat.RecordCycleCompleted(Start); + time.Advance(TimeSpan.FromSeconds(4)); + var result = await check.CheckHealthAsync(Context()); + + Assert.Equal(HealthStatus.Healthy, result.Status); + Assert.Equal(4d, result.Data["ageSeconds"]); + Assert.Equal(26d, result.Data["graceSeconds"]); + } + + [Fact] + public async Task PollCycle_StoppedSweeping_IsUnhealthy() + { + // The failure this check exists for: Kestrel is fine, the page still renders, and every + // card on it is quietly frozen. + var time = new FakeTimeProvider(Start); + var heartbeat = new PollCycleHeartbeat(time); + var check = new PollCycleHealthCheck(heartbeat, Options.Create(Registry()), time); + + heartbeat.RecordCycleCompleted(Start); + time.Advance(TimeSpan.FromMinutes(2)); + var result = await check.CheckHealthAsync(Context()); + + Assert.Equal(HealthStatus.Unhealthy, result.Status); + Assert.Contains("stale", result.Description, StringComparison.Ordinal); + } + + [Fact] + public void Heartbeat_ReportsNothingUntilACycleCompletes() + { + var heartbeat = new PollCycleHeartbeat(new FakeTimeProvider(Start)); + + Assert.Null(heartbeat.LastCompletedUtc); + Assert.Equal(Start, heartbeat.StartedUtc); + + heartbeat.RecordCycleCompleted(Start.AddSeconds(7)); + + Assert.Equal(Start.AddSeconds(7), heartbeat.LastCompletedUtc); + } + + // ---------------------------------------------------------------- the fleet gauge + + [Fact] + public void Gauge_ReportsOneMeasurementPerProbedInstance() + { + var store = new OverviewSnapshotStore(); + store.Publish(SnapshotOf( + Snapshots.Instance("central-a", group: "central", status: InstanceStatus.Up), + Snapshots.Instance("central-b", group: "central", status: InstanceStatus.Degraded), + Snapshots.Instance("site-a-a", group: "site-a", status: InstanceStatus.Unreachable))); + + using var factory = new DummyMeterFactory(); + using var metrics = new OverviewMetrics(factory, store); + + var measured = CollectStatusGauge(factory); + + Assert.Equal(3, measured.Count); + Assert.Equal(0, measured.Single(m => m.Tags["node"] as string == "central-a").Value); + Assert.Equal(1, measured.Single(m => m.Tags["node"] as string == "central-b").Value); + Assert.Equal(3, measured.Single(m => m.Tags["node"] as string == "site-a-a").Value); + Assert.Equal("central", measured.First().Tags["group"]); + Assert.Equal("App", measured.First().Tags["application"]); + } + + [Fact] + public void Gauge_SkipsInstancesThatHaveNeverBeenProbed() + { + // The enum's zero ordinal is Up, so emitting anything for an unprobed instance would + // publish the single most reassuring answer about a node nobody has heard from. + var store = new OverviewSnapshotStore(); + store.Publish(SnapshotOf( + Snapshots.Instance("probed", status: InstanceStatus.Down), + Snapshots.Instance("never-probed", status: null))); + + using var factory = new DummyMeterFactory(); + using var metrics = new OverviewMetrics(factory, store); + + var measured = CollectStatusGauge(factory); + + Assert.Equal("probed", Assert.Single(measured).Tags["node"]); + } + + [Fact] + public void Gauge_TracksTheStoreRatherThanASnapshotTakenAtConstruction() + { + // It is an OBSERVABLE gauge: read at scrape time from whatever the store currently holds, + // so the metric and the page can never disagree about what the dashboard believes. + var store = new OverviewSnapshotStore(); + store.Publish(SnapshotOf(Snapshots.Instance("a", status: InstanceStatus.Up))); + + using var factory = new DummyMeterFactory(); + using var metrics = new OverviewMetrics(factory, store); + + Assert.Equal(0, Assert.Single(CollectStatusGauge(factory)).Value); + + store.Publish(SnapshotOf(Snapshots.Instance("a", status: InstanceStatus.Down))); + + Assert.Equal(2, Assert.Single(CollectStatusGauge(factory)).Value); + } + + [Fact] + public async Task Sweep_StampsTheHeartbeatSoTheSelfCheckCanSeeIt() + { + // Ties the two halves together: without this the health check would be measuring a value + // nothing ever writes, and would report a healthy poller forever. + var time = new FakeTimeProvider(Start); + var store = new OverviewSnapshotStore(); + var heartbeat = new PollCycleHeartbeat(time); + var handler = new RoutingHandler() + .Json("http://a:8080/health/ready", System.Net.HttpStatusCode.OK, HealthBody.Ready("Healthy")); + + using var factory = new DummyMeterFactory(); + var poller = new OverviewPollerService( + Options.Create(Registry()), + new StubHttpClientFactory(handler), + store, + time, + new OverviewMetrics(factory, store), + heartbeat, + Microsoft.Extensions.Logging.Abstractions.NullLogger.Instance); + + Assert.Null(heartbeat.LastCompletedUtc); + + await poller.SweepAsync(); + + Assert.Equal(Start, heartbeat.LastCompletedUtc); + } + + private static OverviewSnapshot SnapshotOf(params InstanceSnapshot[] instances) => + new(Start, [new ApplicationSnapshot("App", instances, [])]); + + private static List<(int Value, Dictionary Tags)> CollectStatusGauge( + DummyMeterFactory factory) + { + var measured = new List<(int, Dictionary)>(); + + using var listener = new MeterListener + { + InstrumentPublished = (instrument, l) => + { + // Scoped to THIS test's factory: meter names are process-global, so matching on + // the name alone would also collect instruments from tests running in parallel. + if (ReferenceEquals(instrument.Meter.Scope, factory) && + instrument.Name == OverviewMetrics.InstanceStatusInstrument) + { + l.EnableMeasurementEvents(instrument); + } + }, + }; + + listener.SetMeasurementEventCallback((_, value, tags, _) => + measured.Add((value, tags.ToArray().ToDictionary(t => t.Key, t => t.Value, StringComparer.Ordinal)))); + + listener.Start(); + listener.RecordObservableInstruments(); + + return measured; + } +} diff --git a/ZB.MOM.WW.Overview/tests/ZB.MOM.WW.Overview.Tests/OverviewPollerServiceTests.cs b/ZB.MOM.WW.Overview/tests/ZB.MOM.WW.Overview.Tests/OverviewPollerServiceTests.cs index 54e12b0..0da8a69 100644 --- a/ZB.MOM.WW.Overview/tests/ZB.MOM.WW.Overview.Tests/OverviewPollerServiceTests.cs +++ b/ZB.MOM.WW.Overview/tests/ZB.MOM.WW.Overview.Tests/OverviewPollerServiceTests.cs @@ -1,6 +1,7 @@ using System.Net; using Microsoft.Extensions.Logging.Abstractions; using Microsoft.Extensions.Options; +using ZB.MOM.WW.Overview.Observability; using ZB.MOM.WW.Overview.Polling; using ZB.MOM.WW.Overview.Registry; @@ -52,6 +53,8 @@ public class OverviewPollerServiceTests new StubHttpClientFactory(handler), store, timeProvider, + new OverviewMetrics(new DummyMeterFactory(), store), + new PollCycleHeartbeat(timeProvider), NullLogger.Instance); return (poller, store); diff --git a/ZB.MOM.WW.Overview/tests/ZB.MOM.WW.Overview.Tests/PollingTestSupport.cs b/ZB.MOM.WW.Overview/tests/ZB.MOM.WW.Overview.Tests/PollingTestSupport.cs index 52b74ed..f6ead8e 100644 --- a/ZB.MOM.WW.Overview/tests/ZB.MOM.WW.Overview.Tests/PollingTestSupport.cs +++ b/ZB.MOM.WW.Overview/tests/ZB.MOM.WW.Overview.Tests/PollingTestSupport.cs @@ -1,3 +1,4 @@ +using System.Diagnostics.Metrics; using System.Net; using System.Text; using System.Text.Json; @@ -5,6 +6,34 @@ using ZB.MOM.WW.Overview.Polling; namespace ZB.MOM.WW.Overview.Tests; +/// A meter factory with no listener attached, for tests that only need the poller to run. +internal sealed class DummyMeterFactory : IMeterFactory +{ + private readonly List _meters = []; + + public Meter Create(MeterOptions options) + { + ArgumentNullException.ThrowIfNull(options); + + // Stamp ourselves as the scope, exactly as the real factory does. Meter names are global, + // so a MeterListener that matched on name alone would pick up meters created by tests + // running in parallel; the scope is what makes one test's instruments identifiable. + options.Scope = this; + + var meter = new Meter(options); + _meters.Add(meter); + return meter; + } + + public void Dispose() + { + foreach (var meter in _meters) + meter.Dispose(); + + _meters.Clear(); + } +} + /// A clock the test drives by hand. /// /// Only is overridden: the poller's sweep — the part with rules worth diff --git a/docs/plans/2026-07-22-overview-dashboard-impl-plan.md.tasks.json b/docs/plans/2026-07-22-overview-dashboard-impl-plan.md.tasks.json index 7826a0f..2105704 100644 --- a/docs/plans/2026-07-22-overview-dashboard-impl-plan.md.tasks.json +++ b/docs/plans/2026-07-22-overview-dashboard-impl-plan.md.tasks.json @@ -130,7 +130,7 @@ { "id": "17", "subject": "3.7 Overview: self-observability", - "status": "pending", + "status": "completed", "blockedBy": [ "14" ] @@ -138,7 +138,7 @@ { "id": "18", "subject": "3.8 Overview: tests", - "status": "pending", + "status": "completed", "blockedBy": [ "16", "17" @@ -147,7 +147,7 @@ { "id": "19", "subject": "3.9 Overview: config + docker", - "status": "pending", + "status": "completed", "blockedBy": [ "17" ] @@ -182,7 +182,38 @@ ], "deviations": [ "Task 3.6 said 'no antiforgery'. AddRazorComponents stamps antiforgery metadata on every component endpoint unconditionally and the endpoint middleware hard-fails without a matching middleware, so every page returned 500. Resolved with app.UseAntiforgery() (inert here, no forms) rather than MapRazorComponents(...).DisableAntiforgery(), which would silently expose the first form anyone adds. No authentication was added.", - "Bootstrap IS vendored (copied from HistorianGateway, v5.3.3). Not optional: ThemeShell and TechButton are built on Bootstrap utility classes, so the shared kit does not render without it." + "Bootstrap IS vendored (copied from HistorianGateway, v5.3.3). Not optional: ThemeShell and TechButton are built on Bootstrap utility classes, so the shared kit does not render without it.", + { + "task": "3.7", + "what": "Metric names follow components/observability/spec/METRIC-CONVENTIONS.md (dot-separated overview.instance.status / overview.poll.duration, unit s) instead of the plan's literal zb_overview_instance_status.", + "why": "That spec is marked Standardized and authoritative for the family; the Prometheus exporter renders them as overview_instance_status and overview_poll_duration_seconds. Gauge labels are application/node/group \u2014 'node', not 'instance', because Prometheus stamps its own instance label on every scraped series and a colliding metric label is silently renamed exported_instance." + }, + { + "task": "3.9", + "what": "Three registries, not one: appsettings.json ships an EMPTY Applications list, appsettings.Development.json a host-reachable subset (4 instances), appsettings.Docker.json the full 16-instance fleet.", + "why": "The dev rigs publish only some node ports to the host. OtOpcUa publishes none per-node (only Traefik :9200, load-balanced across the central pair, so it cannot identify a node) and ScadaBridge's site nodes keep :8084 container-internal. A host-run dashboard can genuinely only see part of the fleet; the containerised one joins the rig networks and sees all of it. Empty Applications in appsettings.json keeps a registry-less production deploy failing fast at ConfigPreflight instead of booting into an empty dashboard." + }, + { + "task": "3.9", + "what": "appsettings.json declares real Serilog sinks and MinimumLevel overrides rather than the family's empty \"Serilog\": {}.", + "why": "AddZbSerilog drives sinks entirely from configuration and adds none of its own, so an empty section produces an app that logs NOWHERE \u2014 verified against the running HistorianGateway container, whose log stream is completely empty for exactly this reason (OtOpcUa, which populates the section, logs normally). The HttpClient override is needed too: 16 instances on a 10s cadence emit several Information lines a second of pure probe noise." + }, + { + "task": "3.9", + "what": "Dockerfile clears ASPNETCORE_HTTP_PORTS as well as ASPNETCORE_URLS.", + "why": "On .NET 8+ the aspnet base image declares its binding via ASPNETCORE_HTTP_PORTS=8080, not ASPNETCORE_URLS. Clearing only URLS leaves Kestrel logging 'Overriding address(es) http://*:8080' on every boot \u2014 the URLs-override trap arriving by default rather than by mistake." + }, + { + "task": "3.9", + "what": "No container healthcheck.", + "why": "aspnet:10.0 ships neither curl nor wget, and adding a --healthcheck mode to the app would put a second entry point into production code purely to satisfy Compose. The plan explicitly allows skipping." + } + ], + "phase4_findings": [ + "Rig ports verified by probing, not by reading compose (they disagree): OtOpcUa central-1/2 serve health on :9000 but site-* on :8080 (their explicit Kestrel listeners for LocalDb sync re-bind the primary HTTP port); ScadaBridge central on :5000, sites on :8084.", + "ScadaBridge site nodes currently answer :8084/health/ready with 404 \u2014 MapZbHealth on sites is on the feat/site-node-health branch and the rig runs main. Phase 4 needs the rigs redeployed from the 0.2.0 branches.", + "No leader chip renders on any group yet: the rigs run pre-0.2.0 builds whose akka check carries no data object. This is the designed graceful degradation, and it confirms Phase 4 must redeploy before checking design 9 item 2.", + "SUSPECTED RIG SPLIT-BRAIN: scadabridge-central-a AND central-b both return /health/active 200 with 'oldest Up member (singleton host)'. Two nodes cannot both be the oldest of one cluster. Verify during Phase 4 after redeploy \u2014 this is exactly the condition the dashboard exists to surface." ] } -} +} \ No newline at end of file