Joseph Doherty
92997b40dc
Closes Phase 3 of the overview-dashboard plan (tasks 3.7-3.9).
Self-observability
- OverviewMetrics: an observable gauge read from the published snapshot at scrape
time, so the metric and the page cannot disagree about what the dashboard
believes, plus a sweep-duration histogram. Instrument names follow the family
METRIC-CONVENTIONS spec (overview.instance.status / overview.poll.duration,
unit s) rather than the plan's literal zb_ names; Prometheus renders them as
overview_instance_status and overview_poll_duration_seconds. Labels are
application/node/group - "node", not "instance", because Prometheus stamps its
own instance label on every scraped series and a colliding metric label is
silently renamed exported_instance. Instances never probed emit nothing: the
enum's zero ordinal is Up, so a placeholder would publish the single most
reassuring answer about a node nobody has heard from.
- Two readiness checks make the dashboard's own /health/ready mean something:
registry-loaded (reports the counts it bound) and last-poll-cycle-completed,
which catches the failure where Kestrel is fine, the page still renders, and
every card on it is quietly frozen. The heartbeat is stamped by the sweep, not
derived from the snapshot's timestamp - the poller publishes the registry
before it probes anything, so a stuck poller would otherwise look healthy.
No cycle yet is Degraded inside the grace window and Unhealthy past it, so a
restart does not fail readiness for its first seconds and a poller that never
started cannot hide behind "still starting".
Configuration
- appsettings.json ships tuning defaults with an EMPTY Applications list, so a
deployment that supplies no registry fails at ConfigPreflight naming the key
instead of booting into a dashboard of nothing.
- Three registries, because reachability genuinely differs: Development is the
host-reachable subset (4 instances), Docker is the full 16-instance fleet by
container DNS name. The rigs publish almost no per-node HTTP port to the host
- OtOpcUa publishes none (only Traefik :9200, load-balanced across the central
pair, so it cannot identify a node) and ScadaBridge keeps site :8084
container-internal. Ports were established by probing the running rigs, not
read off the compose files, because the two disagree: OtOpcUa site nodes serve
health on :8080, not the :9000 their compose anchor implies.
- Real Serilog sinks, not the family's empty "Serilog": {}. AddZbSerilog drives
sinks entirely from configuration and adds none of its own, so the empty
section produces an app that logs NOWHERE - verified against the running
HistorianGateway container, whose log stream is completely empty for exactly
this reason. The HttpClient level override matters too: 16 instances on a 10s
cadence emit several Information lines a second of pure probe noise.
Container
- HistorianGateway-pattern runtime-only image; the compose stack joins the three
rig networks as external, so bringing the dashboard up or down cannot disturb
a rig. The Dockerfile clears ASPNETCORE_HTTP_PORTS as well as ASPNETCORE_URLS:
on .NET 8+ the base image declares its binding via the former, and clearing
only the latter leaves Kestrel logging "Overriding address(es)" every boot.
No healthcheck - aspnet:10.0 has neither curl nor wget, and adding a
--healthcheck entry point to production code to satisfy Compose is not a
trade worth making.
Tests
- BootTests boot the real Program.cs: every endpoint anonymous, the page
rendering from cache while nothing it watches is reachable, the Meters
allowlist proven by an end-to-end /metrics scrape, and both refuse-to-boot
paths (missing registry, stale window inside the poll interval).
The registry is supplied via environment variables, not
ConfigureAppConfiguration, because ConfigPreflight runs before Build() - when
the factory's callbacks are applied - and would not see them.
170 tests, 0 warnings. Verified live against the running rigs: 16 cards, 16
gauge series, page and gauge agreeing exactly.
219 lines
8.9 KiB
JSON
219 lines
8.9 KiB
JSON
{
|
|
"plan": "docs/plans/2026-07-22-overview-dashboard-impl-plan.md",
|
|
"tasks": [
|
|
{
|
|
"id": "1",
|
|
"subject": "0.1 Health writer: optional per-entry `data` object",
|
|
"status": "completed",
|
|
"blockedBy": []
|
|
},
|
|
{
|
|
"id": "2",
|
|
"subject": "0.2 AkkaClusterHealthCheck: populate cluster-view data",
|
|
"status": "completed",
|
|
"blockedBy": []
|
|
},
|
|
{
|
|
"id": "3",
|
|
"subject": "0.3 Health 0.2.0: version, test, pack, publish, verify",
|
|
"status": "completed",
|
|
"blockedBy": [
|
|
"1",
|
|
"2"
|
|
]
|
|
},
|
|
{
|
|
"id": "4",
|
|
"subject": "1.1 ScadaBridge: bump Health packages to 0.2.0",
|
|
"status": "completed",
|
|
"blockedBy": [
|
|
"3"
|
|
]
|
|
},
|
|
{
|
|
"id": "5",
|
|
"subject": "1.2 ScadaBridge: site health checks (akka-cluster, localdb, active-node)",
|
|
"status": "completed",
|
|
"blockedBy": [
|
|
"4"
|
|
]
|
|
},
|
|
{
|
|
"id": "6",
|
|
"subject": "1.3 ScadaBridge: map MapZbHealth on the site pipeline",
|
|
"status": "completed",
|
|
"blockedBy": [
|
|
"5"
|
|
]
|
|
},
|
|
{
|
|
"id": "7",
|
|
"subject": "1.4 ScadaBridge: site health tests",
|
|
"status": "completed",
|
|
"blockedBy": [
|
|
"6"
|
|
]
|
|
},
|
|
{
|
|
"id": "8",
|
|
"subject": "2.1 OtOpcUa: verify `akka` check is the shared AkkaClusterHealthCheck",
|
|
"status": "completed",
|
|
"blockedBy": []
|
|
},
|
|
{
|
|
"id": "9",
|
|
"subject": "2.2 OtOpcUa: bump Health to 0.2.0",
|
|
"status": "completed",
|
|
"blockedBy": [
|
|
"3",
|
|
"8"
|
|
]
|
|
},
|
|
{
|
|
"id": "10",
|
|
"subject": "2.3 Alignment bumps: mxgw + HistorianGateway Health 0.2.0",
|
|
"status": "completed",
|
|
"blockedBy": [
|
|
"3"
|
|
]
|
|
},
|
|
{
|
|
"id": "11",
|
|
"subject": "3.1 Overview: scaffold project",
|
|
"status": "completed",
|
|
"blockedBy": [
|
|
"3"
|
|
]
|
|
},
|
|
{
|
|
"id": "12",
|
|
"subject": "3.2 Overview: registry options + validation",
|
|
"status": "completed",
|
|
"blockedBy": [
|
|
"11"
|
|
]
|
|
},
|
|
{
|
|
"id": "13",
|
|
"subject": "3.3 Overview: health client + status model",
|
|
"status": "completed",
|
|
"blockedBy": [
|
|
"11"
|
|
]
|
|
},
|
|
{
|
|
"id": "14",
|
|
"subject": "3.4 Overview: poller + snapshot store",
|
|
"status": "completed",
|
|
"blockedBy": [
|
|
"12",
|
|
"13"
|
|
]
|
|
},
|
|
{
|
|
"id": "15",
|
|
"subject": "3.5 Overview: leader aggregation",
|
|
"status": "completed",
|
|
"blockedBy": [
|
|
"13"
|
|
]
|
|
},
|
|
{
|
|
"id": "16",
|
|
"subject": "3.6 Overview: UI (ThemeShell, Overview page, InstanceCard)",
|
|
"status": "completed",
|
|
"blockedBy": [
|
|
"14",
|
|
"15"
|
|
]
|
|
},
|
|
{
|
|
"id": "17",
|
|
"subject": "3.7 Overview: self-observability",
|
|
"status": "completed",
|
|
"blockedBy": [
|
|
"14"
|
|
]
|
|
},
|
|
{
|
|
"id": "18",
|
|
"subject": "3.8 Overview: tests",
|
|
"status": "completed",
|
|
"blockedBy": [
|
|
"16",
|
|
"17"
|
|
]
|
|
},
|
|
{
|
|
"id": "19",
|
|
"subject": "3.9 Overview: config + docker",
|
|
"status": "completed",
|
|
"blockedBy": [
|
|
"17"
|
|
]
|
|
},
|
|
{
|
|
"id": "20",
|
|
"subject": "4 Live acceptance (design \u00a79)",
|
|
"status": "pending",
|
|
"blockedBy": [
|
|
"7",
|
|
"9",
|
|
"18",
|
|
"19"
|
|
]
|
|
}
|
|
],
|
|
"notes": {
|
|
"deferred": [
|
|
"Task 2.2 rig check (admin node /health/ready shows data.leader) \u2014 live docker-dev, folded into Phase 4.",
|
|
"Phase 1 DoD rig curls (site :8084 ready/active split) \u2014 live docker rig, folded into Phase 4."
|
|
],
|
|
"branches": {
|
|
"scadaproj": "feat/health-0.2.0-cluster-data (80668a0) -> feat/overview-dashboard, stacked",
|
|
"ScadaBridge": "feat/site-node-health (commit 47850c0f)",
|
|
"OtOpcUa": "feat/health-0.2.0-bump (commit 2e515c34, made in an isolated git worktree \u2014 feat/mesh-phase6 working tree untouched)",
|
|
"HistorianGateway": "chore/health-0.2.0-bump (commit 51f0c2b)",
|
|
"MxAccessGateway": "chore/health-0.2.0-bump (commit 2d54ace)"
|
|
},
|
|
"front_loaded": [
|
|
"Validator / health-client / status-derivation tests from task 3.8 were written with tasks 3.2-3.3 so each batch is verified; 3.8 covers the remainder (poller, leader, store, bUnit, boot).",
|
|
"Poller / snapshot-store / leader-resolver / bUnit InstanceCard + Overview page tests were written with tasks 3.4-3.6; task 3.8 now covers only the WebApplicationFactory boot tests."
|
|
],
|
|
"deviations": [
|
|
"Task 3.6 said 'no antiforgery'. AddRazorComponents stamps antiforgery metadata on every component endpoint unconditionally and the endpoint middleware hard-fails without a matching middleware, so every page returned 500. Resolved with app.UseAntiforgery() (inert here, no forms) rather than MapRazorComponents(...).DisableAntiforgery(), which would silently expose the first form anyone adds. No authentication was added.",
|
|
"Bootstrap IS vendored (copied from HistorianGateway, v5.3.3). Not optional: ThemeShell and TechButton are built on Bootstrap utility classes, so the shared kit does not render without it.",
|
|
{
|
|
"task": "3.7",
|
|
"what": "Metric names follow components/observability/spec/METRIC-CONVENTIONS.md (dot-separated overview.instance.status / overview.poll.duration, unit s) instead of the plan's literal zb_overview_instance_status.",
|
|
"why": "That spec is marked Standardized and authoritative for the family; the Prometheus exporter renders them as overview_instance_status and overview_poll_duration_seconds. Gauge labels are application/node/group \u2014 'node', not 'instance', because Prometheus stamps its own instance label on every scraped series and a colliding metric label is silently renamed exported_instance."
|
|
},
|
|
{
|
|
"task": "3.9",
|
|
"what": "Three registries, not one: appsettings.json ships an EMPTY Applications list, appsettings.Development.json a host-reachable subset (4 instances), appsettings.Docker.json the full 16-instance fleet.",
|
|
"why": "The dev rigs publish only some node ports to the host. OtOpcUa publishes none per-node (only Traefik :9200, load-balanced across the central pair, so it cannot identify a node) and ScadaBridge's site nodes keep :8084 container-internal. A host-run dashboard can genuinely only see part of the fleet; the containerised one joins the rig networks and sees all of it. Empty Applications in appsettings.json keeps a registry-less production deploy failing fast at ConfigPreflight instead of booting into an empty dashboard."
|
|
},
|
|
{
|
|
"task": "3.9",
|
|
"what": "appsettings.json declares real Serilog sinks and MinimumLevel overrides rather than the family's empty \"Serilog\": {}.",
|
|
"why": "AddZbSerilog drives sinks entirely from configuration and adds none of its own, so an empty section produces an app that logs NOWHERE \u2014 verified against the running HistorianGateway container, whose log stream is completely empty for exactly this reason (OtOpcUa, which populates the section, logs normally). The HttpClient override is needed too: 16 instances on a 10s cadence emit several Information lines a second of pure probe noise."
|
|
},
|
|
{
|
|
"task": "3.9",
|
|
"what": "Dockerfile clears ASPNETCORE_HTTP_PORTS as well as ASPNETCORE_URLS.",
|
|
"why": "On .NET 8+ the aspnet base image declares its binding via ASPNETCORE_HTTP_PORTS=8080, not ASPNETCORE_URLS. Clearing only URLS leaves Kestrel logging 'Overriding address(es) http://*:8080' on every boot \u2014 the URLs-override trap arriving by default rather than by mistake."
|
|
},
|
|
{
|
|
"task": "3.9",
|
|
"what": "No container healthcheck.",
|
|
"why": "aspnet:10.0 ships neither curl nor wget, and adding a --healthcheck mode to the app would put a second entry point into production code purely to satisfy Compose. The plan explicitly allows skipping."
|
|
}
|
|
],
|
|
"phase4_findings": [
|
|
"Rig ports verified by probing, not by reading compose (they disagree): OtOpcUa central-1/2 serve health on :9000 but site-* on :8080 (their explicit Kestrel listeners for LocalDb sync re-bind the primary HTTP port); ScadaBridge central on :5000, sites on :8084.",
|
|
"ScadaBridge site nodes currently answer :8084/health/ready with 404 \u2014 MapZbHealth on sites is on the feat/site-node-health branch and the rig runs main. Phase 4 needs the rigs redeployed from the 0.2.0 branches.",
|
|
"No leader chip renders on any group yet: the rigs run pre-0.2.0 builds whose akka check carries no data object. This is the designed graceful degradation, and it confirms Phase 4 must redeploy before checking design 9 item 2.",
|
|
"SUSPECTED RIG SPLIT-BRAIN: scadabridge-central-a AND central-b both return /health/active 200 with 'oldest Up member (singleton host)'. Two nodes cannot both be the oldest of one cluster. Verify during Phase 4 after redeploy \u2014 this is exactly the condition the dashboard exists to surface."
|
|
]
|
|
}
|
|
} |