Joseph Doherty
41013dcab3
Found during the live acceptance run. The host calls UseStaticWebAssets only when the environment is literally "Development", so running the build output under any other name - checking a staging registry locally, say - leaves the Theme RCL's assets under _content/ unresolvable. The failure is silent and unhelpful: the page returns 200 with its markup fully present and renders as a blank white screen, because every stylesheet 404s. Calling it unconditionally is a no-op once published (the manifest is gone and the assets are real files on disk), so the only thing it changes is that failure. The container was never affected. Phase 4 results recorded in the ledger: 7 of 9 design section 9 checks fully pass, 2 pass for ScadaBridge and are blocked for OtOpcUa pending a rig redeploy. Notable live evidence: - Leader chip moved from site-c-a to site-c-b when site-c-a was stopped, and the split-brain chip appeared while the pair was self-formed then cleared once it genuinely joined. - Kill to Unreachable took two failing sweeps (the damping working as designed); recovery landed on the first good poll. - Five consecutive page loads in 0.00-0.02s all showed the same "last sweep" timestamp, which is cache-first stated as a measurement. - Freezing the probed endpoints while auto-refresh stayed RUNNING froze the poller mid-sweep and the cards aged into STALE on their own - staleness is computed at render, so a stuck poller needs no explicit stall detection. The run also surfaced a pre-existing rig defect in both products: self-first Akka seed lists mean a pair booted cold self-forms two 1-member clusters that never join. After a --force-recreate every ScadaBridge node named ITSELF leader and every node answered /health/active 200; restarting ONE node of each pair converged them to one Leader plus one Active plus one Standby. That is the user's own open backlog item (docs/plans/2026-07-22-initjoin-selfform-fallback.md, unexecuted), and the dashboard made it visible fleet-wide in one screen. 170 tests, 0 warnings.
234 lines
13 KiB
JSON
234 lines
13 KiB
JSON
{
|
|
"plan": "docs/plans/2026-07-22-overview-dashboard-impl-plan.md",
|
|
"tasks": [
|
|
{
|
|
"id": "1",
|
|
"subject": "0.1 Health writer: optional per-entry `data` object",
|
|
"status": "completed",
|
|
"blockedBy": []
|
|
},
|
|
{
|
|
"id": "2",
|
|
"subject": "0.2 AkkaClusterHealthCheck: populate cluster-view data",
|
|
"status": "completed",
|
|
"blockedBy": []
|
|
},
|
|
{
|
|
"id": "3",
|
|
"subject": "0.3 Health 0.2.0: version, test, pack, publish, verify",
|
|
"status": "completed",
|
|
"blockedBy": [
|
|
"1",
|
|
"2"
|
|
]
|
|
},
|
|
{
|
|
"id": "4",
|
|
"subject": "1.1 ScadaBridge: bump Health packages to 0.2.0",
|
|
"status": "completed",
|
|
"blockedBy": [
|
|
"3"
|
|
]
|
|
},
|
|
{
|
|
"id": "5",
|
|
"subject": "1.2 ScadaBridge: site health checks (akka-cluster, localdb, active-node)",
|
|
"status": "completed",
|
|
"blockedBy": [
|
|
"4"
|
|
]
|
|
},
|
|
{
|
|
"id": "6",
|
|
"subject": "1.3 ScadaBridge: map MapZbHealth on the site pipeline",
|
|
"status": "completed",
|
|
"blockedBy": [
|
|
"5"
|
|
]
|
|
},
|
|
{
|
|
"id": "7",
|
|
"subject": "1.4 ScadaBridge: site health tests",
|
|
"status": "completed",
|
|
"blockedBy": [
|
|
"6"
|
|
]
|
|
},
|
|
{
|
|
"id": "8",
|
|
"subject": "2.1 OtOpcUa: verify `akka` check is the shared AkkaClusterHealthCheck",
|
|
"status": "completed",
|
|
"blockedBy": []
|
|
},
|
|
{
|
|
"id": "9",
|
|
"subject": "2.2 OtOpcUa: bump Health to 0.2.0",
|
|
"status": "completed",
|
|
"blockedBy": [
|
|
"3",
|
|
"8"
|
|
]
|
|
},
|
|
{
|
|
"id": "10",
|
|
"subject": "2.3 Alignment bumps: mxgw + HistorianGateway Health 0.2.0",
|
|
"status": "completed",
|
|
"blockedBy": [
|
|
"3"
|
|
]
|
|
},
|
|
{
|
|
"id": "11",
|
|
"subject": "3.1 Overview: scaffold project",
|
|
"status": "completed",
|
|
"blockedBy": [
|
|
"3"
|
|
]
|
|
},
|
|
{
|
|
"id": "12",
|
|
"subject": "3.2 Overview: registry options + validation",
|
|
"status": "completed",
|
|
"blockedBy": [
|
|
"11"
|
|
]
|
|
},
|
|
{
|
|
"id": "13",
|
|
"subject": "3.3 Overview: health client + status model",
|
|
"status": "completed",
|
|
"blockedBy": [
|
|
"11"
|
|
]
|
|
},
|
|
{
|
|
"id": "14",
|
|
"subject": "3.4 Overview: poller + snapshot store",
|
|
"status": "completed",
|
|
"blockedBy": [
|
|
"12",
|
|
"13"
|
|
]
|
|
},
|
|
{
|
|
"id": "15",
|
|
"subject": "3.5 Overview: leader aggregation",
|
|
"status": "completed",
|
|
"blockedBy": [
|
|
"13"
|
|
]
|
|
},
|
|
{
|
|
"id": "16",
|
|
"subject": "3.6 Overview: UI (ThemeShell, Overview page, InstanceCard)",
|
|
"status": "completed",
|
|
"blockedBy": [
|
|
"14",
|
|
"15"
|
|
]
|
|
},
|
|
{
|
|
"id": "17",
|
|
"subject": "3.7 Overview: self-observability",
|
|
"status": "completed",
|
|
"blockedBy": [
|
|
"14"
|
|
]
|
|
},
|
|
{
|
|
"id": "18",
|
|
"subject": "3.8 Overview: tests",
|
|
"status": "completed",
|
|
"blockedBy": [
|
|
"16",
|
|
"17"
|
|
]
|
|
},
|
|
{
|
|
"id": "19",
|
|
"subject": "3.9 Overview: config + docker",
|
|
"status": "completed",
|
|
"blockedBy": [
|
|
"17"
|
|
]
|
|
},
|
|
{
|
|
"id": "20",
|
|
"subject": "4 Live acceptance (design \u00a79)",
|
|
"status": "pending",
|
|
"blockedBy": [
|
|
"7",
|
|
"9",
|
|
"18",
|
|
"19"
|
|
]
|
|
}
|
|
],
|
|
"notes": {
|
|
"deferred": [
|
|
"Task 2.2 rig check (admin node /health/ready shows data.leader) \u2014 live docker-dev, folded into Phase 4.",
|
|
"Phase 1 DoD rig curls (site :8084 ready/active split) \u2014 live docker rig, folded into Phase 4."
|
|
],
|
|
"branches": {
|
|
"scadaproj": "feat/health-0.2.0-cluster-data (80668a0) -> feat/overview-dashboard, stacked",
|
|
"ScadaBridge": "feat/site-node-health (commit 47850c0f)",
|
|
"OtOpcUa": "feat/health-0.2.0-bump (commit 2e515c34, made in an isolated git worktree \u2014 feat/mesh-phase6 working tree untouched)",
|
|
"HistorianGateway": "chore/health-0.2.0-bump (commit 51f0c2b)",
|
|
"MxAccessGateway": "chore/health-0.2.0-bump (commit 2d54ace)"
|
|
},
|
|
"front_loaded": [
|
|
"Validator / health-client / status-derivation tests from task 3.8 were written with tasks 3.2-3.3 so each batch is verified; 3.8 covers the remainder (poller, leader, store, bUnit, boot).",
|
|
"Poller / snapshot-store / leader-resolver / bUnit InstanceCard + Overview page tests were written with tasks 3.4-3.6; task 3.8 now covers only the WebApplicationFactory boot tests."
|
|
],
|
|
"deviations": [
|
|
"Task 3.6 said 'no antiforgery'. AddRazorComponents stamps antiforgery metadata on every component endpoint unconditionally and the endpoint middleware hard-fails without a matching middleware, so every page returned 500. Resolved with app.UseAntiforgery() (inert here, no forms) rather than MapRazorComponents(...).DisableAntiforgery(), which would silently expose the first form anyone adds. No authentication was added.",
|
|
"Bootstrap IS vendored (copied from HistorianGateway, v5.3.3). Not optional: ThemeShell and TechButton are built on Bootstrap utility classes, so the shared kit does not render without it.",
|
|
{
|
|
"task": "3.7",
|
|
"what": "Metric names follow components/observability/spec/METRIC-CONVENTIONS.md (dot-separated overview.instance.status / overview.poll.duration, unit s) instead of the plan's literal zb_overview_instance_status.",
|
|
"why": "That spec is marked Standardized and authoritative for the family; the Prometheus exporter renders them as overview_instance_status and overview_poll_duration_seconds. Gauge labels are application/node/group \u2014 'node', not 'instance', because Prometheus stamps its own instance label on every scraped series and a colliding metric label is silently renamed exported_instance."
|
|
},
|
|
{
|
|
"task": "3.9",
|
|
"what": "Three registries, not one: appsettings.json ships an EMPTY Applications list, appsettings.Development.json a host-reachable subset (4 instances), appsettings.Docker.json the full 16-instance fleet.",
|
|
"why": "The dev rigs publish only some node ports to the host. OtOpcUa publishes none per-node (only Traefik :9200, load-balanced across the central pair, so it cannot identify a node) and ScadaBridge's site nodes keep :8084 container-internal. A host-run dashboard can genuinely only see part of the fleet; the containerised one joins the rig networks and sees all of it. Empty Applications in appsettings.json keeps a registry-less production deploy failing fast at ConfigPreflight instead of booting into an empty dashboard."
|
|
},
|
|
{
|
|
"task": "3.9",
|
|
"what": "appsettings.json declares real Serilog sinks and MinimumLevel overrides rather than the family's empty \"Serilog\": {}.",
|
|
"why": "AddZbSerilog drives sinks entirely from configuration and adds none of its own, so an empty section produces an app that logs NOWHERE \u2014 verified against the running HistorianGateway container, whose log stream is completely empty for exactly this reason (OtOpcUa, which populates the section, logs normally). The HttpClient override is needed too: 16 instances on a 10s cadence emit several Information lines a second of pure probe noise."
|
|
},
|
|
{
|
|
"task": "3.9",
|
|
"what": "Dockerfile clears ASPNETCORE_HTTP_PORTS as well as ASPNETCORE_URLS.",
|
|
"why": "On .NET 8+ the aspnet base image declares its binding via ASPNETCORE_HTTP_PORTS=8080, not ASPNETCORE_URLS. Clearing only URLS leaves Kestrel logging 'Overriding address(es) http://*:8080' on every boot \u2014 the URLs-override trap arriving by default rather than by mistake."
|
|
},
|
|
{
|
|
"task": "3.9",
|
|
"what": "No container healthcheck.",
|
|
"why": "aspnet:10.0 ships neither curl nor wget, and adding a --healthcheck mode to the app would put a second entry point into production code purely to satisfy Compose. The plan explicitly allows skipping."
|
|
}
|
|
],
|
|
"phase4_findings": [
|
|
"Rig ports verified by probing, not by reading compose (they disagree): OtOpcUa central-1/2 serve health on :9000 but site-* on :8080 (their explicit Kestrel listeners for LocalDb sync re-bind the primary HTTP port); ScadaBridge central on :5000, sites on :8084.",
|
|
"ScadaBridge site nodes currently answer :8084/health/ready with 404 \u2014 MapZbHealth on sites is on the feat/site-node-health branch and the rig runs main. Phase 4 needs the rigs redeployed from the 0.2.0 branches.",
|
|
"No leader chip renders on any group yet: the rigs run pre-0.2.0 builds whose akka check carries no data object. This is the designed graceful degradation, and it confirms Phase 4 must redeploy before checking design 9 item 2.",
|
|
"SUSPECTED RIG SPLIT-BRAIN: scadabridge-central-a AND central-b both return /health/active 200 with 'oldest Up member (singleton host)'. Two nodes cannot both be the oldest of one cluster. Verify during Phase 4 after redeploy \u2014 this is exactly the condition the dashboard exists to surface.",
|
|
"PRODUCT FIX during acceptance: Program.cs now calls builder.WebHost.UseStaticWebAssets() unconditionally. The host only calls it when the environment is literally 'Development', so running the build output under any other name (checking a staging registry locally) left the Theme RCL's _content assets 404 and the page rendered as a blank white screen with its markup fully present. No-op once published, so the container was never affected.",
|
|
"RIG DEFECT (pre-existing, both products): self-first Akka seed lists mean a pair booted cold self-forms two 1-member clusters that never join. Observed fleet-wide on ScadaBridge after a --force-recreate: every node named ITSELF leader and every node answered /health/active 200. Restarting ONE node of a pair makes it join the survivor and the pair converges to one Leader + one Active + one Standby. This is the user's own open backlog item (docs/plans/2026-07-22-initjoin-selfform-fallback.md, unexecuted). The dashboard surfaced it fleet-wide in one screen, which is what it is for.",
|
|
"Leader and Active legitimately diverge after a restart: Akka leader is lowest-address, ScadaBridge's active-node check is oldest Up member. site-c-a shows LEADER + STANDBY while site-c-b shows ACTIVE. Same divergence CLAUDE.md documents for OtOpcUa's Primary vs RoleLeader.",
|
|
"OUTSTANDING: the OtOpcUa docker-dev rig still runs a pre-0.2.0 build. Redeploying it needs a decision because the repo is checked out on feat/mesh-phase6, which carries 4 commits and an untracked Phase 7 plan beyond the base of feat/health-0.2.0-bump."
|
|
],
|
|
"phase4_results": {
|
|
"1_all_render_single_active_per_pair": "PASS (ScadaBridge, all 4 pairs: one Active + one Standby, verified live) / BLOCKED (OtOpcUa rig runs pre-0.2.0 build with self-formed 1-member meshes, so all 6 nodes report Active)",
|
|
"2_leader_chip_and_split_brain": "PASS (ScadaBridge). Leader chip moved from site-c-a to site-c-b when site-c-a was stopped, and the split-brain chip appeared while the pair was self-formed then cleared once it genuinely joined. BLOCKED for OtOpcUa (deployed build emits no data object, so no leader chip \u2014 the designed graceful degradation, confirmed)",
|
|
"3_kill_and_restart": "PASS. Stopped scadabridge-site-c-a 12:01:27; Unreachable by 12:02:33 (two-strike damping = 2 failing sweeps). Restarted 12:04:01; Up again by 12:04:16, i.e. on the first good poll (recovery is deliberately asymmetric)",
|
|
"4_degraded_with_failing_entry": "PASS. Canned Degraded and Unhealthy endpoints render as Degraded/Down with the failing entry and its description visible, and check-dot ok/warn/bad classes per check",
|
|
"5_cache_first": "PASS. Five consecutive page loads in 0.00-0.02s, all showing the identical 'last sweep 12:17:44' header \u2014 the render never triggers a poll (an inline sweep could not finish in 10ms with two unreachable targets on 3s/10s timeouts)",
|
|
"6_pause_and_stuck_poller": "PASS both halves. Pause -> both cards STALE within the 10s window, KPI stale=2, 'Stale since'; Resume -> clean recovery. Separately, SIGSTOPping the endpoints while auto-refresh stayed RUNNING froze the poller mid-sweep and the cards aged into STALE on their own \u2014 staleness is computed at render, so a stuck poller needs no explicit stall detection",
|
|
"7_management_links_no_login": "PASS. 'Open CentralUI' opened http://localhost:9001 in a new tab (target=_blank rel=noopener); the dashboard itself never presented or redirected to a login",
|
|
"8_malformed_registry": "PASS both branches. No registry -> ConfigPreflight names Overview:Applications:0:Name and :Instances:0:BaseUrl. Stale window inside the poll interval -> validator names the keys and the consequence",
|
|
"9_health_and_metrics": "PASS. /health/ready, /health/active, /healthz, /metrics all 200; 16 gauge series with application/node/group labels"
|
|
}
|
|
}
|
|
} |