diff --git a/docs/mockups/overview-dashboard-mockup.html b/docs/mockups/overview-dashboard-mockup.html new file mode 100644 index 0000000..7850b5a --- /dev/null +++ b/docs/mockups/overview-dashboard-mockup.html @@ -0,0 +1,554 @@ +ZB Overview — dashboard mockup + + +
+ + +
+
+

Fleet overview

+ last cycle 14:52:07 · 312 ms · every 10 s · stale after 45 s + + +
+ +
+
Instances
12registered
+
Up
8
+
Degraded
1
+
Down
1
+
Unreachable
1
+
Stale
1
+
+ + +
+
+

OtOpcUa

+ Degraded + + 4 instances · 2 clusters +
+ +
+
+ central pair + Leader · central-1 + 2 members · 0 unreachable +
+
+
+
+
central-1otopcua-central-1:9200
+
LeaderActiveUp
+
+
+
Latency18 ms
+
Last poll4 s ago
+
Checks4 / 4 healthy
+
+ Check detail +
configdbSQL Server reachable
+
akkaAkka cluster member status: Up
+
admin-leaderthis node holds the admin singletons
+
localdb-replicationpeer connected · backlog 0
+
leader akka.tcp://OtOpcUa@otopcua-central-1:4053 · members 2 · unreachable 0 · roles admin,driver
+
+
+
Open AdminUI ↗ready 200 · active 200
+
+ +
+
+
central-2otopcua-central-2:9200
+
StandbyUp
+
+
+
Latency21 ms
+
Last poll4 s ago
+
Checks4 / 4 healthy
+
+ Check detail +
configdbSQL Server reachable
+
akkaAkka cluster member status: Up
+
admin-leaderstandby — singletons on central-1
+
localdb-replicationpeer connected · backlog 0
+
leader akka.tcp://OtOpcUa@otopcua-central-1:4053 · members 2 · unreachable 0 · roles admin,driver
+
+
+
Open AdminUI ↗ready 200 · active 503
+
+
+
+ +
+
+ site-a pair + Leader · site-a-1 + 2 members · 0 unreachable +
+
+
+
+
site-a-1otopcua-site-a-1:9210
+
LeaderActiveUp
+
+
+
Latency33 ms
+
Last poll5 s ago
+
Checks3 / 3 healthy
+
+ Check detail +
akkaAkka cluster member status: Up
+
deployed-configartifact cache rev 41 · verified
+
localdb-replicationpeer connected · backlog 0
+
leader akka.tcp://OtOpcUa@otopcua-site-a-1:4053 · members 2 · unreachable 0 · roles driver
+
+
+
Open AdminUI ↗ready 200 · active 200
+
+ +
+
+
site-a-2otopcua-site-a-2:9210
+
StandbyDegraded
+
+
+
Latency29 ms
+
Last poll5 s ago
+
Checks2 / 3 healthy
+
+ Check detail +
akkaAkka cluster member status: Up
+
deployed-configartifact cache rev 41 · verified
+
localdb-replicationpeer unreachable for 38 s · backlog 12 ops
+
leader akka.tcp://OtOpcUa@otopcua-site-a-1:4053 · members 2 · unreachable 0 · roles driver
+
+
+
Open AdminUI ↗ready 200 · active 503
+
+
+
+
+ + +
+
+

ScadaBridge

+ Down + + 6 instances · 3 clusters +
+ +
+
+ central pair + Leader · central-a + 2 members · 0 unreachable +
+
+
+
+
central-asb-central-a:5000
+
LeaderActiveUp
+
+
+
Latency24 ms
+
Last poll3 s ago
+
Checks3 / 3 healthy
+
+ Check detail +
databaseScadaBridge config DB reachable
+
akka-clusterAkka cluster member status: Up
+
required-singletonsall central singletons started
+
leader akka.tcp://scadabridge@sb-central-a:8081 · members 2 · unreachable 0 · roles Central
+
+
+
Open CentralUI ↗ready 200 · active 200
+
+ +
+
+
central-bsb-central-b:5000
+
StandbyUp
+
+
+
Latency26 ms
+
Last poll3 s ago
+
Checks3 / 3 healthy
+
+ Check detail +
databaseScadaBridge config DB reachable
+
akka-clusterAkka cluster member status: Up
+
required-singletonsstandby — singletons on central-a
+
leader akka.tcp://scadabridge@sb-central-a:8081 · members 2 · unreachable 0 · roles Central
+
+
+
Open CentralUI ↗ready 200 · active 503
+
+
+
+ +
+
+ site-a pair + Leader · site-a-1 + 2 members · 1 unreachable +
+
+
+
+
site-a-1sb-site-a-1:8084
+
LeaderActiveUp
+
+
+
Latency31 ms
+
Last poll3 s ago
+
Checks3 / 3 healthy
+
+ Check detail +
akka-clusterAkka cluster member status: Up
+
localdbsite store reachable · replication backlog 0
+
active-nodeprimary of pair site-a
+
leader akka.tcp://scadabridge@sb-site-a-1:8082 · members 2 · unreachable 1 · roles Site,site-a
+
+
+
no management UI — site nodeready 200 · active 200
+
+ +
+
+
site-a-2sb-site-a-2:8084
+
Unreachable
+
+
+
Errorconnection refused
+
Failing since14:49:31 · 2 polls
+
Last good poll14:49:27
+
+
no management UI — site nodeprobe failed
+
+
+
+ +
+
+ site-b pair + Leader disagreement — possible split-brain + site-b-1 → site-b-1 · site-b-2 → site-b-2 +
+
+
+
+
site-b-1sb-site-b-1:8084
+
ActiveUp
+
+
+
Latency40 ms
+
Last poll3 s ago
+
Checks3 / 3 healthy
+
+ Check detail +
akka-clusterAkka cluster member status: Up
+
localdbsite store reachable
+
active-nodeprimary of pair site-b
+
leader akka.tcp://scadabridge@sb-site-b-1:8082 · members 1 · unreachable 0
+
+
+
no management UI — site nodeready 200 · active 200
+
+ +
+
+
site-b-2sb-site-b-2:8084
+
ActiveUp
+
+
+
Latency38 ms
+
Last poll3 s ago
+
Checks3 / 3 healthy
+
+ Check detail +
akka-clusterAkka cluster member status: Up
+
localdbsite store reachable
+
active-nodeprimary of pair site-b
+
leader akka.tcp://scadabridge@sb-site-b-2:8082 · members 1 · unreachable 0
+
+
+
no management UI — site nodeready 200 · active 200
+
+
+
+
+ + +
+
+

MxAccessGateway

+ Down + + 1 instance +
+
+
+
+
windevwindev:5130
+
Down
+
+
+
Latency57 ms
+
Failing since14:47:12 · 30 polls
+
Checks0 / 1 healthy
+
+ Check detail +
auth-storeSQLite key store unreachable: database is locked
+
+
+
Open Dashboard ↗ready 503
+
+
+
+ + +
+
+

HistorianGateway

+ Stale + + 1 instance +
+
+
+
+
wonder-app-vd03wonder-app-vd03:5222
+
Stale
+
+
+
Data age1 m 58 s — stale since 14:50:54
+
Last known stateUp (14:50:09)
+
Checks (last known)4 / 4 healthy
+
+ Last known check detail +
historian-connectionwonder-sql-vd03:32565 session pool healthy
+
galaxy-sqlGalaxy repository DB reachable
+
store-forward-drainoutbox empty
+
session-pool3 leased / 8 max
+
+
+
Open Dashboard ↗data older than 45 s threshold — VPN?
+
+
+
+ +

+ Mockup — static data. Registry: appsettings Overview:Applications · poll 10 s · timeout 3 s · stale 45 s. + Leader/member data from the akka-cluster check's data object (ZB.MOM.WW.Health 0.2.0). +

+
+
+ + diff --git a/docs/plans/2026-07-22-overview-dashboard-design.md b/docs/plans/2026-07-22-overview-dashboard-design.md new file mode 100644 index 0000000..0421896 --- /dev/null +++ b/docs/plans/2026-07-22-overview-dashboard-design.md @@ -0,0 +1,153 @@ +# ZB.MOM.WW.Overview — family overview dashboard, v1 design + +**Date:** 2026-07-22 +**Decision context:** `overview_dashboard_options.md` (same date) — Option A chosen: a custom in-house pane, matching the look and feel of the other scadaproj apps. Aspire standalone rejected (resource/status UI disabled outside an AppHost; no persistence; dev-tool positioning). +**Implementation plan:** `2026-07-22-overview-dashboard-impl-plan.md` (same directory) — phased, code-verified tasks incl. the sister-project changes. + +## 1. What it is + +A small Blazor Server app, hosted in scadaproj like the other shared components, that shows **every registered application instance on one page**: Up/Degraded/Down/Unreachable, Active/Standby where applicable, the **current Akka cluster leader** for each cluster group, per-check detail, and a deep link to each instance's own management UI. The page always renders instantly from a cached snapshot — a background poller on a **configurable timer** does all the HTTP work, and a **configurable staleness timeout** marks any data the poller hasn't refreshed as out-of-date/offline. **No login: the dashboard is anonymous and strictly read-only** (it only GETs endpoints that are already anonymous, and has no mutating actions). Configured entirely from an appsettings registry. **Every instance reports through the same channel:** the anonymous `/health/ready` + `/health/active` endpoints of `MapZbHealth`, parsing the canonical `ZbHealthWriter` JSON. Three of the four apps already expose these everywhere; the one gap — ScadaBridge *site* nodes — is closed by a small prerequisite change in ScadaBridge (§4) rather than worked around, so the dashboard has exactly one probe model. + +Non-goals for v1: history/uptime graphs, alerting/webhooks, telemetry viewing (OTLP/traces), and any push-based ingestion. If ops-grade monitoring is ever needed, that's the Grafana/Prometheus phase in the options doc — this registry ports straight to a scrape config. + +## 2. Project shape + +``` +ZB.MOM.WW.Overview/ # plain directory in scadaproj — NOT a nested git repo + ZB.MOM.WW.Overview.slnx + src/ZB.MOM.WW.Overview/ # single ASP.NET Core + Blazor Server project + Program.cs + Registry/ (options + validator) + Polling/ (poller BackgroundService, health client, snapshot store) + Components/ (App.razor, MainLayout → ThemeShell, Pages/Overview.razor, widgets) + tests/ZB.MOM.WW.Overview.Tests/ + docker/ # HistorianGateway-pattern runtime-only image +``` + +- .NET 10, global InteractiveServer render mode (same router style as ScadaBridge/HG/mxgw — no per-page render-mode traps). +- Package references (Gitea feed): `ZB.MOM.WW.Theme` 0.3.1, `ZB.MOM.WW.Health` (0.2.0 — see §4a), `ZB.MOM.WW.Telemetry(.Serilog)` 0.1.0, `ZB.MOM.WW.Configuration` 0.1.0. **No Auth, no Audit, no Secrets**: the dashboard is anonymous read-only by requirement — there is no login, no mutating action to audit, and nothing secret in the registry. (If a login is ever wanted later, `AddZbLdapAuth` + `LoginCard` drop in the family way.) +- It is an **app**, not a published library — nothing gets packed to the feed. + +## 3. Registry (appsettings) + +```jsonc +"Overview": { + "PollIntervalSeconds": 10, // global poll cadence (configurable timer) + "TimeoutSeconds": 3, // per-request timeout + "StaleAfterSeconds": 45, // snapshot older than this ⇒ instance rendered Stale (out of date/offline) + "Applications": [ + { + "Name": "OtOpcUa", + "ManagementLabel": "AdminUI", + "Instances": [ + { "Name": "central-1", "Group": "central", + "BaseUrl": "http://otopcua-central-1:9000", + "ManagementUrl": "http://otopcua-central-1:9000/", + "HasActiveRole": true }, + { "Name": "site-a-1", "Group": "site-a", "BaseUrl": "http://...", "HasActiveRole": true } + ] + }, + { + "Name": "ScadaBridge", + "Instances": [ + { "Name": "central-a", "Group": "central", "BaseUrl": "http://sb-central-a:5000", + "ManagementUrl": "http://sb-central-a:5000/", "HasActiveRole": true }, + { "Name": "site-a-1", "Group": "site-a", + "BaseUrl": "http://sb-site-a-1:8084", "HasActiveRole": true } // health on the HTTP/1.1 listener — see §4 + ] + }, + { + "Name": "MxAccessGateway", + "ManagementLabel": "Dashboard", + "Instances": [ + { "Name": "windev", "BaseUrl": "http://windev:5130", + "ManagementUrl": "http://windev:5130/" } // single-instance; no active/standby role + ] + }, + { + "Name": "HistorianGateway", + "ManagementLabel": "Dashboard", + "Instances": [ + { "Name": "wonder-app-vd03", "BaseUrl": "https://wonder-app-vd03:5222", + "ManagementUrl": "https://wonder-app-vd03:5222/" } // prod single TLS ALPN port; dev :5220 + ] + } + ] +} +``` + +All four family apps are first-class registry citizens; `HasActiveRole` is simply omitted for the two single-instance gateways (mxgw, HistorianGateway), which have no active/standby concept — their cards show status without the Active badge. + +- Every instance is probed the same way: GET `{BaseUrl}/health/ready`, plus `{BaseUrl}/health/active` when `HasActiveRole`. There is deliberately no per-instance probe-mode switch — one probe model, one status derivation. +- `PollIntervalSeconds`/`TimeoutSeconds`/`StaleAfterSeconds` are overridable per application and per instance (instance wins) for slow links or chatty pairs; `StaleAfterSeconds` must exceed `PollIntervalSeconds` (validated). +- Bound + validated with `AddValidatedOptions` (`ZB.MOM.WW.Configuration`): non-empty registry, absolute http(s) URLs, unique app/instance names, positive intervals, stale > poll. Malformed registry fails boot via `ConfigPreflight`, not the first poll. + +## 4. Prerequisite target-app changes (health from ALL apps) + +The dashboard's contract is "every registered instance serves `MapZbHealth`". Current reality and the deltas: + +| App / node kind | Today | Delta needed | +|---|---|---| +| OtOpcUa admin + driver nodes | `MapZbHealth` mapped unconditionally | None in code. **Deployment note:** a driver-only Windows-service node with no explicit URLs falls back to Kestrel's `localhost:5000` — every registered node must be given a reachable binding (`ASPNETCORE_URLS`/`HTTP_PORTS`), which the docker rig already does. | +| ScadaBridge **Central** | `MapZbHealth` (ready: database/akka-cluster/required-singletons; active: active-node) | None. | +| ScadaBridge **Site** nodes | **No health endpoints** — only the gRPC `:8083` (HTTP/2-only) + `/metrics` on the `:8084` HTTP/1.1 listener | **Small ScadaBridge PR (the one prerequisite):** map `MapZbHealth` on the `:8084` HTTP/1.1 listener alongside `/metrics`. Ready checks: `akka-cluster` (site pair) + site SQLite store (+ `localdb-replication` where enabled); active check: the site pair's oldest-Up active-node rule via `ActiveNodeHealthCheck` — the Host already references all three `ZB.MOM.WW.Health` packages, so this is pure wiring, no new dependencies. Gives site rows full Up/Degraded detail **and** Active/Standby, same as everything else. | +| MxAccessGateway | `MapZbHealth` (ready: auth-store) | None. | +| HistorianGateway | `MapZbHealth` (ready: historian-connection, galaxy-sql, conditional runtime/store-forward/pool checks) | None. | + +Sequencing: the ScadaBridge PR can land any time before the dashboard's acceptance run; the dashboard itself needs no knowledge of it beyond registry entries. + +### 4a. Shared-lib prerequisite: `ZB.MOM.WW.Health` 0.2.0 — cluster-leader data + +Requirement: the dashboard must show the **current leader** of each Akka cluster. The canonical health JSON today carries only `status`/`description`/`durationMs` per entry — no structured data — so this needs a small **additive** change to the shared lib (which is exactly where the family wants such things to live): + +- **Core (`ZbHealthWriter`):** emit an optional `"data": { ... }` object per entry, sourced from `HealthReportEntry.Data`, **only when non-empty** — existing payloads stay byte-identical, so this is non-breaking for every current consumer. +- **`ZB.MOM.WW.Health.Akka` (`AkkaClusterHealthCheck`):** populate the check's data dictionary from local cluster state: `leader` (address), `selfAddress`, `selfRoles`, `memberCount`, `unreachableCount` (+ `roleLeader:` where the app cares). Each node reports *its own view*; the dashboard reads the leader from any Up node of the group and flags disagreement (a visible split-brain tell — a nice free diagnostic for the 2-node pairs). +- **Version/rollout:** bump lib to `0.2.0`, publish the 3 packages to the Gitea feed. **Required bumps:** OtOpcUa + ScadaBridge (the Akka apps — leader data appears with the package bump alone, no app code changes, since both register the shared `AkkaClusterHealthCheck`). **Recommended alignment bumps:** mxgw + HistorianGateway (no behavior change for them; keeps the family version matrix aligned). + +## 5. Polling and status model + +One `OverviewPollerService : BackgroundService`: + +- Every cadence tick, polls **all instances in parallel** (`Task.WhenAll`, per-instance `CancellationTokenSource` timeout; named `HttpClient` via `IHttpClientFactory`, no retries — the next tick is the retry). +- Parses the canonical shape `{ status, totalDurationMs, entries: { name: { status, description, durationMs } } }`. Parsing lives in a small `ZbHealthReportClient` inside this app for v1; promoting it into a `ZB.MOM.WW.Health.Client` package is a deliberate later step, not a v1 dependency (recommendation: promote only when a second consumer appears). +- Derived per-instance state: + - **Up** — ready 200 + `"Healthy"`; **Degraded** — 200 + `"Degraded"`; **Down** — 503 / `"Unhealthy"`; **Unreachable** — timeout, refused, DNS, non-JSON (kept distinct from Down: it usually means network/VPN/registry-typo, not the app). + - **Active/Standby** — `/health/active` 200 → Active, 503 → Standby, error → Unknown (only when `HasActiveRole`). + - Plus: last poll UTC, latency ms, per-check entries, consecutive-failure count (state flips to Down/Unreachable only after **2** consecutive failures to avoid single-blip flapping; recovery is immediate). +- **Cache-first rendering:** results land in an in-memory `OverviewSnapshotStore` (single immutable snapshot swapped atomically); an event notifies circuits to re-render. Page loads NEVER trigger a poll — they read the cached snapshot and render instantly; the poller timer is the only thing that does HTTP. No persistence — a restart repolls the world within one cadence. +- **Staleness:** every instance snapshot carries `LastUpdatedUtc`. When the UI renders (or a periodic sweep runs), any instance whose snapshot age exceeds its effective `StaleAfterSeconds` is shown as **Stale (out of date/offline)** regardless of its last known status — this catches a stuck poller, a paused refresh, or a machine-slept dashboard, not just failed probes. +- **Leader extraction:** for each `Group` (one Akka cluster = one group, e.g. `OtOpcUa/central`, `ScadaBridge/site-a`), the aggregator reads the `akka-cluster` entry's `data.leader` from each Up member and surfaces the group's leader (resolving the leader address back to a registry instance name where possible). If Up members disagree on the leader, the group is flagged (split-brain indicator). +- StatusPill mapping: Up→`Ok`, Degraded→`Warn`, Down→`Bad`, Unreachable→`Bad` (distinct label), Stale→`Idle`, Active/Leader badges→`Info`. + +## 6. UI + +> **Mockup:** [`docs/mockups/overview-dashboard-mockup.html`](../mockups/overview-dashboard-mockup.html) (static, self-contained; real Theme tokens + embedded IBM Plex) — the visual reference for `InstanceCard`, group headers, leader/split-brain chips, and every status state incl. Stale. Open it in a browser; the implementation should match it. + +Single page (`/`), `ThemeShell` chassis with its own product name/accent so it reads as a sibling of the other four apps: + +- One section per application (name + rollup pill = worst instance status), instances as a grid of cards (`TechCard` + `StatusPill` composition — a new `InstanceCard` component, since Theme has no prebuilt tile widget). Cluster groups render a group header with the **Leader: ** chip (and the split-brain warning when member views disagree). +- Card: instance name + group, status pill, Active badge, **Leader badge** on the leader node's card, latency, relative last-seen (turns into an explicit "stale since …" once past `StaleAfterSeconds`); expandable check list (`entries` with per-check pill + description); a `TechButton` "Open " → `ManagementUrl` (new tab). +- A header row: total counts (n Up / n Degraded / n Down / n Unreachable / n Stale), last-refresh clock, pause/resume auto-refresh (paused long enough, cards go Stale by design). +- **No login.** The dashboard is anonymous and read-only end to end; there is nothing to authorize and nothing it can change. It still matches the sister apps' look exactly (ThemeShell + tokens + IBM Plex), just without the login gate. + +## 7. Self-observability and hosting + +- The dashboard eats the family dog food: `MapZbHealth` (ready check: "registry loaded" + "last poll cycle completed"), `MapZbMetrics`, `AddZbTelemetry`/`AddZbSerilog`; a `zb_overview_instance_status` gauge (per instance, labeled) so even v1 status is scrapeable — and becomes the Grafana bridge later. +- Ports: dev `http://localhost:5320` (clear of every family port in use: 5000/5001, 5120/7121, 5220–5222, 8081–8085, 9000, 9200, 4053, 4840); production single TLS ALPN endpoint like HistorianGateway, warn-only if terminated upstream. +- `docker/` mirrors the HistorianGateway pattern: publish framework-dependent on the host (authed Gitea feed), COPY into `aspnet:10.0`, gitignored env_file. Natural first deployment: the docker-dev rig host, registry pointed at the existing OtOpcUa + ScadaBridge rig nodes — that is also the acceptance test. + +## 8. Open decisions (settled recommendations) + +1. **ScadaBridge site rows** — resolved (2026-07-22, user direction): site nodes gain `MapZbHealth` as the one prerequisite PR (§4), so every row is a full health row. *Chosen over* a metrics-liveness fallback (two probe models) and over reading Central's site aggregator (couples the dashboard to ScadaBridge internals and its transport). +2. **Health-client ownership** — in-app for v1; promote to `ZB.MOM.WW.Health.Client` on second consumer. *(Best practice: contract-owning lib should host the client, but a package bump + publish for one consumer is ceremony without benefit.)* +3. **History/alerting** — explicitly out of v1; the `zb_overview_instance_status` gauge is the forward hook (Prometheus can scrape the dashboard alone and get fleet status history for free). +4. **Leader transport** — resolved (2026-07-22, user requirement): via a `data` object in the canonical health JSON (`ZB.MOM.WW.Health` 0.2.0, §4a), populated by the shared `AkkaClusterHealthCheck`. *Chosen over* description-string parsing (fragile) and a separate leader endpoint (second contract for one field). +5. **Auth** — resolved (2026-07-22, user requirement): none. Anonymous, read-only. + +## 9. Acceptance (v1 definition of done) + +- The §4 ScadaBridge site-node health PR is merged and site rows render with full check detail + Active/Standby (no special-cased rows anywhere). +- Registry pointed at the live docker-dev rigs (OtOpcUa 6-node + ScadaBridge local cluster) renders every instance correctly, including: each Active/Standby pair (central **and** site pairs) showing exactly one Active; **each cluster group showing its current leader, and the leader chip moving when the leader node is stopped**; killing a node flips its card to Unreachable within ~2 cadences and back on restart; a Degraded check (e.g. stop a dependency) shows Warn with the failing entry visible; management links open the right app UIs; the page is reachable with no login. +- **Cache/staleness proven live:** first page load renders instantly from cache (no poll on request); with auto-refresh paused (or the poller stopped) past `StaleAfterSeconds`, every card flips to Stale, and resumes cleanly. +- Boot fails with a clear `ConfigPreflight` message on a malformed registry. +- `dotnet test` green: options validator, status derivation (incl. flap damping, staleness marking, leader extraction + disagreement flag, JSON parsing golden tests against captured real payloads from all four apps — with and without `data`), snapshot store, and a bUnit render test of `InstanceCard`. diff --git a/docs/plans/2026-07-22-overview-dashboard-impl-plan.md b/docs/plans/2026-07-22-overview-dashboard-impl-plan.md new file mode 100644 index 0000000..09410ba --- /dev/null +++ b/docs/plans/2026-07-22-overview-dashboard-impl-plan.md @@ -0,0 +1,171 @@ +# ZB.MOM.WW.Overview — implementation plan + +**Date:** 2026-07-22 +**Design:** `2026-07-22-overview-dashboard-design.md` (same directory). Read it first; this plan assumes its decisions (custom app, family look/feel, anonymous read-only, cache-first polling, staleness timeout, Akka leader display). +**Repos touched:** `scadaproj/ZB.MOM.WW.Health` (shared lib 0.2.0), `~/Desktop/ScadaBridge` (site-node health PR), `~/Desktop/OtOpcUa` (Health bump + check verification), optional alignment bumps in `~/Desktop/MxAccessGateway` + `~/Desktop/HistorianGateway`, and the new `scadaproj/ZB.MOM.WW.Overview/`. +**Sequencing:** Phase 0 → Phases 1 + 2 (parallel, independent repos) → Phase 3 (app; can start in parallel once 0.2.0 is on the feed) → Phase 4 (live acceptance). Phases 1/2 only gate Phase 4, not Phase 3 development. + +All facts below were code-verified 2026-07-22 (file:line cites are from that pass); re-check a cite if the target repo has moved since. + +**How to run this in a fresh session:** read the design doc first, then execute phases in order (0 → 1‖2 → 3 → 4), one phase per sitting with its DoD met before moving on. Work in each repo on a feature branch (`feat/health-0.2.0-cluster-data` in scadaproj, `feat/site-node-health` in ScadaBridge, `feat/health-0.2.0-bump` in OtOpcUa); commit scadaproj work (lib + new app + these docs) on `main` per family habit only when asked. Nothing in this plan requires user input except merges/pushes and the Phase 4 rig runs; if a cited line has drifted, re-locate by the quoted identifier, not the line number. + +--- + +## Phase 0 — `ZB.MOM.WW.Health` 0.2.0: entry `data` + Akka cluster-view + +Repo: `scadaproj/ZB.MOM.WW.Health/` (plain directory — do NOT `git init`; commits happen in scadaproj). Akka.Cluster pin 1.5.62 (`Directory.Packages.props:9`); version is centralized at `Directory.Build.props:8` (`0.1.0`). + +### Task 0.1 — writer: optional per-entry `data` object +`src/ZB.MOM.WW.Health/ZbHealthWriter.cs`: +- Add to the private `HealthEntryDto` (lines ~76-81): + `[JsonIgnore(Condition = JsonIgnoreCondition.WhenWritingNull)] public IReadOnlyDictionary? Data { get; init; }` +- In the projection (line ~62): `Data = e.Value.Data.Count > 0 ? e.Value.Data : null` (`HealthReportEntry.Data` is never null; empty when unset). +- **Invariant: old payloads stay byte-identical.** The per-property `JsonIgnore` is essential — the global serializer options deliberately do NOT ignore nulls (the null-`description` behavior is pinned by `ResponseWriterTests.cs:69-85`); do not change `SerializerOptions`. + +Tests (`tests/ZB.MOM.WW.Health.Tests/ResponseWriterTests.cs`): extend `StubHealthCheck` (lines 21-31) to accept an optional data dictionary; add (a) non-empty data → `"data": {...}` emitted with camelCase envelope keys but **verbatim data keys**, (b) empty data → no `data` key at all (assert on raw JSON string), (c) existing null-description test still green. + +### Task 0.2 — `AkkaClusterHealthCheck`: populate cluster-view data +`src/ZB.MOM.WW.Health.Akka/AkkaClusterHealthCheck.cs` (currently reads only `SelfMember.Status`, returns description-only — lines 38-70): +- Add an internal static helper `BuildClusterData(Cluster cluster)` returning `IReadOnlyDictionary` with JSON-friendly values only (string/int/string[]): + - `leader` — `cluster.State.Leader?.ToString()` (omit the key when null: pre-join/unknown), + - `selfAddress` — `cluster.SelfAddress.ToString()`, + - `selfRoles` — `cluster.SelfMember.Roles.OrderBy(r => r).ToArray()`, + - `memberCount` — `cluster.State.Members.Count`, + - `unreachableCount` — `cluster.State.Unreachable.Count`. + All property names confirmed compiling in this repo (`ActiveNodeHealthCheck.cs:116-132`, `AkkaActiveNodeGate.cs:33-49`). +- Pass it through every result path: `HealthCheckResult.Healthy/Degraded/Unhealthy(description, data)`. The no-ActorSystem / cluster-inaccessible paths keep returning description-only (no data) — pinned by `AkkaClusterStatusPolicyTests.cs:63-96`. +- ASP.NET Core copies `HealthCheckResult.Data` → `HealthReportEntry.Data` automatically, and `MapZbHealth` already wires `ZbHealthWriter` (`ZbHealthEndpointExtensions.cs:42-54`) — **no consumer code change needed beyond the package bump**, provided the app registers this shared check (verified for ScadaBridge; Task 2.1 verifies OtOpcUa). + +Tests (`tests/ZB.MOM.WW.Health.Akka.Tests/`): the `InternalsVisibleTo` hook (`ActiveNodeHealthCheck.cs:7`) exposes the helper. Use `Akka.TestKit.Xunit2` (pin 1.5.62; conventions per existing tests — inherit `TestKit`, never hand-roll `ActorSystem.Create` + manual join) with a single-node self-joined cluster: assert `leader` == self address, `memberCount` == 1, `selfRoles` round-trip; plus the check-level test that a healthy result now carries data and the degraded no-system path carries none. + +### Task 0.3 — version, pack, publish +- `Directory.Build.props`: `0.1.0` → `0.2.0`. Note the change in the repo's CHANGELOG/README if present. +- `dotnet test` from the repo root (baseline 58 tests across 3 test projects — all green plus the new ones), then `dotnet pack -c Release -o artifacts` (3 nupkgs @ 0.2.0). +- Publish: `dotnet nuget push artifacts/ZB.MOM.WW.Health*.0.2.0.nupkg --source https://gitea.dohertylan.com/api/packages/dohertj2/nuget/index.json --api-key ` (one command per nupkg or glob). +- Verify like every prior lib release (the family gotcha: "published" claims must be proven): `curl -s https://gitea.dohertylan.com/api/packages/dohertj2/nuget/registration/zb.mom.ww.health/index.json` shows 0.2.0 (NOT `/nuget/v3/registration/` — that 404s for everything), then restore-verify from a scratch consumer project that pins 0.2.0. + +**Phase DoD:** suite green; 3 packages at 0.2.0 restore-verified from the feed; a manual smoke (any sample host + the check) shows `entries["akka-cluster"].data.leader` in `/health/ready`. + +--- + +## Phase 1 — ScadaBridge PR: site-node health endpoints + +Repo: `~/Desktop/ScadaBridge` (branch off `main`, e.g. `feat/site-node-health`). Site branch of `src/ZB.MOM.WW.ScadaBridge.Host/Program.cs` is lines 485-570: Kestrel `:8083` gRPC Http2-only (507-512), `:8084` `Http1AndHttp2` (516-518); site DI via `SiteServiceRegistration.Configure` (:532); site pipeline maps only `MapZbMetrics` (:545), gRPC (:548), `MapZbLocalDbSync` (:555). No auth middleware and no FallbackPolicy exist on the site pipeline, so anonymous health needs nothing special. + +### Task 1.1 — bump Health packages +`Directory.Packages.props:93-95`: all three `ZB.MOM.WW.Health*` 0.1.0 → **0.2.0**. + +### Task 1.2 — site health checks (register between `:532` and `var app = builder.Build()` at `:534`) +Mirror the central registration style (`Program.cs:277-304`, `AddTypeActivatedCheck`): +1. **`akka-cluster`** (tags: Ready) — the shared `AkkaClusterHealthCheck` with `AkkaClusterStatusPolicy.Default`, exactly as central does at 282-286. The site ActorSystem→DI bridge already exists (`SiteServiceRegistration.cs:168-169`). This is also what carries `data.leader` for site pairs. +2. **`localdb`** (tags: Ready) — **new** check class in the Host (site namespace): opens the consolidated site SQLite (`LocalDb:Path`) and runs `SELECT 1`. There is no EF context on site (central's `DatabaseHealthCheck` does NOT apply — that context is central-only, `Program.cs:266`). Optionally enrich `data` with `ISyncStatus.Connected`/`OplogBacklog` (`SiteServiceRegistration.cs:139-140`) when replication is enabled — enrich only; replication-off must NOT fail the check (replication is default-OFF by design). +3. **`active-node`** (tags: Active) — **new** thin `SitePairActiveNodeHealthCheck` delegating to `IClusterNodeProvider.SelfIsPrimary` (registered on site, `SiteServiceRegistration.cs:172-178`; role-scoped `site-{SiteId}` oldest-Up). **Do NOT reuse central's `OldestNodeActiveHealthCheck`** — it calls `SelfIsOldest(cluster)` with no role scope (`OldestNodeActiveHealthCheck.cs:30`) and would compute oldest across the wrong member set. Follow the existing delegation precedent `SiteEventLogActiveNodeCheck` (`SiteServiceRegistration.cs:187-191`). Active semantics: primary → Healthy, standby → Unhealthy (503 = Standby to the dashboard — same convention as central). + +### Task 1.3 — map endpoints +`app.MapZbHealth();` in the site pipeline after `UseRouting()` (:539), alongside `MapZbMetrics()` (:545). Served on `:8084` (Http1AndHttp2). No auth changes. + +### Task 1.4 — tests +Extend `tests/ZB.MOM.WW.ScadaBridge.Host.Tests/`: +- Imitate `HealthCheckTests.cs` (`WebApplicationFactory`, `[Collection(HostBootCollection.Name)]`, `UseSetting("ScadaBridge:Node:Role", ...)`, `SkipMigrations=true`, registration assertions via `IOptions.Value.Registrations` + `registration.Factory(services)` type checks). **Caveat found in recon:** every existing factory test boots the Central role; there is no site-role factory bootstrap yet. Build one (Role=Site + minimal `ScadaBridge:Node` config + temp `LocalDb:Path`); if the full site host proves too heavy to boot in-test, fall back to the `CompositionRootTests.cs`/`SiteLocalDbWiringTests.cs` style over `SiteServiceRegistration.Configure` for registrations, plus a factory test for just the endpoint mapping. Include a positive control for any "central-only check absent on site" assertion. +- Assert: site maps `/health/ready` + `/health/active` (non-404); registrations exactly {akka-cluster→Ready, localdb→Ready, active-node→Active} with the intended check types; central registrations unchanged. + +**Phase DoD:** full ScadaBridge suite green; on the local docker rig, `curl http://:8084/health/ready` returns the canonical JSON with `akka-cluster` carrying `data.leader`, and `/health/active` splits 200/503 across the pair; central `/health/ready` also shows `data.leader` (free via the bump). PR merged to `main`. + +--- + +## Phase 2 — OtOpcUa bump (+ optional family alignment) + +### Task 2.1 — verify the `akka` check is the shared class +OtOpcUa registers ready checks `configdb`, `akka`, `admin-leader`, `localdb-replication` (registration reachable from `MapOtOpcUaHealth` / Program wiring, `src/Server/ZB.MOM.WW.OtOpcUa.Host`). **Verify** the `akka` check is `ZB.MOM.WW.Health.Akka.AkkaClusterHealthCheck`. If it is → nothing more. If bespoke → either swap to the shared check (preferred, same policy semantics) or populate the same `data` keys (`leader`/`selfAddress`/`selfRoles`/`memberCount`/`unreachableCount`) so the dashboard sees one contract. + +### Task 2.2 — bump +`~/Desktop/OtOpcUa/Directory.Packages.props`: `ZB.MOM.WW.Health*` → 0.2.0. Build + run the Host/AdminUI test suites touched by health. Rig check: any admin node `/health/ready` shows `data.leader`. + +### Task 2.3 (recommended) — alignment bumps, mxgw + HistorianGateway +Neither uses CPM (**no `Directory.Packages.props` — the known trap**): bump the inline `ZB.MOM.WW.Health` pins in `MxGateway.Server.csproj` and `ZB.MOM.WW.HistorianGateway.Server.csproj` to 0.2.0. No behavior change (no Akka checks there); keeps the family version matrix aligned. Skippable if either repo has unrelated in-flight work. + +**Phase DoD:** OtOpcUa on 0.2.0 with leader data live on the rig; alignment bumps merged or explicitly deferred with a note. + +--- + +## Phase 3 — the `ZB.MOM.WW.Overview` app + +Location: `scadaproj/ZB.MOM.WW.Overview/` — **plain directory, NOT a nested git repo** (family rule). App conventions (copied from HistorianGateway, the reference implementation — cites from recon): + +### Task 3.1 — scaffold +- `ZB.MOM.WW.Overview.slnx` with `/src/` + `/tests/` folders (LocalDb's slnx is the minimal template). +- `src/ZB.MOM.WW.Overview/ZB.MOM.WW.Overview.csproj`: `Sdk="Microsoft.NET.Sdk.Web"`, net10.0, **inline pinned versions, no CPM** (the app convention): `ZB.MOM.WW.Theme` 0.3.1, `ZB.MOM.WW.Health` 0.2.0, `ZB.MOM.WW.Telemetry` + `.Serilog` 0.1.0, `ZB.MOM.WW.Configuration` 0.1.0, `Serilog.AspNetCore` + console/file sinks. NO Auth/Audit/Secrets packages. +- `Directory.Build.props` with the family zero-warnings gate (`TreatWarningsAsErrors`, `AnalysisLevel=latest`, `EnforceCodeStyleInBuild`, `Deterministic`, Nullable/ImplicitUsings) — copy HistorianGateway's. +- `nuget.config`: copy HistorianGateway's (nuget.org `*` + `dohertj2-gitea` source with `ZB.MOM.WW.*` packageSourceMapping; credentials stay user-level, never committed). + +### Task 3.2 — registry options + validation +`Registry/OverviewOptions.cs` (+ `ApplicationEntry`, `InstanceEntry`) binding section `Overview` per the design §3 schema: global `PollIntervalSeconds`(10)/`TimeoutSeconds`(3)/`StaleAfterSeconds`(45); per-application and per-instance overrides (instance wins); instance fields `Name`, `Group?`, `BaseUrl`, `ManagementUrl?`, `HasActiveRole`(false). +`Registry/OverviewOptionsValidator.cs` via `OptionsValidatorBase` + `AddValidatedOptions` (`ZB.MOM.WW.Configuration`): non-empty applications, ≥1 instance each, unique app names + unique instance names within an app, absolute http/https `BaseUrl`/`ManagementUrl`, all intervals positive, **effective stale > effective poll for every instance**. Plus `ConfigPreflight.For(builder.Configuration).RequireValue("Overview:Applications:0:Name")…`-style presence check pre-`Build()` (HistorianGateway `Program.cs:62-66` is the pattern). + +### Task 3.3 — health client + status model +`Polling/ZbHealthReportClient.cs`: GETs a URL, deserializes the canonical shape `{status, totalDurationMs, entries:{name:{status, description, durationMs, data?}}}` — `data` values as `JsonElement` (they arrive as arbitrary JSON; convert to string/int on read). Treat both 200 and 503 as *parseable* responses (503 carries a body too); classify non-JSON/undeserializable as Unreachable. In-app for v1 (design §8.2). +`Polling/InstanceStatus.cs`: `Up | Degraded | Down | Unreachable` (+ separate `ActiveState: Active|Standby|Unknown|NotApplicable`, `IsStale` computed at render). Derivation per design §5, including two-strike flap damping (state degrades only after 2 consecutive failures; recovery immediate). + +### Task 3.4 — poller + snapshot store +`Polling/OverviewPollerService.cs : BackgroundService`: `PeriodicTimer(PollIntervalSeconds)`; each tick polls all instances in parallel (`Task.WhenAll`; per-instance `CancellationTokenSource` with its effective timeout; named `HttpClient` via `IHttpClientFactory`; no retries — next tick is the retry). Per instance: `/health/ready` always; `/health/active` when `HasActiveRole`. +`Polling/OverviewSnapshotStore.cs`: single immutable snapshot object swapped atomically (`Volatile.Write`/`Interlocked.Exchange`); each instance snapshot carries `LastUpdatedUtc`, latency, parsed entries, derived status, consecutive-failure counter. A C# `event`/`Action` notifies circuits to re-render on swap. **Pages never poll** — they read the store only (cache-first requirement). +Staleness: computed at render (`now - LastUpdatedUtc > effective StaleAfterSeconds` ⇒ render Stale regardless of stored status); the page also runs a small UI tick (~5 s) so ages/staleness advance without a poll event. + +### Task 3.5 — leader aggregation +`Polling/LeaderResolver.cs`: for each `(Application, Group)` with any `HasActiveRole` instance, collect `entries["akka-cluster"].data.leader` from members whose status is Up/Degraded; resolve the Akka address (`akka.tcp://@host:port`) to a registry instance by **host match against `BaseUrl`** (fall back to showing the raw address when unmatched — different ports are expected, the Akka port is not the HTTP port); if reporting members disagree → set the group's `LeaderDisagreement` flag (split-brain tell). Pure static logic → heavily unit-testable. + +### Task 3.6 — UI +**Visual reference (match it):** `docs/mockups/overview-dashboard-mockup.html` — a static, self-contained mockup built from the real Theme tokens + embedded IBM Plex; open it in a browser before styling. It defines: the KPI strip treatment (alert/caution tints), the instance-card anatomy (status stripe on the top edge; **dashed border = Unreachable, solid = Down**; faded body + "stale since" for Stale), group headers with `Leader · ` chips and the split-brain warning chip, footer `ready/active` raw-signal line, and the check-detail `
` layout with the monospace `data` line. Reuse Theme classes where they exist (`chip-*`, `panel`, `kv`, `agg-*`); the mockup's extra classes (`inst*`, `group*`, `check-*`) go into the app's own `site.css`. + +Copy HistorianGateway's Blazor skeleton, minus every auth piece: +- `Components/App.razor` (head order matters: Bootstrap CSS → `` → `site.css` → `HeadOutlet @rendermode=InteractiveServer`; body: `` → Bootstrap JS → `` → `blazor.web.js`), `Routes.razor` with plain `RouteView` (no `AuthorizeRouteView`), `_Imports.razor` incl. `@using ZB.MOM.WW.Theme`. +- `Components/Layout/MainLayout.razor` → `ThemeShell Product="SCADA Overview" Accent="…"` (pick an unused accent; HG is `#2f855a`), Nav = single `NavRailItem Href="/" Match="NavLinkMatch.All"` (+ a Health link later if wanted), no AuthorizeView anywhere. +- `Components/Pages/Overview.razor` (`@page "/"`): header strip (counts n Up/Degraded/Down/Unreachable/Stale, last-refresh clock, pause/resume toggle — pause just stops applying poll events + lets staleness paint), then per-application sections: app name + worst-status rollup pill; cluster groups get a sub-header with `Leader: ` chip (`StatusPill Info`) and the disagreement warning chip when flagged; instance grid of `InstanceCard`s. +- `Components/Widgets/InstanceCard.razor`: `TechCard` + `StatusPill` composition — name/group, status pill (mapping per design §5), Active badge, Leader badge, latency, relative last-seen ("stale since …" once stale), native `
` check list (per-check pill + description + data leader line for akka-cluster), `TechButton` (Ghost/Secondary) → `ManagementUrl` (`target="_blank" rel="noopener"`). +- Program wiring: `AddRazorComponents().AddInteractiveServerComponents()`; `MapRazorComponents().AddInteractiveServerRenderMode()` — **no** `RequireAuthorization`, no antiforgery/cascading auth state; `UseStaticFiles()` before endpoints (Theme assets come from `_content/ZB.MOM.WW.Theme/`). + +### Task 3.7 — self-observability +`AddZbSerilog(o => o.ServiceName = "overview")`; `AddZbTelemetry(o => { o.ServiceName = "overview"; o.Meters = [OverviewMetrics.MeterName]; })` — **`Meters` is a silent allowlist: forgetting the custom meter name silently drops the gauge** (known family gotcha). `OverviewMetrics`: observable gauge `zb_overview_instance_status` (labels app/instance/group; value = status enum ordinal) + poll-duration histogram. Register own checks: `registry-loaded` and `last-poll-cycle-completed` (Unhealthy if no completed cycle within 2× poll interval — makes the dashboard's own `/health/ready` honest). `MapZbHealth()` + `MapZbMetrics()`. + +### Task 3.8 — tests (`tests/ZB.MOM.WW.Overview.Tests`) +xunit 2.9.3 + `xunit.runner.visualstudio` 3.1.4 + `Microsoft.NET.Test.Sdk` 17.14.1 + **bunit 1.40.0** (Theme's own test stack — imitate `StatusPillTests.cs`: `class X : TestContext`, `RenderComponent`, class-list asserts): +- Validator table tests (incl. stale ≤ poll rejected, relative URL rejected). +- Health-client golden tests against **captured real payloads** from all four apps (capture during Phase 4 prep; until then use handcrafted fixtures matching `ZbHealthWriter` incl. entries with and without `data`). +- Status derivation: 200/Healthy→Up, 200/Degraded→Degraded, 503→Down, timeout/refused/garbage→Unreachable, flap damping (1 failure holds, 2 flips, recovery immediate), active 200/503/error → Active/Standby/Unknown. +- Staleness: fresh vs. aged snapshot with per-instance override. +- LeaderResolver: agreement, disagreement flag, address→instance host matching, unmatched fallback. +- Snapshot store: atomic swap + event fired. +- bUnit: `InstanceCard` renders pill class/badges/link per state; Overview page smoke over a stubbed store. +- `WebApplicationFactory` boot tests: page + `/health/ready` + `/metrics` reachable **with no auth**; malformed registry → boot fails with the `ConfigPreflight`/validator message. + +### Task 3.9 — config + docker +- `appsettings.json`: Logging skeleton, empty `Serilog`/`Telemetry` sections (libs default them), `Overview` registry sample; `appsettings.Development.json`: Kestrel `Http` endpoint `http://localhost:5320` + a dev registry pointing at the docker rigs. **Derive the rig instance URLs from the compose files, don't guess:** OtOpcUa docker-dev rig → `~/Desktop/OtOpcUa` docker-dev compose (6 nodes; AdminUI reachable on host :9200 — per-node published HTTP ports are in the compose port mappings); ScadaBridge local cluster → `~/Desktop/ScadaBridge/docker/` compose (central pair UI ports + site-node :8084 mappings); mxgw → `windev:5130`; HistorianGateway → its docker compose (:5220) or `wonder-app-vd03:5222` (VPN). Production Kestrel via env (`Kestrel__Endpoints__Http__Url`), the HG pattern; **don't set `ASPNETCORE_URLS` alongside Kestrel endpoint config** (URLs-override trap). +- `docker/`: HG-pattern runtime-only image — host-side `dotnet publish -c Release -o docker/publish -p:UseAppHost=false` (host has the Gitea feed creds), `FROM mcr.microsoft.com/dotnet/aspnet:10.0`, `COPY publish/ ./`, `EXPOSE 5320`, `ENTRYPOINT ["dotnet","ZB.MOM.WW.Overview.dll"]`; compose project name `zb-overview`, port `5320:5320`, `ASPNETCORE_ENVIRONMENT` + Kestrel env vars. No env_file needed (no secrets). Note: `aspnet:10.0` has no `curl` — healthcheck via `wget`/dotnet or skip. + +**Phase DoD:** `dotnet build` 0 warnings; `dotnet test` green; `dotnet run` on :5320 renders the page instantly from cache against a dev registry; container builds and runs. + +--- + +## Phase 4 — live acceptance (design §9) + +Prep: capture real `/health/ready` + `/health/active` payloads from all four apps (post-0.2.0) into the golden-test fixtures. Then run the design §9 checklist end-to-end on the rigs (OtOpcUa 6-node docker-dev + ScadaBridge local cluster + windev mxgw + wonder-app-vd03 HG if reachable/VPN): +1. Every registered instance renders correctly; single Active per pair (central AND site). +2. **Leader chip correct per cluster group and moves when the leader node is stopped**; split-brain flag not shown in normal operation. +3. Kill a node → Unreachable within ~2 cadences; restart → recovers. +4. Force a Degraded check → Warn with failing entry visible. +5. Cache-first: first page load instant, no poll triggered by the request. +6. Pause auto-refresh past `StaleAfterSeconds` → all cards Stale; resume → clean recovery. Also stop the poller (kill the service loop) → same. +7. Management links open the right UIs; page reachable with **no login**. +8. Malformed registry → boot fails with a clear preflight message. +9. `curl :5320/health/ready` + `/metrics` (gauge present with per-instance labels). + +**Program DoD:** all 9 pass; ScadaBridge PR merged; OtOpcUa bump merged; Health 0.2.0 on the feed; scadaproj commits for the lib + new app + docs. + +--- + +## Cross-cutting notes for the executor + +- **Family gotchas that WILL bite:** `ZbTelemetryOptions.Meters` silent allowlist; mxgw/HG have no `Directory.Packages.props` (inline pins); never host-`sqlite3` a live container WAL DB during rig checks; `Akka.TestKit.Xunit2` is xunit-v2-only (fine here — everything in this plan is xunit 2.9.3); Gitea feed verification uses `/api/packages/dohertj2/nuget/registration/…`. +- **Do not** add auth "while at it" — anonymous read-only is a requirement, not an omission. +- **Contract discipline:** the `data` field is additive and optional; the dashboard must render fully (minus leader) against 0.1.0-era payloads — that's what keeps Phase 3 unblocked by Phases 1/2 and tolerant of un-bumped deployments. +- Commit style: lib change + publish in scadaproj (`feat(health): 0.2.0 …`); ScadaBridge and OtOpcUa changes as PRs in their own repos referencing this plan; new app committed in scadaproj. diff --git a/overview_dashboard_options.md b/overview_dashboard_options.md new file mode 100644 index 0000000..38e8148 --- /dev/null +++ b/overview_dashboard_options.md @@ -0,0 +1,86 @@ +# Central overview dashboard for the scadaproj family — options investigation + +**Date:** 2026-07-22 +**Goal:** a single pane showing status of every app instance (OtOpcUa, ScadaBridge, MxAccessGateway, HistorianGateway — each with multiple nodes/pairs), with deep links to each instance's own management UI, pre-configured via an appsettings registry of applications and instances. "Aspire-dashboard-like." + +--- + +## 1. What the family already gives us (verified in code, 2026-07-22) + +The reconnaissance result is unusually favorable: **the data plane for a poll-based overview already exists on all four apps, with zero changes needed in the target apps.** + +- **Health endpoints are uniform and anonymous everywhere.** All four apps call the shared `MapZbHealth()` (`ZB.MOM.WW.Health`), which maps `/health/ready`, `/health/active`, and `/healthz`, all `.AllowAnonymous()` in the library itself (`ZbHealthEndpointExtensions.cs:35-64`). Even OtOpcUa's global `FallbackPolicy = RequireAuthenticatedUser` doesn't gate them (endpoint `AllowAnonymous` wins). +- **The JSON contract is canonical and stable** (`ZbHealthWriter.cs:45-81`, camelCase): + ```json + { "status": "Healthy|Degraded|Unhealthy", + "totalDurationMs": 12.3, + "entries": { "": { "status": "...", "description": null, "durationMs": 1.2 } } } + ``` + Healthy/Degraded → HTTP 200, Unhealthy → 503. +- **`/health/active` is the active/standby signal.** ScadaBridge Central's `active` tier runs the oldest-member `active-node` check; OtOpcUa has `admin-leader`. No off-the-shelf product can express active/standby natively — this endpoint makes it a first-class dashboard column for us. +- **`/metrics` (Prometheus) is anonymous on all four** (explicitly on OtOpcUa; by absence of a fallback policy on the others). OTLP push is config-only on three apps (`:Telemetry:Exporter=Otlp` + `OtlpEndpoint`); **HistorianGateway hardcodes its telemetry setup and would need a one-line change** to gain the config keys. +- **Nothing aggregates across apps today.** The closest is ScadaBridge's `/monitoring/health` (CentralUI), but it's push-based over the site transport and scoped to ScadaBridge sites in one cluster. OtOpcUa/HG/mxgw pages show only their own instance. A multi-app registry + remote HTTP poller is genuinely new work. +- **Reusable building blocks for an in-house pane:** `ZB.MOM.WW.Theme` (`ThemeShell`, `StatusPill` with Ok/Warn/Bad/Idle/Info, `TechCard`), `ZB.MOM.WW.Auth.AspNetCore` (the same hardened LDAP cookie login all four apps use), `ZB.MOM.WW.Configuration` (validate the registry at startup), `ZB.MOM.WW.Telemetry`/`Health` (the dashboard monitors itself the family way). No prebuilt tile/grid widget exists — a node-card composite would be new, but it's `TechCard` + `StatusPill` composition. +- **Known ports for the registry:** OtOpcUa AdminUI dev `:9000` (driver-only nodes fall back to `:5000`); ScadaBridge Central `:5000/:5001` (site nodes expose only `:8084` metrics — no health/UI); mxgw `:5120/:7121`; HistorianGateway dashboard `:5220` dev / single TLS ALPN port (rec. `:5222`) in production. + +**Watch-out for the registry design:** ScadaBridge *site* nodes map no health endpoints at all (metrics only), so a site-pair row can't be polled the same way as a central pair — either poll `:8084/metrics` for liveness, read site status from Central's aggregator, or (later) add `MapZbHealth` to site nodes. + +## 2. Off-the-shelf candidates (facts verified by fetch, 2026-07-22) + +| Option | Version / status | Config-file registry | Persistence | Auth | Verdict | +|---|---|---|---|---|---| +| **Aspire dashboard (standalone)** | Aspire 13 (Nov 2025); `mcr.microsoft.com/dotnet/aspire-dashboard` | **No** — resource UI is disabled in standalone; needs a custom gRPC resource service to light up | None (in-memory, lost on restart) | BrowserToken / OIDC | Not the ops pane. MS docs: dev/short-term diagnostic tool. `AddExternalService()`/custom URLs are AppHost run-mode features only. | +| **AspNetCore.HealthChecks.UI (Xabaril)** | 9.0.0 (Dec 2024), net8.0 TFM only; upstream coasting on dependabot since Aug 2025 ("looking for maintainers"); tiny `DotNetDiag` fork exists | **Yes** — `HealthChecksUI:HealthChecks` array in appsettings; exactly the requested model | SQLite/SQL Server/InMemory | In-process — **our LDAP cookie works unmodified** | Closest functional match, but it **requires the `UIResponseWriter` payload** (issue #285) — our `ZbHealthWriter` shape is close but not identical (`durationMs` vs `duration`, no `data`/`tags`), so every app would need a second UI-format endpoint. Plus a slow-moving upstream dependency. | +| **Gatus** | v5.36.0 (May 2026), active | **Yes** — YAML; conditions incl. `[BODY]` JSONPath, so it can evaluate our health JSON **with zero app changes** (incl. degraded) | SQLite/Postgres | Built-in basic/OIDC (no LDAP cookie) | Best zero-effort comparable: registry + JSON conditions + uptime history + 30+ alert providers in one Go binary. Trade: non-.NET, non-Theme UI, outside our auth stack. | +| **Homepage (gethomepage.dev)** | v1.13.2 (Jun 2026) | Yes — YAML link tiles | n/a | None (reverse proxy) | Link portal only; `siteMonitor` is status-code-only — cannot express Degraded or active/standby. Out. | +| **Grafana + Prometheus + blackbox** | Grafana 13.0 (Apr 2026) | Yes — scrape config | Full TSDB history | **Native LDAP** | Most durable, real alerting, grows into actual ops monitoring. Highest setup (3 containers + dashboard authoring); Degraded is invisible to blackbox (HTTP 200) without a status gauge metric or `ResultStatusCodes` remap; deep links via data links are clunkier than a real portal. | + +## 3. Options + +### Option A — build a small in-house overview app (`ZB.MOM.WW.Overview`) — **RECOMMENDED** + +A thin Blazor Server app hosted in scadaproj alongside the other shared components, composed almost entirely from the family's own libraries: + +- **Registry:** `Overview:Applications[]` in appsettings — per app: name, kind, accent; per instance: display name, site/role, `healthBaseUrl`, `managementUrl`, optional `metricsUrl`, `expectActive` flag, poll interval/timeout overrides. Validated at startup via `ZB.MOM.WW.Configuration` (`AddValidatedOptions` + `ConfigPreflight`) — malformed registry fails the boot, not the first poll. +- **Poller:** one `BackgroundService` + `IHttpClientFactory`; polls each instance's `/health/ready` + `/health/active` in parallel on a fixed cadence (5–15 s), deserializes the canonical `ZbHealthWriter` JSON (a ~30-line typed client — worth putting in `ZB.MOM.WW.Health` as `ZbHealthReportClient` so it's contract-owned), and keeps an in-memory snapshot. Statuses per instance: **Up / Degraded / Down / Unreachable**, plus **Active/Standby** from `/health/active` — the column no off-the-shelf option can render. +- **UI:** `ThemeShell` + a new node-card grid built from `TechCard` + `StatusPill`; per-check expandable detail (the `entries` dict); every card deep-links to the instance's own management page. LDAP cookie login via `AddZbLdapAuth` — same UX as the other four apps (the dashboard reads only anonymous endpoints, but the pane itself should sit behind the family login). +- **Self-hosting:** it registers its own `MapZbHealth`/`MapZbMetrics` like any family app; a `docker/` folder mirroring HistorianGateway's runtime-only image pattern. + +**Fit:** best-practice fit for this shop. The requested feature set (appsettings registry, status, deep links) maps 1:1 onto libraries that exist precisely so new apps can be assembled this way; the only genuinely new code is the poller + snapshot + one grid page — realistically **2–4 days** for a first useful version. It also keeps auth, theming, and ops conventions identical to the rest of the family, which none of the off-the-shelf options can do. Deliberate scope cut for v1: no history/alerting (that's Option D's job if it's ever needed — and the registry file ports straight to a Prometheus scrape config). + +**Costs / risks:** you own the code (small); no built-in uptime history or webhooks in v1; the ScadaBridge-site-node gap (§1 watch-out) needs a decision in the registry design. + +### Option B — AspNetCore.HealthChecks.UI embedded in a thin family host + +Host the Xabaril UI inside a small ASP.NET app behind our LDAP cookie. Gets registry-from-appsettings, polling, SQLite history, and webhooks for near-zero UI code. + +**Fit:** the strongest off-the-shelf functional match, and the only one where `ZB.MOM.WW.Auth` works unmodified. But it costs a change in **all four target apps** (a second health endpoint in `UIResponseWriter` format, because the UI hard-requires that payload), buys a visibly unmaintained upstream (net8.0-only TFM, dependabot-only commits since Aug 2025), has no active/standby concept, and its UI will never look like Theme. For a shop that already owns Health + Theme + Auth libs, it saves less than it costs. + +### Option C — Gatus container as a zero-app-change status page + +Deploy Gatus with a YAML registry; its `[BODY]` JSONPath conditions consume our existing health JSON directly (including Degraded), and it brings uptime history + alerting for free. + +**Fit:** the best effort-to-value stopgap — running in an afternoon, no app changes, actively maintained. Best practice would call this the "buy" answer if the family didn't have its own UI/auth stack. Trades: separate basic/OIDC auth outside the LDAP cookie, un-Theme-able UI, and active/standby only expressible as another endpoint row rather than a paired-node concept. Reasonable as an interim pane while Option A is built, or as a permanent low-ceremony answer if the dashboard turns out to be rarely used. + +### Option D — Grafana + Prometheus + blackbox exporter + +**Fit:** the right long-term answer **if** the real goal grows into ops monitoring (history, alerting, SLOs) rather than a convenience pane — native LDAP auth is a genuine plus. As a *first* deliverable for "status + links" it's the highest effort and the worst at deep links/branding, and Degraded/active-standby need extra plumbing (a health-status gauge in `ZB.MOM.WW.Telemetry` would be the clean fix, and is worth doing eventually regardless). Not first; a natural phase 2 that Option A's registry feeds directly. + +### Option E — Aspire dashboard standalone + +**Fit:** rejected as the overview pane — in standalone mode the resource/status/links UI is disabled (it's an OTLP telemetry viewer only), nothing persists across restarts, and Microsoft positions it explicitly as a dev/short-term diagnostic tool. **Worth keeping as a free companion tool:** three of the four apps can push OTLP with config only, so an ad-hoc `aspire-dashboard` container gives structured logs + traces during debugging for zero code. (HistorianGateway needs the one-line telemetry-config change to join.) + +> **DECISION (2026-07-22):** Option A chosen — custom app, family look-and-feel (Theme/Auth/Health kit). Aspire rejected for the pane. V1 design: [`docs/plans/2026-07-22-overview-dashboard-design.md`](docs/plans/2026-07-22-overview-dashboard-design.md). + +## 4. Recommendation + +**Option A** — build the thin `ZB.MOM.WW.Overview` app from the family's own parts, with two riders: + +1. If an interim pane is wanted *this week*, stand up **Gatus (Option C)** against the existing health endpoints — zero app changes, discard later. +2. Keep **Aspire standalone (Option E)** in the toolbox as a dev-time OTLP viewer, and make HistorianGateway's telemetry config-driven (one line) so all four apps can join it. + +Decisions to make before building A: how site-node ScadaBridge rows are sourced (metrics-liveness vs Central-aggregator vs adding MapZbHealth to sites), and whether v1 needs any history/alerting (recommended: no — defer to Option D if ever needed). + +--- + +*Sources: family code (ZB.MOM.WW.Health `ZbHealthEndpointExtensions.cs`/`ZbHealthWriter.cs`, per-app Program.cs wiring); aspire.dev standalone/configuration/security docs + Aspire 13 announcement; Xabaril repo + NuGet + issue #285 + DotNetDiag fork; TwiN/gatus releases; gethomepage.dev docs; Grafana/blackbox dashboards. All web claims verified by fetch 2026-07-22.*