dohertj2/scadaproj
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs(overview): family overview dashboard — options, design, impl plan, mockup

Browse Source 
Options investigation (Aspire standalone/HealthChecks.UI/Gatus/Homepage/
Grafana vs custom) → decision: custom ZB.MOM.WW.Overview Blazor app in the
family look. Design: anonymous read-only single pane, appsettings registry
of all four apps' instances, cache-first poller on a configurable timer,
configurable staleness timeout, Active/Standby from /health/active, and
per-cluster Akka leader chips (needs Health 0.2.0 optional per-entry data).
Impl plan: phased + code-verified — Phase 0 Health 0.2.0 (writer data field
+ AkkaClusterHealthCheck cluster view), Phase 1 ScadaBridge site-node
MapZbHealth PR (:8084; role-scoped active check), Phase 2 OtOpcUa bump,
Phase 3 the app (:5320), Phase 4 live rig acceptance. Mockup is the visual
reference: real Theme tokens + embedded IBM Plex, all card states incl.
stale + split-brain. Ready to execute in a fresh session.
This commit is contained in:
Joseph Doherty
2026-07-22 09:22:13 -04:00
parent ff58ad4f60
commit 6dd7858619
4 changed files with 964 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/mockups/overview-dashboard-mockup.html
+554
View File
File diff suppressed because one or more lines are too long
docs/plans/2026-07-22-overview-dashboard-design.md
+153
View File
@@ -0,0 +1,153 @@
# ZB.MOM.WW.Overview — family overview dashboard, v1 design
**Date:** 2026-07-22
**Decision context:** `overview_dashboard_options.md` (same date) — Option A chosen: a custom in-house pane, matching the look and feel of the other scadaproj apps. Aspire standalone rejected (resource/status UI disabled outside an AppHost; no persistence; dev-tool positioning).
**Implementation plan:** `2026-07-22-overview-dashboard-impl-plan.md` (same directory) — phased, code-verified tasks incl. the sister-project changes.
## 1. What it is
A small Blazor Server app, hosted in scadaproj like the other shared components, that shows **every registered application instance on one page**: Up/Degraded/Down/Unreachable, Active/Standby where applicable, the **current Akka cluster leader** for each cluster group, per-check detail, and a deep link to each instance's own management UI. The page always renders instantly from a cached snapshot — a background poller on a **configurable timer** does all the HTTP work, and a **configurable staleness timeout** marks any data the poller hasn't refreshed as out-of-date/offline. **No login: the dashboard is anonymous and strictly read-only** (it only GETs endpoints that are already anonymous, and has no mutating actions). Configured entirely from an appsettings registry. **Every instance reports through the same channel:** the anonymous `/health/ready` + `/health/active` endpoints of `MapZbHealth`, parsing the canonical `ZbHealthWriter` JSON. Three of the four apps already expose these everywhere; the one gap — ScadaBridge *site* nodes — is closed by a small prerequisite change in ScadaBridge (§4) rather than worked around, so the dashboard has exactly one probe model.
Non-goals for v1: history/uptime graphs, alerting/webhooks, telemetry viewing (OTLP/traces), and any push-based ingestion. If ops-grade monitoring is ever needed, that's the Grafana/Prometheus phase in the options doc — this registry ports straight to a scrape config.
## 2. Project shape
```
ZB.MOM.WW.Overview/ # plain directory in scadaproj — NOT a nested git repo
ZB.MOM.WW.Overview.slnx
src/ZB.MOM.WW.Overview/ # single ASP.NET Core + Blazor Server project
Program.cs
Registry/ (options + validator)
Polling/ (poller BackgroundService, health client, snapshot store)
Components/ (App.razor, MainLayout → ThemeShell, Pages/Overview.razor, widgets)
tests/ZB.MOM.WW.Overview.Tests/
docker/ # HistorianGateway-pattern runtime-only image
```
- .NET 10, global InteractiveServer render mode (same router style as ScadaBridge/HG/mxgw — no per-page render-mode traps).
- Package references (Gitea feed): `ZB.MOM.WW.Theme` 0.3.1, `ZB.MOM.WW.Health` (0.2.0 — see §4a), `ZB.MOM.WW.Telemetry(.Serilog)` 0.1.0, `ZB.MOM.WW.Configuration` 0.1.0. **No Auth, no Audit, no Secrets**: the dashboard is anonymous read-only by requirement — there is no login, no mutating action to audit, and nothing secret in the registry. (If a login is ever wanted later, `AddZbLdapAuth` + `LoginCard` drop in the family way.)
- It is an **app**, not a published library — nothing gets packed to the feed.
## 3. Registry (appsettings)
```jsonc
"Overview": {
"PollIntervalSeconds": 10, // global poll cadence (configurable timer)
"TimeoutSeconds": 3, // per-request timeout
"StaleAfterSeconds": 45, // snapshot older than this ⇒ instance rendered Stale (out of date/offline)
"Applications": [
{
"Name": "OtOpcUa",
"ManagementLabel": "AdminUI",
"Instances": [
{ "Name": "central-1", "Group": "central",
"BaseUrl": "http://otopcua-central-1:9000",
"ManagementUrl": "http://otopcua-central-1:9000/",
"HasActiveRole": true },
{ "Name": "site-a-1", "Group": "site-a", "BaseUrl": "http://...", "HasActiveRole": true }
]
},
{
"Name": "ScadaBridge",
"Instances": [
{ "Name": "central-a", "Group": "central", "BaseUrl": "http://sb-central-a:5000",
"ManagementUrl": "http://sb-central-a:5000/", "HasActiveRole": true },
{ "Name": "site-a-1", "Group": "site-a",
"BaseUrl": "http://sb-site-a-1:8084", "HasActiveRole": true } // health on the HTTP/1.1 listener — see §4
]
},
{
"Name": "MxAccessGateway",
"ManagementLabel": "Dashboard",
"Instances": [
{ "Name": "windev", "BaseUrl": "http://windev:5130",
"ManagementUrl": "http://windev:5130/" } // single-instance; no active/standby role
]
},
{
"Name": "HistorianGateway",
"ManagementLabel": "Dashboard",
"Instances": [
{ "Name": "wonder-app-vd03", "BaseUrl": "https://wonder-app-vd03:5222",
"ManagementUrl": "https://wonder-app-vd03:5222/" } // prod single TLS ALPN port; dev :5220
]
}
]
}
```
All four family apps are first-class registry citizens; `HasActiveRole` is simply omitted for the two single-instance gateways (mxgw, HistorianGateway), which have no active/standby concept — their cards show status without the Active badge.
- Every instance is probed the same way: GET `{BaseUrl}/health/ready`, plus `{BaseUrl}/health/active` when `HasActiveRole`. There is deliberately no per-instance probe-mode switch — one probe model, one status derivation.
- `PollIntervalSeconds`/`TimeoutSeconds`/`StaleAfterSeconds` are overridable per application and per instance (instance wins) for slow links or chatty pairs; `StaleAfterSeconds` must exceed `PollIntervalSeconds` (validated).
- Bound + validated with `AddValidatedOptions<OverviewOptions, OverviewOptionsValidator>` (`ZB.MOM.WW.Configuration`): non-empty registry, absolute http(s) URLs, unique app/instance names, positive intervals, stale > poll. Malformed registry fails boot via `ConfigPreflight`, not the first poll.
## 4. Prerequisite target-app changes (health from ALL apps)
The dashboard's contract is "every registered instance serves `MapZbHealth`". Current reality and the deltas:
| App / node kind | Today | Delta needed |
|---|---|---|
| OtOpcUa admin + driver nodes | `MapZbHealth` mapped unconditionally | None in code. **Deployment note:** a driver-only Windows-service node with no explicit URLs falls back to Kestrel's `localhost:5000` — every registered node must be given a reachable binding (`ASPNETCORE_URLS`/`HTTP_PORTS`), which the docker rig already does. |
| ScadaBridge **Central** | `MapZbHealth` (ready: database/akka-cluster/required-singletons; active: active-node) | None. |
| ScadaBridge **Site** nodes | **No health endpoints** — only the gRPC `:8083` (HTTP/2-only) + `/metrics` on the `:8084` HTTP/1.1 listener | **Small ScadaBridge PR (the one prerequisite):** map `MapZbHealth` on the `:8084` HTTP/1.1 listener alongside `/metrics`. Ready checks: `akka-cluster` (site pair) + site SQLite store (+ `localdb-replication` where enabled); active check: the site pair's oldest-Up active-node rule via `ActiveNodeHealthCheck` — the Host already references all three `ZB.MOM.WW.Health` packages, so this is pure wiring, no new dependencies. Gives site rows full Up/Degraded detail **and** Active/Standby, same as everything else. |
| MxAccessGateway | `MapZbHealth` (ready: auth-store) | None. |
| HistorianGateway | `MapZbHealth` (ready: historian-connection, galaxy-sql, conditional runtime/store-forward/pool checks) | None. |
Sequencing: the ScadaBridge PR can land any time before the dashboard's acceptance run; the dashboard itself needs no knowledge of it beyond registry entries.
### 4a. Shared-lib prerequisite: `ZB.MOM.WW.Health` 0.2.0 — cluster-leader data
Requirement: the dashboard must show the **current leader** of each Akka cluster. The canonical health JSON today carries only `status`/`description`/`durationMs` per entry — no structured data — so this needs a small **additive** change to the shared lib (which is exactly where the family wants such things to live):
- **Core (`ZbHealthWriter`):** emit an optional `"data": { ... }` object per entry, sourced from `HealthReportEntry.Data`, **only when non-empty** — existing payloads stay byte-identical, so this is non-breaking for every current consumer.
- **`ZB.MOM.WW.Health.Akka` (`AkkaClusterHealthCheck`):** populate the check's data dictionary from local cluster state: `leader` (address), `selfAddress`, `selfRoles`, `memberCount`, `unreachableCount` (+ `roleLeader:<role>` where the app cares). Each node reports *its own view*; the dashboard reads the leader from any Up node of the group and flags disagreement (a visible split-brain tell — a nice free diagnostic for the 2-node pairs).
- **Version/rollout:** bump lib to `0.2.0`, publish the 3 packages to the Gitea feed. **Required bumps:** OtOpcUa + ScadaBridge (the Akka apps — leader data appears with the package bump alone, no app code changes, since both register the shared `AkkaClusterHealthCheck`). **Recommended alignment bumps:** mxgw + HistorianGateway (no behavior change for them; keeps the family version matrix aligned).
## 5. Polling and status model
One `OverviewPollerService : BackgroundService`:
- Every cadence tick, polls **all instances in parallel** (`Task.WhenAll`, per-instance `CancellationTokenSource` timeout; named `HttpClient` via `IHttpClientFactory`, no retries — the next tick is the retry).
- Parses the canonical shape `{ status, totalDurationMs, entries: { name: { status, description, durationMs } } }`. Parsing lives in a small `ZbHealthReportClient` inside this app for v1; promoting it into a `ZB.MOM.WW.Health.Client` package is a deliberate later step, not a v1 dependency (recommendation: promote only when a second consumer appears).
- Derived per-instance state:
- **Up** — ready 200 + `"Healthy"`; **Degraded** — 200 + `"Degraded"`; **Down** — 503 / `"Unhealthy"`; **Unreachable** — timeout, refused, DNS, non-JSON (kept distinct from Down: it usually means network/VPN/registry-typo, not the app).
- **Active/Standby** — `/health/active` 200 → Active, 503 → Standby, error → Unknown (only when `HasActiveRole`).
- Plus: last poll UTC, latency ms, per-check entries, consecutive-failure count (state flips to Down/Unreachable only after **2** consecutive failures to avoid single-blip flapping; recovery is immediate).
- **Cache-first rendering:** results land in an in-memory `OverviewSnapshotStore` (single immutable snapshot swapped atomically); an event notifies circuits to re-render. Page loads NEVER trigger a poll — they read the cached snapshot and render instantly; the poller timer is the only thing that does HTTP. No persistence — a restart repolls the world within one cadence.
- **Staleness:** every instance snapshot carries `LastUpdatedUtc`. When the UI renders (or a periodic sweep runs), any instance whose snapshot age exceeds its effective `StaleAfterSeconds` is shown as **Stale (out of date/offline)** regardless of its last known status — this catches a stuck poller, a paused refresh, or a machine-slept dashboard, not just failed probes.
- **Leader extraction:** for each `Group` (one Akka cluster = one group, e.g. `OtOpcUa/central`, `ScadaBridge/site-a`), the aggregator reads the `akka-cluster` entry's `data.leader` from each Up member and surfaces the group's leader (resolving the leader address back to a registry instance name where possible). If Up members disagree on the leader, the group is flagged (split-brain indicator).
- StatusPill mapping: Up→`Ok`, Degraded→`Warn`, Down→`Bad`, Unreachable→`Bad` (distinct label), Stale→`Idle`, Active/Leader badges→`Info`.
## 6. UI
> **Mockup:** [`docs/mockups/overview-dashboard-mockup.html`](../mockups/overview-dashboard-mockup.html) (static, self-contained; real Theme tokens + embedded IBM Plex) — the visual reference for `InstanceCard`, group headers, leader/split-brain chips, and every status state incl. Stale. Open it in a browser; the implementation should match it.
Single page (`/`), `ThemeShell` chassis with its own product name/accent so it reads as a sibling of the other four apps:
- One section per application (name + rollup pill = worst instance status), instances as a grid of cards (`TechCard` + `StatusPill` composition — a new `InstanceCard` component, since Theme has no prebuilt tile widget). Cluster groups render a group header with the **Leader: <instance>** chip (and the split-brain warning when member views disagree).
- Card: instance name + group, status pill, Active badge, **Leader badge** on the leader node's card, latency, relative last-seen (turns into an explicit "stale since …" once past `StaleAfterSeconds`); expandable check list (`entries` with per-check pill + description); a `TechButton` "Open <ManagementLabel>" → `ManagementUrl` (new tab).
- A header row: total counts (n Up / n Degraded / n Down / n Unreachable / n Stale), last-refresh clock, pause/resume auto-refresh (paused long enough, cards go Stale by design).
- **No login.** The dashboard is anonymous and read-only end to end; there is nothing to authorize and nothing it can change. It still matches the sister apps' look exactly (ThemeShell + tokens + IBM Plex), just without the login gate.
## 7. Self-observability and hosting
- The dashboard eats the family dog food: `MapZbHealth` (ready check: "registry loaded" + "last poll cycle completed"), `MapZbMetrics`, `AddZbTelemetry`/`AddZbSerilog`; a `zb_overview_instance_status` gauge (per instance, labeled) so even v1 status is scrapeable — and becomes the Grafana bridge later.
- Ports: dev `http://localhost:5320` (clear of every family port in use: 5000/5001, 5120/7121, 5220–5222, 8081–8085, 9000, 9200, 4053, 4840); production single TLS ALPN endpoint like HistorianGateway, warn-only if terminated upstream.
- `docker/` mirrors the HistorianGateway pattern: publish framework-dependent on the host (authed Gitea feed), COPY into `aspnet:10.0`, gitignored env_file. Natural first deployment: the docker-dev rig host, registry pointed at the existing OtOpcUa + ScadaBridge rig nodes — that is also the acceptance test.
## 8. Open decisions (settled recommendations)
1. **ScadaBridge site rows** — resolved (2026-07-22, user direction): site nodes gain `MapZbHealth` as the one prerequisite PR (§4), so every row is a full health row. *Chosen over* a metrics-liveness fallback (two probe models) and over reading Central's site aggregator (couples the dashboard to ScadaBridge internals and its transport).
2. **Health-client ownership** — in-app for v1; promote to `ZB.MOM.WW.Health.Client` on second consumer. *(Best practice: contract-owning lib should host the client, but a package bump + publish for one consumer is ceremony without benefit.)*
3. **History/alerting** — explicitly out of v1; the `zb_overview_instance_status` gauge is the forward hook (Prometheus can scrape the dashboard alone and get fleet status history for free).
4. **Leader transport** — resolved (2026-07-22, user requirement): via a `data` object in the canonical health JSON (`ZB.MOM.WW.Health` 0.2.0, §4a), populated by the shared `AkkaClusterHealthCheck`. *Chosen over* description-string parsing (fragile) and a separate leader endpoint (second contract for one field).
5. **Auth** — resolved (2026-07-22, user requirement): none. Anonymous, read-only.
## 9. Acceptance (v1 definition of done)
- The §4 ScadaBridge site-node health PR is merged and site rows render with full check detail + Active/Standby (no special-cased rows anywhere).
- Registry pointed at the live docker-dev rigs (OtOpcUa 6-node + ScadaBridge local cluster) renders every instance correctly, including: each Active/Standby pair (central **and** site pairs) showing exactly one Active; **each cluster group showing its current leader, and the leader chip moving when the leader node is stopped**; killing a node flips its card to Unreachable within ~2 cadences and back on restart; a Degraded check (e.g. stop a dependency) shows Warn with the failing entry visible; management links open the right app UIs; the page is reachable with no login.
- **Cache/staleness proven live:** first page load renders instantly from cache (no poll on request); with auto-refresh paused (or the poller stopped) past `StaleAfterSeconds`, every card flips to Stale, and resumes cleanly.
- Boot fails with a clear `ConfigPreflight` message on a malformed registry.
- `dotnet test` green: options validator, status derivation (incl. flap damping, staleness marking, leader extraction + disagreement flag, JSON parsing golden tests against captured real payloads from all four apps — with and without `data`), snapshot store, and a bUnit render test of `InstanceCard`.
docs/plans/2026-07-22-overview-dashboard-impl-plan.md
+171
View File
@@ -0,0 +1,171 @@
# ZB.MOM.WW.Overview — implementation plan
**Date:** 2026-07-22
**Design:** `2026-07-22-overview-dashboard-design.md` (same directory). Read it first; this plan assumes its decisions (custom app, family look/feel, anonymous read-only, cache-first polling, staleness timeout, Akka leader display).
**Repos touched:** `scadaproj/ZB.MOM.WW.Health` (shared lib 0.2.0), `~/Desktop/ScadaBridge` (site-node health PR), `~/Desktop/OtOpcUa` (Health bump + check verification), optional alignment bumps in `~/Desktop/MxAccessGateway` + `~/Desktop/HistorianGateway`, and the new `scadaproj/ZB.MOM.WW.Overview/`.
**Sequencing:** Phase 0 → Phases 1 + 2 (parallel, independent repos) → Phase 3 (app; can start in parallel once 0.2.0 is on the feed) → Phase 4 (live acceptance). Phases 1/2 only gate Phase 4, not Phase 3 development.
All facts below were code-verified 2026-07-22 (file:line cites are from that pass); re-check a cite if the target repo has moved since.
**How to run this in a fresh session:** read the design doc first, then execute phases in order (0 → 1‖2 → 3 → 4), one phase per sitting with its DoD met before moving on. Work in each repo on a feature branch (`feat/health-0.2.0-cluster-data` in scadaproj, `feat/site-node-health` in ScadaBridge, `feat/health-0.2.0-bump` in OtOpcUa); commit scadaproj work (lib + new app + these docs) on `main` per family habit only when asked. Nothing in this plan requires user input except merges/pushes and the Phase 4 rig runs; if a cited line has drifted, re-locate by the quoted identifier, not the line number.
---
## Phase 0 — `ZB.MOM.WW.Health` 0.2.0: entry `data` + Akka cluster-view
Repo: `scadaproj/ZB.MOM.WW.Health/` (plain directory — do NOT `git init`; commits happen in scadaproj). Akka.Cluster pin 1.5.62 (`Directory.Packages.props:9`); version is centralized at `Directory.Build.props:8` (`<Version>0.1.0</Version>`).
### Task 0.1 — writer: optional per-entry `data` object
`src/ZB.MOM.WW.Health/ZbHealthWriter.cs`:
- Add to the private `HealthEntryDto` (lines ~76-81):
`[JsonIgnore(Condition = JsonIgnoreCondition.WhenWritingNull)] public IReadOnlyDictionary<string, object>? Data { get; init; }`
- In the projection (line ~62): `Data = e.Value.Data.Count > 0 ? e.Value.Data : null` (`HealthReportEntry.Data` is never null; empty when unset).
- **Invariant: old payloads stay byte-identical.** The per-property `JsonIgnore` is essential — the global serializer options deliberately do NOT ignore nulls (the null-`description` behavior is pinned by `ResponseWriterTests.cs:69-85`); do not change `SerializerOptions`.
Tests (`tests/ZB.MOM.WW.Health.Tests/ResponseWriterTests.cs`): extend `StubHealthCheck` (lines 21-31) to accept an optional data dictionary; add (a) non-empty data → `"data": {...}` emitted with camelCase envelope keys but **verbatim data keys**, (b) empty data → no `data` key at all (assert on raw JSON string), (c) existing null-description test still green.
### Task 0.2 — `AkkaClusterHealthCheck`: populate cluster-view data
`src/ZB.MOM.WW.Health.Akka/AkkaClusterHealthCheck.cs` (currently reads only `SelfMember.Status`, returns description-only — lines 38-70):
- Add an internal static helper `BuildClusterData(Cluster cluster)` returning `IReadOnlyDictionary<string, object>` with JSON-friendly values only (string/int/string[]):
- `leader` — `cluster.State.Leader?.ToString()` (omit the key when null: pre-join/unknown),
- `selfAddress` — `cluster.SelfAddress.ToString()`,
- `selfRoles` — `cluster.SelfMember.Roles.OrderBy(r => r).ToArray()`,
- `memberCount` — `cluster.State.Members.Count`,
- `unreachableCount` — `cluster.State.Unreachable.Count`.
All property names confirmed compiling in this repo (`ActiveNodeHealthCheck.cs:116-132`, `AkkaActiveNodeGate.cs:33-49`).
- Pass it through every result path: `HealthCheckResult.Healthy/Degraded/Unhealthy(description, data)`. The no-ActorSystem / cluster-inaccessible paths keep returning description-only (no data) — pinned by `AkkaClusterStatusPolicyTests.cs:63-96`.
- ASP.NET Core copies `HealthCheckResult.Data` → `HealthReportEntry.Data` automatically, and `MapZbHealth` already wires `ZbHealthWriter` (`ZbHealthEndpointExtensions.cs:42-54`) — **no consumer code change needed beyond the package bump**, provided the app registers this shared check (verified for ScadaBridge; Task 2.1 verifies OtOpcUa).
Tests (`tests/ZB.MOM.WW.Health.Akka.Tests/`): the `InternalsVisibleTo` hook (`ActiveNodeHealthCheck.cs:7`) exposes the helper. Use `Akka.TestKit.Xunit2` (pin 1.5.62; conventions per existing tests — inherit `TestKit`, never hand-roll `ActorSystem.Create` + manual join) with a single-node self-joined cluster: assert `leader` == self address, `memberCount` == 1, `selfRoles` round-trip; plus the check-level test that a healthy result now carries data and the degraded no-system path carries none.
### Task 0.3 — version, pack, publish
- `Directory.Build.props`: `0.1.0` → `0.2.0`. Note the change in the repo's CHANGELOG/README if present.
- `dotnet test` from the repo root (baseline 58 tests across 3 test projects — all green plus the new ones), then `dotnet pack -c Release -o artifacts` (3 nupkgs @ 0.2.0).
- Publish: `dotnet nuget push artifacts/ZB.MOM.WW.Health*.0.2.0.nupkg --source https://gitea.dohertylan.com/api/packages/dohertj2/nuget/index.json --api-key <from ~/.nuget user config / keychain — same credential every prior publish used>` (one command per nupkg or glob).
- Verify like every prior lib release (the family gotcha: "published" claims must be proven): `curl -s https://gitea.dohertylan.com/api/packages/dohertj2/nuget/registration/zb.mom.ww.health/index.json` shows 0.2.0 (NOT `/nuget/v3/registration/` — that 404s for everything), then restore-verify from a scratch consumer project that pins 0.2.0.
**Phase DoD:** suite green; 3 packages at 0.2.0 restore-verified from the feed; a manual smoke (any sample host + the check) shows `entries["akka-cluster"].data.leader` in `/health/ready`.
---
## Phase 1 — ScadaBridge PR: site-node health endpoints
Repo: `~/Desktop/ScadaBridge` (branch off `main`, e.g. `feat/site-node-health`). Site branch of `src/ZB.MOM.WW.ScadaBridge.Host/Program.cs` is lines 485-570: Kestrel `:8083` gRPC Http2-only (507-512), `:8084` `Http1AndHttp2` (516-518); site DI via `SiteServiceRegistration.Configure` (:532); site pipeline maps only `MapZbMetrics` (:545), gRPC (:548), `MapZbLocalDbSync` (:555). No auth middleware and no FallbackPolicy exist on the site pipeline, so anonymous health needs nothing special.
### Task 1.1 — bump Health packages
`Directory.Packages.props:93-95`: all three `ZB.MOM.WW.Health*` 0.1.0 → **0.2.0**.
### Task 1.2 — site health checks (register between `:532` and `var app = builder.Build()` at `:534`)
Mirror the central registration style (`Program.cs:277-304`, `AddTypeActivatedCheck`):
1. **`akka-cluster`** (tags: Ready) — the shared `AkkaClusterHealthCheck` with `AkkaClusterStatusPolicy.Default`, exactly as central does at 282-286. The site ActorSystem→DI bridge already exists (`SiteServiceRegistration.cs:168-169`). This is also what carries `data.leader` for site pairs.
2. **`localdb`** (tags: Ready) — **new** check class in the Host (site namespace): opens the consolidated site SQLite (`LocalDb:Path`) and runs `SELECT 1`. There is no EF context on site (central's `DatabaseHealthCheck<ScadaBridgeDbContext>` does NOT apply — that context is central-only, `Program.cs:266`). Optionally enrich `data` with `ISyncStatus.Connected`/`OplogBacklog` (`SiteServiceRegistration.cs:139-140`) when replication is enabled — enrich only; replication-off must NOT fail the check (replication is default-OFF by design).
3. **`active-node`** (tags: Active) — **new** thin `SitePairActiveNodeHealthCheck` delegating to `IClusterNodeProvider.SelfIsPrimary` (registered on site, `SiteServiceRegistration.cs:172-178`; role-scoped `site-{SiteId}` oldest-Up). **Do NOT reuse central's `OldestNodeActiveHealthCheck`** — it calls `SelfIsOldest(cluster)` with no role scope (`OldestNodeActiveHealthCheck.cs:30`) and would compute oldest across the wrong member set. Follow the existing delegation precedent `SiteEventLogActiveNodeCheck` (`SiteServiceRegistration.cs:187-191`). Active semantics: primary → Healthy, standby → Unhealthy (503 = Standby to the dashboard — same convention as central).
### Task 1.3 — map endpoints
`app.MapZbHealth();` in the site pipeline after `UseRouting()` (:539), alongside `MapZbMetrics()` (:545). Served on `:8084` (Http1AndHttp2). No auth changes.
### Task 1.4 — tests
Extend `tests/ZB.MOM.WW.ScadaBridge.Host.Tests/`:
- Imitate `HealthCheckTests.cs` (`WebApplicationFactory<Program>`, `[Collection(HostBootCollection.Name)]`, `UseSetting("ScadaBridge:Node:Role", ...)`, `SkipMigrations=true`, registration assertions via `IOptions<HealthCheckServiceOptions>.Value.Registrations` + `registration.Factory(services)` type checks). **Caveat found in recon:** every existing factory test boots the Central role; there is no site-role factory bootstrap yet. Build one (Role=Site + minimal `ScadaBridge:Node` config + temp `LocalDb:Path`); if the full site host proves too heavy to boot in-test, fall back to the `CompositionRootTests.cs`/`SiteLocalDbWiringTests.cs` style over `SiteServiceRegistration.Configure` for registrations, plus a factory test for just the endpoint mapping. Include a positive control for any "central-only check absent on site" assertion.
- Assert: site maps `/health/ready` + `/health/active` (non-404); registrations exactly {akka-cluster→Ready, localdb→Ready, active-node→Active} with the intended check types; central registrations unchanged.
**Phase DoD:** full ScadaBridge suite green; on the local docker rig, `curl http://<site-node>:8084/health/ready` returns the canonical JSON with `akka-cluster` carrying `data.leader`, and `/health/active` splits 200/503 across the pair; central `/health/ready` also shows `data.leader` (free via the bump). PR merged to `main`.
---
## Phase 2 — OtOpcUa bump (+ optional family alignment)
### Task 2.1 — verify the `akka` check is the shared class
OtOpcUa registers ready checks `configdb`, `akka`, `admin-leader`, `localdb-replication` (registration reachable from `MapOtOpcUaHealth` / Program wiring, `src/Server/ZB.MOM.WW.OtOpcUa.Host`). **Verify** the `akka` check is `ZB.MOM.WW.Health.Akka.AkkaClusterHealthCheck`. If it is → nothing more. If bespoke → either swap to the shared check (preferred, same policy semantics) or populate the same `data` keys (`leader`/`selfAddress`/`selfRoles`/`memberCount`/`unreachableCount`) so the dashboard sees one contract.
### Task 2.2 — bump
`~/Desktop/OtOpcUa/Directory.Packages.props`: `ZB.MOM.WW.Health*` → 0.2.0. Build + run the Host/AdminUI test suites touched by health. Rig check: any admin node `/health/ready` shows `data.leader`.
### Task 2.3 (recommended) — alignment bumps, mxgw + HistorianGateway
Neither uses CPM (**no `Directory.Packages.props` — the known trap**): bump the inline `ZB.MOM.WW.Health` pins in `MxGateway.Server.csproj` and `ZB.MOM.WW.HistorianGateway.Server.csproj` to 0.2.0. No behavior change (no Akka checks there); keeps the family version matrix aligned. Skippable if either repo has unrelated in-flight work.
**Phase DoD:** OtOpcUa on 0.2.0 with leader data live on the rig; alignment bumps merged or explicitly deferred with a note.
---
## Phase 3 — the `ZB.MOM.WW.Overview` app
Location: `scadaproj/ZB.MOM.WW.Overview/` — **plain directory, NOT a nested git repo** (family rule). App conventions (copied from HistorianGateway, the reference implementation — cites from recon):
### Task 3.1 — scaffold
- `ZB.MOM.WW.Overview.slnx` with `/src/` + `/tests/` folders (LocalDb's slnx is the minimal template).
- `src/ZB.MOM.WW.Overview/ZB.MOM.WW.Overview.csproj`: `Sdk="Microsoft.NET.Sdk.Web"`, net10.0, **inline pinned versions, no CPM** (the app convention): `ZB.MOM.WW.Theme` 0.3.1, `ZB.MOM.WW.Health` 0.2.0, `ZB.MOM.WW.Telemetry` + `.Serilog` 0.1.0, `ZB.MOM.WW.Configuration` 0.1.0, `Serilog.AspNetCore` + console/file sinks. NO Auth/Audit/Secrets packages.
- `Directory.Build.props` with the family zero-warnings gate (`TreatWarningsAsErrors`, `AnalysisLevel=latest`, `EnforceCodeStyleInBuild`, `Deterministic`, Nullable/ImplicitUsings) — copy HistorianGateway's.
- `nuget.config`: copy HistorianGateway's (nuget.org `*` + `dohertj2-gitea` source with `ZB.MOM.WW.*` packageSourceMapping; credentials stay user-level, never committed).
### Task 3.2 — registry options + validation
`Registry/OverviewOptions.cs` (+ `ApplicationEntry`, `InstanceEntry`) binding section `Overview` per the design §3 schema: global `PollIntervalSeconds`(10)/`TimeoutSeconds`(3)/`StaleAfterSeconds`(45); per-application and per-instance overrides (instance wins); instance fields `Name`, `Group?`, `BaseUrl`, `ManagementUrl?`, `HasActiveRole`(false).
`Registry/OverviewOptionsValidator.cs` via `OptionsValidatorBase` + `AddValidatedOptions<OverviewOptions, OverviewOptionsValidator>` (`ZB.MOM.WW.Configuration`): non-empty applications, ≥1 instance each, unique app names + unique instance names within an app, absolute http/https `BaseUrl`/`ManagementUrl`, all intervals positive, **effective stale > effective poll for every instance**. Plus `ConfigPreflight.For(builder.Configuration).RequireValue("Overview:Applications:0:Name")…`-style presence check pre-`Build()` (HistorianGateway `Program.cs:62-66` is the pattern).
### Task 3.3 — health client + status model
`Polling/ZbHealthReportClient.cs`: GETs a URL, deserializes the canonical shape `{status, totalDurationMs, entries:{name:{status, description, durationMs, data?}}}` — `data` values as `JsonElement` (they arrive as arbitrary JSON; convert to string/int on read). Treat both 200 and 503 as *parseable* responses (503 carries a body too); classify non-JSON/undeserializable as Unreachable. In-app for v1 (design §8.2).
`Polling/InstanceStatus.cs`: `Up | Degraded | Down | Unreachable` (+ separate `ActiveState: Active|Standby|Unknown|NotApplicable`, `IsStale` computed at render). Derivation per design §5, including two-strike flap damping (state degrades only after 2 consecutive failures; recovery immediate).
### Task 3.4 — poller + snapshot store
`Polling/OverviewPollerService.cs : BackgroundService`: `PeriodicTimer(PollIntervalSeconds)`; each tick polls all instances in parallel (`Task.WhenAll`; per-instance `CancellationTokenSource` with its effective timeout; named `HttpClient` via `IHttpClientFactory`; no retries — next tick is the retry). Per instance: `/health/ready` always; `/health/active` when `HasActiveRole`.
`Polling/OverviewSnapshotStore.cs`: single immutable snapshot object swapped atomically (`Volatile.Write`/`Interlocked.Exchange`); each instance snapshot carries `LastUpdatedUtc`, latency, parsed entries, derived status, consecutive-failure counter. A C# `event`/`Action` notifies circuits to re-render on swap. **Pages never poll** — they read the store only (cache-first requirement).
Staleness: computed at render (`now - LastUpdatedUtc > effective StaleAfterSeconds` ⇒ render Stale regardless of stored status); the page also runs a small UI tick (~5 s) so ages/staleness advance without a poll event.
### Task 3.5 — leader aggregation
`Polling/LeaderResolver.cs`: for each `(Application, Group)` with any `HasActiveRole` instance, collect `entries["akka-cluster"].data.leader` from members whose status is Up/Degraded; resolve the Akka address (`akka.tcp://<system>@host:port`) to a registry instance by **host match against `BaseUrl`** (fall back to showing the raw address when unmatched — different ports are expected, the Akka port is not the HTTP port); if reporting members disagree → set the group's `LeaderDisagreement` flag (split-brain tell). Pure static logic → heavily unit-testable.
### Task 3.6 — UI
**Visual reference (match it):** `docs/mockups/overview-dashboard-mockup.html` — a static, self-contained mockup built from the real Theme tokens + embedded IBM Plex; open it in a browser before styling. It defines: the KPI strip treatment (alert/caution tints), the instance-card anatomy (status stripe on the top edge; **dashed border = Unreachable, solid = Down**; faded body + "stale since" for Stale), group headers with `Leader · <instance>` chips and the split-brain warning chip, footer `ready/active` raw-signal line, and the check-detail `<details>` layout with the monospace `data` line. Reuse Theme classes where they exist (`chip-*`, `panel`, `kv`, `agg-*`); the mockup's extra classes (`inst*`, `group*`, `check-*`) go into the app's own `site.css`.
Copy HistorianGateway's Blazor skeleton, minus every auth piece:
- `Components/App.razor` (head order matters: Bootstrap CSS → `<ThemeHead/>` → `site.css` → `HeadOutlet @rendermode=InteractiveServer`; body: `<Routes @rendermode=InteractiveServer/>` → Bootstrap JS → `<ThemeScripts/>` → `blazor.web.js`), `Routes.razor` with plain `RouteView` (no `AuthorizeRouteView`), `_Imports.razor` incl. `@using ZB.MOM.WW.Theme`.
- `Components/Layout/MainLayout.razor` → `ThemeShell Product="SCADA Overview" Accent="…"` (pick an unused accent; HG is `#2f855a`), Nav = single `NavRailItem Href="/" Match="NavLinkMatch.All"` (+ a Health link later if wanted), no AuthorizeView anywhere.
- `Components/Pages/Overview.razor` (`@page "/"`): header strip (counts n Up/Degraded/Down/Unreachable/Stale, last-refresh clock, pause/resume toggle — pause just stops applying poll events + lets staleness paint), then per-application sections: app name + worst-status rollup pill; cluster groups get a sub-header with `Leader: <instance>` chip (`StatusPill Info`) and the disagreement warning chip when flagged; instance grid of `InstanceCard`s.
- `Components/Widgets/InstanceCard.razor`: `TechCard` + `StatusPill` composition — name/group, status pill (mapping per design §5), Active badge, Leader badge, latency, relative last-seen ("stale since …" once stale), native `<details>` check list (per-check pill + description + data leader line for akka-cluster), `TechButton` (Ghost/Secondary) → `ManagementUrl` (`target="_blank" rel="noopener"`).
- Program wiring: `AddRazorComponents().AddInteractiveServerComponents()`; `MapRazorComponents<App>().AddInteractiveServerRenderMode()` — **no** `RequireAuthorization`, no antiforgery/cascading auth state; `UseStaticFiles()` before endpoints (Theme assets come from `_content/ZB.MOM.WW.Theme/`).
### Task 3.7 — self-observability
`AddZbSerilog(o => o.ServiceName = "overview")`; `AddZbTelemetry(o => { o.ServiceName = "overview"; o.Meters = [OverviewMetrics.MeterName]; })` — **`Meters` is a silent allowlist: forgetting the custom meter name silently drops the gauge** (known family gotcha). `OverviewMetrics`: observable gauge `zb_overview_instance_status` (labels app/instance/group; value = status enum ordinal) + poll-duration histogram. Register own checks: `registry-loaded` and `last-poll-cycle-completed` (Unhealthy if no completed cycle within 2× poll interval — makes the dashboard's own `/health/ready` honest). `MapZbHealth()` + `MapZbMetrics()`.
### Task 3.8 — tests (`tests/ZB.MOM.WW.Overview.Tests`)
xunit 2.9.3 + `xunit.runner.visualstudio` 3.1.4 + `Microsoft.NET.Test.Sdk` 17.14.1 + **bunit 1.40.0** (Theme's own test stack — imitate `StatusPillTests.cs`: `class X : TestContext`, `RenderComponent<T>`, class-list asserts):
- Validator table tests (incl. stale ≤ poll rejected, relative URL rejected).
- Health-client golden tests against **captured real payloads** from all four apps (capture during Phase 4 prep; until then use handcrafted fixtures matching `ZbHealthWriter` incl. entries with and without `data`).
- Status derivation: 200/Healthy→Up, 200/Degraded→Degraded, 503→Down, timeout/refused/garbage→Unreachable, flap damping (1 failure holds, 2 flips, recovery immediate), active 200/503/error → Active/Standby/Unknown.
- Staleness: fresh vs. aged snapshot with per-instance override.
- LeaderResolver: agreement, disagreement flag, address→instance host matching, unmatched fallback.
- Snapshot store: atomic swap + event fired.
- bUnit: `InstanceCard` renders pill class/badges/link per state; Overview page smoke over a stubbed store.
- `WebApplicationFactory<Program>` boot tests: page + `/health/ready` + `/metrics` reachable **with no auth**; malformed registry → boot fails with the `ConfigPreflight`/validator message.
### Task 3.9 — config + docker
- `appsettings.json`: Logging skeleton, empty `Serilog`/`Telemetry` sections (libs default them), `Overview` registry sample; `appsettings.Development.json`: Kestrel `Http` endpoint `http://localhost:5320` + a dev registry pointing at the docker rigs. **Derive the rig instance URLs from the compose files, don't guess:** OtOpcUa docker-dev rig → `~/Desktop/OtOpcUa` docker-dev compose (6 nodes; AdminUI reachable on host :9200 — per-node published HTTP ports are in the compose port mappings); ScadaBridge local cluster → `~/Desktop/ScadaBridge/docker/` compose (central pair UI ports + site-node :8084 mappings); mxgw → `windev:5130`; HistorianGateway → its docker compose (:5220) or `wonder-app-vd03:5222` (VPN). Production Kestrel via env (`Kestrel__Endpoints__Http__Url`), the HG pattern; **don't set `ASPNETCORE_URLS` alongside Kestrel endpoint config** (URLs-override trap).
- `docker/`: HG-pattern runtime-only image — host-side `dotnet publish -c Release -o docker/publish -p:UseAppHost=false` (host has the Gitea feed creds), `FROM mcr.microsoft.com/dotnet/aspnet:10.0`, `COPY publish/ ./`, `EXPOSE 5320`, `ENTRYPOINT ["dotnet","ZB.MOM.WW.Overview.dll"]`; compose project name `zb-overview`, port `5320:5320`, `ASPNETCORE_ENVIRONMENT` + Kestrel env vars. No env_file needed (no secrets). Note: `aspnet:10.0` has no `curl` — healthcheck via `wget`/dotnet or skip.
**Phase DoD:** `dotnet build` 0 warnings; `dotnet test` green; `dotnet run` on :5320 renders the page instantly from cache against a dev registry; container builds and runs.
---
## Phase 4 — live acceptance (design §9)
Prep: capture real `/health/ready` + `/health/active` payloads from all four apps (post-0.2.0) into the golden-test fixtures. Then run the design §9 checklist end-to-end on the rigs (OtOpcUa 6-node docker-dev + ScadaBridge local cluster + windev mxgw + wonder-app-vd03 HG if reachable/VPN):
1. Every registered instance renders correctly; single Active per pair (central AND site).
2. **Leader chip correct per cluster group and moves when the leader node is stopped**; split-brain flag not shown in normal operation.
3. Kill a node → Unreachable within ~2 cadences; restart → recovers.
4. Force a Degraded check → Warn with failing entry visible.
5. Cache-first: first page load instant, no poll triggered by the request.
6. Pause auto-refresh past `StaleAfterSeconds` → all cards Stale; resume → clean recovery. Also stop the poller (kill the service loop) → same.
7. Management links open the right UIs; page reachable with **no login**.
8. Malformed registry → boot fails with a clear preflight message.
9. `curl :5320/health/ready` + `/metrics` (gauge present with per-instance labels).
**Program DoD:** all 9 pass; ScadaBridge PR merged; OtOpcUa bump merged; Health 0.2.0 on the feed; scadaproj commits for the lib + new app + docs.
---
## Cross-cutting notes for the executor
- **Family gotchas that WILL bite:** `ZbTelemetryOptions.Meters` silent allowlist; mxgw/HG have no `Directory.Packages.props` (inline pins); never host-`sqlite3` a live container WAL DB during rig checks; `Akka.TestKit.Xunit2` is xunit-v2-only (fine here — everything in this plan is xunit 2.9.3); Gitea feed verification uses `/api/packages/dohertj2/nuget/registration/…`.
- **Do not** add auth "while at it" — anonymous read-only is a requirement, not an omission.
- **Contract discipline:** the `data` field is additive and optional; the dashboard must render fully (minus leader) against 0.1.0-era payloads — that's what keeps Phase 3 unblocked by Phases 1/2 and tolerant of un-bumped deployments.
- Commit style: lib change + publish in scadaproj (`feat(health): 0.2.0 …`); ScadaBridge and OtOpcUa changes as PRs in their own repos referencing this plan; new app committed in scadaproj.