e538ea36e33d8d61d5b15cf6ede4e4a73fa2a34f
12 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Joseph Doherty
|
c72271aa8a |
docs: split per-library history out of the root index into ZB.MOM.WW.*/CLAUDE.md files
Claude-Session: https://claude.ai/code/session_01Wzg6NUCWHipSexLs2cgRFv |
||
|
Joseph Doherty
|
846b77f5b4 |
feat(overview): show historian session-pool depths on the HistorianGateway card
Asked whether the gateway's new session row also covered HistorianGateway. It did not, and could not: HistorianGateway publishes its pool depths on a different TIER with a different KEY SET. mxaccessgw active tier total / ready / faulted one check historiangw READY tier warmSessions / pool / … TWO checks The session row is located by the total/ready/faulted triple, so it correctly ignored these rather than half-matching them. This adds a sibling row instead of unifying the two behind a shared shape, because there is no such shape - inventing one would mean rendering a contract neither app publishes. No poller change was needed: the ready tier's body is already retained, so this is entirely a rendering change. Labelled "Sessions pooled", deliberately not "connections". warmSessions is pool DEPTH - warm pre-authenticated sessions held ready - and sessions are leased on demand, so the event pool legitimately sits at 0 until something reads events. Calling that a connection count would make a normal idle pool read as an outage, so 0 is shown plainly and the row goes red only on consecutiveFactoryFailures, which is an actual fault. Pools render read-then-event rather than in dictionary order, so the line is stable across polls and the pool an operator cares about comes first. 188 tests (183 + 5): ordering, that an idle pool is not painted red, that a failing factory is, a positive control on the omission assertion, and one pinning that neither app's row can render the other's data. |
||
|
Joseph Doherty
|
da276ff15a |
feat(overview): show MXAccess session counts on the gateway card
The gateway publishes its live MXAccess session counts, and the dashboard could not reach them. Four things stood between the data and the card, and only the last one is cosmetic: The active tier was probed only for instances declared HasActiveRole. A standalone gateway has no active/standby role, so its active tier was never requested at all. The tempting fix is to declare mxgw HasActiveRole, and it is wrong: that flag also paints an Active/Standby badge, so it would buy the data by asserting a role that does not exist. The probe is now unconditional for any reachable instance while ObserveActive stays role-gated, separating "what does this tier report" from "am I the active half". The active probe's parsed body was then discarded - only ready.Report was kept - so InstanceSnapshot gains ActiveReport. The two tiers stay separate rather than merged, because a check name is only unique within a tier. The card's one data-rendering path, ClusterDataLine, is located by the Akka leader/activeNode keys and would ignore session data even once present. The new row follows that helper's own precedent and locates the check by the data it carries - the total/ready/faulted triple - not by name, requiring all three so a future check publishing a lone "total" cannot be mistaken for it. Rendered on the card FACE, not inside the collapsed check list: a count behind a disclosure triangle is one nobody reads. Zero is shown rather than hidden, since a gateway holding no sessions is the interesting case and suppressing it would make the row appear only when the news is good. Cost, stated plainly: every reachable instance now makes one extra HTTP request per poll. The unreachable-instance skip is untouched, so the worst-case sweep duration - which is set by timeouts, not by healthy round-trips - is unchanged. Four cadence tests were counting raw requests as a proxy for "instances polled"; they now count ready-tier probes, which is what they always meant and is robust to tier changes. The Wonder registry comment claiming the active tier ran an empty check set was true when written and stopped being true when the gateway added the check; it now warns against the HasActiveRole shortcut instead. 183 tests pass (177 + 6), including a positive control on the absence assertion so "row not rendered" cannot pass vacuously. |
||
|
Joseph Doherty
|
4f97e72ac9 |
fix(overview): wonder registry — 127.0.0.1 BaseUrls so leader names cannot resolve wrongly
Live on the box, the ScadaBridge *site* card claimed "Leader · central". Central and site are separate Akka clusters, so that name was simply wrong. Cause is in the registry, not the app. ResolveToInstanceName matches a reported Akka leader address to an instance on HOST ONLY - correct by design, because an app's Akka remoting port and its HTTP port differ, so comparing ports would never match. It assumes what holds on the docker rig and not here: distinct hosts per instance. All five instances are on one box, and the scan is fleet-wide in registry order, so with every BaseUrl on "localhost" the first matching entry won the chip for every group. No host value makes this resolve CORRECTLY, because the instances really are co-located. So make it never match instead: ScadaBridge's Akka addresses use host "localhost" and OtOpcUa's uses the FQDN, and "127.0.0.1" is string-unequal to both. Every chip now falls back to the raw Akka address - unambiguous and always true. A raw address beats a confident wrong name. Using the FQDN for OtOpcUa's BaseUrl would have resolved its chip correctly and was rejected: that name resolves to a link-local IPv6 on this box, and routing the real health probe through it to improve a label trades the primary function for a cosmetic one. ManagementUrl values are unchanged - they are followed by the operator's browser from another machine and must keep the box address. |
||
|
Joseph Doherty
|
c582508ae5 |
feat(overview): wonder-app-vd03 registry overlay
Adds appsettings.Wonder.json for the first deployment of this dashboard to a real box. Named "Wonder" rather than "Production", mirroring the existing Docker overlay: this is ONE deployment's fleet, not a claim about what every production deployment looks like. Every port was established by probing the running box, not inferred from the docker rig - and the two disagree, so inference would have been wrong: OtOpcUa :9000 (AdminUI on the same listener) ScadaBridge Central :8085 (rig serves this on :5000) ScadaBridge Site :8084 (no UI - / is 404, so the card carries no link) MxAccessGateway :5130 (:5120 is gRPC) HistorianGateway :5220 (:5221 is gRPC h2c) :8083 and :8086 answer 400 to an HTTP/1.1 GET - h2c gRPC - and are excluded. Both gateways are HasActiveRole:false. Their /health/active does return 200, but only because the active tier runs an EMPTY check set when no Active-tagged check is registered, so an Active badge would assert a role neither has. Central and site are separate groups because they are separate Akka clusters; one shared group would manufacture a leader disagreement. BaseUrl is localhost (probed by the service, on the box) while ManagementUrl carries the box address (followed by the operator's browser, elsewhere). Serilog's file path is absolute and outside the Server directory an upgrade replaces - a relative path resolves against the service AppDirectory, so every upgrade would silently eat the log history. Same class of defect that cost HistorianGateway its secrets store on a swap. Boot-verified locally under ASPNETCORE_ENVIRONMENT=Wonder: registry passes OverviewOptionsValidator and the host starts. |
||
|
Joseph Doherty
|
8f31184b7d |
chore(overview): bump Theme 0.3.1 -> 0.4.1, Health 0.2.0 -> 0.3.0
Family alignment ahead of the first wonder-app-vd03 deploy. Both bumps are
inert for this app, established rather than assumed:
- Theme 0.3.1 -> 0.4.1 is two commits: the upstreamed Bootstrap .btn sizing
block and the rail-btn-block modifier. Overview answers NO to both of the
questions the kit README says to ask - it declares no .btn sizing of its own
and has ZERO call sites carrying a bare `btn` token (its single button is
class="rail-btn"), so the new .btn rule matches nothing here. The .rail-btn
refactor that introduced --rail-btn-inset resolves to the same 0.6rem the
literal margin used, so the one call site is unchanged. Same answer as
HistorianGateway. Note a naive \bbtn\b grep is a FALSE POSITIVE on
"rail-btn" - the hyphen is a word boundary; match a whitespace-delimited
token instead.
- Health 0.2.0 -> 0.3.0 is a pure version-number alignment: the core
ZB.MOM.WW.Health package is byte-identical between the two, with the entire
delta confined to ZB.MOM.WW.Health.Akka (ActiveNodeHealthCheck,
ClusterActiveNode, AkkaActiveNodeGate), which this app does not reference.
Verified with a positive control on the diff path. The 0.2.1 active-tier
503 change therefore cannot reach this app's own endpoints; it registers no
Active-tagged check, so /health/active runs an empty set and stays 200.
Overview consumes the 0.3.0 `activeNode` data key over the WIRE from other
apps' health JSON, which is independent of the package it compiles against -
LeaderResolver already ranked it (
|
||
|
Joseph Doherty
|
344bdc4da5 |
fix(overview): rank activeNode over leader for the Active chip
The dashboard derived every leader chip from AkkaClusterHealthCheck's `leader` data — ClusterState.Leader, the LOWEST-ADDRESSED Up member. That is not the node in charge. The node holding the singletons, gated for writes, and answering 200 on /health/active is the OLDEST Up member. The two agree only until some node restarts, after which the restarted node rejoins youngest but keeps its address. Observed live on the rebuilt OtOpcUa rig during the Health 0.3.0 work: leader central-1, active node central-2. The dashboard would have put the Active chip on central-1 while /health/active said central-2 — the chip contradicting the tier it exists to visualise. The Phase 4 acceptance run missed it because the rig had just been formed, the one state in which the two orderings agree. Health 0.3.0's active tier now publishes `activeNode` on every member, including standbys, so the fix is to prefer it and keep `leader` as the fallback for an instance on an older Health or whose active tier was not probed. The instance card now shows `active` and `leader` side by side rather than swapping one for the other: they are legitimately different nodes, and showing both turns a silent contradiction into something readable at 3am. Tests: 177 (+3) — precedence when the two disagree, a standby naming the active node rather than itself, and the fallback still working with no activeNode. |
||
|
Joseph Doherty
|
d95292c95d |
fix(overview): locate the cluster view by data key, not by check name
Found by the live rig, and invisible to every unit test that existed.
LeaderResolver and InstanceCard.ClusterDataLine both looked the cluster view up
under the check name "akka-cluster". ScadaBridge registers the shared
AkkaClusterHealthCheck under that name; OtOpcUa registers the SAME check as
"akka". So once the OtOpcUa rig was rebuilt on Health 0.2.0 and started
publishing data.leader, its groups still rendered with no leader chip and no
evidence line - and a group with no votes is a legitimate state, so nothing
looked wrong. ScadaBridge's four groups were fully populated the whole time,
which is what made it look like the feature worked.
Both now find the entry by the DATA KEY ("leader"), which is the part of the
contract the shared check actually owns rather than the part each consumer names
for itself.
Regression tests: the leader is found under akka-cluster, akka, and an arbitrary
future name; and an entry that publishes data WITHOUT a leader (ScadaBridge's
localdb check, whose data carries replication counters) does not get picked in
preference to the one that answers the question. Every prior fixture used
ScadaBridge's name, which is precisely why this got through.
After the fix all seven cluster groups render a leader chip with zero
split-brain flags - design section 9 check 2 now passes for BOTH products.
174 tests, 0 warnings.
|
||
|
Joseph Doherty
|
41013dcab3 |
fix(overview): call UseStaticWebAssets unconditionally; record Phase 4 results
Found during the live acceptance run. The host calls UseStaticWebAssets only when the environment is literally "Development", so running the build output under any other name - checking a staging registry locally, say - leaves the Theme RCL's assets under _content/ unresolvable. The failure is silent and unhelpful: the page returns 200 with its markup fully present and renders as a blank white screen, because every stylesheet 404s. Calling it unconditionally is a no-op once published (the manifest is gone and the assets are real files on disk), so the only thing it changes is that failure. The container was never affected. Phase 4 results recorded in the ledger: 7 of 9 design section 9 checks fully pass, 2 pass for ScadaBridge and are blocked for OtOpcUa pending a rig redeploy. Notable live evidence: - Leader chip moved from site-c-a to site-c-b when site-c-a was stopped, and the split-brain chip appeared while the pair was self-formed then cleared once it genuinely joined. - Kill to Unreachable took two failing sweeps (the damping working as designed); recovery landed on the first good poll. - Five consecutive page loads in 0.00-0.02s all showed the same "last sweep" timestamp, which is cache-first stated as a measurement. - Freezing the probed endpoints while auto-refresh stayed RUNNING froze the poller mid-sweep and the cards aged into STALE on their own - staleness is computed at render, so a stuck poller needs no explicit stall detection. The run also surfaced a pre-existing rig defect in both products: self-first Akka seed lists mean a pair booted cold self-forms two 1-member clusters that never join. After a --force-recreate every ScadaBridge node named ITSELF leader and every node answered /health/active 200; restarting ONE node of each pair converged them to one Leader plus one Active plus one Standby. That is the user's own open backlog item (docs/plans/2026-07-22-initjoin-selfform-fallback.md, unexecuted), and the dashboard made it visible fleet-wide in one screen. 170 tests, 0 warnings. |
||
|
Joseph Doherty
|
92997b40dc |
feat(overview): self-observability, config and the container image
Closes Phase 3 of the overview-dashboard plan (tasks 3.7-3.9).
Self-observability
- OverviewMetrics: an observable gauge read from the published snapshot at scrape
time, so the metric and the page cannot disagree about what the dashboard
believes, plus a sweep-duration histogram. Instrument names follow the family
METRIC-CONVENTIONS spec (overview.instance.status / overview.poll.duration,
unit s) rather than the plan's literal zb_ names; Prometheus renders them as
overview_instance_status and overview_poll_duration_seconds. Labels are
application/node/group - "node", not "instance", because Prometheus stamps its
own instance label on every scraped series and a colliding metric label is
silently renamed exported_instance. Instances never probed emit nothing: the
enum's zero ordinal is Up, so a placeholder would publish the single most
reassuring answer about a node nobody has heard from.
- Two readiness checks make the dashboard's own /health/ready mean something:
registry-loaded (reports the counts it bound) and last-poll-cycle-completed,
which catches the failure where Kestrel is fine, the page still renders, and
every card on it is quietly frozen. The heartbeat is stamped by the sweep, not
derived from the snapshot's timestamp - the poller publishes the registry
before it probes anything, so a stuck poller would otherwise look healthy.
No cycle yet is Degraded inside the grace window and Unhealthy past it, so a
restart does not fail readiness for its first seconds and a poller that never
started cannot hide behind "still starting".
Configuration
- appsettings.json ships tuning defaults with an EMPTY Applications list, so a
deployment that supplies no registry fails at ConfigPreflight naming the key
instead of booting into a dashboard of nothing.
- Three registries, because reachability genuinely differs: Development is the
host-reachable subset (4 instances), Docker is the full 16-instance fleet by
container DNS name. The rigs publish almost no per-node HTTP port to the host
- OtOpcUa publishes none (only Traefik :9200, load-balanced across the central
pair, so it cannot identify a node) and ScadaBridge keeps site :8084
container-internal. Ports were established by probing the running rigs, not
read off the compose files, because the two disagree: OtOpcUa site nodes serve
health on :8080, not the :9000 their compose anchor implies.
- Real Serilog sinks, not the family's empty "Serilog": {}. AddZbSerilog drives
sinks entirely from configuration and adds none of its own, so the empty
section produces an app that logs NOWHERE - verified against the running
HistorianGateway container, whose log stream is completely empty for exactly
this reason. The HttpClient level override matters too: 16 instances on a 10s
cadence emit several Information lines a second of pure probe noise.
Container
- HistorianGateway-pattern runtime-only image; the compose stack joins the three
rig networks as external, so bringing the dashboard up or down cannot disturb
a rig. The Dockerfile clears ASPNETCORE_HTTP_PORTS as well as ASPNETCORE_URLS:
on .NET 8+ the base image declares its binding via the former, and clearing
only the latter leaves Kestrel logging "Overriding address(es)" every boot.
No healthcheck - aspnet:10.0 has neither curl nor wget, and adding a
--healthcheck entry point to production code to satisfy Compose is not a
trade worth making.
Tests
- BootTests boot the real Program.cs: every endpoint anonymous, the page
rendering from cache while nothing it watches is reachable, the Meters
allowlist proven by an end-to-end /metrics scrape, and both refuse-to-boot
paths (missing registry, stale window inside the poll interval).
The registry is supplied via environment variables, not
ConfigureAppConfiguration, because ConfigPreflight runs before Build() - when
the factory's callbacks are applied - and would not see them.
170 tests, 0 warnings. Verified live against the running rigs: 16 cards, 16
gauge series, page and gauge agreeing exactly.
|
||
|
Joseph Doherty
|
5fe7135e2c |
feat(overview): poller, leader aggregation and the dashboard UI
Tasks 3.4-3.6 of the overview-dashboard plan. Polling (3.4/3.5): - OverviewPollerService: one timer at the fastest configured cadence, each target polled when its own interval is due, fan-out per sweep, no retries (the next tick is the retry, so damping stays the only place that decides whether a failure is worth showing). - OverviewSnapshotStore: atomic swap of a complete immutable graph plus a change event. Pages read it and never probe — the cache-first requirement. - LeaderResolver: per-group leader from akka-cluster data, with the split-brain flag scoped to members that are actually answering, so an ordinary failover is not misread as a disagreement. UI (3.6): ThemeShell + StatusPill/TechCard composition against the mockup; instance cards carry the status stripe (dashed for Unreachable), the check detail list and the raw ready/active signal line. Two defects found by verification rather than by review: - The active tier answers with a status code and an EMPTY body, but the client demanded parseable JSON on every probe — so a healthy pair rendered as "role unknown" instead of Active/Standby. Split into ProbeAsync (ready tier, body is the payload) and ProbeStatusAsync (active tier, code is the answer). Caught by a live smoke run against two fake health endpoints; the poller tests now serve an empty body so they hold the fix. - SweepAsync published even when cancellation was already requested, leaving a half-probed registry as the store's final state on shutdown. It now checks the token itself rather than relying on the transport to throw. Also: app.UseAntiforgery() is required despite the anonymous-by-design pipeline — AddRazorComponents stamps antiforgery metadata unconditionally and the endpoint middleware hard-fails without it (every page was 500). Chosen over DisableAntiforgery() so a future form is protected by default. 147 tests, 0 warnings. Live-verified end to end against fake endpoints: Up/Unreachable/Active/Standby, leader chip, cluster-data line, KPI counts. |
||
|
Joseph Doherty
|
d0110cf406 |
feat(overview): scaffold the family dashboard + registry + health client
Phase 3 tasks 3.1-3.3 of docs/plans/2026-07-22-overview-dashboard-impl-plan.md. Scaffold (3.1): a plain directory in scadaproj (NOT a nested git repo), slnx over src/ + tests/, HistorianGateway-pattern Directory.Build.props (warnings-as-errors) and nuget.config (nuget.org * + the Gitea feed scoped to ZB.MOM.WW.*). Inline package pins, no CPM — the app convention here. NO Auth/Audit/Secrets packages: the dashboard is anonymous and read-only by requirement. Registry (3.2): OverviewOptions/ApplicationEntry/InstanceEntry bound from the "Overview" section, plus EffectiveTimings resolving instance > application > global overrides in ONE place so no caller re-implements the precedence. OverviewOptionsValidator (OptionsValidatorBase + AddValidatedOptions) rejects an empty registry, applications with no instances, duplicate names, non-absolute or non-http(s) URLs, and non-positive intervals — and checks stale > poll on the EFFECTIVE values, since an override at either level can invert that on one instance while the globals look fine. A ConfigPreflight presence check runs before the host is built so a missing registry fails with the key name, not with "Applications must have at least 1 entry". Health client (3.3): ZbHealthReport DTOs for the canonical ZbHealthWriter body, with per-entry `data` kept as JsonElement — it is an open contract, and binding it to concrete types would break the moment a check adds a field. 200 AND 503 are both parseable answers (503 carries the body naming the failing check); only a transport failure or a non-canonical body is a failed probe, which is why Unreachable stays distinct from Down. Host shutdown propagates as cancellation rather than being recorded as every instance timing out. Status model: Up/Degraded/Down/Unreachable + Active/Standby/Unknown/NotApplicable, with two-strike flap damping — a failing observation only replaces a good state after 2 consecutive failures, while recovery is immediate (damping suppresses false alarms; symmetric damping would just make them linger). 53 tests green, 0 warnings in Debug and Release. Tests for the validator, the client golden payloads (with AND without `data`, so a partly-bumped fleet renders) and the derivation table are front-loaded here from task 3.8 so this batch is verified rather than pending. Also pins AngleSharp 1.5.2 test-side: bunit 1.40.0 pulls 1.2.0, which trips NU1902 and therefore the warnings-as-errors gate (same fix HistorianGateway made in 6bc005d). |