docs(overview): family overview dashboard — options, design, impl plan, mockup

Options investigation (Aspire standalone/HealthChecks.UI/Gatus/Homepage/ Grafana vs custom) → decision: custom ZB.MOM.WW.Overview Blazor app in the family look. Design: anonymous read-only single pane, appsettings registry of all four apps' instances, cache-first poller on a configurable timer, configurable staleness timeout, Active/Standby from /health/active, and per-cluster Akka leader chips (needs Health 0.2.0 optional per-entry data). Impl plan: phased + code-verified — Phase 0 Health 0.2.0 (writer data field + AkkaClusterHealthCheck cluster view), Phase 1 ScadaBridge site-node MapZbHealth PR (:8084; role-scoped active check), Phase 2 OtOpcUa bump, Phase 3 the app (:5320), Phase 4 live rig acceptance. Mockup is the visual reference: real Theme tokens + embedded IBM Plex, all card states incl. stale + split-brain. Ready to execute in a fresh session.
docs(akka-msquic): 1.6 status update — Artery-TCP implementation began 2026-07-03
2026-07-22 09:22:13 -04:00 · 2026-07-22 06:49:39 -04:00 · 2026-07-22 06:41:36 -04:00
6 changed files with 1524 additions and 0 deletions
@@ -0,0 +1,319 @@
 # Deep dive: an MsQuic transport for Akka.NET 1.5.62
 *Research report, 2026-07-22. Question: what would it take to implement the Akka.NET 1.6 MsQuic
 transport (akka.net #7466) against the Akka.NET version currently used by ScadaBridge and OtOpcUa,
 and would features like higher throughput and larger-message support reduce the family's reliance
 on gRPC?*
 **TL;DR:** Building a QUIC transport against the current Akka.NET (1.5.62) is genuinely feasible —
 the classic transport SPI is public, and Akka's own docs literally use a hypothetical QUIC transport
 as the custom-transport example. Roughly 3–5 weeks to production quality. But there's a structural
 catch: plugged in under the classic 1.5 remoting pipeline, QUIC is reduced to "TCP with TLS 1.3 and
 cheaper handshakes." The two things actually wanted — throughput multiplexing and non-blocking large
 messages — come from the *Artery pipeline rewrite above the transport*, which is the part of #7466
 that doesn't exist yet and can't be replicated through the 1.5 SPI. And most of the family's gRPC
 wouldn't go away regardless, because most of it isn't Akka-adjacent. Details below, options and a
 recommendation at the end.
 ---
 ## 1. Where #7466 actually stands (checked 2026-07-22)
 - [Issue #7466](https://github.com/akkadotnet/akka.net/issues/7466) is the **v1.6 "Artery" epic**,
  opened Jan 2025: MsQuic-based Remoting V2 replacing the never-built Aeron plan. Design goals:
  per-stream multiplexing ("this is where all of the significant performance gains come from" per
  the [Petabridge v1.6 roadmap](https://petabridge.com/blog/akkadotnet-v1.6-roadmap/)), mandatory
  TLS 1.3, automatic chunking of large messages onto dedicated streams, a system-message channel
  that works even while quarantined, Akka.Streams backpressure, actor-ref compression, and a
  2–3M msg/sec target. Requires .NET 9+.
 - **Status: design-only.** No linked PRs, no branches, no assignee; the only substantive activity is
  a January 2025 design discussion (to11mtm/Aaronontheweb) about channel counts, ordering
  guarantees, and serialization. The issue itself says a protocol spec is needed before
  implementation starts.
 - **The timeline has slipped badly.** The roadmap targeted alphas end of Q1 2025 and completion
  early Q1 2026. As of July 2026 the
  [releases page](https://github.com/akkadotnet/akka.net/releases) shows **zero 1.6 previews** —
  latest is 1.5.70 (July 3, 2026), and a 1.5.71 milestone is open. The
  [1.6.0 milestone](https://github.com/akkadotnet/akka.net/milestones) reads 44% complete, but the
  closed items are the AOT/trimming/serialization workstreams, not the transport. A
  [PR search for "quic"](https://github.com/akkadotnet/akka.net/pulls?q=is%3Apr+quic) confirms no
  transport code has ever landed.
 So "wait for 1.6" is not a near-term plan. QUIC benefits this year would mean building it in-house.
 > **STATUS UPDATE (2026-07-22, same day, deeper look):** the two paragraphs above are correct about
 > *QUIC* but stale about *Artery* — searching PRs for "quic" missed work named "artery".
 > **Artery implementation began 2026-07-03 and is landing rapidly on `dev`, as Artery-TCP** (the
 > JVM-parallel move): design spec via openspec (#8313, "artery-tcp-remoting"), transport-substrate
 > validation gate G0 (#8315), TCP framing + envelope codec G1 (#8319), handshake/association G2
 > (#8320), parallel inbound/outbound lanes (#8356/#8357, Jul 11), dedicated large-message stream
 > streamId 3 (#8352, Jul 10), quarantine port in flight (#8416), MNTR/CI integration for both
 > transports (#8373/#8413). 192/192 Artery specs passing as of #8352. In parallel,
 > **Serialization.V2** (source-generated + MessagePack internals) is in heavy development — ~30
 > merged PRs with wire-format snapshot tests and an open cross-component integration proof suite
 > (#8407). TCP-first also confirms §7's forward-looking note: the Artery benefits (lanes,
 > large-message stream, control stream) are arriving **transport-agnostic over TCP** — no libmsquic
 > required — with QUIC as a later substrate swap. No preview release or public benchmark numbers
 > yet; see the 1.6-status section of the conversation record for the remaining-work list.
 ## 2. What the projects run today
 Both ScadaBridge and OtOpcUa pin **Akka 1.5.62** and use the classic **`dot-netty.tcp`** transport,
 plaintext, with all size defaults — meaning the **128 KB `maximum-frame-size` cap** applies
 (`ScadaBridge/src/ZB.MOM.WW.ScadaBridge.Host/Actors/AkkaHostedService.cs:310`;
 `OtOpcUa/src/Core/ZB.MOM.WW.OtOpcUa.Cluster/Resources/akka.conf:25`). Neither config touches TLS or
 frame size. The Akka traffic is cluster gossip, heartbeats, singleton/pub-sub coordination, and
 (ScadaBridge) actor messaging in a hub-and-spoke — modest volume, nowhere near transport-bound.
 Two facts that matter for the "do we even need this" question:
 - **The frame size can be raised today.** `akka.remote.dot-netty.tcp.maximum-frame-size = 10MiB`
  (plus matching send/receive buffer sizes) is a config change, not a transport change. The 128 KB
  default is a *guardrail*, not a TCP limitation — large messages on the single ordered association
  channel delay heartbeats behind them, which is exactly the false-failure-detection risk the
  2s/10s failure-detector tuning cares about.
 - **TLS on the existing transport is already available.** Akka.Remote's DotNetty transport gained
  **mutual TLS** in [PR #7851](https://github.com/akkadotnet/akka.net/pull/7851) (merged Oct 2025 —
  included in the pinned 1.5.62). If encryption-in-flight is a driver, QUIC is not required for it.
 ## 3. How a transport plugs into 1.5 — and what it would take
 Akka.Remote has a public custom-transport SPI
 ([docs](https://getakka.net/articles/remoting/transports.html)): implement
 `Akka.Remote.Transport.Transport`, register it via HOCON, and everything above it (handshake,
 endpoint management, quarantine, cluster) is unchanged:
 ```hocon
 akka.remote {
  enabled-transports = ["akka.remote.zb-quic"]
  zb-quic {
    transport-class = "ZB.MOM.WW.Akka.Transport.Quic.QuicTransport, ZB.MOM.WW.Akka.Transport.Quic"
    transport-protocol = quic     # addresses become akka.quic://System@host:port
    hostname = "0.0.0.0"
    port = 4053                   # UDP now, not TCP
  }
 }
 ```
 The concrete build, as a shared lib in scadaproj (the usual pattern):
 **Core (~1–2 weeks):**
 - `QuicTransport : Transport` — ctor `(ActorSystem, Config)` (instantiated reflectively); implement
  `SchemeIdentifier` (`"quic"`), `MaximumPayloadBytes` (configurable — this is where "large
  messages" becomes trivial), `Listen()` (a `QuicListener.ListenAsync` with a private ALPN like
  `"akka-remote"`, surfacing `InboundAssociation` events), `Associate(Address)`
  (`QuicConnection.ConnectAsync` + open one bidirectional `QuicStream`), `Shutdown()`.
 - `QuicAssociationHandle : AssociationHandle` — `Write(ByteString)` must be **non-blocking with a
  boolean backpressure signal** (return `false` when the send queue is full; Akka's `EndpointWriter`
  buffers and retries — getting this semantic right is the subtle part), `Disassociate()` (map to
  `CloseAsync` with an app error code).
 - **Framing** — QUIC streams are byte streams like TCP, so the 4-byte length-prefix framing that
  dot-netty does must be re-implemented on both sides. The Akka protocol handshake/PDU layer rides
  on top automatically.
 **Security config (~0.5–1 week):** QUIC **mandates TLS 1.3** — there is no plaintext mode. Every
 node needs a certificate, and a validation policy is required (mutual TLS with a pinned internal CA
 is the sane one; the Secrets lib is a natural home for cert material). This is new operational
 surface the rigs don't have today.
 **Hardening + tests (~1–2 weeks):** association teardown races, quarantine behavior, reconnect
 storms, failure-detector behavior under load, big-frame tests, and soak on the docker-dev rig.
 Historical note for calibration: the Helios→DotNetty transition took Akka.NET years of corner-case
 fixes (dissociation races, quarantine storms). A custom transport is its own support island for
 that class of bug.
 **Migration:** the address scheme changes (`akka.tcp://` → `akka.quic://`), so seed nodes and every
 stored address change; mixed-transport rolling upgrade isn't practical for a pair — but *both*
 transports can be listed in `enabled-transports` during transition, and the site pairs already
 stop/start together by policy, so cutover is easy here.
 ## 4. The structural catch
 The classic SPI models an association as **one ordered, opaque byte channel**. The remoting layer
 above it (system-message resend, handshake, cluster failure detection) *depends* on per-association
 ordering, and the transport never sees message recipients — just framed byte blobs. Consequences:
 - **No multiplexing win.** One association cannot be fanned across N QUIC streams without either
  breaking ordering guarantees or parsing Akka's PDU envelope inside the transport to route by
  recipient while preserving per-pair ordering — at which point you are rewriting Artery's
  dispatcher against internals with no stability contract.
 - **No dedicated large-message or control channel.** A 10 MB frame still queues ahead of heartbeats
  on the same stream — identical head-of-line behavior to TCP today. The 1.6 design fixes this
  *above* the transport; the 1.5 SPI can't express it.
 - What you *do* get: TLS 1.3 built in, faster connection establishment (nice for
  reconnect-after-failover, though failover latency is dominated by the 10–15s
  failure-detector/downing windows, not handshakes), connection migration, and an easy big
  `MaximumPayloadBytes`.
 Net: a 1.5-SPI QUIC transport ≈ **dot-netty.tcp + mTLS + a raised frame size** — both of which are
 available today with config on the stock transport.
 ## 5. What 1.6 actually changes — the layer split, made explicit
 1.6 doesn't just swap the transport; it rewrites the remoting pipeline *above* the transport
 (Artery), and that pipeline is where every benefit §4 marked "not achievable on 1.5" actually lives.
 **In 1.5**, remoting is two layers: the pluggable `Transport` SPI at the bottom (a dumb ordered byte
 pipe — `Associate()`, `Write(ByteString)`, `Disassociate()`) and above it the fixed pipeline:
 `AkkaProtocolTransport` handshake, one `EndpointWriter`/`EndpointReader` actor pair per association,
 system-message resend buffering, and quarantine. That pipeline bakes in three assumptions no custom
 transport can change:
 1. **One association = one ordered channel** — the system-message resend protocol and cluster
   failure detection depend on strict ordering over that single channel.
 2. **Payloads are opaque to the transport** — recipient, message class, and size are invisible by
   the time bytes reach `Write()`, so "big message → own lane" or "heartbeat → priority lane"
   routing is impossible without parsing Akka's internal PDU format (i.e. reimplementing the layer
   above against unstable internals).
 3. **Everything shares the lane** — user messages, system messages, cluster heartbeats. The 128 KB
   frame guardrail exists exactly because a big message parking in front of heartbeats trips the
   failure detector.
 **In 1.6/Artery**, that middle layer is replaced with a recipient-aware, multi-lane, Akka.Streams-
 based pipeline: it knows recipient and size *before* the wire, assigns messages to QUIC streams
 accordingly (ordering preserved per sender→receiver pair rather than per node pair), chunks
 oversized messages onto **dedicated large-message streams**, keeps **system/control messages on
 their own stream** (including a quarantine-bypass channel), adds headers for version negotiation,
 and compresses actor refs to integers. QUIC is almost incidental — it is chosen because it gives
 cheap independent streams for that pipeline to map onto.
 | Benefit | 1.5 + custom QUIC transport | 1.6 Artery | Why the pipeline is the difference |
 |---|---|---|---|
 | Throughput / multiplexing | ✗ (single ordered lane) | ✓ (the 2–3M msg/s target) | Lane assignment needs recipient visibility, which only the pipeline has |
 | Large messages without side channels | ~ (raise the frame cap; heartbeats still queue behind big frames) | ✓ (dedicated chunked streams; control lane unaffected) | Size-based routing happens above the transport |
 | Honest failure detection under load | ✗ (heartbeats share the data lane) | ✓ (isolated control stream) | Same |
 | Split-brain handling while quarantined | ✗ | ✓ (quarantine-bypass channel) | Quarantine lives in the pipeline, not the transport |
 | TLS 1.3 | ✓ (but dot-netty mTLS gets there too) | ✓ | Transport-level — the one benefit 1.5 delivers fully |
 | Fast handshake / connection migration | ✓ | ✓ | Transport-level |
 Consequences for this report's conclusions:
 - **The "don't build it on 1.5" recommendation strengthens** — a 1.5 transport captures only the
  bottom two rows, and both have cheaper paths (dot-netty mTLS; config).
 - **The gRPC-alleviation answer partially flips at 1.6, but only for Akka-adjacent channels.**
  ScadaBridge's HTTP notify-and-fetch for deploy configs, and any "too big for the frame cap"
  workaround, exist purely because of rows 2–3; under Artery those payloads could legitimately ride
  remoting. The genuinely non-Akka gRPC (mxaccessgw, HistorianGateway, LocalDb's deliberate
  cluster-independence) stays gRPC regardless — see the mapping table below.
 - **New costs arrive with the benefits:** Artery is a new wire protocol (flag-day cutover, no
  rolling mix with 1.5 remoting — tolerable given the stop-pairs-together discipline), .NET 9+
  only, and it drags in the platform/ops constraints below (libmsquic, UDP firewall rules, Windows
  Server 2022+, macOS dev friction) — those attach to *any* QUIC adoption, 1.5 or 1.6.
 ## 6. Platform and ops constraints ([System.Net.Quic](https://learn.microsoft.com/en-us/dotnet/fundamentals/networking/quic/quic-overview))
 - APIs are stable since .NET 9; the apps are .NET 10 — fine.
 - **Windows: requires Windows 11 / Server 2022+** (older versions lack the TLS 1.3 crypto APIs).
  ⚠️ Check the `wonder-*` boxes — if any production VM is Server 2019 or older, QUIC remoting is a
  hard no there.
 - **Linux:** `libmsquic` must be installed in every image — the `aspnet:10.0`-based images and the
  docker-dev rig would all need `apt-get install libmsquic` from packages.microsoft.com.
 - **macOS (dev machine): only partially supported** — Homebrew `libmsquic` + a
  `DYLD_FALLBACK_LIBRARY_PATH` export, outside Microsoft's test matrix. Offline test suites that
  spin up real ActorSystems with remoting would grow a fragile native dependency on the Mac.
 - **Firewalls:** remoting moves from TCP to **UDP** — every rule for 4053/etc. needs a UDP twin,
  and OT network gear is notoriously unkind to UDP.
 ## 7. .NET-native alternatives to MsQuic
 Under the 1.5 SPI every transport degenerates into the same ordered byte pipe (§5), so candidate
 wire protocols differ on **dependencies, platform reach, and TLS posture** — not on features. By
 that metric, ranked:
 **1. Managed sockets + `SslStream` + `System.IO.Pipelines` — the best fit if anything gets built.**
 100% in-box managed .NET: zero native dependencies, runs everywhere the apps run today (macOS test
 suites, any Windows Server version, stock `aspnet:10.0` images, TCP firewall rules unchanged).
 TLS 1.3 via `SslStream` is *optional* — can match the current plaintext posture with an mTLS
 upgrade path, unlike QUIC's hard TLS mandate. Arbitrary frame sizes. Strategically this is a
 "modern DotNetty replacement": it sheds the effectively unmaintained DotNetty dependency, which is
 one of 1.6's own goals. JVM precedent that this is respectable: JVM Artery ships TCP and TLS-TCP
 modes alongside Aeron, and TCP Artery reached ~650k msg/s — most of Artery's gain is the pipeline,
 not the wire. What it gives up vs QUIC: fast handshake and connection migration — neither of which
 was a driver here.
 *Throughput expectations, from real data:* this exact idea was prototyped by to11mtm in
 [PR #4594](https://github.com/akkadotnet/akka.net/pull/4594) (Oct 2020, draft, never merged) — an
 Akka.Streams/managed-socket TCP transport under the classic pipeline. On the author's i7-8750H:
 DotNetty baseline ~220–240k msg/s; the managed transport alone ~170–240k (parity at best); adding
 aggressive zero-copy work + a hand-written protobuf deserializer reached ~240–300k (+10–25%), and
 the residual bottlenecks identified were all *pipeline*, not wire: serialization, the PDU codec,
 `EndpointWriter` dispatch, blocking `.Result` in serialization paths. I.e. the transport swap is
 throughput-neutral; only attacking the layer above moves the number, and even extreme effort caps
 near +25%. (Reference points: the official
 [RemotePingPong docs](https://getakka.net/articles/remoting/performance.html) show 82k → 141k msg/s
 from flush-batching alone on older hardware; modern hardware on 1.5.x lands roughly 300–600k per
 association; the 1.6 Artery target is 2–3M.) Two mitigating notes: per-*association* ceilings apply
 per site pair, so hub-and-spoke aggregate scales with site count; and for large frames (the raised
 `maximum-frame-size` case) a managed transport saturates the wire in bytes/sec — the msg/s ceiling
 is a small-message phenomenon.
 **2. HTTP/2 — fully managed multiplexing, but the SPI can't use it.** Kestrel (server) +
 `SocketsHttpHandler` (client), or gRPC bidirectional streaming as the framing, are fully managed
 h2 with real stream multiplexing and h2c (no cert mandate). But the 1.5 SPI locks the multiplexing
 away for exactly the §5 reasons, h2 streams still share one TCP connection (packet-level
 head-of-line blocking remains), and it adds protocol overhead for no captured benefit. Only
 distinct win is riding existing Kestrel/gRPC infrastructure. Not preferred over raw sockets.
 **3. WebSockets (`System.Net.WebSockets`)** — managed, but a single ordered stream, i.e. TCP with
 extra framing. Only interesting for exotic proxy/firewall traversal. Skip.
 **No managed QUIC exists.** `System.Net.Quic` *is* the msquic interop — there is no fully managed
 QUIC implementation in .NET, and nothing else in the box provides independent streams. So "QUIC
 without libmsquic" is not on the table, on any .NET version.
 Forward-looking note — **resolved 2026-07-22**: upstream did exactly this. The Artery port that
 began landing on `dev` in July 2026 is **Artery-TCP** (openspec #8313), so the pipeline benefits
 arrive without libmsquic, and QUIC becomes a later optional substrate — the outcome to11mtm
 advocated in the #7466 design discussion. This also removes the platform/ops constraints of §6
 from the 1.6 adoption path (TCP, no native deps, no UDP firewall changes) unless/until a QUIC
 substrate is chosen deliberately.
 ## 8. Would it actually alleviate the gRPC?
 Mostly no — mapping the family's gRPC surfaces:
 | gRPC usage | Could Akka-over-QUIC replace it? |
 |---|---|
 | mxaccessgw ↔ clients | **No.** It exists for COM bitness isolation; the worker is .NET 4.8 x86, which can't even load a .NET 9+ QUIC stack, and callers aren't cluster members. |
 | HistorianGateway API | **No.** A standalone service contract for multiple heterogeneous consumers — service RPC is the right shape. |
 | LocalDb pair sync | **Technically closest, but deliberately no.** It runs between the same nodes as the Akka cluster, but its whole value is being independent of cluster state (it's what lets a node boot from cache when things are degraded), and it's consumable by non-Akka apps. Coupling it to remoting would weaken it. Its 4 MB cap is a gRPC config choice, not a law. |
 | ScadaBridge site→central (S&F etc.) | Already Akka where Akka fits; the bulk/artifact paths that avoid remoting avoid it because of the single-channel head-of-line problem — which, per §4, a 1.5 QUIC transport does not fix. |
 The "big messages so we don't need side channels" hope is really an Artery feature (dedicated
 chunked large-message streams), not a QUIC-transport feature.
 ## 9. Options
 **A. Don't build it; tune what we have; track 1.6 (recommended).** Best-practice fit:
 transport-layer plumbing is exactly the code an OT product wants to own least, and the upstream
 design explicitly targets the wishlist (large-message channels, quarantine-bypass control channel —
 the latter directly relevant to the 2-node availability-first failover posture). Concretely: bump
 1.5.62 → 1.5.70 (eight releases of remoting/streams fixes, including a cluster-wedging
 consistent-hash-router bug), and if specific pains exist now, raise `maximum-frame-size` and/or
 enable the new dot-netty mTLS. Subscribe to #7466; when real alphas ship, the pair-restart
 discipline makes adoption cheap.
 **B. Build a classic-SPI QUIC transport now.** ~3–5 weeks to production quality per §3. Choose this
 only if the *specific* wins (TLS 1.3 with modern cert handling, UDP traversal, fast reconnect)
 matter enough to own a transport — knowing it delivers no throughput/multiplexing gain and gets
 obsoleted by 1.6.
 **C. Weekend-scale spike, not production.** Implement the minimal `Transport`/`AssociationHandle`
 pair, run a two-node pair on the rig, benchmark against dot-netty. Genuinely worthwhile as
 de-risking/learning for the eventual 1.6 migration (certs, libmsquic packaging, UDP firewall rules
 all get exercised) without owning it in production. A reasonable middle path.
 **Recommendation: A**, with **C** if hands-on evidence is wanted — the honest summary is that for
 this workload, an MsQuic transport on Akka 1.5 buys security and handshake ergonomics that are
 mostly available from config today, while the features that would actually reduce the gRPC
 side-channels only arrive with the Artery pipeline rewrite, which upstream hasn't started coding.
 ## Sources
 - [akka.net #7466 — Epic: Akka.Remote Quic-based Transport (Artery)](https://github.com/akkadotnet/akka.net/issues/7466)
 - [Petabridge — Akka.NET v1.6 Roadmap](https://petabridge.com/blog/akkadotnet-v1.6-roadmap/)
 - [Akka.NET releases](https://github.com/akkadotnet/akka.net/releases)
 - [Akka.NET milestones](https://github.com/akkadotnet/akka.net/milestones)
 - [Akka.NET remoting transports documentation](https://getakka.net/articles/remoting/transports.html)
 - [Akka.NET PR #7851 — mutual TLS for DotNetty transport](https://github.com/akkadotnet/akka.net/pull/7851)
 - [QUIC support in .NET (System.Net.Quic)](https://learn.microsoft.com/en-us/dotnet/fundamentals/networking/quic/quic-overview)
@@ -0,0 +1,241 @@
 # Deep dive: Akka Multi-DC cluster support, ported to Akka.NET, applied to our topology
 *Research report, 2026-07-22. Question: Akka (JVM) has multi-DC cluster support that Akka.NET does
 not. Given our layout (central 2-node cluster + N site 2-node clusters), what would porting it take,
 and how would it be used in our systems?*
 **TL;DR:** Multi-DC maps onto our topology almost perfectly — "one DC per pair" gives per-site
 leadership, per-site singletons, per-site oldest-node primaries, and WAN-partition tolerance inside
 ONE cluster, which is precisely the problem set the OtOpcUa per-cluster mesh program and
 ScadaBridge's three bespoke bridge transports were built to solve. But the port is core-gossip
 surgery that has already defeated the Akka.NET maintainers twice (PRs #3284 and #4111, the latter a
 67-commit functionally-complete draft abandoned in 2019 over multi-node-test instability, now
 milestoned to the far-future 2.0.0). And two of our hardest cross-cluster problems — the 128 KB
 frame limit and bulk data movement — are *not* clustering problems and survive multi-DC untouched.
 Recommendation: don't port; keep the mesh-program direction; treat #4111 as a candidate for an
 upstream-contribution play only if the mesh program's transport phases (2/5) prove heavier than
 expected. Details and options below.
 ---
 ## 1. What Akka multi-DC actually is (JVM; GA since Akka 2.5.6, 2017)
 One correction to the framing: this isn't an upcoming JVM feature — it has been GA on the JVM side
 since 2017. What's true is that it never made it into Akka.NET. Semantics (from the
 [Akka 2.6 docs](https://doc.akka.io/docs/akka/2.6/typed/cluster-dc.html) — 2.6.20 is the last
 Apache-2.0 version, which matters for porting; 2.7+ is BSL-licensed):
 - **DC membership is just a role.** `akka.cluster.multi-data-center.self-data-center = "site-a"`
  adds role `dc-site-a`. One cluster, every member tagged. Default DC is `"default"` — which is why
  untagged and tagged nodes can't mix (JVM rolling 2.4→2.5 upgrades hit exactly this, akka/akka
  #25496).
 - **One leader per DC.** Each DC's leader performs joining/up/leaving/removal transitions for *its
  own* members only. Crucially: **members can join and leave while DCs are partitioned from each
  other** — impossible in a plain cluster, where any unreachable member blocks convergence and
  therefore blocks all leader actions fleet-wide.
 - **Two failure detectors.** `akka.cluster.failure-detector` intra-DC (all-to-all within the DC,
  as today) and `akka.cluster.multi-data-center.failure-detector` cross-DC, run only by the
  `cross-data-center-connections` (default 5) oldest nodes per DC. Cross-DC unreachability raises
  `UnreachableDataCenter` / `ReachableDataCenter` events — it is treated as "the WAN link is bad,"
  explicitly NOT fed into member downing: *"For network partitions between data centers the system
  should typically not down the unreachable nodes, but instead wait until it heals or a decision is
  made by a human or external monitoring system."*
 - **Cluster Singleton is per-DC.** Start the manager on all nodes with 3 DCs defined → 3 live
  singletons, one per DC. Proxies default to the local DC and take a `dataCenter` parameter to
  target another DC's singleton. A "global" singleton = start its manager only in one designated DC
  and accept unavailability when that DC is cut off.
 - **Cluster Sharding is per-DC** (own coordinator/regions per DC; cross-DC proxies available).
  Not relevant to us — neither app uses sharding.
 - **Distributed Pub/Sub is NOT DC-scoped.** The mediator replicates its registry among *all*
  cluster nodes (optionally restricted by role) — so topics span every DC by default. This is the
  single most valuable property for OtOpcUa (§4).
 - Singleton placement and per-DC leadership both key off **oldest member within the DC** — i.e.
  the JVM feature's per-DC primary rule is literally our `ActiveNodeEvaluator.SelfIsOldestUp` /
  OtOpcUa Phase-0b `IsDriverPrimary` rule, applied per pair.
 ## 2. Where Akka.NET stands on it
 - [Issue #3261](https://github.com/akkadotnet/akka.net/issues/3261) (opened Jan 2018 by
  Aaronontheweb) is the tracking issue. Current milestone: **2.0.0 — Akka.Typed**, i.e. no
  committed timeline; it is not in the 1.6 plan (1.6 = Artery/QUIC, .NET Standard drop, new
  Streams materializer — see the companion report `akka_msquic_transport.md`).
 - It was **implemented once**: Horusiath's [PR #3284](https://github.com/akkadotnet/akka.net/pull/3284)
  (2018, milestoned 1.4.0, closed), preserved by Aaron as draft
  [PR #4111](https://github.com/akkadotnet/akka.net/pull/4111) (Dec 2019, still open). 67 commits
  touching membership state, gossip, cross-DC heartbeats, singleton/sharding DC-awareness, failure
  detection, cluster events, and serialization, plus multi-node specs for split-brain and
  sunny-weather scenarios. It reportedly passed on .NET Framework but **died on Multi-Node TestKit
  instability on .NET Core** ("151 tests failing" at one point; deadlocks in MembershipState lazy
  init, convergence races, CoordinatedShutdown exit-hook problems). It was dropped from 1.4.0 for
  review/verification bandwidth and never revived. Its base is **mid-2018 `dev`** — six years and
  two major version lines behind today's 1.5.70.
 So "porting the support" concretely means: re-porting from the JVM 2.6.20 sources (Apache 2.0 —
 do NOT copy from 2.7+/BSL) with #4111 as a .NET-idiom reference, onto Akka.NET 1.5.x, and carrying
 it as a fork until upstream lands its own in 2.0.0.
 ## 3. The mapping onto our topology
 The fit is genuinely striking. Model: **every 2-node pair is a DC** in one spanning cluster.
 | DC | Members | Today |
 |---|---|---|
 | `dc-central` | central-a, central-b | ScadaBridge central cluster / OtOpcUa admin pair |
 | `dc-site-a` | site-a-1, site-a-2 | separate site cluster (SB) / same mesh (OtOpcUa) |
 | `dc-site-b` … | … | … |
 What each multi-DC property buys us:
 - **Per-DC oldest = pair primary, by construction.** The exact rule both products converged on
  (ScadaBridge `ActiveNodeEvaluator.SelfIsOldestUp`; OtOpcUa Phase 0b) becomes the framework's own
  scoping — the mesh design's core motivation ("every Primary-gated decision asks a cluster-wide
  election about a pair-local resource") dissolves.
 - **Per-DC singletons.** OtOpcUa's `admin-operations` singleton stays effectively central by
  starting its manager only on `dc-central` nodes; per-pair singletons (e.g. a per-site drain
  coordinator) come free.
 - **WAN-safe failure handling.** Intra-pair failure detection + our `auto-down`(15s)
  availability-over-partition-safety posture stays *within the pair*, where we want it. A WAN blip
  to a site raises `UnreachableDataCenter` — no downing, no quarantine churn, and (the big one)
  **membership operations at other sites keep working during the outage**. Today's OtOpcUa single
  mesh has neither property, which is a large part of why the mesh program exists.
 - **Pub/sub spans DCs.** OtOpcUa's nine DPS topics + deploy notify channel keep working unchanged
  across the whole fleet. The Secrets Akka replicator (DPS-based) would also span everything —
  today it explicitly cannot cross ScadaBridge's separate clusters (why secrets use SQL-hub mode
  there).
 - **Direct actor messaging replaces ClusterClient.** In one spanning cluster, receptionist
  contact-point management, per-site client caching, and DB-driven address refresh
  (`CentralCommunicationActor`'s `_siteClients`, the 60 s `Site`-row polling) reduce to ordinary
  cluster-aware messaging.
 ### What multi-DC does NOT fix (important)
 - **The 128 KB frame limit and serialization posture are unchanged.** ScadaBridge's HTTP
  notify-and-fetch (deploy configs) and gRPC streaming (`:8083`, high-volume attribute/alarm
  events) exist because Akka messaging is the wrong channel for bulk payloads — that remains true
  in a multi-DC cluster. Those channels stay. (Same conclusion as the MsQuic report: bulk channels
  aren't clustering problems.)
 - **Store-and-forward stays.** At-least-once site→central delivery with a durable buffer
  (`StoreAndForwardService` + replicated `sf_messages`) is an application guarantee; multi-DC
  changes the pipe, not the need for the buffer.
 - **Security got harder, not easier.** Today an attacker on the WAN can reach ClusterClient/gRPC
  seams; in a spanning cluster they can attempt to *join the cluster*. Akka remoting is
  unauthenticated h2c-equivalent in both products (accepted risk, mesh design §6.3) — a spanning
  cluster raises the value of revisiting that.
 - **Persistence single-writer caveats** (the JVM docs' sharding-across-DCs warning) don't bite us —
  no sharding, and pair-local stores are LocalDb-replicated with HLC/LWW already.
 ## 4. Per-system assessment
 ### OtOpcUa — the big beneficiary, but the timing is awkward
 OtOpcUa *today* is one mesh spanning everything — i.e. it is already shaped like a multi-DC
 cluster, minus the DC semantics. The per-cluster mesh program
 (`OtOpcUa/docs/plans/2026-07-22-per-cluster-mesh-program.md`, Phases 1–7) exists to fix exactly the
 things DC-tagging would fix in place:
 | Mesh-program phase | Under multi-DC |
 |---|---|
 | 0a auto-down / 0b oldest-primary | Already done; both carry over unchanged (intra-DC) |
 | 1 `ClusterNode` address columns, DB-sourced ack set | **Mostly unnecessary** — membership stays visible fleet-wide |
 | 2 ClusterClient comm actors + receptionists | **Unnecessary** — one cluster, direct messaging |
 | 3 config fetch-and-cache from central | **Still required** — driven by "sites have no SQL Server," not by clustering |
 | 4 cut driver-side ConfigDb | **Still required** — same reason |
 | 5 gRPC migration of 7 observability topics | **Unnecessary** — DPS keeps working |
 | 6 mesh partition into pairs | Replaced by DC tagging + seed layout (much smaller) |
 | 7 failover drills / live gates | Still required, plus new cross-DC drills |
 So multi-DC would delete the two transport-migration phases (2 and 5 — the heaviest pure-plumbing
 work) and shrink 1 and 6, while leaving the genuinely deployment-driven phases (3, 4) intact. That
 is a real prize. It is also exactly why the timing is awkward: the program was just sequenced on
 the ScadaBridge-proven separate-cluster shape, and switching horses means betting those phases on a
 forked cluster core instead of on patterns already running in production-adjacent code.
 ### ScadaBridge — already solved this; migration value is low
 ScadaBridge built the separate-cluster bridges years-of-work deep and they are live: per-site
 `ClusterClient`s with DB-driven contact refresh, per-node receptionist registration, gRPC
 server-on-site/client-on-central streaming, token-gated HTTP deploy fetch, S&F with active-node
 gating. Two details actually make it multi-DC-*ready* — every cluster already shares the one
 ActorSystem name `"scadabridge"` (separation is purely seed-node partitioning), and the roles
 already include `site-{SiteId}` — but what multi-DC would replace is only the ClusterClient control
 channel and the address-book machinery. The gRPC and HTTP channels stay (frame limit), S&F stays,
 and the CLI's ClusterClient entry point would need rework. Replacing a proven, tested seam with a
 forked gossip layer for a cleaner control channel is a poor trade on its own.
 ## 5. What the port itself takes
 Scope (from the JVM 2.6.20 implementation and the #4111 diff):
 1. **Membership/gossip core** — `Member` gains a `dataCenter` (derived from the `dc-` role, so the
   gossip **wire format is nearly unchanged** — the DC rides in roles); `MembershipState`/gossip
   convergence, leader election, and reachability calculations all become DC-filtered (convergence
   ignores cross-DC observations; per-DC leader actions permitted during inter-DC splits). This is
   the delicate 20% that is 80% of the risk.
 2. **Heartbeating** — keep today's all-to-all within a DC; add the cross-DC heartbeater run by the
   N oldest per DC with its own failure-detector config; new `UnreachableDataCenter` /
   `ReachableDataCenter` events.
 3. **Downing** — `AutoDowning`/SBR must become DC-scoped (JVM SBR is; our `auto-down` posture then
   applies per pair, which is exactly the HA stance we want). A non-DC-aware downing provider in a
   multi-DC cluster is actively dangerous (downs the far side on WAN blips).
 4. **Cluster tools** — Singleton manager/proxy DC-awareness (small — it is role-filtering);
   sharding proxies (skippable for us); DPS needs nothing.
 5. **Serialization** — additive protobuf changes for the new heartbeat/gossip fields; whole-fleet
   cutover required (mixed DC-aware/unaware nodes don't interoperate — the JVM had the same
   constraint). Our stop/start-pairs-together discipline makes this tolerable, but it is a
   flag-day, not rolling.
 6. **Verification — the real cost.** This is what killed it upstream twice: MNTK specs for
   split-brain, join-during-partition, cross-DC unreachability, singleton-per-DC handover. Our
   substitute would be rig-based live gates (docker-dev with real WAN-partition injection via
   `tc`/iptables), which we're good at — but the failure modes are convergence races that soak
   tests catch and unit tests don't.
 **Effort honestly:** the code port is plausibly 4–8 weeks against the Apache-2.0 JVM reference with
 #4111 as a map. The validation to *trust* it under an OT platform's availability requirements is
 the same again or more, and then it's a **permanent fork of Akka.Cluster** — every 1.5.x bump
 (there have been 8 since our pin) becomes a merge with the most intricate code in the framework,
 with no upstream support for the modified layer, until upstream's own version lands in 2.0.0 and
 strands the fork entirely. Precedent matters here: the framework's own authors, with the MNTK
 infrastructure we lack, shelved it twice.
 ## 6. Options
 **A. Don't port. Continue the current direction (recommended).** ScadaBridge keeps its proven
 bridges; OtOpcUa executes the mesh program as sequenced (selfform prerequisite → Phases 1–7). Every
 pattern in that program is already running in ScadaBridge production code; risk lives in
 application code we own rather than framework internals we'd fork. Cost: OtOpcUa still pays for
 Phases 2 and 5 (ClusterClient + gRPC plumbing), and cross-cluster pub/sub stays unavailable
 (secrets replication keeps SQL-hub mode on ScadaBridge).
 **B. Port multi-DC (fork Akka.Cluster).** Scope per §5, ~2–4 months including validation, plus the
 standing fork tax. Payoff concentrated in OtOpcUa (skip Phases 2/5, keep DPS, keep one mesh with
 correct per-pair scoping). Only rational as an **upstream-contribution play** — rebase/re-port
 against current `dev`, revive #4111 with Petabridge (possibly as sponsored work), so the fork has
 an exit. As a private fork: not worth it for two products with a working alternative.
 **C. Multi-DC-lite on stock Akka.NET** (keep OtOpcUa's single mesh, emulate the scoping
 app-side): role-scoped singletons and role-scoped DPS groups per pair (supported today), the
 already-done oldest-Up-per-role primary rule, plus a **custom `IDowningProvider`** that only downs
 same-site peers (pluggable, no fork). What cannot be emulated: convergence — one unreachable node
 anywhere still blocks joins/leaves fleet-wide, and cross-site heartbeating stays all-to-all over
 the WAN. Tolerable for a small, static fleet that rarely adds nodes mid-outage; it is a
 known-sharp-edges compromise, not a fix. Worth considering only if the mesh program stalls.
 **Recommendation: A.** The mesh program should keep going — it's sequenced, its prerequisite plan
 exists, and its shape is production-proven next door. Revisit B strictly as "fund/contribute the
 upstream port" if Phases 2/5 blow up in practice, and note in that case the decision point comes
 *before* Phase 2 starts, because B makes exactly those phases unnecessary.
 ## Sources
 - [Akka 2.6 Multi-DC Cluster docs](https://doc.akka.io/docs/akka/2.6/typed/cluster-dc.html) (last Apache-2.0 line)
 - [Akka classic Distributed Pub/Sub docs](https://doc.akka.io/docs/akka/2.6/distributed-pub-sub.html) (mediator scope: all nodes / role-filtered)
 - [akka.net #3261 — Multi-Data Center Support in Akka.Cluster](https://github.com/akkadotnet/akka.net/issues/3261) (milestone 2.0.0)
 - [akka.net PR #3284 — original multi-DC implementation (Horusiath)](https://github.com/akkadotnet/akka.net/pull/3284)
 - [akka.net PR #4111 — preserved 67-commit draft (Aaronontheweb)](https://github.com/akkadotnet/akka.net/pull/4111)
 - [akka/akka #25496 — 2.4→2.5 rolling-upgrade DC requirement failure](https://github.com/akka/akka/issues/25496)
 - ScadaBridge cross-cluster seam map: `CentralCommunicationActor.cs` / `SiteCommunicationActor.cs` (ClusterClient),
  `SiteStreamGrpcClient/Server.cs` + `sitestream.proto` (gRPC :8083), `HttpDeploymentConfigFetcher.cs` +
  `DeploymentConfigEndpoints.cs` (token-gated HTTP), `StoreAndForwardService.cs`, `ActiveNodeEvaluator.cs`;
  ActorSystem name shared at `AkkaHostedService.cs:191`
 - OtOpcUa mesh design + program: `OtOpcUa/docs/plans/2026-07-21-per-cluster-mesh-design.md`,
  `2026-07-22-per-cluster-mesh-program.md`
@@ -0,0 +1,153 @@
 # ZB.MOM.WW.Overview — family overview dashboard, v1 design
 **Date:** 2026-07-22
 **Decision context:** `overview_dashboard_options.md` (same date) — Option A chosen: a custom in-house pane, matching the look and feel of the other scadaproj apps. Aspire standalone rejected (resource/status UI disabled outside an AppHost; no persistence; dev-tool positioning).
 **Implementation plan:** `2026-07-22-overview-dashboard-impl-plan.md` (same directory) — phased, code-verified tasks incl. the sister-project changes.
 ## 1. What it is
 A small Blazor Server app, hosted in scadaproj like the other shared components, that shows **every registered application instance on one page**: Up/Degraded/Down/Unreachable, Active/Standby where applicable, the **current Akka cluster leader** for each cluster group, per-check detail, and a deep link to each instance's own management UI. The page always renders instantly from a cached snapshot — a background poller on a **configurable timer** does all the HTTP work, and a **configurable staleness timeout** marks any data the poller hasn't refreshed as out-of-date/offline. **No login: the dashboard is anonymous and strictly read-only** (it only GETs endpoints that are already anonymous, and has no mutating actions). Configured entirely from an appsettings registry. **Every instance reports through the same channel:** the anonymous `/health/ready` + `/health/active` endpoints of `MapZbHealth`, parsing the canonical `ZbHealthWriter` JSON. Three of the four apps already expose these everywhere; the one gap — ScadaBridge *site* nodes — is closed by a small prerequisite change in ScadaBridge (§4) rather than worked around, so the dashboard has exactly one probe model.
 Non-goals for v1: history/uptime graphs, alerting/webhooks, telemetry viewing (OTLP/traces), and any push-based ingestion. If ops-grade monitoring is ever needed, that's the Grafana/Prometheus phase in the options doc — this registry ports straight to a scrape config.
 ## 2. Project shape
 ```
 ZB.MOM.WW.Overview/                      # plain directory in scadaproj — NOT a nested git repo
  ZB.MOM.WW.Overview.slnx
  src/ZB.MOM.WW.Overview/               # single ASP.NET Core + Blazor Server project
    Program.cs
    Registry/     (options + validator)
    Polling/      (poller BackgroundService, health client, snapshot store)
    Components/   (App.razor, MainLayout → ThemeShell, Pages/Overview.razor, widgets)
  tests/ZB.MOM.WW.Overview.Tests/
  docker/                                # HistorianGateway-pattern runtime-only image
 ```
 - .NET 10, global InteractiveServer render mode (same router style as ScadaBridge/HG/mxgw — no per-page render-mode traps).
 - Package references (Gitea feed): `ZB.MOM.WW.Theme` 0.3.1, `ZB.MOM.WW.Health` (0.2.0 — see §4a), `ZB.MOM.WW.Telemetry(.Serilog)` 0.1.0, `ZB.MOM.WW.Configuration` 0.1.0. **No Auth, no Audit, no Secrets**: the dashboard is anonymous read-only by requirement — there is no login, no mutating action to audit, and nothing secret in the registry. (If a login is ever wanted later, `AddZbLdapAuth` + `LoginCard` drop in the family way.)
 - It is an **app**, not a published library — nothing gets packed to the feed.
 ## 3. Registry (appsettings)
 ```jsonc
 "Overview": {
  "PollIntervalSeconds": 10,          // global poll cadence (configurable timer)
  "TimeoutSeconds": 3,                // per-request timeout
  "StaleAfterSeconds": 45,            // snapshot older than this ⇒ instance rendered Stale (out of date/offline)
  "Applications": [
    {
      "Name": "OtOpcUa",
      "ManagementLabel": "AdminUI",
      "Instances": [
        { "Name": "central-1", "Group": "central",
          "BaseUrl": "http://otopcua-central-1:9000",
          "ManagementUrl": "http://otopcua-central-1:9000/",
          "HasActiveRole": true },
        { "Name": "site-a-1", "Group": "site-a", "BaseUrl": "http://...", "HasActiveRole": true }
      ]
    },
    {
      "Name": "ScadaBridge",
      "Instances": [
        { "Name": "central-a", "Group": "central", "BaseUrl": "http://sb-central-a:5000",
          "ManagementUrl": "http://sb-central-a:5000/", "HasActiveRole": true },
        { "Name": "site-a-1", "Group": "site-a",
          "BaseUrl": "http://sb-site-a-1:8084", "HasActiveRole": true }   // health on the HTTP/1.1 listener — see §4
      ]
    },
    {
      "Name": "MxAccessGateway",
      "ManagementLabel": "Dashboard",
      "Instances": [
        { "Name": "windev", "BaseUrl": "http://windev:5130",
          "ManagementUrl": "http://windev:5130/" }          // single-instance; no active/standby role
      ]
    },
    {
      "Name": "HistorianGateway",
      "ManagementLabel": "Dashboard",
      "Instances": [
        { "Name": "wonder-app-vd03", "BaseUrl": "https://wonder-app-vd03:5222",
          "ManagementUrl": "https://wonder-app-vd03:5222/" } // prod single TLS ALPN port; dev :5220
      ]
    }
  ]
 }
 ```
 All four family apps are first-class registry citizens; `HasActiveRole` is simply omitted for the two single-instance gateways (mxgw, HistorianGateway), which have no active/standby concept — their cards show status without the Active badge.
 - Every instance is probed the same way: GET `{BaseUrl}/health/ready`, plus `{BaseUrl}/health/active` when `HasActiveRole`. There is deliberately no per-instance probe-mode switch — one probe model, one status derivation.
 - `PollIntervalSeconds`/`TimeoutSeconds`/`StaleAfterSeconds` are overridable per application and per instance (instance wins) for slow links or chatty pairs; `StaleAfterSeconds` must exceed `PollIntervalSeconds` (validated).
 - Bound + validated with `AddValidatedOptions<OverviewOptions, OverviewOptionsValidator>` (`ZB.MOM.WW.Configuration`): non-empty registry, absolute http(s) URLs, unique app/instance names, positive intervals, stale > poll. Malformed registry fails boot via `ConfigPreflight`, not the first poll.
 ## 4. Prerequisite target-app changes (health from ALL apps)
 The dashboard's contract is "every registered instance serves `MapZbHealth`". Current reality and the deltas:
 | App / node kind | Today | Delta needed |
 |---|---|---|
 | OtOpcUa admin + driver nodes | `MapZbHealth` mapped unconditionally | None in code. **Deployment note:** a driver-only Windows-service node with no explicit URLs falls back to Kestrel's `localhost:5000` — every registered node must be given a reachable binding (`ASPNETCORE_URLS`/`HTTP_PORTS`), which the docker rig already does. |
 | ScadaBridge **Central** | `MapZbHealth` (ready: database/akka-cluster/required-singletons; active: active-node) | None. |
 | ScadaBridge **Site** nodes | **No health endpoints** — only the gRPC `:8083` (HTTP/2-only) + `/metrics` on the `:8084` HTTP/1.1 listener | **Small ScadaBridge PR (the one prerequisite):** map `MapZbHealth` on the `:8084` HTTP/1.1 listener alongside `/metrics`. Ready checks: `akka-cluster` (site pair) + site SQLite store (+ `localdb-replication` where enabled); active check: the site pair's oldest-Up active-node rule via `ActiveNodeHealthCheck` — the Host already references all three `ZB.MOM.WW.Health` packages, so this is pure wiring, no new dependencies. Gives site rows full Up/Degraded detail **and** Active/Standby, same as everything else. |
 | MxAccessGateway | `MapZbHealth` (ready: auth-store) | None. |
 | HistorianGateway | `MapZbHealth` (ready: historian-connection, galaxy-sql, conditional runtime/store-forward/pool checks) | None. |
 Sequencing: the ScadaBridge PR can land any time before the dashboard's acceptance run; the dashboard itself needs no knowledge of it beyond registry entries.
 ### 4a. Shared-lib prerequisite: `ZB.MOM.WW.Health` 0.2.0 — cluster-leader data
 Requirement: the dashboard must show the **current leader** of each Akka cluster. The canonical health JSON today carries only `status`/`description`/`durationMs` per entry — no structured data — so this needs a small **additive** change to the shared lib (which is exactly where the family wants such things to live):
 - **Core (`ZbHealthWriter`):** emit an optional `"data": { ... }` object per entry, sourced from `HealthReportEntry.Data`, **only when non-empty** — existing payloads stay byte-identical, so this is non-breaking for every current consumer.
 - **`ZB.MOM.WW.Health.Akka` (`AkkaClusterHealthCheck`):** populate the check's data dictionary from local cluster state: `leader` (address), `selfAddress`, `selfRoles`, `memberCount`, `unreachableCount` (+ `roleLeader:<role>` where the app cares). Each node reports *its own view*; the dashboard reads the leader from any Up node of the group and flags disagreement (a visible split-brain tell — a nice free diagnostic for the 2-node pairs).
 - **Version/rollout:** bump lib to `0.2.0`, publish the 3 packages to the Gitea feed. **Required bumps:** OtOpcUa + ScadaBridge (the Akka apps — leader data appears with the package bump alone, no app code changes, since both register the shared `AkkaClusterHealthCheck`). **Recommended alignment bumps:** mxgw + HistorianGateway (no behavior change for them; keeps the family version matrix aligned).
 ## 5. Polling and status model
 One `OverviewPollerService : BackgroundService`:
 - Every cadence tick, polls **all instances in parallel** (`Task.WhenAll`, per-instance `CancellationTokenSource` timeout; named `HttpClient` via `IHttpClientFactory`, no retries — the next tick is the retry).
 - Parses the canonical shape `{ status, totalDurationMs, entries: { name: { status, description, durationMs } } }`. Parsing lives in a small `ZbHealthReportClient` inside this app for v1; promoting it into a `ZB.MOM.WW.Health.Client` package is a deliberate later step, not a v1 dependency (recommendation: promote only when a second consumer appears).
 - Derived per-instance state:
  - **Up** — ready 200 + `"Healthy"`; **Degraded** — 200 + `"Degraded"`; **Down** — 503 / `"Unhealthy"`; **Unreachable** — timeout, refused, DNS, non-JSON (kept distinct from Down: it usually means network/VPN/registry-typo, not the app).
  - **Active/Standby** — `/health/active` 200 → Active, 503 → Standby, error → Unknown (only when `HasActiveRole`).
  - Plus: last poll UTC, latency ms, per-check entries, consecutive-failure count (state flips to Down/Unreachable only after **2** consecutive failures to avoid single-blip flapping; recovery is immediate).
 - **Cache-first rendering:** results land in an in-memory `OverviewSnapshotStore` (single immutable snapshot swapped atomically); an event notifies circuits to re-render. Page loads NEVER trigger a poll — they read the cached snapshot and render instantly; the poller timer is the only thing that does HTTP. No persistence — a restart repolls the world within one cadence.
 - **Staleness:** every instance snapshot carries `LastUpdatedUtc`. When the UI renders (or a periodic sweep runs), any instance whose snapshot age exceeds its effective `StaleAfterSeconds` is shown as **Stale (out of date/offline)** regardless of its last known status — this catches a stuck poller, a paused refresh, or a machine-slept dashboard, not just failed probes.
 - **Leader extraction:** for each `Group` (one Akka cluster = one group, e.g. `OtOpcUa/central`, `ScadaBridge/site-a`), the aggregator reads the `akka-cluster` entry's `data.leader` from each Up member and surfaces the group's leader (resolving the leader address back to a registry instance name where possible). If Up members disagree on the leader, the group is flagged (split-brain indicator).
 - StatusPill mapping: Up→`Ok`, Degraded→`Warn`, Down→`Bad`, Unreachable→`Bad` (distinct label), Stale→`Idle`, Active/Leader badges→`Info`.
 ## 6. UI
 > **Mockup:** [`docs/mockups/overview-dashboard-mockup.html`](../mockups/overview-dashboard-mockup.html) (static, self-contained; real Theme tokens + embedded IBM Plex) — the visual reference for `InstanceCard`, group headers, leader/split-brain chips, and every status state incl. Stale. Open it in a browser; the implementation should match it.
 Single page (`/`), `ThemeShell` chassis with its own product name/accent so it reads as a sibling of the other four apps:
 - One section per application (name + rollup pill = worst instance status), instances as a grid of cards (`TechCard` + `StatusPill` composition — a new `InstanceCard` component, since Theme has no prebuilt tile widget). Cluster groups render a group header with the **Leader: <instance>** chip (and the split-brain warning when member views disagree).
 - Card: instance name + group, status pill, Active badge, **Leader badge** on the leader node's card, latency, relative last-seen (turns into an explicit "stale since …" once past `StaleAfterSeconds`); expandable check list (`entries` with per-check pill + description); a `TechButton` "Open <ManagementLabel>" → `ManagementUrl` (new tab).
 - A header row: total counts (n Up / n Degraded / n Down / n Unreachable / n Stale), last-refresh clock, pause/resume auto-refresh (paused long enough, cards go Stale by design).
 - **No login.** The dashboard is anonymous and read-only end to end; there is nothing to authorize and nothing it can change. It still matches the sister apps' look exactly (ThemeShell + tokens + IBM Plex), just without the login gate.
 ## 7. Self-observability and hosting
 - The dashboard eats the family dog food: `MapZbHealth` (ready check: "registry loaded" + "last poll cycle completed"), `MapZbMetrics`, `AddZbTelemetry`/`AddZbSerilog`; a `zb_overview_instance_status` gauge (per instance, labeled) so even v1 status is scrapeable — and becomes the Grafana bridge later.
 - Ports: dev `http://localhost:5320` (clear of every family port in use: 5000/5001, 5120/7121, 5220–5222, 8081–8085, 9000, 9200, 4053, 4840); production single TLS ALPN endpoint like HistorianGateway, warn-only if terminated upstream.
 - `docker/` mirrors the HistorianGateway pattern: publish framework-dependent on the host (authed Gitea feed), COPY into `aspnet:10.0`, gitignored env_file. Natural first deployment: the docker-dev rig host, registry pointed at the existing OtOpcUa + ScadaBridge rig nodes — that is also the acceptance test.
 ## 8. Open decisions (settled recommendations)
 1. **ScadaBridge site rows** — resolved (2026-07-22, user direction): site nodes gain `MapZbHealth` as the one prerequisite PR (§4), so every row is a full health row. *Chosen over* a metrics-liveness fallback (two probe models) and over reading Central's site aggregator (couples the dashboard to ScadaBridge internals and its transport).
 2. **Health-client ownership** — in-app for v1; promote to `ZB.MOM.WW.Health.Client` on second consumer. *(Best practice: contract-owning lib should host the client, but a package bump + publish for one consumer is ceremony without benefit.)*
 3. **History/alerting** — explicitly out of v1; the `zb_overview_instance_status` gauge is the forward hook (Prometheus can scrape the dashboard alone and get fleet status history for free).
 4. **Leader transport** — resolved (2026-07-22, user requirement): via a `data` object in the canonical health JSON (`ZB.MOM.WW.Health` 0.2.0, §4a), populated by the shared `AkkaClusterHealthCheck`. *Chosen over* description-string parsing (fragile) and a separate leader endpoint (second contract for one field).
 5. **Auth** — resolved (2026-07-22, user requirement): none. Anonymous, read-only.
 ## 9. Acceptance (v1 definition of done)
 - The §4 ScadaBridge site-node health PR is merged and site rows render with full check detail + Active/Standby (no special-cased rows anywhere).
 - Registry pointed at the live docker-dev rigs (OtOpcUa 6-node + ScadaBridge local cluster) renders every instance correctly, including: each Active/Standby pair (central **and** site pairs) showing exactly one Active; **each cluster group showing its current leader, and the leader chip moving when the leader node is stopped**; killing a node flips its card to Unreachable within ~2 cadences and back on restart; a Degraded check (e.g. stop a dependency) shows Warn with the failing entry visible; management links open the right app UIs; the page is reachable with no login.
 - **Cache/staleness proven live:** first page load renders instantly from cache (no poll on request); with auto-refresh paused (or the poller stopped) past `StaleAfterSeconds`, every card flips to Stale, and resumes cleanly.
 - Boot fails with a clear `ConfigPreflight` message on a malformed registry.
 - `dotnet test` green: options validator, status derivation (incl. flap damping, staleness marking, leader extraction + disagreement flag, JSON parsing golden tests against captured real payloads from all four apps — with and without `data`), snapshot store, and a bUnit render test of `InstanceCard`.
@@ -0,0 +1,171 @@
 # ZB.MOM.WW.Overview — implementation plan
 **Date:** 2026-07-22
 **Design:** `2026-07-22-overview-dashboard-design.md` (same directory). Read it first; this plan assumes its decisions (custom app, family look/feel, anonymous read-only, cache-first polling, staleness timeout, Akka leader display).
 **Repos touched:** `scadaproj/ZB.MOM.WW.Health` (shared lib 0.2.0), `~/Desktop/ScadaBridge` (site-node health PR), `~/Desktop/OtOpcUa` (Health bump + check verification), optional alignment bumps in `~/Desktop/MxAccessGateway` + `~/Desktop/HistorianGateway`, and the new `scadaproj/ZB.MOM.WW.Overview/`.
 **Sequencing:** Phase 0 → Phases 1 + 2 (parallel, independent repos) → Phase 3 (app; can start in parallel once 0.2.0 is on the feed) → Phase 4 (live acceptance). Phases 1/2 only gate Phase 4, not Phase 3 development.
 All facts below were code-verified 2026-07-22 (file:line cites are from that pass); re-check a cite if the target repo has moved since.
 **How to run this in a fresh session:** read the design doc first, then execute phases in order (0 → 1‖2 → 3 → 4), one phase per sitting with its DoD met before moving on. Work in each repo on a feature branch (`feat/health-0.2.0-cluster-data` in scadaproj, `feat/site-node-health` in ScadaBridge, `feat/health-0.2.0-bump` in OtOpcUa); commit scadaproj work (lib + new app + these docs) on `main` per family habit only when asked. Nothing in this plan requires user input except merges/pushes and the Phase 4 rig runs; if a cited line has drifted, re-locate by the quoted identifier, not the line number.
 ---
 ## Phase 0 — `ZB.MOM.WW.Health` 0.2.0: entry `data` + Akka cluster-view
 Repo: `scadaproj/ZB.MOM.WW.Health/` (plain directory — do NOT `git init`; commits happen in scadaproj). Akka.Cluster pin 1.5.62 (`Directory.Packages.props:9`); version is centralized at `Directory.Build.props:8` (`<Version>0.1.0</Version>`).
 ### Task 0.1 — writer: optional per-entry `data` object
 `src/ZB.MOM.WW.Health/ZbHealthWriter.cs`:
 - Add to the private `HealthEntryDto` (lines ~76-81):
  `[JsonIgnore(Condition = JsonIgnoreCondition.WhenWritingNull)] public IReadOnlyDictionary<string, object>? Data { get; init; }`
 - In the projection (line ~62): `Data = e.Value.Data.Count > 0 ? e.Value.Data : null` (`HealthReportEntry.Data` is never null; empty when unset).
 - **Invariant: old payloads stay byte-identical.** The per-property `JsonIgnore` is essential — the global serializer options deliberately do NOT ignore nulls (the null-`description` behavior is pinned by `ResponseWriterTests.cs:69-85`); do not change `SerializerOptions`.
 Tests (`tests/ZB.MOM.WW.Health.Tests/ResponseWriterTests.cs`): extend `StubHealthCheck` (lines 21-31) to accept an optional data dictionary; add (a) non-empty data → `"data": {...}` emitted with camelCase envelope keys but **verbatim data keys**, (b) empty data → no `data` key at all (assert on raw JSON string), (c) existing null-description test still green.
 ### Task 0.2 — `AkkaClusterHealthCheck`: populate cluster-view data
 `src/ZB.MOM.WW.Health.Akka/AkkaClusterHealthCheck.cs` (currently reads only `SelfMember.Status`, returns description-only — lines 38-70):
 - Add an internal static helper `BuildClusterData(Cluster cluster)` returning `IReadOnlyDictionary<string, object>` with JSON-friendly values only (string/int/string[]):
  - `leader` — `cluster.State.Leader?.ToString()` (omit the key when null: pre-join/unknown),
  - `selfAddress` — `cluster.SelfAddress.ToString()`,
  - `selfRoles` — `cluster.SelfMember.Roles.OrderBy(r => r).ToArray()`,
  - `memberCount` — `cluster.State.Members.Count`,
  - `unreachableCount` — `cluster.State.Unreachable.Count`.
  All property names confirmed compiling in this repo (`ActiveNodeHealthCheck.cs:116-132`, `AkkaActiveNodeGate.cs:33-49`).
 - Pass it through every result path: `HealthCheckResult.Healthy/Degraded/Unhealthy(description, data)`. The no-ActorSystem / cluster-inaccessible paths keep returning description-only (no data) — pinned by `AkkaClusterStatusPolicyTests.cs:63-96`.
 - ASP.NET Core copies `HealthCheckResult.Data` → `HealthReportEntry.Data` automatically, and `MapZbHealth` already wires `ZbHealthWriter` (`ZbHealthEndpointExtensions.cs:42-54`) — **no consumer code change needed beyond the package bump**, provided the app registers this shared check (verified for ScadaBridge; Task 2.1 verifies OtOpcUa).
 Tests (`tests/ZB.MOM.WW.Health.Akka.Tests/`): the `InternalsVisibleTo` hook (`ActiveNodeHealthCheck.cs:7`) exposes the helper. Use `Akka.TestKit.Xunit2` (pin 1.5.62; conventions per existing tests — inherit `TestKit`, never hand-roll `ActorSystem.Create` + manual join) with a single-node self-joined cluster: assert `leader` == self address, `memberCount` == 1, `selfRoles` round-trip; plus the check-level test that a healthy result now carries data and the degraded no-system path carries none.
 ### Task 0.3 — version, pack, publish
 - `Directory.Build.props`: `0.1.0` → `0.2.0`. Note the change in the repo's CHANGELOG/README if present.
 - `dotnet test` from the repo root (baseline 58 tests across 3 test projects — all green plus the new ones), then `dotnet pack -c Release -o artifacts` (3 nupkgs @ 0.2.0).
 - Publish: `dotnet nuget push artifacts/ZB.MOM.WW.Health*.0.2.0.nupkg --source https://gitea.dohertylan.com/api/packages/dohertj2/nuget/index.json --api-key <from ~/.nuget user config / keychain — same credential every prior publish used>` (one command per nupkg or glob).
 - Verify like every prior lib release (the family gotcha: "published" claims must be proven): `curl -s https://gitea.dohertylan.com/api/packages/dohertj2/nuget/registration/zb.mom.ww.health/index.json` shows 0.2.0 (NOT `/nuget/v3/registration/` — that 404s for everything), then restore-verify from a scratch consumer project that pins 0.2.0.
 **Phase DoD:** suite green; 3 packages at 0.2.0 restore-verified from the feed; a manual smoke (any sample host + the check) shows `entries["akka-cluster"].data.leader` in `/health/ready`.
 ---
 ## Phase 1 — ScadaBridge PR: site-node health endpoints
 Repo: `~/Desktop/ScadaBridge` (branch off `main`, e.g. `feat/site-node-health`). Site branch of `src/ZB.MOM.WW.ScadaBridge.Host/Program.cs` is lines 485-570: Kestrel `:8083` gRPC Http2-only (507-512), `:8084` `Http1AndHttp2` (516-518); site DI via `SiteServiceRegistration.Configure` (:532); site pipeline maps only `MapZbMetrics` (:545), gRPC (:548), `MapZbLocalDbSync` (:555). No auth middleware and no FallbackPolicy exist on the site pipeline, so anonymous health needs nothing special.
 ### Task 1.1 — bump Health packages
 `Directory.Packages.props:93-95`: all three `ZB.MOM.WW.Health*` 0.1.0 → **0.2.0**.
 ### Task 1.2 — site health checks (register between `:532` and `var app = builder.Build()` at `:534`)
 Mirror the central registration style (`Program.cs:277-304`, `AddTypeActivatedCheck`):
 1. **`akka-cluster`** (tags: Ready) — the shared `AkkaClusterHealthCheck` with `AkkaClusterStatusPolicy.Default`, exactly as central does at 282-286. The site ActorSystem→DI bridge already exists (`SiteServiceRegistration.cs:168-169`). This is also what carries `data.leader` for site pairs.
 2. **`localdb`** (tags: Ready) — **new** check class in the Host (site namespace): opens the consolidated site SQLite (`LocalDb:Path`) and runs `SELECT 1`. There is no EF context on site (central's `DatabaseHealthCheck<ScadaBridgeDbContext>` does NOT apply — that context is central-only, `Program.cs:266`). Optionally enrich `data` with `ISyncStatus.Connected`/`OplogBacklog` (`SiteServiceRegistration.cs:139-140`) when replication is enabled — enrich only; replication-off must NOT fail the check (replication is default-OFF by design).
 3. **`active-node`** (tags: Active) — **new** thin `SitePairActiveNodeHealthCheck` delegating to `IClusterNodeProvider.SelfIsPrimary` (registered on site, `SiteServiceRegistration.cs:172-178`; role-scoped `site-{SiteId}` oldest-Up). **Do NOT reuse central's `OldestNodeActiveHealthCheck`** — it calls `SelfIsOldest(cluster)` with no role scope (`OldestNodeActiveHealthCheck.cs:30`) and would compute oldest across the wrong member set. Follow the existing delegation precedent `SiteEventLogActiveNodeCheck` (`SiteServiceRegistration.cs:187-191`). Active semantics: primary → Healthy, standby → Unhealthy (503 = Standby to the dashboard — same convention as central).
 ### Task 1.3 — map endpoints
 `app.MapZbHealth();` in the site pipeline after `UseRouting()` (:539), alongside `MapZbMetrics()` (:545). Served on `:8084` (Http1AndHttp2). No auth changes.
 ### Task 1.4 — tests
 Extend `tests/ZB.MOM.WW.ScadaBridge.Host.Tests/`:
 - Imitate `HealthCheckTests.cs` (`WebApplicationFactory<Program>`, `[Collection(HostBootCollection.Name)]`, `UseSetting("ScadaBridge:Node:Role", ...)`, `SkipMigrations=true`, registration assertions via `IOptions<HealthCheckServiceOptions>.Value.Registrations` + `registration.Factory(services)` type checks). **Caveat found in recon:** every existing factory test boots the Central role; there is no site-role factory bootstrap yet. Build one (Role=Site + minimal `ScadaBridge:Node` config + temp `LocalDb:Path`); if the full site host proves too heavy to boot in-test, fall back to the `CompositionRootTests.cs`/`SiteLocalDbWiringTests.cs` style over `SiteServiceRegistration.Configure` for registrations, plus a factory test for just the endpoint mapping. Include a positive control for any "central-only check absent on site" assertion.
 - Assert: site maps `/health/ready` + `/health/active` (non-404); registrations exactly {akka-cluster→Ready, localdb→Ready, active-node→Active} with the intended check types; central registrations unchanged.
 **Phase DoD:** full ScadaBridge suite green; on the local docker rig, `curl http://<site-node>:8084/health/ready` returns the canonical JSON with `akka-cluster` carrying `data.leader`, and `/health/active` splits 200/503 across the pair; central `/health/ready` also shows `data.leader` (free via the bump). PR merged to `main`.
 ---
 ## Phase 2 — OtOpcUa bump (+ optional family alignment)
 ### Task 2.1 — verify the `akka` check is the shared class
 OtOpcUa registers ready checks `configdb`, `akka`, `admin-leader`, `localdb-replication` (registration reachable from `MapOtOpcUaHealth` / Program wiring, `src/Server/ZB.MOM.WW.OtOpcUa.Host`). **Verify** the `akka` check is `ZB.MOM.WW.Health.Akka.AkkaClusterHealthCheck`. If it is → nothing more. If bespoke → either swap to the shared check (preferred, same policy semantics) or populate the same `data` keys (`leader`/`selfAddress`/`selfRoles`/`memberCount`/`unreachableCount`) so the dashboard sees one contract.
 ### Task 2.2 — bump
 `~/Desktop/OtOpcUa/Directory.Packages.props`: `ZB.MOM.WW.Health*` → 0.2.0. Build + run the Host/AdminUI test suites touched by health. Rig check: any admin node `/health/ready` shows `data.leader`.
 ### Task 2.3 (recommended) — alignment bumps, mxgw + HistorianGateway
 Neither uses CPM (**no `Directory.Packages.props` — the known trap**): bump the inline `ZB.MOM.WW.Health` pins in `MxGateway.Server.csproj` and `ZB.MOM.WW.HistorianGateway.Server.csproj` to 0.2.0. No behavior change (no Akka checks there); keeps the family version matrix aligned. Skippable if either repo has unrelated in-flight work.
 **Phase DoD:** OtOpcUa on 0.2.0 with leader data live on the rig; alignment bumps merged or explicitly deferred with a note.
 ---
 ## Phase 3 — the `ZB.MOM.WW.Overview` app
 Location: `scadaproj/ZB.MOM.WW.Overview/` — **plain directory, NOT a nested git repo** (family rule). App conventions (copied from HistorianGateway, the reference implementation — cites from recon):
 ### Task 3.1 — scaffold
 - `ZB.MOM.WW.Overview.slnx` with `/src/` + `/tests/` folders (LocalDb's slnx is the minimal template).
 - `src/ZB.MOM.WW.Overview/ZB.MOM.WW.Overview.csproj`: `Sdk="Microsoft.NET.Sdk.Web"`, net10.0, **inline pinned versions, no CPM** (the app convention): `ZB.MOM.WW.Theme` 0.3.1, `ZB.MOM.WW.Health` 0.2.0, `ZB.MOM.WW.Telemetry` + `.Serilog` 0.1.0, `ZB.MOM.WW.Configuration` 0.1.0, `Serilog.AspNetCore` + console/file sinks. NO Auth/Audit/Secrets packages.
 - `Directory.Build.props` with the family zero-warnings gate (`TreatWarningsAsErrors`, `AnalysisLevel=latest`, `EnforceCodeStyleInBuild`, `Deterministic`, Nullable/ImplicitUsings) — copy HistorianGateway's.
 - `nuget.config`: copy HistorianGateway's (nuget.org `*` + `dohertj2-gitea` source with `ZB.MOM.WW.*` packageSourceMapping; credentials stay user-level, never committed).
 ### Task 3.2 — registry options + validation
 `Registry/OverviewOptions.cs` (+ `ApplicationEntry`, `InstanceEntry`) binding section `Overview` per the design §3 schema: global `PollIntervalSeconds`(10)/`TimeoutSeconds`(3)/`StaleAfterSeconds`(45); per-application and per-instance overrides (instance wins); instance fields `Name`, `Group?`, `BaseUrl`, `ManagementUrl?`, `HasActiveRole`(false).
 `Registry/OverviewOptionsValidator.cs` via `OptionsValidatorBase` + `AddValidatedOptions<OverviewOptions, OverviewOptionsValidator>` (`ZB.MOM.WW.Configuration`): non-empty applications, ≥1 instance each, unique app names + unique instance names within an app, absolute http/https `BaseUrl`/`ManagementUrl`, all intervals positive, **effective stale > effective poll for every instance**. Plus `ConfigPreflight.For(builder.Configuration).RequireValue("Overview:Applications:0:Name")…`-style presence check pre-`Build()` (HistorianGateway `Program.cs:62-66` is the pattern).
 ### Task 3.3 — health client + status model
 `Polling/ZbHealthReportClient.cs`: GETs a URL, deserializes the canonical shape `{status, totalDurationMs, entries:{name:{status, description, durationMs, data?}}}` — `data` values as `JsonElement` (they arrive as arbitrary JSON; convert to string/int on read). Treat both 200 and 503 as *parseable* responses (503 carries a body too); classify non-JSON/undeserializable as Unreachable. In-app for v1 (design §8.2).
 `Polling/InstanceStatus.cs`: `Up | Degraded | Down | Unreachable` (+ separate `ActiveState: Active|Standby|Unknown|NotApplicable`, `IsStale` computed at render). Derivation per design §5, including two-strike flap damping (state degrades only after 2 consecutive failures; recovery immediate).
 ### Task 3.4 — poller + snapshot store
 `Polling/OverviewPollerService.cs : BackgroundService`: `PeriodicTimer(PollIntervalSeconds)`; each tick polls all instances in parallel (`Task.WhenAll`; per-instance `CancellationTokenSource` with its effective timeout; named `HttpClient` via `IHttpClientFactory`; no retries — next tick is the retry). Per instance: `/health/ready` always; `/health/active` when `HasActiveRole`.
 `Polling/OverviewSnapshotStore.cs`: single immutable snapshot object swapped atomically (`Volatile.Write`/`Interlocked.Exchange`); each instance snapshot carries `LastUpdatedUtc`, latency, parsed entries, derived status, consecutive-failure counter. A C# `event`/`Action` notifies circuits to re-render on swap. **Pages never poll** — they read the store only (cache-first requirement).
 Staleness: computed at render (`now - LastUpdatedUtc > effective StaleAfterSeconds` ⇒ render Stale regardless of stored status); the page also runs a small UI tick (~5 s) so ages/staleness advance without a poll event.
 ### Task 3.5 — leader aggregation
 `Polling/LeaderResolver.cs`: for each `(Application, Group)` with any `HasActiveRole` instance, collect `entries["akka-cluster"].data.leader` from members whose status is Up/Degraded; resolve the Akka address (`akka.tcp://<system>@host:port`) to a registry instance by **host match against `BaseUrl`** (fall back to showing the raw address when unmatched — different ports are expected, the Akka port is not the HTTP port); if reporting members disagree → set the group's `LeaderDisagreement` flag (split-brain tell). Pure static logic → heavily unit-testable.
 ### Task 3.6 — UI
 **Visual reference (match it):** `docs/mockups/overview-dashboard-mockup.html` — a static, self-contained mockup built from the real Theme tokens + embedded IBM Plex; open it in a browser before styling. It defines: the KPI strip treatment (alert/caution tints), the instance-card anatomy (status stripe on the top edge; **dashed border = Unreachable, solid = Down**; faded body + "stale since" for Stale), group headers with `Leader · <instance>` chips and the split-brain warning chip, footer `ready/active` raw-signal line, and the check-detail `<details>` layout with the monospace `data` line. Reuse Theme classes where they exist (`chip-*`, `panel`, `kv`, `agg-*`); the mockup's extra classes (`inst*`, `group*`, `check-*`) go into the app's own `site.css`.
 Copy HistorianGateway's Blazor skeleton, minus every auth piece:
 - `Components/App.razor` (head order matters: Bootstrap CSS → `<ThemeHead/>` → `site.css` → `HeadOutlet @rendermode=InteractiveServer`; body: `<Routes @rendermode=InteractiveServer/>` → Bootstrap JS → `<ThemeScripts/>` → `blazor.web.js`), `Routes.razor` with plain `RouteView` (no `AuthorizeRouteView`), `_Imports.razor` incl. `@using ZB.MOM.WW.Theme`.
 - `Components/Layout/MainLayout.razor` → `ThemeShell Product="SCADA Overview" Accent="…"` (pick an unused accent; HG is `#2f855a`), Nav = single `NavRailItem Href="/" Match="NavLinkMatch.All"` (+ a Health link later if wanted), no AuthorizeView anywhere.
 - `Components/Pages/Overview.razor` (`@page "/"`): header strip (counts n Up/Degraded/Down/Unreachable/Stale, last-refresh clock, pause/resume toggle — pause just stops applying poll events + lets staleness paint), then per-application sections: app name + worst-status rollup pill; cluster groups get a sub-header with `Leader: <instance>` chip (`StatusPill Info`) and the disagreement warning chip when flagged; instance grid of `InstanceCard`s.
 - `Components/Widgets/InstanceCard.razor`: `TechCard` + `StatusPill` composition — name/group, status pill (mapping per design §5), Active badge, Leader badge, latency, relative last-seen ("stale since …" once stale), native `<details>` check list (per-check pill + description + data leader line for akka-cluster), `TechButton` (Ghost/Secondary) → `ManagementUrl` (`target="_blank" rel="noopener"`).
 - Program wiring: `AddRazorComponents().AddInteractiveServerComponents()`; `MapRazorComponents<App>().AddInteractiveServerRenderMode()` — **no** `RequireAuthorization`, no antiforgery/cascading auth state; `UseStaticFiles()` before endpoints (Theme assets come from `_content/ZB.MOM.WW.Theme/`).
 ### Task 3.7 — self-observability
 `AddZbSerilog(o => o.ServiceName = "overview")`; `AddZbTelemetry(o => { o.ServiceName = "overview"; o.Meters = [OverviewMetrics.MeterName]; })` — **`Meters` is a silent allowlist: forgetting the custom meter name silently drops the gauge** (known family gotcha). `OverviewMetrics`: observable gauge `zb_overview_instance_status` (labels app/instance/group; value = status enum ordinal) + poll-duration histogram. Register own checks: `registry-loaded` and `last-poll-cycle-completed` (Unhealthy if no completed cycle within 2× poll interval — makes the dashboard's own `/health/ready` honest). `MapZbHealth()` + `MapZbMetrics()`.
 ### Task 3.8 — tests (`tests/ZB.MOM.WW.Overview.Tests`)
 xunit 2.9.3 + `xunit.runner.visualstudio` 3.1.4 + `Microsoft.NET.Test.Sdk` 17.14.1 + **bunit 1.40.0** (Theme's own test stack — imitate `StatusPillTests.cs`: `class X : TestContext`, `RenderComponent<T>`, class-list asserts):
 - Validator table tests (incl. stale ≤ poll rejected, relative URL rejected).
 - Health-client golden tests against **captured real payloads** from all four apps (capture during Phase 4 prep; until then use handcrafted fixtures matching `ZbHealthWriter` incl. entries with and without `data`).
 - Status derivation: 200/Healthy→Up, 200/Degraded→Degraded, 503→Down, timeout/refused/garbage→Unreachable, flap damping (1 failure holds, 2 flips, recovery immediate), active 200/503/error → Active/Standby/Unknown.
 - Staleness: fresh vs. aged snapshot with per-instance override.
 - LeaderResolver: agreement, disagreement flag, address→instance host matching, unmatched fallback.
 - Snapshot store: atomic swap + event fired.
 - bUnit: `InstanceCard` renders pill class/badges/link per state; Overview page smoke over a stubbed store.
 - `WebApplicationFactory<Program>` boot tests: page + `/health/ready` + `/metrics` reachable **with no auth**; malformed registry → boot fails with the `ConfigPreflight`/validator message.
 ### Task 3.9 — config + docker
 - `appsettings.json`: Logging skeleton, empty `Serilog`/`Telemetry` sections (libs default them), `Overview` registry sample; `appsettings.Development.json`: Kestrel `Http` endpoint `http://localhost:5320` + a dev registry pointing at the docker rigs. **Derive the rig instance URLs from the compose files, don't guess:** OtOpcUa docker-dev rig → `~/Desktop/OtOpcUa` docker-dev compose (6 nodes; AdminUI reachable on host :9200 — per-node published HTTP ports are in the compose port mappings); ScadaBridge local cluster → `~/Desktop/ScadaBridge/docker/` compose (central pair UI ports + site-node :8084 mappings); mxgw → `windev:5130`; HistorianGateway → its docker compose (:5220) or `wonder-app-vd03:5222` (VPN). Production Kestrel via env (`Kestrel__Endpoints__Http__Url`), the HG pattern; **don't set `ASPNETCORE_URLS` alongside Kestrel endpoint config** (URLs-override trap).
 - `docker/`: HG-pattern runtime-only image — host-side `dotnet publish -c Release -o docker/publish -p:UseAppHost=false` (host has the Gitea feed creds), `FROM mcr.microsoft.com/dotnet/aspnet:10.0`, `COPY publish/ ./`, `EXPOSE 5320`, `ENTRYPOINT ["dotnet","ZB.MOM.WW.Overview.dll"]`; compose project name `zb-overview`, port `5320:5320`, `ASPNETCORE_ENVIRONMENT` + Kestrel env vars. No env_file needed (no secrets). Note: `aspnet:10.0` has no `curl` — healthcheck via `wget`/dotnet or skip.
 **Phase DoD:** `dotnet build` 0 warnings; `dotnet test` green; `dotnet run` on :5320 renders the page instantly from cache against a dev registry; container builds and runs.
 ---
 ## Phase 4 — live acceptance (design §9)
 Prep: capture real `/health/ready` + `/health/active` payloads from all four apps (post-0.2.0) into the golden-test fixtures. Then run the design §9 checklist end-to-end on the rigs (OtOpcUa 6-node docker-dev + ScadaBridge local cluster + windev mxgw + wonder-app-vd03 HG if reachable/VPN):
 1. Every registered instance renders correctly; single Active per pair (central AND site).
 2. **Leader chip correct per cluster group and moves when the leader node is stopped**; split-brain flag not shown in normal operation.
 3. Kill a node → Unreachable within ~2 cadences; restart → recovers.
 4. Force a Degraded check → Warn with failing entry visible.
 5. Cache-first: first page load instant, no poll triggered by the request.
 6. Pause auto-refresh past `StaleAfterSeconds` → all cards Stale; resume → clean recovery. Also stop the poller (kill the service loop) → same.
 7. Management links open the right UIs; page reachable with **no login**.
 8. Malformed registry → boot fails with a clear preflight message.
 9. `curl :5320/health/ready` + `/metrics` (gauge present with per-instance labels).
 **Program DoD:** all 9 pass; ScadaBridge PR merged; OtOpcUa bump merged; Health 0.2.0 on the feed; scadaproj commits for the lib + new app + docs.
 ---
 ## Cross-cutting notes for the executor
 - **Family gotchas that WILL bite:** `ZbTelemetryOptions.Meters` silent allowlist; mxgw/HG have no `Directory.Packages.props` (inline pins); never host-`sqlite3` a live container WAL DB during rig checks; `Akka.TestKit.Xunit2` is xunit-v2-only (fine here — everything in this plan is xunit 2.9.3); Gitea feed verification uses `/api/packages/dohertj2/nuget/registration/…`.
 - **Do not** add auth "while at it" — anonymous read-only is a requirement, not an omission.
 - **Contract discipline:** the `data` field is additive and optional; the dashboard must render fully (minus leader) against 0.1.0-era payloads — that's what keeps Phase 3 unblocked by Phases 1/2 and tolerant of un-bumped deployments.
 - Commit style: lib change + publish in scadaproj (`feat(health): 0.2.0 …`); ScadaBridge and OtOpcUa changes as PRs in their own repos referencing this plan; new app committed in scadaproj.
@@ -0,0 +1,86 @@
 # Central overview dashboard for the scadaproj family — options investigation
 **Date:** 2026-07-22
 **Goal:** a single pane showing status of every app instance (OtOpcUa, ScadaBridge, MxAccessGateway, HistorianGateway — each with multiple nodes/pairs), with deep links to each instance's own management UI, pre-configured via an appsettings registry of applications and instances. "Aspire-dashboard-like."
 ---
 ## 1. What the family already gives us (verified in code, 2026-07-22)
 The reconnaissance result is unusually favorable: **the data plane for a poll-based overview already exists on all four apps, with zero changes needed in the target apps.**
 - **Health endpoints are uniform and anonymous everywhere.** All four apps call the shared `MapZbHealth()` (`ZB.MOM.WW.Health`), which maps `/health/ready`, `/health/active`, and `/healthz`, all `.AllowAnonymous()` in the library itself (`ZbHealthEndpointExtensions.cs:35-64`). Even OtOpcUa's global `FallbackPolicy = RequireAuthenticatedUser` doesn't gate them (endpoint `AllowAnonymous` wins).
 - **The JSON contract is canonical and stable** (`ZbHealthWriter.cs:45-81`, camelCase):
  ```json
  { "status": "Healthy|Degraded|Unhealthy",
    "totalDurationMs": 12.3,
    "entries": { "<check>": { "status": "...", "description": null, "durationMs": 1.2 } } }
  ```
  Healthy/Degraded → HTTP 200, Unhealthy → 503.
 - **`/health/active` is the active/standby signal.** ScadaBridge Central's `active` tier runs the oldest-member `active-node` check; OtOpcUa has `admin-leader`. No off-the-shelf product can express active/standby natively — this endpoint makes it a first-class dashboard column for us.
 - **`/metrics` (Prometheus) is anonymous on all four** (explicitly on OtOpcUa; by absence of a fallback policy on the others). OTLP push is config-only on three apps (`<App>:Telemetry:Exporter=Otlp` + `OtlpEndpoint`); **HistorianGateway hardcodes its telemetry setup and would need a one-line change** to gain the config keys.
 - **Nothing aggregates across apps today.** The closest is ScadaBridge's `/monitoring/health` (CentralUI), but it's push-based over the site transport and scoped to ScadaBridge sites in one cluster. OtOpcUa/HG/mxgw pages show only their own instance. A multi-app registry + remote HTTP poller is genuinely new work.
 - **Reusable building blocks for an in-house pane:** `ZB.MOM.WW.Theme` (`ThemeShell`, `StatusPill` with Ok/Warn/Bad/Idle/Info, `TechCard`), `ZB.MOM.WW.Auth.AspNetCore` (the same hardened LDAP cookie login all four apps use), `ZB.MOM.WW.Configuration` (validate the registry at startup), `ZB.MOM.WW.Telemetry`/`Health` (the dashboard monitors itself the family way). No prebuilt tile/grid widget exists — a node-card composite would be new, but it's `TechCard` + `StatusPill` composition.
 - **Known ports for the registry:** OtOpcUa AdminUI dev `:9000` (driver-only nodes fall back to `:5000`); ScadaBridge Central `:5000/:5001` (site nodes expose only `:8084` metrics — no health/UI); mxgw `:5120/:7121`; HistorianGateway dashboard `:5220` dev / single TLS ALPN port (rec. `:5222`) in production.
 **Watch-out for the registry design:** ScadaBridge *site* nodes map no health endpoints at all (metrics only), so a site-pair row can't be polled the same way as a central pair — either poll `:8084/metrics` for liveness, read site status from Central's aggregator, or (later) add `MapZbHealth` to site nodes.
 ## 2. Off-the-shelf candidates (facts verified by fetch, 2026-07-22)
 | Option | Version / status | Config-file registry | Persistence | Auth | Verdict |
 |---|---|---|---|---|---|
 | **Aspire dashboard (standalone)** | Aspire 13 (Nov 2025); `mcr.microsoft.com/dotnet/aspire-dashboard` | **No** — resource UI is disabled in standalone; needs a custom gRPC resource service to light up | None (in-memory, lost on restart) | BrowserToken / OIDC | Not the ops pane. MS docs: dev/short-term diagnostic tool. `AddExternalService()`/custom URLs are AppHost run-mode features only. |
 | **AspNetCore.HealthChecks.UI (Xabaril)** | 9.0.0 (Dec 2024), net8.0 TFM only; upstream coasting on dependabot since Aug 2025 ("looking for maintainers"); tiny `DotNetDiag` fork exists | **Yes** — `HealthChecksUI:HealthChecks` array in appsettings; exactly the requested model | SQLite/SQL Server/InMemory | In-process — **our LDAP cookie works unmodified** | Closest functional match, but it **requires the `UIResponseWriter` payload** (issue #285) — our `ZbHealthWriter` shape is close but not identical (`durationMs` vs `duration`, no `data`/`tags`), so every app would need a second UI-format endpoint. Plus a slow-moving upstream dependency. |
 | **Gatus** | v5.36.0 (May 2026), active | **Yes** — YAML; conditions incl. `[BODY]` JSONPath, so it can evaluate our health JSON **with zero app changes** (incl. degraded) | SQLite/Postgres | Built-in basic/OIDC (no LDAP cookie) | Best zero-effort comparable: registry + JSON conditions + uptime history + 30+ alert providers in one Go binary. Trade: non-.NET, non-Theme UI, outside our auth stack. |
 | **Homepage (gethomepage.dev)** | v1.13.2 (Jun 2026) | Yes — YAML link tiles | n/a | None (reverse proxy) | Link portal only; `siteMonitor` is status-code-only — cannot express Degraded or active/standby. Out. |
 | **Grafana + Prometheus + blackbox** | Grafana 13.0 (Apr 2026) | Yes — scrape config | Full TSDB history | **Native LDAP** | Most durable, real alerting, grows into actual ops monitoring. Highest setup (3 containers + dashboard authoring); Degraded is invisible to blackbox (HTTP 200) without a status gauge metric or `ResultStatusCodes` remap; deep links via data links are clunkier than a real portal. |
 ## 3. Options
 ### Option A — build a small in-house overview app (`ZB.MOM.WW.Overview`) — **RECOMMENDED**
 A thin Blazor Server app hosted in scadaproj alongside the other shared components, composed almost entirely from the family's own libraries:
 - **Registry:** `Overview:Applications[]` in appsettings — per app: name, kind, accent; per instance: display name, site/role, `healthBaseUrl`, `managementUrl`, optional `metricsUrl`, `expectActive` flag, poll interval/timeout overrides. Validated at startup via `ZB.MOM.WW.Configuration` (`AddValidatedOptions` + `ConfigPreflight`) — malformed registry fails the boot, not the first poll.
 - **Poller:** one `BackgroundService` + `IHttpClientFactory`; polls each instance's `/health/ready` + `/health/active` in parallel on a fixed cadence (5–15 s), deserializes the canonical `ZbHealthWriter` JSON (a ~30-line typed client — worth putting in `ZB.MOM.WW.Health` as `ZbHealthReportClient` so it's contract-owned), and keeps an in-memory snapshot. Statuses per instance: **Up / Degraded / Down / Unreachable**, plus **Active/Standby** from `/health/active` — the column no off-the-shelf option can render.
 - **UI:** `ThemeShell` + a new node-card grid built from `TechCard` + `StatusPill`; per-check expandable detail (the `entries` dict); every card deep-links to the instance's own management page. LDAP cookie login via `AddZbLdapAuth` — same UX as the other four apps (the dashboard reads only anonymous endpoints, but the pane itself should sit behind the family login).
 - **Self-hosting:** it registers its own `MapZbHealth`/`MapZbMetrics` like any family app; a `docker/` folder mirroring HistorianGateway's runtime-only image pattern.
 **Fit:** best-practice fit for this shop. The requested feature set (appsettings registry, status, deep links) maps 1:1 onto libraries that exist precisely so new apps can be assembled this way; the only genuinely new code is the poller + snapshot + one grid page — realistically **2–4 days** for a first useful version. It also keeps auth, theming, and ops conventions identical to the rest of the family, which none of the off-the-shelf options can do. Deliberate scope cut for v1: no history/alerting (that's Option D's job if it's ever needed — and the registry file ports straight to a Prometheus scrape config).
 **Costs / risks:** you own the code (small); no built-in uptime history or webhooks in v1; the ScadaBridge-site-node gap (§1 watch-out) needs a decision in the registry design.
 ### Option B — AspNetCore.HealthChecks.UI embedded in a thin family host
 Host the Xabaril UI inside a small ASP.NET app behind our LDAP cookie. Gets registry-from-appsettings, polling, SQLite history, and webhooks for near-zero UI code.
 **Fit:** the strongest off-the-shelf functional match, and the only one where `ZB.MOM.WW.Auth` works unmodified. But it costs a change in **all four target apps** (a second health endpoint in `UIResponseWriter` format, because the UI hard-requires that payload), buys a visibly unmaintained upstream (net8.0-only TFM, dependabot-only commits since Aug 2025), has no active/standby concept, and its UI will never look like Theme. For a shop that already owns Health + Theme + Auth libs, it saves less than it costs.
 ### Option C — Gatus container as a zero-app-change status page
 Deploy Gatus with a YAML registry; its `[BODY]` JSONPath conditions consume our existing health JSON directly (including Degraded), and it brings uptime history + alerting for free.
 **Fit:** the best effort-to-value stopgap — running in an afternoon, no app changes, actively maintained. Best practice would call this the "buy" answer if the family didn't have its own UI/auth stack. Trades: separate basic/OIDC auth outside the LDAP cookie, un-Theme-able UI, and active/standby only expressible as another endpoint row rather than a paired-node concept. Reasonable as an interim pane while Option A is built, or as a permanent low-ceremony answer if the dashboard turns out to be rarely used.
 ### Option D — Grafana + Prometheus + blackbox exporter
 **Fit:** the right long-term answer **if** the real goal grows into ops monitoring (history, alerting, SLOs) rather than a convenience pane — native LDAP auth is a genuine plus. As a *first* deliverable for "status + links" it's the highest effort and the worst at deep links/branding, and Degraded/active-standby need extra plumbing (a health-status gauge in `ZB.MOM.WW.Telemetry` would be the clean fix, and is worth doing eventually regardless). Not first; a natural phase 2 that Option A's registry feeds directly.
 ### Option E — Aspire dashboard standalone
 **Fit:** rejected as the overview pane — in standalone mode the resource/status/links UI is disabled (it's an OTLP telemetry viewer only), nothing persists across restarts, and Microsoft positions it explicitly as a dev/short-term diagnostic tool. **Worth keeping as a free companion tool:** three of the four apps can push OTLP with config only, so an ad-hoc `aspire-dashboard` container gives structured logs + traces during debugging for zero code. (HistorianGateway needs the one-line telemetry-config change to join.)
 > **DECISION (2026-07-22):** Option A chosen — custom app, family look-and-feel (Theme/Auth/Health kit). Aspire rejected for the pane. V1 design: [`docs/plans/2026-07-22-overview-dashboard-design.md`](docs/plans/2026-07-22-overview-dashboard-design.md).
 ## 4. Recommendation
 **Option A** — build the thin `ZB.MOM.WW.Overview` app from the family's own parts, with two riders:
 1. If an interim pane is wanted *this week*, stand up **Gatus (Option C)** against the existing health endpoints — zero app changes, discard later.
 2. Keep **Aspire standalone (Option E)** in the toolbox as a dev-time OTLP viewer, and make HistorianGateway's telemetry config-driven (one line) so all four apps can join it.
 Decisions to make before building A: how site-node ScadaBridge rows are sourced (metrics-liveness vs Central-aggregator vs adding MapZbHealth to sites), and whether v1 needs any history/alerting (recommended: no — defer to Option D if ever needed).
 ---
 *Sources: family code (ZB.MOM.WW.Health `ZbHealthEndpointExtensions.cs`/`ZbHealthWriter.cs`, per-app Program.cs wiring); aspire.dev standalone/configuration/security docs + Aspire 13 announcement; Xabaril repo + NuGet + issue #285 + DotNetDiag fork; TwiN/gatus releases; gethomepage.dev docs; Grafana/blackbox dashboards. All web claims verified by fetch 2026-07-22.*