Compare commits
9 Commits
5919e29969
..
main
| Author | SHA1 | Date | |
|---|---|---|---|
 Joseph Doherty
|
6721c8a224 | ||
|
Joseph Doherty
|
3dbd6cd57c | ||
|
Joseph Doherty
|
92015da4e0 | ||
|
Joseph Doherty
|
1911de2c3c | ||
|
Joseph Doherty
|
b5e7bc8b3c | ||
|
Joseph Doherty
|
4d93d2580b | ||
|
Joseph Doherty
|
6dd7858619 | ||
|
Joseph Doherty
|
ff58ad4f60 | ||
|
Joseph Doherty
|
dbfd5e7a7e |
@@ -0,0 +1,369 @@
|
||||
# Deep dive: an MsQuic transport for Akka.NET 1.5.62
|
||||
|
||||
*Research report, 2026-07-22. Question: what would it take to implement the Akka.NET 1.6 MsQuic
|
||||
transport (akka.net #7466) against the Akka.NET version currently used by ScadaBridge and OtOpcUa,
|
||||
and would features like higher throughput and larger-message support reduce the family's reliance
|
||||
on gRPC?*
|
||||
|
||||
**TL;DR:** Building a QUIC transport against the current Akka.NET (1.5.62) is genuinely feasible —
|
||||
the classic transport SPI is public, and Akka's own docs literally use a hypothetical QUIC transport
|
||||
as the custom-transport example. Roughly 3–5 weeks to production quality. But there's a structural
|
||||
catch: plugged in under the classic 1.5 remoting pipeline, QUIC is reduced to "TCP with TLS 1.3 and
|
||||
cheaper handshakes." The two things actually wanted — throughput multiplexing and non-blocking large
|
||||
messages — come from the *Artery pipeline rewrite above the transport*, which is the part of #7466
|
||||
that doesn't exist yet and can't be replicated through the 1.5 SPI. And most of the family's gRPC
|
||||
wouldn't go away regardless, because most of it isn't Akka-adjacent. Details below, options and a
|
||||
recommendation at the end.
|
||||
|
||||
---
|
||||
|
||||
## 1. Where #7466 actually stands (checked 2026-07-22)
|
||||
|
||||
- [Issue #7466](https://github.com/akkadotnet/akka.net/issues/7466) is the **v1.6 "Artery" epic**,
|
||||
opened Jan 2025: MsQuic-based Remoting V2 replacing the never-built Aeron plan. Design goals:
|
||||
per-stream multiplexing ("this is where all of the significant performance gains come from" per
|
||||
the [Petabridge v1.6 roadmap](https://petabridge.com/blog/akkadotnet-v1.6-roadmap/)), mandatory
|
||||
TLS 1.3, automatic chunking of large messages onto dedicated streams, a system-message channel
|
||||
that works even while quarantined, Akka.Streams backpressure, actor-ref compression, and a
|
||||
2–3M msg/sec target. Requires .NET 9+.
|
||||
- **Status: design-only.** No linked PRs, no branches, no assignee; the only substantive activity is
|
||||
a January 2025 design discussion (to11mtm/Aaronontheweb) about channel counts, ordering
|
||||
guarantees, and serialization. The issue itself says a protocol spec is needed before
|
||||
implementation starts.
|
||||
- **The timeline has slipped badly.** The roadmap targeted alphas end of Q1 2025 and completion
|
||||
early Q1 2026. As of July 2026 the
|
||||
[releases page](https://github.com/akkadotnet/akka.net/releases) shows **zero 1.6 previews** —
|
||||
latest is 1.5.70 (July 3, 2026), and a 1.5.71 milestone is open. The
|
||||
[1.6.0 milestone](https://github.com/akkadotnet/akka.net/milestones) reads 44% complete, but the
|
||||
closed items are the AOT/trimming/serialization workstreams, not the transport. A
|
||||
[PR search for "quic"](https://github.com/akkadotnet/akka.net/pulls?q=is%3Apr+quic) confirms no
|
||||
transport code has ever landed.
|
||||
|
||||
So "wait for 1.6" is not a near-term plan. QUIC benefits this year would mean building it in-house.
|
||||
|
||||
> **STATUS UPDATE (2026-07-22, same day, deeper look):** the two paragraphs above are correct about
|
||||
> *QUIC* but stale about *Artery* — searching PRs for "quic" missed work named "artery".
|
||||
> **Artery implementation began 2026-07-03 and is landing rapidly on `dev`, as Artery-TCP** (the
|
||||
> JVM-parallel move): design spec via openspec (#8313, "artery-tcp-remoting"), transport-substrate
|
||||
> validation gate G0 (#8315), TCP framing + envelope codec G1 (#8319), handshake/association G2
|
||||
> (#8320), parallel inbound/outbound lanes (#8356/#8357, Jul 11), dedicated large-message stream
|
||||
> streamId 3 (#8352, Jul 10), quarantine port in flight (#8416), MNTR/CI integration for both
|
||||
> transports (#8373/#8413). 192/192 Artery specs passing as of #8352. In parallel,
|
||||
> **Serialization.V2** (source-generated + MessagePack internals) is in heavy development — ~30
|
||||
> merged PRs with wire-format snapshot tests and an open cross-component integration proof suite
|
||||
> (#8407). TCP-first also confirms §7's forward-looking note: the Artery benefits (lanes,
|
||||
> large-message stream, control stream) are arriving **transport-agnostic over TCP** — no libmsquic
|
||||
> required — with QUIC as a later substrate swap. No preview release or public benchmark numbers
|
||||
> yet; see the 1.6-status section of the conversation record for the remaining-work list.
|
||||
|
||||
## 2. What the projects run today
|
||||
|
||||
Both ScadaBridge and OtOpcUa pin **Akka 1.5.62** and use the classic **`dot-netty.tcp`** transport,
|
||||
plaintext, with all size defaults — meaning the **128 KB `maximum-frame-size` cap** applies
|
||||
(`ScadaBridge/src/ZB.MOM.WW.ScadaBridge.Host/Actors/AkkaHostedService.cs:310`;
|
||||
`OtOpcUa/src/Core/ZB.MOM.WW.OtOpcUa.Cluster/Resources/akka.conf:25`). Neither config touches TLS or
|
||||
frame size. The Akka traffic is cluster gossip, heartbeats, singleton/pub-sub coordination, and
|
||||
(ScadaBridge) actor messaging in a hub-and-spoke — modest volume, nowhere near transport-bound.
|
||||
|
||||
Two facts that matter for the "do we even need this" question:
|
||||
|
||||
- **The frame size can be raised today.** `akka.remote.dot-netty.tcp.maximum-frame-size = 10MiB`
|
||||
(plus matching send/receive buffer sizes) is a config change, not a transport change. The 128 KB
|
||||
default is a *guardrail*, not a TCP limitation — large messages on the single ordered association
|
||||
channel delay heartbeats behind them, which is exactly the false-failure-detection risk the
|
||||
2s/10s failure-detector tuning cares about.
|
||||
- **TLS on the existing transport is already available.** Akka.Remote's DotNetty transport gained
|
||||
**mutual TLS** in [PR #7851](https://github.com/akkadotnet/akka.net/pull/7851) (merged Oct 2025 —
|
||||
included in the pinned 1.5.62). If encryption-in-flight is a driver, QUIC is not required for it.
|
||||
|
||||
## 3. How a transport plugs into 1.5 — and what it would take
|
||||
|
||||
Akka.Remote has a public custom-transport SPI
|
||||
([docs](https://getakka.net/articles/remoting/transports.html)): implement
|
||||
`Akka.Remote.Transport.Transport`, register it via HOCON, and everything above it (handshake,
|
||||
endpoint management, quarantine, cluster) is unchanged:
|
||||
|
||||
```hocon
|
||||
akka.remote {
|
||||
enabled-transports = ["akka.remote.zb-quic"]
|
||||
zb-quic {
|
||||
transport-class = "ZB.MOM.WW.Akka.Transport.Quic.QuicTransport, ZB.MOM.WW.Akka.Transport.Quic"
|
||||
transport-protocol = quic # addresses become akka.quic://System@host:port
|
||||
hostname = "0.0.0.0"
|
||||
port = 4053 # UDP now, not TCP
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
The concrete build, as a shared lib in scadaproj (the usual pattern):
|
||||
|
||||
**Core (~1–2 weeks):**
|
||||
- `QuicTransport : Transport` — ctor `(ActorSystem, Config)` (instantiated reflectively); implement
|
||||
`SchemeIdentifier` (`"quic"`), `MaximumPayloadBytes` (configurable — this is where "large
|
||||
messages" becomes trivial), `Listen()` (a `QuicListener.ListenAsync` with a private ALPN like
|
||||
`"akka-remote"`, surfacing `InboundAssociation` events), `Associate(Address)`
|
||||
(`QuicConnection.ConnectAsync` + open one bidirectional `QuicStream`), `Shutdown()`.
|
||||
- `QuicAssociationHandle : AssociationHandle` — `Write(ByteString)` must be **non-blocking with a
|
||||
boolean backpressure signal** (return `false` when the send queue is full; Akka's `EndpointWriter`
|
||||
buffers and retries — getting this semantic right is the subtle part), `Disassociate()` (map to
|
||||
`CloseAsync` with an app error code).
|
||||
- **Framing** — QUIC streams are byte streams like TCP, so the 4-byte length-prefix framing that
|
||||
dot-netty does must be re-implemented on both sides. The Akka protocol handshake/PDU layer rides
|
||||
on top automatically.
|
||||
|
||||
**Security config (~0.5–1 week):** QUIC **mandates TLS 1.3** — there is no plaintext mode. Every
|
||||
node needs a certificate, and a validation policy is required (mutual TLS with a pinned internal CA
|
||||
is the sane one; the Secrets lib is a natural home for cert material). This is new operational
|
||||
surface the rigs don't have today.
|
||||
|
||||
**Hardening + tests (~1–2 weeks):** association teardown races, quarantine behavior, reconnect
|
||||
storms, failure-detector behavior under load, big-frame tests, and soak on the docker-dev rig.
|
||||
Historical note for calibration: the Helios→DotNetty transition took Akka.NET years of corner-case
|
||||
fixes (dissociation races, quarantine storms). A custom transport is its own support island for
|
||||
that class of bug.
|
||||
|
||||
**Migration:** the address scheme changes (`akka.tcp://` → `akka.quic://`), so seed nodes and every
|
||||
stored address change; mixed-transport rolling upgrade isn't practical for a pair — but *both*
|
||||
transports can be listed in `enabled-transports` during transition, and the site pairs already
|
||||
stop/start together by policy, so cutover is easy here.
|
||||
|
||||
## 4. The structural catch
|
||||
|
||||
The classic SPI models an association as **one ordered, opaque byte channel**. The remoting layer
|
||||
above it (system-message resend, handshake, cluster failure detection) *depends* on per-association
|
||||
ordering, and the transport never sees message recipients — just framed byte blobs. Consequences:
|
||||
|
||||
- **No multiplexing win.** One association cannot be fanned across N QUIC streams without either
|
||||
breaking ordering guarantees or parsing Akka's PDU envelope inside the transport to route by
|
||||
recipient while preserving per-pair ordering — at which point you are rewriting Artery's
|
||||
dispatcher against internals with no stability contract.
|
||||
- **No dedicated large-message or control channel.** A 10 MB frame still queues ahead of heartbeats
|
||||
on the same stream — identical head-of-line behavior to TCP today. The 1.6 design fixes this
|
||||
*above* the transport; the 1.5 SPI can't express it.
|
||||
- What you *do* get: TLS 1.3 built in, faster connection establishment (nice for
|
||||
reconnect-after-failover, though failover latency is dominated by the 10–15s
|
||||
failure-detector/downing windows, not handshakes), connection migration, and an easy big
|
||||
`MaximumPayloadBytes`.
|
||||
|
||||
Net: a 1.5-SPI QUIC transport ≈ **dot-netty.tcp + mTLS + a raised frame size** — both of which are
|
||||
available today with config on the stock transport.
|
||||
|
||||
## 5. What 1.6 actually changes — the layer split, made explicit
|
||||
|
||||
1.6 doesn't just swap the transport; it rewrites the remoting pipeline *above* the transport
|
||||
(Artery), and that pipeline is where every benefit §4 marked "not achievable on 1.5" actually lives.
|
||||
|
||||
**In 1.5**, remoting is two layers: the pluggable `Transport` SPI at the bottom (a dumb ordered byte
|
||||
pipe — `Associate()`, `Write(ByteString)`, `Disassociate()`) and above it the fixed pipeline:
|
||||
`AkkaProtocolTransport` handshake, one `EndpointWriter`/`EndpointReader` actor pair per association,
|
||||
system-message resend buffering, and quarantine. That pipeline bakes in three assumptions no custom
|
||||
transport can change:
|
||||
|
||||
1. **One association = one ordered channel** — the system-message resend protocol and cluster
|
||||
failure detection depend on strict ordering over that single channel.
|
||||
2. **Payloads are opaque to the transport** — recipient, message class, and size are invisible by
|
||||
the time bytes reach `Write()`, so "big message → own lane" or "heartbeat → priority lane"
|
||||
routing is impossible without parsing Akka's internal PDU format (i.e. reimplementing the layer
|
||||
above against unstable internals).
|
||||
3. **Everything shares the lane** — user messages, system messages, cluster heartbeats. The 128 KB
|
||||
frame guardrail exists exactly because a big message parking in front of heartbeats trips the
|
||||
failure detector.
|
||||
|
||||
**In 1.6/Artery**, that middle layer is replaced with a recipient-aware, multi-lane, Akka.Streams-
|
||||
based pipeline: it knows recipient and size *before* the wire, assigns messages to QUIC streams
|
||||
accordingly (ordering preserved per sender→receiver pair rather than per node pair), chunks
|
||||
oversized messages onto **dedicated large-message streams**, keeps **system/control messages on
|
||||
their own stream** (including a quarantine-bypass channel), adds headers for version negotiation,
|
||||
and compresses actor refs to integers. QUIC is almost incidental — it is chosen because it gives
|
||||
cheap independent streams for that pipeline to map onto.
|
||||
|
||||
| Benefit | 1.5 + custom QUIC transport | 1.6 Artery | Why the pipeline is the difference |
|
||||
|---|---|---|---|
|
||||
| Throughput / multiplexing | ✗ (single ordered lane) | ✓ (the 2–3M msg/s target) | Lane assignment needs recipient visibility, which only the pipeline has |
|
||||
| Large messages without side channels | ~ (raise the frame cap; heartbeats still queue behind big frames) | ✓ (dedicated chunked streams; control lane unaffected) | Size-based routing happens above the transport |
|
||||
| Honest failure detection under load | ✗ (heartbeats share the data lane) | ✓ (isolated control stream) | Same |
|
||||
| Split-brain handling while quarantined | ✗ | ✓ (quarantine-bypass channel) | Quarantine lives in the pipeline, not the transport |
|
||||
| TLS 1.3 | ✓ (but dot-netty mTLS gets there too) | ✓ | Transport-level — the one benefit 1.5 delivers fully |
|
||||
| Fast handshake / connection migration | ✓ | ✓ | Transport-level |
|
||||
|
||||
Consequences for this report's conclusions:
|
||||
|
||||
- **The "don't build it on 1.5" recommendation strengthens** — a 1.5 transport captures only the
|
||||
bottom two rows, and both have cheaper paths (dot-netty mTLS; config).
|
||||
- **The gRPC-alleviation answer partially flips at 1.6, but only for Akka-adjacent channels.**
|
||||
ScadaBridge's HTTP notify-and-fetch for deploy configs, and any "too big for the frame cap"
|
||||
workaround, exist purely because of rows 2–3; under Artery those payloads could legitimately ride
|
||||
remoting. The genuinely non-Akka gRPC (mxaccessgw, HistorianGateway, LocalDb's deliberate
|
||||
cluster-independence) stays gRPC regardless — see the mapping table below.
|
||||
- **New costs arrive with the benefits:** Artery is a new wire protocol (flag-day cutover, no
|
||||
rolling mix with 1.5 remoting — tolerable given the stop-pairs-together discipline), .NET 9+
|
||||
only, and it drags in the platform/ops constraints below (libmsquic, UDP firewall rules, Windows
|
||||
Server 2022+, macOS dev friction) — those attach to *any* QUIC adoption, 1.5 or 1.6.
|
||||
|
||||
## 6. Platform and ops constraints ([System.Net.Quic](https://learn.microsoft.com/en-us/dotnet/fundamentals/networking/quic/quic-overview))
|
||||
|
||||
- APIs are stable since .NET 9; the apps are .NET 10 — fine.
|
||||
- **Windows: requires Windows 11 / Server 2022+** (older versions lack the TLS 1.3 crypto APIs).
|
||||
⚠️ Check the `wonder-*` boxes — if any production VM is Server 2019 or older, QUIC remoting is a
|
||||
hard no there.
|
||||
- **Linux:** `libmsquic` must be installed in every image — the `aspnet:10.0`-based images and the
|
||||
docker-dev rig would all need `apt-get install libmsquic` from packages.microsoft.com.
|
||||
- **macOS (dev machine): only partially supported** — Homebrew `libmsquic` + a
|
||||
`DYLD_FALLBACK_LIBRARY_PATH` export, outside Microsoft's test matrix. Offline test suites that
|
||||
spin up real ActorSystems with remoting would grow a fragile native dependency on the Mac.
|
||||
- **Firewalls:** remoting moves from TCP to **UDP** — every rule for 4053/etc. needs a UDP twin,
|
||||
and OT network gear is notoriously unkind to UDP.
|
||||
|
||||
## 7. .NET-native alternatives to MsQuic
|
||||
|
||||
Under the 1.5 SPI every transport degenerates into the same ordered byte pipe (§5), so candidate
|
||||
wire protocols differ on **dependencies, platform reach, and TLS posture** — not on features. By
|
||||
that metric, ranked:
|
||||
|
||||
**1. Managed sockets + `SslStream` + `System.IO.Pipelines` — the best fit if anything gets built.**
|
||||
100% in-box managed .NET: zero native dependencies, runs everywhere the apps run today (macOS test
|
||||
suites, any Windows Server version, stock `aspnet:10.0` images, TCP firewall rules unchanged).
|
||||
TLS 1.3 via `SslStream` is *optional* — can match the current plaintext posture with an mTLS
|
||||
upgrade path, unlike QUIC's hard TLS mandate. Arbitrary frame sizes. Strategically this is a
|
||||
"modern DotNetty replacement": it sheds the effectively unmaintained DotNetty dependency, which is
|
||||
one of 1.6's own goals. JVM precedent that this is respectable: JVM Artery ships TCP and TLS-TCP
|
||||
modes alongside Aeron, and TCP Artery reached ~650k msg/s — most of Artery's gain is the pipeline,
|
||||
not the wire. What it gives up vs QUIC: fast handshake and connection migration — neither of which
|
||||
was a driver here.
|
||||
|
||||
*Throughput expectations, from real data:* this exact idea was prototyped by to11mtm in
|
||||
[PR #4594](https://github.com/akkadotnet/akka.net/pull/4594) (Oct 2020, draft, never merged) — an
|
||||
Akka.Streams/managed-socket TCP transport under the classic pipeline. On the author's i7-8750H:
|
||||
DotNetty baseline ~220–240k msg/s; the managed transport alone ~170–240k (parity at best); adding
|
||||
aggressive zero-copy work + a hand-written protobuf deserializer reached ~240–300k (+10–25%), and
|
||||
the residual bottlenecks identified were all *pipeline*, not wire: serialization, the PDU codec,
|
||||
`EndpointWriter` dispatch, blocking `.Result` in serialization paths. I.e. the transport swap is
|
||||
throughput-neutral; only attacking the layer above moves the number, and even extreme effort caps
|
||||
near +25%. (Reference points: the official
|
||||
[RemotePingPong docs](https://getakka.net/articles/remoting/performance.html) show 82k → 141k msg/s
|
||||
from flush-batching alone on older hardware; modern hardware on 1.5.x lands roughly 300–600k per
|
||||
association; the 1.6 Artery target is 2–3M.) Two mitigating notes: per-*association* ceilings apply
|
||||
per site pair, so hub-and-spoke aggregate scales with site count; and for large frames (the raised
|
||||
`maximum-frame-size` case) a managed transport saturates the wire in bytes/sec — the msg/s ceiling
|
||||
is a small-message phenomenon.
|
||||
|
||||
**2. HTTP/2 — fully managed multiplexing, but the SPI can't use it.** Kestrel (server) +
|
||||
`SocketsHttpHandler` (client), or gRPC bidirectional streaming as the framing, are fully managed
|
||||
h2 with real stream multiplexing and h2c (no cert mandate). But the 1.5 SPI locks the multiplexing
|
||||
away for exactly the §5 reasons, h2 streams still share one TCP connection (packet-level
|
||||
head-of-line blocking remains), and it adds protocol overhead for no captured benefit. Only
|
||||
distinct win is riding existing Kestrel/gRPC infrastructure. Not preferred over raw sockets.
|
||||
|
||||
**3. WebSockets (`System.Net.WebSockets`)** — managed, but a single ordered stream, i.e. TCP with
|
||||
extra framing. Only interesting for exotic proxy/firewall traversal. Skip.
|
||||
|
||||
**No managed QUIC exists.** `System.Net.Quic` *is* the msquic interop — there is no fully managed
|
||||
QUIC implementation in .NET, and nothing else in the box provides independent streams. So "QUIC
|
||||
without libmsquic" is not on the table, on any .NET version.
|
||||
|
||||
Forward-looking note — **resolved 2026-07-22**: upstream did exactly this. The Artery port that
|
||||
began landing on `dev` in July 2026 is **Artery-TCP** (openspec #8313), so the pipeline benefits
|
||||
arrive without libmsquic, and QUIC becomes a later optional substrate — the outcome to11mtm
|
||||
advocated in the #7466 design discussion. This also removes the platform/ops constraints of §6
|
||||
from the 1.6 adoption path (TCP, no native deps, no UDP firewall changes) unless/until a QUIC
|
||||
substrate is chosen deliberately.
|
||||
|
||||
## 8. Would it actually alleviate the gRPC?
|
||||
|
||||
Mostly no — mapping the family's gRPC surfaces:
|
||||
|
||||
| gRPC usage | Could Akka-over-QUIC replace it? |
|
||||
|---|---|
|
||||
| mxaccessgw ↔ clients | **No.** It exists for COM bitness isolation; the worker is .NET 4.8 x86, which can't even load a .NET 9+ QUIC stack, and callers aren't cluster members. |
|
||||
| HistorianGateway API | **No.** A standalone service contract for multiple heterogeneous consumers — service RPC is the right shape. |
|
||||
| LocalDb pair sync | **Technically closest, but deliberately no.** It runs between the same nodes as the Akka cluster, but its whole value is being independent of cluster state (it's what lets a node boot from cache when things are degraded), and it's consumable by non-Akka apps. Coupling it to remoting would weaken it. Its 4 MB cap is a gRPC config choice, not a law. |
|
||||
| ScadaBridge site→central (S&F etc.) | Already Akka where Akka fits; the bulk/artifact paths that avoid remoting avoid it because of the single-channel head-of-line problem — which, per §4, a 1.5 QUIC transport does not fix. |
|
||||
|
||||
The "big messages so we don't need side channels" hope is really an Artery feature (dedicated
|
||||
chunked large-message streams), not a QUIC-transport feature.
|
||||
|
||||
### 8a. The overlooked 1.5-era tool: Akka.Cluster.Chunking (added 2026-07-22)
|
||||
|
||||
One mechanism was missing from the original analysis — Petabridge's
|
||||
**[Akka.Cluster.Chunking](https://github.com/petabridge/Akka.Cluster.Chunking)** (Apache-2.0),
|
||||
which exists precisely to bridge the pre-Artery gap this report is about: sending large messages
|
||||
(5 MB+) over Akka.Remote **without raising `maximum-frame-size` and without head-of-line
|
||||
blocking**. It chunks at the application layer on top of `Akka.Delivery`'s reliable-delivery
|
||||
machinery (ordered, ack'd, resend-on-loss), so every chunk is an ordinary sub-frame-cap remoting
|
||||
message and heartbeats/system messages interleave between chunks — directly defusing the §2
|
||||
guardrail concern with zero transport work.
|
||||
|
||||
Usage is pleasantly transparent: `ChunkingManager.For(system)` then
|
||||
`chunker.DeliverChunked(message, recipient, sender)` (Akka.Hosting: `.AddChunkingManager()`);
|
||||
config via `akka.cluster.chunking.chunk-size` (default 64 kB), `outbound-queue-capacity`
|
||||
(default 20), `request-timeout` (default 5 s). Unlike raw `Akka.Delivery`, no per-flow
|
||||
`ProducerController`/`ConsumerController` pair has to be hand-wired — the manager runs the
|
||||
delivery channels per node pair.
|
||||
|
||||
The honest caveats for our context:
|
||||
|
||||
- **Maturity.** v0.4.0 (Aug 2025), and the README says plainly it is "currently an experiment and
|
||||
is available on an as-is basis." For an OT product that's a real adoption bar; budget for
|
||||
owning/forking it if Petabridge doesn't graduate it.
|
||||
- **Acceptance ≠ delivery.** `DeliverChunked`'s task completes when the message is *accepted* for
|
||||
delivery, not when the far side has it — callers needing end-to-end confirmation still build
|
||||
their own ack on top.
|
||||
- **Requires Akka.Cluster, in-cluster only.** It rides remoting between cluster members. It does
|
||||
not cross the ScadaBridge site↔central boundary — two separate Akka clusters bridged by
|
||||
ClusterClient/gRPC — which is exactly where the family's big payloads actually flow.
|
||||
**Nor does ClusterClient itself chunk** (the tunneled payload + receptionist envelope must fit
|
||||
one frame). It *could* be bolted on — ClusterClient replies carry real IActorRefs, so raw
|
||||
`Akka.Delivery` could then run point-to-point over a direct cross-cluster remoting association —
|
||||
but that reopens exactly the direct-remoting coupling (firewall surface, quarantine interplay)
|
||||
the seed-partitioned boundary exists to avoid; large cross-boundary payloads stay on the gRPC
|
||||
side-channels.
|
||||
- **Correctness tool, not throughput tool** — shallow queues and the reliable-delivery handshake
|
||||
make it slower than plain tells; it removes a ceiling, it doesn't add speed.
|
||||
|
||||
Re-mapping §8 with this in hand changes no conclusion: mxaccessgw/HistorianGateway peers aren't
|
||||
cluster members; LocalDb sync is independent of cluster state *by design*; the ScadaBridge
|
||||
bulk/artifact side channels are cross-cluster, out of reach. Where it **is** the right answer is
|
||||
any *future within-cluster* large-message need (e.g. a big payload between the members of one
|
||||
pair): reach for Akka.Cluster.Chunking (or raw `Akka.Delivery` with
|
||||
`ProducerController.Settings.ChunkLargeMessagesBytes`, the in-box primitive it wraps) instead of
|
||||
raising the frame cap — until 1.6's Artery large-message lanes make both workarounds unnecessary.
|
||||
|
||||
## 9. Options
|
||||
|
||||
**A. Don't build it; tune what we have; track 1.6 (recommended).** Best-practice fit:
|
||||
transport-layer plumbing is exactly the code an OT product wants to own least, and the upstream
|
||||
design explicitly targets the wishlist (large-message channels, quarantine-bypass control channel —
|
||||
the latter directly relevant to the 2-node availability-first failover posture). Concretely: bump
|
||||
1.5.62 → 1.5.70 (eight releases of remoting/streams fixes, including a cluster-wedging
|
||||
consistent-hash-router bug), and if specific pains exist now, prefer Akka.Cluster.Chunking /
|
||||
`Akka.Delivery` chunking (§8a) over raising `maximum-frame-size` for any within-cluster large
|
||||
message, and/or enable the new dot-netty mTLS. Subscribe to #7466; when real alphas ship, the pair-restart
|
||||
discipline makes adoption cheap.
|
||||
|
||||
**B. Build a classic-SPI QUIC transport now.** ~3–5 weeks to production quality per §3. Choose this
|
||||
only if the *specific* wins (TLS 1.3 with modern cert handling, UDP traversal, fast reconnect)
|
||||
matter enough to own a transport — knowing it delivers no throughput/multiplexing gain and gets
|
||||
obsoleted by 1.6.
|
||||
|
||||
**C. Weekend-scale spike, not production.** Implement the minimal `Transport`/`AssociationHandle`
|
||||
pair, run a two-node pair on the rig, benchmark against dot-netty. Genuinely worthwhile as
|
||||
de-risking/learning for the eventual 1.6 migration (certs, libmsquic packaging, UDP firewall rules
|
||||
all get exercised) without owning it in production. A reasonable middle path.
|
||||
|
||||
**Recommendation: A**, with **C** if hands-on evidence is wanted — the honest summary is that for
|
||||
this workload, an MsQuic transport on Akka 1.5 buys security and handshake ergonomics that are
|
||||
mostly available from config today, while the features that would actually reduce the gRPC
|
||||
side-channels only arrive with the Artery pipeline rewrite — now in active development upstream
|
||||
(see the §1 status update) — with Akka.Cluster.Chunking (§8a) as the interim in-cluster answer.
|
||||
|
||||
## Sources
|
||||
|
||||
- [akka.net #7466 — Epic: Akka.Remote Quic-based Transport (Artery)](https://github.com/akkadotnet/akka.net/issues/7466)
|
||||
- [Petabridge — Akka.NET v1.6 Roadmap](https://petabridge.com/blog/akkadotnet-v1.6-roadmap/)
|
||||
- [Akka.NET releases](https://github.com/akkadotnet/akka.net/releases)
|
||||
- [Akka.NET milestones](https://github.com/akkadotnet/akka.net/milestones)
|
||||
- [Akka.NET remoting transports documentation](https://getakka.net/articles/remoting/transports.html)
|
||||
- [Akka.NET PR #7851 — mutual TLS for DotNetty transport](https://github.com/akkadotnet/akka.net/pull/7851)
|
||||
- [QUIC support in .NET (System.Net.Quic)](https://learn.microsoft.com/en-us/dotnet/fundamentals/networking/quic/quic-overview)
|
||||
- [petabridge/Akka.Cluster.Chunking](https://github.com/petabridge/Akka.Cluster.Chunking) — v0.4.0, Apache-2.0, experimental
|
||||
- [Akka.NET reliable delivery (Akka.Delivery)](https://getakka.net/articles/actors/reliable-delivery.html)
|
||||
@@ -0,0 +1,241 @@
|
||||
# Deep dive: Akka Multi-DC cluster support, ported to Akka.NET, applied to our topology
|
||||
|
||||
*Research report, 2026-07-22. Question: Akka (JVM) has multi-DC cluster support that Akka.NET does
|
||||
not. Given our layout (central 2-node cluster + N site 2-node clusters), what would porting it take,
|
||||
and how would it be used in our systems?*
|
||||
|
||||
**TL;DR:** Multi-DC maps onto our topology almost perfectly — "one DC per pair" gives per-site
|
||||
leadership, per-site singletons, per-site oldest-node primaries, and WAN-partition tolerance inside
|
||||
ONE cluster, which is precisely the problem set the OtOpcUa per-cluster mesh program and
|
||||
ScadaBridge's three bespoke bridge transports were built to solve. But the port is core-gossip
|
||||
surgery that has already defeated the Akka.NET maintainers twice (PRs #3284 and #4111, the latter a
|
||||
67-commit functionally-complete draft abandoned in 2019 over multi-node-test instability, now
|
||||
milestoned to the far-future 2.0.0). And two of our hardest cross-cluster problems — the 128 KB
|
||||
frame limit and bulk data movement — are *not* clustering problems and survive multi-DC untouched.
|
||||
Recommendation: don't port; keep the mesh-program direction; treat #4111 as a candidate for an
|
||||
upstream-contribution play only if the mesh program's transport phases (2/5) prove heavier than
|
||||
expected. Details and options below.
|
||||
|
||||
---
|
||||
|
||||
## 1. What Akka multi-DC actually is (JVM; GA since Akka 2.5.6, 2017)
|
||||
|
||||
One correction to the framing: this isn't an upcoming JVM feature — it has been GA on the JVM side
|
||||
since 2017. What's true is that it never made it into Akka.NET. Semantics (from the
|
||||
[Akka 2.6 docs](https://doc.akka.io/docs/akka/2.6/typed/cluster-dc.html) — 2.6.20 is the last
|
||||
Apache-2.0 version, which matters for porting; 2.7+ is BSL-licensed):
|
||||
|
||||
- **DC membership is just a role.** `akka.cluster.multi-data-center.self-data-center = "site-a"`
|
||||
adds role `dc-site-a`. One cluster, every member tagged. Default DC is `"default"` — which is why
|
||||
untagged and tagged nodes can't mix (JVM rolling 2.4→2.5 upgrades hit exactly this, akka/akka
|
||||
#25496).
|
||||
- **One leader per DC.** Each DC's leader performs joining/up/leaving/removal transitions for *its
|
||||
own* members only. Crucially: **members can join and leave while DCs are partitioned from each
|
||||
other** — impossible in a plain cluster, where any unreachable member blocks convergence and
|
||||
therefore blocks all leader actions fleet-wide.
|
||||
- **Two failure detectors.** `akka.cluster.failure-detector` intra-DC (all-to-all within the DC,
|
||||
as today) and `akka.cluster.multi-data-center.failure-detector` cross-DC, run only by the
|
||||
`cross-data-center-connections` (default 5) oldest nodes per DC. Cross-DC unreachability raises
|
||||
`UnreachableDataCenter` / `ReachableDataCenter` events — it is treated as "the WAN link is bad,"
|
||||
explicitly NOT fed into member downing: *"For network partitions between data centers the system
|
||||
should typically not down the unreachable nodes, but instead wait until it heals or a decision is
|
||||
made by a human or external monitoring system."*
|
||||
- **Cluster Singleton is per-DC.** Start the manager on all nodes with 3 DCs defined → 3 live
|
||||
singletons, one per DC. Proxies default to the local DC and take a `dataCenter` parameter to
|
||||
target another DC's singleton. A "global" singleton = start its manager only in one designated DC
|
||||
and accept unavailability when that DC is cut off.
|
||||
- **Cluster Sharding is per-DC** (own coordinator/regions per DC; cross-DC proxies available).
|
||||
Not relevant to us — neither app uses sharding.
|
||||
- **Distributed Pub/Sub is NOT DC-scoped.** The mediator replicates its registry among *all*
|
||||
cluster nodes (optionally restricted by role) — so topics span every DC by default. This is the
|
||||
single most valuable property for OtOpcUa (§4).
|
||||
- Singleton placement and per-DC leadership both key off **oldest member within the DC** — i.e.
|
||||
the JVM feature's per-DC primary rule is literally our `ActiveNodeEvaluator.SelfIsOldestUp` /
|
||||
OtOpcUa Phase-0b `IsDriverPrimary` rule, applied per pair.
|
||||
|
||||
## 2. Where Akka.NET stands on it
|
||||
|
||||
- [Issue #3261](https://github.com/akkadotnet/akka.net/issues/3261) (opened Jan 2018 by
|
||||
Aaronontheweb) is the tracking issue. Current milestone: **2.0.0 — Akka.Typed**, i.e. no
|
||||
committed timeline; it is not in the 1.6 plan (1.6 = Artery/QUIC, .NET Standard drop, new
|
||||
Streams materializer — see the companion report `akka_msquic_transport.md`).
|
||||
- It was **implemented once**: Horusiath's [PR #3284](https://github.com/akkadotnet/akka.net/pull/3284)
|
||||
(2018, milestoned 1.4.0, closed), preserved by Aaron as draft
|
||||
[PR #4111](https://github.com/akkadotnet/akka.net/pull/4111) (Dec 2019, still open). 67 commits
|
||||
touching membership state, gossip, cross-DC heartbeats, singleton/sharding DC-awareness, failure
|
||||
detection, cluster events, and serialization, plus multi-node specs for split-brain and
|
||||
sunny-weather scenarios. It reportedly passed on .NET Framework but **died on Multi-Node TestKit
|
||||
instability on .NET Core** ("151 tests failing" at one point; deadlocks in MembershipState lazy
|
||||
init, convergence races, CoordinatedShutdown exit-hook problems). It was dropped from 1.4.0 for
|
||||
review/verification bandwidth and never revived. Its base is **mid-2018 `dev`** — six years and
|
||||
two major version lines behind today's 1.5.70.
|
||||
|
||||
So "porting the support" concretely means: re-porting from the JVM 2.6.20 sources (Apache 2.0 —
|
||||
do NOT copy from 2.7+/BSL) with #4111 as a .NET-idiom reference, onto Akka.NET 1.5.x, and carrying
|
||||
it as a fork until upstream lands its own in 2.0.0.
|
||||
|
||||
## 3. The mapping onto our topology
|
||||
|
||||
The fit is genuinely striking. Model: **every 2-node pair is a DC** in one spanning cluster.
|
||||
|
||||
| DC | Members | Today |
|
||||
|---|---|---|
|
||||
| `dc-central` | central-a, central-b | ScadaBridge central cluster / OtOpcUa admin pair |
|
||||
| `dc-site-a` | site-a-1, site-a-2 | separate site cluster (SB) / same mesh (OtOpcUa) |
|
||||
| `dc-site-b` … | … | … |
|
||||
|
||||
What each multi-DC property buys us:
|
||||
|
||||
- **Per-DC oldest = pair primary, by construction.** The exact rule both products converged on
|
||||
(ScadaBridge `ActiveNodeEvaluator.SelfIsOldestUp`; OtOpcUa Phase 0b) becomes the framework's own
|
||||
scoping — the mesh design's core motivation ("every Primary-gated decision asks a cluster-wide
|
||||
election about a pair-local resource") dissolves.
|
||||
- **Per-DC singletons.** OtOpcUa's `admin-operations` singleton stays effectively central by
|
||||
starting its manager only on `dc-central` nodes; per-pair singletons (e.g. a per-site drain
|
||||
coordinator) come free.
|
||||
- **WAN-safe failure handling.** Intra-pair failure detection + our `auto-down`(15s)
|
||||
availability-over-partition-safety posture stays *within the pair*, where we want it. A WAN blip
|
||||
to a site raises `UnreachableDataCenter` — no downing, no quarantine churn, and (the big one)
|
||||
**membership operations at other sites keep working during the outage**. Today's OtOpcUa single
|
||||
mesh has neither property, which is a large part of why the mesh program exists.
|
||||
- **Pub/sub spans DCs.** OtOpcUa's nine DPS topics + deploy notify channel keep working unchanged
|
||||
across the whole fleet. The Secrets Akka replicator (DPS-based) would also span everything —
|
||||
today it explicitly cannot cross ScadaBridge's separate clusters (why secrets use SQL-hub mode
|
||||
there).
|
||||
- **Direct actor messaging replaces ClusterClient.** In one spanning cluster, receptionist
|
||||
contact-point management, per-site client caching, and DB-driven address refresh
|
||||
(`CentralCommunicationActor`'s `_siteClients`, the 60 s `Site`-row polling) reduce to ordinary
|
||||
cluster-aware messaging.
|
||||
|
||||
### What multi-DC does NOT fix (important)
|
||||
|
||||
- **The 128 KB frame limit and serialization posture are unchanged.** ScadaBridge's HTTP
|
||||
notify-and-fetch (deploy configs) and gRPC streaming (`:8083`, high-volume attribute/alarm
|
||||
events) exist because Akka messaging is the wrong channel for bulk payloads — that remains true
|
||||
in a multi-DC cluster. Those channels stay. (Same conclusion as the MsQuic report: bulk channels
|
||||
aren't clustering problems.)
|
||||
- **Store-and-forward stays.** At-least-once site→central delivery with a durable buffer
|
||||
(`StoreAndForwardService` + replicated `sf_messages`) is an application guarantee; multi-DC
|
||||
changes the pipe, not the need for the buffer.
|
||||
- **Security got harder, not easier.** Today an attacker on the WAN can reach ClusterClient/gRPC
|
||||
seams; in a spanning cluster they can attempt to *join the cluster*. Akka remoting is
|
||||
unauthenticated h2c-equivalent in both products (accepted risk, mesh design §6.3) — a spanning
|
||||
cluster raises the value of revisiting that.
|
||||
- **Persistence single-writer caveats** (the JVM docs' sharding-across-DCs warning) don't bite us —
|
||||
no sharding, and pair-local stores are LocalDb-replicated with HLC/LWW already.
|
||||
|
||||
## 4. Per-system assessment
|
||||
|
||||
### OtOpcUa — the big beneficiary, but the timing is awkward
|
||||
|
||||
OtOpcUa *today* is one mesh spanning everything — i.e. it is already shaped like a multi-DC
|
||||
cluster, minus the DC semantics. The per-cluster mesh program
|
||||
(`OtOpcUa/docs/plans/2026-07-22-per-cluster-mesh-program.md`, Phases 1–7) exists to fix exactly the
|
||||
things DC-tagging would fix in place:
|
||||
|
||||
| Mesh-program phase | Under multi-DC |
|
||||
|---|---|
|
||||
| 0a auto-down / 0b oldest-primary | Already done; both carry over unchanged (intra-DC) |
|
||||
| 1 `ClusterNode` address columns, DB-sourced ack set | **Mostly unnecessary** — membership stays visible fleet-wide |
|
||||
| 2 ClusterClient comm actors + receptionists | **Unnecessary** — one cluster, direct messaging |
|
||||
| 3 config fetch-and-cache from central | **Still required** — driven by "sites have no SQL Server," not by clustering |
|
||||
| 4 cut driver-side ConfigDb | **Still required** — same reason |
|
||||
| 5 gRPC migration of 7 observability topics | **Unnecessary** — DPS keeps working |
|
||||
| 6 mesh partition into pairs | Replaced by DC tagging + seed layout (much smaller) |
|
||||
| 7 failover drills / live gates | Still required, plus new cross-DC drills |
|
||||
|
||||
So multi-DC would delete the two transport-migration phases (2 and 5 — the heaviest pure-plumbing
|
||||
work) and shrink 1 and 6, while leaving the genuinely deployment-driven phases (3, 4) intact. That
|
||||
is a real prize. It is also exactly why the timing is awkward: the program was just sequenced on
|
||||
the ScadaBridge-proven separate-cluster shape, and switching horses means betting those phases on a
|
||||
forked cluster core instead of on patterns already running in production-adjacent code.
|
||||
|
||||
### ScadaBridge — already solved this; migration value is low
|
||||
|
||||
ScadaBridge built the separate-cluster bridges years-of-work deep and they are live: per-site
|
||||
`ClusterClient`s with DB-driven contact refresh, per-node receptionist registration, gRPC
|
||||
server-on-site/client-on-central streaming, token-gated HTTP deploy fetch, S&F with active-node
|
||||
gating. Two details actually make it multi-DC-*ready* — every cluster already shares the one
|
||||
ActorSystem name `"scadabridge"` (separation is purely seed-node partitioning), and the roles
|
||||
already include `site-{SiteId}` — but what multi-DC would replace is only the ClusterClient control
|
||||
channel and the address-book machinery. The gRPC and HTTP channels stay (frame limit), S&F stays,
|
||||
and the CLI's ClusterClient entry point would need rework. Replacing a proven, tested seam with a
|
||||
forked gossip layer for a cleaner control channel is a poor trade on its own.
|
||||
|
||||
## 5. What the port itself takes
|
||||
|
||||
Scope (from the JVM 2.6.20 implementation and the #4111 diff):
|
||||
|
||||
1. **Membership/gossip core** — `Member` gains a `dataCenter` (derived from the `dc-` role, so the
|
||||
gossip **wire format is nearly unchanged** — the DC rides in roles); `MembershipState`/gossip
|
||||
convergence, leader election, and reachability calculations all become DC-filtered (convergence
|
||||
ignores cross-DC observations; per-DC leader actions permitted during inter-DC splits). This is
|
||||
the delicate 20% that is 80% of the risk.
|
||||
2. **Heartbeating** — keep today's all-to-all within a DC; add the cross-DC heartbeater run by the
|
||||
N oldest per DC with its own failure-detector config; new `UnreachableDataCenter` /
|
||||
`ReachableDataCenter` events.
|
||||
3. **Downing** — `AutoDowning`/SBR must become DC-scoped (JVM SBR is; our `auto-down` posture then
|
||||
applies per pair, which is exactly the HA stance we want). A non-DC-aware downing provider in a
|
||||
multi-DC cluster is actively dangerous (downs the far side on WAN blips).
|
||||
4. **Cluster tools** — Singleton manager/proxy DC-awareness (small — it is role-filtering);
|
||||
sharding proxies (skippable for us); DPS needs nothing.
|
||||
5. **Serialization** — additive protobuf changes for the new heartbeat/gossip fields; whole-fleet
|
||||
cutover required (mixed DC-aware/unaware nodes don't interoperate — the JVM had the same
|
||||
constraint). Our stop/start-pairs-together discipline makes this tolerable, but it is a
|
||||
flag-day, not rolling.
|
||||
6. **Verification — the real cost.** This is what killed it upstream twice: MNTK specs for
|
||||
split-brain, join-during-partition, cross-DC unreachability, singleton-per-DC handover. Our
|
||||
substitute would be rig-based live gates (docker-dev with real WAN-partition injection via
|
||||
`tc`/iptables), which we're good at — but the failure modes are convergence races that soak
|
||||
tests catch and unit tests don't.
|
||||
|
||||
**Effort honestly:** the code port is plausibly 4–8 weeks against the Apache-2.0 JVM reference with
|
||||
#4111 as a map. The validation to *trust* it under an OT platform's availability requirements is
|
||||
the same again or more, and then it's a **permanent fork of Akka.Cluster** — every 1.5.x bump
|
||||
(there have been 8 since our pin) becomes a merge with the most intricate code in the framework,
|
||||
with no upstream support for the modified layer, until upstream's own version lands in 2.0.0 and
|
||||
strands the fork entirely. Precedent matters here: the framework's own authors, with the MNTK
|
||||
infrastructure we lack, shelved it twice.
|
||||
|
||||
## 6. Options
|
||||
|
||||
**A. Don't port. Continue the current direction (recommended).** ScadaBridge keeps its proven
|
||||
bridges; OtOpcUa executes the mesh program as sequenced (selfform prerequisite → Phases 1–7). Every
|
||||
pattern in that program is already running in ScadaBridge production code; risk lives in
|
||||
application code we own rather than framework internals we'd fork. Cost: OtOpcUa still pays for
|
||||
Phases 2 and 5 (ClusterClient + gRPC plumbing), and cross-cluster pub/sub stays unavailable
|
||||
(secrets replication keeps SQL-hub mode on ScadaBridge).
|
||||
|
||||
**B. Port multi-DC (fork Akka.Cluster).** Scope per §5, ~2–4 months including validation, plus the
|
||||
standing fork tax. Payoff concentrated in OtOpcUa (skip Phases 2/5, keep DPS, keep one mesh with
|
||||
correct per-pair scoping). Only rational as an **upstream-contribution play** — rebase/re-port
|
||||
against current `dev`, revive #4111 with Petabridge (possibly as sponsored work), so the fork has
|
||||
an exit. As a private fork: not worth it for two products with a working alternative.
|
||||
|
||||
**C. Multi-DC-lite on stock Akka.NET** (keep OtOpcUa's single mesh, emulate the scoping
|
||||
app-side): role-scoped singletons and role-scoped DPS groups per pair (supported today), the
|
||||
already-done oldest-Up-per-role primary rule, plus a **custom `IDowningProvider`** that only downs
|
||||
same-site peers (pluggable, no fork). What cannot be emulated: convergence — one unreachable node
|
||||
anywhere still blocks joins/leaves fleet-wide, and cross-site heartbeating stays all-to-all over
|
||||
the WAN. Tolerable for a small, static fleet that rarely adds nodes mid-outage; it is a
|
||||
known-sharp-edges compromise, not a fix. Worth considering only if the mesh program stalls.
|
||||
|
||||
**Recommendation: A.** The mesh program should keep going — it's sequenced, its prerequisite plan
|
||||
exists, and its shape is production-proven next door. Revisit B strictly as "fund/contribute the
|
||||
upstream port" if Phases 2/5 blow up in practice, and note in that case the decision point comes
|
||||
*before* Phase 2 starts, because B makes exactly those phases unnecessary.
|
||||
|
||||
## Sources
|
||||
|
||||
- [Akka 2.6 Multi-DC Cluster docs](https://doc.akka.io/docs/akka/2.6/typed/cluster-dc.html) (last Apache-2.0 line)
|
||||
- [Akka classic Distributed Pub/Sub docs](https://doc.akka.io/docs/akka/2.6/distributed-pub-sub.html) (mediator scope: all nodes / role-filtered)
|
||||
- [akka.net #3261 — Multi-Data Center Support in Akka.Cluster](https://github.com/akkadotnet/akka.net/issues/3261) (milestone 2.0.0)
|
||||
- [akka.net PR #3284 — original multi-DC implementation (Horusiath)](https://github.com/akkadotnet/akka.net/pull/3284)
|
||||
- [akka.net PR #4111 — preserved 67-commit draft (Aaronontheweb)](https://github.com/akkadotnet/akka.net/pull/4111)
|
||||
- [akka/akka #25496 — 2.4→2.5 rolling-upgrade DC requirement failure](https://github.com/akka/akka/issues/25496)
|
||||
- ScadaBridge cross-cluster seam map: `CentralCommunicationActor.cs` / `SiteCommunicationActor.cs` (ClusterClient),
|
||||
`SiteStreamGrpcClient/Server.cs` + `sitestream.proto` (gRPC :8083), `HttpDeploymentConfigFetcher.cs` +
|
||||
`DeploymentConfigEndpoints.cs` (token-gated HTTP), `StoreAndForwardService.cs`, `ActiveNodeEvaluator.cs`;
|
||||
ActorSystem name shared at `AkkaHostedService.cs:191`
|
||||
- OtOpcUa mesh design + program: `OtOpcUa/docs/plans/2026-07-21-per-cluster-mesh-design.md`,
|
||||
`2026-07-22-per-cluster-mesh-program.md`
|
||||
File diff suppressed because one or more lines are too long
@@ -0,0 +1,153 @@
|
||||
# ZB.MOM.WW.Overview — family overview dashboard, v1 design
|
||||
|
||||
**Date:** 2026-07-22
|
||||
**Decision context:** `overview_dashboard_options.md` (same date) — Option A chosen: a custom in-house pane, matching the look and feel of the other scadaproj apps. Aspire standalone rejected (resource/status UI disabled outside an AppHost; no persistence; dev-tool positioning).
|
||||
**Implementation plan:** `2026-07-22-overview-dashboard-impl-plan.md` (same directory) — phased, code-verified tasks incl. the sister-project changes.
|
||||
|
||||
## 1. What it is
|
||||
|
||||
A small Blazor Server app, hosted in scadaproj like the other shared components, that shows **every registered application instance on one page**: Up/Degraded/Down/Unreachable, Active/Standby where applicable, the **current Akka cluster leader** for each cluster group, per-check detail, and a deep link to each instance's own management UI. The page always renders instantly from a cached snapshot — a background poller on a **configurable timer** does all the HTTP work, and a **configurable staleness timeout** marks any data the poller hasn't refreshed as out-of-date/offline. **No login: the dashboard is anonymous and strictly read-only** (it only GETs endpoints that are already anonymous, and has no mutating actions). Configured entirely from an appsettings registry. **Every instance reports through the same channel:** the anonymous `/health/ready` + `/health/active` endpoints of `MapZbHealth`, parsing the canonical `ZbHealthWriter` JSON. Three of the four apps already expose these everywhere; the one gap — ScadaBridge *site* nodes — is closed by a small prerequisite change in ScadaBridge (§4) rather than worked around, so the dashboard has exactly one probe model.
|
||||
|
||||
Non-goals for v1: history/uptime graphs, alerting/webhooks, telemetry viewing (OTLP/traces), and any push-based ingestion. If ops-grade monitoring is ever needed, that's the Grafana/Prometheus phase in the options doc — this registry ports straight to a scrape config.
|
||||
|
||||
## 2. Project shape
|
||||
|
||||
```
|
||||
ZB.MOM.WW.Overview/ # plain directory in scadaproj — NOT a nested git repo
|
||||
ZB.MOM.WW.Overview.slnx
|
||||
src/ZB.MOM.WW.Overview/ # single ASP.NET Core + Blazor Server project
|
||||
Program.cs
|
||||
Registry/ (options + validator)
|
||||
Polling/ (poller BackgroundService, health client, snapshot store)
|
||||
Components/ (App.razor, MainLayout → ThemeShell, Pages/Overview.razor, widgets)
|
||||
tests/ZB.MOM.WW.Overview.Tests/
|
||||
docker/ # HistorianGateway-pattern runtime-only image
|
||||
```
|
||||
|
||||
- .NET 10, global InteractiveServer render mode (same router style as ScadaBridge/HG/mxgw — no per-page render-mode traps).
|
||||
- Package references (Gitea feed): `ZB.MOM.WW.Theme` 0.3.1, `ZB.MOM.WW.Health` (0.2.0 — see §4a), `ZB.MOM.WW.Telemetry(.Serilog)` 0.1.0, `ZB.MOM.WW.Configuration` 0.1.0. **No Auth, no Audit, no Secrets**: the dashboard is anonymous read-only by requirement — there is no login, no mutating action to audit, and nothing secret in the registry. (If a login is ever wanted later, `AddZbLdapAuth` + `LoginCard` drop in the family way.)
|
||||
- It is an **app**, not a published library — nothing gets packed to the feed.
|
||||
|
||||
## 3. Registry (appsettings)
|
||||
|
||||
```jsonc
|
||||
"Overview": {
|
||||
"PollIntervalSeconds": 10, // global poll cadence (configurable timer)
|
||||
"TimeoutSeconds": 3, // per-request timeout
|
||||
"StaleAfterSeconds": 45, // snapshot older than this ⇒ instance rendered Stale (out of date/offline)
|
||||
"Applications": [
|
||||
{
|
||||
"Name": "OtOpcUa",
|
||||
"ManagementLabel": "AdminUI",
|
||||
"Instances": [
|
||||
{ "Name": "central-1", "Group": "central",
|
||||
"BaseUrl": "http://otopcua-central-1:9000",
|
||||
"ManagementUrl": "http://otopcua-central-1:9000/",
|
||||
"HasActiveRole": true },
|
||||
{ "Name": "site-a-1", "Group": "site-a", "BaseUrl": "http://...", "HasActiveRole": true }
|
||||
]
|
||||
},
|
||||
{
|
||||
"Name": "ScadaBridge",
|
||||
"Instances": [
|
||||
{ "Name": "central-a", "Group": "central", "BaseUrl": "http://sb-central-a:5000",
|
||||
"ManagementUrl": "http://sb-central-a:5000/", "HasActiveRole": true },
|
||||
{ "Name": "site-a-1", "Group": "site-a",
|
||||
"BaseUrl": "http://sb-site-a-1:8084", "HasActiveRole": true } // health on the HTTP/1.1 listener — see §4
|
||||
]
|
||||
},
|
||||
{
|
||||
"Name": "MxAccessGateway",
|
||||
"ManagementLabel": "Dashboard",
|
||||
"Instances": [
|
||||
{ "Name": "windev", "BaseUrl": "http://windev:5130",
|
||||
"ManagementUrl": "http://windev:5130/" } // single-instance; no active/standby role
|
||||
]
|
||||
},
|
||||
{
|
||||
"Name": "HistorianGateway",
|
||||
"ManagementLabel": "Dashboard",
|
||||
"Instances": [
|
||||
{ "Name": "wonder-app-vd03", "BaseUrl": "https://wonder-app-vd03:5222",
|
||||
"ManagementUrl": "https://wonder-app-vd03:5222/" } // prod single TLS ALPN port; dev :5220
|
||||
]
|
||||
}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
All four family apps are first-class registry citizens; `HasActiveRole` is simply omitted for the two single-instance gateways (mxgw, HistorianGateway), which have no active/standby concept — their cards show status without the Active badge.
|
||||
|
||||
- Every instance is probed the same way: GET `{BaseUrl}/health/ready`, plus `{BaseUrl}/health/active` when `HasActiveRole`. There is deliberately no per-instance probe-mode switch — one probe model, one status derivation.
|
||||
- `PollIntervalSeconds`/`TimeoutSeconds`/`StaleAfterSeconds` are overridable per application and per instance (instance wins) for slow links or chatty pairs; `StaleAfterSeconds` must exceed `PollIntervalSeconds` (validated).
|
||||
- Bound + validated with `AddValidatedOptions<OverviewOptions, OverviewOptionsValidator>` (`ZB.MOM.WW.Configuration`): non-empty registry, absolute http(s) URLs, unique app/instance names, positive intervals, stale > poll. Malformed registry fails boot via `ConfigPreflight`, not the first poll.
|
||||
|
||||
## 4. Prerequisite target-app changes (health from ALL apps)
|
||||
|
||||
The dashboard's contract is "every registered instance serves `MapZbHealth`". Current reality and the deltas:
|
||||
|
||||
| App / node kind | Today | Delta needed |
|
||||
|---|---|---|
|
||||
| OtOpcUa admin + driver nodes | `MapZbHealth` mapped unconditionally | None in code. **Deployment note:** a driver-only Windows-service node with no explicit URLs falls back to Kestrel's `localhost:5000` — every registered node must be given a reachable binding (`ASPNETCORE_URLS`/`HTTP_PORTS`), which the docker rig already does. |
|
||||
| ScadaBridge **Central** | `MapZbHealth` (ready: database/akka-cluster/required-singletons; active: active-node) | None. |
|
||||
| ScadaBridge **Site** nodes | **No health endpoints** — only the gRPC `:8083` (HTTP/2-only) + `/metrics` on the `:8084` HTTP/1.1 listener | **Small ScadaBridge PR (the one prerequisite):** map `MapZbHealth` on the `:8084` HTTP/1.1 listener alongside `/metrics`. Ready checks: `akka-cluster` (site pair) + site SQLite store (+ `localdb-replication` where enabled); active check: the site pair's oldest-Up active-node rule via `ActiveNodeHealthCheck` — the Host already references all three `ZB.MOM.WW.Health` packages, so this is pure wiring, no new dependencies. Gives site rows full Up/Degraded detail **and** Active/Standby, same as everything else. |
|
||||
| MxAccessGateway | `MapZbHealth` (ready: auth-store) | None. |
|
||||
| HistorianGateway | `MapZbHealth` (ready: historian-connection, galaxy-sql, conditional runtime/store-forward/pool checks) | None. |
|
||||
|
||||
Sequencing: the ScadaBridge PR can land any time before the dashboard's acceptance run; the dashboard itself needs no knowledge of it beyond registry entries.
|
||||
|
||||
### 4a. Shared-lib prerequisite: `ZB.MOM.WW.Health` 0.2.0 — cluster-leader data
|
||||
|
||||
Requirement: the dashboard must show the **current leader** of each Akka cluster. The canonical health JSON today carries only `status`/`description`/`durationMs` per entry — no structured data — so this needs a small **additive** change to the shared lib (which is exactly where the family wants such things to live):
|
||||
|
||||
- **Core (`ZbHealthWriter`):** emit an optional `"data": { ... }` object per entry, sourced from `HealthReportEntry.Data`, **only when non-empty** — existing payloads stay byte-identical, so this is non-breaking for every current consumer.
|
||||
- **`ZB.MOM.WW.Health.Akka` (`AkkaClusterHealthCheck`):** populate the check's data dictionary from local cluster state: `leader` (address), `selfAddress`, `selfRoles`, `memberCount`, `unreachableCount` (+ `roleLeader:<role>` where the app cares). Each node reports *its own view*; the dashboard reads the leader from any Up node of the group and flags disagreement (a visible split-brain tell — a nice free diagnostic for the 2-node pairs).
|
||||
- **Version/rollout:** bump lib to `0.2.0`, publish the 3 packages to the Gitea feed. **Required bumps:** OtOpcUa + ScadaBridge (the Akka apps — leader data appears with the package bump alone, no app code changes, since both register the shared `AkkaClusterHealthCheck`). **Recommended alignment bumps:** mxgw + HistorianGateway (no behavior change for them; keeps the family version matrix aligned).
|
||||
|
||||
## 5. Polling and status model
|
||||
|
||||
One `OverviewPollerService : BackgroundService`:
|
||||
|
||||
- Every cadence tick, polls **all instances in parallel** (`Task.WhenAll`, per-instance `CancellationTokenSource` timeout; named `HttpClient` via `IHttpClientFactory`, no retries — the next tick is the retry).
|
||||
- Parses the canonical shape `{ status, totalDurationMs, entries: { name: { status, description, durationMs } } }`. Parsing lives in a small `ZbHealthReportClient` inside this app for v1; promoting it into a `ZB.MOM.WW.Health.Client` package is a deliberate later step, not a v1 dependency (recommendation: promote only when a second consumer appears).
|
||||
- Derived per-instance state:
|
||||
- **Up** — ready 200 + `"Healthy"`; **Degraded** — 200 + `"Degraded"`; **Down** — 503 / `"Unhealthy"`; **Unreachable** — timeout, refused, DNS, non-JSON (kept distinct from Down: it usually means network/VPN/registry-typo, not the app).
|
||||
- **Active/Standby** — `/health/active` 200 → Active, 503 → Standby, error → Unknown (only when `HasActiveRole`).
|
||||
- Plus: last poll UTC, latency ms, per-check entries, consecutive-failure count (state flips to Down/Unreachable only after **2** consecutive failures to avoid single-blip flapping; recovery is immediate).
|
||||
- **Cache-first rendering:** results land in an in-memory `OverviewSnapshotStore` (single immutable snapshot swapped atomically); an event notifies circuits to re-render. Page loads NEVER trigger a poll — they read the cached snapshot and render instantly; the poller timer is the only thing that does HTTP. No persistence — a restart repolls the world within one cadence.
|
||||
- **Staleness:** every instance snapshot carries `LastUpdatedUtc`. When the UI renders (or a periodic sweep runs), any instance whose snapshot age exceeds its effective `StaleAfterSeconds` is shown as **Stale (out of date/offline)** regardless of its last known status — this catches a stuck poller, a paused refresh, or a machine-slept dashboard, not just failed probes.
|
||||
- **Leader extraction:** for each `Group` (one Akka cluster = one group, e.g. `OtOpcUa/central`, `ScadaBridge/site-a`), the aggregator reads the `akka-cluster` entry's `data.leader` from each Up member and surfaces the group's leader (resolving the leader address back to a registry instance name where possible). If Up members disagree on the leader, the group is flagged (split-brain indicator).
|
||||
- StatusPill mapping: Up→`Ok`, Degraded→`Warn`, Down→`Bad`, Unreachable→`Bad` (distinct label), Stale→`Idle`, Active/Leader badges→`Info`.
|
||||
|
||||
## 6. UI
|
||||
|
||||
> **Mockup:** [`docs/mockups/overview-dashboard-mockup.html`](../mockups/overview-dashboard-mockup.html) (static, self-contained; real Theme tokens + embedded IBM Plex) — the visual reference for `InstanceCard`, group headers, leader/split-brain chips, and every status state incl. Stale. Open it in a browser; the implementation should match it.
|
||||
|
||||
Single page (`/`), `ThemeShell` chassis with its own product name/accent so it reads as a sibling of the other four apps:
|
||||
|
||||
- One section per application (name + rollup pill = worst instance status), instances as a grid of cards (`TechCard` + `StatusPill` composition — a new `InstanceCard` component, since Theme has no prebuilt tile widget). Cluster groups render a group header with the **Leader: <instance>** chip (and the split-brain warning when member views disagree).
|
||||
- Card: instance name + group, status pill, Active badge, **Leader badge** on the leader node's card, latency, relative last-seen (turns into an explicit "stale since …" once past `StaleAfterSeconds`); expandable check list (`entries` with per-check pill + description); a `TechButton` "Open <ManagementLabel>" → `ManagementUrl` (new tab).
|
||||
- A header row: total counts (n Up / n Degraded / n Down / n Unreachable / n Stale), last-refresh clock, pause/resume auto-refresh (paused long enough, cards go Stale by design).
|
||||
- **No login.** The dashboard is anonymous and read-only end to end; there is nothing to authorize and nothing it can change. It still matches the sister apps' look exactly (ThemeShell + tokens + IBM Plex), just without the login gate.
|
||||
|
||||
## 7. Self-observability and hosting
|
||||
|
||||
- The dashboard eats the family dog food: `MapZbHealth` (ready check: "registry loaded" + "last poll cycle completed"), `MapZbMetrics`, `AddZbTelemetry`/`AddZbSerilog`; a `zb_overview_instance_status` gauge (per instance, labeled) so even v1 status is scrapeable — and becomes the Grafana bridge later.
|
||||
- Ports: dev `http://localhost:5320` (clear of every family port in use: 5000/5001, 5120/7121, 5220–5222, 8081–8085, 9000, 9200, 4053, 4840); production single TLS ALPN endpoint like HistorianGateway, warn-only if terminated upstream.
|
||||
- `docker/` mirrors the HistorianGateway pattern: publish framework-dependent on the host (authed Gitea feed), COPY into `aspnet:10.0`, gitignored env_file. Natural first deployment: the docker-dev rig host, registry pointed at the existing OtOpcUa + ScadaBridge rig nodes — that is also the acceptance test.
|
||||
|
||||
## 8. Open decisions (settled recommendations)
|
||||
|
||||
1. **ScadaBridge site rows** — resolved (2026-07-22, user direction): site nodes gain `MapZbHealth` as the one prerequisite PR (§4), so every row is a full health row. *Chosen over* a metrics-liveness fallback (two probe models) and over reading Central's site aggregator (couples the dashboard to ScadaBridge internals and its transport).
|
||||
2. **Health-client ownership** — in-app for v1; promote to `ZB.MOM.WW.Health.Client` on second consumer. *(Best practice: contract-owning lib should host the client, but a package bump + publish for one consumer is ceremony without benefit.)*
|
||||
3. **History/alerting** — explicitly out of v1; the `zb_overview_instance_status` gauge is the forward hook (Prometheus can scrape the dashboard alone and get fleet status history for free).
|
||||
4. **Leader transport** — resolved (2026-07-22, user requirement): via a `data` object in the canonical health JSON (`ZB.MOM.WW.Health` 0.2.0, §4a), populated by the shared `AkkaClusterHealthCheck`. *Chosen over* description-string parsing (fragile) and a separate leader endpoint (second contract for one field).
|
||||
5. **Auth** — resolved (2026-07-22, user requirement): none. Anonymous, read-only.
|
||||
|
||||
## 9. Acceptance (v1 definition of done)
|
||||
|
||||
- The §4 ScadaBridge site-node health PR is merged and site rows render with full check detail + Active/Standby (no special-cased rows anywhere).
|
||||
- Registry pointed at the live docker-dev rigs (OtOpcUa 6-node + ScadaBridge local cluster) renders every instance correctly, including: each Active/Standby pair (central **and** site pairs) showing exactly one Active; **each cluster group showing its current leader, and the leader chip moving when the leader node is stopped**; killing a node flips its card to Unreachable within ~2 cadences and back on restart; a Degraded check (e.g. stop a dependency) shows Warn with the failing entry visible; management links open the right app UIs; the page is reachable with no login.
|
||||
- **Cache/staleness proven live:** first page load renders instantly from cache (no poll on request); with auto-refresh paused (or the poller stopped) past `StaleAfterSeconds`, every card flips to Stale, and resumes cleanly.
|
||||
- Boot fails with a clear `ConfigPreflight` message on a malformed registry.
|
||||
- `dotnet test` green: options validator, status derivation (incl. flap damping, staleness marking, leader extraction + disagreement flag, JSON parsing golden tests against captured real payloads from all four apps — with and without `data`), snapshot store, and a bUnit render test of `InstanceCard`.
|
||||
@@ -0,0 +1,171 @@
|
||||
# ZB.MOM.WW.Overview — implementation plan
|
||||
|
||||
**Date:** 2026-07-22
|
||||
**Design:** `2026-07-22-overview-dashboard-design.md` (same directory). Read it first; this plan assumes its decisions (custom app, family look/feel, anonymous read-only, cache-first polling, staleness timeout, Akka leader display).
|
||||
**Repos touched:** `scadaproj/ZB.MOM.WW.Health` (shared lib 0.2.0), `~/Desktop/ScadaBridge` (site-node health PR), `~/Desktop/OtOpcUa` (Health bump + check verification), optional alignment bumps in `~/Desktop/MxAccessGateway` + `~/Desktop/HistorianGateway`, and the new `scadaproj/ZB.MOM.WW.Overview/`.
|
||||
**Sequencing:** Phase 0 → Phases 1 + 2 (parallel, independent repos) → Phase 3 (app; can start in parallel once 0.2.0 is on the feed) → Phase 4 (live acceptance). Phases 1/2 only gate Phase 4, not Phase 3 development.
|
||||
|
||||
All facts below were code-verified 2026-07-22 (file:line cites are from that pass); re-check a cite if the target repo has moved since.
|
||||
|
||||
**How to run this in a fresh session:** read the design doc first, then execute phases in order (0 → 1‖2 → 3 → 4), one phase per sitting with its DoD met before moving on. Work in each repo on a feature branch (`feat/health-0.2.0-cluster-data` in scadaproj, `feat/site-node-health` in ScadaBridge, `feat/health-0.2.0-bump` in OtOpcUa); commit scadaproj work (lib + new app + these docs) on `main` per family habit only when asked. Nothing in this plan requires user input except merges/pushes and the Phase 4 rig runs; if a cited line has drifted, re-locate by the quoted identifier, not the line number.
|
||||
|
||||
---
|
||||
|
||||
## Phase 0 — `ZB.MOM.WW.Health` 0.2.0: entry `data` + Akka cluster-view
|
||||
|
||||
Repo: `scadaproj/ZB.MOM.WW.Health/` (plain directory — do NOT `git init`; commits happen in scadaproj). Akka.Cluster pin 1.5.62 (`Directory.Packages.props:9`); version is centralized at `Directory.Build.props:8` (`<Version>0.1.0</Version>`).
|
||||
|
||||
### Task 0.1 — writer: optional per-entry `data` object
|
||||
`src/ZB.MOM.WW.Health/ZbHealthWriter.cs`:
|
||||
- Add to the private `HealthEntryDto` (lines ~76-81):
|
||||
`[JsonIgnore(Condition = JsonIgnoreCondition.WhenWritingNull)] public IReadOnlyDictionary<string, object>? Data { get; init; }`
|
||||
- In the projection (line ~62): `Data = e.Value.Data.Count > 0 ? e.Value.Data : null` (`HealthReportEntry.Data` is never null; empty when unset).
|
||||
- **Invariant: old payloads stay byte-identical.** The per-property `JsonIgnore` is essential — the global serializer options deliberately do NOT ignore nulls (the null-`description` behavior is pinned by `ResponseWriterTests.cs:69-85`); do not change `SerializerOptions`.
|
||||
|
||||
Tests (`tests/ZB.MOM.WW.Health.Tests/ResponseWriterTests.cs`): extend `StubHealthCheck` (lines 21-31) to accept an optional data dictionary; add (a) non-empty data → `"data": {...}` emitted with camelCase envelope keys but **verbatim data keys**, (b) empty data → no `data` key at all (assert on raw JSON string), (c) existing null-description test still green.
|
||||
|
||||
### Task 0.2 — `AkkaClusterHealthCheck`: populate cluster-view data
|
||||
`src/ZB.MOM.WW.Health.Akka/AkkaClusterHealthCheck.cs` (currently reads only `SelfMember.Status`, returns description-only — lines 38-70):
|
||||
- Add an internal static helper `BuildClusterData(Cluster cluster)` returning `IReadOnlyDictionary<string, object>` with JSON-friendly values only (string/int/string[]):
|
||||
- `leader` — `cluster.State.Leader?.ToString()` (omit the key when null: pre-join/unknown),
|
||||
- `selfAddress` — `cluster.SelfAddress.ToString()`,
|
||||
- `selfRoles` — `cluster.SelfMember.Roles.OrderBy(r => r).ToArray()`,
|
||||
- `memberCount` — `cluster.State.Members.Count`,
|
||||
- `unreachableCount` — `cluster.State.Unreachable.Count`.
|
||||
All property names confirmed compiling in this repo (`ActiveNodeHealthCheck.cs:116-132`, `AkkaActiveNodeGate.cs:33-49`).
|
||||
- Pass it through every result path: `HealthCheckResult.Healthy/Degraded/Unhealthy(description, data)`. The no-ActorSystem / cluster-inaccessible paths keep returning description-only (no data) — pinned by `AkkaClusterStatusPolicyTests.cs:63-96`.
|
||||
- ASP.NET Core copies `HealthCheckResult.Data` → `HealthReportEntry.Data` automatically, and `MapZbHealth` already wires `ZbHealthWriter` (`ZbHealthEndpointExtensions.cs:42-54`) — **no consumer code change needed beyond the package bump**, provided the app registers this shared check (verified for ScadaBridge; Task 2.1 verifies OtOpcUa).
|
||||
|
||||
Tests (`tests/ZB.MOM.WW.Health.Akka.Tests/`): the `InternalsVisibleTo` hook (`ActiveNodeHealthCheck.cs:7`) exposes the helper. Use `Akka.TestKit.Xunit2` (pin 1.5.62; conventions per existing tests — inherit `TestKit`, never hand-roll `ActorSystem.Create` + manual join) with a single-node self-joined cluster: assert `leader` == self address, `memberCount` == 1, `selfRoles` round-trip; plus the check-level test that a healthy result now carries data and the degraded no-system path carries none.
|
||||
|
||||
### Task 0.3 — version, pack, publish
|
||||
- `Directory.Build.props`: `0.1.0` → `0.2.0`. Note the change in the repo's CHANGELOG/README if present.
|
||||
- `dotnet test` from the repo root (baseline 58 tests across 3 test projects — all green plus the new ones), then `dotnet pack -c Release -o artifacts` (3 nupkgs @ 0.2.0).
|
||||
- Publish: `dotnet nuget push artifacts/ZB.MOM.WW.Health*.0.2.0.nupkg --source https://gitea.dohertylan.com/api/packages/dohertj2/nuget/index.json --api-key <from ~/.nuget user config / keychain — same credential every prior publish used>` (one command per nupkg or glob).
|
||||
- Verify like every prior lib release (the family gotcha: "published" claims must be proven): `curl -s https://gitea.dohertylan.com/api/packages/dohertj2/nuget/registration/zb.mom.ww.health/index.json` shows 0.2.0 (NOT `/nuget/v3/registration/` — that 404s for everything), then restore-verify from a scratch consumer project that pins 0.2.0.
|
||||
|
||||
**Phase DoD:** suite green; 3 packages at 0.2.0 restore-verified from the feed; a manual smoke (any sample host + the check) shows `entries["akka-cluster"].data.leader` in `/health/ready`.
|
||||
|
||||
---
|
||||
|
||||
## Phase 1 — ScadaBridge PR: site-node health endpoints
|
||||
|
||||
Repo: `~/Desktop/ScadaBridge` (branch off `main`, e.g. `feat/site-node-health`). Site branch of `src/ZB.MOM.WW.ScadaBridge.Host/Program.cs` is lines 485-570: Kestrel `:8083` gRPC Http2-only (507-512), `:8084` `Http1AndHttp2` (516-518); site DI via `SiteServiceRegistration.Configure` (:532); site pipeline maps only `MapZbMetrics` (:545), gRPC (:548), `MapZbLocalDbSync` (:555). No auth middleware and no FallbackPolicy exist on the site pipeline, so anonymous health needs nothing special.
|
||||
|
||||
### Task 1.1 — bump Health packages
|
||||
`Directory.Packages.props:93-95`: all three `ZB.MOM.WW.Health*` 0.1.0 → **0.2.0**.
|
||||
|
||||
### Task 1.2 — site health checks (register between `:532` and `var app = builder.Build()` at `:534`)
|
||||
Mirror the central registration style (`Program.cs:277-304`, `AddTypeActivatedCheck`):
|
||||
1. **`akka-cluster`** (tags: Ready) — the shared `AkkaClusterHealthCheck` with `AkkaClusterStatusPolicy.Default`, exactly as central does at 282-286. The site ActorSystem→DI bridge already exists (`SiteServiceRegistration.cs:168-169`). This is also what carries `data.leader` for site pairs.
|
||||
2. **`localdb`** (tags: Ready) — **new** check class in the Host (site namespace): opens the consolidated site SQLite (`LocalDb:Path`) and runs `SELECT 1`. There is no EF context on site (central's `DatabaseHealthCheck<ScadaBridgeDbContext>` does NOT apply — that context is central-only, `Program.cs:266`). Optionally enrich `data` with `ISyncStatus.Connected`/`OplogBacklog` (`SiteServiceRegistration.cs:139-140`) when replication is enabled — enrich only; replication-off must NOT fail the check (replication is default-OFF by design).
|
||||
3. **`active-node`** (tags: Active) — **new** thin `SitePairActiveNodeHealthCheck` delegating to `IClusterNodeProvider.SelfIsPrimary` (registered on site, `SiteServiceRegistration.cs:172-178`; role-scoped `site-{SiteId}` oldest-Up). **Do NOT reuse central's `OldestNodeActiveHealthCheck`** — it calls `SelfIsOldest(cluster)` with no role scope (`OldestNodeActiveHealthCheck.cs:30`) and would compute oldest across the wrong member set. Follow the existing delegation precedent `SiteEventLogActiveNodeCheck` (`SiteServiceRegistration.cs:187-191`). Active semantics: primary → Healthy, standby → Unhealthy (503 = Standby to the dashboard — same convention as central).
|
||||
|
||||
### Task 1.3 — map endpoints
|
||||
`app.MapZbHealth();` in the site pipeline after `UseRouting()` (:539), alongside `MapZbMetrics()` (:545). Served on `:8084` (Http1AndHttp2). No auth changes.
|
||||
|
||||
### Task 1.4 — tests
|
||||
Extend `tests/ZB.MOM.WW.ScadaBridge.Host.Tests/`:
|
||||
- Imitate `HealthCheckTests.cs` (`WebApplicationFactory<Program>`, `[Collection(HostBootCollection.Name)]`, `UseSetting("ScadaBridge:Node:Role", ...)`, `SkipMigrations=true`, registration assertions via `IOptions<HealthCheckServiceOptions>.Value.Registrations` + `registration.Factory(services)` type checks). **Caveat found in recon:** every existing factory test boots the Central role; there is no site-role factory bootstrap yet. Build one (Role=Site + minimal `ScadaBridge:Node` config + temp `LocalDb:Path`); if the full site host proves too heavy to boot in-test, fall back to the `CompositionRootTests.cs`/`SiteLocalDbWiringTests.cs` style over `SiteServiceRegistration.Configure` for registrations, plus a factory test for just the endpoint mapping. Include a positive control for any "central-only check absent on site" assertion.
|
||||
- Assert: site maps `/health/ready` + `/health/active` (non-404); registrations exactly {akka-cluster→Ready, localdb→Ready, active-node→Active} with the intended check types; central registrations unchanged.
|
||||
|
||||
**Phase DoD:** full ScadaBridge suite green; on the local docker rig, `curl http://<site-node>:8084/health/ready` returns the canonical JSON with `akka-cluster` carrying `data.leader`, and `/health/active` splits 200/503 across the pair; central `/health/ready` also shows `data.leader` (free via the bump). PR merged to `main`.
|
||||
|
||||
---
|
||||
|
||||
## Phase 2 — OtOpcUa bump (+ optional family alignment)
|
||||
|
||||
### Task 2.1 — verify the `akka` check is the shared class
|
||||
OtOpcUa registers ready checks `configdb`, `akka`, `admin-leader`, `localdb-replication` (registration reachable from `MapOtOpcUaHealth` / Program wiring, `src/Server/ZB.MOM.WW.OtOpcUa.Host`). **Verify** the `akka` check is `ZB.MOM.WW.Health.Akka.AkkaClusterHealthCheck`. If it is → nothing more. If bespoke → either swap to the shared check (preferred, same policy semantics) or populate the same `data` keys (`leader`/`selfAddress`/`selfRoles`/`memberCount`/`unreachableCount`) so the dashboard sees one contract.
|
||||
|
||||
### Task 2.2 — bump
|
||||
`~/Desktop/OtOpcUa/Directory.Packages.props`: `ZB.MOM.WW.Health*` → 0.2.0. Build + run the Host/AdminUI test suites touched by health. Rig check: any admin node `/health/ready` shows `data.leader`.
|
||||
|
||||
### Task 2.3 (recommended) — alignment bumps, mxgw + HistorianGateway
|
||||
Neither uses CPM (**no `Directory.Packages.props` — the known trap**): bump the inline `ZB.MOM.WW.Health` pins in `MxGateway.Server.csproj` and `ZB.MOM.WW.HistorianGateway.Server.csproj` to 0.2.0. No behavior change (no Akka checks there); keeps the family version matrix aligned. Skippable if either repo has unrelated in-flight work.
|
||||
|
||||
**Phase DoD:** OtOpcUa on 0.2.0 with leader data live on the rig; alignment bumps merged or explicitly deferred with a note.
|
||||
|
||||
---
|
||||
|
||||
## Phase 3 — the `ZB.MOM.WW.Overview` app
|
||||
|
||||
Location: `scadaproj/ZB.MOM.WW.Overview/` — **plain directory, NOT a nested git repo** (family rule). App conventions (copied from HistorianGateway, the reference implementation — cites from recon):
|
||||
|
||||
### Task 3.1 — scaffold
|
||||
- `ZB.MOM.WW.Overview.slnx` with `/src/` + `/tests/` folders (LocalDb's slnx is the minimal template).
|
||||
- `src/ZB.MOM.WW.Overview/ZB.MOM.WW.Overview.csproj`: `Sdk="Microsoft.NET.Sdk.Web"`, net10.0, **inline pinned versions, no CPM** (the app convention): `ZB.MOM.WW.Theme` 0.3.1, `ZB.MOM.WW.Health` 0.2.0, `ZB.MOM.WW.Telemetry` + `.Serilog` 0.1.0, `ZB.MOM.WW.Configuration` 0.1.0, `Serilog.AspNetCore` + console/file sinks. NO Auth/Audit/Secrets packages.
|
||||
- `Directory.Build.props` with the family zero-warnings gate (`TreatWarningsAsErrors`, `AnalysisLevel=latest`, `EnforceCodeStyleInBuild`, `Deterministic`, Nullable/ImplicitUsings) — copy HistorianGateway's.
|
||||
- `nuget.config`: copy HistorianGateway's (nuget.org `*` + `dohertj2-gitea` source with `ZB.MOM.WW.*` packageSourceMapping; credentials stay user-level, never committed).
|
||||
|
||||
### Task 3.2 — registry options + validation
|
||||
`Registry/OverviewOptions.cs` (+ `ApplicationEntry`, `InstanceEntry`) binding section `Overview` per the design §3 schema: global `PollIntervalSeconds`(10)/`TimeoutSeconds`(3)/`StaleAfterSeconds`(45); per-application and per-instance overrides (instance wins); instance fields `Name`, `Group?`, `BaseUrl`, `ManagementUrl?`, `HasActiveRole`(false).
|
||||
`Registry/OverviewOptionsValidator.cs` via `OptionsValidatorBase` + `AddValidatedOptions<OverviewOptions, OverviewOptionsValidator>` (`ZB.MOM.WW.Configuration`): non-empty applications, ≥1 instance each, unique app names + unique instance names within an app, absolute http/https `BaseUrl`/`ManagementUrl`, all intervals positive, **effective stale > effective poll for every instance**. Plus `ConfigPreflight.For(builder.Configuration).RequireValue("Overview:Applications:0:Name")…`-style presence check pre-`Build()` (HistorianGateway `Program.cs:62-66` is the pattern).
|
||||
|
||||
### Task 3.3 — health client + status model
|
||||
`Polling/ZbHealthReportClient.cs`: GETs a URL, deserializes the canonical shape `{status, totalDurationMs, entries:{name:{status, description, durationMs, data?}}}` — `data` values as `JsonElement` (they arrive as arbitrary JSON; convert to string/int on read). Treat both 200 and 503 as *parseable* responses (503 carries a body too); classify non-JSON/undeserializable as Unreachable. In-app for v1 (design §8.2).
|
||||
`Polling/InstanceStatus.cs`: `Up | Degraded | Down | Unreachable` (+ separate `ActiveState: Active|Standby|Unknown|NotApplicable`, `IsStale` computed at render). Derivation per design §5, including two-strike flap damping (state degrades only after 2 consecutive failures; recovery immediate).
|
||||
|
||||
### Task 3.4 — poller + snapshot store
|
||||
`Polling/OverviewPollerService.cs : BackgroundService`: `PeriodicTimer(PollIntervalSeconds)`; each tick polls all instances in parallel (`Task.WhenAll`; per-instance `CancellationTokenSource` with its effective timeout; named `HttpClient` via `IHttpClientFactory`; no retries — next tick is the retry). Per instance: `/health/ready` always; `/health/active` when `HasActiveRole`.
|
||||
`Polling/OverviewSnapshotStore.cs`: single immutable snapshot object swapped atomically (`Volatile.Write`/`Interlocked.Exchange`); each instance snapshot carries `LastUpdatedUtc`, latency, parsed entries, derived status, consecutive-failure counter. A C# `event`/`Action` notifies circuits to re-render on swap. **Pages never poll** — they read the store only (cache-first requirement).
|
||||
Staleness: computed at render (`now - LastUpdatedUtc > effective StaleAfterSeconds` ⇒ render Stale regardless of stored status); the page also runs a small UI tick (~5 s) so ages/staleness advance without a poll event.
|
||||
|
||||
### Task 3.5 — leader aggregation
|
||||
`Polling/LeaderResolver.cs`: for each `(Application, Group)` with any `HasActiveRole` instance, collect `entries["akka-cluster"].data.leader` from members whose status is Up/Degraded; resolve the Akka address (`akka.tcp://<system>@host:port`) to a registry instance by **host match against `BaseUrl`** (fall back to showing the raw address when unmatched — different ports are expected, the Akka port is not the HTTP port); if reporting members disagree → set the group's `LeaderDisagreement` flag (split-brain tell). Pure static logic → heavily unit-testable.
|
||||
|
||||
### Task 3.6 — UI
|
||||
**Visual reference (match it):** `docs/mockups/overview-dashboard-mockup.html` — a static, self-contained mockup built from the real Theme tokens + embedded IBM Plex; open it in a browser before styling. It defines: the KPI strip treatment (alert/caution tints), the instance-card anatomy (status stripe on the top edge; **dashed border = Unreachable, solid = Down**; faded body + "stale since" for Stale), group headers with `Leader · <instance>` chips and the split-brain warning chip, footer `ready/active` raw-signal line, and the check-detail `<details>` layout with the monospace `data` line. Reuse Theme classes where they exist (`chip-*`, `panel`, `kv`, `agg-*`); the mockup's extra classes (`inst*`, `group*`, `check-*`) go into the app's own `site.css`.
|
||||
|
||||
Copy HistorianGateway's Blazor skeleton, minus every auth piece:
|
||||
- `Components/App.razor` (head order matters: Bootstrap CSS → `<ThemeHead/>` → `site.css` → `HeadOutlet @rendermode=InteractiveServer`; body: `<Routes @rendermode=InteractiveServer/>` → Bootstrap JS → `<ThemeScripts/>` → `blazor.web.js`), `Routes.razor` with plain `RouteView` (no `AuthorizeRouteView`), `_Imports.razor` incl. `@using ZB.MOM.WW.Theme`.
|
||||
- `Components/Layout/MainLayout.razor` → `ThemeShell Product="SCADA Overview" Accent="…"` (pick an unused accent; HG is `#2f855a`), Nav = single `NavRailItem Href="/" Match="NavLinkMatch.All"` (+ a Health link later if wanted), no AuthorizeView anywhere.
|
||||
- `Components/Pages/Overview.razor` (`@page "/"`): header strip (counts n Up/Degraded/Down/Unreachable/Stale, last-refresh clock, pause/resume toggle — pause just stops applying poll events + lets staleness paint), then per-application sections: app name + worst-status rollup pill; cluster groups get a sub-header with `Leader: <instance>` chip (`StatusPill Info`) and the disagreement warning chip when flagged; instance grid of `InstanceCard`s.
|
||||
- `Components/Widgets/InstanceCard.razor`: `TechCard` + `StatusPill` composition — name/group, status pill (mapping per design §5), Active badge, Leader badge, latency, relative last-seen ("stale since …" once stale), native `<details>` check list (per-check pill + description + data leader line for akka-cluster), `TechButton` (Ghost/Secondary) → `ManagementUrl` (`target="_blank" rel="noopener"`).
|
||||
- Program wiring: `AddRazorComponents().AddInteractiveServerComponents()`; `MapRazorComponents<App>().AddInteractiveServerRenderMode()` — **no** `RequireAuthorization`, no antiforgery/cascading auth state; `UseStaticFiles()` before endpoints (Theme assets come from `_content/ZB.MOM.WW.Theme/`).
|
||||
|
||||
### Task 3.7 — self-observability
|
||||
`AddZbSerilog(o => o.ServiceName = "overview")`; `AddZbTelemetry(o => { o.ServiceName = "overview"; o.Meters = [OverviewMetrics.MeterName]; })` — **`Meters` is a silent allowlist: forgetting the custom meter name silently drops the gauge** (known family gotcha). `OverviewMetrics`: observable gauge `zb_overview_instance_status` (labels app/instance/group; value = status enum ordinal) + poll-duration histogram. Register own checks: `registry-loaded` and `last-poll-cycle-completed` (Unhealthy if no completed cycle within 2× poll interval — makes the dashboard's own `/health/ready` honest). `MapZbHealth()` + `MapZbMetrics()`.
|
||||
|
||||
### Task 3.8 — tests (`tests/ZB.MOM.WW.Overview.Tests`)
|
||||
xunit 2.9.3 + `xunit.runner.visualstudio` 3.1.4 + `Microsoft.NET.Test.Sdk` 17.14.1 + **bunit 1.40.0** (Theme's own test stack — imitate `StatusPillTests.cs`: `class X : TestContext`, `RenderComponent<T>`, class-list asserts):
|
||||
- Validator table tests (incl. stale ≤ poll rejected, relative URL rejected).
|
||||
- Health-client golden tests against **captured real payloads** from all four apps (capture during Phase 4 prep; until then use handcrafted fixtures matching `ZbHealthWriter` incl. entries with and without `data`).
|
||||
- Status derivation: 200/Healthy→Up, 200/Degraded→Degraded, 503→Down, timeout/refused/garbage→Unreachable, flap damping (1 failure holds, 2 flips, recovery immediate), active 200/503/error → Active/Standby/Unknown.
|
||||
- Staleness: fresh vs. aged snapshot with per-instance override.
|
||||
- LeaderResolver: agreement, disagreement flag, address→instance host matching, unmatched fallback.
|
||||
- Snapshot store: atomic swap + event fired.
|
||||
- bUnit: `InstanceCard` renders pill class/badges/link per state; Overview page smoke over a stubbed store.
|
||||
- `WebApplicationFactory<Program>` boot tests: page + `/health/ready` + `/metrics` reachable **with no auth**; malformed registry → boot fails with the `ConfigPreflight`/validator message.
|
||||
|
||||
### Task 3.9 — config + docker
|
||||
- `appsettings.json`: Logging skeleton, empty `Serilog`/`Telemetry` sections (libs default them), `Overview` registry sample; `appsettings.Development.json`: Kestrel `Http` endpoint `http://localhost:5320` + a dev registry pointing at the docker rigs. **Derive the rig instance URLs from the compose files, don't guess:** OtOpcUa docker-dev rig → `~/Desktop/OtOpcUa` docker-dev compose (6 nodes; AdminUI reachable on host :9200 — per-node published HTTP ports are in the compose port mappings); ScadaBridge local cluster → `~/Desktop/ScadaBridge/docker/` compose (central pair UI ports + site-node :8084 mappings); mxgw → `windev:5130`; HistorianGateway → its docker compose (:5220) or `wonder-app-vd03:5222` (VPN). Production Kestrel via env (`Kestrel__Endpoints__Http__Url`), the HG pattern; **don't set `ASPNETCORE_URLS` alongside Kestrel endpoint config** (URLs-override trap).
|
||||
- `docker/`: HG-pattern runtime-only image — host-side `dotnet publish -c Release -o docker/publish -p:UseAppHost=false` (host has the Gitea feed creds), `FROM mcr.microsoft.com/dotnet/aspnet:10.0`, `COPY publish/ ./`, `EXPOSE 5320`, `ENTRYPOINT ["dotnet","ZB.MOM.WW.Overview.dll"]`; compose project name `zb-overview`, port `5320:5320`, `ASPNETCORE_ENVIRONMENT` + Kestrel env vars. No env_file needed (no secrets). Note: `aspnet:10.0` has no `curl` — healthcheck via `wget`/dotnet or skip.
|
||||
|
||||
**Phase DoD:** `dotnet build` 0 warnings; `dotnet test` green; `dotnet run` on :5320 renders the page instantly from cache against a dev registry; container builds and runs.
|
||||
|
||||
---
|
||||
|
||||
## Phase 4 — live acceptance (design §9)
|
||||
|
||||
Prep: capture real `/health/ready` + `/health/active` payloads from all four apps (post-0.2.0) into the golden-test fixtures. Then run the design §9 checklist end-to-end on the rigs (OtOpcUa 6-node docker-dev + ScadaBridge local cluster + windev mxgw + wonder-app-vd03 HG if reachable/VPN):
|
||||
1. Every registered instance renders correctly; single Active per pair (central AND site).
|
||||
2. **Leader chip correct per cluster group and moves when the leader node is stopped**; split-brain flag not shown in normal operation.
|
||||
3. Kill a node → Unreachable within ~2 cadences; restart → recovers.
|
||||
4. Force a Degraded check → Warn with failing entry visible.
|
||||
5. Cache-first: first page load instant, no poll triggered by the request.
|
||||
6. Pause auto-refresh past `StaleAfterSeconds` → all cards Stale; resume → clean recovery. Also stop the poller (kill the service loop) → same.
|
||||
7. Management links open the right UIs; page reachable with **no login**.
|
||||
8. Malformed registry → boot fails with a clear preflight message.
|
||||
9. `curl :5320/health/ready` + `/metrics` (gauge present with per-instance labels).
|
||||
|
||||
**Program DoD:** all 9 pass; ScadaBridge PR merged; OtOpcUa bump merged; Health 0.2.0 on the feed; scadaproj commits for the lib + new app + docs.
|
||||
|
||||
---
|
||||
|
||||
## Cross-cutting notes for the executor
|
||||
|
||||
- **Family gotchas that WILL bite:** `ZbTelemetryOptions.Meters` silent allowlist; mxgw/HG have no `Directory.Packages.props` (inline pins); never host-`sqlite3` a live container WAL DB during rig checks; `Akka.TestKit.Xunit2` is xunit-v2-only (fine here — everything in this plan is xunit 2.9.3); Gitea feed verification uses `/api/packages/dohertj2/nuget/registration/…`.
|
||||
- **Do not** add auth "while at it" — anonymous read-only is a requirement, not an omission.
|
||||
- **Contract discipline:** the `data` field is additive and optional; the dashboard must render fully (minus leader) against 0.1.0-era payloads — that's what keeps Phase 3 unblocked by Phases 1/2 and tolerant of un-bumped deployments.
|
||||
- Commit style: lib change + publish in scadaproj (`feat(health): 0.2.0 …`); ScadaBridge and OtOpcUa changes as PRs in their own repos referencing this plan; new app committed in scadaproj.
|
||||
@@ -0,0 +1,86 @@
|
||||
# Central overview dashboard for the scadaproj family — options investigation
|
||||
|
||||
**Date:** 2026-07-22
|
||||
**Goal:** a single pane showing status of every app instance (OtOpcUa, ScadaBridge, MxAccessGateway, HistorianGateway — each with multiple nodes/pairs), with deep links to each instance's own management UI, pre-configured via an appsettings registry of applications and instances. "Aspire-dashboard-like."
|
||||
|
||||
---
|
||||
|
||||
## 1. What the family already gives us (verified in code, 2026-07-22)
|
||||
|
||||
The reconnaissance result is unusually favorable: **the data plane for a poll-based overview already exists on all four apps, with zero changes needed in the target apps.**
|
||||
|
||||
- **Health endpoints are uniform and anonymous everywhere.** All four apps call the shared `MapZbHealth()` (`ZB.MOM.WW.Health`), which maps `/health/ready`, `/health/active`, and `/healthz`, all `.AllowAnonymous()` in the library itself (`ZbHealthEndpointExtensions.cs:35-64`). Even OtOpcUa's global `FallbackPolicy = RequireAuthenticatedUser` doesn't gate them (endpoint `AllowAnonymous` wins).
|
||||
- **The JSON contract is canonical and stable** (`ZbHealthWriter.cs:45-81`, camelCase):
|
||||
```json
|
||||
{ "status": "Healthy|Degraded|Unhealthy",
|
||||
"totalDurationMs": 12.3,
|
||||
"entries": { "<check>": { "status": "...", "description": null, "durationMs": 1.2 } } }
|
||||
```
|
||||
Healthy/Degraded → HTTP 200, Unhealthy → 503.
|
||||
- **`/health/active` is the active/standby signal.** ScadaBridge Central's `active` tier runs the oldest-member `active-node` check; OtOpcUa has `admin-leader`. No off-the-shelf product can express active/standby natively — this endpoint makes it a first-class dashboard column for us.
|
||||
- **`/metrics` (Prometheus) is anonymous on all four** (explicitly on OtOpcUa; by absence of a fallback policy on the others). OTLP push is config-only on three apps (`<App>:Telemetry:Exporter=Otlp` + `OtlpEndpoint`); **HistorianGateway hardcodes its telemetry setup and would need a one-line change** to gain the config keys.
|
||||
- **Nothing aggregates across apps today.** The closest is ScadaBridge's `/monitoring/health` (CentralUI), but it's push-based over the site transport and scoped to ScadaBridge sites in one cluster. OtOpcUa/HG/mxgw pages show only their own instance. A multi-app registry + remote HTTP poller is genuinely new work.
|
||||
- **Reusable building blocks for an in-house pane:** `ZB.MOM.WW.Theme` (`ThemeShell`, `StatusPill` with Ok/Warn/Bad/Idle/Info, `TechCard`), `ZB.MOM.WW.Auth.AspNetCore` (the same hardened LDAP cookie login all four apps use), `ZB.MOM.WW.Configuration` (validate the registry at startup), `ZB.MOM.WW.Telemetry`/`Health` (the dashboard monitors itself the family way). No prebuilt tile/grid widget exists — a node-card composite would be new, but it's `TechCard` + `StatusPill` composition.
|
||||
- **Known ports for the registry:** OtOpcUa AdminUI dev `:9000` (driver-only nodes fall back to `:5000`); ScadaBridge Central `:5000/:5001` (site nodes expose only `:8084` metrics — no health/UI); mxgw `:5120/:7121`; HistorianGateway dashboard `:5220` dev / single TLS ALPN port (rec. `:5222`) in production.
|
||||
|
||||
**Watch-out for the registry design:** ScadaBridge *site* nodes map no health endpoints at all (metrics only), so a site-pair row can't be polled the same way as a central pair — either poll `:8084/metrics` for liveness, read site status from Central's aggregator, or (later) add `MapZbHealth` to site nodes.
|
||||
|
||||
## 2. Off-the-shelf candidates (facts verified by fetch, 2026-07-22)
|
||||
|
||||
| Option | Version / status | Config-file registry | Persistence | Auth | Verdict |
|
||||
|---|---|---|---|---|---|
|
||||
| **Aspire dashboard (standalone)** | Aspire 13 (Nov 2025); `mcr.microsoft.com/dotnet/aspire-dashboard` | **No** — resource UI is disabled in standalone; needs a custom gRPC resource service to light up | None (in-memory, lost on restart) | BrowserToken / OIDC | Not the ops pane. MS docs: dev/short-term diagnostic tool. `AddExternalService()`/custom URLs are AppHost run-mode features only. |
|
||||
| **AspNetCore.HealthChecks.UI (Xabaril)** | 9.0.0 (Dec 2024), net8.0 TFM only; upstream coasting on dependabot since Aug 2025 ("looking for maintainers"); tiny `DotNetDiag` fork exists | **Yes** — `HealthChecksUI:HealthChecks` array in appsettings; exactly the requested model | SQLite/SQL Server/InMemory | In-process — **our LDAP cookie works unmodified** | Closest functional match, but it **requires the `UIResponseWriter` payload** (issue #285) — our `ZbHealthWriter` shape is close but not identical (`durationMs` vs `duration`, no `data`/`tags`), so every app would need a second UI-format endpoint. Plus a slow-moving upstream dependency. |
|
||||
| **Gatus** | v5.36.0 (May 2026), active | **Yes** — YAML; conditions incl. `[BODY]` JSONPath, so it can evaluate our health JSON **with zero app changes** (incl. degraded) | SQLite/Postgres | Built-in basic/OIDC (no LDAP cookie) | Best zero-effort comparable: registry + JSON conditions + uptime history + 30+ alert providers in one Go binary. Trade: non-.NET, non-Theme UI, outside our auth stack. |
|
||||
| **Homepage (gethomepage.dev)** | v1.13.2 (Jun 2026) | Yes — YAML link tiles | n/a | None (reverse proxy) | Link portal only; `siteMonitor` is status-code-only — cannot express Degraded or active/standby. Out. |
|
||||
| **Grafana + Prometheus + blackbox** | Grafana 13.0 (Apr 2026) | Yes — scrape config | Full TSDB history | **Native LDAP** | Most durable, real alerting, grows into actual ops monitoring. Highest setup (3 containers + dashboard authoring); Degraded is invisible to blackbox (HTTP 200) without a status gauge metric or `ResultStatusCodes` remap; deep links via data links are clunkier than a real portal. |
|
||||
|
||||
## 3. Options
|
||||
|
||||
### Option A — build a small in-house overview app (`ZB.MOM.WW.Overview`) — **RECOMMENDED**
|
||||
|
||||
A thin Blazor Server app hosted in scadaproj alongside the other shared components, composed almost entirely from the family's own libraries:
|
||||
|
||||
- **Registry:** `Overview:Applications[]` in appsettings — per app: name, kind, accent; per instance: display name, site/role, `healthBaseUrl`, `managementUrl`, optional `metricsUrl`, `expectActive` flag, poll interval/timeout overrides. Validated at startup via `ZB.MOM.WW.Configuration` (`AddValidatedOptions` + `ConfigPreflight`) — malformed registry fails the boot, not the first poll.
|
||||
- **Poller:** one `BackgroundService` + `IHttpClientFactory`; polls each instance's `/health/ready` + `/health/active` in parallel on a fixed cadence (5–15 s), deserializes the canonical `ZbHealthWriter` JSON (a ~30-line typed client — worth putting in `ZB.MOM.WW.Health` as `ZbHealthReportClient` so it's contract-owned), and keeps an in-memory snapshot. Statuses per instance: **Up / Degraded / Down / Unreachable**, plus **Active/Standby** from `/health/active` — the column no off-the-shelf option can render.
|
||||
- **UI:** `ThemeShell` + a new node-card grid built from `TechCard` + `StatusPill`; per-check expandable detail (the `entries` dict); every card deep-links to the instance's own management page. LDAP cookie login via `AddZbLdapAuth` — same UX as the other four apps (the dashboard reads only anonymous endpoints, but the pane itself should sit behind the family login).
|
||||
- **Self-hosting:** it registers its own `MapZbHealth`/`MapZbMetrics` like any family app; a `docker/` folder mirroring HistorianGateway's runtime-only image pattern.
|
||||
|
||||
**Fit:** best-practice fit for this shop. The requested feature set (appsettings registry, status, deep links) maps 1:1 onto libraries that exist precisely so new apps can be assembled this way; the only genuinely new code is the poller + snapshot + one grid page — realistically **2–4 days** for a first useful version. It also keeps auth, theming, and ops conventions identical to the rest of the family, which none of the off-the-shelf options can do. Deliberate scope cut for v1: no history/alerting (that's Option D's job if it's ever needed — and the registry file ports straight to a Prometheus scrape config).
|
||||
|
||||
**Costs / risks:** you own the code (small); no built-in uptime history or webhooks in v1; the ScadaBridge-site-node gap (§1 watch-out) needs a decision in the registry design.
|
||||
|
||||
### Option B — AspNetCore.HealthChecks.UI embedded in a thin family host
|
||||
|
||||
Host the Xabaril UI inside a small ASP.NET app behind our LDAP cookie. Gets registry-from-appsettings, polling, SQLite history, and webhooks for near-zero UI code.
|
||||
|
||||
**Fit:** the strongest off-the-shelf functional match, and the only one where `ZB.MOM.WW.Auth` works unmodified. But it costs a change in **all four target apps** (a second health endpoint in `UIResponseWriter` format, because the UI hard-requires that payload), buys a visibly unmaintained upstream (net8.0-only TFM, dependabot-only commits since Aug 2025), has no active/standby concept, and its UI will never look like Theme. For a shop that already owns Health + Theme + Auth libs, it saves less than it costs.
|
||||
|
||||
### Option C — Gatus container as a zero-app-change status page
|
||||
|
||||
Deploy Gatus with a YAML registry; its `[BODY]` JSONPath conditions consume our existing health JSON directly (including Degraded), and it brings uptime history + alerting for free.
|
||||
|
||||
**Fit:** the best effort-to-value stopgap — running in an afternoon, no app changes, actively maintained. Best practice would call this the "buy" answer if the family didn't have its own UI/auth stack. Trades: separate basic/OIDC auth outside the LDAP cookie, un-Theme-able UI, and active/standby only expressible as another endpoint row rather than a paired-node concept. Reasonable as an interim pane while Option A is built, or as a permanent low-ceremony answer if the dashboard turns out to be rarely used.
|
||||
|
||||
### Option D — Grafana + Prometheus + blackbox exporter
|
||||
|
||||
**Fit:** the right long-term answer **if** the real goal grows into ops monitoring (history, alerting, SLOs) rather than a convenience pane — native LDAP auth is a genuine plus. As a *first* deliverable for "status + links" it's the highest effort and the worst at deep links/branding, and Degraded/active-standby need extra plumbing (a health-status gauge in `ZB.MOM.WW.Telemetry` would be the clean fix, and is worth doing eventually regardless). Not first; a natural phase 2 that Option A's registry feeds directly.
|
||||
|
||||
### Option E — Aspire dashboard standalone
|
||||
|
||||
**Fit:** rejected as the overview pane — in standalone mode the resource/status/links UI is disabled (it's an OTLP telemetry viewer only), nothing persists across restarts, and Microsoft positions it explicitly as a dev/short-term diagnostic tool. **Worth keeping as a free companion tool:** three of the four apps can push OTLP with config only, so an ad-hoc `aspire-dashboard` container gives structured logs + traces during debugging for zero code. (HistorianGateway needs the one-line telemetry-config change to join.)
|
||||
|
||||
> **DECISION (2026-07-22):** Option A chosen — custom app, family look-and-feel (Theme/Auth/Health kit). Aspire rejected for the pane. V1 design: [`docs/plans/2026-07-22-overview-dashboard-design.md`](docs/plans/2026-07-22-overview-dashboard-design.md).
|
||||
|
||||
## 4. Recommendation
|
||||
|
||||
**Option A** — build the thin `ZB.MOM.WW.Overview` app from the family's own parts, with two riders:
|
||||
|
||||
1. If an interim pane is wanted *this week*, stand up **Gatus (Option C)** against the existing health endpoints — zero app changes, discard later.
|
||||
2. Keep **Aspire standalone (Option E)** in the toolbox as a dev-time OTLP viewer, and make HistorianGateway's telemetry config-driven (one line) so all four apps can join it.
|
||||
|
||||
Decisions to make before building A: how site-node ScadaBridge rows are sourced (metrics-liveness vs Central-aggregator vs adding MapZbHealth to sites), and whether v1 needs any history/alerting (recommended: no — defer to Option D if ever needed).
|
||||
|
||||
---
|
||||
|
||||
*Sources: family code (ZB.MOM.WW.Health `ZbHealthEndpointExtensions.cs`/`ZbHealthWriter.cs`, per-app Program.cs wiring); aspire.dev standalone/configuration/security docs + Aspire 13 announcement; Xabaril repo + NuGet + issue #285 + DotNetDiag fork; TwiN/gatus releases; gethomepage.dev docs; Grafana/blackbox dashboards. All web claims verified by fetch 2026-07-22.*
|
||||
@@ -0,0 +1,351 @@
|
||||
# ScadaBridge: migrating cross-cluster ClusterClient traffic to gRPC-only
|
||||
|
||||
**Date:** 2026-07-22. Code-verified inventory of every ClusterClient touchpoint in
|
||||
`~/Desktop/ScadaBridge`, then the migration assessment. Companion to
|
||||
`akka_msquic_transport.md` §8/§8a (why chunking can't fix this path) and
|
||||
`akka_multidc_clusters.md` (why the site↔central boundary stays two clusters).
|
||||
|
||||
## 1. What ClusterClient actually carries today
|
||||
|
||||
Everything crossing the site↔central boundary funnels through **two actors** —
|
||||
`SiteCommunicationActor` (site) and `CentralCommunicationActor` (central) — which makes the
|
||||
migration surface unusually well-bounded.
|
||||
|
||||
**Receptionist registrations (3, all in `AkkaHostedService.cs`):**
|
||||
|
||||
| Role | Actor | Path | Note |
|
||||
|---|---|---|---|
|
||||
| Central | `CentralCommunicationActor` | `/user/central-communication` (`:436`) | the real workhorse |
|
||||
| Central | `ManagementActor` | `/user/management` (`:459`) | **vestigial** — intended for an out-of-cluster CLI (REQ-HOST-6a), but the in-repo CLI has no ClusterClient; nothing in the repo sends to it |
|
||||
| Site | `SiteCommunicationActor` | `/user/site-communication` (`:935`) | central→site target |
|
||||
|
||||
**Site → central** (single `central-cluster-client`, contacts from
|
||||
`CommunicationOptions.CentralContactPoints`, `AkkaHostedService.cs:942-953`) — 7 message types,
|
||||
all Ask/reply except the heartbeat (`SiteCommunicationActor.cs`):
|
||||
|
||||
| Message | Purpose | Guarantee layered on top |
|
||||
|---|---|---|
|
||||
| `NotificationSubmit` → `NotificationSubmitAck` | S&F notification delivery (central owns SMTP) | not-accepted/timeout → rows stay buffered; active-node-gated retry sweep |
|
||||
| `NotificationStatusQuery` → response | S&F status lookup | — |
|
||||
| `IngestAuditEventsCommand` → reply | audit batch push | ack = accepted-id list; throw-on-failure keeps SQLite rows Pending |
|
||||
| `IngestCachedTelemetryCommand` → reply | audit+SiteCall dual-write push | same contract |
|
||||
| `ReconcileSiteRequest` → response | per-node startup self-heal | — |
|
||||
| `SiteHealthReport` → ack | rich periodic health → `CentralHealthAggregator` | monotonic sequence numbers + dedup; failed-send counter restore |
|
||||
| `HeartbeatMessage` (Tell) | liveness, stamps `IsActive` | fire-and-forget by design |
|
||||
|
||||
**Central → site** (one ClusterClient **per site**, contacts = NodeA/NodeB from the config DB,
|
||||
`CentralCommunicationActor.cs:469-598`): every command wrapped in
|
||||
`SiteEnvelope(siteId, message)` — **29 command types** produced by `CommunicationService`
|
||||
(instance lifecycle, artifact deploy, OPC-UA browse/verify/tag read-write, cert trust,
|
||||
event-log/parked-message queries, integration routing, `TriggerSiteFailover`).
|
||||
|
||||
**Serialization: the default Newtonsoft JSON fallback** — no `serialization-bindings` anywhere.
|
||||
This is the amplifier in the known 128 KB frame bug
|
||||
(`docs/known-issues/2026-06-26-deploy-config-exceeds-akka-frame-size.md`): oversized
|
||||
ClusterClient messages are **silently dropped**, which is why deploy config already moved to
|
||||
HTTP notify-and-fetch.
|
||||
|
||||
**Not ClusterClient (stays untouched):** `DistributedPubSub` replica publishes
|
||||
(`SiteHeartbeatReplica`/`SiteHealthReportReplica`) are intra-central-cluster; all pair-internal
|
||||
Akka (singletons, membership, S&F actors) is intra-cluster.
|
||||
|
||||
## 2. Why gRPC-only is a natural completion, not a rewrite
|
||||
|
||||
The repo has already walked 3 of the 5 steps:
|
||||
|
||||
1. **`docs/plans/grpc_streams.md`** already rules ClusterClient "not built for high-throughput
|
||||
streaming" and moved all streaming values to gRPC — its stated end-state ("ClusterClient keeps
|
||||
command/control") is a waypoint, not a law.
|
||||
2. **Deploy config** already left ClusterClient for token-gated HTTP pull after the frame-size
|
||||
incident — with the rationale "fits the existing pull-based gRPC pattern".
|
||||
3. **The audit path is half-migrated already:** `sitestream.proto` *already defines*
|
||||
`IngestAuditEvents`/`IngestCachedTelemetry` with proto DTOs and an
|
||||
`ISiteStreamAuditClient` seam whose ClusterClient implementation is swappable — the contract
|
||||
and the seam exist; only the transport binding is Akka.
|
||||
4. Upstream direction agrees: JVM Akka deprecated ClusterClient in favor of gRPC for
|
||||
cross-cluster traffic; Akka.NET keeps it working but it is not where investment goes.
|
||||
5. Structural wins on arrival: the **silent-drop-over-128 KB failure class disappears** (gRPC
|
||||
default 4 MB cap + an explicit `ResourceExhausted` error instead of silence), Ask timeouts
|
||||
become real per-call deadlines, and **Akka remoting never crosses the boundary** — the
|
||||
site↔central firewall surface reduces to gRPC/HTTP ports, and the two clusters' quarantine
|
||||
lifecycles fully decouple (the coupling `akka_msquic_transport.md` §8a warns about).
|
||||
|
||||
## 3. What the migration takes
|
||||
|
||||
### 3.1 Topology decision — who serves what
|
||||
|
||||
Today the **site** is the only gRPC server (`SiteStreamService` on `:8083` h2c) and central the
|
||||
only client; but both sides already dial each other at the Akka level, and sites already dial
|
||||
central over HTTP for deploy fetch. So the clean shape is symmetric, no new firewall direction:
|
||||
|
||||
- **New `CentralControlService`** (gRPC) hosted on central's existing Kestrel (same host as the
|
||||
deploy-fetch endpoint): the 7 site→central RPCs.
|
||||
- **Extended site service** (grow `SiteStreamService` or a sibling `SiteCommandService` on the
|
||||
same `:8083` listener): the central→site command envelope.
|
||||
|
||||
(The alternative — a central-initiated duplex stream tunneling site→central pushes — avoids the
|
||||
central listener but re-implements connection management the receptionist gave for free; not
|
||||
worth it when sites already reach central over HTTP.)
|
||||
|
||||
### 3.2 Contract work
|
||||
|
||||
- **Site→central (small):** 7 RPCs. Audit/telemetry are a near-free move — proto DTOs exist and
|
||||
the swap is a new `ISiteStreamAuditClient` implementation (the `NoOp`/`ClusterClient`
|
||||
implementations prove the seam). Notifications, health report, reconcile, heartbeat need
|
||||
proto messages written (they're modest DTOs today).
|
||||
- **Central→site (the bulk):** 29 `SiteEnvelope` commands. Don't write 30 RPCs and don't smuggle
|
||||
JSON in one generic RPC — group by domain into a few RPCs with **`oneof` request/response
|
||||
envelopes** (lifecycle, deploy, OPC-UA browse/tag, queries, failover). Mirrors `SiteEnvelope`
|
||||
semantics with wire-level typing; mapper + golden tests are the real cost here.
|
||||
- **Transport seams:** the site side has precedent (`AkkaHealthReportTransport`,
|
||||
`ISiteStreamAuditClient`); `CommunicationService` (central) needs an `ISiteCommandTransport`
|
||||
seam so the 29 call sites don't change — the seam is introduced first, Akka behind it, then
|
||||
the gRPC implementation swaps in per domain group.
|
||||
|
||||
### 3.3 Semantics to preserve (all transport-agnostic, all carry over)
|
||||
|
||||
- S&F transient-failure contract: gRPC `Unavailable`/`DeadlineExceeded` → rows stay buffered —
|
||||
identical to today's not-accepted/timeout handling, with better error taxonomy.
|
||||
- Health sequence numbers + dedup + counter-restore: unchanged (they live above the transport).
|
||||
- Heartbeat stays fire-and-forget (unary, errors logged not thrown).
|
||||
- Dual-central routing: the receptionist let a site reach "whichever central answers"; the gRPC
|
||||
client needs try-in-order failover across the central A/B endpoints (`CentralContactPoints`
|
||||
becomes `CentralGrpcEndpoints`). The intra-central replica pub/sub already reconciles which
|
||||
central saw the report — unchanged.
|
||||
- Per-site channel management on central: the DB-driven contact cache refresh loop
|
||||
(`CentralCommunicationActor.cs:532-598`) carries over as a `GrpcChannel`-per-site factory —
|
||||
which `grpc_streams.md` (§167, §810) already models for the streaming path.
|
||||
|
||||
### 3.4 Work you must do anyway: auth — preshared key from the Secrets store (decided 2026-07-22)
|
||||
|
||||
`SiteStreamService` today has **no auth at all** — plaintext h2c, no interceptor (only the
|
||||
LocalDb sync path is ApiKey-gated, and only for its own service). This gap exists today
|
||||
regardless of migration — it just becomes indefensible after it. **Decision: the gRPC channels
|
||||
authenticate with a preshared key stored in `ZB.MOM.WW.Secrets`** (which ScadaBridge already
|
||||
adopted, `${secret:NAME}` resolution, fail-closed). Design:
|
||||
|
||||
- **Key scope: one PSK per site relationship** (e.g. secret name `SB-GRPC-PSK-SITE-A`), covering
|
||||
both directions of that pair's traffic (central→site commands AND that site's calls to
|
||||
`CentralControlService`). Central's store holds one secret per site; each site's local SQLite
|
||||
store holds only its own. Blast radius: a compromised site never yields another site's key.
|
||||
(A single fleet-wide PSK is simpler to seed but is rejected for exactly that reason.)
|
||||
- **Client side:** resolve `${secret:SB-GRPC-PSK-<site>}` at startup **fail-closed** (the
|
||||
Layer-B pattern the family already uses for driver secrets — boot fails loudly on a missing/
|
||||
undecryptable ref, no silent unauthenticated fallback) and attach it per-call as a bearer
|
||||
metadata header via a `CallCredentials`/interceptor on the channel pair from §3.5.
|
||||
- **Server side:** a fail-closed interceptor modeled on `LocalDbSyncAuthInterceptor` (the
|
||||
in-repo precedent) on **both** services — constant-time comparison, applied to the SiteStream/
|
||||
SiteCommand services on `:8083` and to `CentralControlService` on central. The LocalDb sync
|
||||
path keeps its own existing key; no sharing between the two mechanisms.
|
||||
- **Seeding/ops:** the `secret` CLI's *Reference audit & seed* flow (interactive console 0.3.0)
|
||||
is the deployment-setup tool — it classifies every `${secret:}` ref and seeds the gaps per
|
||||
node. Rotation: set the new value on both sides' stores, then restart the pair (pairs restart
|
||||
together by established discipline); if zero-blip rotation is ever needed, the server
|
||||
interceptor can accept old+new during a configured window.
|
||||
- **Transport honesty:** a bearer PSK over plaintext h2c is readable/replayable by anyone on
|
||||
path — today that's accepted via network isolation (same posture as the unauthenticated status
|
||||
quo, but now with authentication). TLS on these listeners is the follow-on hardening; the PSK
|
||||
design doesn't change if/when TLS lands.
|
||||
|
||||
### 3.5 Active-node resolution — does the gRPC client need to find the pair's active node?
|
||||
|
||||
**No — and this is load-bearing good news, verified in code.** Inbound commands on a site node
|
||||
are not executed locally: `SiteCommunicationActor` `Forward`s essentially everything to the
|
||||
**Deployment Manager cluster-singleton proxy** (`SiteCommunicationActor.cs:101-181`), and the
|
||||
code comments are explicit — the singleton "always lands on the active site node"; DCL
|
||||
connections/browse sessions "only exist on the singleton's node," so commands hop through the
|
||||
proxy by design (`:153-178`). The same shape holds centralward: `CentralCommunicationActor` runs
|
||||
on both central nodes, and heartbeat/health state is reconciled across the pair via the
|
||||
intra-cluster `DistributedPubSub` replicas. In other words, **each cluster already solves
|
||||
active-node routing internally with singleton proxies + pub/sub — the cross-cluster transport
|
||||
never had to know, and ClusterClient never did** (the receptionist delivers to whichever node
|
||||
answers).
|
||||
|
||||
A gRPC server on either pair node therefore keeps the exact same contract: accept the call, hand
|
||||
it to the same actor, let the singleton proxy route. What the migration must provide instead is
|
||||
cheap **either-node failover at the client**:
|
||||
|
||||
**Client failover/failback policy (per pair — applies to central's per-site clients AND the
|
||||
site's central client alike):**
|
||||
|
||||
- **Two channels, one preference.** Hold a `GrpcChannel` per endpoint (NodeA, NodeB) rather than
|
||||
one channel with re-dial logic; mark one *preferred* (config order, or `/health/active` probe
|
||||
once site-node health lands — Overview plan Phase 1). All calls go to the current channel.
|
||||
- **Failover:** on `Unavailable`/connect failure/readiness rejection
|
||||
(`SiteStreamGrpcServer` already returns `StatusCode.Unavailable` when not ready — the one
|
||||
state where the singleton proxy can't route), flip to the secondary **if available** and stay
|
||||
there (sticky-until-failure, so a flapping primary doesn't ping-pong every call).
|
||||
- **Failback:** a background probe (30–60 s cadence) checks the preferred endpoint
|
||||
(`grpc.health` check or `/health/ready`); when it answers again, new calls return to it —
|
||||
in-flight calls and open streams finish where they are. Failback is an optimization, never a
|
||||
correctness requirement: either node accepts and singleton-routes (§ above).
|
||||
- **Retry safety:** automatic cross-node retry ONLY for failures that provably never reached a
|
||||
server (connect refused, `Unavailable` before response headers). **Never blind-retry
|
||||
`DeadlineExceeded` on non-idempotent commands** (`WriteTag`, `DeployArtifacts`,
|
||||
`TriggerSiteFailover`) — ClusterClient's Ask had the same ambiguity and the layers above
|
||||
already tolerate it (S&F dedup, sequence numbers), but the migration must not *add* an eager
|
||||
retry that turns one write into two. Queries/browses are freely retryable.
|
||||
- **Streams don't fail over transparently:** an open `SubscribeInstance`/`SubscribeSite` stream
|
||||
dies with its channel; the subscriber re-subscribes via the policy above (the
|
||||
`grpc_streams.md` design already owns re-subscribe semantics — reuse, don't reinvent).
|
||||
- Cost of landing on the standby is one intra-pair remoting hop (as today).
|
||||
- One verification item for Phase 2: confirm none of the 29 command handlers bypasses the proxy
|
||||
and assumes local state (the audit/parked-message query paths are the ones to eyeball).
|
||||
|
||||
### 3.6 Deletions at the end
|
||||
|
||||
`ClusterClient`/`ClusterClientReceptionist` creation + registrations, `DefaultSiteClientFactory`,
|
||||
the `SiteEnvelope` Akka path, `CommunicationOptions.CentralContactPoints`. The
|
||||
`/user/management` registration is **deletable today** independent of everything else (no sender
|
||||
exists in the repo). `Akka.Cluster.Tools` stays for ClusterSingleton.
|
||||
|
||||
### 3.7 The management CLI is unaffected (deliberately out of scope)
|
||||
|
||||
`scadabridge.exe` never used ClusterClient: it speaks HTTP Basic to a Central's `/management`
|
||||
endpoints, which ask `ManagementActor` **in-process** (`ManagementEndpoints.cs:117` via
|
||||
`ManagementActorHolder`) — the `/user/management` receptionist registration was built for an
|
||||
out-of-cluster CLI that was never wired (REQ-HOST-6a) and has no sender in the repo. So this
|
||||
migration requires **zero CLI changes**; the CLI's transport is already boundary-independent.
|
||||
Retargeting the CLI onto the new gRPC services later is a separate, optional decision — it would
|
||||
buy one consistent management wire, but costs the CLI a gRPC stack and auth story for no current
|
||||
functional gain, so it is explicitly **not** part of this migration's scope or estimate.
|
||||
|
||||
## 4. Phasing and effort
|
||||
|
||||
| Phase | Content | Effort |
|
||||
|---|---|---|
|
||||
| 0 | Delete vestigial `ManagementActor` registration; add the §3.4 PSK-from-Secrets auth interceptor to `SiteStreamService` (standalone hardening win — central client attaches the PSK it resolves from its store) | days |
|
||||
| 1 | `CentralControlService` + the 7 site→central RPCs behind existing seams; audit swap; heartbeat/health/notifications/reconcile | ~1–2 wks |
|
||||
| 2 | `ISiteCommandTransport` seam + `oneof` command envelopes; migrate the 29 commands by domain group (deploy+lifecycle first, queries last) | ~2–3 wks |
|
||||
| 3 | Remove ClusterClient/receptionist + config cutover; live gate on the docker rig: kill central-a mid-S&F drain, site-node failover, deploy, `TriggerSiteFailover`, dual-central failover of site→central calls | ~1 wk |
|
||||
|
||||
Total ≈ **4–6 weeks** with live gating. Both transports can coexist per-path throughout (the
|
||||
seams make each path a one-line flip), so there is no big-bang cutover; rollback per phase is a
|
||||
config/DI revert.
|
||||
|
||||
**Risks, honestly:** the `oneof` envelope mappers are tedious and where bugs will hide (golden
|
||||
tests against captured payloads); ordering assumptions are low-risk (everything is request/reply
|
||||
or sequence-numbered, verify none of the 30 commands relies on ClusterClient's per-connection
|
||||
ordering across *different* commands); Ask-sender-forwarding tricks disappear (fine — RPC replies
|
||||
are direct, but check `SiteCallAuditActor`'s relay paths); the auth addition must not break the
|
||||
rig's current unauthenticated compose setup (env-gated like LocalDb's).
|
||||
|
||||
## 5. The other knob: raising the 128 KB frame cap
|
||||
|
||||
Since the frame limit motivates part of this migration, the obvious question — why not just
|
||||
raise `akka.remote.dot-netty.tcp.maximum-frame-size`? It is a legitimate knob, and if
|
||||
ClusterClient stays (Option B) it is worth doing deliberately, but know exactly what it buys:
|
||||
|
||||
- **How:** set `maximum-frame-size` (default 128000b) in the generated HOCON
|
||||
(`AkkaHostedService.BuildHocon` — today it never overrides it) and raise
|
||||
`send-buffer-size`/`receive-buffer-size` (default 256000b) to at least the new frame size.
|
||||
**The limit is enforced by the receiver**, so it must be raised on **every node of both
|
||||
clusters and rolled out everywhere before any sender relies on it** — a half-rolled-out raise
|
||||
reproduces the silent-drop bug in one direction. The pair-restart discipline makes the rollout
|
||||
cheap; the coordination requirement is the trap.
|
||||
- **What it buys:** headroom for today's fat-ish commands (browse results, query replies) with
|
||||
zero code change; the deploy-config class of payload would fit at, say, 2–4 MB.
|
||||
- **What it does not fix:** (1) the **silent-drop failure class survives** — the threshold moves,
|
||||
it doesn't disappear; anything over the new cap still vanishes without an error. (2)
|
||||
**Head-of-line gets worse, not better:** all ClusterClient traffic, heartbeats, and
|
||||
failure-detector signals share one ordered dot-netty channel, so a single 4 MB frame stalls
|
||||
everything behind it — on a 2-node availability-first posture with tight failure-detector
|
||||
timeouts, big frames are exactly what you don't want ahead of heartbeats (this is why the
|
||||
deploy fix chose notify-and-fetch over a cap raise, and why `akka_msquic_transport.md` §2
|
||||
treats the default cap as a guardrail, not a bug). (3) The default Newtonsoft serializer's
|
||||
double-JSON-escaping roughly **halves the effective budget** for JSON-bearing payloads. (4) It
|
||||
is a dead-end investment: Artery (1.6) replaces the pipeline, and Option A removes the
|
||||
constraint entirely (gRPC's 4 MB default cap fails loudly with `ResourceExhausted`, and is
|
||||
itself configurable per-call-type).
|
||||
|
||||
Verdict: as a *stopgap under Option B*, a modest raise (e.g. 512 KB–1 MB, buffers to match,
|
||||
cluster-wide) plus an explicit size check before `ClusterClient.Send` (log-and-reject instead of
|
||||
silent drop) is defensible. As part of Option A it's unnecessary — don't spend the rollout.
|
||||
|
||||
## 6. Options
|
||||
|
||||
**A. Full migration, phased as §4 (recommended).** Best-practice fit: it completes a direction
|
||||
the repo already chose twice under pressure (streaming, deploy), eliminates the silent-drop
|
||||
failure class on the control path, closes the unauthenticated-control-channel gap as part of the
|
||||
work, decouples the clusters' failure domains entirely, and lands on the transport the whole
|
||||
family already operates (gRPC + Kestrel + interceptors). Aligns with upstream's own deprecation
|
||||
of ClusterClient.
|
||||
|
||||
**B. Status quo per `grpc_streams.md` (ClusterClient keeps command/control).** Defensible if no
|
||||
current pain: the frame-size hazard is mostly defused (big payloads already moved) and the
|
||||
receptionist's discovery/failover is free; pair it with the deliberate cap raise + pre-send size
|
||||
check from §5. But it leaves two cross-cluster transports to operate,
|
||||
the silent-drop class alive for any future fat command, an unauthenticated gRPC service, and a
|
||||
dependency on the least-invested-in corner of Akka.NET. If choosing B, still do Phase 0 (auth +
|
||||
vestigial-registration deletion) — it stands alone.
|
||||
|
||||
**C. Wait for Akka.NET 1.6.** Not a reason to wait: Artery changes intra-cluster transport;
|
||||
ClusterClient's cross-cluster story doesn't improve. Deferring buys nothing here.
|
||||
|
||||
**Recommendation: A** — with Phase 0 done immediately regardless of the rest.
|
||||
|
||||
## 7. Design-completion deep dive (2026-07-22) — corrections and refinements
|
||||
|
||||
A three-track code deep dive (full message inventory; hosting/config/Secrets/tests; existing
|
||||
gRPC machinery) closed the remaining unknowns. The implementation plan lives at
|
||||
`ScadaBridge/docs/plans/2026-07-22-clusterclient-to-grpc-plan.md`; corrections that supersede
|
||||
earlier sections:
|
||||
|
||||
1. **The seam is smaller than §3.2 assumed — one choke point per direction.** Every central→site
|
||||
producer (all of `CommunicationService` AND `SiteCallAuditActor`'s two relays) funnels
|
||||
`SiteEnvelope` through **`CentralCommunicationActor`**, which unwraps and routes via the
|
||||
per-site ClusterClient; every site→central path funnels through **`SiteCommunicationActor`**.
|
||||
So the transport swap happens *inside the two communication actors* (an injected
|
||||
`ISiteCommandTransport` / `ICentralTransport` behind a config flag) — `CommunicationService`
|
||||
needs **no interface extraction** and its ~20 consumers are untouched. (It has no interface
|
||||
today and is DI-registered concrete; extracting one is unnecessary.)
|
||||
2. **Exact command count: 29, and one is dead.** 27 from `CommunicationService` + 2
|
||||
`SiteCallAuditActor` relays. **Finding: `IntegrationCallRequest` is unwired in production** —
|
||||
`RegisterLocalHandler(Integration, …)` exists only in tests, so the command always answers
|
||||
"Integration handler not available." It is excluded from the gRPC contract (28 commands
|
||||
migrate) and filed as its own issue rather than silently ported or fixed.
|
||||
3. **§3.5 verification item RESOLVED.** Of the 29: 22 are singleton-/cluster-routed (incl.
|
||||
artifact deploy and event-log query, whose "local handler" slots are actually registered to
|
||||
singleton proxies — the event-log one deliberately, per wiring comments); `TriggerSiteFailover`
|
||||
resolves its target from cluster state, node-agnostic. The **only node-local target is the
|
||||
parked-message handler** (5 command types) — and it is standby-safe *by replication*, not
|
||||
routing: it reads/mutates the LocalDb-replicated S&F store, and the delivery sweep is
|
||||
active-gated. The gRPC design must preserve exactly that (dispatch on the receiving node, no
|
||||
forced active-node routing) — which either-node gRPC does.
|
||||
4. **Central hosting is net-new work.** Central has **no** `AddGrpc`/`MapGrpcService` and no
|
||||
explicit Kestrel listener (env-driven `:5000`, Traefik in front on HTTP/1). The migration adds
|
||||
an explicit HTTP/2 (h2c) listener on central (port `8083`, symmetric with sites; rig publishes
|
||||
it), `AddGrpc` + the PSK interceptor, and `CentralControlService`. Sites also get a **new**
|
||||
standing config `ScadaBridge:Communication:CentralGrpcEndpoints` — today the only central URL
|
||||
a site sees arrives *inside* `RefreshDeploymentCommand` (`CentralFetchBaseUrl`), so gRPC
|
||||
endpoint discovery genuinely needs new config.
|
||||
5. **PSK mechanics, concretized from the in-repo templates.** Server side copies
|
||||
`LocalDbSyncAuthInterceptor` exactly (service-prefix match, `authorization: Bearer`,
|
||||
`CryptographicOperations.FixedTimeEquals`, fail-closed on unset key, `PermissionDenied`);
|
||||
clients attach the key as call-level `Metadata` (the LocalDb sync client pattern). Key
|
||||
sourcing: **site nodes** read a static config key supplied as `${secret:SB-GRPC-PSK-<site>}`
|
||||
via the existing pre-host `SecretReferenceExpander` (`Program.cs:56-65`); **central** — whose
|
||||
site set is dynamic from the DB — resolves `SB-GRPC-PSK-{siteId}` at channel-build time via
|
||||
the runtime `ISecretResolver` (the MxGateway G-3 pattern), cached, fail-closed. Sites identify
|
||||
themselves to `CentralControlService` with an `x-scadabridge-site` metadata header so central
|
||||
verifies against that site's key. The rig mirrors the LocalDb dev pattern (same literal on
|
||||
both pair nodes; prod uses `${secret:}`).
|
||||
6. **Reuse map for §3.5's client policy:** per-site `GrpcNodeAAddress`/`GrpcNodeBAddress` columns
|
||||
already exist on the `Site` entity (streaming path reads them); primary/fallback single-retry
|
||||
exists in the pull clients; sticky flip + stability-window retry-budget exists in
|
||||
`DebugStreamBridgeActor`/`SiteAlarmAggregatorActor`. The plan extracts this into one shared
|
||||
`SitePairChannelProvider` rather than a fourth hand-rolled copy. **Caution — doc-vs-code gap:**
|
||||
`grpc_streams.md` names `CentralCommunicationActor.LoadSiteAddressesFromDb()` and
|
||||
singleton-aware node selection; neither exists in code (selection is blind A-then-B flip).
|
||||
7. **Deadlines are net-new.** No gRPC call in the repo sets a deadline or retry policy today; the
|
||||
new services introduce per-call deadlines mirroring the eight `CommunicationOptions` Ask
|
||||
timeouts (`DeploymentTimeout` 2 m … `HealthReportTimeout` 10 s), preserving the documented
|
||||
inner-before-outer timeout ordering (e.g. `SiteCallAuditOptions.RelayTimeout` 10 s <
|
||||
`QueryTimeout` 30 s).
|
||||
8. **Proto codegen is checked-in, not build-time** (protoc segfaults in the linux_arm64 build
|
||||
image): new protos follow the sitestream recipe — commented-out `<Protobuf>` item, generate on
|
||||
macOS, commit the generated C#, `docker/regen-proto.sh`.
|
||||
9. **Test assets to extend:** `Communication.Tests` already covers both communication actors with
|
||||
Akka TestKit (xunit **v2** — TestKit.Xunit2 compatible) + NSubstitute;
|
||||
`DirectActorSiteStreamAuditClient` is an explicitly-reusable wire-bypassing double for
|
||||
ingest-path integration tests.
|
||||
Reference in New Issue
Block a user