Joseph Doherty
3b7489a7b0
The other half of the rebuilt-peer story, and only reachable once 0.1.2 made back-fill work: with the wiped node repopulated and writing again, its writes never reached its peer. last_applied_remote_seq is a watermark in the PEER's seq space, and a rebuilt peer is a new database whose oplog numbers from 1. Two things then went wrong, and both had to be fixed — the first alone leaves data stuck: - The healthy node kept advertising the OLD peer's watermark, and the sending side seeds its pump from exactly that number, so the rebuilt node skipped its entire oplog. The watermark (and the observed peer clock) is now reset when the peer's node id changes. last_acked_seq is deliberately NOT reset: it describes our own oplog's pruned horizon and is what tells a rebuilt peer it needs a snapshot — clearing it would turn 0.1.2's back-fill back off. - A peer claiming to have applied more of our stream than we have ever produced can only be remembering a previous incarnation of us. That claim is now rejected in favour of starting from the beginning; LWW makes the re-send harmless. This mirrors the clamp RecordPeerAckAsync already applies to acks. Found on the OtOpcUa docker-dev rig, where the healthy node held watermark 10 while the rebuilt peer's oplog was seqs 1-3 and its last_acked stayed 0 — the peer never accepted a single one. Covered by a test that converges with one peer, replaces it with a fresh database, and requires a write on the new peer to arrive: RED before this fix, green after. 149/149 pass. Version 0.1.3. Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
scadaproj
Umbrella workspace for a family of related SCADA / OT (operational-technology) projects built around AVEVA System Platform (Wonderware), OPC UA, and Akka.NET.
This repository is part index — a map of the sister projects, which are their own repositories — and part home for cross-project normalization work and the shared code it produces.
Working in here with Claude Code? See
CLAUDE.mdfor the machine-oriented index and guidance. This README is the human-facing overview.
What's in here
| Path | What it is |
|---|---|
CLAUDE.md |
High-level index of the sister projects + working guidance |
components/ |
Component-normalization framework (per concern: target spec, current state, gaps) — 7 components |
docs/plans/ |
Implementation + design plans (incl. Galaxy Repository and LocalDb, which have no components/ entry) |
infra/glauth/ |
The shared GLAuth every app's dev/test LDAP points at (10.100.0.35:3893) |
ZB.MOM.WW.Auth/ |
Built shared library (.NET 10) — login / identity / API keys |
ZB.MOM.WW.Theme/ |
Built shared RCL (.NET 10) — Technical-Light design system |
ZB.MOM.WW.Audit/ |
Built shared library — canonical audit event + writer seam |
ZB.MOM.WW.Health/ |
Built shared library — readiness / liveness / active-node probes |
ZB.MOM.WW.Telemetry/ |
Built shared library — OTel metrics / traces + Serilog |
ZB.MOM.WW.Configuration/ |
Built shared library — options validation + startup preflight |
ZB.MOM.WW.Secrets/ |
Built shared library — AES-256-GCM secret store + ${secret:} resolution + CLI + SQL-Server / Akka cluster replication |
ZB.MOM.WW.GalaxyRepository/ |
Built shared library — Galaxy object-hierarchy SQL browse + gRPC service |
ZB.MOM.WW.LocalDb/ |
Built shared library — embedded SQLite cache + 2-node gRPC replication |
The sister projects
Independent repositories, coupled only at runtime over wire protocols (gRPC + OPC UA) — not by a shared build. scadaproj indexes them; it does not contain them.
- OtOpcUa (
lmxopcua) — OPC UA server (.NET 10) that republishes Wonderware Galaxy tags as an OPC UA address space. - MxAccessGateway (
mxaccessgw) — gRPC gateway giving modern x64 / .NET-10 clients full MXAccess parity without loading 32-bit COM. The linchpin both others depend on. - ScadaBridge — distributed SCADA platform on Akka.NET (hub-and-spoke: one central cluster
- N site clusters); its Data Connection Layer ingests OPC UA and MxAccess.
- HistorianGateway (
historiangw) — gRPC sidecar over the AVEVA Historian (read/write) plus read-only Galaxy browse; no COM, no x86 worker. OtOpcUa's sole historian backend.
Data flow:
Wonderware Galaxy ──gRPC──▶ mxaccessgw ──▶ OtOpcUa (republishes as OPC UA) ─┐
└────────────────────────────────────┴─▶ ScadaBridge clusters
(or ScadaBridge's MxGateway adapter, direct)
Full relationship map: CLAUDE.md.
Component normalization
The sister repos kept re-implementing the same cross-cutting concerns and drifting apart.
components/ captures, per component, the one target spec, each project's
code-verified current state, and a gaps / adoption backlog. Convention + workflow in
components/README.md.
Status verified 2026-07-18 against the feed + consumer branches (see Roadmap). "All four" = OtOpcUa, MxAccessGateway, ScadaBridge, HistorianGateway.
| Component | Feed | Adoption | Folder |
|---|---|---|---|
| Auth (login / identity / authz) | 0.1.0–0.1.5 |
mxaccessgw / ScadaBridge / HistorianGateway @ 0.1.5; OtOpcUa @ 0.1.1 |
components/auth/ |
| UI Theme (layout / tokens / components) | 0.2.0–0.3.1 |
all four @ 0.3.1 |
components/ui-theme/ |
| Audit (event model + writer seam) | 0.1.0 |
all four (DEEP) | components/audit/ |
| Config + validation | 0.1.0 |
all four | components/configuration/ |
| Observability (metrics / traces / logs) | 0.1.0 |
all four | components/observability/ |
| Health (readiness / liveness / active-node) | 0.1.0 |
all four | components/health/ |
Secrets (encrypted store + ${secret:}) |
0.1.0–0.2.0 |
all four @ 0.1.2 (0.1.3/0.2.0 published, not yet consumed) |
components/secrets/ |
| Galaxy Repository (SQL browse + gRPC) | 0.1.0, 0.2.0 |
HistorianGateway + mxaccessgw | (design in docs/plans/) |
| LocalDb (embedded cache + 2-node sync) | 0.1.0 |
none yet | (design in docs/plans/) |
ZB.MOM.WW.Theme — the shared UI kit
The UI-theme component, realized as a single-package .NET 10 Razor Class Library that replaced
each app's copy-pasted Technical-Light design system. Adopted by all four apps, currently on
0.3.1 (feed carries 0.2.0–0.3.1); adoption history in
components/ui-theme/GAPS.md.
| Asset / Component | Purpose | Used by |
|---|---|---|
_content/ZB.MOM.WW.Theme/css/theme.css |
Design tokens, IBM Plex typography, Bootstrap overrides | all |
_content/ZB.MOM.WW.Theme/css/layout.css |
Side-rail shell layout, nav CSS, chip/card helpers | all |
_content/ZB.MOM.WW.Theme/fonts/*.woff2 |
IBM Plex Sans 400/600 + Mono 500, vendored | all |
ThemeHead, ThemeShell, BrandBar |
Shell entry point and chassis components | all |
NavRailItem, NavRailSection |
Rail nav components | all |
StatusPill (StatusState) |
Inline status chip — replaces per-app StatusBadge |
all |
LoginCard |
Static form-POST sign-in card | OtOpcUa, ScadaBridge (MxGateway when login page added) |
TechButton, TechCard, TechField |
Common controls (Bootstrap 5 wrappers) | all |
Consumer matrix: all four apps consume the single ZB.MOM.WW.Theme package —
OtOpcUa AdminUI, MxAccessGateway Server, ScadaBridge Host + CentralUI, HistorianGateway Server.
Build & test
cd ZB.MOM.WW.Theme
dotnet build -c Release # 0 warnings (TreatWarningsAsErrors)
dotnet test # 48 bUnit tests (verified 2026-07-18)
./build/pack.sh # → ./artifacts/ZB.MOM.WW.Theme.<version>.nupkg (feed is at 0.3.1)
Stack: .NET 10 · Razor Class Library · bUnit · xUnit · central package management.
More detail: ZB.MOM.WW.Theme/README.md.
ZB.MOM.WW.Auth — the shared library
The auth component, realized as a four-package .NET 10 library that replaced each app's
re-implemented authentication. Adopted by all four apps — pins drift deliberately
(OtOpcUa 0.1.1, ScadaBridge 0.1.3, MxAccessGateway + HistorianGateway 0.1.4); history in
components/auth/GAPS.md.
| Package | Purpose | Used by |
|---|---|---|
ZB.MOM.WW.Auth.Abstractions |
Contracts, CanonicalRole, API-key types (zero deps) |
all |
ZB.MOM.WW.Auth.Ldap |
Bind-then-search LDAP authentication (fail-closed) | all |
ZB.MOM.WW.Auth.ApiKeys |
Peppered-HMAC API keys + SQLite store + verifier + admin commands | MxAccessGateway, ScadaBridge |
ZB.MOM.WW.Auth.AspNetCore |
Canonical claim types, cookie defaults, LDAP DI helpers | all (web UIs) |
Consumer matrix: OtOpcUa → Abstractions + Ldap + AspNetCore (no ApiKeys / SQLite);
MxAccessGateway, ScadaBridge & HistorianGateway → all four packages.
Identity (LDAP / GLAuth, the canonical role set) is normalized; each app keeps its own
authorization enforcement (OPC-UA permissions / gRPC scopes / roles + site-scoping) and maps
it onto the canonical roles. Design: components/auth/spec/SPEC.md
and CANONICAL-ROLES.md.
Build & test
cd ZB.MOM.WW.Auth
dotnet build -c Release # 0 warnings
dotnet test # 215 passing, 1 skipped (opt-in LDAP integration test) — verified 2026-07-18
dotnet pack -c Release -o ./artifacts # → 4 nupkgs (feed is at 0.1.4)
Stack: .NET 10 · xUnit · Novell.Directory.Ldap.NETStandard · Microsoft.Data.Sqlite · central
package management. The packages are libraries linked into each consumer — there is no
central auth service. More detail: ZB.MOM.WW.Auth/README.md.
The optional LDAP integration test is opt-in:
ZB_LDAP_IT=1 dotnet test # requires a reachable GLAuth (e.g. a sister repo's infra/glauth)
Roadmap
Verified 2026-07-18 against the Gitea feed (
nuget/query?q=ZB.MOM.WW) and the actualPackageReferences on each consumer's default branch — not from status claims in docs. Per-component detail in eachcomponents/<name>/GAPS.md.
Shipped
- ✅ Auth normalized +
ZB.MOM.WW.Authbuilt (feed:0.1.0–0.1.5) and adopted by all four apps.0.1.5(2026-07-18) pins patched SQLitePCLRaw; mxaccessgw, ScadaBridge and HistorianGateway bumped to it. OtOpcUa stays at0.1.1— it takes noApiKeys/SQLite dependency, so it was never exposed. - ✅ UI Theme normalized +
ZB.MOM.WW.Themebuilt (feed:0.2.0–0.3.1); all four apps on0.3.1. - ✅ Audit —
ZB.MOM.WW.Audit0.1.0on the feed; adopted by all four (DEEP: canonical record everywhere). - ✅ Configuration —
ZB.MOM.WW.Configuration0.1.0on the feed; adopted by all four. - ✅ Observability —
ZB.MOM.WW.Telemetry+.Serilog0.1.0on the feed; adopted by all four. - ✅ Health —
ZB.MOM.WW.Health(+.Akka,.EntityFrameworkCore)0.1.0on the feed; adopted by all four (MapZbHealthwired in each; previously documented as "not yet adopted" — that was stale). - ✅ Galaxy Repository —
ZB.MOM.WW.GalaxyRepository0.2.0on the feed; consumed by HistorianGateway and mxaccessgw (which wiresAddZbGalaxyRepositoryin its Server; previously documented as a pending follow-on — also stale). - ✅ Secrets —
ZB.MOM.WW.Secrets(+.Abstractions,.Ui) published through0.1.3(KEK rotation, 2026-07-18); all four apps still pinned at0.1.2. - ✅ Secrets G-7 clustered replication — built 2026-07-18 at
0.2.0, both fork options:ZB.MOM.WW.Secrets.Replicator.SqlServer(shared store and local-plus-hub) andZB.MOM.WW.Secrets.Replicator.AkkaDotNet(peer-to-peer, partition-tolerant). Verified against a real SQL Server 2022 and an in-process 2-node Akka cluster. Published to the feed 2026-07-18 (restore-verified). - ✅ LocalDb —
ZB.MOM.WW.LocalDb(+.Contracts,.Replication) built at0.1.0and published to the feed 2026-07-18 (restore-verified from a scratch consumer).
Open
- ⬜ Adopt a clustered secrets topology in ScadaBridge / OtOpcUa — config + a shared KEK, no code
change.
0.2.0is on the feed.components/secrets/GAPS.md. - ⬜ Bump the four Secrets consumers
0.1.2→0.1.3/0.2.0(KEK rotation + the same security fix). - ⬜ Clear the pre-existing
AngleSharpNU1902 build break in ScadaBridgeCentralUI.Testsand HistorianGatewayTests— both fail full-solution builds underTreatWarningsAsErrors. - ⬜ Adopt
ZB.MOM.WW.LocalDbin an app — no consumer references it yet. - ⬜ Give LocalDb a
components/localdb/entry; it is the one shared lib whose design lives only indocs/plans/2026-07-17-localdb-design.md. - ⬜ Converge the Auth version pins (OtOpcUa is three minors behind at
0.1.1). - ⬜ Normalize the next cross-cutting component.