Joseph Doherty
522cfbfda6
Two adversarial-review findings against 0.2.0. Wire-compatible with 0.1.x/0.2.0 peers: no proto change, no lib_schema_version bump, no on-disk schema change. F4 — DeregisterReplicated destroyed the HLC clock's only durable crash anchor. Clock recovery on open is MAX(hlc_high_water, MAX(oplog.hlc), MAX(row_version.hlc), last_seen_hlc); hlc_high_water is written only by Dispose, and deregistration deletes the table's rows from BOTH ledgers. On a file whose registered tables have all been deregistered, an ungraceful exit (crash, SIGKILL, power loss) therefore left nothing above zero to recover from: the reopened clock restarts from wall-clock time and can re-issue stamps below HLCs peers already hold, which LWW discards as older. The current clock is now flushed inside the deregistration transaction, before the prune, as a separate command so it cannot inflate the "did anything change?" answer. FlushHlcHighWater takes an optional transaction (SQLite rejects a command with no transaction on a connection that has one) and is now MAX-guarded — every writer of that column is flushing a monotonic clock, so a lower value can only be stale. F3 — the sync inbox was Channel.CreateUnbounded, making inbound memory a function of what the PEER sends. The bad case is not hypothetical: a snapshot streams every row of every registered table, needs_snapshot is per-database, and both sides of a pair that enables replication late baseline unconditionally — so a mutual full snapshot of a large dataset (an event log near its storage cap) was buffered whole, in memory, on both nodes at once. The inbox is now bounded at 64 messages with wait-mode backpressure, so the ceiling is ~64 x MaxBatchBytes — a configured number rather than a peer-controlled one — and a full inbox stops draining the gRPC stream, which closes the peer's flow-control window. The pump writes with WriteAsync, not TryWrite: on a full bounded channel TryWrite drops the message, and every inbound message is protocol state. Bounding alone would have deadlocked the very case it was aimed at. The snapshot a node owes was streamed from INSIDE the handshake, before the receive loop existed, so neither side drained its inbox while sending; a mutual snapshot converged only because the inbox was unbounded. The snapshot now runs as the first phase of the outbound task, started alongside the receive loop, so the inbox is drained for the whole life of the session. Interleaving needed no new safety: snapshot rows and deltas both merge through LWW and the snapshot's watermark write is MAX-guarded. Ordering within the outbound lane is preserved (the pump still starts only after the snapshot, from its as-of seq). Per-table needs_snapshot (the finding's optional third part) is NOT done: the flag lives in __localdb_peer_state and narrowing it needs an on-disk schema change, which this fix deliberately avoids. Documented as a follow-up, and the cost is now written down where it is decided — RegisterReplicated(baselineExistingRows) remarks and the README both say that baselining one table re-streams every registered table in both directions, and that high-volume tables should be trimmed before enabling replication late. 188/188 tests pass (was 184). New coverage: the deregistered clock anchor surviving an ungraceful reopen (a second instance over the same file with no Dispose — fails without the flush, recovering at 0), the flush not counting as work done, bounded-inbox backpressure under a slow reader (the pump stalls at the bound and loses nothing, proven by contiguous in-order delivery), and a mutual snapshot over a capacity-1 transport converging — that last one deadlocks and times out against the old handshake-phase snapshot ordering. Version 0.2.1.
scadaproj
Umbrella workspace for a family of related SCADA / OT (operational-technology) projects built around AVEVA System Platform (Wonderware), OPC UA, and Akka.NET.
This repository is part index — a map of the sister projects, which are their own repositories — and part home for cross-project normalization work and the shared code it produces.
Working in here with Claude Code? See
CLAUDE.mdfor the machine-oriented index and guidance. This README is the human-facing overview.
What's in here
| Path | What it is |
|---|---|
CLAUDE.md |
High-level index of the sister projects + working guidance |
components/ |
Component-normalization framework (per concern: target spec, current state, gaps) — 7 components |
docs/plans/ |
Implementation + design plans (incl. Galaxy Repository and LocalDb, which have no components/ entry) |
infra/glauth/ |
The shared GLAuth every app's dev/test LDAP points at (10.100.0.35:3893) |
ZB.MOM.WW.Auth/ |
Built shared library (.NET 10) — login / identity / API keys |
ZB.MOM.WW.Theme/ |
Built shared RCL (.NET 10) — Technical-Light design system |
ZB.MOM.WW.Audit/ |
Built shared library — canonical audit event + writer seam |
ZB.MOM.WW.Health/ |
Built shared library — readiness / liveness / active-node probes |
ZB.MOM.WW.Telemetry/ |
Built shared library — OTel metrics / traces + Serilog |
ZB.MOM.WW.Configuration/ |
Built shared library — options validation + startup preflight |
ZB.MOM.WW.Secrets/ |
Built shared library — AES-256-GCM secret store + ${secret:} resolution + CLI + SQL-Server / Akka cluster replication |
ZB.MOM.WW.GalaxyRepository/ |
Built shared library — Galaxy object-hierarchy SQL browse + gRPC service |
ZB.MOM.WW.LocalDb/ |
Built shared library — embedded SQLite cache + 2-node gRPC replication |
The sister projects
Independent repositories, coupled only at runtime over wire protocols (gRPC + OPC UA) — not by a shared build. scadaproj indexes them; it does not contain them.
- OtOpcUa (
lmxopcua) — OPC UA server (.NET 10) that republishes Wonderware Galaxy tags as an OPC UA address space. - MxAccessGateway (
mxaccessgw) — gRPC gateway giving modern x64 / .NET-10 clients full MXAccess parity without loading 32-bit COM. The linchpin both others depend on. - ScadaBridge — distributed SCADA platform on Akka.NET (hub-and-spoke: one central cluster
- N site clusters); its Data Connection Layer ingests OPC UA and MxAccess.
- HistorianGateway (
historiangw) — gRPC sidecar over the AVEVA Historian (read/write) plus read-only Galaxy browse; no COM, no x86 worker. OtOpcUa's sole historian backend.
Data flow:
Wonderware Galaxy ──gRPC──▶ mxaccessgw ──▶ OtOpcUa (republishes as OPC UA) ─┐
└────────────────────────────────────┴─▶ ScadaBridge clusters
(or ScadaBridge's MxGateway adapter, direct)
Full relationship map: CLAUDE.md.
Component normalization
The sister repos kept re-implementing the same cross-cutting concerns and drifting apart.
components/ captures, per component, the one target spec, each project's
code-verified current state, and a gaps / adoption backlog. Convention + workflow in
components/README.md.
Status verified 2026-07-18 against the feed + consumer branches (see Roadmap). "All four" = OtOpcUa, MxAccessGateway, ScadaBridge, HistorianGateway.
| Component | Feed | Adoption | Folder |
|---|---|---|---|
| Auth (login / identity / authz) | 0.1.0–0.1.5 |
mxaccessgw / ScadaBridge / HistorianGateway @ 0.1.5; OtOpcUa @ 0.1.1 |
components/auth/ |
| UI Theme (layout / tokens / components) | 0.2.0–0.3.1 |
all four @ 0.3.1 |
components/ui-theme/ |
| Audit (event model + writer seam) | 0.1.0 |
all four (DEEP) | components/audit/ |
| Config + validation | 0.1.0 |
all four | components/configuration/ |
| Observability (metrics / traces / logs) | 0.1.0 |
all four | components/observability/ |
| Health (readiness / liveness / active-node) | 0.1.0 |
all four | components/health/ |
Secrets (encrypted store + ${secret:}) |
0.1.0–0.2.0 |
all four @ 0.1.2 (0.1.3/0.2.0 published, not yet consumed) |
components/secrets/ |
| Galaxy Repository (SQL browse + gRPC) | 0.1.0, 0.2.0 |
HistorianGateway + mxaccessgw | (design in docs/plans/) |
| LocalDb (embedded cache + 2-node sync) | 0.1.0 |
none yet | (design in docs/plans/) |
ZB.MOM.WW.Theme — the shared UI kit
The UI-theme component, realized as a single-package .NET 10 Razor Class Library that replaced
each app's copy-pasted Technical-Light design system. Adopted by all four apps, currently on
0.3.1 (feed carries 0.2.0–0.3.1); adoption history in
components/ui-theme/GAPS.md.
| Asset / Component | Purpose | Used by |
|---|---|---|
_content/ZB.MOM.WW.Theme/css/theme.css |
Design tokens, IBM Plex typography, Bootstrap overrides | all |
_content/ZB.MOM.WW.Theme/css/layout.css |
Side-rail shell layout, nav CSS, chip/card helpers | all |
_content/ZB.MOM.WW.Theme/fonts/*.woff2 |
IBM Plex Sans 400/600 + Mono 500, vendored | all |
ThemeHead, ThemeShell, BrandBar |
Shell entry point and chassis components | all |
NavRailItem, NavRailSection |
Rail nav components | all |
StatusPill (StatusState) |
Inline status chip — replaces per-app StatusBadge |
all |
LoginCard |
Static form-POST sign-in card | OtOpcUa, ScadaBridge (MxGateway when login page added) |
TechButton, TechCard, TechField |
Common controls (Bootstrap 5 wrappers) | all |
Consumer matrix: all four apps consume the single ZB.MOM.WW.Theme package —
OtOpcUa AdminUI, MxAccessGateway Server, ScadaBridge Host + CentralUI, HistorianGateway Server.
Build & test
cd ZB.MOM.WW.Theme
dotnet build -c Release # 0 warnings (TreatWarningsAsErrors)
dotnet test # 48 bUnit tests (verified 2026-07-18)
./build/pack.sh # → ./artifacts/ZB.MOM.WW.Theme.<version>.nupkg (feed is at 0.3.1)
Stack: .NET 10 · Razor Class Library · bUnit · xUnit · central package management.
More detail: ZB.MOM.WW.Theme/README.md.
ZB.MOM.WW.Auth — the shared library
The auth component, realized as a four-package .NET 10 library that replaced each app's
re-implemented authentication. Adopted by all four apps — pins drift deliberately
(OtOpcUa 0.1.1, ScadaBridge 0.1.3, MxAccessGateway + HistorianGateway 0.1.4); history in
components/auth/GAPS.md.
| Package | Purpose | Used by |
|---|---|---|
ZB.MOM.WW.Auth.Abstractions |
Contracts, CanonicalRole, API-key types (zero deps) |
all |
ZB.MOM.WW.Auth.Ldap |
Bind-then-search LDAP authentication (fail-closed) | all |
ZB.MOM.WW.Auth.ApiKeys |
Peppered-HMAC API keys + SQLite store + verifier + admin commands | MxAccessGateway, ScadaBridge |
ZB.MOM.WW.Auth.AspNetCore |
Canonical claim types, cookie defaults, LDAP DI helpers | all (web UIs) |
Consumer matrix: OtOpcUa → Abstractions + Ldap + AspNetCore (no ApiKeys / SQLite);
MxAccessGateway, ScadaBridge & HistorianGateway → all four packages.
Identity (LDAP / GLAuth, the canonical role set) is normalized; each app keeps its own
authorization enforcement (OPC-UA permissions / gRPC scopes / roles + site-scoping) and maps
it onto the canonical roles. Design: components/auth/spec/SPEC.md
and CANONICAL-ROLES.md.
Build & test
cd ZB.MOM.WW.Auth
dotnet build -c Release # 0 warnings
dotnet test # 215 passing, 1 skipped (opt-in LDAP integration test) — verified 2026-07-18
dotnet pack -c Release -o ./artifacts # → 4 nupkgs (feed is at 0.1.4)
Stack: .NET 10 · xUnit · Novell.Directory.Ldap.NETStandard · Microsoft.Data.Sqlite · central
package management. The packages are libraries linked into each consumer — there is no
central auth service. More detail: ZB.MOM.WW.Auth/README.md.
The optional LDAP integration test is opt-in:
ZB_LDAP_IT=1 dotnet test # requires a reachable GLAuth (e.g. a sister repo's infra/glauth)
Roadmap
Verified 2026-07-18 against the Gitea feed (
nuget/query?q=ZB.MOM.WW) and the actualPackageReferences on each consumer's default branch — not from status claims in docs. Per-component detail in eachcomponents/<name>/GAPS.md.
Shipped
- ✅ Auth normalized +
ZB.MOM.WW.Authbuilt (feed:0.1.0–0.1.5) and adopted by all four apps.0.1.5(2026-07-18) pins patched SQLitePCLRaw; mxaccessgw, ScadaBridge and HistorianGateway bumped to it. OtOpcUa stays at0.1.1— it takes noApiKeys/SQLite dependency, so it was never exposed. - ✅ UI Theme normalized +
ZB.MOM.WW.Themebuilt (feed:0.2.0–0.3.1); all four apps on0.3.1. - ✅ Audit —
ZB.MOM.WW.Audit0.1.0on the feed; adopted by all four (DEEP: canonical record everywhere). - ✅ Configuration —
ZB.MOM.WW.Configuration0.1.0on the feed; adopted by all four. - ✅ Observability —
ZB.MOM.WW.Telemetry+.Serilog0.1.0on the feed; adopted by all four. - ✅ Health —
ZB.MOM.WW.Health(+.Akka,.EntityFrameworkCore)0.1.0on the feed; adopted by all four (MapZbHealthwired in each; previously documented as "not yet adopted" — that was stale). - ✅ Galaxy Repository —
ZB.MOM.WW.GalaxyRepository0.2.0on the feed; consumed by HistorianGateway and mxaccessgw (which wiresAddZbGalaxyRepositoryin its Server; previously documented as a pending follow-on — also stale). - ✅ Secrets —
ZB.MOM.WW.Secrets(+.Abstractions,.Ui) published through0.1.3(KEK rotation, 2026-07-18); all four apps still pinned at0.1.2. - ✅ Secrets G-7 clustered replication — built 2026-07-18 at
0.2.0, both fork options:ZB.MOM.WW.Secrets.Replicator.SqlServer(shared store and local-plus-hub) andZB.MOM.WW.Secrets.Replicator.AkkaDotNet(peer-to-peer, partition-tolerant). Verified against a real SQL Server 2022 and an in-process 2-node Akka cluster. Published to the feed 2026-07-18 (restore-verified). - ✅ LocalDb —
ZB.MOM.WW.LocalDb(+.Contracts,.Replication) built at0.1.0and published to the feed 2026-07-18 (restore-verified from a scratch consumer).
Open
- ⬜ Adopt a clustered secrets topology in ScadaBridge / OtOpcUa — config + a shared KEK, no code
change.
0.2.0is on the feed.components/secrets/GAPS.md. - ⬜ Bump the four Secrets consumers
0.1.2→0.1.3/0.2.0(KEK rotation + the same security fix). - ⬜ Clear the pre-existing
AngleSharpNU1902 build break in ScadaBridgeCentralUI.Testsand HistorianGatewayTests— both fail full-solution builds underTreatWarningsAsErrors. - ⬜ Adopt
ZB.MOM.WW.LocalDbin an app — no consumer references it yet. - ⬜ Give LocalDb a
components/localdb/entry; it is the one shared lib whose design lives only indocs/plans/2026-07-17-localdb-design.md. - ⬜ Converge the Auth version pins (OtOpcUa is three minors behind at
0.1.1). - ⬜ Normalize the next cross-cutting component.