Joseph Doherty
9377fa1678
WP3.3 of the ScadaBridge architectural-review remediation. Six changes, all wire-compatible with 0.1.x peers — no proto change, no lib_schema_version bump. The on-disk bookkeeping schema goes to v2 and is upgraded in place when a file is opened. DeregisterReplicated(table). There was no way to stop capturing once a file had been registered, so a node that does not replicate kept paying three triggers inside every write transaction, appending to an oplog nothing would ever drain. Drops the triggers and prunes the table's oplog + row-version rows in one transaction; idempotent, and deliberately usable on a table this process never registered, since stale triggers left by an EARLIER build are the primary case. Deregistration must be symmetric across a pair, and already is enforced: the handshake compares registered-table digests fail-closed, so a node that drops a table its peer still replicates stops syncing loudly rather than diverging. RegisterReplicated(table, baselineExistingRows: true). Closes the late-opt-in hole: the snapshot streamer pages from __localdb_row_version, which only triggers populate, so a node that started replicating LATE never sent its pre-existing rows — silently, forever. Seeding alone would not have fixed it, because a node whose oplog starts at seq 1 presents no gap at the handshake and is told it needs no snapshot; the seed therefore also flags needs_snapshot, and the existing resync machinery carries the rows. The floor is HLC 0 stamped with this node's id: every genuine HLC is a UTC millisecond shifted left 16 bits and so is strictly greater, which makes a baselined row lose every comparison against a real remote write and win only where the peer has no version of that key at all. Two nodes baselining the same key both hold 0 and resolve by the node-id tie-break — convergent, and nothing better is available for two rows of unordered antiquity. Opt-in, so the documented CDC contract is unchanged unless asked. Pruning the ledger on deregister is safe precisely because re-registration can re-baseline; the two APIs are a pair. O(1) backlog depth. GetOplogDepth ran COUNT(*) over the entire unacked backlog, once a minute from maintenance AND on every metrics scrape — a full scan of a million rows exactly when a node is already behind. Now MAX(seq) - MAX(last_acked_seq, MIN(seq) - 1), floored at 0: three b-tree seeks, measured at 58 VM steps against 100,020 for the count on a 50k backlog. Exact, because seq is AUTOINCREMENT and every prune path removes a prefix (ack pruning by watermark, the row cap by count, the age cap by an HLC cutoff that rises with seq). The MIN clamp is what keeps it exact after a cap prune has moved the horizon above a stalled watermark — without it the caps would re-flag on every tick forever. Byte-budget batching. MaxBatchSize was the only cap on an outbound message, which made it a proxy for a byte budget that consumers had to tune against their widest row: ScadaBridge pinned it to 16 because 70 KB config_json times the 500 default is ~35 MB against a 4 MB gRPC limit. MaxBatchBytes (default 2 MB) now bounds delta and snapshot messages by summed serialized size, with the row count demoted to a secondary cap. A row larger than the budget is sent alone rather than stalling the stream on it. Since more, smaller messages would otherwise mean more apply transactions and more acks, inbound delta batches and acks ALREADY queued are coalesced — one transaction and one ack per burst, bounded by the sender's own caps, and a no-op under a trickle. LwwApplier prepares each statement once. It went through the generic transaction façade, which builds a fresh SqliteCommand per call: three SQL parses per applied row, on the path that drains a reconnect backlog. Statements are now held for the batch and rebound. Dropped __localdb_oplog_hlc. It indexed (table_name, pk_json, hlc), which no query has ever used — every oplog access path is by seq or a full hlc-predicate delete — while costing a b-tree insert inside every captured consumer write. 184/184 tests pass (was 149). New coverage: schema upgrade drops the index and preserves identity/watermarks/data, deregistration idempotency and trigger absence and cross-process cleanup, baseline seeding at the floor with LWW safety proved through the real applier, depth asserted equal to the counting query across every prune scenario plus a constant-time assertion over a 200k backlog, byte-budget chunking including the one-giant-row edge on both the delta and snapshot paths, prepare-once statement counting, and coalescing that must not swallow the message that ends its run. Version 0.2.0.
scadaproj
Umbrella workspace for a family of related SCADA / OT (operational-technology) projects built around AVEVA System Platform (Wonderware), OPC UA, and Akka.NET.
This repository is part index — a map of the sister projects, which are their own repositories — and part home for cross-project normalization work and the shared code it produces.
Working in here with Claude Code? See
CLAUDE.mdfor the machine-oriented index and guidance. This README is the human-facing overview.
What's in here
| Path | What it is |
|---|---|
CLAUDE.md |
High-level index of the sister projects + working guidance |
components/ |
Component-normalization framework (per concern: target spec, current state, gaps) — 7 components |
docs/plans/ |
Implementation + design plans (incl. Galaxy Repository and LocalDb, which have no components/ entry) |
infra/glauth/ |
The shared GLAuth every app's dev/test LDAP points at (10.100.0.35:3893) |
ZB.MOM.WW.Auth/ |
Built shared library (.NET 10) — login / identity / API keys |
ZB.MOM.WW.Theme/ |
Built shared RCL (.NET 10) — Technical-Light design system |
ZB.MOM.WW.Audit/ |
Built shared library — canonical audit event + writer seam |
ZB.MOM.WW.Health/ |
Built shared library — readiness / liveness / active-node probes |
ZB.MOM.WW.Telemetry/ |
Built shared library — OTel metrics / traces + Serilog |
ZB.MOM.WW.Configuration/ |
Built shared library — options validation + startup preflight |
ZB.MOM.WW.Secrets/ |
Built shared library — AES-256-GCM secret store + ${secret:} resolution + CLI + SQL-Server / Akka cluster replication |
ZB.MOM.WW.GalaxyRepository/ |
Built shared library — Galaxy object-hierarchy SQL browse + gRPC service |
ZB.MOM.WW.LocalDb/ |
Built shared library — embedded SQLite cache + 2-node gRPC replication |
The sister projects
Independent repositories, coupled only at runtime over wire protocols (gRPC + OPC UA) — not by a shared build. scadaproj indexes them; it does not contain them.
- OtOpcUa (
lmxopcua) — OPC UA server (.NET 10) that republishes Wonderware Galaxy tags as an OPC UA address space. - MxAccessGateway (
mxaccessgw) — gRPC gateway giving modern x64 / .NET-10 clients full MXAccess parity without loading 32-bit COM. The linchpin both others depend on. - ScadaBridge — distributed SCADA platform on Akka.NET (hub-and-spoke: one central cluster
- N site clusters); its Data Connection Layer ingests OPC UA and MxAccess.
- HistorianGateway (
historiangw) — gRPC sidecar over the AVEVA Historian (read/write) plus read-only Galaxy browse; no COM, no x86 worker. OtOpcUa's sole historian backend.
Data flow:
Wonderware Galaxy ──gRPC──▶ mxaccessgw ──▶ OtOpcUa (republishes as OPC UA) ─┐
└────────────────────────────────────┴─▶ ScadaBridge clusters
(or ScadaBridge's MxGateway adapter, direct)
Full relationship map: CLAUDE.md.
Component normalization
The sister repos kept re-implementing the same cross-cutting concerns and drifting apart.
components/ captures, per component, the one target spec, each project's
code-verified current state, and a gaps / adoption backlog. Convention + workflow in
components/README.md.
Status verified 2026-07-18 against the feed + consumer branches (see Roadmap). "All four" = OtOpcUa, MxAccessGateway, ScadaBridge, HistorianGateway.
| Component | Feed | Adoption | Folder |
|---|---|---|---|
| Auth (login / identity / authz) | 0.1.0–0.1.5 |
mxaccessgw / ScadaBridge / HistorianGateway @ 0.1.5; OtOpcUa @ 0.1.1 |
components/auth/ |
| UI Theme (layout / tokens / components) | 0.2.0–0.3.1 |
all four @ 0.3.1 |
components/ui-theme/ |
| Audit (event model + writer seam) | 0.1.0 |
all four (DEEP) | components/audit/ |
| Config + validation | 0.1.0 |
all four | components/configuration/ |
| Observability (metrics / traces / logs) | 0.1.0 |
all four | components/observability/ |
| Health (readiness / liveness / active-node) | 0.1.0 |
all four | components/health/ |
Secrets (encrypted store + ${secret:}) |
0.1.0–0.2.0 |
all four @ 0.1.2 (0.1.3/0.2.0 published, not yet consumed) |
components/secrets/ |
| Galaxy Repository (SQL browse + gRPC) | 0.1.0, 0.2.0 |
HistorianGateway + mxaccessgw | (design in docs/plans/) |
| LocalDb (embedded cache + 2-node sync) | 0.1.0 |
none yet | (design in docs/plans/) |
ZB.MOM.WW.Theme — the shared UI kit
The UI-theme component, realized as a single-package .NET 10 Razor Class Library that replaced
each app's copy-pasted Technical-Light design system. Adopted by all four apps, currently on
0.3.1 (feed carries 0.2.0–0.3.1); adoption history in
components/ui-theme/GAPS.md.
| Asset / Component | Purpose | Used by |
|---|---|---|
_content/ZB.MOM.WW.Theme/css/theme.css |
Design tokens, IBM Plex typography, Bootstrap overrides | all |
_content/ZB.MOM.WW.Theme/css/layout.css |
Side-rail shell layout, nav CSS, chip/card helpers | all |
_content/ZB.MOM.WW.Theme/fonts/*.woff2 |
IBM Plex Sans 400/600 + Mono 500, vendored | all |
ThemeHead, ThemeShell, BrandBar |
Shell entry point and chassis components | all |
NavRailItem, NavRailSection |
Rail nav components | all |
StatusPill (StatusState) |
Inline status chip — replaces per-app StatusBadge |
all |
LoginCard |
Static form-POST sign-in card | OtOpcUa, ScadaBridge (MxGateway when login page added) |
TechButton, TechCard, TechField |
Common controls (Bootstrap 5 wrappers) | all |
Consumer matrix: all four apps consume the single ZB.MOM.WW.Theme package —
OtOpcUa AdminUI, MxAccessGateway Server, ScadaBridge Host + CentralUI, HistorianGateway Server.
Build & test
cd ZB.MOM.WW.Theme
dotnet build -c Release # 0 warnings (TreatWarningsAsErrors)
dotnet test # 48 bUnit tests (verified 2026-07-18)
./build/pack.sh # → ./artifacts/ZB.MOM.WW.Theme.<version>.nupkg (feed is at 0.3.1)
Stack: .NET 10 · Razor Class Library · bUnit · xUnit · central package management.
More detail: ZB.MOM.WW.Theme/README.md.
ZB.MOM.WW.Auth — the shared library
The auth component, realized as a four-package .NET 10 library that replaced each app's
re-implemented authentication. Adopted by all four apps — pins drift deliberately
(OtOpcUa 0.1.1, ScadaBridge 0.1.3, MxAccessGateway + HistorianGateway 0.1.4); history in
components/auth/GAPS.md.
| Package | Purpose | Used by |
|---|---|---|
ZB.MOM.WW.Auth.Abstractions |
Contracts, CanonicalRole, API-key types (zero deps) |
all |
ZB.MOM.WW.Auth.Ldap |
Bind-then-search LDAP authentication (fail-closed) | all |
ZB.MOM.WW.Auth.ApiKeys |
Peppered-HMAC API keys + SQLite store + verifier + admin commands | MxAccessGateway, ScadaBridge |
ZB.MOM.WW.Auth.AspNetCore |
Canonical claim types, cookie defaults, LDAP DI helpers | all (web UIs) |
Consumer matrix: OtOpcUa → Abstractions + Ldap + AspNetCore (no ApiKeys / SQLite);
MxAccessGateway, ScadaBridge & HistorianGateway → all four packages.
Identity (LDAP / GLAuth, the canonical role set) is normalized; each app keeps its own
authorization enforcement (OPC-UA permissions / gRPC scopes / roles + site-scoping) and maps
it onto the canonical roles. Design: components/auth/spec/SPEC.md
and CANONICAL-ROLES.md.
Build & test
cd ZB.MOM.WW.Auth
dotnet build -c Release # 0 warnings
dotnet test # 215 passing, 1 skipped (opt-in LDAP integration test) — verified 2026-07-18
dotnet pack -c Release -o ./artifacts # → 4 nupkgs (feed is at 0.1.4)
Stack: .NET 10 · xUnit · Novell.Directory.Ldap.NETStandard · Microsoft.Data.Sqlite · central
package management. The packages are libraries linked into each consumer — there is no
central auth service. More detail: ZB.MOM.WW.Auth/README.md.
The optional LDAP integration test is opt-in:
ZB_LDAP_IT=1 dotnet test # requires a reachable GLAuth (e.g. a sister repo's infra/glauth)
Roadmap
Verified 2026-07-18 against the Gitea feed (
nuget/query?q=ZB.MOM.WW) and the actualPackageReferences on each consumer's default branch — not from status claims in docs. Per-component detail in eachcomponents/<name>/GAPS.md.
Shipped
- ✅ Auth normalized +
ZB.MOM.WW.Authbuilt (feed:0.1.0–0.1.5) and adopted by all four apps.0.1.5(2026-07-18) pins patched SQLitePCLRaw; mxaccessgw, ScadaBridge and HistorianGateway bumped to it. OtOpcUa stays at0.1.1— it takes noApiKeys/SQLite dependency, so it was never exposed. - ✅ UI Theme normalized +
ZB.MOM.WW.Themebuilt (feed:0.2.0–0.3.1); all four apps on0.3.1. - ✅ Audit —
ZB.MOM.WW.Audit0.1.0on the feed; adopted by all four (DEEP: canonical record everywhere). - ✅ Configuration —
ZB.MOM.WW.Configuration0.1.0on the feed; adopted by all four. - ✅ Observability —
ZB.MOM.WW.Telemetry+.Serilog0.1.0on the feed; adopted by all four. - ✅ Health —
ZB.MOM.WW.Health(+.Akka,.EntityFrameworkCore)0.1.0on the feed; adopted by all four (MapZbHealthwired in each; previously documented as "not yet adopted" — that was stale). - ✅ Galaxy Repository —
ZB.MOM.WW.GalaxyRepository0.2.0on the feed; consumed by HistorianGateway and mxaccessgw (which wiresAddZbGalaxyRepositoryin its Server; previously documented as a pending follow-on — also stale). - ✅ Secrets —
ZB.MOM.WW.Secrets(+.Abstractions,.Ui) published through0.1.3(KEK rotation, 2026-07-18); all four apps still pinned at0.1.2. - ✅ Secrets G-7 clustered replication — built 2026-07-18 at
0.2.0, both fork options:ZB.MOM.WW.Secrets.Replicator.SqlServer(shared store and local-plus-hub) andZB.MOM.WW.Secrets.Replicator.AkkaDotNet(peer-to-peer, partition-tolerant). Verified against a real SQL Server 2022 and an in-process 2-node Akka cluster. Published to the feed 2026-07-18 (restore-verified). - ✅ LocalDb —
ZB.MOM.WW.LocalDb(+.Contracts,.Replication) built at0.1.0and published to the feed 2026-07-18 (restore-verified from a scratch consumer).
Open
- ⬜ Adopt a clustered secrets topology in ScadaBridge / OtOpcUa — config + a shared KEK, no code
change.
0.2.0is on the feed.components/secrets/GAPS.md. - ⬜ Bump the four Secrets consumers
0.1.2→0.1.3/0.2.0(KEK rotation + the same security fix). - ⬜ Clear the pre-existing
AngleSharpNU1902 build break in ScadaBridgeCentralUI.Testsand HistorianGatewayTests— both fail full-solution builds underTreatWarningsAsErrors. - ⬜ Adopt
ZB.MOM.WW.LocalDbin an app — no consumer references it yet. - ⬜ Give LocalDb a
components/localdb/entry; it is the one shared lib whose design lives only indocs/plans/2026-07-17-localdb-design.md. - ⬜ Converge the Auth version pins (OtOpcUa is three minors behind at
0.1.1). - ⬜ Normalize the next cross-cutting component.