WP3.3 of the ScadaBridge architectural-review remediation. Six changes, all
wire-compatible with 0.1.x peers — no proto change, no lib_schema_version bump. The
on-disk bookkeeping schema goes to v2 and is upgraded in place when a file is opened.
DeregisterReplicated(table). There was no way to stop capturing once a file had been
registered, so a node that does not replicate kept paying three triggers inside every
write transaction, appending to an oplog nothing would ever drain. Drops the triggers and
prunes the table's oplog + row-version rows in one transaction; idempotent, and
deliberately usable on a table this process never registered, since stale triggers left by
an EARLIER build are the primary case. Deregistration must be symmetric across a pair, and
already is enforced: the handshake compares registered-table digests fail-closed, so a node
that drops a table its peer still replicates stops syncing loudly rather than diverging.
RegisterReplicated(table, baselineExistingRows: true). Closes the late-opt-in hole: the
snapshot streamer pages from __localdb_row_version, which only triggers populate, so a node
that started replicating LATE never sent its pre-existing rows — silently, forever. Seeding
alone would not have fixed it, because a node whose oplog starts at seq 1 presents no gap at
the handshake and is told it needs no snapshot; the seed therefore also flags needs_snapshot,
and the existing resync machinery carries the rows. The floor is HLC 0 stamped with this
node's id: every genuine HLC is a UTC millisecond shifted left 16 bits and so is strictly
greater, which makes a baselined row lose every comparison against a real remote write and
win only where the peer has no version of that key at all. Two nodes baselining the same key
both hold 0 and resolve by the node-id tie-break — convergent, and nothing better is
available for two rows of unordered antiquity. Opt-in, so the documented CDC contract is
unchanged unless asked. Pruning the ledger on deregister is safe precisely because
re-registration can re-baseline; the two APIs are a pair.
O(1) backlog depth. GetOplogDepth ran COUNT(*) over the entire unacked backlog, once a
minute from maintenance AND on every metrics scrape — a full scan of a million rows exactly
when a node is already behind. Now MAX(seq) - MAX(last_acked_seq, MIN(seq) - 1), floored at
0: three b-tree seeks, measured at 58 VM steps against 100,020 for the count on a 50k
backlog. Exact, because seq is AUTOINCREMENT and every prune path removes a prefix (ack
pruning by watermark, the row cap by count, the age cap by an HLC cutoff that rises with
seq). The MIN clamp is what keeps it exact after a cap prune has moved the horizon above a
stalled watermark — without it the caps would re-flag on every tick forever.
Byte-budget batching. MaxBatchSize was the only cap on an outbound message, which made it a
proxy for a byte budget that consumers had to tune against their widest row: ScadaBridge
pinned it to 16 because 70 KB config_json times the 500 default is ~35 MB against a 4 MB
gRPC limit. MaxBatchBytes (default 2 MB) now bounds delta and snapshot messages by summed
serialized size, with the row count demoted to a secondary cap. A row larger than the budget
is sent alone rather than stalling the stream on it. Since more, smaller messages would
otherwise mean more apply transactions and more acks, inbound delta batches and acks ALREADY
queued are coalesced — one transaction and one ack per burst, bounded by the sender's own
caps, and a no-op under a trickle.
LwwApplier prepares each statement once. It went through the generic transaction façade,
which builds a fresh SqliteCommand per call: three SQL parses per applied row, on the path
that drains a reconnect backlog. Statements are now held for the batch and rebound.
Dropped __localdb_oplog_hlc. It indexed (table_name, pk_json, hlc), which no query has ever
used — every oplog access path is by seq or a full hlc-predicate delete — while costing a
b-tree insert inside every captured consumer write.
184/184 tests pass (was 149). New coverage: schema upgrade drops the index and preserves
identity/watermarks/data, deregistration idempotency and trigger absence and cross-process
cleanup, baseline seeding at the floor with LWW safety proved through the real applier,
depth asserted equal to the counting query across every prune scenario plus a constant-time
assertion over a 200k backlog, byte-budget chunking including the one-giant-row edge on both
the delta and snapshot paths, prepare-once statement counting, and coalescing that must not
swallow the message that ends its run.
Version 0.2.0.
The other half of the rebuilt-peer story, and only reachable once 0.1.2 made
back-fill work: with the wiped node repopulated and writing again, its writes
never reached its peer.
last_applied_remote_seq is a watermark in the PEER's seq space, and a rebuilt
peer is a new database whose oplog numbers from 1. Two things then went wrong,
and both had to be fixed — the first alone leaves data stuck:
- The healthy node kept advertising the OLD peer's watermark, and the sending
side seeds its pump from exactly that number, so the rebuilt node skipped its
entire oplog. The watermark (and the observed peer clock) is now reset when
the peer's node id changes. last_acked_seq is deliberately NOT reset: it
describes our own oplog's pruned horizon and is what tells a rebuilt peer it
needs a snapshot — clearing it would turn 0.1.2's back-fill back off.
- A peer claiming to have applied more of our stream than we have ever produced
can only be remembering a previous incarnation of us. That claim is now
rejected in favour of starting from the beginning; LWW makes the re-send
harmless. This mirrors the clamp RecordPeerAckAsync already applies to acks.
Found on the OtOpcUa docker-dev rig, where the healthy node held watermark 10
while the rebuilt peer's oplog was seqs 1-3 and its last_acked stayed 0 — the
peer never accepted a single one. Covered by a test that converges with one
peer, replaces it with a fresh database, and requires a write on the new peer
to arrive: RED before this fix, green after. 149/149 pass.
Version 0.1.3.
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
A converged pair prunes its oplog on ack, so "no oplog rows at all" is the
steady state of a healthy pair — not an anomaly. ComputeSnapshotRequiredAsync
measured the peer's gap against the oldest surviving oplog row and treated an
empty oplog as "no gap possible", which made that steady state the one state
from which a wiped peer could never be healed: it reconnected with an empty
database and a zero watermark, was told it needed no snapshot, and stayed
empty indefinitely.
With an empty oplog the oldest seq we could still stream is one past
last_acked_seq — ack-pruning is the only thing that empties the oplog without
also flagging needs_snapshot, and it deletes exactly seq <= last_acked_seq.
Measuring the gap against that heals the wiped peer and leaves every other
case identical: a converged peer's watermark equals last_acked_seq (no
snapshot), and a pair that has never written anything has both at zero.
Found by the OtOpcUa LocalDb Phase 1 live gate (check 4), which recorded it as
a documented limitation. Covered by a test that reproduces the live-gate
scenario through the real session stack: RED before this fix (the wiped side
never converges, 15 s timeout), green after. 148/148 pass.
Version 0.1.2.
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
Closes criterion 5 of ScadaBridge/docs/known-issues/2026-07-20-localdb-disk-io-
error-under-load.md §9: Phase 1 shipped only correctness and convergence tests,
so nothing exercised many pooled per-operation connections writing a REGISTERED
table (capture trigger + zb_hlc_next UDF) concurrently — which is exactly the
shape the ScadaBridge site nodes run under load.
8 writers x 250 inserts, one fresh pooled connection per operation, with
concurrent readers churning the pool and the WAL index throughout. Asserts both
that every write lands and that the oplog captured each exactly once, so a
trigger that dropped or double-counted under contention fails it.
Written during the 2026-07-20 disk-I/O incident investigation and left
uncommitted; the incident itself was root-caused to host-side sqlite3 access on
a bind-mounted WAL database, not a library defect.
Claude-Session: https://claude.ai/code/session_01BL2Vu1ESDQ9SCN4gVKkdts
SQLite creates the database FILE on demand but never its parent DIRECTORY, and
SqliteLocalDb opens the file eagerly in its constructor — so a configured path
under a not-yet-existing directory was a hard startup failure ("SQLite Error
14: unable to open database file") rather than a first run that just worked.
Relative defaults like ./data/app.db hit it routinely; a containerized consumer
escapes only when a volume mount happens to create the directory first.
ScadaBridge found this during LocalDb Phase 2 and worked around it app-side
(SiteLocalDbDirectory.Ensure). Every other consumer still had the gap, so the
guarantee belongs here.
The standing objection to a library that creates directories is that it can
mask a mis-typed path by silently starting an empty database. That does not
apply: SQLite ALREADY does exactly that for a mis-typed file NAME. Refusing to
create the directory protects nothing — it only makes the two halves of one
path behave inconsistently and turns the directory half into an opaque error.
A creation failure (permissions, read-only mount) now throws an
InvalidOperationException naming the directory and the cause, instead of
surfacing as SQLite error 14. That is a change of exception type, but only on a
path that already failed hard.
Both behaviours are pinned and verified to fail with the call removed.
147 tests pass, 0 warnings. Published to the Gitea feed at 0.1.1.
Claude-Session: https://claude.ai/code/session_01BL2Vu1ESDQ9SCN4gVKkdts
Publishes the three ZB.MOM.WW.LocalDb packages to the Gitea feed (restore-verified
from a scratch consumer) and adds the build/push.sh the other shared libs already have.
The status prose across CLAUDE.md, README.md and components/*/GAPS.md had drifted from
reality, so it was re-derived from the feed listing and the actual PackageReferences +
registration calls on each consumer's default branch rather than from prior claims.
Five claims were false: Health "not yet adopted" (all four apps wire MapZbHealth),
GalaxyRepository's mxaccessgw adoption "a follow-on" (its Server wires
AddZbGalaxyRepository), Configuration "not yet pushed", Secrets G-8 "not yet
committed", and Theme pinned at 0.2.0 (all four are on 0.3.1). Every doc also said
"three apps" while HistorianGateway is a fourth consumer of seven libs, and all
eight libraries' test counts were stale (re-ran each suite; all green).
Surfaces one previously unrecorded gap: Secrets source is at 0.1.3 with KEK rotation
committed, but the feed tops out at 0.1.2, so no app can consume rotation until it
is published.
Health and observability divergence tables are labelled historical, not re-verified —
the libraries are proven wired, but per-app probe coverage vs spec was not re-walked.
Claude-Session: https://claude.ai/code/session_01BL2Vu1ESDQ9SCN4gVKkdts
Joseph Doherty