The other half of the rebuilt-peer story, and only reachable once 0.1.2 made
back-fill work: with the wiped node repopulated and writing again, its writes
never reached its peer.
last_applied_remote_seq is a watermark in the PEER's seq space, and a rebuilt
peer is a new database whose oplog numbers from 1. Two things then went wrong,
and both had to be fixed — the first alone leaves data stuck:
- The healthy node kept advertising the OLD peer's watermark, and the sending
side seeds its pump from exactly that number, so the rebuilt node skipped its
entire oplog. The watermark (and the observed peer clock) is now reset when
the peer's node id changes. last_acked_seq is deliberately NOT reset: it
describes our own oplog's pruned horizon and is what tells a rebuilt peer it
needs a snapshot — clearing it would turn 0.1.2's back-fill back off.
- A peer claiming to have applied more of our stream than we have ever produced
can only be remembering a previous incarnation of us. That claim is now
rejected in favour of starting from the beginning; LWW makes the re-send
harmless. This mirrors the clamp RecordPeerAckAsync already applies to acks.
Found on the OtOpcUa docker-dev rig, where the healthy node held watermark 10
while the rebuilt peer's oplog was seqs 1-3 and its last_acked stayed 0 — the
peer never accepted a single one. Covered by a test that converges with one
peer, replaces it with a fresh database, and requires a write on the new peer
to arrive: RED before this fix, green after. 149/149 pass.
Version 0.1.3.
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
The lib row still claimed ScadaBridge was the only consumer (verified
2026-07-20); OtOpcUa's Phase 1 adoption merged 2026-07-21. Records 0.1.2 and
the wiped-peer snapshot fix it carries.
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
A converged pair prunes its oplog on ack, so "no oplog rows at all" is the
steady state of a healthy pair — not an anomaly. ComputeSnapshotRequiredAsync
measured the peer's gap against the oldest surviving oplog row and treated an
empty oplog as "no gap possible", which made that steady state the one state
from which a wiped peer could never be healed: it reconnected with an empty
database and a zero watermark, was told it needed no snapshot, and stayed
empty indefinitely.
With an empty oplog the oldest seq we could still stream is one past
last_acked_seq — ack-pruning is the only thing that empties the oplog without
also flagging needs_snapshot, and it deletes exactly seq <= last_acked_seq.
Measuring the gap against that heals the wiped peer and leaves every other
case identical: a converged peer's watermark equals last_acked_seq (no
snapshot), and a pair that has never written anything has both at zero.
Found by the OtOpcUa LocalDb Phase 1 live gate (check 4), which recorded it as
a documented limitation. Covered by a test that reproduces the live-gate
scenario through the real session stack: RED before this fix (the wiped side
never converges, 15 s timeout), green after. 148/148 pass.
Version 0.1.2.
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
Phases 1 and 2 landed on ScadaBridge main together as PR #23
(merge 28ca04d7) on 2026-07-20 and were pushed to origin. The
component row still described both as unmerged branches.
Verified before editing: main..feat/localdb-phase1 and
main..feat/localdb-phase2 are both empty, main == origin/main,
and SiteReplicationActor/ReplicationService have no hits left in
main's tree.
Claude-Session: https://claude.ai/code/session_01BL2Vu1ESDQ9SCN4gVKkdts
The row still said "Adopted by ScadaBridge (Phase 1 only)" and "Phase 2 is NOT
started". Both phases are now complete with a 10-check live gate PASS.
Also records what the row could not have said before: the 0.1.1 parent-
directory boot fix, that ScadaBridge is the library's ONLY consumer (verified —
OtOpcUa, mxgw and HistorianGateway have no LocalDb reference at all), the
deliberate never-registered status of notification_lists/smtp_configurations
and why, and the three operational constraints an operator has to know
(stop/start a pair together, the TombstoneRetention resurrection bound, and
MaxBatchSize batching by row count against a 4 MB gRPC cap).
Replication remains default-OFF and no production deployment has it enabled.
Claude-Session: https://claude.ai/code/session_01BL2Vu1ESDQ9SCN4gVKkdts
Closes criterion 5 of ScadaBridge/docs/known-issues/2026-07-20-localdb-disk-io-
error-under-load.md §9: Phase 1 shipped only correctness and convergence tests,
so nothing exercised many pooled per-operation connections writing a REGISTERED
table (capture trigger + zb_hlc_next UDF) concurrently — which is exactly the
shape the ScadaBridge site nodes run under load.
8 writers x 250 inserts, one fresh pooled connection per operation, with
concurrent readers churning the pool and the WAL index throughout. Asserts both
that every write lands and that the oplog captured each exactly once, so a
trigger that dropped or double-counted under contention fails it.
Written during the 2026-07-20 disk-I/O incident investigation and left
uncommitted; the incident itself was root-caused to host-side sqlite3 access on
a bind-mounted WAL database, not a library defect.
Claude-Session: https://claude.ai/code/session_01BL2Vu1ESDQ9SCN4gVKkdts
SQLite creates the database FILE on demand but never its parent DIRECTORY, and
SqliteLocalDb opens the file eagerly in its constructor — so a configured path
under a not-yet-existing directory was a hard startup failure ("SQLite Error
14: unable to open database file") rather than a first run that just worked.
Relative defaults like ./data/app.db hit it routinely; a containerized consumer
escapes only when a volume mount happens to create the directory first.
ScadaBridge found this during LocalDb Phase 2 and worked around it app-side
(SiteLocalDbDirectory.Ensure). Every other consumer still had the gap, so the
guarantee belongs here.
The standing objection to a library that creates directories is that it can
mask a mis-typed path by silently starting an empty database. That does not
apply: SQLite ALREADY does exactly that for a mis-typed file NAME. Refusing to
create the directory protects nothing — it only makes the two halves of one
path behave inconsistently and turns the directory half into an opaque error.
A creation failure (permissions, read-only mount) now throws an
InvalidOperationException naming the directory and the cause, instead of
surfacing as SQLite error 14. That is a change of exception type, but only on a
path that already failed hard.
Both behaviours are pinned and verified to fail with the call removed.
147 tests pass, 0 warnings. Published to the Gitea feed at 0.1.1.
Claude-Session: https://claude.ai/code/session_01BL2Vu1ESDQ9SCN4gVKkdts
Environment.SetEnvironmentVariable/GetEnvironmentVariable mutate the
process-wide environ; concurrent setenv/getenv across parallel xunit test
classes is a documented crash/flake source on glibc even with unique
variable names. Group the six classes that touch process env vars into a
DisableParallelization=true collection so they run serially against each
other while everything else keeps running in parallel.
Was 'no app adoption yet'. Now records the honest state: ScadaBridge site nodes
adopted Phase 1 on an UNMERGED branch, replication wired but default-OFF and
live-proven on the docker rig's site-a pair only, Phase 2 not started, no other
app adopted, no production deployment replicating.
Claude-Session: https://claude.ai/code/session_01BL2Vu1ESDQ9SCN4gVKkdts
Fixes scadaproj#2. ConfirmDeleteModal used the bare `modal` class and the
Secrets.Ui RCL shipped no CSS at all, silently depending on the host having
no opinion about that selector. Bootstrap 5 does: `.modal { display: none }`
made the delete modal permanently invisible on every Bootstrap host
(ScadaBridge, OtOpcUa, HistorianGateway — confirmed live on the first two)
while every @onclick handler kept working. Only MxGateway, the lone
Bootstrap-free host, could ever have rendered it.
Fix, entirely inside the RCL so a package bump repairs every host with no
host-side changes:
- class vocabulary renamed to zb-secrets-modal / -backdrop / -card / -title
so no host framework selector can match the elements;
- the component emits its own <style> block alongside the markup. Scoped
.razor.css was deliberately NOT used: three of the four family hosts never
link a scoped-CSS bundle, so isolation CSS would silently fail to load —
the exact defect class being fixed. Theme tokens with fallbacks keep the
card legible even on a host without the kit stylesheet.
- data-testids unchanged, so existing Playwright gates and bUnit tests keep
their selectors.
New bUnit pins: no Bootstrap-reserved class name appears in the rendered
markup, and the component ships a style block that both positions the
overlay and gives it a display mode. Suite 182 pass / 0 fail / 0.2.3.
Claude-Session: https://claude.ai/code/session_01BL2Vu1ESDQ9SCN4gVKkdts
Joseph Doherty