fix(localdb): flush the HLC anchor on deregister, bound the sync inbox (0.2.1)

Two adversarial-review findings against 0.2.0. Wire-compatible with 0.1.x/0.2.0 peers:
no proto change, no lib_schema_version bump, no on-disk schema change.

F4 — DeregisterReplicated destroyed the HLC clock's only durable crash anchor. Clock
recovery on open is MAX(hlc_high_water, MAX(oplog.hlc), MAX(row_version.hlc),
last_seen_hlc); hlc_high_water is written only by Dispose, and deregistration deletes the
table's rows from BOTH ledgers. On a file whose registered tables have all been
deregistered, an ungraceful exit (crash, SIGKILL, power loss) therefore left nothing above
zero to recover from: the reopened clock restarts from wall-clock time and can re-issue
stamps below HLCs peers already hold, which LWW discards as older. The current clock is now
flushed inside the deregistration transaction, before the prune, as a separate command so
it cannot inflate the "did anything change?" answer. FlushHlcHighWater takes an optional
transaction (SQLite rejects a command with no transaction on a connection that has one) and
is now MAX-guarded — every writer of that column is flushing a monotonic clock, so a lower
value can only be stale.

F3 — the sync inbox was Channel.CreateUnbounded, making inbound memory a function of what
the PEER sends. The bad case is not hypothetical: a snapshot streams every row of every
registered table, needs_snapshot is per-database, and both sides of a pair that enables
replication late baseline unconditionally — so a mutual full snapshot of a large dataset
(an event log near its storage cap) was buffered whole, in memory, on both nodes at once.
The inbox is now bounded at 64 messages with wait-mode backpressure, so the ceiling is
~64 x MaxBatchBytes — a configured number rather than a peer-controlled one — and a full
inbox stops draining the gRPC stream, which closes the peer's flow-control window. The
pump writes with WriteAsync, not TryWrite: on a full bounded channel TryWrite drops the
message, and every inbound message is protocol state.

Bounding alone would have deadlocked the very case it was aimed at. The snapshot a node
owes was streamed from INSIDE the handshake, before the receive loop existed, so neither
side drained its inbox while sending; a mutual snapshot converged only because the inbox
was unbounded. The snapshot now runs as the first phase of the outbound task, started
alongside the receive loop, so the inbox is drained for the whole life of the session.
Interleaving needed no new safety: snapshot rows and deltas both merge through LWW and the
snapshot's watermark write is MAX-guarded. Ordering within the outbound lane is preserved
(the pump still starts only after the snapshot, from its as-of seq).

Per-table needs_snapshot (the finding's optional third part) is NOT done: the flag lives in
__localdb_peer_state and narrowing it needs an on-disk schema change, which this fix
deliberately avoids. Documented as a follow-up, and the cost is now written down where it
is decided — RegisterReplicated(baselineExistingRows) remarks and the README both say that
baselining one table re-streams every registered table in both directions, and that
high-volume tables should be trimmed before enabling replication late.

188/188 tests pass (was 184). New coverage: the deregistered clock anchor surviving an
ungraceful reopen (a second instance over the same file with no Dispose — fails without the
flush, recovering at 0), the flush not counting as work done, bounded-inbox backpressure
under a slow reader (the pump stalls at the bound and loses nothing, proven by contiguous
in-order delivery), and a mutual snapshot over a capacity-1 transport converging — that
last one deadlocks and times out against the old handshake-phase snapshot ordering.

Version 0.2.1.
This commit is contained in:
Joseph Doherty
2026-08-14 23:34:55 -04:00
parent 1ce0c5955f
commit 522cfbfda6
10 changed files with 376 additions and 43 deletions
+48 -12
View File
@@ -10,7 +10,7 @@ Change capture uses SQLite triggers writing a single HLC-stamped oplog inside th
consumer's own write transaction (crash-consistent, non-bypassable). Convergence is
last-writer-wins over a hybrid logical clock. All timestamps are UTC.
## Packages (all `0.2.0`)
## Packages (all `0.2.1`)
| Package | Contents |
|---|---|
@@ -20,6 +20,27 @@ last-writer-wins over a hybrid logical clock. All timestamps are UTC.
> Status: **published to the Gitea feed; adopted by ScadaBridge (Phases 1+2) and OtOpcUa (Phase 1).**
>
> - `0.2.1` (2026-08-14) — two adversarial-review findings against `0.2.0`. Wire-compatible with
> `0.1.x`/`0.2.0` peers; no proto change, no `lib_schema_version` bump, no on-disk schema change.
> - **`DeregisterReplicated` no longer leaves the HLC without a crash anchor.** Clock recovery on
> open is `MAX(hlc_high_water, oplog HLCs, ledger HLCs, last_seen_hlc)`, and `hlc_high_water` was
> written only by graceful disposal — while deregistration deletes the table's oplog AND ledger
> rows. A crash after a deregister boot could therefore recover the clock below stamps the peer
> already holds, and LWW discards those as older. The current clock is now flushed inside the
> deregistration transaction, before the prune (and the flush is MAX-guarded, so it can never
> regress the anchor).
> - **The sync inbox is bounded (64 messages) instead of unbounded**, so inbound memory is capped
> at roughly `64 x MaxBatchBytes` rather than "whatever the peer sends" — the bad case being a
> mutual full snapshot of a large dataset, buffered whole on both nodes at once. A full inbox now
> stops draining the gRPC stream, which closes the peer's flow-control window.
> - **A snapshot this node owes is streamed alongside a running receive loop**, not from inside the
> handshake. That is what makes bounding safe: previously neither side drained its inbox while
> sending, so a mutual snapshot converged only because the inbox was unbounded. Interleaving was
> already safe (snapshot rows and deltas both merge through LWW, the watermark write is
> MAX-guarded) — only the ordering changed.
> - **Documented the cost of late baselining**: the snapshot flag is per-database, so baselining
> one table re-streams every registered table, in both directions on a late-enabled pair. Trim
> high-volume tables (event logs) before turning replication on late.
> - `0.2.0` (2026-08-14) — performance + lifecycle work from the ScadaBridge architectural review.
> All of it is wire-compatible with `0.1.x` peers (no proto change, no `lib_schema_version` bump);
> the on-disk bookkeeping schema goes to **v2** and is upgraded in place on open.
@@ -157,7 +178,9 @@ All options are validated at startup (`ZB.MOM.WW.Configuration` validators).
- **Resync** — when a peer's watermark falls below the pruned oplog horizon, it is sent a
full snapshot of the registered tables (streamed, keyset-paged) and deltas resume above
the snapshot's as-of seq. **Each node decides from its OWN state** whether it owes the
peer a snapshot.
peer a snapshot. The snapshot is streamed *after* the handshake, alongside a running
receive loop, so a node sending one still applies what the peer sends it — a mutual
snapshot exchanges under flow control instead of being buffered whole in memory.
- **Maintenance** — a background service on **both** roles enforces the oplog row/age caps
and prunes expired tombstones every `MaintenanceInterval`.
@@ -191,11 +214,20 @@ All options are validated at startup (`ZB.MOM.WW.Configuration` validators).
every comparison against a real remote write and wins only where the peer has no version
of that key at all. Two nodes that baseline the same key both hold HLC `0` and resolve by
the node-id tie-break — convergent, but arbitrary as to which content survives.
- **The snapshot flag is per-database, not per-table.** Baselining one table flags a resync
that streams **every** registered table in full, and on a pair enabling replication late
both nodes do that to each other simultaneously — the largest transfer this library
performs. Bounded (batch budget out, bounded inbox in), but **purge or trim high-volume
tables such as event logs before enabling replication late**, and expect one slow first
sync. A per-table flag needs an on-disk schema change and is not in `0.2.x`.
- **REAL ±Infinity** is carried in SQLite's JSON dialect on the wire (fine for the
SQLite↔SQLite peers this library targets).
- `AddZbLocalDb` / `AddZbLocalDbReplication` are **first-registration-wins** (idempotent).
- A **mutual simultaneous snapshot** buffers the peer's inbound snapshot in memory while
this node is still sending its own (acceptable for the small datasets this lib targets).
- **Inbound buffering is bounded, not zero.** The duplex inbox holds up to 64 messages, so
peak inbound memory is roughly `64 x MaxBatchBytes` (~128 MB at the 2 MB default, far
less in practice — messages are only that large when the sender has that much backlog).
Lower `MaxBatchBytes` to lower the ceiling. Beyond the bound the stream stops being
drained and the peer is slowed by flow control.
## Observability
@@ -231,7 +263,10 @@ provider is wired or the poll threw, so a DB failure never reads as "0 backlog")
registered-table digests fail-closed, so a node that drops a table its peer still
replicates stops syncing with a schema-mismatch error. Deregister on both nodes, or on
neither. Re-enabling later should pass `baselineExistingRows: true`, or the rows that
accumulated while capture was off stay invisible to the peer.
accumulated while capture was off stay invisible to the peer — but trim high-volume
tables first (see Limitations: baselining flags a full, all-tables snapshot). The HLC's
durable anchor is flushed in the same transaction, so a crash after a deregister boot
cannot bring the clock back below stamps peers already hold.
- **On-disk schema upgrades** are applied in place when the file is opened, in one
transaction, and every step is safe to re-run. `schema_version` in `__localdb_meta` is
local-file state only — it is NOT the wire `lib_schema_version`, so a pair mid-upgrade
@@ -246,13 +281,14 @@ provider is wired or the poll threw, so a DB failure never reads as "0 backlog")
## Testing
184 tests, all offline / no live dependencies — unit (HLC ordering + restart
persistence, trigger SQL generation, LWW decision table, watermark/pruning, digest
computation, schema-version upgrade, deregistration idempotency, baseline-floor LWW
safety, O(1) depth vs a seeded backlog, byte-budget chunking), integration over two real
SQLite DBs on an in-memory (and real) gRPC pipe (bidirectional convergence, delete/update
races, offline-accumulate → reconnect, prune → snapshot resync), plus seeded randomized
property tests (random interleaved op sequences → both DBs identical after quiesce).
188 tests, all offline / no live dependencies — unit (HLC ordering + restart
persistence, deregistration's HLC anchor surviving an ungraceful reopen, trigger SQL
generation, LWW decision table, watermark/pruning, digest computation, schema-version
upgrade, deregistration idempotency, baseline-floor LWW safety, O(1) depth vs a seeded
backlog, byte-budget chunking, bounded-inbox backpressure under a slow reader), integration
over two real SQLite DBs on an in-memory (and real) gRPC pipe (bidirectional convergence,
delete/update races, offline-accumulate → reconnect, prune → snapshot resync, mutual
snapshot over a capacity-1 transport), plus seeded randomized property tests (random interleaved op sequences → both DBs identical after quiesce).
Family conventions apply (`TreatWarningsAsErrors`).
Build / test from this directory: