fix(localdb): flush the HLC anchor on deregister, bound the sync inbox (0.2.1)
Two adversarial-review findings against 0.2.0. Wire-compatible with 0.1.x/0.2.0 peers: no proto change, no lib_schema_version bump, no on-disk schema change. F4 — DeregisterReplicated destroyed the HLC clock's only durable crash anchor. Clock recovery on open is MAX(hlc_high_water, MAX(oplog.hlc), MAX(row_version.hlc), last_seen_hlc); hlc_high_water is written only by Dispose, and deregistration deletes the table's rows from BOTH ledgers. On a file whose registered tables have all been deregistered, an ungraceful exit (crash, SIGKILL, power loss) therefore left nothing above zero to recover from: the reopened clock restarts from wall-clock time and can re-issue stamps below HLCs peers already hold, which LWW discards as older. The current clock is now flushed inside the deregistration transaction, before the prune, as a separate command so it cannot inflate the "did anything change?" answer. FlushHlcHighWater takes an optional transaction (SQLite rejects a command with no transaction on a connection that has one) and is now MAX-guarded — every writer of that column is flushing a monotonic clock, so a lower value can only be stale. F3 — the sync inbox was Channel.CreateUnbounded, making inbound memory a function of what the PEER sends. The bad case is not hypothetical: a snapshot streams every row of every registered table, needs_snapshot is per-database, and both sides of a pair that enables replication late baseline unconditionally — so a mutual full snapshot of a large dataset (an event log near its storage cap) was buffered whole, in memory, on both nodes at once. The inbox is now bounded at 64 messages with wait-mode backpressure, so the ceiling is ~64 x MaxBatchBytes — a configured number rather than a peer-controlled one — and a full inbox stops draining the gRPC stream, which closes the peer's flow-control window. The pump writes with WriteAsync, not TryWrite: on a full bounded channel TryWrite drops the message, and every inbound message is protocol state. Bounding alone would have deadlocked the very case it was aimed at. The snapshot a node owes was streamed from INSIDE the handshake, before the receive loop existed, so neither side drained its inbox while sending; a mutual snapshot converged only because the inbox was unbounded. The snapshot now runs as the first phase of the outbound task, started alongside the receive loop, so the inbox is drained for the whole life of the session. Interleaving needed no new safety: snapshot rows and deltas both merge through LWW and the snapshot's watermark write is MAX-guarded. Ordering within the outbound lane is preserved (the pump still starts only after the snapshot, from its as-of seq). Per-table needs_snapshot (the finding's optional third part) is NOT done: the flag lives in __localdb_peer_state and narrowing it needs an on-disk schema change, which this fix deliberately avoids. Documented as a follow-up, and the cost is now written down where it is decided — RegisterReplicated(baselineExistingRows) remarks and the README both say that baselining one table re-streams every registered table in both directions, and that high-volume tables should be trimmed before enabling replication late. 188/188 tests pass (was 184). New coverage: the deregistered clock anchor surviving an ungraceful reopen (a second instance over the same file with no Dispose — fails without the flush, recovering at 0), the flush not counting as work done, bounded-inbox backpressure under a slow reader (the pump stalls at the bound and loses nothing, proven by contiguous in-order delivery), and a mutual snapshot over a capacity-1 transport converging — that last one deadlocks and times out against the old handshake-phase snapshot ordering. Version 0.2.1.
This commit is contained in:
+48
-12
@@ -10,7 +10,7 @@ Change capture uses SQLite triggers writing a single HLC-stamped oplog inside th
|
||||
consumer's own write transaction (crash-consistent, non-bypassable). Convergence is
|
||||
last-writer-wins over a hybrid logical clock. All timestamps are UTC.
|
||||
|
||||
## Packages (all `0.2.0`)
|
||||
## Packages (all `0.2.1`)
|
||||
|
||||
| Package | Contents |
|
||||
|---|---|
|
||||
@@ -20,6 +20,27 @@ last-writer-wins over a hybrid logical clock. All timestamps are UTC.
|
||||
|
||||
> Status: **published to the Gitea feed; adopted by ScadaBridge (Phases 1+2) and OtOpcUa (Phase 1).**
|
||||
>
|
||||
> - `0.2.1` (2026-08-14) — two adversarial-review findings against `0.2.0`. Wire-compatible with
|
||||
> `0.1.x`/`0.2.0` peers; no proto change, no `lib_schema_version` bump, no on-disk schema change.
|
||||
> - **`DeregisterReplicated` no longer leaves the HLC without a crash anchor.** Clock recovery on
|
||||
> open is `MAX(hlc_high_water, oplog HLCs, ledger HLCs, last_seen_hlc)`, and `hlc_high_water` was
|
||||
> written only by graceful disposal — while deregistration deletes the table's oplog AND ledger
|
||||
> rows. A crash after a deregister boot could therefore recover the clock below stamps the peer
|
||||
> already holds, and LWW discards those as older. The current clock is now flushed inside the
|
||||
> deregistration transaction, before the prune (and the flush is MAX-guarded, so it can never
|
||||
> regress the anchor).
|
||||
> - **The sync inbox is bounded (64 messages) instead of unbounded**, so inbound memory is capped
|
||||
> at roughly `64 x MaxBatchBytes` rather than "whatever the peer sends" — the bad case being a
|
||||
> mutual full snapshot of a large dataset, buffered whole on both nodes at once. A full inbox now
|
||||
> stops draining the gRPC stream, which closes the peer's flow-control window.
|
||||
> - **A snapshot this node owes is streamed alongside a running receive loop**, not from inside the
|
||||
> handshake. That is what makes bounding safe: previously neither side drained its inbox while
|
||||
> sending, so a mutual snapshot converged only because the inbox was unbounded. Interleaving was
|
||||
> already safe (snapshot rows and deltas both merge through LWW, the watermark write is
|
||||
> MAX-guarded) — only the ordering changed.
|
||||
> - **Documented the cost of late baselining**: the snapshot flag is per-database, so baselining
|
||||
> one table re-streams every registered table, in both directions on a late-enabled pair. Trim
|
||||
> high-volume tables (event logs) before turning replication on late.
|
||||
> - `0.2.0` (2026-08-14) — performance + lifecycle work from the ScadaBridge architectural review.
|
||||
> All of it is wire-compatible with `0.1.x` peers (no proto change, no `lib_schema_version` bump);
|
||||
> the on-disk bookkeeping schema goes to **v2** and is upgraded in place on open.
|
||||
@@ -157,7 +178,9 @@ All options are validated at startup (`ZB.MOM.WW.Configuration` validators).
|
||||
- **Resync** — when a peer's watermark falls below the pruned oplog horizon, it is sent a
|
||||
full snapshot of the registered tables (streamed, keyset-paged) and deltas resume above
|
||||
the snapshot's as-of seq. **Each node decides from its OWN state** whether it owes the
|
||||
peer a snapshot.
|
||||
peer a snapshot. The snapshot is streamed *after* the handshake, alongside a running
|
||||
receive loop, so a node sending one still applies what the peer sends it — a mutual
|
||||
snapshot exchanges under flow control instead of being buffered whole in memory.
|
||||
- **Maintenance** — a background service on **both** roles enforces the oplog row/age caps
|
||||
and prunes expired tombstones every `MaintenanceInterval`.
|
||||
|
||||
@@ -191,11 +214,20 @@ All options are validated at startup (`ZB.MOM.WW.Configuration` validators).
|
||||
every comparison against a real remote write and wins only where the peer has no version
|
||||
of that key at all. Two nodes that baseline the same key both hold HLC `0` and resolve by
|
||||
the node-id tie-break — convergent, but arbitrary as to which content survives.
|
||||
- **The snapshot flag is per-database, not per-table.** Baselining one table flags a resync
|
||||
that streams **every** registered table in full, and on a pair enabling replication late
|
||||
both nodes do that to each other simultaneously — the largest transfer this library
|
||||
performs. Bounded (batch budget out, bounded inbox in), but **purge or trim high-volume
|
||||
tables such as event logs before enabling replication late**, and expect one slow first
|
||||
sync. A per-table flag needs an on-disk schema change and is not in `0.2.x`.
|
||||
- **REAL ±Infinity** is carried in SQLite's JSON dialect on the wire (fine for the
|
||||
SQLite↔SQLite peers this library targets).
|
||||
- `AddZbLocalDb` / `AddZbLocalDbReplication` are **first-registration-wins** (idempotent).
|
||||
- A **mutual simultaneous snapshot** buffers the peer's inbound snapshot in memory while
|
||||
this node is still sending its own (acceptable for the small datasets this lib targets).
|
||||
- **Inbound buffering is bounded, not zero.** The duplex inbox holds up to 64 messages, so
|
||||
peak inbound memory is roughly `64 x MaxBatchBytes` (~128 MB at the 2 MB default, far
|
||||
less in practice — messages are only that large when the sender has that much backlog).
|
||||
Lower `MaxBatchBytes` to lower the ceiling. Beyond the bound the stream stops being
|
||||
drained and the peer is slowed by flow control.
|
||||
|
||||
## Observability
|
||||
|
||||
@@ -231,7 +263,10 @@ provider is wired or the poll threw, so a DB failure never reads as "0 backlog")
|
||||
registered-table digests fail-closed, so a node that drops a table its peer still
|
||||
replicates stops syncing with a schema-mismatch error. Deregister on both nodes, or on
|
||||
neither. Re-enabling later should pass `baselineExistingRows: true`, or the rows that
|
||||
accumulated while capture was off stay invisible to the peer.
|
||||
accumulated while capture was off stay invisible to the peer — but trim high-volume
|
||||
tables first (see Limitations: baselining flags a full, all-tables snapshot). The HLC's
|
||||
durable anchor is flushed in the same transaction, so a crash after a deregister boot
|
||||
cannot bring the clock back below stamps peers already hold.
|
||||
- **On-disk schema upgrades** are applied in place when the file is opened, in one
|
||||
transaction, and every step is safe to re-run. `schema_version` in `__localdb_meta` is
|
||||
local-file state only — it is NOT the wire `lib_schema_version`, so a pair mid-upgrade
|
||||
@@ -246,13 +281,14 @@ provider is wired or the poll threw, so a DB failure never reads as "0 backlog")
|
||||
|
||||
## Testing
|
||||
|
||||
184 tests, all offline / no live dependencies — unit (HLC ordering + restart
|
||||
persistence, trigger SQL generation, LWW decision table, watermark/pruning, digest
|
||||
computation, schema-version upgrade, deregistration idempotency, baseline-floor LWW
|
||||
safety, O(1) depth vs a seeded backlog, byte-budget chunking), integration over two real
|
||||
SQLite DBs on an in-memory (and real) gRPC pipe (bidirectional convergence, delete/update
|
||||
races, offline-accumulate → reconnect, prune → snapshot resync), plus seeded randomized
|
||||
property tests (random interleaved op sequences → both DBs identical after quiesce).
|
||||
188 tests, all offline / no live dependencies — unit (HLC ordering + restart
|
||||
persistence, deregistration's HLC anchor surviving an ungraceful reopen, trigger SQL
|
||||
generation, LWW decision table, watermark/pruning, digest computation, schema-version
|
||||
upgrade, deregistration idempotency, baseline-floor LWW safety, O(1) depth vs a seeded
|
||||
backlog, byte-budget chunking, bounded-inbox backpressure under a slow reader), integration
|
||||
over two real SQLite DBs on an in-memory (and real) gRPC pipe (bidirectional convergence,
|
||||
delete/update races, offline-accumulate → reconnect, prune → snapshot resync, mutual
|
||||
snapshot over a capacity-1 transport), plus seeded randomized property tests (random interleaved op sequences → both DBs identical after quiesce).
|
||||
Family conventions apply (`TreatWarningsAsErrors`).
|
||||
|
||||
Build / test from this directory:
|
||||
|
||||
Reference in New Issue
Block a user