Two adversarial-review findings against 0.2.0. Wire-compatible with 0.1.x/0.2.0 peers:
no proto change, no lib_schema_version bump, no on-disk schema change.
F4 — DeregisterReplicated destroyed the HLC clock's only durable crash anchor. Clock
recovery on open is MAX(hlc_high_water, MAX(oplog.hlc), MAX(row_version.hlc),
last_seen_hlc); hlc_high_water is written only by Dispose, and deregistration deletes the
table's rows from BOTH ledgers. On a file whose registered tables have all been
deregistered, an ungraceful exit (crash, SIGKILL, power loss) therefore left nothing above
zero to recover from: the reopened clock restarts from wall-clock time and can re-issue
stamps below HLCs peers already hold, which LWW discards as older. The current clock is now
flushed inside the deregistration transaction, before the prune, as a separate command so
it cannot inflate the "did anything change?" answer. FlushHlcHighWater takes an optional
transaction (SQLite rejects a command with no transaction on a connection that has one) and
is now MAX-guarded — every writer of that column is flushing a monotonic clock, so a lower
value can only be stale.
F3 — the sync inbox was Channel.CreateUnbounded, making inbound memory a function of what
the PEER sends. The bad case is not hypothetical: a snapshot streams every row of every
registered table, needs_snapshot is per-database, and both sides of a pair that enables
replication late baseline unconditionally — so a mutual full snapshot of a large dataset
(an event log near its storage cap) was buffered whole, in memory, on both nodes at once.
The inbox is now bounded at 64 messages with wait-mode backpressure, so the ceiling is
~64 x MaxBatchBytes — a configured number rather than a peer-controlled one — and a full
inbox stops draining the gRPC stream, which closes the peer's flow-control window. The
pump writes with WriteAsync, not TryWrite: on a full bounded channel TryWrite drops the
message, and every inbound message is protocol state.
Bounding alone would have deadlocked the very case it was aimed at. The snapshot a node
owes was streamed from INSIDE the handshake, before the receive loop existed, so neither
side drained its inbox while sending; a mutual snapshot converged only because the inbox
was unbounded. The snapshot now runs as the first phase of the outbound task, started
alongside the receive loop, so the inbox is drained for the whole life of the session.
Interleaving needed no new safety: snapshot rows and deltas both merge through LWW and the
snapshot's watermark write is MAX-guarded. Ordering within the outbound lane is preserved
(the pump still starts only after the snapshot, from its as-of seq).
Per-table needs_snapshot (the finding's optional third part) is NOT done: the flag lives in
__localdb_peer_state and narrowing it needs an on-disk schema change, which this fix
deliberately avoids. Documented as a follow-up, and the cost is now written down where it
is decided — RegisterReplicated(baselineExistingRows) remarks and the README both say that
baselining one table re-streams every registered table in both directions, and that
high-volume tables should be trimmed before enabling replication late.
188/188 tests pass (was 184). New coverage: the deregistered clock anchor surviving an
ungraceful reopen (a second instance over the same file with no Dispose — fails without the
flush, recovering at 0), the flush not counting as work done, bounded-inbox backpressure
under a slow reader (the pump stalls at the bound and loses nothing, proven by contiguous
in-order delivery), and a mutual snapshot over a capacity-1 transport converging — that
last one deadlocks and times out against the old handshake-phase snapshot ordering.
Version 0.2.1.
The other half of the rebuilt-peer story, and only reachable once 0.1.2 made
back-fill work: with the wiped node repopulated and writing again, its writes
never reached its peer.
last_applied_remote_seq is a watermark in the PEER's seq space, and a rebuilt
peer is a new database whose oplog numbers from 1. Two things then went wrong,
and both had to be fixed — the first alone leaves data stuck:
- The healthy node kept advertising the OLD peer's watermark, and the sending
side seeds its pump from exactly that number, so the rebuilt node skipped its
entire oplog. The watermark (and the observed peer clock) is now reset when
the peer's node id changes. last_acked_seq is deliberately NOT reset: it
describes our own oplog's pruned horizon and is what tells a rebuilt peer it
needs a snapshot — clearing it would turn 0.1.2's back-fill back off.
- A peer claiming to have applied more of our stream than we have ever produced
can only be remembering a previous incarnation of us. That claim is now
rejected in favour of starting from the beginning; LWW makes the re-send
harmless. This mirrors the clamp RecordPeerAckAsync already applies to acks.
Found on the OtOpcUa docker-dev rig, where the healthy node held watermark 10
while the rebuilt peer's oplog was seqs 1-3 and its last_acked stayed 0 — the
peer never accepted a single one. Covered by a test that converges with one
peer, replaces it with a fresh database, and requires a write on the new peer
to arrive: RED before this fix, green after. 149/149 pass.
Version 0.1.3.
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
A converged pair prunes its oplog on ack, so "no oplog rows at all" is the
steady state of a healthy pair — not an anomaly. ComputeSnapshotRequiredAsync
measured the peer's gap against the oldest surviving oplog row and treated an
empty oplog as "no gap possible", which made that steady state the one state
from which a wiped peer could never be healed: it reconnected with an empty
database and a zero watermark, was told it needed no snapshot, and stayed
empty indefinitely.
With an empty oplog the oldest seq we could still stream is one past
last_acked_seq — ack-pruning is the only thing that empties the oplog without
also flagging needs_snapshot, and it deletes exactly seq <= last_acked_seq.
Measuring the gap against that heals the wiped peer and leaves every other
case identical: a converged peer's watermark equals last_acked_seq (no
snapshot), and a pair that has never written anything has both at zero.
Found by the OtOpcUa LocalDb Phase 1 live gate (check 4), which recorded it as
a documented limitation. Covered by a test that reproduces the live-gate
scenario through the real session stack: RED before this fix (the wiped side
never converges, 15 s timeout), green after. 148/148 pass.
Version 0.1.2.
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
Joseph Doherty